Dev links #2

  1. XmlHttpRequests via Javascript
  2. HTTP requests using PHP
  3. Base64 encoding/decoding in Javascript
  4. Getting installation date of an app (Android)

I’m attempting to resuscitate my TweetPuller chrome extension to use v. 1.1. of Twitter’s api. So far, I’ve created a PHP script to retrieve the needed access token and calling out to that in order to prevent exposure of my consumer key and/or secret on the client. So far so good. 

Oh and merry Christmas! 🙂

Advertisements

Dev links and sweet finds

  1. Personal Blocklist: An extension from Google that lets you block certain sites. *cough*w3schools*cough*
  2. ThreeDubMedia‘s collection of jQuery plugins for implementing dragging & dropping functionality via jQuery
  3. draggable.js is another simple plugin for jQuery that lets you drag and drop stuff.
  4. yet more plugins that help you drag and drop stuff!

Enjoy! 🙂

Stashing stuff in localStorage and other dev links

  1. Hiding tt-hint when using typeahead: This is useful because when using typeahead, the top hint for a query is on automatically. There’s no option to turn this off so a workaround I found online was to set the font size to 0 and turn off the visibility for the .tt-hint class.
  2. Using transitions: Since I just learned how to apply (basic) transitions, I’ll abuse them for a few weeks and then forget about them. I’ve been bad about providing feedback on actions in my web application so I’ll be applying my new found power liberally. 🙂

Image

In the latest version of CoursePicker on the web (v2), I decided to add a bit of flair to the application by mapping the building numbers to actual building names. By default, the building numbers are provided with sections and for me, it has always been a pain to find a section I’m interested in and then have to go to the UGA Campus Mapping application to find out the building details. So, I set about finding a document that contained the mapping of building numbers to the actual building names. My first hit turned up a service (I think it was Scribd) which wanted me to pay to view the list. Well, don’t be fooled into paying money for it! There’s a UGA Building Index document for free here and I used the free Nitrocloud online pdf conversion service to export the pdf to a csv file. Once I had a csv file, it was trivial to format the file into a valid JSON file. If you’re new to JSON file formats, you want to bookmark JSONLint.com and this JSON Formatter & Validator.

So, for use in my application, I was essentially loading the JSON file from my webserver on every page load and wasn’t “caching” it anywhere. Granted, my web traffic is minuscule but I want to actually learn good practices. So, today, I decided to revisit my naive implementation and rediscovered localStorage. This JSON file is fairly static so I am not too worried about the information being out of date although I may need to specify an expiration date manually (I learned that the expiration date of items in localStorage is really up to the user). There’s lscache by Pamela Fox which uses localStorage but allows you to specify expiration dates, etc. I could also use cookies to store my data, etc. 

I went with lscache because it was easy enough and it works for now. Here’s the localStorage way to store stuff:

if (localStorage.getItem("uga_buildings") === null) {
$.getJSON("/uga_building_names.json", function(data){
var uga_buildings = data;
localStorage.setItem('uga_buildings', JSON.stringify(data));
});
console.log("stored list of uga buildings in local storage.");
}else{
var uga_buildings = JSON.parse(localStorage.getItem("uga_buildings"));
console.log("retrieved list of uga buildings from localStorage.");
}

The emphasized text is emphasized because you can only store strings in localStorage. So, you need to convert your object to a string (JSON.stringify) and when retrieving your object, you convert it back to the original form (JSON.parse).

You can check out the source code for CoursePicker here. As always, bug reports and pull requests are welcome! 🙂

CoursePicker – The Saga Continues …

I’m working on a couple of enhancements to my CoursePicker web application to address some shortcomings of the original application.  On my current backlog are two major items:

  1. Architecting the backend to handle multiple semesters. This enhancement is almost complete but I need to work on automating the process. Because I’m grabbing large CSV files and dumping them into my database, I ran into memory problems on dreamhost and had to use my local computer. A tentative solution is: splitting the larger CSV files and upload separately. This is a high priority item because during the available sections will change pretty frequently.
  2. Redesign the webpage using my pitiful CSS skills. I’ve taken this to mean liberal use of divs and rounded corners. 🙂 This is coming along quite nicely and I’ll post some new screenshots in this post.
  3. Restructure/refactor/consolidate code.

The Backend

The current production version of CoursePicker refreshes the database on a schedule (using cron). A bash script uses curl to grab the Fall 2013 csv file for the Athens campus from the Registrar Reporting site.

Then, this csv file is sorted; first according to the 3rd column which is the 4-letter course abbreviation and then on the section call number.

After the sorting process, the csv file is further processed using the cut [e.g. cut -d’,’ -f3-5 old.csv > new.csv ] command to create a new csv file with the wanted columns (columns 3 – 5). The output of this cut step is then fed into a PHP script which converts the data into a JSON file. Thereafter, this JSON file serves as the source of the course autocomplete field in the application.

Separately, the sorted csv file is then loaded into the database using a Java application which uses the mysql load statement to perform the database population step [e.g. -load data local infile ‘sorted_courses.csv’ into table Table fields terminated by ‘,’ enclosed by ‘\”‘ -]. Note: on Dreamhost, files larger than 6mb must be imported (as opposed to using phpMyAdmin to import the csv file).

Enhancements:

  1. Instead of just getting 1 semester’s worth of courses, the new version will be pulling CSV files from the Athens and Gwinnett campuses. Note, the CSV files of interest begin with course_offering.
  2. Since data from different semesters is being loaded into the same table, I need to ensure that any assumptions (e.g. that no call number is duplicated across Athens and Gwinnett campuses) still hold to prevent any bugs.

The Frontend

I’m limiting my application to the Athens and Gwinnett campus and the CSV files of interest begin with course_offering (it has over 20 fields). So, this frontend needed to be changed in response to this new requirement. However, I’ll describe the current production version’s setup (frontend) first and then describe the enhancements.

When the page is first loaded, a UserSchedule object is created and stored in the $_SESSION object. This UserSchedule is a fancy array which contains a unique id and a list of Section objects.

In the current/live version of CoursePicker, the user can narrow down their course selection through the official UGA requirements OR using a plain ole’ textbox which has autocompletion available. The end goal is: when the user selects a course (XXXX-2345), this value is sent to a controller (a PHP script) to retrieve the list of sections for that particular course. This list of Sections is returned to the View (web page) as a String which is converted to a valid JSON object and available for manipulating via JavaScript.

From here on, it’s trivial to add a Section object to the UserSchedule and there is a lot of DOM addition/insertion/deletion going on via JavaScript (coursepicker.js).

Enhancements:

  1. Allow user to select the campus and semester of interest.
  2. Remove the feature allowing users to drill down and simply allow them to search for courses by entering part of the course name or course number. Typehead.js allows you to specify tokens to be searched again so I’m using the course prefix (e.g. CSCI), the course number (e.g. 1302), and splitting up the abbreviated course name so that “Software Development” becomes “Software” “Development” and each word is now available as a token.”
  3. Breakup coursepicker.js into themes e.g. a script to handle drawing items on the canvas, script to handle DOM modifications, etc
  4. Breakup CSS similarly e.g. CSS that handles styling of my Section divs is one CSS file, styling the Canvas element is handled by another, etc.
  5. Breakup controllers so that one controller handles getting items from the database and another deals with managing the UserSchedule object.
  6. Make the user interface more “friendly”. This is definitely still a WIP (Work In Progress) but I think some changes I have in store will make things more userful. For instance, the live version of CoursePicker displays the list of sections for a chosen Course in a dropdown list. While this is great for courses with a long list of sections, it didn’t help my user plan better i.e. they had to click through all the sections to see the times, etc. Granted, I could alter the dropdown box to also display the times, but that would just be lazy. 🙂 So, I’m displaying each section as a styled div and allowing the user to hide or unhide the list of sections.

For the first frontend enhancement, I used the prefetch option in Twitter’s typeahead.js which caches the retrieved item in localStorage. The downside to this which I discovered was when the user selected a new semester, reloading the page with the new item didn’t work. I finally had to simple clear localStorage each time the user changed the selected item to something different and that forced the json file to be refetched.

$('#courseEntry').typeahead({
 name: 'courses',
 limit: 7,
 prefetch: $('#jsonURL').val(), 
 template: [ 
 '<p class="tt-courseShortname">{{coursePrefix}}-{{courseNumber}}</p>', 
 '<p class="tt-courseName">{{courseName}}</p>' 
 ].join(''), 
 engine: Hogan 
 }).
$('#jsonURL').val() //is a hidden input field which is updated whenever the selection changes

The first bonus for using typeahead.js is the ability to style the autocomplete entries! It’s pretty slick and once I get better at it, I’ll be sure to trick mine out even more. 🙂

The second is the fact that I can specify the actual value returned when the user chooses an item from the autocompete menu. In my case, a single autocomplete entry is composed of the course prefix, course number and the course short name but on selection, the course prefix concatenated to the course number is selected. The immediate benefit is that I can make my autocomplete suggestions much more descriptive (and more userfriendly) without making the application a hacky mess.

The third incentive from switching to typehead.js is the fact that instead of doing this to get the selected value, I can add an event listener to the object to get the value neatly.

The last reason (for me) for using typeahead.js is moving from over 50 LOC to less than 15 LOC to have autocomplete. Up until last week, I hadn’t been able to get typeahead.js to work. Part of this is due to the fact that I kept relying on my old (but working) code to get autocomplete working. However, last week, something changed: I read the README for typeahead.js. I mean I really sat down with it to understand what it was doing and once I did, it clicked. So I went from following the instructions here (https://gist.github.com/rn0/1848844) for remote JSON loading (if you use the typeahead module in Bootstrap 2.X) to something simpler and more elegant.

.on('typeahead:selected',function(obj,datum){
 var courseValue = datum.value; //Gets the XXXX-1234 returned by typeahead.js
 $.ajax({
     type: "POST",
     url: 'path/to/controller',
     data: { action : "getSections", courseEntry : courseValue},
     dataType: "json"
     })
 .done(function(msg){
     console.log("Success!");
 })
 .fail(function(msg){
     console.log(msg + "Error getting sections.");
 });

I didn’t mean for this post to get too long but I’m glad I’m finally semi-documenting how this project works. 🙂 Hopefully, my rewrite will be ready for the Spring semester craziness in a few weeks. As before, I’m putting the latest changes on Github but I’m aware that the readme sucks and doesn’t actually help you setup a replica of this app. However, this post should get you on the road to doing this yourself if you choose.

You can check out the work-in-progress here: the basic functionality (adding/deleting sections) has been added but the data is stale because I’m still testing.

Screenshot from 2013-12-13 23:59:58

The Last Hurrah

The end of the semester is nigh as I have my last final in a few hours. Naturally, I’m not studying but blogging my anxiety away. As of 12/11/2013, I will be done with my CS curriculum which means no more CS classes are required for my degree but I have some non-CS classes to take through out next year.

I managed to talk myself out of taking any extra CS classes. Instead, I will focus on learning other skills and working on my “main” web apps/ projects.

  • DawgTransit.com was a site I started over a year ago to help people get familiar with UGA bus routes. The code base isn’t public and that’s deliberate because it’s quite ugly. 🙂 The back-end is written in PHP and I mostly wrote it as I went i.e. no real backend design went into it. So, eventually, I want to rewrite the backend using OO principles and work on a more sophisticated routing algorithm that takes into account the distance between bus stops in recommending routes for users.
  • CoursePicker is my next ‘big’ web app which is also written in PHP. The source code for this is available online and it’s less hacky than Dawgtransit i.e. I thought about the back-end design just as much as the frontend and I’m pretty happy with the result. However, the site was built with the Fall 2013 semester data only. An immediate improvement I want to roll out is: allowing the user to select the semester based on either the Athens & Gwinnett campuses (Griffin campus will have to wait). I’m currently working on populating my database with the data automatically.
  • Health Inspection Records is my latest baby (native Android app) which I built with data from just 10 counties. I obtained the data through scraping the html pages that contained the inspection reports into a database. A future improvement for this app is including more counties, but the challenge is that other counties aren’t publishing their reports as simple/scrapeable html pages. So, I would need to pick my next batch of counties to target carefully. In doing some user testing for my app, I realized that the UX of the app needs work so that is one immediate improvement I could begin work on. An example of this is I observed a user having difficulty submitting their restaurant selection.Why? Well, I assumed that the user’s workflow would be: start typing restaurant name, pick out the full name from the autocomplete list and press the big blue button to submit. Well, after typing the restaurant name, the user kept selecting “done” on the onscreen keyboard (Android) and I did not map the “done” softkey to any actions in my app! For the record, the big blue button is not named “Submit” so for all I know that renaming the button to “Submit” may help the user know what next to do but there are 2 big blue buttons on that screen so that might introduce more problems. I could go on, but the point is: I’ve got some work to do. 🙂
  • DawgDictionary /UGA Slang was another simple web site I threw together which scraped the data from http://english.uga.edu/def/ into a database in order to present the info in a more user friendly way. Well, I recently had the idea to make it into an Android app (a dictionary of sorts) and I’ve begun work on this. Screenshot_2013-12-09-19-14-23 Screenshot_2013-12-09-19-14-34Nowhere near done, but it’s in the works. Also, the “browse” function on the website hasn’t implemented and I’m not too happy with the website’s look so a redesign will be in the cards.
  • TweetPuller – I made a Chrome extension using Twitter’s old api. Shortly after, Twitter started requiring all apps using twitter to move towards oauth authentication and I never got around to doing this so my extension doesn’t work anymore. 😦 My todo for this extension is to make the extension comply with Twitter’s new rules and it’ll be good to go.

My point in enumerating these projects is really to reinforce to myself that there’s a lot of work to be done and to not feel guilty for not taking any CS classes in the Spring 2014 semester. Course credits are expensive! (~ $255 per credit hour) and if I can educate myself without paying anyone, that’s a win in my book.

Google App Engine PHP Preview

I signed up for this a while back and never quite got around to taking advantage of it. Well, I went through the HelloWorld tutorial which took longer than I care to admit. One of the things that tripped me up was the php executable path. I didn’t realize it needed to be specified when launching the app (locally). Here’s what it looks like:

me:~$ python ~/google_appengine/dev_appserver.py –php_executable_path=/home/me/php-5.4.15/installdir/bin/php-cgi /home/me/Documents/Coding/PHP/GAEPHP/helloworld/