A Fuel Price Mashup: Web Development Goes Entirely Online

I put this mashup together to play with two new online tools -- the Google Mashup Editor (GME) and Yahoo Pipes. If you're interested in the process (and the effort involved), read-on in part 2 below. If you live in Perth, use the mashup!

Part 1: For Those in WA Who Want Cheap Petrol…

I'm not one for driving 50km out of my way to save 5c/L on petrol, but with unleaded approaching $1.50/L it's probably worth scouting out who's offering it cheapest around where you usually fill up anyway.

Fuel Watch has published fuel prices for years, you've probably seen the updates after the news alerting you to the cheapest petrol 'North of the river' (as though Subiaco was next to Mindarie).



Sign in and create a list of suburbs where you usually 'fillerup', then check which of the stations in your local areas have the best prices on the handy Google Map.

Part 2: ...Piping Chocolate into Your Mashup Peanut Butter

Lunchtime last Monday I got my invite to the Google Mashups Editor (GME) beta (Check out Philipp's introduction), that evening I decided to try it out.

In 48 hours this technology test project completely changed my view of web development. I've written mashups before (Live Cricket, Cricket Venues) but they're pretty basic -- static and read-only, with no personalisation. Worse, each took over a week to finish. This time, before I went to bed on Monday I was done. A couple more hours on Tuesday for the 20% stuff and I'm finished. In less than a day. Done.

The Technology
Overview

Development, debugging, hosting and deployment are done entirely online -- no local environment, editors, or builds. I moved between three different computers during the development and needed only to login to Google and Yahoo to have my development environment ready to go. For me this is a revelation. Gone, the time-wasting installation and configuration of a development environment, no finding, changing, and configuring paths and dependencies.

If you're a coder, chances are you remember the first moment that a change in a line of text affected what happened on the computer in front of you. That shift from passive viewer to active participant is what's driven a lot of us into the world of software development. These tools have given me that same feeling of empowerment over technology for the first time in years.

The Peanut Butter: Google Mashups Editor (GME)

The GME is an online development environment that lets you build Web 2.0 style apps based on RSS feeds. Crucially, it hosts your app and lets you define data structures and hosts read / write data sources on an application wide and a per user basis.

In real-life I'm a C# desktop developer, though I've tinkered in web development and I've used the Google Web Toolkit. The WA Petrol Price Map had a development time, from scratch to working prototype, of about 5 hours, probably a couple more hours for the polish.

A Step-By-Step to Creating a Mashup in GME

(Here’s the source code so you can follow along at home)
Google provide excellent sample projects that you can use as templates. Each editor page is essentially HTML, you use 'tags' to add all the goodies. So I start by laying out my page -- a header and footer, and a table to hold the map and lists.

You use gm:template tags to define data structures and how to display them. You also set the editability (if and how to display the add/edit/delete buttons). So next I create a template for the data to store - a list of suburbs w/ descriptions.

With the data structure defined, I add an instance of this to my UI with a gm:list tag, specifying the template and the data source. In this case I want it stored per user, so I use the ${user} reference. Then I add the petrol price list by adding another gm:list, but this time I don't specify a template, and I point the data source to the address of my RSS feed.

Then I add a gm:map tag for my Google map, and tie its data source directly to the petrol price list data -- ${petrolPriceList}. The map has built in support for goecoded fields, so I just need to specify the feed tags for lat / long.

Now I've got a list of user editable suburbs, a list of the cheapest petrol stations for the suburb selected, and a map of all the stations. The only thing missing is interactivity. GME uses gm:handleEvent tags for this. I tie the map and petrol station lists together by adding a 'select' event handler to both (they share the same data source, a select on one now selects the same item on the other.)

Finally, I make selecting a suburb update the RSS query I use to access the petrol prices. This needs JavaScript, so I add a select handleEvent tag to the suburb list and instruct it to execute a JavaScript function that updates the data source based on my suburb selection.

AND THAT'S IT. Done and dusted. I've seen the future of web development, and this is the way forward.

Some Quick GME Tips & Tricks:

• Use Pipes for RSS feed manipulation, especially anything involving regex. Pipe’s visual interface is a thousand times better than writing JavaScript functions.
• Get usage stats with Google Analytics.
• Make sure users can make some use your mashup without signing in. People are reluctant to sign in without knowing what they're getting first.
• Use a sensible title when you create your new project as that's what will appear in the Google powered login page.
• You can copy a project in its entirety. Useful for using a sample project as a template, or renaming an existing project.
• Host your data in Google Spreadsheets and use the Spreadsheets API for a feed to use in your mashups.
• GME caches requests so you don't hammer your feed providers. This can result in stale data. Add a &frequency= parameter to your feed requests to ensure you get timely data. Concatenate part of the date/time for the update rate you need (Monthly? Daily? Hourly?)
• You can store data either per user (using the ${user} data store) or per application (using ${app}). You can set access levels to application wide stored data to public, read-only, or members’ access only. Create a private mashup for family vacations by adding them all as members and cutting off public access.
• The GME creates a Google Code project for each GME project, including a version controlled SVN repository.
• Use JavaScript for access to the Map functions not exposed in the gm:map tag. (like turning on Traffic View, etc).

…and some existing problems:

• The GME really didn't like either IE6 or IE7. I haven't experimented in FF yet.
• Code formatting was wonky at best, and it was very easy to 'accidentally' delete text by using seemingly innocuous key sequences (shift-home deletes a line!).
• This is definitely still in Beta. The generated code works flawlessly, but the editor itself is buggy as hell (which may be why they recently allowed you to edit offline). Luckily, the team is very receptive to comments and bug reports via the support forums.
• Functionality is lacking, but more (by way of new tags) is on the way. With luck they’ll expand it to provide all the GWT functionality, and eventually make this the backend for Google Page Maker.

The Chocolate: Yahoo Pipes

Yahoo Pipes was a revelation. Manipulating feed items to modify and annotate fields, looking up and incorporating information based on feed item fields is the 'hard work' behind much of mashup development -- really any data driven development. Pipes makes this the easy part.

Pipes lets you make any RSS feed your bitch. Join feeds together, kick their contents with a regex boot, annotate them with lookups from other web services (like Google's geocoder), sort, filter, take user inputs -- whatever. This is real power, and it's implemented simply, visually and hosted entirely by Yahoo. No more spending hours writing and hosting a web service that does the same thing, say goodbye caching and bandwidth issues.

Here's how it worked for me.

The FuelWatch feed takes a suburb name as a parameter and returns an RSS feed in the form, [price]: [Station Name] - [Station Address].

Bring on the Pipes. I specify an input parameter (suburb) and join that with the FuelWatch feed address to generate my input feed.

Everything from here-on acts on every item of the feed, so I start by copying the title field into a new address field. Whack it with a regex stick to trim the address from the title field and clean up the address field. The foreach transaction lets me pass this address to the Google Geocoder, adding the result as a 'geocode' section on each feed item. A little more copying and a little more regex -- voila -- I've got glat and glong fields at the root level of my feed, perfect for a mashup.

Check out the completed pipe. Total creation time including registering with Yahoo and figuring out how to use pipes (and re-learning regex)? 2 hours.

Pipe Tips and Tricks

• Use the 'copy' filter to duplicate fields, then you can perform different actions on each copy.
• Use regex to modify each field until it contains exactly what you want.
• The regex control works sequentially, so you can perform multiple changes on the same field one after the other.
• Use the 'foreach' transform to add the results of a web-based lookup based on a field into each feed item (Eg. Add geocoordinates by passing an address to the Google Geocoder. Add photos by passing tags to Flickr).
• Pipes lets you take CSV, RSS, and ATOM data as feed sources, then outputs it as either RSS or JSON.

The Future of Development?

This time last week I'd started a draft on how the move towards Internet applications was raising barriers for entry into the world of software development. On Monday afternoon my perspective changed.

I develop desktop apps in Windows; have done for the last decade. I can see the world moving to web based applications, but before Monday I was staying in the shallows, debating the value of investing the time and infrastructure necessary to develop a real Web 2.0 app.

See, if you want to write a desktop app all you need is a computer and a compiler. Distribution requires little more than an email account. To write a web app (and widgets != applications), you also need a server, database, and bandwidth. Without any server infrastructure, doing things like saving user data is non-trivial. Even if you host it yourself, you need an Internet connection with suitable bandwidth and a 24/7 server. That's a lot of overhead for a curious 12 year old, an experimenting student living on campus, or a consultant developer working on a laptop.

No longer. Google's Mashup Editor provides a framework for developers to define and save 'per user' and 'per application' data structures and hosts whatever application you produce. It is a Good Thing. Used with a data manipulation framework as powerful as Pipes and you don’t need anything but a computer connected to the Internet to develop and deploy a fully interactive web app without even owning a computer, let alone installing a compiler.

Now your development environment moves with you. If you're computer access consists of a computer you share with your family and the workstation at the school library, you can still write fully featured web apps just by logging in.

I'm still only dipping my foot in the deep end of web development, but if Google and Yahoo continue to develop tools like these, it won't be long before anyone with the desire can dive into full scale web development without a second thought.

Update (18/07/07) Read this new article on how to use Pipes to turn this mashup into an even more useful Mapplet.