An archive of Bus Open Data

If you've been following our work for a while you'll know that there are some topics we get very animated about. One of those is buses.

We've shared our experiments with bus open data, we've written thoughts on franchising, we've investigated phantom buses, hosted workshops about bus open data, and much more. We think buses are very important.

Thanks to the Bus Services Act, and the subsequent Bus Open Data Service (BODS), getting live bus locations isn't hard. BODS data is perfect for showing people their current bus location on a map. But many of the projects we've worked on have needed archival information about bus locations. For instance, to assess the impact of bus franchising in Manchester, you need data for a period before and after franchising. To look at the effective size of Birmingham, you want to calculate the average real journey times of thousands of bus services over a large period. If we wanted to compare average bus speeds to speed limits in Cambridge, we'd need high-resolution location data over time too.

The Department for Transport don't provide an archive of the location data; every researcher is left to collect their own. Unfortunately, that means you can't have location data before you thought to start archiving. It also means that multiple transport analysis groups are duplicating effort by continuously polling the BODS servers to build their own archives.

A national data library

Earlier in the year, our small team met up for lunch in Leeds. As we were waiting we got chatting about proposals we'd heard for a National Data Library. What would it consist of? How long would it take to create? How much would it cost? Was there a chance it could be hosted outside London? Would it include the datasets we've felt the UK has been lacking for years?

Before our food arrived at the table we had "accidentally" started our own National Data Library by registering a domain name - datalibrary.uk. By the next day we had a simple webpage online asking for feedback on the datasets we and others might like to see. Our top suggestion was a historical archive of Bus Open Data.

Do it yourself

Time matters. Rather than wait for the Department for Transport, central Government, or an official National Data Library to start providing an archive of BODS data, we thought we'd just start building an archive ourselves. We'd make it openly available.

Our colleague Tom told us he had an old computer sitting around, an 8TB SSD, and a strong broadband connection. A quick back-of-the-envelope calculation suggested that that hard drive might hold a year's worth of location data. We would start collecting and we'd have time to figure out what to do about the limits later.

Our first task was to write some scripts to archive the BODS data. We opted to keep things as simple as possible by using shell scripts that run from a cronjob. We created one script to download all the timetable data once a day and another script to download both the SIRI-VM and GTFS-RT live location data every 30 seconds*. Both scripts save the data to a directory structure based on date with filenames that contain datetime stamps. We started collecting on a laptop that we left running.

Next we set up an nginx server on Tom's spare computer. Given the very limited resources from doing this on no budget, this server was going to both collect the data from BODS and serve up the archive. Ideally, if money became available in the future, we could split these two tasks onto separate machines to make sure that the critical archival job wasn't affected by people accessing the archive. But, for now, it'd have to do.

With this web server up-and-running we planned to transfer our temporary archive - 18 days of data - over to it. Unfortunately, a mistyped command meant we accidentally wiped everything we had collected! Eighteen days gone! We were annoyed at ourselves but we had learned a lesson. We started collecting the archive again but now we had another item on our future wish list; have an offsite backup of the entire archive too.

Improving our archive

The archive has been growing almost continuously since June 18th. There is one gap of a few days in early July when a cat knocked the server offline. That's a risk when you are doing things on less than a shoestring. But that incident added another thing to our wish list; have at least two geographically-separated archiving computers so that we can fill in gaps if one experiences downtime.

We've improved the website in various ways over the past month:

• We present calendar charts to let us easily check the health of the archive over time e.g. spot gaps such as the one due to the cat.
• We present an estimate of how many days storage we have left on the disk to help us know how long we have left to address that.
• We've added a task that zips up all the live location data into daily downloads rather than have to download 2880 files per day.
• We set rate limiting and have limited download speeds to try to pre-emptively limit the impact of people setting up inconsiderate scripts/bots that overwhelm the server.

We have lots of ideas for how this could be improved but we'd need time/funding to do them all. Aside from our time, we roughly estimate the setup costs to be: Tom's spare server (let's estimate £500 if it was new), an 8TB disk drive (£500 new), and Tom's internet (say £40/month?). So that'd be roughly £1500 for the first year. For a long-term archive we'd need to add roughly one new 8TB drive every year. And if we wanted more reliability we'd double this. These costs aren't nothing but are far less than the amounts a national body would no doubt spend on a pilot or consultation.

We had a tiny team and no budget but we got something working. It isn't that hard. The question is: what do we do next?

*The live data is published every 10 seconds and this can mean that we are missing bus locations that are fired off between our 30 second polling. However three times as fast polling would give us a third of the time to address our storage limits. It is a compromise for now and could be addressed with a bit of money to buy more hard disk space.