Building the General Election Hexmaps

In 2019, we built a general election results tracker for Britain Elects and the New Statesman. Since then, we've invested in building the Open Innovations technical platform which we use to create data visualisation sites for our customers and sponsors. For this election, we thought we would see if we could build a new results tracker using this platform. This is a technical blog about how we went about doing that.

Our technical platform is based on some software called Lume. This software builds websites by combining data and templates to create the individual files that define the page, styles and interactivity. Importantly, this is done once, and can be hosted on any web server, requiring no databases or application servers. This is known as static site generation. Preparing the data can happen in advance, via reproducible pipelines, and then when we decide we want to create a new version of the site we simply set off a build.

All of the source data and templates are kept in GitHub. This is a really nice choice because it means we can keep track of changes to data and revert to old versions if we need. GitHub also includes a feature called Actions, which can run jobs in the cloud. We use these to prepare data or build the site, as they can be triggered by events (such as a change in the content of the GitHub repository), or on a schedule.

The site was built using data that we were collecting about votes cast, combined with reference data on candidates, electorate and prior results. These were used to populate our Lume visualisation components to create images such as hex cartograms and waffle charts. As results were announced and the team captured the data, the visualisations were updated and the site re-published.

One of the most powerful features of Lume is the ability to create multiple pages with different data populating the same template. We use this extensively when creating pages for the 650 constituencies. Each showed a unique list of candidates and as soon as results were available presented the winning candidate and headline colours to denote the winning party.

Because the site was only periodically build, Stuart created some front-end JavaScript which would refresh the latest information as it became available. This not only kept the data as up to date as it could be, but meant that if the build failed for any reason we were still presenting live data. This turned out to be really useful on the night!

The missing piece of the puzzle was how to get data in the first place. Most of the Open Innovations team have some software development skills and are well-used to working with raw data directly in git. The team collecting the data, however, were less familiar with the tools, and we'd be asking them to edit data files directly. There was a risk that data quality issues would creep in, so we decided that we needed to provide a more user-friendly experience for capturing results. The was doubly important given that the data capture team would be working into the small hours of the evening as the results were announced.

Luckily, Lume has a CMS component that works directly with GitHub. All we needed to do was design the form and host it somewhere with appropriate password protection. With some brief training, the team were quickly up to speed and able to capture the details we needed.

A brief aside on the process by which they were capturing data. The form included the ability to capture votes for all candidates in each constituency, as well as additional information like turnout, electorate and the time that the declaration was made. We realised that capturing all that data and doing so accurately was a pretty mammoth task, particularly given this would be the middle of the night and results would be coming thick and fast. We decided that the one piece of information we needed to capture is the winning party. This we could do by entering the value 1 as the votes cast for the winning candidate. This would then update the map and tables, enabling the team to go back and fill in missing detail when they had a bit more time.

Overall, we made over 900 updates to GitHub during Friday, compared with below 100 the rest of the week.

We did run into a few problems with the process, one of which was that GitHub actions stopped building on a schedule at some point. We don't know why this was, but it could be down to available capacity or rate limiting on GitHub servers. In the future we would probably run similar jobs on a dedicated environment purely for reliability reasons.

We also ran into some issues with the social media sharing images we created, in that the social sites were not recognising our images, even though they worked in preview sites. Something to look at for the future

Finally, we had one or two bugs emerged during the night. It's testament to the skills of the team and their familiarisation with the platform that they were able to fix them on the fly.

Overall, this proved to be an excellent test of the platform, and like the space race, we've had to stretch ourselves and develop new technologies and better understand the existing ones. This will certainly pay dividends in our future work.