When visualisations go viral
On the evening of Wednesday 20th March, one petition on the Parliament Petitions website suddenly started to gain momentum. When our hexmap - which shows the geographic distribution of signatures - was shared as a way to track signatures, we had no idea what was about to unfold.
Hex appeal
We first started dabbling in hexmaps during the 2017 General Election, where we wanted to explore ways of creating a visually more equal representation of the UK. Constituencies vary greatly in size and population - for example, many London constituencies look geographically 'small' on a map but might have a large population, with rural areas looking much bigger on a map but perhaps containing half the number of people. When it comes to things like election results, it can visually look very dominant for the leading party on a traditional geographical map. Hexmaps help reduce this visual bias because each hex has the same visual weight.
As we've seen since we made our tool a year ago, many Brexit-related petitions tend to correlate strongly with how constituencies are estimated to have voted in the 2016 Referendum (Hanretty et al 2016).
Credit: ODI Leeds
Open data lets you do more
We can create our hex maps thanks to the simple API provided by the House of Commons Petitions website. They provide JSON breakdowns by constituency and country for every petition. Those always provide the most up-to-date state of the petition and, late on Wednesday night, Stuart (Data Projects, ODI Leeds) set up a simple automated job to download a copy of the JSON every 5 minutes to create an archive. He also made a very basic graph using HTML/CSS that would show the trend live. The fact that the graph was really basic didn't matter; the key thing was having the data stored so that a better job could be done later. He compiled the totals into a simple CSV file which he linked to from the graph page so that others could take the data and do their own visualisations. He then went to sleep.
Thursday morning arrived and the petition was still rising in signatories. As one of the few sites with a ready made map visualisation and a graph (however basic) of the temporal trend, the ODI Leeds website was quickly becoming very popular. We had some useful advice on making the map work better on mobile devices and had implemented that before 8am. As the petition itself started to go more viral, the main Petitions website was struggling to function. It fell over around 9am.
Dealing with demand
By the start of the work day our website was also becoming less and less responsive under the weight of visits from people sharing it on Twitter. The first thing we needed to do was to change the way our website got updates to the petitions data. Previously, with so few people using our tool, it had been OK to make a request to the petitions website each time someone loaded a petition so that we always had the latest data. Stuart live-edited the code behind the tool so that, for this particularly popular petition, we used the local copies we were saving every 5 minutes. This instantly reduced our load on the main Petitions site down to a maximum of one request every 5 minutes in total regardless of how many people were looking at our site. That put us in a vastly different situation to the main Petitions website where the petition page was making one request for the total every 20 seconds or so. Multiply that by a few hundred thousand people and you've got a lot of requests when your servers are also trying to let people sign the petition.
So, we became an unofficial "mirror" of the total that didn't strain the main Petitions website. Unfortunately, that didn't stop one angry, anonymous, member of the public calling us up to demand that we take down our website because he was convinced that it was our map causing the overwhelming demand on the Petitions website. Thankfully, hundreds of other people realised that we were actually reducing the strain and were directing people to use our graph to monitor the total once they'd signed.
We'd reduced the load on the main Petitions website but our own website was struggling. Stuart likes the bandwidth-footprint of our website to be low so the petition page took around 250kB in total (on Thursday). That is small by the standards of many web pages e.g. the geographic version of the petition uses 5 times more bandwidth. Our server was simply struggling under the shear number of simultaneous requests for resources that make the page (CSS, Javascript, images etc). So, we needed to move requests off our server. We quickly set up a Github repository, dumped copies of some of the resources there, and then updated our hex map to get them from Github pages rather than our own server. This was all done in a matter of minutes and our server quickly got more responsive.
Meanwhile, over on the main Petitions website, they were also making changes to their code to deal with the demand and prioritise the ability of the website to let people sign petitions rather than have up-to-the-minute reporting. That can be seen in the gaps in updates in our CSV file of the 5-minute totals. The code behind the Petitions website is open source and you can see all the changes to their code they made to deal with it.
Open data leads to better visualisations
One of the key things we did was to provide the 5-minute totals as a CSV file and encourage others to use it (as well as providing a mirror of the JSON snapshot). Several people took up the challenge and created their own graphs. Independent Consultant Jenny Shipway has created lots of interesting graphs showing the total, the rate of signing over time, and how the rate has varied by hour of day. You can clearly see how the rate of signing drops dramatically between midnight and 6am UK-time.
Credit: Jenny Shipway
Credit: Jenny Shipway
A few people weren't fans of our visualisation and that is fine too. No single visualisation is going to answer every question or fit everyone's preferences for colours/styles. Our colour scale was picked to be OK for those with colour deficiency but wasn't to everyone's taste. Some people had specific questions that a hex map doesn't make it easy to answer. In those cases we tried to send people to alternative visualisations that did e.g. an ordered table of constituencies, the geographic map, or the growth with time. As we've used data that is open and uses a hex map layout that is open, it is also possible for people to take the building blocks of what we've done and make their own visualisation to suit different needs/uses.
Summary
Since Thursday 21st March, our petitions hex map has had over 400,000 views. Our basic graph showing the growth of that petition has had over 230,000 views. Our website has had more visits in the week or so following the petition than over its whole lifetime before.
Credit: ODI Leeds
When people want something to go viral, they perhaps aren't aware of the snowball effect that takes place. We certainly weren't! But by working in the open, using open data, and using flexible services like GitHub, we were able to respond quickly to not only our benefit (keeping our website running) but to the benefit of others (providing a way for people to watch the signatures rise that didn't break the Petitions website). We share the data that we've made and used, we want to see what others can make with it, and we want to see people improve it. Being radically open makes things better.
