Listing UK Government Organisations
For a side project we wanted an official list of UK Government organisations, their websites, and preferably unique codes for each governmental organisation. As central government isn't our area of expertise, we weren't already aware of such a list so we asked others for advice.
Before we go further, what did we mean by "UK Government organisations"? We probably don't know exactly what we mean - because Government is complicated - but we didn't want "local" things such as Local Authorities, NHS Trusts, GPs, universities etc. Ideally we hoped there would be an official, centrally maintained and updated, register of UK (central) Government organisations. There were lots of suggestions.
Registers, lists, and APIs
Noel Dempsey told us that Parliament's written questions API returns the IDs for the body/department responsible for answering questions but that only has about 34 bodies listed and their IDs are likely specific to Parliament's internal processes.
We were told that there used to be a Register at https://government-organisation.register.gov.uk/records which was archived in March 2021. This Register had 989 entries. It is referenced by the International Aid Transparency Initiative's Organisation Registration Agency as GB-GOR and they say it is "preferable" to GB-GOV (21 organisations; hosted at data.gov.uk) and GB-GOVUK. Given the number of entries, it is also most likely the source for Democracy Club's repository for Government Organisations (CSV and JSON) which David Kane pointed out to us and which was last updated in September 2020.
Audree Fletcher suggested the webpage gov.uk/government/organisation which lists 578 results. Alex Shillemore pointed out that gov.uk/government/organisation doesn't include the Royal Navy, the British Army or the Royal Air Force and we suspect this is because they are part of the Crown rather than the Government - just as Parliament is separate to the "Government". It isn't clear how up-to-date the list is because, for instance, it includes the Disabled People's Employment Corporation which went into liquidation in October 2017 and the General Advisory Committee on Science which closed in April 2017.
Sym Roe had looked at a central list of organisations in 2017 and found no single custodian. Sym shared a link to an openregister repository that was last updated in 2017 (TSV) and mentioned an API at gov.uk/api/organisations. That API currently gives 1130 results and includes closed organisations. Everything in the list of 578 is included in this list of 1130. Paul Downey (MHCLG) mentioned that the analytics_identifier key given in the API results is a good bet for a unique identifier and this seems to be the same ID referred to by the archived GB-GOR Register. It looks as though the slug values may also be unique keys but we don't know how stable they are compared to analytics_identifier e.g. if someone renames Sports Council for Wales into Sport Wales, does it change its slug and what happens to its analytics_identifier?
After attempting to read through a bunch of Ruby code we've established that gov.uk/government/organisation filters the API using the following rules to get from 1130 to 578 entries:
- exclude if
- exclude if
details.govuk_statusis "devolved" and
- exclude if
formatis "Court" or "Civil Service"
- exclude if it is a child organsiation of HM Courts & Tribunals Service
Paul Downey also suggested WhatDoTheyKnow's list of authorities that you can send an FOI request to (WDTK also provide a CSV file although the link seems to time-out). That has 38,395 entries in total and does split them up by some categories e.g. 55 Ministerial departments. Each organisation's page on WhatDoTheyKnow links to the organisation homepage although we're not sure how up-to-date those are either. Interestingly, Sport Wales are listed by WDTK as "Sports Wales" [sic] with a note saying that it was "Formerly known as the Sport Council for Wales" so they could very well be more up-to-date than the gov.uk list/API.
Just as we thought we'd got our heads around organisations, we then discovered that gov.uk also has a list of 621 "Groups" at gov.uk/government/groups. What is the distinction between "Group" and "Organisation"? Presumably someone in Government knows.
Bringing it all together
As we've seen, there are several different lists of UK government organisations.The API seems to be the best bet although gov.uk/government/organisations mentions when an organisation has a separate website (a website not under gov.uk) - the API doesn't tell you this.
To get the list of websites/IDs/name that we needed for our side project, we have created a Github repository which screen-scrapes gov.uk/government/organisations to find all the organisations which have a "separate website" note. We then go to the gov.uk pages for those organisations and extract the "real" website URL for each of them. We then add in the IDs from the API results by matching against the organisation slug. As the "separate website" URLs aren't necessarily current (e.g. we noticed that the Advisory Committees on Justices of the Peace holding page links to a 404 page and HM Inspectorate of Prisons has moved) we have made a list of alternate URLs that was initially generated from gov.uk/government/organisations but which we can manually update to fix broken links. This repository should automatically update on the 1st of each month and produces a cut-down-CSV version of the API results that has been augmented with the alternate URLs.