Scotland’s Open Data, February 2019. An update.

Note: this blog post first appeared on codethecity.co.uk in February 2019 and has been archived here with a redirect from the original URL.

Scotland’s provision of open data may be slowly improving, but it is a long way behind the rest of the UK. In my most recent trawl through websites and portals I found a few minor improvements, which are positive, but progress is too slow; some data providers are slipping backwards; and most others are still ignoring the issue altogether. Now is the time for the Scottish Government to act to fix this drag on the Scottish economy and society, and stop inhibiting innovation.

Latest review

Over the last week, I have conducted yet another trawl of Scottish Open Data websites and portals. I keep this updated on this Github Repo.  I’ve carried out this research without assistance, in my own time. The review could be more comprehensive, frequent and robust if I was supported to do it.

This work builds on previous pieces of research I’ve carried out and articles that I have written. Recently, I’ve created an index of those blog posts here as much for my own convenience of finding and linking to them as anything.

During this latest trawl, I’ve tried to better capture the wide spread of Scottish Government departments, agencies, non-departmental public bodies, health boards, local authorities, health and social care partnerships and academic institutions;  and assess each sector using quite conservative measures.

The output of that, as we will see below, does not paint a good picture of Scotland’s performance, despite a few very good examples of people doing good work despite a clear policy gap.

Let us look at this sector by sector, following the list of findings here.

Local Authorities

Of Scotland’s 32 local authorities, only 19 produce open data of any kind.  This group uses a mixture of open data portals (10), web landing pages (7) and GIS systems (2). This leaves 13 who produce no open data whatsoever.

Those 19 councils (ignoring the other 13) produce a total of 731 datasets, giving a mean for the group of 38 and a median of 17 datasets. This total is only six more than I found three months ago, despite Dumfries and Galloway launching a new portal with 33 datasets !

Also, stagnation is a real issue. For example, it is worth noting once again that while Edinburgh produces an impressive 234 open data sets, only five of those have been updated in the last six months, and 228 of them date from 2014-2017.  While there is a value in retaining historic data ( allowing comparisons, trends etc to be analysed), the value of data which is not being updated diminishes rapidly.

When I ran the OD programme for Aberdeen City Council (which, like all Scottish councils, is a unitary authority), based on some back-of-the-envelope calculations I reckoned that we could reasonably expect to have about 250 data sets. So, if each of the 32 did the same, as we would expect, then we’d have 8,000 datasets from local authorities alone. This puts the 731 current figure into perspective.

Scottish Government

So far, I have found the following open data being produced:

• 248 datasets on the excellent, and expanding, Statistics.Gov.Scot portal  covering a number of departments, agencies and NDPBs,
• 54 datasets on the Scottish Natural Heritage portal, 53 of which are explicitly covered by OGL and one marked “free to use data.”
• At least 43 OGL-licensed mapping layers on the Marine Scotland portal
• Just four geospatial datasets for download on the Spatial Hub
• Six Linked open data sets, licensed under OGL, on the SEPA site.
• Great interactive mapping of the Scottish Indices of Multiple Deprivation, for which the source Data is included above on the Statistics Portal mentioned above.

That makes a total of 353 datasets. I’ve not tracked these number previously, so can’t say if they are rising, but there certainly appears to be good progress and some good quality work going on to make Scottish Government data available openly. This includes the four newly-opened sets of boundary data by the Spatial Hub, out of 33 data sets.

However, if we look at the breadth of agencies etc that comprises the Scottish Government, it is clear that there are many gaps. In addition to the parent body of the Scottish Government there are a further 33 Directorates, 9 Agencies, and 92 Non-Departmental Public Bodies. That’s a total of 135 business units.

Let’s assume that they could each produce a conservative 80 data sets, and it is arguable that that should be considerably higher, then we’d expect 10,800 datasets to be released. Suddenly, 353 doesn’t seem that great.

Health

Scotland’s Health service is composed, in addition to the parent NHS Scotland body, of 14 Health Boards and 30 joint Health and Social Care Partnerships. That gives a total of 45 bodies.

Again, taking the same modest yardstick, of 80 open data sets for each, we would expect to see 3,600 data sets released.

What I found was 26 data sets on the new NHS Scotland open data portal. This is a great, high-quality resource, which I know from conversations with those behind it has great commitment to adding to its range of data provided.

However, given our yardstick above, we are still 3,574 data sets short on Scottish Health data.

Higher and Further education

Scotland’s HE / FE landscape comprises of 35 Universities and colleges.

Glasgow and Edinburgh Universities each have an open data publication mechanism for data arising out of a business operation, which contain interesting and useful data.

Despite that, there is no operational, statistical or other open data being created by any universities or colleges that I could identify. Again, using the same measure as above, that produces a deficit of (80 x 35) or 2,800 datasets.

Supply versus expectation

If we accept for the moment that the approximate number of data sets that we might expect in the Scottish public sector is as set out above, and that the current provision is, or is close to, what I have found in this trawl, then what is the over all picture?

Sector	Published	Expected	Defecit
Local Government	731	8000	7,269
Scottish Government	353	10,800	10,447
Health	26	3,600	3,574
FE / HE	0	2,800	2,800
Totals	1,110	25,200	24,090

Table 1: Supply versus expectation of Scottish public sector Open Data

As we can see from the table above, it appears that the Scottish public sector is currently publishing 1,110 of 24,090 expected open data sets. This is just 4.6%. So, by those calculations, more than 95% of data that we might reasonably expect to see published as Open Data is not being released.

Scotland is behind the UK generally

Whether you agree with the exact figures or not, and I am open to challenge and discussion, it is clear that we are failing to produce the data that is badly needed to stimulate innovation and deliver the economic and social benefits that we expected when set out to deliver open data for Scotland.

I’ve long argued that in terms of the UK’s performance in Open Data league tables, such as the Open Data Barometer, Scotland is a drag on the UK’s performance, with Scotland’s meagre output falling well short of the rest of the UK’s Open Data.  In addition to existing approaches, we should see Scotland’s OD assessed separately, using the same methodology, in order to be able compare Scotland with the UK as a whole. That would allow us to measure Scotland’s performance on a like-for-like basis, identify shortfalls and target remedial action where needed.

Policy underpinning

I have argued previously that a significant issue which stops the Scottish public sector getting behind open data is the lack of public policy to make it happen, as well as an ignorance, or denial, of the potential economic and social benefits that it would bring. While I was part of the group who wrote the Scottish Government’s 2015 Open Data Strategy, it was, in its final form, toothless and not underpinned by policy.

We now have an Open Government Action Plan for Scotland 2018-2020 (PDF). This is  great step forward but unfortunately it is almost entirely silent on Open Data, as pointed out in my response to the draft in November 2018.

Even when Open Data does make an appearance, on page 19, it is relation to broader topic rather than forming actions on its own merits.  The position is similar in the plan’s detailed commitments.  This is not to denigrate the work that has gone into these, and the early positive engagement between Scottish Government and civic groups, but this is a huge missed opportunity – and we should not have to wait until 2020 to rectify it.

At this point, it is worth contrasting this with the Welsh Government’s Open Government plan 2016-2018 which was reviewed recently (PDF). In that plan, Open Data was the entire focus of the first two sections, and covered pages 4 to 6 of the plan. This was no afterthought: it was a significant driver and a central plank of their open government plan.

The broader community

Scotland still lacks a developed Open Data community. This will come in time as data is made more widely available, is more usable and useful – and also through the engagement with the Open Government process  – but we all need to work to develop that and accelerate the process. I set out suggestions for this in a previous post.

There are significant opportunities to grow the use of open data through the opening of private sector and community-generated and -curated data.

The universities and colleges in Scotland should be adopting open data in their curriculum, raising awareness among students, creating entrepreneurs who can establish businesses on the back of open data.

Schools should be using open data to get their classes involved: using it to explain their environment, climate, and transport system; to understand local demographics, the distribution of local government spending, or comparative attainment of schools.

Government should be  developing the curriculum to use open data to foster a better understanding of data and how it underpins modern society.

There are some positive things going on: the roadshows that the Scottish Government are doing, as well as other Data Fest Fringe events; the regular data hack weekends we’ve been doing in Aberdeen under the Code The City banner; and the major long-term project to build and deploy community-hosted air quality monitoring sensors which provide open data for the local community. These need to become the norm – and to be happening across the country.

Organisations such as The Data Lab, Censis and other innovation centres have a great opportunity here to advance their work, whether in education, community building or fostering innovation, and to support this to achieve their organisational missions.

Bringing people together

Having earlier created a Twitter account for a nascent Scottish Open Data Action Group (@Soda_group), I have reconsidered that. Instead of an action group to pressure, shame or coerce the Scottish Government into action, what we need is a common group that has the Scottish Government onside – and everyone works together. So I have renamed it @opendata_sco. It already has 179 followers and I hope that we can grow that quickly, and use that to generate more interest and engagement.

I have also launched a new open Slack channel for Open Data Scotland, so that a community can better communicate with one another.

Please join, using this form.

As I have said previously this isn’t a them-and-us, supply-and-demand relationship. We’re all in it together, and the better we collaborate as a community the better, and quicker, society as a whole benefits from it.

========================================

Header photo by Andrew Amistad on Unsplash