Code The City is a civic hacking initiative focused on using tech and (open) data for civic good. We use hack weekends, open data, workshops, and idea generation tools. We run Data Meetups, the Aberdeen Python User Group and the annual Scottish Open Data Unconference
Do you ever get the feeling that patterns are emerging, the stars aligning, things coming together, coincidences happening more frequently? Of course there is a lot of magical thinking behind those feelings – but the sessions I’ve attended, books I’ve read, speakers I’ve heard, conversations I’ve participated in this week all point to the importance and significance of SODU 2022 next weekend.
Mid-week I had a conversation with represenatatives from Scottsh Government about a forthcoming refresh of their 2015 Open Data Strategy. I was part of the group which drew up the original strategy. Between the drafting and the publication of final version, any sense that implementing the strategy was mandatory was lost. Some of us warned that this would lead to low adoption and therefore low impact in terms of data being released. We were, it is sad to say, proved correct. Of course the published strategy does say that Scotland’s data is to be open by default, but in practice it is anything but that.
While it is pleasing to hear that the Scottish Government is refreshing its open data strategy, the reason for that refresh is not, as far as I can tell, to make open data manadatory or to put measures in place to benchmark performance. As far as we can ascertain no-one monitors or has monitored the implementation of the 2015 strategy; even to the extent of counting as we did in 2019 the number of datasets released, itself a crude measure, And if I read the runes correctly, that commitment to data open by default might not make it in to the next iteration of the strategy which would be a major leap backwards. Civic society needs to make their feelings known about this in the strongest possible terms.
On Friday I gave a presentation to the University of Aberdeen’s Open Access Week’s Open Data event. I spoke about making open data useful, useable and used. The talk was one of half a dozen. I found the talks by each of the others speakers very good – but the one by Paola Masuzzo very impactful. The scenarios she spoke of in the struggle to get good government data resonated with the fight I and others put up to get Covid Data released in 2020. Paola is on the board of On Data and their website has a lot of great content. Unless you are fluent in Italian you might be better allowing Google to translate it if you open it in Chrome browser.
Another conversation that I have been having is with Jack and Karen, my colleagues in the Open Data Scotland project. This was born at Code The City hack weekends and previous SODU sessions. Should it remain as a CTC project? Should we spin it off as its own Community Interest Company? What is its role beyond being an open data portal? In my view we need to be more of a lobbying, pressure group for OD in Scotland. For example I believe that one week away from SODU2022 we have one person from the whole of the public sector in Scotland attending for only one day. There should at least be be someone there from every council health board, government agency etc. That would mean about 180 government attendees. But if their job doesn’t have open data at its core, and there is no obligation to publish, and no consequences to not doing so, why bother publishing? And why bother attending this weekend event?
It could be argued that since the Digital Strategy 2021 in Scotland does recognise the economic vlaue of open data then some of the core digital team with a responibility for implementing that should be there?
It was higlighted at SODU2021 and 2020, if I recall correctly, that there is a gap between government and civic society, and that the enormous goodwill and positivity of the civic group is a resource which government ignores and squanders. Chatting to someone else this week we puzzled to understand why the public sector is so afraid of engagement? Is there a sense that they know they are failing at OD and seek to avoid questioning? We know that several civil servants say they are committed to OD, but that there is no ministerial support in Scotland. Could that be it? We are a genuinely friendly community – and would welcome them with open arms.
This week I’ve been listening to the audio version of The Moves That Matter: A Grandmaster on the Game of Life by Jonathan Rowson, who I knew when he was a teenager in Aberdeen. At one point Jonathan said something about chess that I’ve long thought about applying to open data. He argues that chess “falls through the cultural cracks and lacks a stable cultural category that honours all of its elements; chess is more than a sport but is less than sport; chess is educational and educative but is not education; chess is full of artistic ideas but is not art; … is in many ways scientific but it not science“. He draws from this that without a cultural category that corresponds to a government department that would fund it, or a media section that would report it, it goes underfunded and unreported. The parallels with open data are deep. What IS open data? How do we get it funding and supported in `government? And recognition and attention?
Today was Wikidata‘s 10th birthday. I’m a big fan of Wikidata – and I train people to use it on behalf of Wikimedia UK. I also give guest lectures at RGU to Master students on the topic. We’ve used Wikidata in many projects at CTC hack events too. I’m convinced that Wikidata has a significant role to play as a place for publicly-curated linked open data – and filling in for missing ‘official’ government data,
Finally, I was contacted by someone who had attended the UK Open Data Camp that we hosted in 2018 in Aberdeen. He’s now living in Orkney and having heard about SODU2022 he’s making the journey down to attend, which is great to hear. We always look forward to catching up with the broader community.
Personally, I can’t wait to be at SODU2022. I hope that you do to and that we can have some meaningful discussions on the points above and much more!
If you haven’t booked yet you can still grab a ticket now.
[Published 29 Oct 2022. Edited 30 Oct to correct spelling errors and for readability.]
“Hear me calling, hear me calling loud, If you don’t come soon, I’ll be wearing a shroud.” – Ten Years After (1969)
Introduction
Today marks the tenth anniversary of my involvement with Open Data in Scotland. As I wrote here, back in 2009-2010 I’d been following the work that Chris Taggart and others were doing with open data, and was inspired by them to create what I now believe to have been the first open data published in the public sector in Scotland.
This piece is a reflection of my own views. These views may be the same as those held by colleagues at Code The City or indeed on the civic side of the Open Government Partnership. I’ve not specifically asked other individuals in either group.
While my involvement in, and championing of, open data in Scotland is now a decade long, my enthusiasm for the subject and in the the social and economic benefits it can deliver, is undiminished by my leaving the public sector in 2017 after thirty four years. In fact the opposite is true: the more I am involved in the OD movement, and study what is being achieved beyond Scotland’s narrow borders, the more I am convinced that we are a country intent on squandering a rich opportunity, regardless of our politicians’ public pronouncements.
But the journey has not been easy. primarily due to a lack of direction from Scottish Government and little commitment, resource or engagement at all levels of public service. A friend who reviewed this blog post suggested that I should replace the picture of a birthday cake (above) with one of a naked human back bearing bleeding scars from the our battles. He’s right – it is STILL a battle ten years on.
It is not as if the position in Scotland is getting better. We are moving at a glacial pace. The gap between Scotland and other countries in this regard is widening. I gave a talk earlier this year in which I showed assessments of Scotland, Romania and Kenya’s performance in Open Government (source: https://www.opengovpartnership.org/campaigns/global-report/ Vol 2) and asked the audience to identify which was Scotland.
Extracts from Vol2 to of the Open Gov Partnership report
In February 2020 the European Data Portal published a report – The Economic Impact of Open Data – which sets out a clear economic case for open data. That paper looks at 15 previous studies between 1999 and 2020 which have examined at the market size of open data at national and international levels, measured in terms of GDP of each study’s geographical area.
Taking the average and median values from those reports (1.33% and 1.19% respectively) and an estimated GDP for Scotland (2018) of £170.4bn we can see that the missed opportunity for Scotland is of the order of £2.027bn to £2.266bn per annum. What is the actual value of the local market created by Scottish-created open data? if pushed for a figure I would estimate that it is currently worth a few hundred thousand pounds per annum, and no more. Quite a gap!
Meantime we have the usual suspect of consultants whispering sweetly in the ears of ministers, senior civil servants and council bosses that we should be monetising data, creating markets, selling it. There will be no mention, I suspect of the heavily-subsidised, private sector led, yet failed Copenhagen Data Exchange, I suspect. (Maybe they can make a few bob back selling the domain name! )
You can buy the failed CityDataExchange.com for just $5195
While this commercial approach to data may plug small gaps in annual funding for Scotland, and line the pockets of some big companies in the process, it won’t deliver the financial benefits at a national level of anything like the figures suggested by that EU Data Portal report but it will, in the process, actively hamper innovation and inhibit societal benefits.
I hear lots of institutions saying “we need to sell data” or “we need to sell access rights to these photos” or similar. Yet, in so many cases, the operation of the mechanisms of control; the staffing, administration, payment processing etc. far outstrips any generated income. When I challenged ex colleagues in local government about this behaviour their response was “but our managers want to see an income line” to which we could add “no matter how much it is costing us.” And this tweet from The Ferret on Tuesday of this week is another excellent example of this!
I have also heard lots of political proclamations of “open and transparent” government in Scotland since 2014. Yet most of the evidence points in exactly the opposite direction. Don’t forget, when Covid 19 struck, Scotland’s government was reportedly the only political administration apart from Bolsonaro’s far right one in Brazil to use the opportunity to limit Freedom of Information.
Openness, really?
It is clear that there is little or no commitment to open data in any meaningful way at a Scottish Government level, in local authorities, or among national agencies. This is not to say that there aren’t civil servants who are doing their best, often fighting against political or senior administration’s actions. Public declarations are rarely matched by delivery of anything of substance and conversations with people in those agencies (of which I have had many) paints a grim picture of political masters saying one thing and doing another, of senior management not backing up public statements of intent with the necessary resource commitment and, on more than occasion, suggestions of bad actors actually going against what is official policy.
I mention below that I joined the Open Government Partnership late in 2019. Initially I was enthusiastic about what we might achieve. While there are civil servants working dedicatedly on open government who want to make it work, I am unconvinced about political commitment to it. We really need to get some positive and practical demonstration that Scottish Government are behind us – otherwise I and the other civil society representatives are just assisting in an open-washing exercise.
In my view (and that of others) the press in Scotland does not provide adequate scrutiny and challenge of government. We have a remarkably ineffective political opposition. We also have a network of agencies and quangos which are reliant on the Scottish Government for funding who are unwilling to push back. All of this gives the political side a free pass to spout encouraging words of “open and transparent” yet do the minimum at all times.
We may have an existing Open Data Strategy for Scotland (2015) stating that Scotland’s data is “open by default”, yet my 2019 calculation was that over 95% of the data that could and should be open was still locked up. And there is little movement on fixing that.
We have many examples of agencies doing one thing and saying another, such as Scottish Enterprise extolling the virtues of Open Data yet producing none. Its one API has been broken for many months, I am told.
My good friends at The Data Lab do amazing work on funding MSc and Phd places, and providing funding for industrial research in the application of data science. Their mission is “to help Scotland maximise value from data …” yet they currently offer no guidance on open data, no targeted programme of support, no championing of open data at all, despite the widely-accepted economic advantages which it can deliver. There is the potential for The Data Lab to lead on how Scotland makes the most of open data and to guide government thinking on this!
All of this is not to pick on specific organisations, or hard working and dedicated employees within them. But it does highlight systemic failures in Scotland from the top of government downwards.
Fixing this is an enormous task: one which can only be done by the development of a fresh strategy for open data in Scotland, which is mandated for all public sector bodies, is funded as an investment (recognising the economic potential), and which is rigorously monitored and enforced.
I could go on…. but let’s look at this year’s survey.
In February 2019 I conducted a survey of the state of open data in Scotland. It didn’t paint an encouraging picture. The data behind that survey has been preserved here. A year on, I started thinking about repeating the review.
In the intervening year I’d been involved in quite a bit activity around open data. I had
joined the civic side of the Open Government group for Scotland and was asked to lead for the next iteration of the plan on Commitment Three (sharing information and data) ,
joined the steering group of Stirling University’s research project, Data Commons Scotland,
trained as a trainer for Wikimedia UK, delivering training in Wikidata, Wikipedia and Wiki Commons, and running multiple sessions for Code The City with a focus on Wikidata,
created an open Slack Group for the open data community in Scotland to engage with one another,
created an Open Data Scotland twitter account which has gained almost 500 followers, and
In restarting this year’s review of open data publishing in Scotland my aims were to see what had changed in the intervening 12 months and to increase the coverage of the survey: going broader and deeper and developing an even more accurate picture. That work spilled into March at which point Covid-19 struck. During lockdown I was distracted by various pieces of work. It wasn’t until August, and with a growing sense of the imminence of this 10-year anniversary, that I was galvanised to finish that review.
I am conscious that the methodology employed here is not the cleverest – one person counting only the numbers of datasets produced. This is something I return to later.
The picture in 2020
I broke the review down into sectoral groupings to make it more managable to conduct. By sticking to that I hope to make this overview more readable. The updated Git Hub repo in which I noted my findings is available publicly, and I encourage anyone who spots errors or omissions to make a pull request to correct them. Each heading below has a link to the Github page for the research.
Overall there is little significant positive change. This is one factor which gives rise to concerns about government’s commitment to openness generally and open data specifically; and to a growing cynicism in the civic community about where we go from here.
I reviewed this area in February 2020 and rechecked it in August. Sadly there has been no significant change in the publication of open data by local government in the eighteen months since I last reviewed this. More than a third of councils (13 out of a total of 32) still make no open data provision.
While the big gain is that Renrewshire Council have launched a new data portal with over fifty datasets, most councils have shown little or no change.
Sadly the Highland Council portal, procured as part of the Scottish Cities Alliance’s Data Cluster programme at £10,000’s cost, has vanished. I dont think it ever saw a dataset being added to it. Searching Highland Council’s website for open data finds nothing.
While big numbers of data sets don’t mean much by themselves, the City of Edinburgh Council has a mighty 236 datasets. Brilliant! BUT … none of them are remotely current. The last update to any of them was September 2019. Over 90% of them haven’t been updated since 2016 or earlier.
Similarly Glasgow, which has 95 datasets listed have a portal which is repeatedly offline for days at a time. A portal which won’t load is useless.
Dundee, Perth and Stirling continue to do well. Their offerings are growing and they demonstrate commitment to the long-haul.
Aberdeen launched a portal, more than three years in the planning, populated it with 16 datasets and immediately let their open data officer leave at the end of a short-term contract. Some of their datasets are interesting and useful – but there was no consultation with the local data community about what they would find useful, or deliver benefits locally; all despite multiple invitations from me to interact with that community at the local data meet-ups which I was running in the city.
It was hoped that the programme under the Scottish Cities alliance would yield uniform datasets, prioritised across all seven Scottish Cities, but there is no sign of that happening, sadly. So what you find on all portals or platforms is pretty much a pot-luck draw.
Where common standards exist – such as the 360 Giving standard for the publication of support for charities – organisations should be universally adopting these. Yet this is only used by two of 32 authorities, all of whom have grant-making services. Surely, during a pandemic especially, it would be advantageous to funders and recipients to know who is funding which body to deliver what project?
Councils – Open Government Licence and RPSI
This is a slight aside from the publication of open data, but an important one. If the Scottish Authorities were to adopt an OGL approach to the publication of data and information on their website (as both the Scottish Government’s core site and the Information Commissioner for Scotland do) then we would be able to at least reuse data obtained from those sites. This is not a replacement for publishing proper open data but it would be a tiny step forward.
The table below (source and review data here) shows the current permissions to reuse the content of Scottish Local Authorities’ websites. Many are lacking in clarity, have messy wording, are vague or misunderstand terminologies. They also, in the main, ignore legislation on fair re-use.
Table of local authority adoption of PGL and RPSI
Open Government Licence
The Scottish Government’s own site is excellent and clear: permitting all content except logos to be be reused under the Open Government Licence. This is not true for local authorities. At present only Falkirk and Orkney Councils – two of the smaller ones – allow, and promote OGL re-use of content. There is no good reason why all of the public sector, including local government, should not be compelled to adopt the terms of OGL.
Re-use of Public Sector Information (RPSI) Regulations
Since 2015 the public sector has been obliged by the RPSI Regulations to permit reasonable reuse of information held by local authorities. So, even if Scottish LAs have not yet adopted OGL for all website content, they should have been making it clear for the last five years how a citizen can re-use their data and information from their website.
In my latest trawl through the T&Cs and Copyright Statements of 32 Scottish Local Authorities, I found only 7 referencing RPSI rights there, with 25 not doing so (see the full table above). I am fairly sure that these authorities are breaking the legal obligation on public bodies to provide that information.
Finally, given the presence of COSLA on the Open Government Scotland steering group, the situation with no open data; poor, missing or outdated data; and OGL and PRSI issues needs to be raised there and some reassurance sought that they will work with their member organisations to fix these issues.
The NHS Scotland Open Data platform continues to be developed as a very useful resource. The number of datasets there has more than doubled since last year (from 26 to 73).
None of the fourteen Health Boards publish their own open data beyond what is on the NHS Scotland portal.
Only one of the thirty Health and Social Care Partnerships (HSCPs) publish anything resembling open data: Angus HSCP.
COVID-19 and open data
While we are on health, I’ve wrote (here and here) early in the pandemic about the need for open data to help the better public understanding of the situation, and stimulate innovative responses to the crisis. The statistics team at Scottish Government responded well to this and we’ve started to develop a good relationship. I’ve not followed that up with a retrospective about what did happen. Perhaps I will in time.
It was clear that the need for open data in CV19 situation caught government and health sector napping. The response was slower than it should have been and patchy, and there are still gaps. People find it difficult to locate data when it is on muliple platforms, spread across Scots Govt, Health and NRS. That is, in a microcosm, one of the real challenges of OD in Scotland.
With an open Slack group for Open Data Scotland there is a direct channel that data providers could use to engage the open data community on their plans and proposals. They could also to sound out what data analysts and dataviz specialists would find useful. That opportunity was not taken during the Covid crisis, and while I was OK in the short term with being used as a human conduit to that group, it was neither efficient nor sustainable. My hope is that post SODU 2020, and as the next iteration of the Open Gov Scotland plan comes together we will see better, more frequent, direct engagement with the data community on the outside of Government, and a more porous border altogether.
There is no significant change across the sector in the past 18 months. The vast majority of institutions make no provision of open data. Some have vague plans, many of them historic – going back four years or more – and not acted on.
Lumping Universities and Colleges together, one might expect at a minimum properly structured and licensed open data from every institution on :
courses
modules
events
performance (perhaps some of this is on HESA and SFC sites?)
physical assets
environmental performance
KPI targets and achievements etc.
Of course, there is none of that.
Universities and colleges
I reviewed open data provision of Universities and Colleges around 17 February 2020. I revisited this on 11 August 2020, making minor changes to the numbers of data sets found.
While five of fifteen universities are publishing increasing amounts of data in relation to research projects, most of which are on a CC-0 or other open basis, there continues to be a very limited amount of real operational open data across the sector with loads of promises and statements of intent, some going back several years.
The Higher Education Statistics Agency publishes a range of potentially useful-looking Open Data under a CC-BY-4.0 licence. This is data about insitutions, course, students etc – and not data published by the institutions themselves. But I could identify none of that. Overall, this was very disappointing.
Further, while there are 20 FE colleges. None produces anything that might be classed as open data. A few have anything beyond vague statement of intent. Perhaps City of Glasgow College not only comes closest, but does link to some sources of info and data.
The Crighton Observatory
While doing all of this, I was reminded of the Crighton Institute’s Regional Observatory which was announced to loud fanfares in 2013 and appears to have quietly been shut down in 2017. Two of the team involved say in their Linked In profiles that they left at the end of the project. Even the domain name to which articles point is now up for grabs (Feb 2020).
It now appears (Aug 2020) that the total initial budget for the project was >£1.1m. Given that the purpose of the observatory was to amass a great deal of open data, I have also attempted to find out where the data is that it collected and where the knowledge and learning arising from the project has been published for posterity? I can’t locate it. This FOI request may help. The big question: what benefits did the £1.1m+ deliver?
In February 2019 I found that The Scottish Parliament had released 121 data sets. This covers motions, petitions, Bills, petitions and other procedural data, and is very interesting. This year we find that they have still 121 data sets, so, there are no new data sources.
In fact that number is misleading. In February 2020 I discovered that while 75 of these have been updated with new data, the remaining 46 (marked BETA) no longer work. As of August 2020 this is still the case. Why not fix them, or at worst clear them out to simplfy the finadbility of working data?
Some of these BETA datasets should contain potentially more interesting / useful data e.g. Register of Members Interests but just don’t work. Returning: [“{message: ‘Data is presently unavailable’}”]
I didn’t note the availability of APIs last year, but there are 186 API calls available. Many of these are year-specific. I tested half a dozen and about a third of those returned error messages. I suspect some of these align with the non-functioning historic BETAs.
Sadly the issues raised a year ago about the lack of clarity of the licensing of the data is unchanged. To find the licence, you have to go to Notes > Policy on Use of SPCB Copyright Material. Following the first link there (to a PDF) you see that you have to add “Contains information licenced under the Scottish Parliament Copyright Licence.” to anything you make with it, which is OK. But if you go to the second link “Scottish Parliament Copyright Licence” (another PDF) the wording (slightly) contradicts that obligation. It then has a chunk about OGL but says, “This Scottish Parliament Licence is aligned with OGLv3.0” whatever that means. Why not just license all of the data under OGL? I can’t see what they are trying to do.
Trying to work out the business units within the structure of Scottish Government is a significant challenge in itself. Attempting to then establish which have published open data, and what those data sets are, and how they are licensed, is almost an impossible task. If my checking, and arithmetic are right, then of 147 discrete business units, only 27 have published any open data and 120 have published none.
So we can say with some confidence that the issue with findability of data raised in Feb 2019 is unchanged, there being no central portal for open data in the Scottish public sector or even for Scottish Government. Searching the main Scottish Government website for open data yields 633 results, none of which are links to data on the first four screenfuls. I didn’t go deeper than that.
The Scottish Government’s Statistics Team have a very good portal with 295 Data Sets from multiple organisational-providers. This is up by 46 datasets on last year and includes a two new organisations: The Care Inspectorate and Registers of Scotland. The latter, so far (Aug 2020), has no datasets on the portal.
There are some interesting new entrants into the list of those parts of Scottish Government publishing data such as David MacBrayne Limited which is, I believe, wholly owned by SG and is the parent, or operator of Calmac Ferries Limited. On 1st March 2020 they released a new data platform to get data about their 29 ferry routes. This is very welcome. After choosing the dates, routes and traffic types you can download a CSV of results. While their intent appears to be to make it Open Data, the website is copyright and there is no specific licensing of the data. This is easily fixable.
It is also interesting to contrast Transport Scotland with work going on in England. Transport Scotland’s publication scheme says of open data “Open data made available by the authority as described by the Scottish Government’s Open Data Strategy and Resource Pack, available under an open licence. We comply with the guidance above when publishing data and other information to our website. Details of publications and statistics can be found in the body of this document or on the Publications section of our website.” I searched both without success for any OD. Why not say “we don’t publish any Open Data”? Compare this complete absence of open data with even the single project Open Bus Data for England. Read the story here. Scotland is yet again so far behind!
Summary
In the review of data I’ve shown that little has changed in 18 months. Very few branches of government are publishing open data at all. The landscape is littered with outdated and forgotten statements of good intent which are not acted on; broken links; portals that vanish or don’t work; out of date data; yawning gaps in publication and so on.
The claim of “Open By Default” in the current (2015) Open Data Strategy is misleading and mostly ignored with consequence. The First Minister may frequently repeat the mantra of “Open and Transparent” when speaking or questioned by journalists, but it is easily demonstrable that the administration frequently act in the directly opposite way to that.
The recent situations with Covid-19 and the SQA exams results show Scotland would have found itself in a much better place this year with a mature and well-developed approach to open data: an approach one might have reasonably expected after five full years of “open by default”.
The social and economic arguments for open data are indisputable. These have been accepted by most other governments of the developed world. Importantly, they have also been taken up and acted on by developing nations who have in many cases overtaken Scotland in their delivery of their Open Government plans.
The work I have done in 2019 and in this review is not a sustainable one – i.e. one single volunteer monitoring the activity of every branch and level of government in Scotland. And the methodology is limited to what is achievable by an individual.
A country which was serious about Open Data would have targets and measures, monitoring and open reporting of progress.
It wouldn’t just count datasets published. It would be looking at engagement, the usefulness of data and its integration into education.
It would fund innovation: specifically in the use of open data; in the creation of tools; in developing services to both support government in creating data pipelines, and in helping citizens in data use.
It would co-develop and mandate the use of data standards across the public sector.
It would develop and share canonical lists of ‘things’ with unique identifiers allowing data sets to be integrated.
It would adopt the concept of data as infrastructure on which new products, services, apps, and insights could be built.
I really want Scotland to make the most of the opportunities afforded by Open Data. I wouldn’t have spent ten years at this if I didn’t believe in the potential this offers; nor if I didn’t have the evidence to show that this can be done. I wouldn’t be giving up my time year-on-year researching this, giving talks, organising groups and creating opportunities for engagement.
What is fundamentally lacking here is some honesty from Scottish Government ministers instead of their pretence of support for open data.
Scotland is B, in the centre. Kenya is A, and Romania C.
I could have chosen Mexico, Honduras, Paraguay, Uruguay – or others. All are doing better than Scotland.
Note: this blog post first appeared on codethecity.co.uk in February 2019 and has been archived here with a redirect from the original URL.
Scotland’s provision of open data may be slowly improving, but it is a long way behind the rest of the UK. In my most recent trawl through websites and portals I found a few minor improvements, which are positive, but progress is too slow; some data providers are slipping backwards; and most others are still ignoring the issue altogether. Now is the time for the Scottish Government to act to fix this drag on the Scottish economy and society, and stop inhibiting innovation.
Latest review
Over the last week, I have conducted yet another trawl of Scottish Open Data websites and portals. I keep this updated on this Github Repo. I’ve carried out this research without assistance, in my own time. The review could be more comprehensive, frequent and robust if I was supported to do it.
This work builds on previous pieces of research I’ve carried out and articles that I have written. Recently, I’ve created an index of those blog posts here as much for my own convenience of finding and linking to them as anything.
During this latest trawl, I’ve tried to better capture the wide spread of Scottish Government departments, agencies, non-departmental public bodies, health boards, local authorities, health and social care partnerships and academic institutions; and assess each sector using quite conservative measures.
The output of that, as we will see below, does not paint a good picture of Scotland’s performance, despite a few very good examples of people doing good work despite a clear policy gap.
Let us look at this sector by sector, following the list of findings here.
Local Authorities
Of Scotland’s 32 local authorities, only 19 produce open data of any kind. This group uses a mixture of open data portals (10), web landing pages (7) and GIS systems (2). This leaves 13 who produce no open data whatsoever.
Those 19 councils (ignoring the other 13) produce a total of 731 datasets, giving a mean for the group of 38 and a median of 17 datasets. This total is only six more than I found three months ago, despite Dumfries and Galloway launching a new portal with 33 datasets !
Also, stagnation is a real issue. For example, it is worth noting once again that while Edinburgh produces an impressive 234 open data sets, only five of those have been updated in the last six months, and 228 of them date from 2014-2017. While there is a value in retaining historic data ( allowing comparisons, trends etc to be analysed), the value of data which is not being updated diminishes rapidly.
When I ran the OD programme for Aberdeen City Council (which, like all Scottish councils, is a unitary authority), based on some back-of-the-envelope calculations I reckoned that we could reasonably expect to have about 250 data sets. So, if each of the 32 did the same, as we would expect, then we’d have 8,000 datasets from local authorities alone. This puts the 731 current figure into perspective.
Scottish Government
So far, I have found the following open data being produced:
248 datasets on the excellent, and expanding, Statistics.Gov.Scot portal covering a number of departments, agencies and NDPBs,
54 datasets on the Scottish Natural Heritage portal, 53 of which are explicitly covered by OGL and one marked “free to use data.”
Just four geospatial datasets for download on the Spatial Hub
Six Linked open data sets, licensed under OGL, on the SEPA site.
Great interactive mapping of the Scottish Indices of Multiple Deprivation, for which the source Data is included above on the Statistics Portal mentioned above.
That makes a total of 353 datasets. I’ve not tracked these number previously, so can’t say if they are rising, but there certainly appears to be good progress and some good quality work going on to make Scottish Government data available openly. This includes the four newly-opened sets of boundary data by the Spatial Hub, out of 33 data sets.
However, if we look at the breadth of agencies etc that comprises the Scottish Government, it is clear that there are many gaps. In addition to the parent body of the Scottish Government there are a further 33 Directorates, 9 Agencies, and 92 Non-Departmental Public Bodies. That’s a total of 135 business units.
Let’s assume that they could each produce a conservative 80 data sets, and it is arguable that that should be considerably higher, then we’d expect 10,800 datasets to be released. Suddenly, 353 doesn’t seem that great.
Health
Scotland’s Health service is composed, in addition to the parent NHS Scotland body, of 14 Health Boards and 30 joint Health and Social Care Partnerships. That gives a total of 45 bodies.
Again, taking the same modest yardstick, of 80 open data sets for each, we would expect to see 3,600 data sets released.
What I found was 26 data sets on the new NHS Scotland open data portal. This is a great, high-quality resource, which I know from conversations with those behind it has great commitment to adding to its range of data provided.
However, given our yardstick above, we are still 3,574 data sets short on Scottish Health data.
Higher and Further education
Scotland’s HE / FE landscape comprises of 35 Universities and colleges.
Glasgow and Edinburgh Universities each have an open data publication mechanism for data arising out of a business operation, which contain interesting and useful data.
Despite that, there is no operational, statistical or other open data being created by any universities or colleges that I could identify. Again, using the same measure as above, that produces a deficit of (80 x 35) or 2,800 datasets.
Supply versus expectation
If we accept for the moment that the approximate number of data sets that we might expect in the Scottish public sector is as set out above, and that the current provision is, or is close to, what I have found in this trawl, then what is the over all picture?
Sector
Published
Expected
Defecit
Local Government
731
8000
7,269
Scottish Government
353
10,800
10,447
Health
26
3,600
3,574
FE / HE
0
2,800
2,800
Totals
1,110
25,200
24,090
Table 1: Supply versus expectation of Scottish public sector Open Data
As we can see from the table above, it appears that the Scottish public sector is currently publishing 1,110 of 24,090 expected open data sets. This is just 4.6%. So, by those calculations, more than 95% of data that we might reasonably expect to see published as Open Data is not being released.
Scotland is behind the UK generally
Whether you agree with the exact figures or not, and I am open to challenge and discussion, it is clear that we are failing to produce the data that is badly needed to stimulate innovation and deliver the economic and social benefits that we expected when set out to deliver open data for Scotland.
I’ve long argued that in terms of the UK’s performance in Open Data league tables, such as the Open Data Barometer, Scotland is a drag on the UK’s performance, with Scotland’s meagre output falling well short of the rest of the UK’s Open Data. In addition to existing approaches, we should see Scotland’s OD assessed separately, using the same methodology, in order to be able compare Scotland with the UK as a whole. That would allow us to measure Scotland’s performance on a like-for-like basis, identify shortfalls and target remedial action where needed.
Policy underpinning
I have argued previously that a significant issue which stops the Scottish public sector getting behind open data is the lack of public policy to make it happen, as well as an ignorance, or denial, of the potential economic and social benefits that it would bring. While I was part of the group who wrote the Scottish Government’s 2015 Open Data Strategy, it was, in its final form, toothless and not underpinned by policy.
Even when Open Data does make an appearance, on page 19, it is relation to broader topic rather than forming actions on its own merits. The position is similar in the plan’s detailed commitments. This is not to denigrate the work that has gone into these, and the early positive engagement between Scottish Government and civic groups, but this is a huge missed opportunity – and we should not have to wait until 2020 to rectify it.
At this point, it is worth contrasting this with the Welsh Government’s Open Government plan 2016-2018 which was reviewed recently (PDF). In that plan, Open Data was the entire focus of the first two sections, and covered pages 4 to 6 of the plan. This was no afterthought: it was a significant driver and a central plank of their open government plan.
The broader community
Scotland still lacks a developed Open Data community. This will come in time as data is made more widely available, is more usable and useful – and also through the engagement with the Open Government process – but we all need to work to develop that and accelerate the process. I set out suggestions for this in a previous post.
There are significant opportunities to grow the use of open data through the opening of private sector and community-generated and -curated data.
The universities and colleges in Scotland should be adopting open data in their curriculum, raising awareness among students, creating entrepreneurs who can establish businesses on the back of open data.
Schools should be using open data to get their classes involved: using it to explain their environment, climate, and transport system; to understand local demographics, the distribution of local government spending, or comparative attainment of schools.
Government should be developing the curriculum to use open data to foster a better understanding of data and how it underpins modern society.
There are some positive things going on: the roadshows that the Scottish Government are doing, as well as other Data Fest Fringe events; the regular data hack weekends we’ve been doing in Aberdeen under the Code The City banner; and the major long-term project to build and deploy community-hosted air quality monitoring sensors which provide open data for the local community. These need to become the norm – and to be happening across the country.
Organisations such as The Data Lab, Censis and other innovation centres have a great opportunity here to advance their work, whether in education, community building or fostering innovation, and to support this to achieve their organisational missions.
Bringing people together
Having earlier created a Twitter account for a nascent Scottish Open Data Action Group (@Soda_group), I have reconsidered that. Instead of an action group to pressure, shame or coerce the Scottish Government into action, what we need is a common group that has the Scottish Government onside – and everyone works together. So I have renamed it @opendata_sco. It already has 179 followers and I hope that we can grow that quickly, and use that to generate more interest and engagement.
As I have said previously this isn’t a them-and-us, supply-and-demand relationship. We’re all in it together, and the better we collaborate as a community the better, and quicker, society as a whole benefits from it.
Note: This blogpost first appeared on codethecity.co.uk in January 2019 and has been archived here with a redirect from the original URL.
I wrote some recent articles about the state of open data in Scotland. Those highlighted the poor current provision and set out some thoughts on how to improve the situation. This post is about a concrete example of the impact of government doing things poorly.
Ennui: a great spur to experimentation
As the Christmas ticked by I started to get restless. Rather than watch a third rerun of Elf, I decided I wanted to practice some new skills in mapping data: specifically how to make Choropleth Maps. Rather than slavishly follow some online tutorials and show unemployment per US state, I thought it would be more interesting to plot some data for Scotland’s 32 local authorities.
Where to get the council boundaries?
If you search Google for “boundary data Scottish Local Authorities” you will be taken to this page on the data.gov.uk website. It is titled “Scottish Local Authority Areas” and the description explains the background to local government boundaries in Scotland. The publisher of the data is the Scottish Government Spatial Data Infrastructure (SDI). Had I started on their home page, which is far from user-friendly, and filtered and searched, I would have eventually been taken back to the page on the data.gov.uk data portal.
This takes you to a page headed, alarmingly, “Order OS Open Data.” After some lengthy text (which warns that DVDs will take about 28 days to arrive but that downloads will normally arrive within an hour), there then follows a list of fifteen data sets to choose. The Boundary Line option looked most appropriate after reading descriptions.
This was described as being in a proprietary ERSI shapefile format, and being 754Mb of files, with another version in the also proprietary Mapinfo format. Importantly, there was no option for downloading data for Scotland only, which I wanted. In order to download it, I had to give some minimal details, and complete a captcha. On completion, I got the message, “Your email containing download links may take up to 2 hours to arrive.”
There was a very welcome message at the foot of the page: “OS OpenData products are free under the Open Government Licence.” This linked not to the usual National Archives definition, but to a page on the OS site itself with some extra, but non-onerous reminders.
Once the link arrived (actually within a few minutes) I then clicked to download the data as a Zip file. Thankfully, I have a reasonably fast connection, and within a few minutes I received and unzipped twelve sets of 4 files each, which now took up 1.13GB on my hard drive.
Partial directory listing of downloaded files
Two sets of files looked relevant: scotland_and_wales_region.shp and scotland_and_wales_const_region.shp. I couldn’t work out what the differences were in these, and it wasn’t clear why Wales data is also bundled with Scotland – but these looked useful.
Wrong data in the wrong format
My first challenge was that I didn’t want Shapefiles, but these were the only thing on offer, it appeared. The tutorials I was going to follow and adapt used a library called Folium, which called for data as GeoJson, which is a neutral, lightweight and human readable file format.
I needed to find a way to check the contents of the Shapefiles: were they even the ones I wanted? If so, then perhaps I could convert them in some way.
To check the shapefile contents, I settled on a library called GeoPandas. One after the other I loaded scotland_and_wales_region.shp and scotland_and_wales_const_region.shp. After viewing the data in tabular form, I could see that these are not what I was looking for.
So, I searched again on the Scottish Spatial Infrastructure and found this page. It has a Download link at the top right. I must have missed that.
SSI Download Link
But when you click on Download it turns out to be a download of the metadata associated with the data, not the data files. Clicking Download link via OS Open Data, further down page, takes you back to the very same link, above.
I did further searching. It appeared that the Scottish Local Government Boundary Commission offered data for wards within councils but not the councils’ own boundaries themselves. For admin boundaries, there were links to OS’ Boundary Line site where I was confronted by same choices as earlier.
Eventually, through frustration I started to check the others of the twelve previously-downloaded Boundary Line data sets and found there was a shape file called “district_borough_unitary_region.shp” On inspection in GeoPandas it appeared that this was what I needed – despite Scottish Local Authorities being neither districts nor boroughs – except that it contained all local authority boundaries for the UK – some 380 (not just the 32 that I needed).
Converting the data
Having downloaded the data I then had to find a way to convert it from Shapefile to Geojson (adapting some code I had discovered on StackOverflow) then subset the data to throw away almost 350 of the 380 boundaries. This was a two stage process: use a conversion script to read in Shapefiles, process and spit out Geojson; write some code to read in the Geojson, covert it to a python dictionary, match elements against a list of Scottish LAs, then write the subset of boundaries back out as a geojson text file.
Code to convert shapefiles to geojson
Using the Geojson to create a choropleth map
I’ll spare the details here, but I then spent many, many hours trying to get the Geojson which I had generated to work with the Folium library. Eventually it dawned on me that while the converted Geojson looked ok, in fact it was not correct. The conversion routine was not producing the correct Geojson.
Another source
Having returned to this about 10 days after my first attempts, and done more hunting around (surely someone else had tried to use Scottish LAs as geojson!) I discovered that Martin Crowley had republished on Github boundaries for UK Administrations as Geojson. This was something that had intended to do for myself later, once I had working conversions, since the OGL licence permits republishing with accreditation.
Had I had access to these two weeks ago, I could have used them. With the Scottish data downloaded as Geojson, producing a simple choropleth map as a test took less than ten minutes!
Choropleth map of Scottish Local Authorities
While there is some tidying to do on the scale of the key, and the shading, the general principle works very well. I will share the code for this in a future post.
Some questions
There is something decidedly user-unfriendly about the SDI approach which is reflective of the Scottish public sector at large when it comes to open data. This raises some specific, and some general questions.
Why can’t the Scottish Government’s SDI team publish data themselves, as the OGL facilitates, rather than have a reliance on OS publishing?
Why are boundary data, and by the looks of it other geographic data, published as ESRI GIS shapefiles or Mapinfo formats rather than the generally more-useable, and much-smaller, GeoJson format?
Why can’t we have Scottish (and English, and Welsh) authority boundaries as individual downloads, rather than bundled as UK-level data, forcing the developer to download unnecessary files? I ended up with 1.13GB (and 48 files) of data instead of a single 8.1MB Scottish geojson file.
What engagement with the wider data science / open community have SDI team done to establish how their data could be useful, useable and used?
How do we, as the broader Open Data community share or signpost resources? Is it all down to government? Should we actively and routinely push things to Google Dataset Search? Had there been a place for me to look, then I would have found the GitHub repo of council boundaries in minutes, and been done in time to see the second half of Elf!
And finally
I am always up for a conversation about how we make open data work as it should in Scotland. If you want to make the right things happen, and need advice, or guidance, for your organisation, business or community, then we can help you. Please get in touch. You can find me here or here or fill in this contact form and we will respond promptly.
Note: this blog post was first published on 10th June 2018 at CodeTheCity.co.uk has has been archived here with redirects from the original URL.
In this, the first of two posts, I look back over eight years of open data in Scotland, showing where ambition and intent mostly didn’t deliver as we hoped.
In the next part I will look forward, examining how we should rectify things, engage the right people, build on current foundations, and how we all can be involved in making it work as we hoped it would all those years ago.
Let our story begin
“The moon was low down, and there was just the glimmer of the false dawn that comes about an hour before the real one.” – Rudyard Kipling, Plain Tales from the Hills, 1888
The story to-date of Open Data in Scotland is one of multiple false dawns. Are we at last about to witness a real sunrise after so much misplaced hope?
The trigger
At Data Lab‘s recent Innovation Week in Glasgow, I found myself among 115 other data science MSc students – some of the brightest and best in Scotland – working on seven different industry challenges. You can read more of how that went on my own blog. In this post I want to mention briefly one of the challenges, and the subsequent conversations which it stirred in the room, then on social media and even in email correspondence, then use that to illustrate my false dawn analogy.
The Challenge
The Innovation Week challenge was a simple one compared to some others, and was composed of two questions: “how might we analyse planning applications in light of biodiversity?”, and, “how might we evaluate the cumulative impact of planning applications across the 32 Scottish Local Authorities?”
These are, on the face of it, fairly easily answered. To make it even simpler, as part of the preparation for the innovation week, Data Lab, Snook and others had done some of the leg work for us. This included identifying the NBN Atlas system as one which contained over 219 million sightings of wildlife species, which could be queried easily and which provided open access to its data.
That should have been the difficult part. The other part, getting current and planning application data from the Scottish Local Authorities should have been the easier task – but it was far from it. In fact, in the context of the time available to us, it was impossible as we could find not a single council, of the 32, offering its planning data as open data. You can read more of the particulars of that on my earlier blog posts, above.
This is about the general – not the specific, so, for now, let us set some context to this, and perhaps see how we got to be this point.
The first false dawn.
We start in August 2010, when I was working in Aberdeen City Council. I’d been reading quite a bit about open data, and following what a few enlightened individuals, such as Chris Taggart were doing. It seemed to me so obvious that open data could deliver so much socially and economically – even if no formal studies had by then been published. So, since it was a no-brainer, I arranged for us to publish the first open data in Scotland – at least from a Scottish City Council.
The UK Coalition Government had, in 2010, put Open Data front and centre. They created http://data.gov.uk and mandated a transparency agenda for England and Wales which necessitated publishing Open Data for all LA transactions over £500.
At some point thereafter, in 2011-12 both Edinburgh and Glasgow councils started to produce some open data. Sally Kerr in Edinburgh became their champion – and began working with Ewan Klein in Edinburgh University to get things moving there. I can’t track the exact dates. If you can help me, please let me know and I will update this post.
Studies, and even mainstream press, were starting to highlight the benefits of open data. Now this was starting to feel like a movement!
Suffering from premature congratulation
In 2012 the Open Data Institute was founded by Nigel Shadbolt and Tim Berners-Lee, and from day one championed open data as a public good, stressing the need for effective governance models to protect it.
During 2012 and 2013 Aberdeen, Edinburgh and others started work with Nesta Scotland, run out of Dundee, by the inspirational Jackie McKenzie and her amazing team. They funded two collaborative programmes: Make It Local Scotland and Open Data Scotland.
The former had Aberdeen City Council using Linked Open Data (another leap forward) to create a citizen-driven alerts system for road travel disruption. This was built by Bill Roberts and his team at Swirrl – who have gone on to do more excellent work in this area.
Around mid 2013 Glasgow had received Technology Strategy Board funding for a future cities demonstrator was was recruiting people to work on its open data programme
The second Nesta programme, Open Data Scotland , saw two cities – Aberdeen and Edinburgh – work with two rural councils, East Lothian and Clackmannanshire. Crucially, it linked us all with the Code For Europe movement, and we were able to see at first-hand the amazing work being done in Amsterdam, Helsinki, Barcelona, Berlin and elsewhere. It felt that we were part of something bigger, and unstoppable.
And it gets real-er
In late 2014 the Scottish Government appeared to suddenly ‘get’ open data. They wanted a strategy – so they pulled a bunch of us together two write one. The group included Sally from Edinburgh and me – and the document was published in March 2015. I had pushed for it to have more teeth than it ended up having, and to commit to defined actions, putting an onus on departments and local government to deliver widely on this in a tight timescale.
It did include –
“To realise our vision and to meet the growing interest from users we encourage all organisations to have an Open Data publication plan in place and published on their website by December 2015. Organisations currently publishing data in a format which does not readily support re-use, should within their plan identify when the data will be made available in a more re-usable format. The ambition is for all data by 2017 to be published in a format of 3* or above.” I will come back to this later.
This MUST be it!
In 2016-2017 the Scottish Cities Alliance, supported by the European Regional Development Fund launched a programme: Scotland’s Eighth City – The Smart City. At its heart was data – and more specifically open data. The data project was to feature all seven of Scotland’s cities, working on four streams of work:
data standards
data platforms
data engagement and
Data analytics.
The perception was also at that time that the Scottish Government had taken its eye off the ball as regards open data. Little if anything had changed as a result of the 2015 strategy. By working together as 7 cities we could lead the way – and get the other 25 councils, and the Scottish Government themselves, not only to take notice, but also to work with us to put Open Data at the heart of Scottish public services.
The programme would run from Jan 2017 to Dec 2018. I was asked to lead it, which I was delighted to do – and remained involved in that way until I retired from Aberdeen City Council in June 2017.
At that point Aberdeen abandoned all commitment to open data and withdrew from the SCA programme. I have no first-hand knowledge of the SCA programme as it stands now.
Six False Dawns Later
So, after six false dawns what is the state of open data in Scotland: is it where we expected it to be? The short answer to that has be a resounding no.
Some of the developments which should have acted as beacons have been abandoned. The few open data portals we have are, with some newer exceptions, looking pretty neglected: data is incomplete or out of date. There is no national co-ordination of effort, no clear sets of guidance, no agreement on standards or terminologies, no technical co-ordination.
Activity, where it happens at all, is localised, and is more often than not grass-roots driven (which is not in itself a bad thing). In some cases local authorities are being shamed into reinstating their programmes by community groups.
The Scottish Government, with the exception of their SIMD Linked Data work, which was again built by Swirrl, and some statistical data, have produced shamefully little Open Data since their 2015 Strategy.
Despite a number of key players in the examples above still being around, in one role of another, and a growing body of evidence demonstrating ROI, there is strong evidence that Senior Managers, Elected Members and others don’t understand the socio-economic benefits that publishing open data can bring. This is particularly disturbing considering the shrinking budgets and the need to be more efficient and effective.
So, what now?
Given that we have witnessed these many false dawns, when will the real sunrise be? What will trigger that, and what can we each do to make it happen?
Code The City 
