Code The City is a civic hacking initiative focused on using tech and (open) data for civic good. We use hack weekends, open data, workshops, and idea generation tools. We run Data Meetups, the Aberdeen Python User Group and the annual Scottish Open Data Unconference
– why so poorly understood or adopted in Scotland?
Public Sector Information (PSI) is information that has been created by Public Bodies. In 2003 The EU published “Directive 2003/98/EC on the re-use of public sector information, known as the PSI Directive. This is an EU directive that stipulates minimum requirements for EU member states regarding making public sector information available for re-use. This directive provides a common legislative framework for this area. The Directive is an attempt to remove barriers that hinder the re-use of public sector information throughout the Union.” [1]
In 2005 the UK Government introduced its own regulations. [2] And ten years later the it introduced the Reuse of Public Sector Information Regulations 2015 [3] which set out the framework for UK public sector bodies sharing information and data in line with those requirements. The National Archives produced guidance for both public bodies, and for would-be re-users of the info and data. [4]
Re-use means using public sector information for a purpose different from the one for which it was originally produced, held or disseminated.
Why is PSI valuable
“Public sector information constitutes a vast, diverse and valuable pool of resources. In Market Assessment of Public Sector Information (commissioned by the Department for Business Innovation and Skills in 2013), the value of public sector information to consumers, businesses and the public sector itself in 2011/12 was estimated to be approximately £1.8 billion (in 2011 prices). [5]
“Re-use of public sector information provides enormous opportunities for economic and social benefits, while also promoting transparency and accountability of the public sector.” [5]
Why PSI must be reusable
Re-use of public sector information stimulates the development of innovative new information products and services in the UK and across Europe, thus boosting the information industry.
“The Re-use of Public Sector Information Regulations 2015 [have been] in force from 18 July 2015. They build on the prior 2005 Regulations which removed obstacles to the re-use of public sector information. The 2015 Regulations harmonise and relax the conditions of re-use for public sector information, and bring the cultural sector into scope. The 2015 Regulations continue to improve transparency, fairness and consistency among public sector bodies and re-use of their information. [6]
How is information and data licensed for re-use?
Public bodies, up until 2010 were obliged to make information and data available under a PSI Click-Use Licence. [7] In 2010 this was superseded by V1.0 of the Open Government Licence ( or OGL). The latter has been updated twice and is now at version 3.0. [8]
Where is information and data in Scotland licenced for re-use?
While several local authorities, The Scottish Government and a few government agencies have published open data either on dedicated portals or in sections of their websites explicitly licensed under Open Government Licence (OGL 3), almost no organisations make website content so available for re-use despite an obligation to do so for almost 20 years.
When I worked at Aberdeen City Council, trying to get senior managers in numerous services and departments to make information and data available under OGL or its predecessor was a non-starter. The legal department at the time didn’t support it. And any conversation pushed it further down the line as a ‘nice-to-do’. And we weren’t alone. No council that I can recall was doing this as standard for their websites.
The one notable, and welcome exception, that I am aware of, was The Scottish Government. Since at least 2015 they’ve permitted reuse under OGL, albeit behind a Crown Copyright link at the foot of each website page. But Since April 2022 they explicitly had an open licence statement on each page. Unfortunately this doesn’t carry through to other public body sites, at least not explicitly. With over 180 to check [8] it is difficult to be absolutely certain.
The OGL and Copyright statement on the current Scottish Government Website
I personally, and with various groups, have campaigned for the Scottish public sector to meet their obligations under RPSI 2015. In November 2018 I responded to the consultation on the Scotland Draft Action Plan on Open Government[9]:
“There is one simple thing that could be done with immediate impact, and minimal effort, to free up large amounts of data and information for public re-use: adopt an Open Government Licence (OGL) for all published website information and data on the Scottish Government’s website(s), and other public sector sites, the only exception being where this cannot legally be done, as would be the case when personal data is involved.”
I continued:
“At present, websites operated by Scottish Government, local authorities, health boards etc. all appear to have blanket copyright statements. I certainly could find no exception to that. With OGL-licensed content, where data is not yet available as Open Data (OD), a page published as HTML could be legitimately scraped and transformed to open data by third parties as the licence would permit that. “
And
“The Scottish Government should mandate this approach not just for the whole of the public sector but also for companies performing contracts on behalf of Government, or who are in receipt of public funding or subsidy.”
In work in 2019, revisited in 2020, I looked in more detail at how local authorities in Scotland conformed to their obligations under RPSI regulations. [10] At that point 25 of 32 did not licence web content under OGL at all. Only one authority got it right, and the remainder, six authorities, had a stab at granting permission.
Why mention this now?
In March 2022 was lead author of a report for The David Hume Institute: “What is Open Data and Why Does it Matter?”[11] In that report, which set out the economic, social, environmental drivers for opening government data, I also highlighted that most councils failed to make their website data and information available for re-use under a clear licence.
Today in a Tweet, Councillor Anthony Carroll, stated “At today’s Glasgow Digital Board, a paper I’ve pushed for passed on @GlasgowCC ‘s website content to have an Open Government License. This means anyone can re-use website content (with attributation) instead of it being copyrighted”. [12] He then reference the DHI paper which I authored.
So, well done to Councillor Carroll and to Glasgow City Council.
Now we need COSLA to push all local authorities to do the same. And, as I asked in 2018 and again in 2022, for Scottish Government to compel all public bodies to fulfil their legal obligations and do this properly.
This page contains quoted text from government pages which are licensed under the Open Government Licence, and a Wikipedia page which is licensed as CC-BY-SA, all of which are referenced at the foot of the article.
Do you ever get the feeling that patterns are emerging, the stars aligning, things coming together, coincidences happening more frequently? Of course there is a lot of magical thinking behind those feelings – but the sessions I’ve attended, books I’ve read, speakers I’ve heard, conversations I’ve participated in this week all point to the importance and significance of SODU 2022 next weekend.
Mid-week I had a conversation with represenatatives from Scottsh Government about a forthcoming refresh of their 2015 Open Data Strategy. I was part of the group which drew up the original strategy. Between the drafting and the publication of final version, any sense that implementing the strategy was mandatory was lost. Some of us warned that this would lead to low adoption and therefore low impact in terms of data being released. We were, it is sad to say, proved correct. Of course the published strategy does say that Scotland’s data is to be open by default, but in practice it is anything but that.
While it is pleasing to hear that the Scottish Government is refreshing its open data strategy, the reason for that refresh is not, as far as I can tell, to make open data manadatory or to put measures in place to benchmark performance. As far as we can ascertain no-one monitors or has monitored the implementation of the 2015 strategy; even to the extent of counting as we did in 2019 the number of datasets released, itself a crude measure, And if I read the runes correctly, that commitment to data open by default might not make it in to the next iteration of the strategy which would be a major leap backwards. Civic society needs to make their feelings known about this in the strongest possible terms.
On Friday I gave a presentation to the University of Aberdeen’s Open Access Week’s Open Data event. I spoke about making open data useful, useable and used. The talk was one of half a dozen. I found the talks by each of the others speakers very good – but the one by Paola Masuzzo very impactful. The scenarios she spoke of in the struggle to get good government data resonated with the fight I and others put up to get Covid Data released in 2020. Paola is on the board of On Data and their website has a lot of great content. Unless you are fluent in Italian you might be better allowing Google to translate it if you open it in Chrome browser.
Another conversation that I have been having is with Jack and Karen, my colleagues in the Open Data Scotland project. This was born at Code The City hack weekends and previous SODU sessions. Should it remain as a CTC project? Should we spin it off as its own Community Interest Company? What is its role beyond being an open data portal? In my view we need to be more of a lobbying, pressure group for OD in Scotland. For example I believe that one week away from SODU2022 we have one person from the whole of the public sector in Scotland attending for only one day. There should at least be be someone there from every council health board, government agency etc. That would mean about 180 government attendees. But if their job doesn’t have open data at its core, and there is no obligation to publish, and no consequences to not doing so, why bother publishing? And why bother attending this weekend event?
It could be argued that since the Digital Strategy 2021 in Scotland does recognise the economic vlaue of open data then some of the core digital team with a responibility for implementing that should be there?
It was higlighted at SODU2021 and 2020, if I recall correctly, that there is a gap between government and civic society, and that the enormous goodwill and positivity of the civic group is a resource which government ignores and squanders. Chatting to someone else this week we puzzled to understand why the public sector is so afraid of engagement? Is there a sense that they know they are failing at OD and seek to avoid questioning? We know that several civil servants say they are committed to OD, but that there is no ministerial support in Scotland. Could that be it? We are a genuinely friendly community – and would welcome them with open arms.
This week I’ve been listening to the audio version of The Moves That Matter: A Grandmaster on the Game of Life by Jonathan Rowson, who I knew when he was a teenager in Aberdeen. At one point Jonathan said something about chess that I’ve long thought about applying to open data. He argues that chess “falls through the cultural cracks and lacks a stable cultural category that honours all of its elements; chess is more than a sport but is less than sport; chess is educational and educative but is not education; chess is full of artistic ideas but is not art; … is in many ways scientific but it not science“. He draws from this that without a cultural category that corresponds to a government department that would fund it, or a media section that would report it, it goes underfunded and unreported. The parallels with open data are deep. What IS open data? How do we get it funding and supported in `government? And recognition and attention?
Today was Wikidata‘s 10th birthday. I’m a big fan of Wikidata – and I train people to use it on behalf of Wikimedia UK. I also give guest lectures at RGU to Master students on the topic. We’ve used Wikidata in many projects at CTC hack events too. I’m convinced that Wikidata has a significant role to play as a place for publicly-curated linked open data – and filling in for missing ‘official’ government data,
Finally, I was contacted by someone who had attended the UK Open Data Camp that we hosted in 2018 in Aberdeen. He’s now living in Orkney and having heard about SODU2022 he’s making the journey down to attend, which is great to hear. We always look forward to catching up with the broader community.
Personally, I can’t wait to be at SODU2022. I hope that you do to and that we can have some meaningful discussions on the points above and much more!
If you haven’t booked yet you can still grab a ticket now.
[Published 29 Oct 2022. Edited 30 Oct to correct spelling errors and for readability.]
This was one of three projects which were worked on during CTC24 – Open In Practice. We asked Jan Ainali, who led the project, to explain it for those who were not present at the weekend event.
Why did we run this project?
On Wikidata, there is a WikiProject for getting all the government agencies of all levels and the whole world properly modeled. Since it is a huge project, every way to try to break it down to bite size pieces are necessary. Work on the UK was already started, so it made a lot of sense trying to complete Scotland.
Work done before CTC24
In October during Scottish Open Data Unconference 2021 we got started on this task. We found some good sources and made fine progress, completing several categories of agencies. By completing here, I mean that we made sure there were items in Wikidata representing the agencies and that they were well enough modeled so that we could query for them. But we weren’t done, and some of the trickiest parts remained.
What we achieved at CTC24, what impact we hope it will have
With a joint effort, we managed to sort out how the judicial system was organized, which was something that remained unclear since the last session. Most time was spent in researching to understand it, and when that was done, it was fairly straightforward to create items for the courts that were missing and to model the others in a way that made it possible to query for them. We also managed to sort out some other small tasks from the last event, and finally we could produce one huge query to get all Scottish agencies at once: https://w.wiki/4TpN
Now we think this is the most complete and up-to-date list of Scottish agencies. If we are wrong about that, we would love feedback so that we can improve it.
What next – how can people get involved?
A few things are happening right now. First, we are importing agencies into the Govdirectory platform that is a more user-friendly view of Wikidata. We have already imported the local authorities, NHS boards and Health and Social Care Partnerships. You can find that data here: https://www.govdirectory.org/united-kingdom/
Since the large query was a bit messy, we will also try to improve the modeling in Wikidata. You are more than welcome to help with this. This will make queries for everybody simpler, and we will continuously be importing more agencies to the platform as we get done.
You can also help by adding contact points like email, official websites, social media accounts to the Wikidata items. You can either use the Wikidata button on the Govdirectory website, or you can go to the WikiProject page on Wikidata and run some queries there to find items to improve. https://www.wikidata.org/wiki/Wikidata:WikiProject_Govdirectory/United_Kingdom
Header Image by Jan Ainali, CC0, via Wikimedia Commons
“Hear me calling, hear me calling loud, If you don’t come soon, I’ll be wearing a shroud.” – Ten Years After (1969)
Introduction
Today marks the tenth anniversary of my involvement with Open Data in Scotland. As I wrote here, back in 2009-2010 I’d been following the work that Chris Taggart and others were doing with open data, and was inspired by them to create what I now believe to have been the first open data published in the public sector in Scotland.
This piece is a reflection of my own views. These views may be the same as those held by colleagues at Code The City or indeed on the civic side of the Open Government Partnership. I’ve not specifically asked other individuals in either group.
While my involvement in, and championing of, open data in Scotland is now a decade long, my enthusiasm for the subject and in the the social and economic benefits it can deliver, is undiminished by my leaving the public sector in 2017 after thirty four years. In fact the opposite is true: the more I am involved in the OD movement, and study what is being achieved beyond Scotland’s narrow borders, the more I am convinced that we are a country intent on squandering a rich opportunity, regardless of our politicians’ public pronouncements.
But the journey has not been easy. primarily due to a lack of direction from Scottish Government and little commitment, resource or engagement at all levels of public service. A friend who reviewed this blog post suggested that I should replace the picture of a birthday cake (above) with one of a naked human back bearing bleeding scars from the our battles. He’s right – it is STILL a battle ten years on.
It is not as if the position in Scotland is getting better. We are moving at a glacial pace. The gap between Scotland and other countries in this regard is widening. I gave a talk earlier this year in which I showed assessments of Scotland, Romania and Kenya’s performance in Open Government (source: https://www.opengovpartnership.org/campaigns/global-report/ Vol 2) and asked the audience to identify which was Scotland.
Extracts from Vol2 to of the Open Gov Partnership report
In February 2020 the European Data Portal published a report – The Economic Impact of Open Data – which sets out a clear economic case for open data. That paper looks at 15 previous studies between 1999 and 2020 which have examined at the market size of open data at national and international levels, measured in terms of GDP of each study’s geographical area.
Taking the average and median values from those reports (1.33% and 1.19% respectively) and an estimated GDP for Scotland (2018) of £170.4bn we can see that the missed opportunity for Scotland is of the order of £2.027bn to £2.266bn per annum. What is the actual value of the local market created by Scottish-created open data? if pushed for a figure I would estimate that it is currently worth a few hundred thousand pounds per annum, and no more. Quite a gap!
Meantime we have the usual suspect of consultants whispering sweetly in the ears of ministers, senior civil servants and council bosses that we should be monetising data, creating markets, selling it. There will be no mention, I suspect of the heavily-subsidised, private sector led, yet failed Copenhagen Data Exchange, I suspect. (Maybe they can make a few bob back selling the domain name! )
You can buy the failed CityDataExchange.com for just $5195
While this commercial approach to data may plug small gaps in annual funding for Scotland, and line the pockets of some big companies in the process, it won’t deliver the financial benefits at a national level of anything like the figures suggested by that EU Data Portal report but it will, in the process, actively hamper innovation and inhibit societal benefits.
I hear lots of institutions saying “we need to sell data” or “we need to sell access rights to these photos” or similar. Yet, in so many cases, the operation of the mechanisms of control; the staffing, administration, payment processing etc. far outstrips any generated income. When I challenged ex colleagues in local government about this behaviour their response was “but our managers want to see an income line” to which we could add “no matter how much it is costing us.” And this tweet from The Ferret on Tuesday of this week is another excellent example of this!
I have also heard lots of political proclamations of “open and transparent” government in Scotland since 2014. Yet most of the evidence points in exactly the opposite direction. Don’t forget, when Covid 19 struck, Scotland’s government was reportedly the only political administration apart from Bolsonaro’s far right one in Brazil to use the opportunity to limit Freedom of Information.
Openness, really?
It is clear that there is little or no commitment to open data in any meaningful way at a Scottish Government level, in local authorities, or among national agencies. This is not to say that there aren’t civil servants who are doing their best, often fighting against political or senior administration’s actions. Public declarations are rarely matched by delivery of anything of substance and conversations with people in those agencies (of which I have had many) paints a grim picture of political masters saying one thing and doing another, of senior management not backing up public statements of intent with the necessary resource commitment and, on more than occasion, suggestions of bad actors actually going against what is official policy.
I mention below that I joined the Open Government Partnership late in 2019. Initially I was enthusiastic about what we might achieve. While there are civil servants working dedicatedly on open government who want to make it work, I am unconvinced about political commitment to it. We really need to get some positive and practical demonstration that Scottish Government are behind us – otherwise I and the other civil society representatives are just assisting in an open-washing exercise.
In my view (and that of others) the press in Scotland does not provide adequate scrutiny and challenge of government. We have a remarkably ineffective political opposition. We also have a network of agencies and quangos which are reliant on the Scottish Government for funding who are unwilling to push back. All of this gives the political side a free pass to spout encouraging words of “open and transparent” yet do the minimum at all times.
We may have an existing Open Data Strategy for Scotland (2015) stating that Scotland’s data is “open by default”, yet my 2019 calculation was that over 95% of the data that could and should be open was still locked up. And there is little movement on fixing that.
We have many examples of agencies doing one thing and saying another, such as Scottish Enterprise extolling the virtues of Open Data yet producing none. Its one API has been broken for many months, I am told.
My good friends at The Data Lab do amazing work on funding MSc and Phd places, and providing funding for industrial research in the application of data science. Their mission is “to help Scotland maximise value from data …” yet they currently offer no guidance on open data, no targeted programme of support, no championing of open data at all, despite the widely-accepted economic advantages which it can deliver. There is the potential for The Data Lab to lead on how Scotland makes the most of open data and to guide government thinking on this!
All of this is not to pick on specific organisations, or hard working and dedicated employees within them. But it does highlight systemic failures in Scotland from the top of government downwards.
Fixing this is an enormous task: one which can only be done by the development of a fresh strategy for open data in Scotland, which is mandated for all public sector bodies, is funded as an investment (recognising the economic potential), and which is rigorously monitored and enforced.
I could go on…. but let’s look at this year’s survey.
In February 2019 I conducted a survey of the state of open data in Scotland. It didn’t paint an encouraging picture. The data behind that survey has been preserved here. A year on, I started thinking about repeating the review.
In the intervening year I’d been involved in quite a bit activity around open data. I had
joined the civic side of the Open Government group for Scotland and was asked to lead for the next iteration of the plan on Commitment Three (sharing information and data) ,
joined the steering group of Stirling University’s research project, Data Commons Scotland,
trained as a trainer for Wikimedia UK, delivering training in Wikidata, Wikipedia and Wiki Commons, and running multiple sessions for Code The City with a focus on Wikidata,
created an open Slack Group for the open data community in Scotland to engage with one another,
created an Open Data Scotland twitter account which has gained almost 500 followers, and
In restarting this year’s review of open data publishing in Scotland my aims were to see what had changed in the intervening 12 months and to increase the coverage of the survey: going broader and deeper and developing an even more accurate picture. That work spilled into March at which point Covid-19 struck. During lockdown I was distracted by various pieces of work. It wasn’t until August, and with a growing sense of the imminence of this 10-year anniversary, that I was galvanised to finish that review.
I am conscious that the methodology employed here is not the cleverest – one person counting only the numbers of datasets produced. This is something I return to later.
The picture in 2020
I broke the review down into sectoral groupings to make it more managable to conduct. By sticking to that I hope to make this overview more readable. The updated Git Hub repo in which I noted my findings is available publicly, and I encourage anyone who spots errors or omissions to make a pull request to correct them. Each heading below has a link to the Github page for the research.
Overall there is little significant positive change. This is one factor which gives rise to concerns about government’s commitment to openness generally and open data specifically; and to a growing cynicism in the civic community about where we go from here.
I reviewed this area in February 2020 and rechecked it in August. Sadly there has been no significant change in the publication of open data by local government in the eighteen months since I last reviewed this. More than a third of councils (13 out of a total of 32) still make no open data provision.
While the big gain is that Renrewshire Council have launched a new data portal with over fifty datasets, most councils have shown little or no change.
Sadly the Highland Council portal, procured as part of the Scottish Cities Alliance’s Data Cluster programme at £10,000’s cost, has vanished. I dont think it ever saw a dataset being added to it. Searching Highland Council’s website for open data finds nothing.
While big numbers of data sets don’t mean much by themselves, the City of Edinburgh Council has a mighty 236 datasets. Brilliant! BUT … none of them are remotely current. The last update to any of them was September 2019. Over 90% of them haven’t been updated since 2016 or earlier.
Similarly Glasgow, which has 95 datasets listed have a portal which is repeatedly offline for days at a time. A portal which won’t load is useless.
Dundee, Perth and Stirling continue to do well. Their offerings are growing and they demonstrate commitment to the long-haul.
Aberdeen launched a portal, more than three years in the planning, populated it with 16 datasets and immediately let their open data officer leave at the end of a short-term contract. Some of their datasets are interesting and useful – but there was no consultation with the local data community about what they would find useful, or deliver benefits locally; all despite multiple invitations from me to interact with that community at the local data meet-ups which I was running in the city.
It was hoped that the programme under the Scottish Cities alliance would yield uniform datasets, prioritised across all seven Scottish Cities, but there is no sign of that happening, sadly. So what you find on all portals or platforms is pretty much a pot-luck draw.
Where common standards exist – such as the 360 Giving standard for the publication of support for charities – organisations should be universally adopting these. Yet this is only used by two of 32 authorities, all of whom have grant-making services. Surely, during a pandemic especially, it would be advantageous to funders and recipients to know who is funding which body to deliver what project?
Councils – Open Government Licence and RPSI
This is a slight aside from the publication of open data, but an important one. If the Scottish Authorities were to adopt an OGL approach to the publication of data and information on their website (as both the Scottish Government’s core site and the Information Commissioner for Scotland do) then we would be able to at least reuse data obtained from those sites. This is not a replacement for publishing proper open data but it would be a tiny step forward.
The table below (source and review data here) shows the current permissions to reuse the content of Scottish Local Authorities’ websites. Many are lacking in clarity, have messy wording, are vague or misunderstand terminologies. They also, in the main, ignore legislation on fair re-use.
Table of local authority adoption of PGL and RPSI
Open Government Licence
The Scottish Government’s own site is excellent and clear: permitting all content except logos to be be reused under the Open Government Licence. This is not true for local authorities. At present only Falkirk and Orkney Councils – two of the smaller ones – allow, and promote OGL re-use of content. There is no good reason why all of the public sector, including local government, should not be compelled to adopt the terms of OGL.
Re-use of Public Sector Information (RPSI) Regulations
Since 2015 the public sector has been obliged by the RPSI Regulations to permit reasonable reuse of information held by local authorities. So, even if Scottish LAs have not yet adopted OGL for all website content, they should have been making it clear for the last five years how a citizen can re-use their data and information from their website.
In my latest trawl through the T&Cs and Copyright Statements of 32 Scottish Local Authorities, I found only 7 referencing RPSI rights there, with 25 not doing so (see the full table above). I am fairly sure that these authorities are breaking the legal obligation on public bodies to provide that information.
Finally, given the presence of COSLA on the Open Government Scotland steering group, the situation with no open data; poor, missing or outdated data; and OGL and PRSI issues needs to be raised there and some reassurance sought that they will work with their member organisations to fix these issues.
The NHS Scotland Open Data platform continues to be developed as a very useful resource. The number of datasets there has more than doubled since last year (from 26 to 73).
None of the fourteen Health Boards publish their own open data beyond what is on the NHS Scotland portal.
Only one of the thirty Health and Social Care Partnerships (HSCPs) publish anything resembling open data: Angus HSCP.
COVID-19 and open data
While we are on health, I’ve wrote (here and here) early in the pandemic about the need for open data to help the better public understanding of the situation, and stimulate innovative responses to the crisis. The statistics team at Scottish Government responded well to this and we’ve started to develop a good relationship. I’ve not followed that up with a retrospective about what did happen. Perhaps I will in time.
It was clear that the need for open data in CV19 situation caught government and health sector napping. The response was slower than it should have been and patchy, and there are still gaps. People find it difficult to locate data when it is on muliple platforms, spread across Scots Govt, Health and NRS. That is, in a microcosm, one of the real challenges of OD in Scotland.
With an open Slack group for Open Data Scotland there is a direct channel that data providers could use to engage the open data community on their plans and proposals. They could also to sound out what data analysts and dataviz specialists would find useful. That opportunity was not taken during the Covid crisis, and while I was OK in the short term with being used as a human conduit to that group, it was neither efficient nor sustainable. My hope is that post SODU 2020, and as the next iteration of the Open Gov Scotland plan comes together we will see better, more frequent, direct engagement with the data community on the outside of Government, and a more porous border altogether.
There is no significant change across the sector in the past 18 months. The vast majority of institutions make no provision of open data. Some have vague plans, many of them historic – going back four years or more – and not acted on.
Lumping Universities and Colleges together, one might expect at a minimum properly structured and licensed open data from every institution on :
courses
modules
events
performance (perhaps some of this is on HESA and SFC sites?)
physical assets
environmental performance
KPI targets and achievements etc.
Of course, there is none of that.
Universities and colleges
I reviewed open data provision of Universities and Colleges around 17 February 2020. I revisited this on 11 August 2020, making minor changes to the numbers of data sets found.
While five of fifteen universities are publishing increasing amounts of data in relation to research projects, most of which are on a CC-0 or other open basis, there continues to be a very limited amount of real operational open data across the sector with loads of promises and statements of intent, some going back several years.
The Higher Education Statistics Agency publishes a range of potentially useful-looking Open Data under a CC-BY-4.0 licence. This is data about insitutions, course, students etc – and not data published by the institutions themselves. But I could identify none of that. Overall, this was very disappointing.
Further, while there are 20 FE colleges. None produces anything that might be classed as open data. A few have anything beyond vague statement of intent. Perhaps City of Glasgow College not only comes closest, but does link to some sources of info and data.
The Crighton Observatory
While doing all of this, I was reminded of the Crighton Institute’s Regional Observatory which was announced to loud fanfares in 2013 and appears to have quietly been shut down in 2017. Two of the team involved say in their Linked In profiles that they left at the end of the project. Even the domain name to which articles point is now up for grabs (Feb 2020).
It now appears (Aug 2020) that the total initial budget for the project was >£1.1m. Given that the purpose of the observatory was to amass a great deal of open data, I have also attempted to find out where the data is that it collected and where the knowledge and learning arising from the project has been published for posterity? I can’t locate it. This FOI request may help. The big question: what benefits did the £1.1m+ deliver?
In February 2019 I found that The Scottish Parliament had released 121 data sets. This covers motions, petitions, Bills, petitions and other procedural data, and is very interesting. This year we find that they have still 121 data sets, so, there are no new data sources.
In fact that number is misleading. In February 2020 I discovered that while 75 of these have been updated with new data, the remaining 46 (marked BETA) no longer work. As of August 2020 this is still the case. Why not fix them, or at worst clear them out to simplfy the finadbility of working data?
Some of these BETA datasets should contain potentially more interesting / useful data e.g. Register of Members Interests but just don’t work. Returning: [“{message: ‘Data is presently unavailable’}”]
I didn’t note the availability of APIs last year, but there are 186 API calls available. Many of these are year-specific. I tested half a dozen and about a third of those returned error messages. I suspect some of these align with the non-functioning historic BETAs.
Sadly the issues raised a year ago about the lack of clarity of the licensing of the data is unchanged. To find the licence, you have to go to Notes > Policy on Use of SPCB Copyright Material. Following the first link there (to a PDF) you see that you have to add “Contains information licenced under the Scottish Parliament Copyright Licence.” to anything you make with it, which is OK. But if you go to the second link “Scottish Parliament Copyright Licence” (another PDF) the wording (slightly) contradicts that obligation. It then has a chunk about OGL but says, “This Scottish Parliament Licence is aligned with OGLv3.0” whatever that means. Why not just license all of the data under OGL? I can’t see what they are trying to do.
Trying to work out the business units within the structure of Scottish Government is a significant challenge in itself. Attempting to then establish which have published open data, and what those data sets are, and how they are licensed, is almost an impossible task. If my checking, and arithmetic are right, then of 147 discrete business units, only 27 have published any open data and 120 have published none.
So we can say with some confidence that the issue with findability of data raised in Feb 2019 is unchanged, there being no central portal for open data in the Scottish public sector or even for Scottish Government. Searching the main Scottish Government website for open data yields 633 results, none of which are links to data on the first four screenfuls. I didn’t go deeper than that.
The Scottish Government’s Statistics Team have a very good portal with 295 Data Sets from multiple organisational-providers. This is up by 46 datasets on last year and includes a two new organisations: The Care Inspectorate and Registers of Scotland. The latter, so far (Aug 2020), has no datasets on the portal.
There are some interesting new entrants into the list of those parts of Scottish Government publishing data such as David MacBrayne Limited which is, I believe, wholly owned by SG and is the parent, or operator of Calmac Ferries Limited. On 1st March 2020 they released a new data platform to get data about their 29 ferry routes. This is very welcome. After choosing the dates, routes and traffic types you can download a CSV of results. While their intent appears to be to make it Open Data, the website is copyright and there is no specific licensing of the data. This is easily fixable.
It is also interesting to contrast Transport Scotland with work going on in England. Transport Scotland’s publication scheme says of open data “Open data made available by the authority as described by the Scottish Government’s Open Data Strategy and Resource Pack, available under an open licence. We comply with the guidance above when publishing data and other information to our website. Details of publications and statistics can be found in the body of this document or on the Publications section of our website.” I searched both without success for any OD. Why not say “we don’t publish any Open Data”? Compare this complete absence of open data with even the single project Open Bus Data for England. Read the story here. Scotland is yet again so far behind!
Summary
In the review of data I’ve shown that little has changed in 18 months. Very few branches of government are publishing open data at all. The landscape is littered with outdated and forgotten statements of good intent which are not acted on; broken links; portals that vanish or don’t work; out of date data; yawning gaps in publication and so on.
The claim of “Open By Default” in the current (2015) Open Data Strategy is misleading and mostly ignored with consequence. The First Minister may frequently repeat the mantra of “Open and Transparent” when speaking or questioned by journalists, but it is easily demonstrable that the administration frequently act in the directly opposite way to that.
The recent situations with Covid-19 and the SQA exams results show Scotland would have found itself in a much better place this year with a mature and well-developed approach to open data: an approach one might have reasonably expected after five full years of “open by default”.
The social and economic arguments for open data are indisputable. These have been accepted by most other governments of the developed world. Importantly, they have also been taken up and acted on by developing nations who have in many cases overtaken Scotland in their delivery of their Open Government plans.
The work I have done in 2019 and in this review is not a sustainable one – i.e. one single volunteer monitoring the activity of every branch and level of government in Scotland. And the methodology is limited to what is achievable by an individual.
A country which was serious about Open Data would have targets and measures, monitoring and open reporting of progress.
It wouldn’t just count datasets published. It would be looking at engagement, the usefulness of data and its integration into education.
It would fund innovation: specifically in the use of open data; in the creation of tools; in developing services to both support government in creating data pipelines, and in helping citizens in data use.
It would co-develop and mandate the use of data standards across the public sector.
It would develop and share canonical lists of ‘things’ with unique identifiers allowing data sets to be integrated.
It would adopt the concept of data as infrastructure on which new products, services, apps, and insights could be built.
I really want Scotland to make the most of the opportunities afforded by Open Data. I wouldn’t have spent ten years at this if I didn’t believe in the potential this offers; nor if I didn’t have the evidence to show that this can be done. I wouldn’t be giving up my time year-on-year researching this, giving talks, organising groups and creating opportunities for engagement.
What is fundamentally lacking here is some honesty from Scottish Government ministers instead of their pretence of support for open data.
Scotland is B, in the centre. Kenya is A, and Romania C.
I could have chosen Mexico, Honduras, Paraguay, Uruguay – or others. All are doing better than Scotland.
I showed how the Italian health authorities were doing just what was needed in the most difficult of circumstances. I explained how, in the absence of an official publication of open data (in an openly-licensed, neutral format, machine-readable format ) I’d taken it on myself on 15th March 2020 to gather and publish the data. My hope was that I’d have to do it for a week or two then the Scottish Government would take over.
And here we are five weeks and one day later and I am still having to do it. Meantime a growing list of websites and applications has developed to use the data which is great but adds to the pressure.
So what’s happened in the intervening time and, more importantly, what hasn’t?
From manual to coded scraping
Originally I was gathering the data manually. Going to the Scottish Government web page and retyping the data into CSVs. This is a terrible practice – open to errors and demanding double and triple checking before pushing to Github. While it looked like my publication was going to be short term that hardly mattered.
But as the weeks dragged by I had to concede that there was no rescue coming from Scottish Government any time soon. Initially it appeared that the daily data was going to be published openly via their statistics platform. That eventually morphed into an additional but different set of data (from National Records of Scotland, not HPS).
So, I resolved to build a scraper – a piece of code that will read the HTML of a webpage and extract the data from that. Sounds easy – but in practice it can be far from it. And when all is said and done it is the most brittle of solutions: as any small change can break the code.
Nested Span Tags on the SG web page
Given how poorly the page was structured (endless nested blank span tags being just one crime against HTML) I didn’t have a great deal of confidence that it could be kept working.
I built it and tested it daily but it wasn’t until 14th April that I was confident enough that it would work daily. Even then it wouldn’t take much to derail it. At that point it was 360 lines of code just to get a few dozens numeric values from a single page.
There is probably some law named after someone wiser than me that says that once you launch a piece of software it will be broken the very next day, and so it did the very next morning. The scraper relies on knowing the structure of a page – finding bulleted lists, tables, and iterating through those structures looking for patterns to match and grabbing the numbers.
Since then the Scottish Government have changed the structure of the page as many as six times, including
making the final item in a bulleted list into a new paragraph on its own right,
removing a table completely,
and today changing the format of numbers in a table to include commas where none were used before.
Last week it looked like we might have an easier solution on our hands: not only did they change the URL of the page with the data, they then without fanfare added a new XLSX spreadsheet with the daily data in it, updated each day. While not a CSV file, it appeared that it would be very useful.
So yesterday I started to code up a routine to
grab the XSLX file,
download that,
save it as a reference copy, then
figure out the worksheet names which have data, not charts,
go to those worksheets,
find the ranges with the data (ignoring comments in rows above the data, to the right of the data, below the data – see image below),
extract that data and write it back to plain CSV files as I was doing wth my original scraper.
A screenshot of one worksheet showing one area of data (green) and three of non-data (red)
Having tested the first part of it yesterday I re-ran it today and it broke. It turns out the URL at which the spreadsheet is published changes from day to day. I suspect that this is as a result of some sort of Content Management System.
All of which means I have to now do another scraper to identify each day’s URL before I can do any of the above.
Why are we in this position?
The current position defies logic. There are so many factors that should have meant that Scottish Government would have this sorted out by now.
I’d identified the need for plain CSV publishing previously, and very publicly, giving good examples.
I’d had an email from contact in SG mentioning two of my CSV files (in the context of the forthcoming NRS data publication).
I work as part of the Civic Society group as part of the Open Government partnership, and I am the lead for open data.
So people know where to find me and interact with me.
I have blogged extensively about what we need – and have emailed contacts at SG.
As part of the planning for Scottish Open Data Unconference I set up a Slack group to which SG contacts were invited – and I believe some signed up.
So there are forms through which, if there was any uncertainty, anyone at SG could ask “what does the data community want?” or “we’re thinking about doing ‘x’ would that work for you?” But there has been no such approach.
Meantime, I’ve spent many 10s of hours for no financial reward doing what Peter Drucker called ‘doing the wrong thing righter.” i.e. allowing the SG to continue to publish data wrongly on the basis that the effort is transferred onto the community to create work-arounds.
After five weeks of doing this daily, I’m absolutely fed up of it. I’m going to formally raise it through the Open Government Partnership with a view to getting the right data, in the right format, published daily.
Code The City 
