Category: open data

The Open Government Licence, and the re-use of Public Sector Information

– why so poorly understood or adopted in Scotland?

Public Sector Information (PSI) is information that has been created by Public Bodies. In 2003 The EU published “Directive 2003/98/EC on the re-use of public sector information, known as the PSI Directive. This is an EU directive that stipulates minimum requirements for EU member states regarding making public sector information available for re-use. This directive provides a common legislative framework for this area. The Directive is an attempt to remove barriers that hinder the re-use of public sector information throughout the Union.” [1]

In 2005 the UK Government introduced its own regulations. [2] And ten years later the it introduced the Reuse of Public Sector Information Regulations 2015 [3] which set out the framework for UK public sector bodies sharing information and data in line with those requirements. The National Archives produced guidance for both public bodies, and for would-be re-users of the info and data. [4]

Re-use means using public sector information for a purpose different from the one for which it was originally produced, held or disseminated.

Why is PSI valuable

“Public sector information constitutes a vast, diverse and valuable pool of resources. In Market Assessment of Public Sector Information (commissioned by the Department for Business Innovation and Skills in 2013), the value of public sector information to consumers, businesses and the public sector itself in 2011/12 was estimated to be approximately £1.8 billion (in 2011 prices). [5]

“Re-use of public sector information provides enormous opportunities for economic and social benefits, while also promoting transparency and accountability of the public sector.” [5]

Why PSI must be reusable

Re-use of public sector information stimulates the development of innovative new information products and services in the UK and across Europe, thus boosting the information industry.

“The Re-use of Public Sector Information Regulations 2015 [have been] in force from 18 July 2015. They build on the prior 2005 Regulations which removed obstacles to the re-use of public sector information. The 2015 Regulations harmonise and relax the conditions of re-use for public sector information, and bring the cultural sector into scope. The 2015 Regulations continue to improve transparency, fairness and consistency among public sector bodies and re-use of their information. [6]

How is information and data licensed for re-use?

Public bodies, up until 2010 were obliged to make information and data available under a PSI Click-Use Licence. [7] In 2010 this was superseded by V1.0 of the Open Government Licence ( or OGL). The latter has been updated twice and is now at version 3.0. [8]

Where is information and data in Scotland licenced for re-use?

While several local authorities, The Scottish Government and a few government agencies have published open data either on dedicated portals or in sections of their websites explicitly licensed under Open Government Licence (OGL 3), almost no organisations make website content so available for re-use despite an obligation to do so for almost 20 years.

When I worked at Aberdeen City Council, trying to get senior managers in numerous services and departments to make information and data available under OGL or its predecessor was a non-starter. The legal department at the time didn’t support it. And any conversation pushed it further down the line as a ‘nice-to-do’. And we weren’t alone. No council that I can recall was doing this as standard for their websites.

The one notable, and welcome exception, that I am aware of, was The Scottish Government. Since at least 2015 they’ve permitted reuse under OGL, albeit behind a Crown Copyright link at the foot of each website page. But Since April 2022 they explicitly had an open licence statement on each page. Unfortunately this doesn’t carry through to other public body sites, at least not explicitly. With over 180 to check [8] it is difficult to be absolutely certain.

The OGL and Copyright statement on the current Scottish Government Website

I personally, and with various groups, have campaigned for the Scottish public sector to meet their obligations under RPSI 2015. In November 2018 I responded to the consultation on the Scotland Draft Action Plan on Open Government[9]:

“There is one simple thing that could be done with immediate impact, and minimal effort, to free up large amounts of data and information for public re-use: adopt an Open Government Licence (OGL) for all published website information and data on the Scottish Government’s website(s), and other public sector sites, the only exception being where this cannot legally be done, as would be the case when personal data is involved.”

I continued:

“At present, websites operated by Scottish Government, local authorities, health boards etc.  all appear to have blanket copyright statements. I certainly could find no exception to that. With OGL-licensed content, where data is not yet available as Open Data (OD), a page published as HTML could be legitimately scraped and transformed to open data by third parties as the licence would permit that. “

And

“The Scottish Government should mandate this approach not just for the whole of the public sector but also for companies performing contracts on behalf of Government, or who are in receipt of public funding or subsidy.”

In work in 2019, revisited in 2020, I looked in more detail at how local authorities in Scotland conformed to their obligations under RPSI regulations. [10] At that point 25 of 32 did not licence web content under OGL at all. Only one authority got it right, and the remainder, six authorities, had a stab at granting permission.

Why mention this now?

In March 2022 was lead author of a report for The David Hume Institute: “What is Open Data and Why Does it Matter?”[11] In that report, which set out the economic, social, environmental drivers for opening government data, I also highlighted that most councils failed to make their website data and information available for re-use under a clear licence.

Today in a Tweet, Councillor Anthony Carroll, stated “At today’s Glasgow Digital Board, a paper I’ve pushed for passed on @GlasgowCC ‘s website content to have an Open Government License. This means anyone can re-use website content (with attributation) instead of it being copyrighted”. [12] He then reference the DHI paper which I authored.

So, well done to Councillor Carroll and to Glasgow City Council.

Now we need COSLA to push all local authorities to do the same. And, as I asked in 2018 and again in 2022, for Scottish Government to compel all public bodies to fulfil their legal obligations and do this properly.

Ian Watt

10 March 2023

References

  1. https://en.wikipedia.org/wiki/Directive_on_the_re-use_of_public_sector_information (CC-BY-SA V3)
  2. https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/regulations/
  3. https://www.legislation.gov.uk/uksi/2015/1415/contents/made
  4. https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/about-psi/psi-valuable/
  5. https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/about-psi/psi-must-re-usable/
  6. https://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/uk-government-licensing-framework/open-government-licence/
  7. https://opendata.scot/organizations/
  8. https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/
  9. https://codethecity.org/2019/11/15/response-to-scotlands-draft-action-plan-on-open-government/
  10. https://github.com/watty62/SOD/blob/master/OGL_content.md
  11. https://davidhumeinstitute.org/research-1/research-what-is-open-data
  12. https://twitter.com/_anthonycarroll/status/1633903590740770841?s=61&t=XsLcJSnpR1TgzOpce8dhhQ

Header Image: http://www.nationalarchives.gov.uk/information-management/government-licensing/ogl-symbol.htm

This page contains quoted text from government pages which are licensed under the Open Government Licence, and a Wikipedia page which is licensed as CC-BY-SA, all of which are referenced at the foot of the article.

[Edited for clarity 11 March 2023]

Statement from SODU2022

The third annual Scottish Open Data Unconference (SODU2022) took place in Aberdeen on 5–6 November 2022 in Aberdeen and online. Over the two days, attendees primarily from civic society, participated in 31 themed discussions. These covered diverse topics such as Open Data’s contribution to the economy; Policy, Strategy and Legislation of Open Data; The Technology behind Open Data Scotland; and The Ownership of Properties in Aberdeen City Centre. The organisers and participants wish to thank The Data Lab for their generosity in sponsoring the event, and ONE Tech Hub for providing the venue for the unconference.

The following statement was developed from the final session of the weekend. It has been collated and published on behalf of those at the unconference and in wider civic society by the organisers of the event, Code the City. 

Why open data matters

Making data (primarily government data) available under an open licence has a number of identified benefits. These may be categorised as humanitarian, social, economic, performance and environmental [1][2]. 

The Current State of Open Data In Scotland

While the Scottish Government publicly accepts the value of open data, the reality is that delivery on strategy and policy lags far behind what should be delivered. 

The introduction to the 2015 Open Data Strategy for Scotland [3] identifies many of the benefits and reasons for making data open. The Scottish Government’s participation in the international Open Government Partnership [4] is based on the social benefits of improving transparency and accountability, and enhancing citizen participation. The Scottish Government’s Digital Strategy 2021 [5] identifies the potential economic benefits of open data through innovation and entrepreneurship. In this context it is estimated that the value to Scotland’s economy of this, if done well, calculated as a percentage of GDP would be of the order of £2.23bn per annum. 

That said, apart from some small and isolated instances of very good open data publishing in Scotland the majority of public bodies (thought to number 179 but ironically there is no central set of open data to corroborate that) including health boards, local councils, universities, government departments etc publish no, or very little open data. 

It is not clear who is the lead for open data for the public sector in Scotland. Where does responsibility sit for implementation of the various strategies and plans? Whose role is it to monitor performance, report on that, or measure impact. And in each of the 179 or thereabouts public bodies, who has the responsibility to ensure that their organisation is implementing open data to deliver the multiple benefits noted above?

When open data is produced it often suffers from many issues:

  • Sporadic publishing
  • Lack of consistency or standards – for example across all 32 local authorities
  • Lack of current and up-to-date datasets  
  • Gaps in categories of published data by organisations  
  • Short-termism which sees projects commit £10,000s of funding then shutdown publishing platforms and delete the data

However, in civic society there is a small but thriving community for open data. They have, in addition to organising and attending three annual unconferences for Open Data, built, maintained and enhanced the Open Data Scotland portal [6].This project has provided a range of opportunities beyond making what open data exists in Scotland findable for the first time. It has also shared knowledge and expertise, and encourages community participation in the development of the site.

Threats or opportunities

Working with open data increases the breadth of experiences of students developers, analysts, businesses, and others, and increases opportunities. Participation in the development of projects such as Open Data Scotland continues to provide educational opportunities as well as societal benefits.

The lack of open data, provided to high standards, universally, and in a sustained manner is a barrier to innovation and a threat to transparency and accountability. 

Short term data publishing projects with no longevity are a risk to business continuity for those who are dependent on it.   

There is a broken feedback loop to publishers. Individuals and organisations, especially businesses, which are using open data need to be vocal about it. Otherwise, it is only natural for publishers to assume that their data is not being used, deem it to be without value, which may lead to termination of publication.

The government has created strategies and plans, but without mandate to publish for every public body. This has been shown over seven years not to work. [7]

There is a double threat to the provision of open data in Scotland. While the Scottish Government is refreshing its 2015 Open Data Strategy, the community fear that the new version will not be fully consulted on and that it will seek to water down the commitment which the Scottish Government made to data being made ‘open by default’. This would be a retrograde step and act against the intentions of the 2021 Digital Strategy and participation in the Open Government Partnership. 

In addition the Retained EU Law (Revocation and Reform) Bill [8] being put before the UK parliament would sweep away the UK’s Re-use of PSI Regulations 2015 on which basis data which has not been explicitly published as open data can be sought and used. As Jeni Tennison of ODI noted “[without these regulations] and with unprecedented pressures on the public purse, public bodies will start charging for data. Services using it will become more expensive. Some will close down. Others just won’t be built.” [9] This would kill open data in Scotland and stifle delivery of the Digital Strategy 2021

What Scotland Needs

We have identified that for Open Data to flourish in Scotland we need better commitment to, and engagement from, all organisations, bodies, or people with an interest in open data and the benefits it can deliver. We also have the following specific requests. 

From Government

We need oversight of delivery of open data in the Scottish Government to be a specific role – not be spread over multiple roles and structures. 

We need the refresh of the Open Data Strategy to be fully consulted on. It needs to strengthen, not weaken, the Scottish Government’s commitment to open data publishing. It needs to introduce regulations obliging each publicly-funded body in Scotland to deliver open data. It needs to identify  a specific role in each public sector organisation which will have a responsibility for implementing the strategy. 

It needs to provide a monitoring framework for ministers on progress in implementing the strategies which involve open data. This needs to be reported on publicly.

The Scottish Government needs to join with civic society in lobbying the UK government to retain R-PSI regulations and be prepared to introduce Scottish legislation to replace it if this fails. 

For businesses to use open data to deliver new products and services as outlined in the Digital Strategy, businesses need reliable, regular publishing without changes. Charging for data is a barrier to innovation and needs to cease. 

From Business

Anyone, including businesses, who are using open data need to be open about it. They should avoid agreeing contracts which oblige them to be silent about open data use to provide services which are only deliverable using open data. If Government is to be persuaded to make data available and keep making it available the use needs to be seen. We need businesses to join with civic society in lobbying for more, better open data.

From Education sector

We need Scotland’s schools, colleges and universities to use open data in their curriculum. We need teaching staff to better understand what open data is, how it can be used and to use it in course work. We need colleges and universities to publish open data which can be re-used too. 

From The Third Sector

The third sector needs to engage with civic society in promoting the use and benefits of open data. They need to demand that data is published openly which they can use. They need to educate their trustees and staff about the benefits of open data – such as the use of standards and platforms including 360 Giving.[10]

From The Press

We need those in the press to participate in the conversations about open data, to get involved with civil society groups, to challenge government and be involved with the broader community.  We need a press which understands the potential and power of open data and to help share the case for it and stories about open data. 

From Civic Society

We need the public to understand what open data is and what it can potentially deliver. We can help by providing access to education, resources such as the Open Data Scotland portal, and success stories from using open data. Government and the education sector have a responsibility here too. We need the public to tell us about challenges and success stories. 

[1]  https://www.europeandataportal.eu/en/training/what-open-data

[2] https://opendatahandbook.org/guide/en/why-open-data/ 

[3] https://www.gov.scot/publications/open-data-strategy/ 

[4] https://www.gov.scot/policies/improving-public-services/open-government-partnership/ 

[5] https://www.gov.scot/publications/a-changing-nation-how-scotland-will-thrive-in-a-digital-world/  

[6] https://opendata.scot 

[7] https://codethecity.org/2019/11/15/scotlands-open-data-february-2019-an-update/ 

[8] https://bills.parliament.uk/bills/3340 

[9] https://twitter.com/jenit/status/1576140360820330496?s=61&t=nI2hETRy_yo0zmhc13ycCg 

[10] https://www.threesixtygiving.org/ 

Header Photo by Hannah Busing on Unsplash

One week to SODU2022 – time to start thinking

Do you ever get the feeling that patterns are emerging, the stars aligning, things coming together, coincidences happening more frequently? Of course there is a lot of magical thinking behind those feelings – but the sessions I’ve attended, books I’ve read, speakers I’ve heard, conversations I’ve participated in this week all point to the importance and significance of SODU 2022 next weekend.

Mid-week I had a conversation with represenatatives from Scottsh Government about a forthcoming refresh of their 2015 Open Data Strategy. I was part of the group which drew up the original strategy. Between the drafting and the publication of final version, any sense that implementing the strategy was mandatory was lost. Some of us warned that this would lead to low adoption and therefore low impact in terms of data being released. We were, it is sad to say, proved correct. Of course the published strategy does say that Scotland’s data is to be open by default, but in practice it is anything but that.

In 2019 I estimated that over 95% of public sector data that should be open remained closed. And one look at the automatically-generated live table of local authority open data coverage shows what a long way there is to go!

While it is pleasing to hear that the Scottish Government is refreshing its open data strategy, the reason for that refresh is not, as far as I can tell, to make open data manadatory or to put measures in place to benchmark performance. As far as we can ascertain no-one monitors or has monitored the implementation of the 2015 strategy; even to the extent of counting as we did in 2019 the number of datasets released, itself a crude measure, And if I read the runes correctly, that commitment to data open by default might not make it in to the next iteration of the strategy which would be a major leap backwards. Civic society needs to make their feelings known about this in the strongest possible terms.

On Friday I gave a presentation to the University of Aberdeen’s Open Access Week’s Open Data event. I spoke about making open data useful, useable and used. The talk was one of half a dozen. I found the talks by each of the others speakers very good – but the one by Paola Masuzzo very impactful. The scenarios she spoke of in the struggle to get good government data resonated with the fight I and others put up to get Covid Data released in 2020. Paola is on the board of On Data and their website has a lot of great content. Unless you are fluent in Italian you might be better allowing Google to translate it if you open it in Chrome browser.

Another conversation that I have been having is with Jack and Karen, my colleagues in the Open Data Scotland project. This was born at Code The City hack weekends and previous SODU sessions. Should it remain as a CTC project? Should we spin it off as its own Community Interest Company? What is its role beyond being an open data portal? In my view we need to be more of a lobbying, pressure group for OD in Scotland. For example I believe that one week away from SODU2022 we have one person from the whole of the public sector in Scotland attending for only one day. There should at least be be someone there from every council health board, government agency etc. That would mean about 180 government attendees. But if their job doesn’t have open data at its core, and there is no obligation to publish, and no consequences to not doing so, why bother publishing? And why bother attending this weekend event?

It could be argued that since the Digital Strategy 2021 in Scotland does recognise the economic vlaue of open data then some of the core digital team with a responibility for implementing that should be there?

It was higlighted at SODU2021 and 2020, if I recall correctly, that there is a gap between government and civic society, and that the enormous goodwill and positivity of the civic group is a resource which government ignores and squanders. Chatting to someone else this week we puzzled to understand why the public sector is so afraid of engagement? Is there a sense that they know they are failing at OD and seek to avoid questioning? We know that several civil servants say they are committed to OD, but that there is no ministerial support in Scotland. Could that be it? We are a genuinely friendly community – and would welcome them with open arms.

This week I’ve been listening to the audio version of The Moves That Matter: A Grandmaster on the Game of Life by Jonathan Rowson, who I knew when he was a teenager in Aberdeen. At one point Jonathan said something about chess that I’ve long thought about applying to open data. He argues that chess “falls through the cultural cracks and lacks a stable cultural category that honours all of its elements; chess is more than a sport but is less than sport; chess is educational and educative but is not education; chess is full of artistic ideas but is not art; … is in many ways scientific but it not science“. He draws from this that without a cultural category that corresponds to a government department that would fund it, or a media section that would report it, it goes underfunded and unreported. The parallels with open data are deep. What IS open data? How do we get it funding and supported in `government? And recognition and attention?

Today was Wikidata‘s 10th birthday. I’m a big fan of Wikidata – and I train people to use it on behalf of Wikimedia UK. I also give guest lectures at RGU to Master students on the topic. We’ve used Wikidata in many projects at CTC hack events too. I’m convinced that Wikidata has a significant role to play as a place for publicly-curated linked open data – and filling in for missing ‘official’ government data,

Finally, I was contacted by someone who had attended the UK Open Data Camp that we hosted in 2018 in Aberdeen. He’s now living in Orkney and having heard about SODU2022 he’s making the journey down to attend, which is great to hear. We always look forward to catching up with the broader community.

Personally, I can’t wait to be at SODU2022. I hope that you do to and that we can have some meaningful discussions on the points above and much more!

If you haven’t booked yet you can still grab a ticket now.

[Published 29 Oct 2022. Edited 30 Oct to correct spelling errors and for readability.]

Header photo by israel palacio on Unsplash

SODU 2022

The Scottish Open Data Unconference (SODU) is back for a third year.

Since our inception in 2014 Code The City has consistently championed Open Data. Our trustees set up ODI Aberdeen, the only Scottish node of the Open Data Institute. As an organisation, and as individual trustees, we’ve worked to highlight the case to government for open data, to use open data, to make data open, and to educate others in the benefits and opportunities of open data. The highly-successful Open Data Scotland portal, built by a team of community volunteers to address a gap in Government provision, grew from conversations at previous SODU events and hack weekends which we hosted.

Why open data is important

Making data, particularly government data, open has very many benefits – from transparency to citizen empowerment; from providing fuel for innovation to improving government efficiency and much more. The Open Data Handbook is a great source of further background on this. In early 2020 the EU Data portal published a meta-study of the economic value of open data in early 2020. From this we extrapolated that the economic potential to Scotland, if OD publishing were done well, could be in excess of £2 billion per annum. Our trustee, Ian Watt published a research paper What is Open Data and Why Does it Matter? with the David Hume Institute which is available here under an CC-VY-SA 4 open licence.

A brief history of SODU

Back in 2020 we came up with the idea of a Scottish Open Data Unconference – an event which anyone with an interest in open data could attend (whether an activist or mildly curious; whether in Government, civic society, academia, or industry). The event was in part inspired by our hosting UK Open Data Camp in 2018 but our intention was that it would target our growing network. It wouldn’t just be about OD in Scotland but would look worldwide to help Scottish attendees develop their understanding of OD, create small networks to address challenges locally, help publishers connect with their data consumers and vice versa.

Then Covid got in the way – and our March 2020 event (for which we had 107 attendees signed up) was pushed back and run as an online event in September with just a third of that number. We repeated it online again in October 2021 with a similar number of attendees.

We’re now delighted that SODU 2022 is back as a physical event at the ONE Tech Hub, Aberdeen which is a great space for meeting and working together. We’ll also have high quality catering to look forward to! And we’re doing all we can to make it as safe as possible here.

What to expect

Based on previous attendance at SODU 2020 and 2021, and similar events we’ve run, we expect a broad mix of attendees. There will be a strong presence from civic society, with others from academia, the press and the IT / Data industry. W are also hopeful that we will see some attendees from government, given their policy / strategy / legislative obligation to publish data.

Each day will be different, depending on who is there and what they want to discuss (see format below). Some will attend to show their work or share research; others to run practical sessions. Some will offer informal educational slots; others will seek support for projects; and others still will seek or provide feedback on plans, or data provision.

The key to a successful event is the mix of attendees and people bringing a positive, open attitude, and a willingness to engage.

Format

This event will follow the open space format, where participants create the agenda each morning for the sessions that will happen that day. This is the time to pitch your questions to the many people who will be there, and to tell others about your project too.

Sessions in open space always work best when there is an interactive aspect so that the participants talk as much as the person chairing the session. This short video will tell you more about open space. The amazing thing is that it scales really well from 20 people to over 1000 people.

Booking


We’ve several types of tickets available.

  • Weekend-long ones for students at just £15,
  • General weekend tickets for just £25,
  • Daily tickets at £15 for either the Saturday or Sunday, and
  • Weekend online tickets for just £10.

And if the price is a barrier to your attendance just tell us and we’ll sort something out for you.

These events work best with a broad mix of attendees. No matter what your skillset, interest area or knowledge there is something for you – and something you can contribute. The write-ups of the 2020 and the 2021 events will remind you of how much can be achieved by a group of dedicated people over just a few hours!

Book here.

We hope that you can join us.

Ian, Karen, Andrew, Pauline, Bruce
Code The City
Charity SC047835

The Od-Bods project: update from CTC24

Why did we run this project? 

Theoretically, with the 2015 Scottish Government commitment to data being “open by default”, we should have universal publication of appropriate data as open data. In reality Scotland is very poorly served with Open Data. Few local authorities publish any, and those who do have little consistency. Beyond councils the picture is, if anything, even worse. Finding data is all but impossible. We set out to make data more findable, identify who is making data available and, perhaps as importantly, those who are not. We began with local government.

This work is a starting point, not an end point. 

Work done before CTC24

  • We wrote this blog post about this project to accompany the work done at CTC23, the forerunner to this event. 

What we achieved at CTC24, what impact we hope it will have

What challenges we have faced/are currently facing?

  • As always, lack of (good) engagement  with public sector.
  • There is no standardisation in how and where local government published its data.
  • Gathering data and cleaning it to output in a presentable format is currently a manual and laborious process.

What next – how can people get involved?

  • We have a page explaining the project, what our objectives are and what the plan is: https://opendata.scot/about/
  • There’s a big list of GitHub issues to be worked on here: https://github.com/OpenDataScotland/the_od_bods/issues 
  • The current milestone for Q1 2022 is to improve our data that we have gathered so far:
    • Fix known bugs with API calls
    • Tidy up inconsistent dataset tags
    • Identify and locate any missing data from the 32 local authorities which we haven’t found yet
    • Add more data features/metadata if possible

Join us at CTC25 to work on the project issues and work toward our next milestone!