scotland – Page 2 – Code The City

Response to Scotland’s Draft Action Plan on Open Government

Note: This blog post originally appeared on codethecity.co.uk in November 2018 and has been archived here with a redirect from the original URL.

The Scottish Government published its draft action plan on 14th November 2018. You can find it here. They are seeking feedback before the 27th November 2018.

Here is my feedback which I sent on 25th November.

Thank you for the chance to feed back on the drafts of the Scottish Open Government Action Plan and Commitments.

These documents are welcome and while they certainly set a path for moving Scotland further in the right direction in terms of openness and transparency, we should remember that those should not be our only aims. We need to ensure that we also address the need to use data and information to fuel innovation, and deliver societal and economic benefits for Scotland.

I have set out below my observations and suggestions in a number of areas which range from the general to the specific.

The public good

Data and information held by the Scottish Government and the public sector should be considered a Public Good. See https://www.nic.org.uk/wp-content/uploads/Data-for-the-Public-Good-NIC-Report.pdf and https://www.gov.uk/government/publications/data-for-the-public-good-government-response/government-response-to-data-for-the-public-good.

To deliver that public good requires freeing up information and data as a matter of course, rather than by exception.

There is one simple thing that could be done with immediate impact, and minimal effort, to free up large amounts of data and information for public re-use: adopt an Open Government Licence (OGL) for all published website information and data on the Scottish Government’s website(s), and other public sector sites, the only exception being where this cannot legally be done, as would be the case when personal data is involved.

The ICO’s own website (http://www.itspublicknowledge.info/home/TermsAndConditions.aspx) takes this approach: “Where the Commissioner is the copyright holder, information is available through the Open Government Licence. This means you have a worldwide, royalty-free, perpetual, non-exclusive licence to use the information, subject to important conditions set out in the licence.”

At present, websites operated by Scottish Government, local authorities, health boards etc. all appear to have blanket copyright statements. I certainly could find no exception to that. With OGL-licensed content, where data is not yet available as Open Data (OD), a page published as HTML could be legitimately scraped and transformed to open data by third parties as the licence would permit that. Currently pages such as this list of planning applications, https://publicaccess.aberdeencity.gov.uk/online-applications/simpleSearchResults.do?action=firstPage contain valuable data but are caught by default, site-wide copyright statements.

Of course, in reality citizens, companies, universities and organisations do scrape website content, but it is done under the radar. This approach results in repeated scraping as the results are not published as open data, and there is consequently limited public benefit. Switching the licensing model to OGL by default, and copyright by exception, would solve this and encourage both innovation and engagement: moving a supplier / consumer relationship to one where data and information are a shared public good.

The Scottish Government should mandate this approach not just for the whole of the public sector but also for companies performing contracts on behalf of Government, or who are in receipt of public funding or subsidy.

Targets for publishing

The Scottish Government’s own Open Data Strategy 2015 commits it to publishing data openly but despite my efforts and those of other contributors to it, the strategy mostly lacks hard targets, and sets overly-modest goals: “The ambitionis for all data by 2017 to be published in a format of 3* or above.” One could ask if all of Scottish Government’s data wasactually published to 3* standard by the end of 2017. If not, how much? Who knows – is this even measured, reported on or published?

Therefore, any new action plan should have harder, more specific targets. It is arguable that the lack of these, and of a clear Open Data Policyfor Government, as I called for in 2015, allows overly-pressed civil servants to have much less focus on publishing open data than is needed, resulting in inadequate resources being applied to that. So, ideally this action plan should be underpinned by policy for the whole of the Scottish public sector to ensure that effort and resource can be targeted on publication.

To support this, the public benefits of open data publishing, both in social and economic terms, should be made clear to all data publishers.

Every FOI request should be assessed on receipt, identifying whether it is for data or whether data publishing would satisfy that and future similar requests. If so, the data set should be set for publication as OD with regular periodic updates.

Statutory obligations

I looked for, but could not see, in the action plan and other document, an acknowledgement of the current statutory obligations on the Scottish Government in this area. Recognising, noting and commenting on these in the document would be a useful reminder of specific existing obligations but would also strengthen broader arguments for OD. The following list is not exhaustive.

There are obligations under the G8 Charter on Open Data https://www.gov.uk/government/publications/open-data-charter.

Further, there are existing clear obligations under The Re-use of Public Sector Information Regulations (2015) https://www.legislation.gov.uk/uksi/2015/1415/contents. There is a handy guide here:

http://www.nationalarchives.gov.uk/documents/information-management/psi–guidance-for-public-sector-bodies.pdf (see pages 22 onwards in particular).

Where specific legislation mandates open publication then this should be made clear, as is the case, for example, under The Public Services Reform (Scotland) (2010), if only to avoid this type of headline: https://www.heraldscotland.com/news/17238918.snp-ministers-missing-their-own-transparency-target/

Another example is the OECD’s “Compendium of good practices on the publication and reuse of open data for Anti-corruption across G20 countries: Towards data-driven public sector integrity and civic auditing”.

https://www.oecd.org/gov/digital-government/g20-oecd-compendium.pdf

Recommendations and best practice

There are many resources available online which demonstrate best practices which Scotland’s public sector should adopt in order to deliver the aims of the action plan. Again, these should be mandated for adoption in the action plan. Some examples follow.

Discoverability

A key part of publishing information and data openly is discoverability. To do this well means understanding and applying best practices. Having standard identifiers, descriptors, taxonomies etc. will aid discoverability. So, all information and data publishing should use best practice, using the correct metadata and appropriate standards such as DCAT / DCAT-AP / DCAT.json.

There are some useful resources to assist in this such as

The Scottish Government has an internal expert on this, who sits on the international standards board. It is imperative that his input is sought, and implemented rigorously, in terms of this application of standards.

Data as infrastructure

We should acknowledge the concept of data as infrastructure. See https://www.nic.org.uk/wp-content/uploads/Data-As-Infrastructure.pdf and https://theodi.org/topic/data-infrastructure/. Publishing to our best ability, based on standards and best practice will allow new products and services be developed for societal and economic benefit, and support innovation.

Reference Data

By using standard identifiers for things, such as UPRNs for properties, USRNs for roads and so on, data from multiple government sources can be aggregated about that object, and we can link items with certainty. If the identifiers are then made public, external data such as those from the private sector, can be amalgamated. There must be a concerted effort to make these identifiers public and re-usable. Instead of what appears to be a starting position of “we can’t do this because of x ” we must shift to “how can we do this and how can we sweep away barriers?” Where no identifiers exist for a specific domain, but it is identified that there would be benefit from having them, these should be created.

General approach to open data

Open Data is not a separate thing or process. The curation, management and publication of data is a continuum starting with the internal processes of the organisation. OD should be seen as the natural end point for all data where it is appropriate to publish openly. By adopting an open data by default approach, as outlined here https://en.wikipedia.org/wiki/Open_by_default effort is expended on publishing, not on finding a reason or way to publish: data will be published as OD unless there are specific legal reasons why it can’t be. There are additional benefits to this, including improvements in data quality, de-duplication and re-use of data internally by other departments or services.

Further, while the draft action plan focuses on statistical data, it needs to be recognised that while publishing statistical data openly, the scope needs to be so much wider: encompassing all branches of the Scottish Government, its directorates, its NDPBs, and other agencies. SG also needs to act as a leader to health boards, local authorities, and to joint health and social care partnerships, and work with others such as Scottish Cities alliance where work is ongoing.

We need to open up reference data, geographical boundaries, transactional data, financial data, in fact anything that need not be closed by default.

National portal

Scotland lacks a national open data portal. While this is not a necessity, in order to aid discovery, it would be an advantage, particularly when we have a growing number of existing places where data is being published across Scotland. Many other countries have national portals (https://www.opendatasoft.com/a-comprehensive-list-of-all-open-data-portals-around-the-world/ ) and some such as Austria have had a federation model of publishing at various levels of government in place for many years. If we get discoverability right, and tools such as Google’s data search engine (https://www.google.com/publicdata/directory) begin to mature, this may be less of an issue.

Geospatial commission

Both the recently-formed geospatial commission and the rapidly changing stance of Ordnance Survey is going to impact on what we can publish – with barriers being removed. This increased liberalism will mean that data which we could not publish 3 months ago will suddenly be publishable. Scottish Government need to be on top of that and acting on it to push out data as soon as it can. Beyond that, they should be routinely pushing OS on issues such as derived data to ensure that barriers to publishing are actively removed. Similarly, if reference data is opened up at a UK level, then the Scottish portion of that data needs to be highlighted by the Scottish Government.

Community Building

The action plan must include commitments to work with the Open Data community in Scotland. It is smaller than it should be since there has been relatively little data of value to work with up to now. Contrast with the position of Transport For London, one single organisation, whose open data as far back 2013 was reported to be responsible for 5,000 developer jobs and 500 apps. The Scots Govt needs to grow the OD community and develop it by being an active part of it; to actively seek input on what data sets would be most useful, to use the community as a sounding board; to gain the trust and support of the community by empowering them to be infomediaries who will build and develop products and services which enable citizens to use the data produced, and make sense of it.

Supporting education

Finally, the publication of open data needs to be seen as an educational resource too. Data should be available for use by schools, colleges and universities. Curricular development should encompass the use of open data. Outreach should work with teachers and lecturers so that children can understand their locality by using data pertinent to them. Honours-year and post-grad students in computing sciences should use open data in their projects. Innovation and entrepreneurship courses should encourage the use of public data. Journalism courses should teach data journalism, and so on.

Ian Watt

etc

Open Data Scotland – a nudge from OD Camp?

Note: This blog post was originally published in November 2018 at CodeTheCity.co.uk and was archived here with redirects from the original URL.

Over the first weekend of November 2018, just over 100 people congregated in Aberdeen to attend the UK Open Data Camp. We’d pushed hard to bring it to Scotland, and specifically Aberdeen, for the first time. The event, the sixth of its type, which follows an unconference model where the attendees set the agenda, has previously taken place in England, Wales and Northern Ireland.

I’m not going to go through what we did over the weekend, you can find plenty of that here and here. There are links to all 44 sessions which took places on this Google doc, and many of those have collaborative notes taken during the sessions.

Instead this is a reflective piece, seeking to understand what OD Camp can show us about the state of Open Data in Scotland and beyond.

Who was there?

Of the 100+ attendees, including camp-makers, we estimate that about 40 were from the public sector. Getting exact numbers is hard – people register in their own name, with their own email addresses, but we think that is a good guess.

While this sounds good, during the pitching session on the first day Rory Gianni asked a question: “Hands up who is here from the Scottish public sector?” Two people’s hands went up out of 100+. Each were from local authorities, Aberdeen and Perth city councils, and a third (also from Aberdeen) joined later on Saturday.

This is really concerning and shows the gulf between what Scotland could, or rather must, be doing and what is actually happening.

The Scottish Public Sector

It is estimated that the Scottish Civil Service encompasses 16,000+ officers. It encompasses 33 directorates, nine executive agencies and around 90 Non-departmental public bodies (NDPBs) plus other odds and ends such as the Crown Office and Procurator Fiscal Service.

Then we have 14 health boards, 32 local authorities, 32 Joint Health and Social Care Partnerships and so on.

All of these should be producing open data.

Reality

Sadly, we are very far from that. Few are of any scale or quality. I’ve written about this extensively in the past including in this blog post and its successor post.

So, if we use attendance by the Scottish public sector, at a free-to-attend event which was arranged for them on their very doorstep, as a barometer of commitment to open data, it is clear that something is rotten in the state of ~~Denmark~~ Scotland.

Three weeks on

Since the event, I’ve reached out to the Scottish Government through two channels. I contacted the Roger Halliday, the Chief Statistician, the senior civil servant with a responsibility for Open data, and responded to a Twitter contact from Kate Forbes, the minister for Public Finance and Digital Economy.

I then had an hour-long conversation with Roger and two of his colleagues. This was a very positive discussion. I took away that there is a genuine commitment to doing things better, underpinned by a realism about capacity and capability to widely deliver publication and engagement with the wider OD community. I have agreed to be part of a round table meeting on OD to be held in the new year – and have expressed a commitment to assist in any way needed to improve things.

Meanwhile

Ironically, in the midst of this three week period, the Scottish Government published its Open Government action plan. This emerged on 14th November and is open for feedback until 27th November. So, if you are quick, you can respond to that – and I encourage you to do so. While this certainly seeks to move things in the right direction in terms of openness and transparency, it is extremely light on open data and committed actions to address some of the issues which I have raised.

My next blog post will be a copy of the feedback which I provide, and on which I am currently working.

And finally

When I started drafting this post I was in a very negative frame of mind as regards the Scottish Open Data scene – and particularly in terms of the public sector. In the intervening period, I launched the Scottish Open Data Action group on Twitter. The thinking behind this was to get together a group of activists to swell the public voice beyond mine and that of ODI Aberdeen.

Given the way things are moving on with the Scottish Government and the positive engagement that has begun, the group, which is in its infancy, may not be needed as a vocal pressure group. Instead we could be a supportive external panel who provide expertise and encouragement as needed. Who knows – let’s see!

It is easier to recycle a fridge than reuse Scottish public sector website content and data!

During the course of Code The City 17: Make Aberdeen Better this weekend we made a startling discovery. It is easier to recycle your old fridge-freezer than to get data and content for re-use from Scottish public sector websites. As a consequence, innovating new solutions to common problems and helping make things easier for citizens is made immeasurably more difficult.

One of the event’s challenges posed was “How do we easily help citizens to find where to recycle item ‘x’ in the most convenient fashion. That was quickly broadened out to ‘dispose of an item” since not everything can be recycled – some might be better reused, and others treated as waste, if it can’t be reused or recycled. With limited kerbside collections, getting rid of domestic items mainly involves taking them somewhere – but where?

With climate change, and the environment on most people’s minds at the moment, and legislative and financial pressures on local authorities to put less to landfill, surely it is in everyone’s interest to make it work as well as it can.

To test how to help people to help themselves by giving advice and guidance, we came up with a list of 12 items to test this on – including a fridge, a phone charger, a glass bottle, and tetra pack carton. On the face of it this should be simple, and probably has been solved already.

The Github Repo

All of Code The City hack weekend projects are based on open data and open source code. We use Github to share that code – and any other digital artefacts created as part of the project. All of this one’s outputs can be found (and shared openly) here.

Initial research

That was where we started: looking to see if the problem has already been solved. There is no point in reinventing the wheel.

We looked for two things – apps for mobile phones, and websites with appropriate guidance.

Aberdeen specific information?

Since we were at an event in Aberdeen we first looked at Aberdeen City Council’s website. What could we find out there?

Not much as it turned out – and certainly not anything useful in an easy-to-use fashion. On the front page there was an icon and group of suggested services for Bins and recycling; none of which were what we were looking for.

Typing recycling into the search box (and note we didn’t at this stage know if our hypothetical item could be recycled) returned the first 15 of 33 results. As shown below.

The results were a strangely unordered list – neither sorted alphabetically nor by obvious themes. So relevant items could be on page 3 of the results. Who wants to read policies if they are trying to dispose of a sofa? Why are two of (we later discovered) five recycling centres shown but three others not? Why would I as a citizen want to find out about trade waste when I just want to get rid of a dodgy phone charger?

Why is there a link to all recycling points (smaller facilities in supermarket carparks or such like, with limited acceptance of items), but apparently not to all centres which cover much more items? Actually there is a link ‘Find Your Nearest Recycling Centre’ (but not your nearest recycling point which are much more numerous). This takes you a map and tabular list of centres and what they accept. And it is easy to miss the search box between the two. No such facility exists for the recycling points.

Open Data?

Perhaps there is open data on the ACC Data portal that we could re-purpose – allowing us to build our own solution? Sadly not – the portal has had the same five data sets for almost two years, and every one of those has a broken link to the WMSes.

If we were in Dundee we could download and use freely their recycling centre data. But not in Aberdeen.

Apps to the rescue?

There are some apps and services that do most of what we are trying to do. For example iRecycle – Iphone and Android is a nice app for Android and iOS that would work were it not for US locations only.

We couldn’t find something for Scotland that worked as an App.

Other sources of information?

Since we drew a blank as far as both Aberdeen City Council and any useable apps, we widened our search.

Recycle For Scotland

The website Recycle For Scotland (RFS) is, on the face of it a useful means to identify what to do with a piece of domestic waste. Oddly, there appears not to be any link to it that we could find from any of the ACC recycling pages.

BUT …… it doesn’t work as well as it could and the content, and data behind it have no clear licence to permit reuse.

The Issues with RFS

Searching the site, or navigating by the menus, for Electrical Items results in a page that is headed “This content was archived on 13th August 2018” – hardly inspiring confidence. No alternative page appears to exist and this page is the one turned up in navigation on the site.

Searching for what to do with batteries in Aberdeen results in a list of shops at least one of which closed down about 18 months ago. Entering a search means entering your location manually – every time you search! This quickly becomes wearing.

While the air of neglect is strong, the site is at least useful compared to the ACC website. But it doesn’t do what we want. Perhaps we could re-use some of the content? No – there is no clear licence regarding reuse of the website’s content.

The site appears to be a rebadged version of Recycle Now, built for Zero Waste Scotland (ZWS). According to ZWS’s Terms and Conditions on their own site, and deeply ironically, you can’t (re)use any materials from that site.

Zero Waste Scotland - zero re-use — Zero Waste Scotland – zero re-use

ZWS are publicly funded by the Scottish Government and the European Regional Development Fund – all public money.

Public funding should equal open licences

We argue that any website operated by a government agency, or department, or NDPB, should automatically be licensed under the Open Government Licence (OGL). And any data behind that site should be licensed as Open Data.

The Scottish Government’s own website is fully licenced under OGL.

Changing the licensing of Recycle For Scotland website, making its code open source, and making its data open would have many benefits.

its functionality could be improved on by anyone
the data could be repurposed in new applications
errors could be corrected by a larger group than a single company maintaining it.

Where did this leave us?

Having failed to identify an app that worked for Scotland, nor interactive guidance on the ACC website, we tried the patchy and, on the face of it, unreliable RFS site. We’d turned to the data and whether we could construct something useable from open data and repurposed, fixed, content over the weekend – this is a hack event after all.

But in this we were defeated – data is wrapped up in web pages: formatted for human readability, not reuse in new apps.

Websites which were set up to encourage re-use and recycling ironically prohibit that as far as their content and data is concerned, and deliberately stifle innovation.

Public funding from the City Council, the Scottish Government and the European Regional Development Fund is used to fund sites which you have paid but elements of which you cannot reuse yourself.

Finally

At a time of climate crisis, which the Scottish Government has announced is a priority action, it can’t be right that not only is it difficult to find ways to divert domestic items from landfill, but also that these Government-funded websites have deliberate measures in place to stop us innovating in order to make access to reuse and recycle easier!

Hopefully politicians, ministers and councillors will read this (please draw it to their attention) and wake up to the fact that Scotland deserves, and needs, better than this.

Only by having an Open Data by default policy for the whole of the Scottish Public Sector, and an open government licence on all websites can we fix these problems through innovation.

After all if the non-functioning Northern Ireland Assembly can come up with an open data strategy that commits the region to open data by default, why on earth can’t Scotland?

See below:

“Northern Ireland public sector data is open by default. Open by default is the first guiding principle that will facilitate and accelerate Open Data publication.”

[Edit – Added 12-Nov-2019]

Postscript

If you are interested to read more about the poor state of Scottish Open Data you might be interested in this post I wrote in February 2019 which also contains links to other posts on the subject:

Scotland’s Open Data, February 2019. An Update.

Sadly, not much has changed in the intervening nine months.

[/Edit]