Note: This blogpost first appeared on codethecity.co.uk in January 2019 and has been archived here with a redirect from the original URL.
I wrote some recent articles about the state of open data in Scotland. Those highlighted the poor current provision and set out some thoughts on how to improve the situation. This post is about a concrete example of the impact of government doing things poorly.
Ennui: a great spur to experimentation
As the Christmas ticked by I started to get restless. Rather than watch a third rerun of Elf, I decided I wanted to practice some new skills in mapping data: specifically how to make Choropleth Maps. Rather than slavishly follow some online tutorials and show unemployment per US state, I thought it would be more interesting to plot some data for Scotland’s 32 local authorities.
Where to get the council boundaries?
If you search Google for “boundary data Scottish Local Authorities” you will be taken to this page on the data.gov.uk website. It is titled “Scottish Local Authority Areas” and the description explains the background to local government boundaries in Scotland. The publisher of the data is the Scottish Government Spatial Data Infrastructure (SDI). Had I started on their home page, which is far from user-friendly, and filtered and searched, I would have eventually been taken back to the page on the data.gov.uk data portal.
This takes you to a page headed, alarmingly, “Order OS Open Data.” After some lengthy text (which warns that DVDs will take about 28 days to arrive but that downloads will normally arrive within an hour), there then follows a list of fifteen data sets to choose. The Boundary Line option looked most appropriate after reading descriptions.
This was described as being in a proprietary ERSI shapefile format, and being 754Mb of files, with another version in the also proprietary Mapinfo format. Importantly, there was no option for downloading data for Scotland only, which I wanted. In order to download it, I had to give some minimal details, and complete a captcha. On completion, I got the message, “Your email containing download links may take up to 2 hours to arrive.”
There was a very welcome message at the foot of the page: “OS OpenData products are free under the Open Government Licence.” This linked not to the usual National Archives definition, but to a page on the OS site itself with some extra, but non-onerous reminders.
Once the link arrived (actually within a few minutes) I then clicked to download the data as a Zip file. Thankfully, I have a reasonably fast connection, and within a few minutes I received and unzipped twelve sets of 4 files each, which now took up 1.13GB on my hard drive.
Two sets of files looked relevant: scotland_and_wales_region.shp and scotland_and_wales_const_region.shp. I couldn’t work out what the differences were in these, and it wasn’t clear why Wales data is also bundled with Scotland – but these looked useful.
Wrong data in the wrong format
My first challenge was that I didn’t want Shapefiles, but these were the only thing on offer, it appeared. The tutorials I was going to follow and adapt used a library called Folium, which called for data as GeoJson, which is a neutral, lightweight and human readable file format.
I needed to find a way to check the contents of the Shapefiles: were they even the ones I wanted? If so, then perhaps I could convert them in some way.
To check the shapefile contents, I settled on a library called GeoPandas. One after the other I loaded scotland_and_wales_region.shp and scotland_and_wales_const_region.shp. After viewing the data in tabular form, I could see that these are not what I was looking for.
So, I searched again on the Scottish Spatial Infrastructure and found this page. It has a Download link at the top right. I must have missed that.
But when you click on Download it turns out to be a download of the metadata associated with the data, not the data files. Clicking Download link via OS Open Data, further down page, takes you back to the very same link, above.
I did further searching. It appeared that the Scottish Local Government Boundary Commission offered data for wards within councils but not the councils’ own boundaries themselves. For admin boundaries, there were links to OS’ Boundary Line site where I was confronted by same choices as earlier.
Eventually, through frustration I started to check the others of the twelve previously-downloaded Boundary Line data sets and found there was a shape file called “district_borough_unitary_region.shp” On inspection in GeoPandas it appeared that this was what I needed – despite Scottish Local Authorities being neither districts nor boroughs – except that it contained all local authority boundaries for the UK – some 380 (not just the 32 that I needed).
Converting the data
Having downloaded the data I then had to find a way to convert it from Shapefile to Geojson (adapting some code I had discovered on StackOverflow) then subset the data to throw away almost 350 of the 380 boundaries. This was a two stage process: use a conversion script to read in Shapefiles, process and spit out Geojson; write some code to read in the Geojson, covert it to a python dictionary, match elements against a list of Scottish LAs, then write the subset of boundaries back out as a geojson text file.
Using the Geojson to create a choropleth map
I’ll spare the details here, but I then spent many, many hours trying to get the Geojson which I had generated to work with the Folium library. Eventually it dawned on me that while the converted Geojson looked ok, in fact it was not correct. The conversion routine was not producing the correct Geojson.
Having returned to this about 10 days after my first attempts, and done more hunting around (surely someone else had tried to use Scottish LAs as geojson!) I discovered that Martin Crowley had republished on Github boundaries for UK Administrations as Geojson. This was something that had intended to do for myself later, once I had working conversions, since the OGL licence permits republishing with accreditation.
Had I had access to these two weeks ago, I could have used them. With the Scottish data downloaded as Geojson, producing a simple choropleth map as a test took less than ten minutes!
While there is some tidying to do on the scale of the key, and the shading, the general principle works very well. I will share the code for this in a future post.
There is something decidedly user-unfriendly about the SDI approach which is reflective of the Scottish public sector at large when it comes to open data. This raises some specific, and some general questions.
Why can’t the Scottish Government’s SDI team publish data themselves, as the OGL facilitates, rather than have a reliance on OS publishing?
Why are boundary data, and by the looks of it other geographic data, published as ESRI GIS shapefiles or Mapinfo formats rather than the generally more-useable, and much-smaller, GeoJson format?
Why can’t we have Scottish (and English, and Welsh) authority boundaries as individual downloads, rather than bundled as UK-level data, forcing the developer to download unnecessary files? I ended up with 1.13GB (and 48 files) of data instead of a single 8.1MB Scottish geojson file.
What engagement with the wider data science / open community have SDI team done to establish how their data could be useful, useable and used?
How do we, as the broader Open Data community share or signpost resources? Is it all down to government? Should we actively and routinely push things to Google Dataset Search? Had there been a place for me to look, then I would have found the GitHub repo of council boundaries in minutes, and been done in time to see the second half of Elf!
I am always up for a conversation about how we make open data work as it should in Scotland. If you want to make the right things happen, and need advice, or guidance, for your organisation, business or community, then we can help you. Please get in touch. You can find me here or here or fill in this contact form and we will respond promptly.
Note: This blog post originally appeared on codethecity.co.uk in November 2018 and has been archived here with a redirect from the original URL.
The Scottish Government published its draft action plan on 14th November 2018. You can find it here. They are seeking feedback before the 27th November 2018.
Here is my feedback which I sent on 25th November.
Thank you for the chance to feed back on the drafts of the Scottish Open Government Action Plan and Commitments.
These documents are welcome and while they certainly set a path for moving Scotland further in the right direction in terms of openness and transparency, we should remember that those should not be our only aims. We need to ensure that we also address the need to use data and information to fuel innovation, and deliver societal and economic benefits for Scotland.
I have set out below my observations and suggestions in a number of areas which range from the general to the specific.
To deliver that public good requires freeing up information and data as a matter of course, rather than by exception.
There is one simple thing that could be done with immediate impact, and minimal effort, to free up large amounts of data and information for public re-use: adopt an Open Government Licence (OGL) for all published website information and data on the Scottish Government’s website(s), and other public sector sites, the only exception being where this cannot legally be done, as would be the case when personal data is involved.
The ICO’s own website (http://www.itspublicknowledge.info/home/TermsAndConditions.aspx) takes this approach: “Where the Commissioner is the copyright holder, information is available through the Open Government Licence. This means you have a worldwide, royalty-free, perpetual, non-exclusive licence to use the information, subject to important conditions set out in the licence.”
At present, websites operated by Scottish Government, local authorities, health boards etc. all appear to have blanket copyright statements. I certainly could find no exception to that. With OGL-licensed content, where data is not yet available as Open Data (OD), a page published as HTML could be legitimately scraped and transformed to open data by third parties as the licence would permit that. Currently pages such as this list of planning applications, https://publicaccess.aberdeencity.gov.uk/online-applications/simpleSearchResults.do?action=firstPage contain valuable data but are caught by default, site-wide copyright statements.
Of course, in reality citizens, companies, universities and organisations do scrape website content, but it is done under the radar. This approach results in repeated scraping as the results are not published as open data, and there is consequently limited public benefit. Switching the licensing model to OGL by default, and copyright by exception, would solve this and encourage both innovation and engagement: moving a supplier / consumer relationship to one where data and information are a shared public good.
The Scottish Government should mandate this approach not just for the whole of the public sector but also for companies performing contracts on behalf of Government, or who are in receipt of public funding or subsidy.
Targets for publishing
The Scottish Government’s own Open Data Strategy 2015 commits it to publishing data openly but despite my efforts and those of other contributors to it, the strategy mostly lacks hard targets, and sets overly-modest goals: “The ambitionis for all data by 2017 to be published in a format of 3* or above.” One could ask if all of Scottish Government’s data wasactually published to 3* standard by the end of 2017. If not, how much? Who knows – is this even measured, reported on or published?
Therefore, any new action plan should have harder, more specific targets. It is arguable that the lack of these, and of a clear Open Data Policyfor Government, as I called for in 2015, allows overly-pressed civil servants to have much less focus on publishing open data than is needed, resulting in inadequate resources being applied to that. So, ideally this action plan should be underpinned by policy for the whole of the Scottish public sector to ensure that effort and resource can be targeted on publication.
To support this, the public benefits of open data publishing, both in social and economic terms, should be made clear to all data publishers.
Every FOI request should be assessed on receipt, identifying whether it is for data or whether data publishing would satisfy that and future similar requests. If so, the data set should be set for publication as OD with regular periodic updates.
I looked for, but could not see, in the action plan and other document, an acknowledgement of the current statutory obligations on the Scottish Government in this area. Recognising, noting and commenting on these in the document would be a useful reminder of specific existing obligations but would also strengthen broader arguments for OD. The following list is not exhaustive.
Another example is the OECD’s “Compendium of good practices on the publication and reuse of open data for Anti-corruption across G20 countries: Towards data-driven public sector integrity and civic auditing”.
There are many resources available online which demonstrate best practices which Scotland’s public sector should adopt in order to deliver the aims of the action plan. Again, these should be mandated for adoption in the action plan. Some examples follow.
A key part of publishing information and data openly is discoverability. To do this well means understanding and applying best practices. Having standard identifiers, descriptors, taxonomies etc. will aid discoverability. So, all information and data publishing should use best practice, using the correct metadata and appropriate standards such as DCAT / DCAT-AP / DCAT.json.
There are some useful resources to assist in this such as
The Scottish Government has an internal expert on this, who sits on the international standards board. It is imperative that his input is sought, and implemented rigorously, in terms of this application of standards.
By using standard identifiers for things, such as UPRNs for properties, USRNs for roads and so on, data from multiple government sources can be aggregated about that object, and we can link items with certainty. If the identifiers are then made public, external data such as those from the private sector, can be amalgamated. There must be a concerted effort to make these identifiers public and re-usable. Instead of what appears to be a starting position of “we can’t do this because of x ” we must shift to “how can we do this and how can we sweep away barriers?” Where no identifiers exist for a specific domain, but it is identified that there would be benefit from having them, these should be created.
General approach to open data
Open Data is not a separate thing or process. The curation, management and publication of data is a continuum starting with the internal processes of the organisation. OD should be seen as the natural end point for all data where it is appropriate to publish openly. By adopting an open data by default approach, as outlined here https://en.wikipedia.org/wiki/Open_by_default effort is expended on publishing, not on finding a reason or way to publish: data will be published as OD unless there are specific legal reasons why it can’t be. There are additional benefits to this, including improvements in data quality, de-duplication and re-use of data internally by other departments or services.
Further, while the draft action plan focuses on statistical data, it needs to be recognised that while publishing statistical data openly, the scope needs to be so much wider: encompassing all branches of the Scottish Government, its directorates, its NDPBs, and other agencies. SG also needs to act as a leader to health boards, local authorities, and to joint health and social care partnerships, and work with others such as Scottish Cities alliance where work is ongoing.
We need to open up reference data, geographical boundaries, transactional data, financial data, in fact anything that need not be closed by default.
Both the recently-formed geospatial commission and the rapidly changing stance of Ordnance Survey is going to impact on what we can publish – with barriers being removed. This increased liberalism will mean that data which we could not publish 3 months ago will suddenly be publishable. Scottish Government need to be on top of that and acting on it to push out data as soon as it can. Beyond that, they should be routinely pushing OS on issues such as derived data to ensure that barriers to publishing are actively removed. Similarly, if reference data is opened up at a UK level, then the Scottish portion of that data needs to be highlighted by the Scottish Government.
The action plan must include commitments to work with the Open Data community in Scotland. It is smaller than it should be since there has been relatively little data of value to work with up to now. Contrast with the position of Transport For London, one single organisation, whose open data as far back 2013 was reported to be responsible for 5,000 developer jobs and 500 apps. The Scots Govt needs to grow the OD community and develop it by being an active part of it; to actively seek input on what data sets would be most useful, to use the community as a sounding board; to gain the trust and support of the community by empowering them to be infomediaries who will build and develop products and services which enable citizens to use the data produced, and make sense of it.
Finally, the publication of open data needs to be seen as an educational resource too. Data should be available for use by schools, colleges and universities. Curricular development should encompass the use of open data. Outreach should work with teachers and lecturers so that children can understand their locality by using data pertinent to them. Honours-year and post-grad students in computing sciences should use open data in their projects. Innovation and entrepreneurship courses should encourage the use of public data. Journalism courses should teach data journalism, and so on.
Note: This blog post was originally published in November 2018 at CodeTheCity.co.uk and was archived here with redirects from the original URL.
Over the first weekend of November 2018, just over 100 people congregated in Aberdeen to attend the UK Open Data Camp. We’d pushed hard to bring it to Scotland, and specifically Aberdeen, for the first time. The event, the sixth of its type, which follows an unconference model where the attendees set the agenda, has previously taken place in England, Wales and Northern Ireland.
I’m not going to go through what we did over the weekend, you can find plenty of that here and here. There are links to all 44 sessions which took places on this Google doc, and many of those have collaborative notes taken during the sessions.
Instead this is a reflective piece, seeking to understand what OD Camp can show us about the state of Open Data in Scotland and beyond.
Who was there?
Of the 100+ attendees, including camp-makers, we estimate that about 40 were from the public sector. Getting exact numbers is hard – people register in their own name, with their own email addresses, but we think that is a good guess.
While this sounds good, during the pitching session on the first day Rory Gianni asked a question: “Hands up who is here from the Scottish public sector?” Two people’s hands went up out of 100+. Each were from local authorities, Aberdeen and Perth city councils, and a third (also from Aberdeen) joined later on Saturday.
This is really concerning and shows the gulf between what Scotland could, or rather must, be doing and what is actually happening.
Sadly, we are very far from that. Few are of any scale or quality. I’ve written about this extensively in the past including in this blog post and its successor post.
So, if we use attendance by the Scottish public sector, at a free-to-attend event which was arranged for them on their very doorstep, as a barometer of commitment to open data, it is clear that something is rotten in the state of Denmark Scotland.
Three weeks on
Since the event, I’ve reached out to the Scottish Government through two channels. I contacted the Roger Halliday, the Chief Statistician, the senior civil servant with a responsibility for Open data, and responded to a Twitter contact from Kate Forbes, the minister for Public Finance and Digital Economy.
I then had an hour-long conversation with Roger and two of his colleagues. This was a very positive discussion. I took away that there is a genuine commitment to doing things better, underpinned by a realism about capacity and capability to widely deliver publication and engagement with the wider OD community. I have agreed to be part of a round table meeting on OD to be held in the new year – and have expressed a commitment to assist in any way needed to improve things.
Ironically, in the midst of this three week period, the Scottish Government published its Open Government action plan. This emerged on 14th November and is open for feedback until 27th November. So, if you are quick, you can respond to that – and I encourage you to do so. While this certainly seeks to move things in the right direction in terms of openness and transparency, it is extremely light on open data and committed actions to address some of the issues which I have raised.
My next blog post will be a copy of the feedback which I provide, and on which I am currently working.
When I started drafting this post I was in a very negative frame of mind as regards the Scottish Open Data scene – and particularly in terms of the public sector. In the intervening period, I launched the Scottish Open Data Action group on Twitter. The thinking behind this was to get together a group of activists to swell the public voice beyond mine and that of ODI Aberdeen.
Given the way things are moving on with the Scottish Government and the positive engagement that has begun, the group, which is in its infancy, may not be needed as a vocal pressure group. Instead we could be a supportive external panel who provide expertise and encouragement as needed. Who knows – let’s see!
Note: This post was first published in in June 2018 on the CodeTheCity.co.uk blog and has been archived here with redirects from the URL.
There is an oft-repeated joke in which a tourist, completely lost in the Irish countryside, asks an old fellow who is leaning on a gate at the edge of a field, “Can you tell me how to get to Dublin?” After a long pause, the old guy replies, “Well, you don’t want to start from here”.
Previously, I covered open data in Scotland from 2010 to the present. Now I look ahead, but to get there we need to start from where we currently find ourselves.
Scottish open data publishing – now
Earlier this week I spent a couple of hours pulling this list together as a first snapshot of the current open data publishing landscape. The intention is to present an accurate precis of the current state, within the available time to do the research. If I have missed anything, or got it wrong, let me know and I will fix it.
There have been sporadic attempts – of varying size, cost, and success – to make Scottish open data available. How these were initiated or funded varies. Examples include bodies such as Nesta, individual local authorities, groups such as the Scottish Cities Alliance (SCA), and by the Scottish Government.
It appears that at the time of writing that the SCA programme (which is scheduled to run from Jan 2017 to Dec 2018) has so far delivered new open data portals for Dundee, Perth, Inverness and Stirling. Some of these have started to publish a few data sets and others, 18 months into the programme, are still waiting to do so. Aberdeen, who dropped out in late 2017, announced in May of this year that they were back on board, but so far there is no sign of anything being delivered. Even the open data landing pages which Aberdeen City Council once hosted have been removed, although I have heard mention of some GIS open data due to be released.
Edinburgh and Glasgow had existing portals prior to the SCA programme. In Edinburgh’s case, while it has an impressive 234 datasets, only four of these have been updated in the last six months, and no new data sets added for over 15 months.
It looks like Glasgow’s open data platform is a new one, replacing the one created as part of the TSB funded £20m+ future cities programme (PDF. Links to original site have disappeared). It used to host over 370 data sets. The new one has far fewer: 72 . While a number of these have been added to the new portal this year, many of them are historic: e.g. house sales data only go to 2013, which suggests that these are ported from the old site and not updated. It also suggests that around 300 data sets have vanished (temporarily, we hope)!
Some considerable recent attention, and an award, has been given to a project carried out on business rates data by North Lanarkshire Council (NLC) with partners Snook and Urban Tide. This is part of a programme funded by the ODI, and the press coverage reiterates NLC’s claim to have an open by default policy. I know both Urban Tide and Snook, and their work – so I am sure that it will be great. In researching this, though, I could find no data.
In response to my enquiries NLC told me that they are testing a platform. Interestingly, Edinburgh has claimed in the past to have an open-by-default policy for data too, which I cannot locate. Sadly this position is not supported by their own portal’s current condition.
Similarly, Renfrewshire have an Open Data in Renfrewshire page, “The Council is taking a lead role in complying with the Scottish Government’s Open Data Strategy“, the Dublin Code data of which show it was created, and last updated in April 2016. They have a 25-page strategy dated 2015 with a commitment to open data by default, but NO open data that I can find; not even an entry in their website A-Z.
When we created the business case for the SCA data programme, I was quite clear that each of the 7 local authorities were procuring a portal for their city, not for the council. This is an important point. When local councils fail to provide a platform, and data, it is not just the local authority’s image it is tarnished – they are failing citizens, academia and businesses alike.
Where can we see best practice in action?
Sadly, the answer isn’t in Scottish local government, at least for now. Perhaps, when the SCA project reaches its conclusion in December, there will more to show for it. Let us hope.
It also has its Scottish Spatial Data Infrastructure hub which presents geospatial data for both local authority and Scottish Government. This is a welcome resource but is not without its challenges. I’ve not found a way to search by licence (as it appears that not all data is licensed for reuse) and some of the data formats (e.g. WMS or WFS) are more suited to other specialists rather than the general public.
If you know of other high quality examples which I have missed, please let me know.
What stops publishers doing better?
I have had many conversations about this over the years. Since I wrote part one of this mini-series several people contacted me with their thoughts about the Scottish Public Sector’s approach to OD.
Issues which get in the way of doing it right (in no particular order) include:
Lack of awareness (or deliberate ignoring) of legal commitments to provide the data
No open data policy, so it is easy to not do it.
No organisational commitment
A lack of understanding by, and therefore no support from, senior managers / elected members
Short term-ism. Too frequently, OD is delivered as a project, not a long-term commitment
No clear responsibility for OD, or the wrong people / roles with responsibility
Lack of awareness of benefits (to organisation, to economy, to society)
Lack of capacity or lack of skills
Lack of engagement with wider data community
Imagined barriers, or no drive to overcome them
Poor data management, and / or siloed structures within the organisation
Data hoarding by services (“data is power and I am not giving mine up”)
Legal restrictions on publishing (real or imagined)
I can’t deal with all of these in this post – and many are cultural, and need to be resolved by the organisations themselves, but I will address a few of these below. It should also be noted that the G8 Charter on Open Data from 2013, and the Scottish Government’s 2015 Open Data Strategy (PDF), mean that not publishing is simply not an option.
But, licensing …
While not all open data is geospatial, a significant proportion is, and particularly useful one at that. A common barrier which is raised when electing not to release geospatial data is the licensing restrictions imposed by Ordnance Survey. Sometimes these are genuine issues but on occasion these difficulties are either thrown up by over-cautious individuals or those who can’t be bothered to research and tackle them.
I do recognise that the issue is a complex one but it is worth comparing the likes of the Surrey Planning Hub which offers a developer-friendly API returning fully-geocoded planning application data for all local authorities in an entire county, with – for example – the Scottish Spatial Hub which hosts 27 amalgamated spatial datasets for the 32 councils. Only three of these are open data. If you try to download the Planning Application data (c.f. Surrey) you are asked for a authentication key. If you try to register for one you are informed that you can only do so if you work for a local authority.
If anyone can explain why Surrey and Hampshire Hub, and other English authorities such as Camden can offer downloads of planning open data, of this quality and Scotland can’t, I would love to hear that. At its heart I believe there a misunderstanding about the OS Licence for Derived Data and presumption to publish.
This recent blog post by Ben Proctor, based on work at OD Camp Belfast, gives as good a set of guidance, and some debunking of myths. His summary hits the nail on the head: “The vast majority of derived data based on OS information can just be published by public bodies under this ‘presumption to publish’.”
The vast majority of derived data based on OS information can just be published by public bodies under this ‘presumption to publish’.
An announcement last week by Ordnance Survey points in the direction of further openness and a more permissive licensing regime (see this post by Owen Boswarva) and this is ahead of the formation and work of the new Geospatial Commission (GC).
So, perceived licence issues will soon be no longer being a barrier behind which the mis-informed can shelter. If I were working in local government data, or in a Scottish Government directorate, I would be proactively planning now how I am going to start to publish it.
Of course, the issues are not just with with publication.
The aim of the Aberdeen meet-up is to create that city-region local data community: bringing together interested, engaged participants from academia, citizens, community groups, developers, councils, Scots Govt departments, private companies and others. Open data is a large part of that conversation as well as data science and other related topics.
Activity such as that should be happening in each of the seven cities, and across Scotland more generally. While it doesn’t have to be driven by the local council – ours wasn’t – it should open up a meaningful dialogue with authorities: demonstrating need, prioritising specific data, providing feedback, creating opportunities for data use, identifying data in others’ hands, providing advocacy etc.
When we created the Scottish Cities Alliances Open Data programme, one of the four planned work streams, which was well-funded, was the nurturing of local data communities. Our aim was to move from the position of council as provider, and citizen / developer as consumer, of data, to one of all interested parties working together. As I said in that piece, “Going beyond publication, the true value of open data will be realised in its re-use and in the innovative uses to which it is put. The SCA partners will work to develop city-region open data eco-systems where the public, third and private sectors collaborate to encourage data use, economic stimulation and creative approaches to solving civic challenges.”
Going beyond publication, the true value of open data will be realised in its re-use and in the innovative uses to which it is put. The SCA partners will work to develop city-region open data eco-systems where the public, third and private sectors collaborate to encourage data use, economic stimulation and creative approaches to solving civic challenges.
As an adjunct to the SCA programme I put forward a proposal in 2017 for funding of a Code For Scotland programme, based on our experience as part of Code For Europe 2014 (PDF). There was a general support for it, but it was put on hold at the time. Part of the idea behind that was to provide seed support for creating a grass-roots movement to work with data in each Scottish city. In the absence of that, or to complement it should it come about, we do need to create informal networks of open data groups across the country.
So, what’s missing?
I subscribe to the notion that data in public hands is a common asset – and should be treated as such: a concept sometimes referred to as a data commons. Getting to that position entails quite a change in thinking and action. A first step is to create open data, publishing that in a way that easily allows, or encourages, re-use, with clear permissive licensing.
Drawing from the points above, to achieve the potential offered by open data (and already realised in more progressive places) Scotland needs the following:
The Scottish Government, and its many branches, Local Government, Health Boards, and others must now demonstrate a commitment to publish open data. This should follow the Enschede model and implement an open-by-default data policy. This means having the policy formally adopted, published, and committed to by all managers and employees.
We need to stop seeing open data as a separate activity to an organisation’s other data governance. It is not. Open data can be regarded to some degree as a barometer of how well an organisation manages its data assets.
Government need to move beyond ‘build-it-and-they-will-come’ attitude to data publishing, and to work with all partners to make it usable, useful and used.
While publishing static open data at three-star level on the five star model is useful starting point, it is not in itself an end. We need
common standards such as DCAT to enable interoperability between data catalogues.
Collaboration is key – and organisations should band together to share some of the heavy lifting. This increases outcomes, improves standards and reduces local cost. We should bin the ‘not invented here mentality’ and look further afield for where work of high quality is taking place. We should share these best practices like this.
While we are on this topic, individual councils should abandon the “we’re special” mentality which surfaces far too often. All unitary authorities essentially provide the same bunch of services, and have the same core systems from few suppliers. Each would benefit from increased co-operation, collaboration and common approaches to data management and publication.
Academia needs to get behind the open data movement. Data Lab and its many partner universities should be actively involved in the Scottish open data eco-system. MSc programmes (and undergraduate courses) should
regularly use open data, and
teach how to make use of it,
show how to build new and innovative services,
encourage students to be advocates for open data, how to request it, and to act as an intermediary between the publisher and the citizen.
We should then extend that to school pupils – linking it to the curriculum, demonstrating how to use data, interpret and understand it, build with it.
Each local city region, at a minimum, should have an active open data group – and links between these should be encouraged. Funding for this core part of the eco-system should be seen by Scottish and Local government as an investment in the economic and social future of Scotland.
The whole is greater than the sum of the parts: recruiting and involving additional local partners, such as local businesses, to make their data open will significantly enhance what the data community can build or create.
We need more meet-ups, events, competitions, challenges, and opportunities for data scientists, coders, analysts to work with government data.
And what will you do?
As the old adage says, “If you are not part of the solution, you are part of the problem.” So my challenge to you is, whatever your role, what are you doing to bring this about?
For local government in particular, please stop boasting about what you are going to do. Do that thing whatever it is, make it live, publish the data, deliver that policy, live up to promises – then you can boast about it.
If you have a responsibility for data and you aren’t actively pushing for its release as open data then you are probably in the wrong job.
If you are a politician, or elected official, and you are not questioning why your organisation is not publishing open data and supporting its use then you should stop down, and let someone who understands this stand for your seat.
If you work in Economic Development, Community Development, Health, Social Care, Transport, Environmental Services or anything else and you aren’t supporting a movement which can positively impact on your area of specialism then your need rethink your commitment to that role.
If you find yourself justifying why you haven’t published, couldn’t get support, would have liked to but , didn’t get a budget, weren’t supported, ‘legal’ said no, the dog ate your data…. please stop. I have heard excuses from all quarters for the last eight years. No more, please.
If you are an academic and your course neither makes use of, nor champions, open data, then revise your course materials (they could probably do with a refresh anyway).
If you are a developer, citizen, journalist, analyst – whatever – and you are not part of a local data meet-up, join one. If there isn’t one, start one.
If your local authority isn’t publishing open data, ask them why: lobby councillors, use FOI, get in the press.
Stop waiting for others to make stuff happen!
My intention is to write a follow up to this section, with a more detailed list of suggestions, links to handy guides, useful publications etc.
I am always up for a conversation about this. If you want to make the right things happen, and need advice, or guidance, for your organisation, business or community, then we can help you. Please get in touch. You can find me here or here or fill in this contact form and we will respond promptly.
Note: this blog post was first published on 10th June 2018 at CodeTheCity.co.uk has has been archived here with redirects from the original URL.
In this, the first of two posts, I look back over eight years of open data in Scotland, showing where ambition and intent mostly didn’t deliver as we hoped.
In the next part I will look forward, examining how we should rectify things, engage the right people, build on current foundations, and how we all can be involved in making it work as we hoped it would all those years ago.
Let our story begin
“The moon was low down, and there was just the glimmer of the false dawn that comes about an hour before the real one.” – Rudyard Kipling, Plain Tales from the Hills, 1888
The story to-date of Open Data in Scotland is one of multiple false dawns. Are we at last about to witness a real sunrise after so much misplaced hope?
At Data Lab‘s recent Innovation Week in Glasgow, I found myself among 115 other data science MSc students – some of the brightest and best in Scotland – working on seven different industry challenges. You can read more of how that went on my own blog. In this post I want to mention briefly one of the challenges, and the subsequent conversations which it stirred in the room, then on social media and even in email correspondence, then use that to illustrate my false dawn analogy.
The Innovation Week challenge was a simple one compared to some others, and was composed of two questions: “how might we analyse planning applications in light of biodiversity?”, and, “how might we evaluate the cumulative impact of planning applications across the 32 Scottish Local Authorities?”
These are, on the face of it, fairly easily answered. To make it even simpler, as part of the preparation for the innovation week, Data Lab, Snook and others had done some of the leg work for us. This included identifying the NBN Atlas system as one which contained over 219 million sightings of wildlife species, which could be queried easily and which provided open access to its data.
That should have been the difficult part. The other part, getting current and planning application data from the Scottish Local Authorities should have been the easier task – but it was far from it. In fact, in the context of the time available to us, it was impossible as we could find not a single council, of the 32, offering its planning data as open data. You can read more of the particulars of that on my earlier blog posts, above.
This is about the general – not the specific, so, for now, let us set some context to this, and perhaps see how we got to be this point.
The first false dawn.
We start in August 2010, when I was working in Aberdeen City Council. I’d been reading quite a bit about open data, and following what a few enlightened individuals, such as Chris Taggart were doing. It seemed to me so obvious that open data could deliver so much socially and economically – even if no formal studies had by then been published. So, since it was a no-brainer, I arranged for us to publish the first open data in Scotland – at least from a Scottish City Council.
The UK Coalition Government had, in 2010, put Open Data front and centre. They created http://data.gov.uk and mandated a transparency agenda for England and Wales which necessitated publishing Open Data for all LA transactions over £500.
At some point thereafter, in 2011-12 both Edinburgh and Glasgow councils started to produce some open data. Sally Kerr in Edinburgh became their champion – and began working with Ewan Klein in Edinburgh University to get things moving there. I can’t track the exact dates. If you can help me, please let me know and I will update this post.
In 2012 the Open Data Institute was founded by Nigel Shadbolt and Tim Berners-Lee, and from day one championed open data as a public good, stressing the need for effective governance models to protect it.
During 2012 and 2013 Aberdeen, Edinburgh and others started work with Nesta Scotland, run out of Dundee, by the inspirational Jackie McKenzie and her amazing team. They funded two collaborative programmes: Make It Local Scotland and Open Data Scotland.
The former had Aberdeen City Council using Linked Open Data (another leap forward) to create a citizen-driven alerts system for road travel disruption. This was built by Bill Roberts and his team at Swirrl – who have gone on to do more excellent work in this area.
Around mid 2013 Glasgow had received Technology Strategy Board funding for a future cities demonstrator was was recruiting people to work on its open data programme
The second Nesta programme, Open Data Scotland , saw two cities – Aberdeen and Edinburgh – work with two rural councils, East Lothian and Clackmannanshire. Crucially, it linked us all with the Code For Europe movement, and we were able to see at first-hand the amazing work being done in Amsterdam, Helsinki, Barcelona, Berlin and elsewhere. It felt that we were part of something bigger, and unstoppable.
And it gets real-er
In late 2014 the Scottish Government appeared to suddenly ‘get’ open data. They wanted a strategy – so they pulled a bunch of us together two write one. The group included Sally from Edinburgh and me – and the document was published in March 2015. I had pushed for it to have more teeth than it ended up having, and to commit to defined actions, putting an onus on departments and local government to deliver widely on this in a tight timescale.
It did include –
“To realise our vision and to meet the growing interest from users we encourage all organisations to have an Open Data publication plan in place and published on their website by December 2015. Organisations currently publishing data in a format which does not readily support re-use, should within their plan identify when the data will be made available in a more re-usable format. The ambition is for all data by 2017 to be published in a format of 3* or above.” I will come back to this later.
This MUST be it!
In 2016-2017 the Scottish Cities Alliance, supported by the European Regional Development Fund launched a programme: Scotland’s Eighth City – The Smart City. At its heart was data – and more specifically open data. The data project was to feature all seven of Scotland’s cities, working on four streams of work:
data engagement and
The perception was also at that time that the Scottish Government had taken its eye off the ball as regards open data. Little if anything had changed as a result of the 2015 strategy. By working together as 7 cities we could lead the way – and get the other 25 councils, and the Scottish Government themselves, not only to take notice, but also to work with us to put Open Data at the heart of Scottish public services.
The programme would run from Jan 2017 to Dec 2018. I was asked to lead it, which I was delighted to do – and remained involved in that way until I retired from Aberdeen City Council in June 2017.
At that point Aberdeen abandoned all commitment to open data and withdrew from the SCA programme. I have no first-hand knowledge of the SCA programme as it stands now.
Six False Dawns Later
So, after six false dawns what is the state of open data in Scotland: is it where we expected it to be? The short answer to that has be a resounding no.
Some of the developments which should have acted as beacons have been abandoned. The few open data portals we have are, with some newer exceptions, looking pretty neglected: data is incomplete or out of date. There is no national co-ordination of effort, no clear sets of guidance, no agreement on standards or terminologies, no technical co-ordination.
Activity, where it happens at all, is localised, and is more often than not grass-roots driven (which is not in itself a bad thing). In some cases local authorities are being shamed into reinstating their programmes by community groups.
The Scottish Government, with the exception of their SIMD Linked Data work, which was again built by Swirrl, and some statistical data, have produced shamefully little Open Data since their 2015 Strategy.
Despite a number of key players in the examples above still being around, in one role of another, and a growing body of evidence demonstrating ROI, there is strong evidence that Senior Managers, Elected Members and others don’t understand the socio-economic benefits that publishing open data can bring. This is particularly disturbing considering the shrinking budgets and the need to be more efficient and effective.
So, what now?
Given that we have witnessed these many false dawns, when will the real sunrise be? What will trigger that, and what can we each do to make it happen?
During the course of Code The City 17: Make Aberdeen Better this weekend we made a startling discovery. It is easier to recycle your old fridge-freezer than to get data and content for re-use from Scottish public sector websites. As a consequence, innovating new solutions to common problems and helping make things easier for citizens is made immeasurably more difficult.
One of the event’s challenges posed was “How do we easily help citizens to find where to recycle item ‘x’ in the most convenient fashion. That was quickly broadened out to ‘dispose of an item” since not everything can be recycled – some might be better reused, and others treated as waste, if it can’t be reused or recycled. With limited kerbside collections, getting rid of domestic items mainly involves taking them somewhere – but where?
With climate change, and the environment on most people’s minds at the moment, and legislative and financial pressures on local authorities to put less to landfill, surely it is in everyone’s interest to make it work as well as it can.
To test how to help people to help themselves by giving advice and guidance, we came up with a list of 12 items to test this on – including a fridge, a phone charger, a glass bottle, and tetra pack carton. On the face of it this should be simple, and probably has been solved already.
The Github Repo
All of Code The City hack weekend projects are based on open data and open source code. We use Github to share that code – and any other digital artefacts created as part of the project. All of this one’s outputs can be found (and shared openly) here.
That was where we started: looking to see if the problem has already been solved. There is no point in reinventing the wheel.
We looked for two things – apps for mobile phones, and websites with appropriate guidance.
Aberdeen specific information?
Since we were at an event in Aberdeen we first looked at Aberdeen City Council’s website. What could we find out there?
Not much as it turned out – and certainly not anything useful in an easy-to-use fashion. On the front page there was an icon and group of suggested services for Bins and recycling; none of which were what we were looking for.
Typing recycling into the search box (and note we didn’t at this stage know if our hypothetical item could be recycled) returned the first 15 of 33 results. As shown below.
The results were a strangely unordered list – neither sorted alphabetically nor by obvious themes. So relevant items could be on page 3 of the results. Who wants to read policies if they are trying to dispose of a sofa? Why are two of (we later discovered) five recycling centres shown but three others not? Why would I as a citizen want to find out about trade waste when I just want to get rid of a dodgy phone charger?
Why is there a link to all recycling points (smaller facilities in supermarket carparks or such like, with limited acceptance of items), but apparently not to all centres which cover much more items? Actually there is a link ‘Find Your Nearest Recycling Centre’ (but not your nearest recycling point which are much more numerous). This takes you a map and tabular list of centres and what they accept. And it is easy to miss the search box between the two. No such facility exists for the recycling points.
Perhaps there is open data on the ACC Data portal that we could re-purpose – allowing us to build our own solution? Sadly not – the portal has had the same five data sets for almost two years, and every one of those has a broken link to the WMSes.
If we were in Dundee we could download and use freely their recycling centre data. But not in Aberdeen.
Apps to the rescue?
There are some apps and services that do most of what we are trying to do. For example iRecycle – Iphone and Android is a nice app for Android and iOS that would work were it not for US locations only.
We couldn’t find something for Scotland that worked as an App.
Other sources of information?
Since we drew a blank as far as both Aberdeen City Council and any useable apps, we widened our search.
Recycle For Scotland
The website Recycle For Scotland (RFS) is, on the face of it a useful means to identify what to do with a piece of domestic waste. Oddly, there appears not to be any link to it that we could find from any of the ACC recycling pages.
BUT …… it doesn’t work as well as it could and the content, and data behind it have no clear licence to permit reuse.
The Issues with RFS
Searching the site, or navigating by the menus, for Electrical Items results in a page that is headed “This content was archived on 13th August 2018” – hardly inspiring confidence. No alternative page appears to exist and this page is the one turned up in navigation on the site.
Searching for what to do with batteries in Aberdeen results in a list of shops at least one of which closed down about 18 months ago. Entering a search means entering your location manually – every time you search! This quickly becomes wearing.
While the air of neglect is strong, the site is at least useful compared to the ACC website. But it doesn’t do what we want. Perhaps we could re-use some of the content? No – there is no clear licence regarding reuse of the website’s content.
ZWS are publicly funded by the Scottish Government and the European Regional Development Fund – all public money.
Public funding should equal open licences
We argue that any website operated by a government agency, or department, or NDPB, should automatically be licensed under the Open Government Licence (OGL). And any data behind that site should be licensed as Open Data.
Changing the licensing of Recycle For Scotland website, making its code open source, and making its data open would have many benefits.
its functionality could be improved on by anyone
the data could be repurposed in new applications
errors could be corrected by a larger group than a single company maintaining it.
Where did this leave us?
Having failed to identify an app that worked for Scotland, nor interactive guidance on the ACC website, we tried the patchy and, on the face of it, unreliable RFS site. We’d turned to the data and whether we could construct something useable from open data and repurposed, fixed, content over the weekend – this is a hack event after all.
But in this we were defeated – data is wrapped up in web pages: formatted for human readability, not reuse in new apps.
Websites which were set up to encourage re-use and recycling ironically prohibit that as far as their content and data is concerned, and deliberately stifle innovation.
Public funding from the City Council, the Scottish Government and the European Regional Development Fund is used to fund sites which you have paid but elements of which you cannot reuse yourself.
At a time of climate crisis, which the Scottish Government has announced is a priority action, it can’t be right that not only is it difficult to find ways to divert domestic items from landfill, but also that these Government-funded websites have deliberate measures in place to stop us innovating in order to make access to reuse and recycle easier!
Hopefully politicians, ministers and councillors will read this (please draw it to their attention) and wake up to the fact that Scotland deserves, and needs, better than this.
Only by having an Open Data by default policy for the whole of the Scottish Public Sector, and an open government licence on all websites can we fix these problems through innovation.
After all if the non-functioning Northern Ireland Assembly can come up with an open data strategy that commits the region to open data by default, why on earth can’t Scotland?
“Northern Ireland public sector data is open by default. Open by default is the first guiding principle that will facilitate and accelerate Open Data publication.”
[Edit – Added 12-Nov-2019]
If you are interested to read more about the poor state of Scottish Open Data you might be interested in this post I wrote in February 2019 which also contains links to other posts on the subject:
At Code The City our objective is help our local community become literate in both technology and data and to use them to full advantage. We help people, organisations and charities to gain the right skills. We are improving what we do at Code The City, and how we do it: changes which are fundamental to making that vision a reality.
Our work up to now
Over the past five years we’ve run 16 Hack Weekends and, in Spring 2018, we started to host monthly data meet-ups. Both things have been very successful but are not the sum total of our ambitions. To deliver those fully we needed a base from which to operate and to grow.
We’re now set up in the ONE Tech Hub, hosted by ONE Codebase. This has cemented our position as part of the local ecosystem. Since moving in six weeks ago we’ve launched the Young City Coders sessions. Our first one, last week attracted 22 keen young people and there is a waiting list for places. We’ll run those twice a month from now on. We’re really grateful for assitance we have received. Inoapps gave us sponsorship to get these sessions started, and both they and the James Hutton Institute donated used laptops.
The immediate future
In another six weeks or so we’ll start a Tech Tribe. That’s the name we’ve given to a programme to get people, and women in particular, into STEM careers and education. Many of them missed the chance first time round. The Data Lab already sponsor our Data Meet-ups and are now sponsoring these sessions, too.
All this educational activity is reliant on volunteer time. Two of our founders, Ian and Bruce, have now become STEM ambassadors. Part of that was getting PVG checks to allow them to work with children and vulnerable adults. We have a handful of others who are going to go through the same process. But, we want to be resilient, and scale up and so we need more people. If you would like to volunteer and get the appropriate certification, please get in touch.
This week also sees the start of the new Aberdeen Python User Group which kicks off on Wednesday. Python is by many measures the most popular, flexible and growing programming language which is used in data science, astronomy, biology, security, web development…. the list is endless.
Our next Hack Weekend will be in November and will address volunteering and civic engagement. We also hope to run another hack weekend in December just before Christmas.
We are planning a springtime event: the Scottish Open Data Unconference. Details will be announced of these very soon.
A picture takes shape
All this is like a jigsaw puzzle, the picture of which is gradually emerging as we fit the pieces together.
By running coding sessions for youngsters and mums, we are starting to help families better understand the potential of data and technology to transform their lives.
By creating Data and Python Meet-ups we are creating networking opportunities. These raise awareness of the good work that is going on in academia and industry. It exposes employers to graduate talent. We help people to share their skills, experience and expertise and to self-organise.
By running hack events we are helping charities and public sector organisations to make the most of the opportunities of digital and data to transform. We also help the local tech community of coders and developers and others to give something back to worthy causes.
By leading projects such as Aberdeen Air Quality we put the creation of data into people’s hands. This demonstrates the potential of collective endeavour for a common cause. The data is made available openly for anyone to build any new product or service. And it offers up the potential for schools and universities to use that data to better understand the local environment.
By running a national unconference we bring specialists, experts, and a wider network to the city to mix with local practitioners. This facilitates discussions at local, regional and national levels and between data users, publishers and academics at every level.
Our charity values. Your values?
In addition to all of the above, Ian, our founder CEO, is a non-executive director of the UK-wide Community Interest Company, Democracy Club. Its strapline is “Our vision is of a country with the digital foundations to support everyone’s participation in democratic life.” Now, Ian has joined the steering group of Scotland’s Open Government Network. He is also now on the board of Stirling University-led project, Data Commons.
The commitment of our charity and its founders is to create that better world underpinned by data and digital, from the ground up. That means running events of many kinds. empowering people, giving them the skills and knowledge they need.
You can do your bit too: come to meet-ups; share your work; be part of a network; becoming a STEM ambassador; coach and mentor others, put something back.