Note: This post was first published in in June 2018 on the CodeTheCity.co.uk blog and has been archived here with redirects from the URL.
There is an oft-repeated joke in which a tourist, completely lost in the Irish countryside, asks an old fellow who is leaning on a gate at the edge of a field, “Can you tell me how to get to Dublin?” After a long pause, the old guy replies, “Well, you don’t want to start from here”.
Previously, I covered open data in Scotland from 2010 to the present. Now I look ahead, but to get there we need to start from where we currently find ourselves.
Scottish open data publishing – now
Earlier this week I spent a couple of hours pulling this list together as a first snapshot of the current open data publishing landscape. The intention is to present an accurate precis of the current state, within the available time to do the research. If I have missed anything, or got it wrong, let me know and I will fix it.
There have been sporadic attempts – of varying size, cost, and success – to make Scottish open data available. How these were initiated or funded varies. Examples include bodies such as Nesta, individual local authorities, groups such as the Scottish Cities Alliance (SCA), and by the Scottish Government.
It appears that at the time of writing that the SCA programme (which is scheduled to run from Jan 2017 to Dec 2018) has so far delivered new open data portals for Dundee, Perth, Inverness and Stirling. Some of these have started to publish a few data sets and others, 18 months into the programme, are still waiting to do so. Aberdeen, who dropped out in late 2017, announced in May of this year that they were back on board, but so far there is no sign of anything being delivered. Even the open data landing pages which Aberdeen City Council once hosted have been removed, although I have heard mention of some GIS open data due to be released.
Edinburgh and Glasgow had existing portals prior to the SCA programme. In Edinburgh’s case, while it has an impressive 234 datasets, only four of these have been updated in the last six months, and no new data sets added for over 15 months.
It looks like Glasgow’s open data platform is a new one, replacing the one created as part of the TSB funded £20m+ future cities programme (PDF. Links to original site have disappeared). It used to host over 370 data sets. The new one has far fewer: 72 . While a number of these have been added to the new portal this year, many of them are historic: e.g. house sales data only go to 2013, which suggests that these are ported from the old site and not updated. It also suggests that around 300 data sets have vanished (temporarily, we hope)!
Some considerable recent attention, and an award, has been given to a project carried out on business rates data by North Lanarkshire Council (NLC) with partners Snook and Urban Tide. This is part of a programme funded by the ODI, and the press coverage reiterates NLC’s claim to have an open by default policy. I know both Urban Tide and Snook, and their work – so I am sure that it will be great. In researching this, though, I could find no data.
In response to my enquiries NLC told me that they are testing a platform. Interestingly, Edinburgh has claimed in the past to have an open-by-default policy for data too, which I cannot locate. Sadly this position is not supported by their own portal’s current condition.
Similarly, Renfrewshire have an Open Data in Renfrewshire page, “The Council is taking a lead role in complying with the Scottish Government’s Open Data Strategy“, the Dublin Code data of which show it was created, and last updated in April 2016. They have a 25-page strategy dated 2015 with a commitment to open data by default, but NO open data that I can find; not even an entry in their website A-Z.
When we created the business case for the SCA data programme, I was quite clear that each of the 7 local authorities were procuring a portal for their city, not for the council. This is an important point. When local councils fail to provide a platform, and data, it is not just the local authority’s image it is tarnished – they are failing citizens, academia and businesses alike.
Where can we see best practice in action?
Sadly, the answer isn’t in Scottish local government, at least for now. Perhaps, when the SCA project reaches its conclusion in December, there will more to show for it. Let us hope.
The Scottish Government has a great resource in the Scottish Indices of Multiple Deprivation site, using Linked Open Data, that Swirrl developed for them – with excellent visualisation.
It also has its Scottish Spatial Data Infrastructure hub which presents geospatial data for both local authority and Scottish Government. This is a welcome resource but is not without its challenges. I’ve not found a way to search by licence (as it appears that not all data is licensed for reuse) and some of the data formats (e.g. WMS or WFS) are more suited to other specialists rather than the general public.
In the third sector, ALISS with help from Rory Gianni have done some very good work on the ALISS v4 API.
If you know of other high quality examples which I have missed, please let me know.
What stops publishers doing better?
I have had many conversations about this over the years. Since I wrote part one of this mini-series several people contacted me with their thoughts about the Scottish Public Sector’s approach to OD.
Issues which get in the way of doing it right (in no particular order) include:
- Lack of awareness (or deliberate ignoring) of legal commitments to provide the data
- No open data policy, so it is easy to not do it.
- No organisational commitment
- A lack of understanding by, and therefore no support from, senior managers / elected members
- Short term-ism. Too frequently, OD is delivered as a project, not a long-term commitment
- No clear responsibility for OD, or the wrong people / roles with responsibility
- Lack of awareness of benefits (to organisation, to economy, to society)
- Lack of capacity or lack of skills
- Lack of engagement with wider data community
- Imagined barriers, or no drive to overcome them
- Poor data management, and / or siloed structures within the organisation
- Data hoarding by services (“data is power and I am not giving mine up”)
- Legal restrictions on publishing (real or imagined)
I can’t deal with all of these in this post – and many are cultural, and need to be resolved by the organisations themselves, but I will address a few of these below. It should also be noted that the G8 Charter on Open Data from 2013, and the Scottish Government’s 2015 Open Data Strategy (PDF), mean that not publishing is simply not an option.
But, licensing …
While not all open data is geospatial, a significant proportion is, and particularly useful one at that. A common barrier which is raised when electing not to release geospatial data is the licensing restrictions imposed by Ordnance Survey. Sometimes these are genuine issues but on occasion these difficulties are either thrown up by over-cautious individuals or those who can’t be bothered to research and tackle them.
I do recognise that the issue is a complex one but it is worth comparing the likes of the Surrey Planning Hub which offers a developer-friendly API returning fully-geocoded planning application data for all local authorities in an entire county, with – for example – the Scottish Spatial Hub which hosts 27 amalgamated spatial datasets for the 32 councils. Only three of these are open data. If you try to download the Planning Application data (c.f. Surrey) you are asked for a authentication key. If you try to register for one you are informed that you can only do so if you work for a local authority.
If anyone can explain why Surrey and Hampshire Hub, and other English authorities such as Camden can offer downloads of planning open data, of this quality and Scotland can’t, I would love to hear that. At its heart I believe there a misunderstanding about the OS Licence for Derived Data and presumption to publish.
This recent blog post by Ben Proctor, based on work at OD Camp Belfast, gives as good a set of guidance, and some debunking of myths. His summary hits the nail on the head: “The vast majority of derived data based on OS information can just be published by public bodies under this ‘presumption to publish’.”
The vast majority of derived data based on OS information can just be published by public bodies under this ‘presumption to publish’.
An announcement last week by Ordnance Survey points in the direction of further openness and a more permissive licensing regime (see this post by Owen Boswarva) and this is ahead of the formation and work of the new Geospatial Commission (GC).
So, perceived licence issues will soon be no longer being a barrier behind which the mis-informed can shelter. If I were working in local government data, or in a Scottish Government directorate, I would be proactively planning now how I am going to start to publish it.
Of course, the issues are not just with with publication.
Is there an open data community in Scotland?
There is currently no easily-defined Scottish network of those interested in OD. Apart from our own ODI Aberdeen, our monthly Data Meetup in Aberdeen and the regular work we do under the charity Code The City banner, there are some groups such as the Open Knowledge Network branches in Edinburgh and Glasgow, currently with no scheduled meetings . There are various tech meet-ups. There is a Scottish Linked Data Interest Group which meets sporadically. And MBN who, with Data Lab, support our Aberdeen Data Meetup, also run a related one in Glasgow. There is no umbrella group, or co-ordination yet.
The aim of the Aberdeen meet-up is to create that city-region local data community: bringing together interested, engaged participants from academia, citizens, community groups, developers, councils, Scots Govt departments, private companies and others. Open data is a large part of that conversation as well as data science and other related topics.
Activity such as that should be happening in each of the seven cities, and across Scotland more generally. While it doesn’t have to be driven by the local council – ours wasn’t – it should open up a meaningful dialogue with authorities: demonstrating need, prioritising specific data, providing feedback, creating opportunities for data use, identifying data in others’ hands, providing advocacy etc.
When we created the Scottish Cities Alliances Open Data programme, one of the four planned work streams, which was well-funded, was the nurturing of local data communities. Our aim was to move from the position of council as provider, and citizen / developer as consumer, of data, to one of all interested parties working together. As I said in that piece, “Going beyond publication, the true value of open data will be realised in its re-use and in the innovative uses to which it is put. The SCA partners will work to develop city-region open data eco-systems where the public, third and private sectors collaborate to encourage data use, economic stimulation and creative approaches to solving civic challenges.”
Going beyond publication, the true value of open data will be realised in its re-use and in the innovative uses to which it is put. The SCA partners will work to develop city-region open data eco-systems where the public, third and private sectors collaborate to encourage data use, economic stimulation and creative approaches to solving civic challenges.
As an adjunct to the SCA programme I put forward a proposal in 2017 for funding of a Code For Scotland programme, based on our experience as part of Code For Europe 2014 (PDF). There was a general support for it, but it was put on hold at the time. Part of the idea behind that was to provide seed support for creating a grass-roots movement to work with data in each Scottish city. In the absence of that, or to complement it should it come about, we do need to create informal networks of open data groups across the country.
So, what’s missing?
I subscribe to the notion that data in public hands is a common asset – and should be treated as such: a concept sometimes referred to as a data commons. Getting to that position entails quite a change in thinking and action. A first step is to create open data, publishing that in a way that easily allows, or encourages, re-use, with clear permissive licensing.
Drawing from the points above, to achieve the potential offered by open data (and already realised in more progressive places) Scotland needs the following:
- The Scottish Government, and its many branches, Local Government, Health Boards, and others must now demonstrate a commitment to publish open data. This should follow the Enschede model and implement an open-by-default data policy. This means having the policy formally adopted, published, and committed to by all managers and employees.
- We need to stop seeing open data as a separate activity to an organisation’s other data governance. It is not. Open data can be regarded to some degree as a barometer of how well an organisation manages its data assets.
- Government need to move beyond ‘build-it-and-they-will-come’ attitude to data publishing, and to work with all partners to make it usable, useful and used.
- While publishing static open data at three-star level on the five star model is useful starting point, it is not in itself an end. We need
- Collaboration is key – and organisations should band together to share some of the heavy lifting. This increases outcomes, improves standards and reduces local cost. We should bin the ‘not invented here mentality’ and look further afield for where work of high quality is taking place. We should share these best practices like this.
- While we are on this topic, individual councils should abandon the “we’re special” mentality which surfaces far too often. All unitary authorities essentially provide the same bunch of services, and have the same core systems from few suppliers. Each would benefit from increased co-operation, collaboration and common approaches to data management and publication.
- Academia needs to get behind the open data movement. Data Lab and its many partner universities should be actively involved in the Scottish open data eco-system. MSc programmes (and undergraduate courses) should
- regularly use open data, and
- teach how to make use of it,
- show how to build new and innovative services,
- encourage students to be advocates for open data, how to request it, and to act as an intermediary between the publisher and the citizen.
- We should then extend that to school pupils – linking it to the curriculum, demonstrating how to use data, interpret and understand it, build with it.
- Each local city region, at a minimum, should have an active open data group – and links between these should be encouraged. Funding for this core part of the eco-system should be seen by Scottish and Local government as an investment in the economic and social future of Scotland.
- The whole is greater than the sum of the parts: recruiting and involving additional local partners, such as local businesses, to make their data open will significantly enhance what the data community can build or create.
- We need more meet-ups, events, competitions, challenges, and opportunities for data scientists, coders, analysts to work with government data.
And what will you do?
As the old adage says, “If you are not part of the solution, you are part of the problem.” So my challenge to you is, whatever your role, what are you doing to bring this about?
For local government in particular, please stop boasting about what you are going to do. Do that thing whatever it is, make it live, publish the data, deliver that policy, live up to promises – then you can boast about it.
If you have a responsibility for data and you aren’t actively pushing for its release as open data then you are probably in the wrong job.
If you are a politician, or elected official, and you are not questioning why your organisation is not publishing open data and supporting its use then you should stop down, and let someone who understands this stand for your seat.
If you work in Economic Development, Community Development, Health, Social Care, Transport, Environmental Services or anything else and you aren’t supporting a movement which can positively impact on your area of specialism then your need rethink your commitment to that role.
If you find yourself justifying why you haven’t published, couldn’t get support, would have liked to but , didn’t get a budget, weren’t supported, ‘legal’ said no, the dog ate your data…. please stop. I have heard excuses from all quarters for the last eight years. No more, please.
If you are an academic and your course neither makes use of, nor champions, open data, then revise your course materials (they could probably do with a refresh anyway).
If you are a developer, citizen, journalist, analyst – whatever – and you are not part of a local data meet-up, join one. If there isn’t one, start one.
If your local authority isn’t publishing open data, ask them why: lobby councillors, use FOI, get in the press.
Stop waiting for others to make stuff happen!
My intention is to write a follow up to this section, with a more detailed list of suggestions, links to handy guides, useful publications etc.
And finally
I am always up for a conversation about this. If you want to make the right things happen, and need advice, or guidance, for your organisation, business or community, then we can help you. Please get in touch. You can find me here or here or fill in this contact form and we will respond promptly.
Header Photo by Hans-Peter Gauster on Unsplash
Code The City 
One thought on “Scottish Open Data – how do we get there?”