During the course of Code The City 17: Make Aberdeen Better this weekend we made a startling discovery. It is easier to recycle your old fridge-freezer than to get data and content for re-use from Scottish public sector websites. As a consequence, innovating new solutions to common problems and helping make things easier for citizens is made immeasurably more difficult.
One of the event’s challenges posed was “How do we easily help citizens to find where to recycle item ‘x’ in the most convenient fashion. That was quickly broadened out to ‘dispose of an item” since not everything can be recycled – some might be better reused, and others treated as waste, if it can’t be reused or recycled. With limited kerbside collections, getting rid of domestic items mainly involves taking them somewhere – but where?
With climate change, and the environment on most people’s minds at the moment, and legislative and financial pressures on local authorities to put less to landfill, surely it is in everyone’s interest to make it work as well as it can.
To test how to help people to help themselves by giving advice and guidance, we came up with a list of 12 items to test this on – including a fridge, a phone charger, a glass bottle, and tetra pack carton. On the face of it this should be simple, and probably has been solved already.
The Github Repo
All of Code The City hack weekend projects are based on open data and open source code. We use Github to share that code – and any other digital artefacts created as part of the project. All of this one’s outputs can be found (and shared openly) here.
That was where we started: looking to see if the problem has already been solved. There is no point in reinventing the wheel.
We looked for two things – apps for mobile phones, and websites with appropriate guidance.
Aberdeen specific information?
Since we were at an event in Aberdeen we first looked at Aberdeen City Council’s website. What could we find out there?
Not much as it turned out – and certainly not anything useful in an easy-to-use fashion. On the front page there was an icon and group of suggested services for Bins and recycling; none of which were what we were looking for.
Typing recycling into the search box (and note we didn’t at this stage know if our hypothetical item could be recycled) returned the first 15 of 33 results. As shown below.
The results were a strangely unordered list – neither sorted alphabetically nor by obvious themes. So relevant items could be on page 3 of the results. Who wants to read policies if they are trying to dispose of a sofa? Why are two of (we later discovered) five recycling centres shown but three others not? Why would I as a citizen want to find out about trade waste when I just want to get rid of a dodgy phone charger?
Why is there a link to all recycling points (smaller facilities in supermarket carparks or such like, with limited acceptance of items), but apparently not to all centres which cover much more items? Actually there is a link ‘Find Your Nearest Recycling Centre’ (but not your nearest recycling point which are much more numerous). This takes you a map and tabular list of centres and what they accept. And it is easy to miss the search box between the two. No such facility exists for the recycling points.
Perhaps there is open data on the ACC Data portal that we could re-purpose – allowing us to build our own solution? Sadly not – the portal has had the same five data sets for almost two years, and every one of those has a broken link to the WMSes.
If we were in Dundee we could download and use freely their recycling centre data. But not in Aberdeen.
Apps to the rescue?
There are some apps and services that do most of what we are trying to do. For example iRecycle – Iphone and Android is a nice app for Android and iOS that would work were it not for US locations only.
We couldn’t find something for Scotland that worked as an App.
Other sources of information?
Since we drew a blank as far as both Aberdeen City Council and any useable apps, we widened our search.
Recycle For Scotland
The website Recycle For Scotland (RFS) is, on the face of it a useful means to identify what to do with a piece of domestic waste. Oddly, there appears not to be any link to it that we could find from any of the ACC recycling pages.
BUT …… it doesn’t work as well as it could and the content, and data behind it have no clear licence to permit reuse.
The Issues with RFS
Searching the site, or navigating by the menus, for Electrical Items results in a page that is headed “This content was archived on 13th August 2018” – hardly inspiring confidence. No alternative page appears to exist and this page is the one turned up in navigation on the site.
Searching for what to do with batteries in Aberdeen results in a list of shops at least one of which closed down about 18 months ago. Entering a search means entering your location manually – every time you search! This quickly becomes wearing.
While the air of neglect is strong, the site is at least useful compared to the ACC website. But it doesn’t do what we want. Perhaps we could re-use some of the content? No – there is no clear licence regarding reuse of the website’s content.
ZWS are publicly funded by the Scottish Government and the European Regional Development Fund – all public money.
Public funding should equal open licences
We argue that any website operated by a government agency, or department, or NDPB, should automatically be licensed under the Open Government Licence (OGL). And any data behind that site should be licensed as Open Data.
Changing the licensing of Recycle For Scotland website, making its code open source, and making its data open would have many benefits.
its functionality could be improved on by anyone
the data could be repurposed in new applications
errors could be corrected by a larger group than a single company maintaining it.
Where did this leave us?
Having failed to identify an app that worked for Scotland, nor interactive guidance on the ACC website, we tried the patchy and, on the face of it, unreliable RFS site. We’d turned to the data and whether we could construct something useable from open data and repurposed, fixed, content over the weekend – this is a hack event after all.
But in this we were defeated – data is wrapped up in web pages: formatted for human readability, not reuse in new apps.
Websites which were set up to encourage re-use and recycling ironically prohibit that as far as their content and data is concerned, and deliberately stifle innovation.
Public funding from the City Council, the Scottish Government and the European Regional Development Fund is used to fund sites which you have paid but elements of which you cannot reuse yourself.
At a time of climate crisis, which the Scottish Government has announced is a priority action, it can’t be right that not only is it difficult to find ways to divert domestic items from landfill, but also that these Government-funded websites have deliberate measures in place to stop us innovating in order to make access to reuse and recycle easier!
Hopefully politicians, ministers and councillors will read this (please draw it to their attention) and wake up to the fact that Scotland deserves, and needs, better than this.
Only by having an Open Data by default policy for the whole of the Scottish Public Sector, and an open government licence on all websites can we fix these problems through innovation.
After all if the non-functioning Northern Ireland Assembly can come up with an open data strategy that commits the region to open data by default, why on earth can’t Scotland?
“Northern Ireland public sector data is open by default. Open by default is the first guiding principle that will facilitate and accelerate Open Data publication.”
[Edit – Added 12-Nov-2019]
If you are interested to read more about the poor state of Scottish Open Data you might be interested in this post I wrote in February 2019 which also contains links to other posts on the subject:
At Code The City our objective is help our local community become literate in both technology and data and to use them to full advantage. We help people, organisations and charities to gain the right skills. We are improving what we do at Code The City, and how we do it: changes which are fundamental to making that vision a reality.
Our work up to now
Over the past five years we’ve run 16 Hack Weekends and, in Spring 2018, we started to host monthly data meet-ups. Both things have been very successful but are not the sum total of our ambitions. To deliver those fully we needed a base from which to operate and to grow.
We’re now set up in the ONE Tech Hub, hosted by ONE Codebase. This has cemented our position as part of the local ecosystem. Since moving in six weeks ago we’ve launched the Young City Coders sessions. Our first one, last week attracted 22 keen young people and there is a waiting list for places. We’ll run those twice a month from now on. We’re really grateful for assitance we have received. Inoapps gave us sponsorship to get these sessions started, and both they and the James Hutton Institute donated used laptops.
The immediate future
In another six weeks or so we’ll start a Tech Tribe. That’s the name we’ve given to a programme to get people, and women in particular, into STEM careers and education. Many of them missed the chance first time round. The Data Lab already sponsor our Data Meet-ups and are now sponsoring these sessions, too.
All this educational activity is reliant on volunteer time. Two of our founders, Ian and Bruce, have now become STEM ambassadors. Part of that was getting PVG checks to allow them to work with children and vulnerable adults. We have a handful of others who are going to go through the same process. But, we want to be resilient, and scale up and so we need more people. If you would like to volunteer and get the appropriate certification, please get in touch.
This week also sees the start of the new Aberdeen Python User Group which kicks off on Wednesday. Python is by many measures the most popular, flexible and growing programming language which is used in data science, astronomy, biology, security, web development…. the list is endless.
Our next Hack Weekend will be in November and will address volunteering and civic engagement. We also hope to run another hack weekend in December just before Christmas.
We are planning a springtime event: the Scottish Open Data Unconference. Details will be announced of these very soon.
A picture takes shape
All this is like a jigsaw puzzle, the picture of which is gradually emerging as we fit the pieces together.
By running coding sessions for youngsters and mums, we are starting to help families better understand the potential of data and technology to transform their lives.
By creating Data and Python Meet-ups we are creating networking opportunities. These raise awareness of the good work that is going on in academia and industry. It exposes employers to graduate talent. We help people to share their skills, experience and expertise and to self-organise.
By running hack events we are helping charities and public sector organisations to make the most of the opportunities of digital and data to transform. We also help the local tech community of coders and developers and others to give something back to worthy causes.
By leading projects such as Aberdeen Air Quality we put the creation of data into people’s hands. This demonstrates the potential of collective endeavour for a common cause. The data is made available openly for anyone to build any new product or service. And it offers up the potential for schools and universities to use that data to better understand the local environment.
By running a national unconference we bring specialists, experts, and a wider network to the city to mix with local practitioners. This facilitates discussions at local, regional and national levels and between data users, publishers and academics at every level.
Our charity values. Your values?
In addition to all of the above, Ian, our founder CEO, is a non-executive director of the UK-wide Community Interest Company, Democracy Club. Its strapline is “Our vision is of a country with the digital foundations to support everyone’s participation in democratic life.” Now, Ian has joined the steering group of Scotland’s Open Government Network. He is also now on the board of Stirling University-led project, Data Commons.
The commitment of our charity and its founders is to create that better world underpinned by data and digital, from the ground up. That means running events of many kinds. empowering people, giving them the skills and knowledge they need.
You can do your bit too: come to meet-ups; share your work; be part of a network; becoming a STEM ambassador; coach and mentor others, put something back.
Last weekend we hosted the second Aberdeen Air Quality hack weekend in recent months. Coming out it there are a number of tasks which we need to work on next. While some of these fall to the community to deliver, there are also significant opportunities for us to work with partners.
While the Air Aberdeen website is better, we still need to apply the styling that was created at the weekend.
We’ve established that the DHT022 chips which we use in the standard Luftdaten device model have challenges in working in our maritime climate. They get saturated and stop reporting meaningful values. There is a fix which is to use BME380 chips in their place. These will continue to give humidity and temperature readings, plus pressure, but due to the different technology used will handle the humidity better. Knowing local humidity is important (see weather data below). So, we need to adapt the design of all new devices to use these chips, and retrofit the existing devices with the new chips.
Placement of new devices
We launched in February with a target of 50 sensors by the end of June and 100 by the end of the year. So far attendees have built 55 devices of which 34 are currently, or have recently been, live. That leaves 21 in people’s hands that are still to be registered and turned on. We’re offering help to those hosts to make them live.
Further, with the generous sponsorship of Converged, Codify, and now IFB we will shortly build 30 more devices, and that will take us to a total of 85. We’ve had an approach by a local company who may be able to sponsor another 40. So, it looks like we will soon exceed the 100 target. Where do we locate these new ones? We need to have a plan to strategically place those around the city where they would be most useful which is where the map, above, comes in.
Community plus council?
We really want to work with the local authority on several aspects of the project. It’s not them versus us. We all gain by working together. There are several areas that we could collaborate on, in addition to the strategic placement of future devices.
For example, we’ve been in discussions with the local authority’s education service with a view to siting a box on every one of the 60 schools in the city. That would take us to about 185 devices – far in excess of the target. Doing that needs funding, and while the technology challenge to get them on the network is trivial, ensuring that the devices survive on the exterior of the buildings might be a challenge.
Also, we’ve asked but had no response to our request to co-locate one of our devices on a roadside monitoring station which would allow us to check the correlation between the outputs of the two. We need to pursue that again.
Comparing our data suggests that we can more than fill in gaps in the local council’s data. The map of the central part of Aberdeen in the image above, shows all of the six official sensors (green) and 12 of the 24 community sensors that we have in the city (in red). You can also see great gaps where there are no sensors which again shows the need for strategic placement of the new ones.
We’ve calculated that with a hundred sensors we’d have 84,096,000 data observations per year for the city, all as open data. The local authority, with six sensors each publishing three items of data hourly, have 157,680 readings per annum – which is 0.18% of the community readings (and if we reach 185 devices then ACC’s data is about 0.10% or 1/1000th of the community data) and the latter of course, besides being properly open-licensed, has much greater granularity and geographic spread.
We need to ensure that we gather historic and new weather data and use that to check if adjustments are needed to PM values. Given that the one-person team who was going to work on this at CTC16 disappeared, we need to first set up that weather data gathering, then apply some algorithms to adjust the data when needed, then make that data available.
Engagement with Academia
We need to get the two local universities aboard, particularly on the data science work. We have some academics and post-grads who attend our events, but how do we get the data used in classes and projects? How do we attract more students to work with us? And , again we need to get schools to only hosting the devices but the pupils using the data to understand their local environment?
The cool stuff
Finally, we when we have the data collected, cleaned, and curated, and APIs in place (from the green up through orange to red layers below) we can start to build some cool things (the blue layers).
These might include, but are not limited to:
data science-driven predictive models of forecast AQ in local areas,
public health alerts,
mobile apps to guide you where it is safe to walk, cycle, jog or suggest cleaner routes to school for children,
logging AQ over time and measuring changes,
correlating local AQ with admissions to hospital of cases of COPD and other health conditions
inform debate and the formulation of local government strategy and policy.
As we saw at CTC16, we could also provide the basis for people to innovate using the data. One great example was the hacked LED table-top lamp which changes colour depending on the AQ outside. Others want to develop personalised dashboards.
Join us on 8th and 9th June for a diverse hack weekend as we work together to identify Air Quality issues and fix them.
According to the last air quality report for Aberdeen, we have some of the worst air in Scotland. Even though we live on the coast, and generally have a nice breeze, we still have bad air days. Particulate matter is so small that we can’t see the risks: clear air does not mean clean air. This leads to health issues for some of us, and our families. It also affects how busy the health services are too.
There are some official monitors available for the city, but they don’t have widespread coverage. This means you are unlikely to be able to check the air quality around your house, your child’s school, your cycle commute or where you go running. With your help we would like to change that.
Clean Air Aberdeen sprang from our February Code the City co-design event focusing on air quality. We built some sensors, developed some analytics, and explored how to take this work further. At the end the participants wanted this to continue through an ad-hoc organisation aimed at monitoring air quality in Aberdeen as written up here. The data from every sensor is published as open data which can be used to create new products and services.
As before, we’ll follow our usual co-design approach to gather interesting ideas of projects, and form mixed ability teams from attending students, professionals, developers, and designers to work on ideas over the weekend.
Activities will probably include
Building sensor kits each day
Gathering weather data
Improve monitoring of devices
Use data science to create predictive models
Match open health data with areas of poor AQ
Creating alerts (hitting triggers)
Test for exceeding daily limits etc
Hosting of data for re-use
Governance activity (Saturday)
Better understanding of device limitations (sensitivity, fog, …)
Discussion of any known issues/bugs (getting people to install units once built)
Comms and promotion
Designing UK network meetup for other city projects
And did we mention food? We’ll have great catering from social enterprise The Bread Maker, thanks to sponsorship by Forty Two Studio.
We hope you will join us and encourage friends and colleagues to get involved.
Please don’t delay as the event is only two weeks away.
Update: A write-up of this event which took place on 16-17th February 2019 is available on this page.
How much do you care about the quality of the air you breathe as you walk to work or university, take the kids to school, cycle or jog, or open your bedroom window?
How good is the air you are breathing? How do you know? What are the levels of particulates (PM2.5 or PM10) and why is this important?
When do these levels go up or down? What does that mean?
Who warns you? Where do they get their data, and how good is it?
Where do you get information, or alerts that you can trust?
We aim to sort this in Aberdeen
Partnering with community groups, Aberdeen University and 57 North Hacklab, we are working on a longterm project to build and deploy community-built, and hosted, sensors for PM2.5 and PM10. We aim to have fifty of these in place in the next few months, across Aberdeen. You can see some early ones in place and generating data here.
The first significant milestone of this will be the community workshop we are holding on 16-17 February 2019. If you want to be part of it, you can get a ticket here. But, be quick; they are going quickly.
There are loads of things you can do if you attend.
For a small cost, you can come along and build your own sensor with someone to help you, and take it home to plug into your home wifi. It will then contribute data for your part of the city.
But we will be doing much more than that.
Working with the data
If you have experience in data science or data analysis, or if you want to work with those who do, there are loads of options to work with the data from existing and future sensors.
Allow historical reading to be analysed against the official government sensors for comparison
Use the data; wind speed, humidity… to build live maps of readings to identify sources of emissions.
Compensate readings from sensors against factors which affect pollution levels to attempt to understand the emissions of pollutants in a given area.
2018 has been a really busy year for us. Here are all the things that we delivered.
Open Data Camp
We hosted UK Open Data Camp’s first ever visit North of the border in November. Over a hundred people travelled to Aberdeen for two days of unconferencing where there were 44 sessions run on a variety of data-related topics. Some people went for an Aberdeen version of the Joy Diversion walk around old Aberdeen, and others discovered the pleasure of logging Open Benches. The feedback was overwhelmingly positive and there were loads of write-ups.
Code the City Hack Weekends
We had two great Code The City Events: CTC13 – Hacking our Relationship with Alcohol, and CTC14 – Archaeology. Both were well attended and produced some very interesting results. The first saw us tackling some interesting real-world problems, helping people to overcome problems, and build a machine learning model to predict whether a beer can design would be more likely to be perceived as alcoholic or not. A report of the weekend is being written as an academic paper for a forthcoming health conference!
The second weekend saw us scanning and creating 3D renders of six real skeletons with mobile phones. We also began to create a 3D model of the church in which the dig took place, and generate data from written logs to populate that.
Well done to all who participated. We got some great feedback on each event.
Wearing our ODI hats, we launched the new monthly Data Meetups in April – and managed to squeeze in nine of them this year. These are really well attended, and saw over 300 people in total coming out on a Tuesday night to hear speakers from across the country on a diverse range of data topics. These ranged from Creating a Data Culture in your business, to public Open Data; from the data of Scottish Football to the use of blockchain in Oil and Gas; and from the use of IoT in Agriculture to extracting data from photos published on Flickr in order to assist conservation.
We’ve also been lobbying the Scottish Government and the city council on Open Data, as Ian has been writing on our sister site. That is starting to bear fruit. Aberdeen City Council have soft-launched a new open data platform, and are recruiting a manager for their open data work. While this is good, it is not as impressive as Dundee and Perth‘s new platforms, yet. The Scottish Cities Alliance are recruiting a new programme manager, and Ian has been invited to be part of a round table discussion on the way forward for Open Data hosted by the Scottish Government next February. It sounds like things will start to move in the right direction in 2019!
Ian and Andrew have worked with ODI HQ to run two local workshops, contributing to two national pieces of research: the first on the effects of Peer to Peer markets on accommodation, and a second on what barriers there are to the better use of Ordnance Survey data and services.
Here’s to an equally successful 2019! Have a great festive break folks!
Throughout 2006 an archaeological dig of the East Kirk of the St Nicholas Church was conducted by a team led by the archaeology service of Aberdeen City Council. You can read more of the history here. A large number of skeletal remains and other artefacts were recovered. Written records were created in the form of plans, and log books, and some of these were drawn, then scanned, and a MS Access 2 Database was also created.
Since the end of the dig, some post-excavation analysis of skeletal remains, and other artefacts, has been conducted, but this is far from complete due to a lack of funds.
Saturday – getting started
Following an introduction from Ali Cameron, the dig director, challenges were identified, ideas for tackling those identified, and teams formed around those.
The teams, and their projects, created a pipeline; one feeding the other.
Below we introduce the teams. Each of these will shortly be linked to individual blog posts for each team.
They had two aims:
to re-label photos and bringing them together for the ‘skelelocator’ team,
to lay out skeletons and explore options for 3D scanning using mobile apps and cameras with photos offloaded to laptops for processing.
Working from CSV files (derived from an Access 2 MDB file), JPEG diagrams, Corel Photopaint files and even using original hand-draw plans and log books, this team aimed to create a complete set of data of all excavated remains, allowing these to be plotted in 3D space using X, Y and Z coordinates.
Team Skeleton Bridge
This team set out to create a schema and mesh diagram which would use the data from team skelocator in a format which the unity burials could use.
Team Unity Burials
The members of this team wanted to create a 3D model of the church interior in Unity, and to place skeletal remains accurately in the 3D space, moving from block models to accurate ones. Their initial focus was on setting up the deployment of their basics to GitHub and to speak to the other teams about what the formats of data they could work with in order to move this along faster
Team PR and Marketing
This team was looking at stories that would help drive any fundraising later, and exploring what data might be possible for visualisations.
Saturday 5pm update
The second round of updates at 5pm on Saturday saw each team make progress.
They were using Qlone for scanning from mobile phones and found it takes only a few minutes per bone – this is also the app recommend for this by Historic Scotland. However, they are blocked by the size of the paper grid that is needed under the object being scanned. An A3 sheet is not quite big enough for larger bones. Another group found it took 20 minutes with laptop to produce low res version from camera photos, and all tried pushing completed models to Sketchfab site.
They explored raw files to see what could be extracted with different tools looking at exif data and whether this could be supplemented with data from the dig books if necessary.
Team Skeleton Bridge
It appeared that there was no need for this team as Skelocator could output neutral format files, with agreed content, directly to team Unity Burials. So the team disbanded and members were absorbed into other teams.
Team Unity Burials
They were experimenting with how they can manipulate data from one app to another to provide reference points for when they do have the skeletons to place in the model, and how they might show the metadata of each skeleton too.
Team PR and Marketing
This team were exploring how to use the Microsoft ML libraries to build a chatbot that would use FAQ information about the dig to answer questions.
The Sunday had teams coming together from 9:30-10:30 and most people returned which was good. We saw an update around noon.
The team used Qlone to scan more bones with mobiles. They discovered the resolution was high enough, even on smaller ones, to be able to notice things which hadn’t previously been noticed such as what appear to be sword cuts to a rib bone (white marks) on a person who was known to have been stabbed in the head.
This highlights the importance of scanning the bodies while they are available before re-internment at a later date. The larger scans with Qlone were found to be too big for detail as you had to stand further away and thus lost resolution.
They are now using the Python library Sloth to bring the images to life by extracting text from them and putting this into a JSON file. They also found this enabled a way to position each of the skeletons by marking their locations on the image and then creating a grid for reference from a known fixed location, which could also be used by the unity burial team.
Team Unity Burials
One member worked out a convoluted, but workable, process to get images into Unity from other file formats, while another team member worked on a UI for the public to navigate the model.
Team PR and Marketing (aka skeleton)
They started a chatbot, but found it needs to have its data in a better format, and are starting to work with with cleaning up a list of skeletons for using with dc.js visualisations, as well as a webpage for holding the data from the other teams.
Final presentations on Sunday afternoon – 4pm
Everything is being brought together in Sketchfab so that all models can be found.
They are exploring how to combine smaller models to create bigger ones by exporting them into another app. They discovered the limits of scaling the grid in Qlone to get the best resolution with devices, and how to use photos and a laptop to get a scan of whole skeleton using the right background cloth.
They further pulled data from processed photos with x,y,z locations from a superimposed grid that could be automated with human double-checking to make up for the lack of GPS being available for their tools in 2006, as it is now. This can then be handed to unity burials.
Team Unity Burials
One member found more ways to bring in scanned data from team scoliosis, while the other improved upon the UI for the VR version and demoed the basic model to people with the VR headset.
Team PR and Marketing (aka skeleton)
They updated their google doc spreadsheet and pulled it into the pages at GitHub.io so that it could be queried for skeleton info and finished adding the basic visualisations of this data with dc.js while the skelebot was improved with some personality, but wasn’t as useful as it was hoped it would be.
Take-aways from the weekend
We saw how well cross-functional teams worked as there was usually someone around during the event who could help with something, and that within each team people were able to bring diverse experience to help with issues. This was most apparent during the ‘round ups’ of team effort when people heard what others were doing or trying to do.
We learned that skeletons hundreds of years old, and paper records of 12 years ago ensured durability of information, while a two year old tech gadget from Google was found to be useless after it updated itself as google had stopped the project, so its built in two-camera scanner couldn’t be used. Similarly file formats which were common at the time of the dig, just 12 years ago, were challenging to access.
We also found that these ‘strongly themed’ events work well for participation. We had 28 attendees on Saturday and 25 on Sunday.