Tag: opendata

Do we know if we know what Open Data is?

A guest post by Karen Jewell, a Data Scientist who attended SODU2020. 

I went into the weekend of SODU, headset and coffee at the ready, thinking that as SODU was both my first experience of an unconference and of Open Data, I wouldn’t be able to participate much but that I could take the opportunity to learn from the brighter and more informed voices around me. Well, it turns out I was quite wrong about my involvement with Open Data.

In the networking sessions of the first day, I introduced myself as someone who didn’t work at all with Open Data, had no experience of it and was here to learn about it. Yet as the event carried on through the day, many discussions and concepts seemed familiar to me and in the afternoon of the first day I had that “ah-hah!” moment. I realised it wasn’t true that I did not work with Open Data, I did, and actually had done so quite a bit in the last 12 months. I just had not realised that is what it was called.

Open Data is data which is not owned or controlled, and is free for use and distribution. Having only just completed my studies in a MSc Data Science at the Robert Gordon University 3 weeks prior, free data was pretty critical to my work as a student. Not only was I able to practice concepts using freely available datasets, 3 of my 8 taught modules required me to source my own dataset for that module’s assessment. To rephrase that, I needed Open Data to complete my degree. Data Scientists are aware of Kaggle, the UCI ML repository, and a quick online search for Scotland’s data will return the Scottish Government’s statistics portal. We see these sources as free data we can practice on, but we may not have recognised it as Open Data, I certainly didn’t until SODU took my blinkers off.

Coming out of SODU, I started to wonder how many other people were in the same metaphorical boat. Were they not answering the call for involvement because they did not realise the availability of Open Data affected them too? To test the idea, I set up a non-scientific survey on Instagram and asked my peers the question “Do you know what Open Data is?” with a simple “Yes/No” response. Of the 22 persons who responded, 3 said Yes (14%) and 19 said No (86%). In a perfect world, I would have also had a follow-up question asking if they had used information from a list of known Open Data sources to confirm the theory, but we will have to do without for now.

Quick Poll
Quick Poll

Yet in the age of Covid-19 where everyone is quite capable of quoting a statistic or method in every online argument for and against, how many of us haven’t realised we are benefiting from the availability of Open Data when we quote new case counts, % positive tests, and infection rates in our conversations on a daily basis?

I attended SODU to learn about Open Data, and I learnt I’d actually been using it all along. Several prominent themes discussed at SODU included the need for a community of practitioners, having a central point of access, and having evidence of the benefits of supporting Open Data. The question that bugs me now is, how do we know who our practitioners and where our success stories are, if they can’t even recognise themselves? Maybe, there is an opportunity to do some work here?

 

Help us make Aberdeen’s Air Quality Better

Join us on 8th and 9th June for a diverse hack weekend as we work together to identify Air Quality issues and fix them.

According to the last air quality report for Aberdeen, we have some of the worst air in Scotland. Even though we live on the coast, and generally have a nice breeze, we still have bad air days. Particulate matter is so small that we can’t see the risks: clear air does not mean clean air. This leads to health issues for some of us, and our families. It also affects how busy the health services are too.

There are some official monitors available for the city, but they don’t have widespread coverage. This means you are unlikely to be able to check the air quality around your house, your child’s school, your cycle commute or where you go running. With your help we would like to change that.

Clean Air Aberdeen sprang from our February Code the City co-design event focusing on air quality. We built some sensors, developed some analytics, and explored how to take this work further. At the end the participants wanted this to continue through an ad-hoc organisation aimed at monitoring air quality in Aberdeen as written up here. The data from every sensor is published as open data which can be used to create new products and services.

We are now ready to continue this air quality campaign to better monitor and analyse the air quality in Aberdeen. We have set up another co-design event at the University of Aberdeen on 8-9 June You can find out more about the event and book tickets here: https://www.eventbrite.co.uk/e/codethecity-16-air-quality-2-tickets-60685063659

As before, we’ll follow our usual co-design approach to gather interesting ideas of projects, and form mixed ability teams from attending students, professionals, developers, and designers to work on ideas over the weekend.

Activities will probably include

  • Building sensor kits each day
  • Gathering weather data
  • Improve monitoring of devices
  • Use data science to create predictive models
  • Match open health data with areas of poor AQ
  • Creating alerts (hitting triggers)
  • Test for exceeding daily limits etc
  • Hosting of data for re-use
  • Governance activity (Saturday)
  • Better understanding of device limitations (sensitivity, fog, …)
  • Discussion of any known issues/bugs (getting people to install units once built)
  • Comms and promotion
  • Designing UK network meetup for other city projects

And did we mention food? We’ll have great catering from social enterprise The Bread Maker, thanks to sponsorship by Forty Two Studio.

We hope you will join us and encourage friends and colleagues to get involved.

Please don’t delay as the event is only two weeks away.

Book a ticket now!

Scraping Goes Off The Rails

This post was originally published on 10ml.com by Ian Watt

The art of scraping websites is one beset by difficulties, as I was reminded this week when re-testing a scraper that I built recently.

Schienenbruch

 

Railway performance

As part of my participation in 100 Days of Code I’ve been working on a few projects.

The first one that I tackled was a scraper to gather data from the PDF performance reports which are published on a four-weekly cycle Scotrail’s website. On the face of it this is a straightforward things to do.

  1. Find the link to the latest PDF on the performance page using the label “Download Monthly Performance Results”.
  2. Grab that PDF to archive it. (Scotrail don’t do that – they vanish each one and replace it with a new one every four weeks, so there is no archive).
  3. Use a service such as PDFTables which has an API, uploading the PDF and getting a CSV file in return (XSLX and XML versions are also available but less useful in this project).
  4. Parse the CSV file and extract a number of values, including headline figures, and four monthly measures for each of the 73 stations in Scotland.
  5. Store those values somewhere. I decided on clean monthly CSV output files as a failsafe, and a relational SQLite database as an additional, better solution.

Creating the scraper

So, I built the bones of the scraper in a few hours over the first couple of days of the year. I tested it on the then current PDF which was for period nine of 2016-17. That worked, first creating the clean CSV, then later adding the DB-write routines.

Boom – number 1

I then remembered that I had downloaded the previous period’s PDF. So I modified the code (to omit the downloading routine) and ran it to test the scraping routine on it – and it blew up my code. The format of the table structure in the PDF had changed with an extra blank link to the right of the first list of station names.

After creating a new version and publishing that, I sat back and waited for the publication of period 10 data. That was published in the middle of this week.

Boom – number 2

I re-ran the scraper to add that new PDF to my database – and guess what? It blew up the scraper again. What had happened? Scotrail had changed the structure of the filename of the PDF – from using dashes (as in ‘performance-display-p1617-09.pdf’) to underscores (‘performance_display_p1617_10.pdf’)

That change meant that my routine for sicking out the year and period, which is used to identify database records, broke. So I had to rewrite it. Not a major hassle – but it means that each new publication has necessitated a tweaking of the code. Hopefully in time the code will be flexible enough to accommodate minor deviations from what is expected without manual changes. We’ll see.

We’re ‘doing the wrong thing righter’ – Drucker

Of course, none of this should be necessary.

In a perfect world Scotrail would publish well structured, machine-readable open data for performance. I did email them on 26th November 2016, long before I started the scraper, both asking for past periods’ data and asking if they wanted assistance in creating Open Data. I got a customer service reply on 7th December saying that a manager would be in touch. To date (15 Jan 2017) I’ve had no further response.

The right thing

Abelio operates the Scotrail franchise under contract to the Scottish Government.

Should the terms of such contracts not put an obligation on the companies not only to put the monthly data into the public domain, but also that it be made available as good open data – and follow the Scottish Government’s on strategy for Open Data ? Extending the government’s open data obligation to those performing contracts for governments would be a welcome step forward for Scotland.

Chatbots and AI – #CTC8

Code the City #8, which will take place in on Sat 25th to Sunday 26th February 2017, will be an exploration of the world of chatbots and AI (or Artificial Intelligence), identifying problems to tackle and quickly prototyping solutions.

>>> Book a ticket on our  Eventbrite page 

What are chat bots?

A chatbot is a piece of software that interacts with a customer or user to directly answer their questions. It uses existing data or information coupled with artificial intelligence to respond in a human-like way, guiding the user to a solution.

There are many examples of live chat bots in this exciting, emerging field. A chatboat could give you travel directions, tell you when its next going to rain in your area, or help you contest parking tickets. It could book you a flight and hotel, or act as a free lawyer to help the homeless get housing . The HBO series Westworld has even launched a bot to help you interact with the (fictional) holiday park!

If you are new to this field and want to get started we suggest you read the Complete Beginners Guide to Chatbots (and some of the links at the end of this article).

Example Travel Bot
Example Travel Bot
Example Waste Bot
Example Waste Bot

How will the weekend run?

We’ll apply our usual  Code The City methodology:

  • Bring together a diverse range of people from various backgrounds, to form teams.
  • Identify problems that we’d like to apply chatbots to solve.
  • Identify approaches,  information and data, to guide how we develop the bots and train them
  • Mix academic thinking, and user need, with open source technology and open data to develop new services
  • Iterate quickly through approaches, testing ideas, failing quickly and refining our approaches.
  • Prototype and demonstrate solutions to an interested audience

Who should attend?

  • Service owners – and service providers
  • Academics and students in the field of chatbots and artificial intelligence
  • Coders
  • Data specialists
  • Front-end and UX designers
  • Bloggers and social media practitioners
  • Anyone with an interest in getting involved in creating bots even for fun!

What you will do?

You will create mixed teams to workshop chatbot solutions to real world issues.  Maybe these will building on the outputs of previous work we’ve done at CodeTheCity. Through rapid prototyping you will create new applications and have some fun in the process.

We’ll show you new techniques for service design, idea generation, prototyping, and rapid iterative application development – and you will show other participants some tricks and approaches, too. We’ll share knowledge and learning.

You might even get a Tshirt, and we can guarantee the best catering of any weekend workshop in the city!

To book a free ticket visit our Eventbrite page   But be quick, tickets will go swiftly!

All attendees will get a year’s free membership of the Open Data Institute.

You can find out more about the previous events on tumblr, on the eventifier, and on flickr.

If you have any questions please get in touch.

How can I support this event?

If you are interested in sponsoring this event please, or providing other support such as access to online tools or services, please  get in touch.

Useful Articles and Resources

>>> Book a ticket on our  Eventbrite page

Our first week as a Node of the ODI

Last week we were delighted to hear the announcement that Code The City have been approved as a node of the Open Data Institute (ODI).

We are the first node in Scotland (and the only one north of Leeds) and will be working with all sectors to ensure that we raise awareness of the best practice in Open Data through training, networking, and organising events across the country. Continue reading Our first week as a Node of the ODI