Final presentations at CTC10 – Perth

We had four presentations at the final pitch session at CTC10.

We have uploaded these to Vimeo below (trimming them for time to just the core presentations, and eliminating intros and questions):

Team one: https://vimeo.com/236647324

Team two: https://vimeo.com/236648125

Team three: https://vimeo.com/236649327

Team four: https://vimeo.com/236650501

These brought an enjoyable and productive couple of days to a close.

Well done to all participants involved!.

CTC9 – Team Presentations

In this close-out post I shall hand over to the teams themselves to walk you through their CTC9 weekend. Check out the videos using the links below. Use the ‘ctc9’ tag to find all other blog posts about the amazing volunteering experience this weekend.

Team: Soul Cats

Team: The Professionals

Team: ALISS API

CTC9 – What a weekend!

I am so glad I joined the CTC9 project as a volunteer. Blogging about this project was a tremendous experience. There are two aspects of this weekend that amazed me beyond the teams’ achievements.

The idea funnel

It was fascinating to witness the journey we all ventured on – from random ideas on post-its to distilling them down into structured approaches.

ideation
ideas ideas ideas
planning
how things fit together

Team work

The teams seemed to develop naturally based on people’s interests. It is remarkable how smoothly people from different sectors and backgrounds worked together in a very productive way. The Code the City staff did a great job in keeping us all on track.

team work

CTC9 – Near the finish line

Here’s a quick update before the big show-and-tell later on.

Team: ALISS API database

The team has developed a draft version of the website tucked away on a test server. They have established the first functional search using the category ‘social isolation’. It returns a list of service providers in the area that is drawn from the three source databases. This is a big step forward, as we now know how to program a search and are able to deliver visible results on a user interface.

The team is also working on searches based on location by postcode or radius.

One expected challenge is the extraction of information from differently formatted data sources. For example, one source database does not provide contact details in dedicated address fields but in a more general description box.

Team: Soul Cats

This group went back to focusing on the public end users. They came up with various names for this new website that make it easy to find. They played with words from Scots dialect and proper King’s English. All suggestions were googled to see whether they exist already or are buried in amongst a ton of other results. Ideally, we want something unique!

The team suggested to submit a selection of words to a public forum in order to collect opinions or votes.

Team: The Professionals

The Professionals are a spin-off group from the Soul Cats. It’s a rollercoaster with those Cats! They went back to focusing on the value this website for health care professionals. In a structured approach they answered 4 key questions:

  1. Who are key stakeholders?
  2. What are key relationships?
  3. What are key challenges?
  4. What are the gains right now if this project went live?

team-gathering

CTC9 – Sunday Morning

What a beautiful sunny morning for making my way over to CTC9 HQ. It’s a slow start today. Hey, it’s Sunday…

Since we didn’t have a close-out meeting last night, we caught up with everybody’s progress in a kick-off meeting this morning. Make sure to read the update from yesterday afternoon beforehand.

Team: ALISS API

Geek MuffinThe data is flowing! We now have access to all 3 data sources: ALISS, GCD and MILO. MILO too? Yes! As it turns computing student Mikko has been working on hooking up MILO to the project as part of Team ALISS API.

Linking up GCD encountered a stumbling block after the initial success because the WiFi network ended up blocking the website used for our API. By the sounds of it, this is in hand though.

screenshot demoNow that we are connected to all databases, they are being combined by matching titles, identifying duplicates etc. The result will provide access to searchable data from all sources via one URL. James has already launched a temporary live demo page that connects to the databases. The first rough draft is based on story boards James designed with input from the user-focused teams last night. The website is currently at an early stage; so some buttons will work, some won’t. Feel free to rummage around.

There is also a shared file repository on github. It harbours user interface code, the backend REST API and photos from our brain storming sessions.

The next big goal is to develop the visual interface further to make search results visible to the website user. At the moment results appear only in code. The team also suggested that functionalities for location-based search and prioritising search results will require more development.

Sunday team photo

Team: Soul Cats

Teams Stripy Tops and Access All Areas have merged under the new name ‘Soul Cats’ (inspired by a T-shirt). This move made sense because both have been targeting user groups – the professional user (Stripy Tops) and the public (Access All Areas) – and now felt that their paths were converging.

The teams have drawn up more specific suggestions on user requirements based on the needs of different target groups. It’s quite impressive how yesterday’s wide-roaming discussions are now funneling into concrete scenarios and solutions. The obvious conclusion is to make the web interface simple – clear language, natural keywords, self-evident icons, sensible menu structure etc.

There was some discussion around: user cases

  • options for geo-location of service providers relative to user addresses
  • including info on mobility/access issues e.g. stairs
  • including info on parking, public and community transport connections
  • including photos of the service location, exteriors and interiors, so that people easily recognise the place once there

The next steps will involve working closer with our coders and coming up with names for the page, categories etc.

Code The City 9 – It’s on!

We kicked off the ‘Code The City 9 – Health Signposting’ weekend this morning bright-eyed and bushy-tailed. There are just under 20 attendees from mixed backgrounds.

We have volunteered to help solve issues around health care data. One problem is that health care data are currently maintained in (at least) three unconnected systems run by different organisations. These are ALISS, GCD (Grampian CareData) and MILO. The ultimate goal of this project is to create an open data source that provides accessible up-to-date information to the public and professionals.

Continue reading Code The City 9 – It’s on!

CTC8 – Chatbots and AI -final presentations

After two days of intense activity and a whole heap of learning for all of us, Code The City #8, our Chatbots and AI weekend came to an end at tea time on Sunday.

It couldn’t have happened without the generous sponsorship of our two sponsors: The Health Alliance, and Fifth Ring, for which we are very grateful.

The weekend rounded off with presentations of each project, four of which we’ve captured on video (see below).

Each of the projects has its own Github repo. Links are included at the end of each project description. And, two days later, the projects are still being worked on!

Team: ALISS

Team ALISS worked on providing a chatbot interface onto healthcare and social data provided via the ALISS system.

ALISS bot project : Code the City 8 from Andrew Sage on Vimeo.

You can find Project ALISS’s code here on Github.

You can also watch this video of Douglas Maxwell from the Alliance being interviewed about the weekend (although at the time of writing the video is offline due to an AWS problem).

Team: City-consult

This team aimed to make the quality of consultations better through using intelligent chatbot interfaces to guide users through the process – and to provide challenge by prompting citizens to comment on previous consultees’ input.

City-Consult bot project : Code the City 8 from Andrew Sage on Vimeo.

You can find the code for City-Consult at this Github repo.

Team: NoBot

The concept for NoBot came from an initial idea which was of a bot which would make scheduling meetings easier. That spawned the idea – what if the Bot’s purpose was to make you have fewer meetings by challenging you at every turn, and in the process the bot’s personality as a sarcastic gatekeeper was born.

NoBot project : Code the City 8 from Andrew Sage on Vimeo.

The code for Nobot lives here on Github.

Team: Seymour

Sadly there is no video of the wind-up talk for Seymour. In short the purpose of Seymour is to help you keep your houseplants alive. (More details to come).

You can find the code for Seymour at this repo on Github.

Team: Stuff Happens

We started this project with the aim to help citizens find out what was happening in the myriad of local events which we each often seem to miss. Many local authorities have a What’s On calendar, sometimes with an RSS feed. None we found had an API unfortunately.

We identified that by pulling multiple RSS feeds into a single database then putting a bot in front of it, and either through scripting or applying some AI, it should be possible to put potential audiences in touch with what is happening.

Further, by enhancing the collected data – enriching it either manually or by applying machine logic, we could make it more easily navigable and intelligible.

Expect a full write-up of the challenges of this project, and what progress was made, on Ian’s blog,

There is no video, but you an find the project code here on Github.

Team: W[oa]nder

This project set out to solve the problem of checking if a shop or business was still open for the day through a Facebook bot interface – as you with wander around, wondering about the question, as it were.

W[oa]nder bot project : Code the City 8 from Andrew Sage on Vimeo.

You can find their code here.

And finally we were joined by Rory on day two who set out to assist team Stuff-Happens through developing some of the AI around terminologies or categories. That became the:

Word Association Scorer

This is now on Github – not a bot but a set of python functions that scores a given text against a set of categories.

And Finally

We had loads of positive feedback from those who attended the weekend (both old hands and newbies) and from those who watched from afar, following progress on Twitter.

We’ve published the dates for CTC9 and subsequent workshops on our front page. We hope you can join us for more creative fun.

Ian, Andrew, Steve and Bruce
@codethecity

CTC8 – Thanks to our sponsors

Code The City Weekends would not happen were it not for the generosity of our sponsors.

As we approach CodeTheCity #8 we must recognise two organisations who have backed this event.

The first is The Health and Social Care Alliance Scotland (the ALLIANCE) who is sponsoring the event through its ALISS Programme.

The ALISS Programme is excited to be sponsoring and attending the Code the City, AI and Chatbots hack weekend. This area of our work allows us to test concepts and prototype real world solutions to problems like; how does a person who is living with sight loss access the great local resources available on ALISS? Or, how does a person who finds it challenging to use normal desk-based computers access the local support available through ALISS? We will hopefully test these types of problems at the hack weekend and follow this up with a blog on our work.

ALISS Logo
ALISS Logo

 

 

 

 

 

Please follow the Alliance and their work on Twitter: at @ALLIANCEScot  @ALISSProgramme or the hashtag search for #ALISS .  If you are attending the weekend please make sure you hook up with @DouglasMaxw3ll and have a chat to him about the great work that his organisation does.

Our other main sponsor is Fifth Ring.

contact___fifth_ring

Fifth Ring is a marketing and communications agency in Aberdeen, Houston and Singapore with a big focus on digital and inbound marketing. Fifth Ring is already experimenting with conversational interfaces for some of their client work.

If you are attending the weekend please make sure to say hello to Steve Milne or Alan Stobie to discuss some of the great work they do.

You can find Fifth Ring on the web, on Twitter, and on LinkedIn.

 

Scraping Goes Off The Rails

This post was originally published on 10ml.com by Ian Watt

The art of scraping websites is one beset by difficulties, as I was reminded this week when re-testing a scraper that I built recently.

Schienenbruch

 

Railway performance

As part of my participation in 100 Days of Code I’ve been working on a few projects.

The first one that I tackled was a scraper to gather data from the PDF performance reports which are published on a four-weekly cycle Scotrail’s website. On the face of it this is a straightforward things to do.

  1. Find the link to the latest PDF on the performance page using the label “Download Monthly Performance Results”.
  2. Grab that PDF to archive it. (Scotrail don’t do that – they vanish each one and replace it with a new one every four weeks, so there is no archive).
  3. Use a service such as PDFTables which has an API, uploading the PDF and getting a CSV file in return (XSLX and XML versions are also available but less useful in this project).
  4. Parse the CSV file and extract a number of values, including headline figures, and four monthly measures for each of the 73 stations in Scotland.
  5. Store those values somewhere. I decided on clean monthly CSV output files as a failsafe, and a relational SQLite database as an additional, better solution.

Creating the scraper

So, I built the bones of the scraper in a few hours over the first couple of days of the year. I tested it on the then current PDF which was for period nine of 2016-17. That worked, first creating the clean CSV, then later adding the DB-write routines.

Boom – number 1

I then remembered that I had downloaded the previous period’s PDF. So I modified the code (to omit the downloading routine) and ran it to test the scraping routine on it – and it blew up my code. The format of the table structure in the PDF had changed with an extra blank link to the right of the first list of station names.

After creating a new version and publishing that, I sat back and waited for the publication of period 10 data. That was published in the middle of this week.

Boom – number 2

I re-ran the scraper to add that new PDF to my database – and guess what? It blew up the scraper again. What had happened? Scotrail had changed the structure of the filename of the PDF – from using dashes (as in ‘performance-display-p1617-09.pdf’) to underscores (‘performance_display_p1617_10.pdf’)

That change meant that my routine for sicking out the year and period, which is used to identify database records, broke. So I had to rewrite it. Not a major hassle – but it means that each new publication has necessitated a tweaking of the code. Hopefully in time the code will be flexible enough to accommodate minor deviations from what is expected without manual changes. We’ll see.

We’re ‘doing the wrong thing righter’ – Drucker

Of course, none of this should be necessary.

In a perfect world Scotrail would publish well structured, machine-readable open data for performance. I did email them on 26th November 2016, long before I started the scraper, both asking for past periods’ data and asking if they wanted assistance in creating Open Data. I got a customer service reply on 7th December saying that a manager would be in touch. To date (15 Jan 2017) I’ve had no further response.

The right thing

Abelio operates the Scotrail franchise under contract to the Scottish Government.

Should the terms of such contracts not put an obligation on the companies not only to put the monthly data into the public domain, but also that it be made available as good open data – and follow the Scottish Government’s on strategy for Open Data ? Extending the government’s open data obligation to those performing contracts for governments would be a welcome step forward for Scotland.