We split up to try and cover as much ground as possible. Here are our individual highlights from the three days::
I was interested to get a handle on ‘Big Data’, something that one hears a lot about – I attended a couple of introductory lectures hosted by companies which were commercialising Hadoop. Hadoop is an Apache (i.e. open source and free) project which provides an infrastructure for consuming massive quantities of data from diverse sources and placing it in a horizontally scaled farm of servers. The trick is that no assumptions are made about the queries that are going to be asked of the data in advance – a schema is applied to the data at the time that a query is made. Because of the way that the data is stored one can get responses to queries that cover masses of data in a predicatable and timely way. Because the data in unstructured in storage it allows for any question to be asked of it. Surely there must be a project somewhere at the School that would benefit from this technology?
The ‘Internet of Things’ is something which one hears ever more about. Network enabled devices are likely to be everywhere soon, and that includes the LSE. Node-RED provides a nice user interface onto a web server, in the demo this was hosted on a Raspberry Pi. Node-RED provides a way of configuring and wiring up diverse Node.js plugins – you can get these for just about anything, from a central heating system to a twitter account to a quadrocopter. Node-RED allows you to, for example, monitor a thermometer and use the data received to switch on the central heating system or open a window.
There was a whole track on people/team work, and others, which overlapped somewhat, on managing complexity, hosting robust systems and managing risk and uncertainty. The overlap: if you have a complex system and need it to be robust, then you really can’t afford to have hidden areas or secrets – everyone has to feel free to fess up if they’ve made a mistake, and be happy to be transparent about how their part of the operation works, otherwise one has no hope of understanding why something has failed. Mistakes therefore have to be welcomed and praised, on the basis that finding weaknesses in the system should be rewarded as it gives one a chance to deal with it.
“Many organizations are childlike. They blithely plan each project as if nothing will go wrong. And then, when something does go wrong, they are shocked and dismayed.” – Tim Lister. Tim talked about identifying risks, then identifying mitigations – if it would take a month to put the mitigation in place, then, a month before the deadline, if the risk is still there then one starts implementing the mitigation. Tim presented some academic research which showed that there is a normal curve for projects – some come in early, most come in half way between early and late, and some come in late: an utterly massive majority of IT project estimates have been shown to assume that the project will come in early, i.e. that everything will go as well as it possibly could.
This was my first QCon and it felt very different from the more traditional, scientific conferences I’ve been to in the past. Keeping each session to a single talk meant the speaker and audience were always engaged and there was plenty of scope to mix and match topics when planning which talks to go to, so I decided to attend as wide a variety as possible.
The first talk I attended was focussed on HTML 5 WebSocket, a new protocol for connecting across the web that offers near real time communication and control, as demonstrated by a WebGL 3D animation hosted in San Francisco which could be controlled from a browser on a device in London with virtually no time lag. This was an impressive demo and a lot of the people in the audience seemed to leave at the end of the talk with the feeling that this could be a technology to watch in the next couple of years.
Like Mladen from the development team who blogged a couple of weeks ago about his day at QCon, I also attended the Real Data Science at NASA talk. It was fascinating to hear how large a proportion of the work undertaken by the computing team at JPL is actually related to getting existing scientific programs written in MATLAB or other packages to run efficiently, for scientific teams unwilling to translate their work to other formats or platforms. Another interesting point to come from this talk was that it is important to always remember that software needs hardware – it’s no use being able to deal with petabytes of data if there’s nowhere to store it!
A surprisingly interesting talk I went to was about regular expressions (for those who are unaware, a regular expression is essentially a pattern to match strings against, for example to find occurrences of a word in a document), where we were encouraged to view a regular expression as a computer program and even draw it. I can honestly say I will never look at a regex in quite the same way again.
A great talk I went to was about making mobile web performance fast. Not only did the speakers give some really useful guidance about how to optimise performance, but they also lifted the lid on some psychological tricks which companies use to make the user feel better about their applications. For example, Facebook did some testing whereby, when waiting, half their iOS users received the Facebook custom ‘throbber’ animation and the other half received the iOS spinner – the half that saw the Facebook animation blamed Facebook for the delay, whilst the other half actually blamed Apple. No prizes for guessing which animation they decided to use. Another example is that of Instagram: when uploading an image a user will get instant feedback saying that the image has been uploaded, giving them the impression that the application is extremely fast, when in reality it is still being uploaded in the background. Of course I would never suggest employing such tactics here at the LSE 😉
Other highlights of the conference for me included the talk introducing Java 8, a talk about including speech recognition in applications and how libraries are now available to make this relatively straightforward, a talk about enhancing books with audio and extended user interaction by harnessing technology originally intended for wearables, and a talk about how to use a Raspberry Pi computer to access your car’s electronic control unit.
Overall, a very interesting and inspiring conference. Roll on QCon 2015.
I’ve also been to the talks covering software designs in some of the largest online service provides, such as Netflix, Twitter and Etsy. Netflix runs a full-on Amazon Web Services based video streaming and discovering service with JVM as their platform, and various JVM languages used. Twitter has transformed their platform from a shaky RubyOnRails app to a solid scalable JVM-based platform. Both companies have released some of their internal innovations as open source projects available for others to use, which may be useful at some point for us. Etsy is a newcomer this year at QCon and they run a LAMP stack with slightly different arrangement towards development and deployment.