As I’m sure you’ve noticed big data has been a hot topic for a while now. Everyone is rushing to use big data technologies like Hadoop and Cassandra and VCs have been pumping money into companies associated with the big data trend for at least the last couple of years. However, if you probe what the excitement is all about then once people move beyond the point that we are now generating data in unbelievable huge volumes and that there must be value in it somewhere then many people draw a blank.
So I was interested when GigaOM put up a post this morning titled 10 ways big data changes everything. I read through the ten ‘case studies’, and summarised them below. I’ve put my opinion on the trend in italics after a summary of the GigaOM case study. There was, in my opinion, a lot of fluff in the examples they chose, and of the ten there were only two that really stood out to me as areas with the depth and breadth to be home to multiple successful startups, and they were business intelligence applications of big data and virtual assistants.
- Mining social media and other music sites to predict the next Lady Gaga – GigaOM highlights Next Big Sound as a key company in this market. Will this idea work? So far Next Big Sound has ‘two undisclosed major record labels’ as customers for this service, making it too early to tell.
- Finding opportunities for energy savings by comparing consumption patterns – observing yourself is powerful in any context and making consumers aware of how the different things they do effect their energy consumption is no different. Comparison with other others is one way to drive behaviour change, but simple displays showing realtime energy consumption may have more impact.
- Virtual assistants – software tools that help us with the more routine tasks we have to deal with. Siri is perhaps the most famous example, but there are many others. GigaOM chose to highlight a datacentre management assistant technology called Autopilot from a German company called Arago. Virtual assistants combine analysis of large volumes of data with artificial intelligence and will, I think, have a huge impact on many aspects of life over the next ten years.
- Data fuelled recommendations – GigaOM gives the example of Foursquare which is making recommendations based on the data it gets from its users checking in and leaving tips and comments. Recommendations get more powerful when they have more data and I think the future will see recommendations based on data from multiple sources – Foursquare, Facebook, Twitter, and anything else that is getting traction with consumers. Ultimately I see this as a subset of the 3. Virtual assistants.
- Tracking disease epidemics – GigaOM tracks the story of how Twitter tracked a recent outbreak of cholera in Haiti and how that can be of use to aid agencies. There are numerous examples now of social media being used to good effect in crisis situations which is great. I’m not sure if it really counts as big data though.
- Business intelligence – GigaOM describes how a startup called Parse.ly is helping publishers analyse publishing data to see what content is driving traffic and predict what might work in the future. I think business intelligence is most obvious near term application for big data and that the most obvious short term startup opportunity is technologies and services for this market. Unsurprisingly this is where most of today’s hot big data companies are playing.
- Mining cellphone billing records – GigaOM describes a number of use cases for this data including helping with malaria education in Kenya. I see cellphone billing records as one more dataset that can be minded to good effect. It is no different in principle to social media data and ultimately the successful services will be the ones that are able to draw value from multiple datasets.
- Using data to predict and create video hits – Netflix is the case study this time. I am sure that by mining user data Netflix will be able to better guess how popular new video content will be, but it isn’t clear from the GigaOM article how much better it will be than traditional human based analyses which focus on success of similar content in the past.
- Touch screen interfaces for interacting with big data – touchscreens offer new possibilities for manipulating large datasets with novel graph and chart interfaces. I’m not so sure about this one. Funky charts have value in presentation situations, but I think that most of the insight from big data will be gleaned by analysts who work by looking at different cuts of the data and zomming in and zooming out.
- Hospitals using big data to improve efficiency – GigaOM describes how electronic patient records have delivered some improvements, but that there is a way to go. There is huge opportunity for IT enabled cost savings in healthcare delivery. The challenges are more organisational and bureaucratic than technical though and the opportunity for startups is to build services highly tailored to the healthcare market in which they work rather than exploiting the latest big data technologies.
One interesting thought which occurred to me whilst compiling this list is that it will likely get much easier for companies to realise the latent value in their data. Over the years we’ve seen a lot of plans from companies that put some value on data generated as a by-product of their main business and only a few of them succeeded in monetising that value. Going forward we will see more businesses dedicated to making money out of data which will be keen to help other companies monetise their data assets. Peer Index and Datasift are two UK based companies that are doing this already.
Finally, some of you may have noted that advertising wasn’t mentioned in the GigaOM list. Better targeting and tracking of advertising is very data intensive and to my mind a clear