You’d have to be living on a desert island to not have heard of “big data”, but what does it really mean, and how can it help your business get better results?
All of us, whether we realise it or not, create big data all the time. Every person who uses a computer, tablet, smartphone or credit card generates vast amounts of information, or a “digital footprint”. We shop online, we access government and council services online, and we upload pictures and personal information onto social media.
According to recent research by the University of Cambridge, computers can judge your personality better than friends and family can, just by analysing your Facebook “likes”. Another familiar use of big data is when retailers like Amazon accurately predict what films or books you might like using information from previous searches and product reviews.
This vast ocean of information about what makes us all tick presents a hidden treasure chest of opportunities for organisations who have the tools to unlock it. Research conducted by the Centre for Economics and Business Research identified that big data could bring around £216 billion of benefits to the UK economy by 2017. The problem is few companies have the right tools yet. Our recent survey of 300 UK businesses on how engaged they were with big data showed only a quarter of Scottish companies felt ready to embrace all aspects of big data analytics.
The art of extracting useful information is known as data science, and without data scientists, big data is useless. The problem is there is a global shortage of data scientists, so much so that according to a recent study by The Tech Partnership and SAS UK, 77 per cent of UK data science positions are hard to fill. By 2020 it is estimated there will be 56,000 job opportunities per year for data scientists in the UK alone. This skills shortage is driving up salaries, with the average salary for a data science professional now sitting at £55,000.
The Data Lab – a new innovation centre focused on helping Scottish industry capitalise on the opportunities offered by data science – is in the middle of conducting a live infographic survey on Big Data on its website. One survey question asks organisations what the biggest barrier to their Big Data progress, and results in so far show that 24 per cent blame lack of skills or access to training, with 15 per cent stating lack of access to technology or investment. However, rather worryingly, another 18 per cent cite “organisational resistance” as the main factor. While the first two barriers are understandable, the third one baffles me. A recent survey by big data firm Accenture Analytics found that 83 per cent of respondents believed their companies were ahead of their peers and had an advantage over their competitors because they were using big data effectively.
Netflix, for example, has invested heavily in making sure it is using big data innovatively, not only to predict what films its customers might like to watch next, but also to improve the quality of its service, in terms of speed and picture quality. By collecting data from users on how the physical location of the content they are downloading affects the viewing experience, it can calculate where to place its data to ensure the best service to as many homes as possible.
An example of how invaluable big data can be to the public sector is the app recently developed by Scottish health data analytics firm Aridhia, which takes just two minutes to calculate a patient’s risk of being readmitted to hospital. The current system in England penalises hospitals when they discharge a patient who is then readmitted again within 30 days. With £390 million in penalties withheld from hospital budgets over just three years, this app could prove to be very attractive indeed.
In our “Reflections” blog this month, our senior director of Education Services, Alok Shrivastava, sums up why all savvy businesses need to embrace the big data challenge. He writes: “Big data analytics enable the most important business asset: information. Every business, regardless of industry must plan to create a big data analytics function. It takes data-based decision making to a whole new level, helping drive efficiency, quality, and growth.” What business wouldn’t want a piece of that?
• Martin Brown is Scottish country manager at EMC