Many of you will of heard the hype around Big Data over recent months and if you haven’t I’m sure you will! It’s fair to say that the rate at which we are creating data is increasing at an almost exponential rate. With increases in use of mobile phone GPS, social media posts and sensors collecting almost everything coupled with decreases in storage costs the volume, variety and velocity of data will remain on its current path.
A study published in 2011 by DiscoveryNews claimed that up to 94% of data worldwide is stored digitally, compared to just 0.8% 25 years ago!!!
However the question is what are we doing with all this data?
A 2011 Gartner survey highlighted corporate data is growing by around 60% each year, however 73% of those interviewed felt their competitors could use the data better. More interesting, or shocking, is that only 17% of the respondents claimed to be using more than 75% of the data they collect. So companies are capturing and storing data they are not even using!
The question to answer is WHY?
Do they fully understand what data they are collecting?
Do they know how to process the data – structured or unstructured?
Is the data actually actually useful?
Many authors have taken the view that whilst the increase in data offers the potential to drive better decision making through business insights, there is a note of caution around this. As the volume and variety of data increases, so does the uncertainty, ambiguity and complexity, i.e. data noise. The removal or correction of data noise requires considerable thought and appropriate tools to ensure the true insights are extracted from the data.
To complicate this further another shift we are seeing within the data growth is that of the increase in unstructured data. By unstructured data I mean data that does not sit with in a database and exists as textual (text in documents, blogs, social media, etc) or non textual data (images, video and music). Obviously analysing this type of data requires different tools to those traditionally used and asks us the same challenging questions above in finding the nuggets of information amongst the noise…
The effort required in filtering and processing the wealth of data we collect requires time and money, which should decrease over time. However the the high start up investments may deter many businesses. Business that are able to take on the challenge in digesting and processing the data will become more efficient in doing so and gain quick wins along the way. Although as seen with other specialisms, technologies will soon become available to automate and improve this activity thus closing the gap and making it more affordable to the masses.
Therefore one of the biggest challenges in the short term to stay ahead is to really understand and focus on the data that is meaningful to you.
Don’t lose sight of the trees within the wood!