“Speak to Lana, she saw something like that last year,” or “Ask Louis to do it, he’s seen that a million times.” We’ve all heard similar phrases within business — phrases that are uttered when something unusual happens, linking people together to help them get more experience in order to solve the challenge of those unusual circumstances. Imagine an 18-year-old being thrown directly into Major League Baseball, facing the best pitcher in the league: Everything that he is about to see is unusual, and the odds of failing are massive. The unusual is what trips us up and can cause us to fail.
While some folks still talk about the three V's — velocity, variety and volume — with regard to big data, I think we are well beyond worrying about what makes data big. Instead the focus is “Why bother with big data?” For me, it comes down to a very simple statement:
What does that mean? Well, any system is able to do the happy path — able to understand how to react well in the circumstances that normally occur, and even able to handle the normal exceptions. But if an event only happens once in a year, the odds are that there will not be a coded exception path, the information won’t be available. If that once-a-year event can occur in any one of 50 countries, then it’s possible that a single country won’t see the event happen for decades. How can you learn and react to something you’ve never seen? A single system can’t. A single person can’t, but the “hive mind” of an organization is able to see the event and find who has seen it before. The strength of the response depends on the strength of the personal networks.
Big data, however, can use techniques such as machine learning to include all the events from every country. So while that event might only occur once a year, the big data system can see 10 such events over a 10-year period and be able to recognize the causes and build a standard response plan to resolve the situation. Thus when it occurs in a new country, it can be automatically detected and resolved, potentially even without a person knowing it occurred.