A decade ago, I thought I understood big data. I had worked in information technology for more than a decade and had run a department that handled docs for some of Boston’s more infamous litigation. I remember having to order new drives and storage appliances to handle the gigabytes and gigabytes of documents and emails that our hapless associates had to search and read through. That was a lot of data… or so I thought.
Fast-forwarding seven years and a career change, I found myself at Amazon running SQL queries against their data warehouse. The scope of that database honestly blew my mind; I had to figure out tricks to even pull down a week of summary data without having it choke or overflow Excel. I thought I’d understood what big data was, but it turns out that I had no clue.
Big data has become a buzzword so prevalent that it’s practically meaningless. At a party last week, I heard someone say, “Every company is a big data company now.” When I asked him to clarify, he said that every company buys and sells big data. While I certainly agree that all companies can use big data or applications based on big data, not all companies base their business models on it. I’ve tripped over this kind of misconception a lot over my career and have even shared some of the misconceptions myself. Now that I work at a big data company , I know better.
Here are six of the biggest mistakes I see execs make when they talk about big data:
1. All data is big data.
According to Gartner , big data must be high-volume, high-velocity and/or high-variety data. This means that if your data can fit in an Excel file, you’re not dealing with big data. If you’re only handling a dataset that measures in the gigabytes and your PC can handle it, you’re not dealing with big data. Maybe you’re dealing with many gigabytes of emails and you can’t figure out how to deal with it, but that doesn’t mean that it’s big data.
2. Big data solves every problem.
I’ve run into a few execs who believe that big data fixes everything. Many of them grasp at big data analysis to solve problems rather than using common sense. I once sat in a room of executives who were trying to figure out why our week-over-week website visit numbers and sales had dipped precipitously during a week in April, but that same week the year before hadn’t experienced the same decrease. They asked for analysis after analysis until someone said, “Well we see a decrease at Easter every year, and Easter was in March last year.” Big data and analysis didn’t help us figure that out, but common sense and a calendar did.
3. Big data is meaningless.
The flip side of the “everything” misconceptions about big data is this one: that big data doesn’t matter. I find this opinion to be more understandable, because the definition of big data indicates that it’s hard to process and understand. If you can’t pull insights out of big data or use it to power your systems, it is, indeed, meaningless. I suspect execs in this camp have learned about big data but have never learned anything from it.
To make big data less meaningless, you need to be able to process and use it, which big data companies make easier. They do this by gathering the data, cleaning it up, organizing it, and outputting it in a way that data scientists or other systems can process. Once a data scientist pulls stories out of the data or your systems use data to execute business operations like supply chains, execs will start seeing value in big data.
4. Big data is easy.
Many things about big data sound easy, like thinking about getting the information and pricing for every single product in the world or tracking every single visitor to every single website.