Is it really true that "Nearly two-thirds of big data projects will fail to get beyond the pilot and experimentation phase in the next two years, and will end up being abandoned," as suggested by Steve Ranger last year in Your big data projects will probably fail, and here's why? My take: to be successful you need a collaborative team with multiple skills, effective leadership, good communication -- and a plan. In other words, don't put the cart before the horse by starting with a technical solution before you understand what problems you'll be trying to solve.
In my consulting and research I help people plan and manage effective and sustainable data and IT management programs. Given how "big data" has risen in importance I'm very interested now in how you plan and govern programs that manage and provide access to organized and useful data.
The question often arises of how much analytical expertise is needed to effectively manage data-intensive programs, regardless of whether new or existing data management and analysis technologies will be employed. While "big data" has made such issues quite visible, you can't expect everyone to be a "data scientist" or professional statistician. At the same time, managers who expect to drive or make sense of improved data analytics are going to need more than a understanding of how spreadsheets operate.
Training and educational programs around data analysis tools and techniques are proliferating and managers in different fields wonder how much analytical expertise is needed. A recent example is Adele Sweetwood's Creative or Analytical? Marketers Must Now Be Both. In it she says, among other things,
"Knowledge of data-management principles and analytical strategies, an understanding of the importance of data quality and data governance, and a solid grasp of the value of data in marketing disciplines are now all essential."
"Today's marketer needs to go well beyond reporting and metrics."
"The successful contributor is proficient in a full range of analytics, which may include optimization, text, sentiment, scoring, modeling, visualization, forecasting, and attribution. That doesn't necessarily mean all marketers must have a PhD in statistics, but they must understand and use such methods."
Sweetwood's overview surprised me a bit. Having managed a lot of quantitative market research early in my career I have always tended to think that marketing is, at its core, a data-dependent function where measuring and predicting customer engagement and/or behavior have always been paramount. As Sweetwood points out, though, the tools now available to the marketer are more sophisticated, complex, and they operate increasingly in real time, i.e., "Today's marketer needs to go well beyond reporting and metrics." Yesterday's number-crunching skills might not be up to the task.