The good news is that 10-year-old Hadoop is maturing quickly. The bad news is that many companies are still struggling to get beyond pilot projects and support many applications on this new data-management platform. That's why I wrote my latest report, "Democratize Big Data: How to Bring Order and Accessibility to Data Lakes."
First, let's consider the industry leaders that are doing great things with data lakes, drawing on examples from the recent Hadoop Summit:
In financial services CapitalOne has been making waves this year, appearing at multiple events to talk about its Hadoop and Spark-based fraud-detection and its big data analytics, streaming, and security work.
In retail Macy's embraced Hadoop more than five years ago to power insights for Macys.com. Today it's doing more sophisticated cross-channel analysis, driving personalized promotions encouraging online customers to shop in stores and in-store customers to obtain out-of-stock and online-only items at Macys.com.
In manufacturing Ford relies on Hadoop for connected car capabilities. Ford does filtering and decision-making at the sensor and car level while uploading crucial data points for centralized insight and analysis. For example, FordPass app users can remotely check their car's fuel level, location, and diagnostic error codes, but detailed data used by service technicians remains in the car's black box.
In insurance Progressive has been a pioneer of usage-based pricing with Progressive SnapShot. The company has more than 15 billion miles' worth of driving data in a Hadoop-based data lake, but it can drill down and offer discounts to individual policy holders based on factors such as their total miles driven, nighttime driving, and speed and braking habits.
These examples are inspiring, but behind every breakthrough there's been a lot of hard work.
Â
