Unless a "neutral" third party publishes them, we tend to view benchmarks as self-serving exercises that vendors typically stack in their own favor. But recent benchmarks issued by Cloudera and Hortonworks for their SQL on Hadoop engines point to something serious going on. In an era of Spark hype, SQL remains table stakes for Hadoop platforms.
Yes, you can perform machine learning, model customer ecosystems as social graphs, run streaming, and conduct sentiment analysis, but for most organizations, the first question they often ask is how fast is the interactive SQL. Using Hadoop only for SQL query might seem like a waste, given its appeal to R or Python developers. But getting buy-in requires satisfying the BI crowd, because in many organizations, SQL's the gateway drug to Hadoop.
And looking at the benchmarking press releases, you get a sense of who's afraid of whom. For Cloudera, it's Amazon. Competitive benchmarks pitted Impala 2.6, Cloudera's SQL-on-Hadoop MPP engine, against Amazon Redshift columnar analytic database. The results, announced a couple weeks back at Strata, showed Impala performing four to 10x faster on either S3 (which Redshift doesn't use) or EBS (which it does).
Cloudera is stating that now even a database that is decoupled from storage (Impala) can perform better than one that followed a traditional tightly coupled data warehouse architecture (Redshift). It's a shot across the bow, given if you want consistent SLAs, high concurrency, or support of very complex SQL syntax, conventional wisdom has been to use a database rather than Hadoop. Cloudera's results don't change that reality, but they do show results in the ballpark with Redshift. And they get the results using AWS's default S3 storage.
But Cloudera's underlying message is not just that Impala has been tuned to go faster. It knows that, while only a minority of customers are deploying to the cloud today, in the long term, the writing is on the wall.
Â
