It's all well and good that Hadoop distribution providers include management consoles for their software. Cloudera offers its Manager product; Hortonworks (along with other ODPi distros) uses Apache Ambari and MapR now provides its MapR Control System. These tools aren't bad, but they're designed primarily for cluster-level operations and metrics. They're not really there to help you manage individual jobs or determine performance over time of scheduled, operationalized jobs.
The relational database world doeshave tools that do that though. What's more, they augment, rather than duplicate, the functionality in the vendors' own management tools. In the Microsoft SQL Server world, for example, Microsoft's own SQL Server Management Studio is a very capable tool, offering a slew of management reports. But that doesn't stop companies like Dell (formerly Quest) Software, Red Gate Software, Idera and SQL Sentry from offering huge suites of their own, all of which add considerable value.
The Big Data world needs tools like this too. And today, it's getting them.
Unravel, in a good wayAfter raising a Series A funding round of $6.5M in December, a Big Data startup called Unravel Data is emerging from `stealth today. Unravel is dedicated to simplifying data operations in the Big Data arena, helping Big Data and IT teams cooperate and collaborate in a fashion similar to the way developers and IT do with DevOps tools (in fact, the company uses the term "DataOps" to refer to the need it addresses).
The company's products don't merely monitor jobs but, according to Unravel Data's founder and CEO, Kunal Agarwal, offers automated root cause analysis for when things go wrong. Unravel also offers guided remedy -- troubleshooting support that is algorithmic, machine learning-driven, with rule- and cost-based optimization as a foundation.
Agarwal says that while everyone wants to run Big Data technology, people are lost in managing it. Most data folks don't know distributed computing and vice-versa. The result, when something goes wrong, is everyone picking through logs and crudely testing theorized diagnoses until the problem is remediated. That just doesn't scale, and Unravel aims to address that need.