Can Big Data Tame the Chaos of Virtualized IT?

Can Big Data Tame the Chaos of Virtualized IT?

The “software-defined” revolution is driving private data centers toward AWS-like efficiency. However, the virtualization of hardware, storage, and networking—not to mention agile coding techniques and a rapid-fire “DevOps” culture–also makes it much more difficult for IT professionals to track down problems. Now some are turning to big data tech to help correlate the various logs and ultimately tame the rising IT chaos.

Visit your friendly neighborhood data center, and you’ll see rack upon rack of servers and storage arrays, all with blinking lights and cabling running everywhere. It’s quite an impressive thing to get a behind-the-scene glimpse of the automation beyond corporate America, to stand in the boiler room of our digitized society.

However, hidden under the impressive hardware is a growing management problem that’s exacerbated by the “software-defined” virtualization trend and near-constant iterations of applications through devotion to DevOps. A data center operator specialist may monitor CPU, memory, and storage utilization for an 8-node Linux cluster, and the administrator sees only the specs of the 20 Docker images he’s managing, while the programmer only cares about the JVM runtime that his Web apps run under.

The whole IT stack has gotten much more complicated over the past decade, and the traditional approach to systems management is broken, according to Bernd Harzog, a systems management industry veteran and the founder and CEO of Atlanta-based OpsDataStore.

“Thirty year ago, we would change software in production once a year because it always created problems,” he says. “Now it’s changed daily, hourly, even every minute. Its’ written in more language than it used to be. And of course everything’s been virtualized.”

The traditional IT management vendors—often dubbed the Big Four of IBM, HP, CA, and BMC—are powerless to keep their frameworks up to date against the pace of relentless change, he says. In fact, even 20 years ago, the pace of innovation was too high for them to keep up with. “They actually never got close to delivering an integrated solution,” Harzog says.

The result is IT management today is a best-of-breed affair. The typical company today runs about 20 to 30 different IT management tools to monitor various pieces of their stack, including the networks, the servers, the storage, the Web servers, the databases, and so on. Some companies run hundreds of different tools.

This best-of-breed approach results in every little thing being instrumented to the n-th degree. But unfortunately, none of the tools were designed to talk to each other, so a specialty monitoring tool may not have visibility into congestion in the network or a growing problem with the storage array.

Harzog developed OpsDataStore to be the glue that brings all of the best-of-breed products together. “Our proposal and our contention is everybody should specialize in what they do best,” he says. “We’re going to be the data plane to knit it all tougher.”

The company built its data plane using (you guessed it), open source technologies.

Apache Kafka serves as the message bus that funnels logs in from the various point vendors. Apache Cassandra serves as highly scalable repository for storing the logs. And Apache Spark provides the compute power to analyze the data and track down problems. All these products are pre-built into the OpsDataStore producdt, which itself is a clustered application that requires nine separate nodes (they can run on virtual servers, naturally).

Harzog’s secret sauce lies in the object model used by the product, and the graph-like topology mapping layer.

Share it:
Share it:

[Social9_Share class=”s9-widget-wrapper”]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

You Might Be Interested In

Should You Use A No-Code AI Platform? Limits and Opportunities

17 Feb, 2022

While a majority of AI projects still don’t reach production, the interest for no-code AI platforms keeps rising. Indeed, a …

Read more

Machine Learning Data for Self-Driving Cars: Shared or Proprietary

15 May, 2017

The crux of any machine learning approach involves data. You need lots and lots of usable data to be able …

Read more

AI and Lab Data: Enabling Decision-Making in Real Time

8 Sep, 2017

Lisa KerberBig data and Artificial Intelligence (AI) undoubtedly have begun disrupting how the business of healthcare is done. As a …

Read more

Do You Want to Share Your Story?

Bring your insights on Data, Visualization, Innovation or Business Agility to our community. Let them learn from your experience.

Get the 3 STEPS

To Drive Analytics Adoption
And manage change

3-steps-to-drive-analytics-adoption

Get Access to Event Discounts

Switch your 7wData account from Subscriber to Event Discount Member by clicking the button below and get access to event discounts. Learn & Grow together with us in a more profitable way!

Get Access to Event Discounts

Create a 7wData account and get access to event discounts. Learn & Grow together with us in a more profitable way!

Don't miss Out!

Stay in touch and receive in depth articles, guides, news & commentary of all things data.