Big Data Moves Toward Real-Time Analysis

Big Data Moves Toward Real-Time Analysis


It's clear there's a transformation in enterprise data handling underway. This was evident among the big data aficionados attending the Hadoop Summit, in San Jose, Calif., and the Spark Summit in San Francisco earlier this month.

One phase of this transformation is in the scale of the data being accumulated, as valuable "machine data" piles up faster than sawdust in a lumber mill. Another phase, one that's less frequently discussed, is the movement of data toward near real-time use.

The data warehouse, as valuable as it is, is history. The most valuable data will be that which is collected and analyzed during the customer interaction, not the review afterward. The analysis that counts is not the results of the last three months, or even the last three days, but the last 30 seconds -- probably less.

In the digital economy, interactions will occur in near real-time. Data analytics will need to be able to keep up. Hadoop and its early implementers, such as Cloudera and Hortonworks, have risen to prominence based on their mastery of scale. They gobble data at a prodigious rate, one that was inconceivable a few years ago.

Read Also:
History of Data Mining

Spark is the new kid on the block, an in-memory system that's not exactly unknown, but is still a stranger in data warehouse circles. IBM said it would pour resources into Spark, an Apache Foundation open source project.

Is it wise to focus as much attention and effort on Spark? The big data field is basically in ferment. There's RethinkDB, an ambitious Redis project or, for that matter, commercial in-memory SAP Hana. With so many initiatives underway, was it wise for IBM to announce that Spark is "potentially the most significant open source project of the next decade"?

At Spark Summit, Amazon Web Services announced a free Spark service running on Amazon Elastic Map Reduce, and IBM announced plans for Spark services on BlueMix (currently in private beta) and SoftLayer. These cloud services will open the floodgates to developers, and IBM’s contributions will surely help to harden the Spark Core for enterprise adoption.

Read Also:
Wordsmith's 'robot journalist' is now available to the public

Chief Data Officer Europe
20 Feb

15% off with code CDO7W17

Read Also:
Microsoft CEO Satya Nadella Has Much To Say About Artificial Intelligence
Predictive Analytics Innovation summit San Diego
22 Feb

$200 off with code DATA200

Read Also:
The Industrys First Enterprise Class Real-Time Streaming Platform
Read Also:
The Internet of Things: a Surveillance State in Disguise
Read Also:
Oracle Service Cloud to Unlock the Value of IoT Data

Leave a Reply

Your email address will not be published. Required fields are marked *