Data analytics platforms may be proving themselves capable of drawing out previously unseen patterns around user behaviour, but organisations wanting to use them for security analytics need to position log data in the broader context of organisational norms, the head of a global analytics innovator has advised.
Increasing standardisation of big-data techniques had driven a surge in the use of the technologies to aggregate both structured and unstructured data for later analysis, Martin Ashby, APAC principal with Hortonworks, told CSO Australia.
This trend – and the proliferation of the Hadoop data platform and related functional tools for governance, security, integration, and more – had eased the learning curve for utilising big data to solve business problems. However, organisations wanting to effectively apply big-data techniques needed to also remember to position security events within a business context so they can formulate and execute appropriate remedial actions.
“We're seeing [growth in big-data security] unilaterally around the region and across industry types,” Ashby explained. “The earliest and fastest moving industries are essentially the ones with the largest data sets and volumes. Once you can start to look at the bigger picture and business information, you can then start to look at things like security threats.”
Those threats often manifest as a range of different activities – for example, repeated incorrect password attempts that have no meaning until they are correlated with a specific IP address – that might go under the radar unless they were tied to higher-level business outcomes.
“When there was a single breach in the system, it wouldn't necessarily have been protected until you start to detect a certain level of things,” Ashby said. “Detection of these threats, and being able to lock those down, becomes crucial and there are a number of cases where people have started to use those at a large-scale level.”
Big-data vendor Splunk recently reported that fully 40 percent of its worldwide business – and half in Australia – is related to the collection and analysis of security data.
Like Hortonworks, Splunk has built a robust analytics business on top of the open-source Hadoop platform – which has rapidly become the go-to analytics platform for large businesses storing masses of security-related information.
Hadoop – which was bolstered by the 2011 split of 24 Yahoo! developers to form Hortonworks on top of the platform – had provided a common technology base for big-data vendors to build on top of, says Ashby.
The company had developed “close to 20” different open-source projects that provide functionality around governance, security, and integration for its Hortonworks Data Platform (HDP), he said, adding that a number of other projects dovetail with the platform – including YARN (Yet Another Resource Negotiator), a 'data operating system' that links the Hadoop Distributed File System (HDFS) with high-level application modules.
