The promise of big data is incredibly enticing and, for most businesses, just as out of reach. The reason is simple: today’s databases are built upon 1970s math that was designed for 20-century data requirements and hardware capabilities. This math has led to tree-structures and associated algorithms that are incapable of delivering the flexibility, scale and performance needed for the dynamic big data world.
Even newer tree-structure variants, which were developed in an attempt to solve these issues, can’t keep up with big data entropy and velocity. Databases have rigid and complex data infrastructures, requiring an enormous amount of hand-holding to run (think calibration treadmill), and often sacrifice features for small upticks in performance and scale. Plus, they’re either read-optimized or write-optimized, not both. Yet if one could devise a tree-structure and algorithm that allowed seek time and write amplification to operate at their theoretical minimum, it would go a long way to empowering big data-capable performance.
That’s why a completely new, adaptive and machine learning-based approach to database technology is gaining traction. With machine learning, any database can become dynamic and optimal. Unlike classic tree-structures, which are governed by fixed mathematical limits and static behavior, this new database science delivers flexible data structures that adapt to behavior based on observed changes in the data and operational capabilities of hardware resources. Machine learning techniques continuously optimize how that data is organized in memory and on disk according to application workload and resource capabilities.
An adaptive architecture turns theoretical read and write minimums into reality. It delivers a write cost of seek zero and a read cost of seek one (point or scan)—enabling the breakthrough, “infinite-oriented” performance and scale big data requires.
An adaptive database relies on machine learning algorithms to model, aggregate and predict across three elements, focusing on resource behavior rather than fixed tree-structure behavior:
An adaptive database takes the workflow of data and, with seek-less writes, achieves maximum storage throughput. Using a Markov-like decision algorithm, it analyzes workflows during ingest to understand how information is changing and predict how it may change in the future. Relevant information is kept hot in memory and non-relevant information cold on disk, where cold data is continually defragmented and/or compressed.