A system must undergo "modernisation" when it can no longer address contemporary problems sufficiently. Many systems that now need overhauling were once the best available options for dealing with certain challenges. But the challenges they solved were confined to the business, technological, and regulatory environments in which they were conceived. Informatica, for example, was founded before the internet. It goes without saying that enterprise integration has changed profoundly since then.
One set of systems that desperately needs modernising is traditional on-premises data architecture. The huge increase in the volume, variety, and velocity of data today is baffling legacy systems. At present, legacy data architectures are bending beneath the weight of these three data-centric challenges. Soon they may break.
Cosmic amounts of data glut our world. Every day, 3.5 billion Google searches are conducted, 300 million photos are uploaded to Facebook, and 2.5 quintillion bytes of data are created. IDC predicts global data will grow ten-fold between 2016 and 2025 to a whopping 163 zettabytes.
We process a remarkable five billion documents a day for just one of our customers. And that’s just for one company.
Managing these surging volumes of data in an on-premises setting is unsustainable. IT ends up pouring valuable time and resources into purchasing, installing, and managing hardware. They also have to write heaps of code to operate the systems in which the data resides (e.g., databases, data warehouses, etc.). Organisations that permit such an approach to data management will never achieve the depth of analytics needed in the digital economy. They will be like surfers endlessly paddling near the shore without ever getting past the breakers.
Most data was of a similar breed in the past. By and large, it was structured and easy to collate. Not so today. Now, some data lives in on-premises databases while other data resides in cloud applications. A given enterprise might collect data that is structured, unstructured, and semi-structured. The variety keeps widening.Â
According to one survey, enterprises use around 1,180 cloud services, many of which produce unique data. In another example, we integrated over 400 applications for a major enterprise IT firm.
The process of integrating all this wildly disparate data alone is too great a task for legacy systems. Within a legacy data architecture, you often have to hand-code your data pipelines, which then need repairing as soon as an API changes. You might also have to oversee an amalgam of integration solutions, ranging from limited point-to-point tools to bulky platforms that must be nurtured through scripting. These traditional approaches are slow, fraught with complexity, and ill-matched for the growing variety of data nowadays. Legacy systems largely thwart companies' efforts to use the data they collect.
Scenarios in which you needed high-speed data processing were far fewer in years past than what we see today. Now, mission-critical operations rely more and more on real-time data processing.
