Data warehousing has reached an impasse. Born in an era of monolithic systems walled off from the outside world, the data warehousing technologies that have dominated the last thirty years stand today as an unsightly reminder of the wear and tear taken by increasingly vain attempts to help them stumble forward into the modern world.
The legacy data warehouse has struggled because it suffers from the limitations of an approach and architecture defined by the constraints of the physical world, a world in which resources were limited, applications ran in isolated silos, and changes were infrequent and carefully controlled.
The limitations born from that world are today the chains that shackle the data warehouse, tying it to limitations of the past and preventing it from addressing the realities of the present and future. Simply put, the legacy warehouse was not designed for the volume, velocity, and variety of data and analytics demanded by the modern enterprise.
Data analytics, once accessible only to the largest and most sophisticated global enterprises, today is a top priority for marketing, finance, development, and sales at organisations of all types and sizes. Stories of the outsized impact of data analytics only increase the demand for better insights, for example, the story of how utilities can realise a 99 percent improvement in accuracy by harnessing big data.
However, the growing demand for data analytics is shining a spotlight on the painful inadequacy of legacy data warehouses. For one, they were never designed for the sheer volume and pace of data that exists today. In large enterprises the volume of data generated within the organisation is overwhelming. For example, in just one year Rolls-Royce generates over three petabytes of data in the manufacturing of fan blades for turbines. Mining that data is essential to design, manufacturing, and after-sales support. Exploding volumes of data are not just a large enterprise problem even mid-sized organisations that may not generate such volumes of data internally have access to large external data sources such as data.gov.uk.
Traditional data warehousing, with its chains of the physical world, is prohibitively expensive and inflexible, limitations that become painfully obvious with the scale of todays data. The limitations force difficult capacity planning exercises and large upfront expenditures to acquire sufficient capacity in advance to avoid a painful, disruptive exercise to scale your data warehouse.
Not only is the scale of data a challenge for organisations, so is the demand for faster results. Not so long ago users would have been happy to get updated analytics on a weekly or daily basis. Today that would be seen as a failure users expect results in seconds and expect those insights to be based on data that is continuously updated, even as the number of people with direct access to data and analytics grows.
Again, the limitations of traditional warehousing become painfully clear.
