On Tuesday, at the 2016 AWS re:Invent conference, Siva Raghupathy, senior manager for solutions architecture at Amazon Web Services, shared five architectural principles that could make it easier for your business to build out a big data solution in the cloud.
Big data is all about the three Vs—volume, velocity, and variety of data. However, those Vs are increasing dramatically, Raghupathy said. And, as big data evolved from batch processing to stream processing to machine learning, the considerations for building a system become much more complex.
When speaking with customers, Raghupathy said that the core challenges usually center around the questions of whether or not there is a reference architecture, what tool should be used, how it should be used, and why it should be used. To better address these questions, Raghupathy outlined the following five architectural principles for building big data solutions in the cloud. AWS customers can then apply these principles as they choose products to fit their use case.
When Raghupathy talked about building decoupled systems, he gave the example of an automobile. The engine and wheels in a car are separate, but they are connected by the gearbox, which helps them work together.
In big data, though, that decoupling mechanism is the storage subsystem. Decoupled systems in big data are important because they allow you to alter one aspect of the system without affecting the other.
Amazon has many different products for big data systems, so it can be very difficult to choose the right one.
