As organizations look to increase their agility, IT and lines of business need to connect faster. Companies need to adopt cloud applications more quickly and they need to be able to access and analyze all their data, whether from a legacy data warehouse, a new SaaS application, or an unstructured data source such as social media. In short, a unified integration platform has become a critical requirement for most enterprises.
According to Gartner, “unnecessarily segregated application and data integration efforts lead to counterproductive practices and escalating deployment costs.”
Don’t let your organization get caught in that trap. Whether you are evaluating what you already have or shopping for something completely new, you should measure any platform by how well it address the “three A’s” of integration: Anything, Anytime, Anywhere.
For today’s enterprise, the spectrum of what needs to be integrated is broader than ever. Most companies are dealing with many different data sources and targets, from software-as-a-service applications to on-premises ERP/CRM, databases and data warehouses, Internet of Things (IoT) sensors, clickstreams, logs, and social media data, just to name a few. Some older sources are being retired, but new sources are being added, so don’t expect simplicity any time soon. Instead, focus on making your enterprise ready for “anything.”
Beyond point-to-point. You may have managed integration before on a point-to-point basis. This approach is labor intensive, requiring hand-coding to get up and running, and additional coding any time there’s a change to either “point.” Integration of your endpoints could run into trouble when this happens, and then you would have to wait for your IT department to get around to fixing the issues. But the more serious problem is that this inflexible approach simply doesn’t scale to support enterprise-wide integration in a time of constant change.
Some modern concepts, when applied to integration, provide this flexibility and scale.
Microservices. An architecture approach in which IT develops a single service as a suite of small services that communicate with each other using lightweight REST APIs, microservices have become, during the past year or so, the standard architecture for developing enterprise applications.
When applied to integrations, these open up tremendous opportunity for achieving large-scale integration at a very low cost. Instead of one big execution engine running all integrations, smaller execution engines run some integrations. This way, you can supply more compute power to the integrations that need it, when they need it. You also can distribute the integrations between nodes on a cluster based on volume variations for horizontal scaling.
The document data model. Today’s modern applications produce more than just row and column data. So how do you achieve loose coupling while simultaneously supporting semi-structured and unstructured data, all without sacrificing performance? You can group data together more naturally and logically, and loosen the restrictions on database schema, by using a document data model to store data. Document-based data models help with loose coupling, brevity in expression, and overall reuse.
In this approach, each record and its associated data are thought of as a “document,” an independent unit that improves performance and makes it easier to distribute data across multiple servers while preserving its locality. You can turn object hierarchical data into a document. But this is not a seamless solution.;
