Four years ago, when we published the second edition of this survey, we saw that not many ETL tools had good, reliable functionality for real-time application integration (EAI) projects. Since then, many ETL tools have added tools for real-time extraction, transformation and integration, and there has been an almost complete convergence between ETL and EAI tools into a new market which is being called Data Integration.
Best of both worlds: a marriage
It’s clear that the vendors think that the future lies in a marriage between ETL and EAI, but when asked most will admit that they have very few customers yet who are using what amounts to real-time ETL. The table below shows that real-time systems and traditional ETL come from very different backgrounds, but there are examples where combining the two produces the best of both worlds.
Likely to merge completely
Many of the suppliers have now integrated these tools, but ETL and EAI tools still satisfy different customer needs. Each market will probably continue to exist on its own, and have its own user population, mainly because that’s what the majority of the customers still want. We’ve seen that these tools increasingly make use of a common metadata layer, a repository of transformations and business logic, and standard adapters to connect to a wide variety of systems.
Data is entered only once in the enterprise
Finally, the nature of ETL for Business Intelligence purposes, differs in many aspects from the nature of EAI. ETL is about moving, transforming, and cleaning large amounts of data no more than a few times a day from many sources into one place – convergence. EAI, meanwhile, often moves relatively small amounts of data, transactions, spreading them across various systems – divergence. We use ETL for data migration, for one view of the truth or data integration to allow better decision making. In general, we use EAI for process optimization and workflows and to make sure that data is entered only once in the enterprise, although some companies are moving towards real-time Business Intelligence and using EAI tools to keep the data warehouse up to date.
ETL and EAI have a strong relationship
The marriage looks set to go ahead, then. From the vendors’ point of view, IBM, Microsoft, Oracle, and other vendors in the EAI and ETL market are producing a unified platform, which makes it look as if the products will at least be living together. Recently, we’ve seen acquisitions where integration and business intelligence tools from companies like BusinessObjects, Cognos, and Datamirror have been bought by platform vendors. With the consolidation in this marketplace, many of the suppliers are no longer dependent on one product for their revenue, so as long as the customer keeps using their revenue-generating product, which may be a database or a particular brand of front-end software (like MS Office), the costs of the ETL and the broker software will continue to be reduced.
In our opinion, ETL and EAI have a strong relationship, should use the same business rules and definitions, but have a different architecture – perhaps it looks like a marriage after all!
Strengths of each: ETL vs EAI:
- Excels at bulk data movement & batch data integration
- Provides for complex transformations, aggregation from multiple sources and sophisticated business rules.
- Assumes considerable data delays.
- Is batch-oriented, making it fast and simple for one-time projects and testing
- Offers little in the way of workflow
- Works primarily at the session layer
EAI: Enterprise application integration systems
- Are limited in data movement capabilities
- Offer less sophisticated transformation and extraction functions
- Operate in real time
- Work better with continuously interacting systems
- Are workflow-oriented at their core
- Work primarily at the transport layer