Data Integration helps organizations to manage their data and data flows in a very efficient way. It is seldom that organizations use one system to capture all their company data. Very often they use different systems for specific purposes like customer relationship management (CRM), Enterprise Resource Planning (ERP), finance, Human Resources and so forth. In many cases the same customer, product or employee data is captured and stored in different systems in different formats for different purposes.
Many organizations faces 3 big data integration challenges
The use of different systems give organizations a lot of flexibility. This and the huge amount of relevant (and also irrelevant) data outside the company creates for every company three big data integration challenges on the other hand:
- How to make sure that data elements are not getting obsolete? Changes in one system are not automatically transferred to other systems.
- How to create one view of the customer or employee data? That data in the different systems is not automatically combined in one database.
- How to combine external (big) data with company data easily? Read more about Big Data.
Different technologies: EAI & ETL
These three challenges are addressed in Data Integration. To get rid of the first challenge, organizations have to implement Enterprise Application Integration (EAI) technology. Often a Service Oriented Architecture (SOA) is used. A data warehouse has to be implemented to overcome the second challenge. In both cases you should consider using ETL tools. To deal with the third challenge you need in addition also a Business Intelligence tool with Big Data capabilities.
The fundament for Business Intelligence
To be able to make better decisions using KPIs and key insights across the enterprise the data needs to be integrated and combined into one single system. In our believe Business Intelligence can’t be successful without a proper data integration layer. That layer takes care of unified definitions, improving the data quality, combining data from different sources into a single table or cube and it should facilitate very fast response times. Data integration is in our vision the beating heart of the Intelligent Organization and it serves several vital functions. Read also our article ‘9 reasons to build a data warehouse’.
Overview of Data Integration tools
Each year from 2007 until now we investigated a large number of data integration tools based on over 90 criteria that play a major role in the quick and successful development of a BI infrastructure. In our study we concentrated on products that could be (relatively) easily used by either business or IT professionals with a minimum of specialized training. In our fully independent study we have divided the products into four separate categories to avoid comparing apples with pears.
1. Pure ETL Products
These data integration tools are independent of the database and the Business Intelligence tool with which they will probably be used. The companies do not rely on another product providing the functionality they offer, and they allow you to migrate to another (relational) database or another Business Intelligence tool without having to substantially change your integration processes.
List of pure ETL products:
- IBM Infosphere Information Server
- Informatica PowerCenter
- Oracle Data Integrator
- Talend Open Studio for Data Integration
- Pervasive Data Integration
2. Database Integrated
These data integration products are supplied as an option when you buy the database software; some of the functionality is built into the database and not available separately in the ETL tool itself. The ETL tool is database dependent and will not work with another database, this is not a restriction it is an element of the design. Replacement of the database would entail completely rewriting the integration processes.
List of database integrated ETL products:
- Oracle Warehouse Builder
- SQL Server Integration Services
3. Business Intelligence Integrated
These are products from the same supplier as the BI software. In many cases these are separate products and the supplier will claim that they can be used independently of the BI tool. This is probably true but because of their use of proprietary metadata, and sometimes the same scripting tools as the BI product they are seldom used separately.
List of BI integrated ETL products:
- SAP Data Services
- SAS Data Integration Studio
4. Niche Products
These are products that don’t fit well into any of the above mentioned groups, but still have considerable ETL functionality in them. In most cases they are not as complete as the other data integration tools we have looked at but they are often in a totally different price class and are very usable to solve certain problems. Traditionally we had included the Open Source products in this category since they lacked the full functionality and customer experience to qualify for the main categories.
Interested in our data integration services and independent advice?
Would you like to be informed about Data Integration and get professional 100% vendor independent advice? You may contact us and we will get back to you as soon as possible.