We monitor the ETL market closely
Since 2003, the consultancy and analyst company Passionned Group has been closely monitoring the market for both ETL and Data Integration tools. In the past, the focus was on the well-known market leaders who were often regarded as visionaries and leaders. Many organizations used to assume that they had automatically made the right choice if they purchased a tool from one of these companies.
The market changed substantially
Since the late nineties, however, the market has changed substantially. Practically all the major Business Intelligence vendors have purchased or developed their own ETL tools. Given that a central data warehouse is one of the major cornerstones of a successful Business Intelligence solution, their choice to move into the ETL marketplace is understandable. After all, 70-80% of the costs involved in a successful Business Intelligence system are infrastructure, creating reliable ETL processes and a well designed method of keeping the data warehouse current is essential.
Survey motivation and set-up
- The first reason is that the focus of many organizations was on Business Intelligence tools for end users, such as OLAP, reporting and dashboards, while ETL, Integration and Data Warehouses form the cornerstones of a successful Business Intelligence solution. The time and energy spent on designing ETL processes correctly and selecting a matching ETL and data integration tool will pay dividends in the future. Thomas Siebel, the founder of Siebel, (now Oracle) was quoted in an interview: “Organizations may save a lot of money when valuable data stored throughout the organization in separate silos can be integrated and combined”.
- The second reason to evaluate and promote ETL tools, is that many organizations still build their data warehouse by hand, writing complex SQL and stored procedures. Many organizations still find themselves in the stone age. Data warehouses are still being developed manually using either SQL or PL/SQL, while developer productivity would be increased by a factor of three to five if a proper ETL tool was used. Additionally, the reliability and stability of data warehouses built using an ETL tool is much higher, since more aspects can be checked and monitored in relation to each other, metadata being a case in point. It should, however, be noted that merely using an ETL tool does not automatically guarantee success.
- Thirdly and finally: the market for Business Intelligence tools is growing again with expectations of large revenue increases this year for some of the suppliers, but there appears to be no increase in the success rate of the warehouses and BI systems that we are building. Despite the availability of better and better Business Intelligence software at lower and lower prices, we are not producing better systems for our users. Our mission is to find the factors that make an ETL project a success.
How our survey is carried out
- A list of 90 criteria was developed as a basis for the comparison, using input from the developer community and focusing on characteristics that play a major role in the quick development of a successful Business Intelligence infrastructure.
- Passionned Group conducted in-depth interviews with all the vendors listed on the ‘List of ETL tools’ page regarding their ETL solutions.
- In addition, vendors were given the opportunity to provide a live, two-hour demonstration in order to present the functionality of their ETL tool. PowerPoint presentations were avoided as much as possible. During these demos, Passionned Group evaluated whether the tool was able to handle particular situations by means of Q&A sessions. Where there was still doubt, Passionned Group gave the tool either zero or half a point.
- Vendors were asked to provide list prices for their tools by an official Request for Proposal (RFP), with two configurations (see also the criteria). This only gave a price indication since all the products are priced in different ways.
- In many cases an intensive dialog took place between Passionned Group and the vendors to obtain answers to outstanding questions. Ambiguous or misleading answers were not accepted. All answers were confirmed in writing by the suppliers to avoid misunderstandings.
- Finally, a number of developers from end-user organizations, as well as Business Intelligence consultants, provided feedback on the results of the ETL study. Whenever the necessary, we inquired further and the evaluation was altered accordingly.
Introduction and structure of the ETL study
Many organizations used to assume that they had automatically made the right choice if they purchased a tool from one of the market leaders. Since the late nineties, however, the market has changed substantially. Practically all the Business Intelligence (BI) vendors have purchased or developed their own ETL tools. Since a centralized data warehouse is one of the cornerstones of a successful BI solution, this has turned out to be a wise choice. Market estimates show that 70-80% of the costs of a successful BI system relate to the creation of reliable ETL processes and data integration.
A large number of ETL tools are compared in our study
Each year from 2007 until now the Passionned Group investigated a large number of ETL tools based on over 90 criteria that play a major role in the quick and successful development of a Business Intelligence infrastructure. In our study we concentrated on products that could be (relatively) easily used by either business or IT professionals with a minimum of specialized training.
A number of products have not been included in the final report for various reasons; Ab Initio did not reply to any of our attempts to contact them. Sybase has currently spread their functionality across different products and it was impossible for us to judge which functionality was in which product. In this edition of the fully independent study we have divided the products into four separate categories to avoid comparing apples with oranges.
1. Pure ETL Products
These products are independent of the database and the Business Intelligence tool with which they will probably be used. The companies do not rely on another product providing the functionality they offer, and they allow you to migrate to another (relational) database or another Business Intelligence tool without having to substantially change your integration processes. This year Oracle’s ODI product has been added to this category, ETL Solutions’ product Transformation Manager has been removed and Pitney Bowes Data Flow product has been moved to the Niche category.
2. Database Integrated
These products are supplied as an option when you buy the database software; some of the functionality is built into the database and not available separately in the ETL tool itself. The ETL tool is database dependent and will not work with another database, this is not a restriction it is an element of the design. Replacement of the database would entail completely rewriting the integration processes.
3. Business Intelligence Integrated
These are products from the same supplier as the Business Analytics software. In many cases these are separate products and the supplier will claim that they can be used independently of the BI tool. This is probably true, but because of their use of proprietary metadata and sometimes the same scripting tools as the BI product, they are seldom used separately.
4. Niche Products
These are products that don’t fit well into any of the above-mentioned groups, but still have considerable ETL functionality in them. In most cases they are not as complete as the other products we have looked at but they are often in a totally different price class and are very usable to solve certain problems. Traditionally we had included the Open Source products in this category since they lacked the full functionality and customer experience to qualify for the main categories. We have moved Talend to the category Pure ETL tools since both their level of functionality and market acceptance justify their inclusion.
The new release of CloverETL has added so much functionality that it would qualify for inclusion as a Pure ETL product, however it still has a very limited user base and as such still fits better into the Niche category, although for companies looking for a very reasonably-priced alternative to the major products this would certainly be a good place to start. After a great deal of discussion we have added Syncsort’s DMExpress product and Expressor’s Data Integrator Platform to this category.
DMExpress has the functionality to be considered a Pure ETL product, but certainly not the user base, and Syncsort themselves position it more as an accelerator than a full-function product. Expressor was bought by QlikTech just after we had done our research and may well be moved to the Business Intelligence Integrated category at some time in the (near) future.