This is our blog about designing and implementing Intelligent organizations. Within this area of expertise we write often about the following subjects BI, analytics, the tools, decision management, data visualization, BI success, Business Intelligence concepts, data management, continuous improvement, and the organization of BI & Analytics.
Of course you can order our book here too.
This article discusses the required project approach to the competencies and roles needed to merge the Business Intelligence processes, the architecture, the tools, and the applications into a lasting, working entity. Business Intelligence projects differ from ‘traditional’ system development projects in several ways, namely:
They often go beyond the boundaries of departments, processes and even business units;
The Intelligent Organization will uses visualization and simulation to ensure that employees can rapidly absorb information and knowledge –which encourages action. After all, a picture is worth a thousand words and “every problem can usually be solved, provided that we use an appropriate model” (Winston, 1992).
Even though people have their own specific preferences when it comes to tables and graphs, a single good quality picture can often communicate a message much faster. A picture may tell us, at a glance, whether our market share increased or decreased in relation to prior periods. Of course you can still use tables (with text), however graphs and charts offer much more variety when it comes to depicting information effectively so that we can quickly interpret that information.
We use dashboards, reports and interactive analysis to enable us to see general and simple relationships within business operations – for example, more customer visits lead to an increase in cross-selling which leads to better financial results – visible. The smaller, more specific and more complex relationships surface when we use data mining.
“Data mining is the uncovering of hidden (unknown) relationships or segments in large data collections that have a predictive value for a specific part of the business operations”.
On the path towards maturity in the field of metadata, an organization can go through four levels of ambition. These are shown in the figure below.
Figure: The four levels of ambition of metadata
Level 0: an organization is on this level if there is hardly any need for metadata, for example because all applications store data in one and the same database from which they also retrieve the data. The applications are more or less isolated, which means that hardly any data is exchanged between them. The organization is structured as such and managers primarily focus on their own departments.
From the outside a portal looks a bit like a stained glass window. The little windows within it – the so-called portlets – represent the different parts of the portal.
Portals offer direct, centralized access to relevant information and business applications through intranets or secured extranets (pull). Portals also ensure that the relevant information – or a reference to it – is sent to users (push).
A Data Vault is a modeling technique for the CDW, designed by Dan Linstedt, which chooses to store all incoming transactions regardless of whether the details are in fact trustworthy and correct: “100% of the data 100% of the time”.
For example: a sales transaction has already taken place, but the corresponding customer does not yet exist in the CRM system. The sales transaction can nonetheless be stored in the CRM system. When the customer becomes known to the system, the transaction changes from a ‘meaningless’ fact into a useful ‘truth’ because now its context is known.
The Business Intelligence architecture will not be able to produce good results if we cannot link tools to it. This blog provides an overview of the most important Business Intelligence tools (BI Tools) in relation to the decision-making cycle. It also describes the purposes, the features and the workings of these tools.
Each Business Intelligence Tool plays a supporting role in one or more phases of the decision-making cycle. The decision-making cycle comprises of the following phases, which can be supported by the following tools (see figure below):
For a proper Business Intelligence system we need to have some sort of ETL in place. When extracting and filtering data from the source systems the following aspects are important:
Indicators and other types of management information need data from tables in the source system. If this requires one or more attributes, it is highly recommended to copy ALL attributes (from a table) to the SA. Why? Well, firstly it is simpler: instead of having to name each attribute explicitly, the table name will do*. Secondly, we can produce indicators or dimensions – which should be based on data in tables that are already in the SA – faster. After all, there is no need to first adjust the extraction. The disadvantage is that more data need to be processed, which may be a burden on the loading time of the SA.
A pharmaceutical wholesaler wanted to find out more about its market performance compared with the performance of other wholesalers in the industry. The data required to calculate the market shares were only partially available in the wholesaler’s source system, namely the order files.
In order to achieve exchange of sales data – anonymous of course – between wholesalers in the same line of business a negotiation took place at board level. As a result, the wholesalers established a joint foundation with the aim to once a month process all sales data – according to a specific format – into one large file containing all revenue data divided by product level and month level.
More and more organizations are wondering what the use is of a data warehouse, and whether or not it’s worth the investment. They also want to know what alternatives are available. A growing number of IT vendors and some “experts” claim that the end of the data warehouse is nigh. When we say vendors, we’re referring to suppliers of data warehouse appliances, data virtualization tools, and data discovery tools. We have a different opinion, though. The data warehouse still is the beating heart of the Intelligent organization and it serves different vital goals.
The aim of the collection process in the Business Intelligence cycle is to collect, filter, cleanse, combine, transform and aggregate data from different internal and external sources in order to increase the likelihood of good management information and useful knowledge during the process of analysis.
This process is usually supported by a data warehouse, which includes a specific architecture that fits in with both the information needs and the technical infrastructure. The data warehouse must ensure that data is transformed into information and knowledge that encourages action.
The architecture of a Business Intelligence system is guided by a good number of basic principles that specifically apply to Business Intelligence systems. A basic umbrella principle here is the well-known phenomenon of ‘structure follows function’: the indicators that are derived from either the business processes or the strategy as well as additional management information largely determine the architectural structure of the Business Intelligence system. In other words: the content of the system determines its architecture and structure.
The Intelligent Organization observes well. It transforms incoming signals into information and knowledge and responds adequately. This is only possible when the different processes within the Business Intelligence cycle are supported by a good architecture.
The architecture ensures that the processes run smoothly and that they are properly aligned. The Business Intelligence architecture forms the link between the processes we described earlier and the applications and Business Intelligence tools. Without a well thought out architecture, we cannot properly organize the processes of the Intelligent Organization and consequently, we cannot apply information and knowledge in the way we should.
An Intelligent organization should keep its eyes open and prick up its ears for data and information. The information it receives should be interpreted, internalized and revised to eventually be distributed across as many processors (people, systems) as possible. However, Business Intelligence does not stop with the distribution of knowledge that encourages actions.
Business Intelligence goes beyond that: ultimately, the information must contribute to a range of aspects such as a higher profitability, better products or faster processing times, all depending on the goals and ambitions. What can or must organizations do in order to actually reap the fruits of their investments in Business Intelligence?
For an organization to become as smart and agile as possible, the process of transforming data into information and knowledge should take as little time as possible and should run with minimal interruptions and errors. An Intelligent Organization will therefore also have to take into account a number of generic requirements that we impose on both consuming information and the Business Intelligence tools. We present you here the top 5 requirements for Business Intelligence.
Only those with the greatest capacity to adapt will survive. Darwin’s famous biological principle ‘survival of the fittest’ examines the passive capacity of living creatures to adapt to their environment. Those who least adapt are the first to be ousted or eaten. This way, individuals that do survive are enabled to reproduce and pass on their genes – and their apparent qualities – to their offspring – with a greater chance for survival as a result. However, minor random mutations in the genes may change this effect. It could be enhanced but also be reduced or even be destroyed.
Most organizations tend to look at only one of the many facets of Business Analytics. Often the focus is on the technical side, or simple reporting, or possibly the internal organization, which is a pity, because Business Intelligence can add significant value in many areas: alignment with and influence on the external environment – things that are going on outside the organization, optimisation of internal processes, productivity improvements of both the people and the machines and last but not least more effective use of the technology and the mountain of data within our walls. The business case for Business Intelligence is both broad and varied.
As a result of the pressure in the current work climate the amount of time that managers and staff have to take the right decisions is decreasing rapidly, although the complexity of the decisions is increasing. This is complicated further by the fact that the amount of data needed to take good decisions is increasing exponentially*, the so-called Big Data. These two factors, time to decide and the volume of data, are diametrically opposed. As illustrated below the Business Intelligence gap will only increase unless organizations taken steps to tackle the problem.
Business Intelligence is not only about developing a better understanding of the organization; it has wider ambitions: improvement and innovation.
Level 1 – understanding: provide insight into what is actually happening in an organization, for example by monitoring the customer response times and the number of complaints received. This gives a better view of how an organization is running and shows how various internal processes are intertwined.
A recent study by one of the analysts firms tells us that “in 2011 the world will create a staggering 1.8 zettabytes” and “by 2020 the world will generate 50 times the amount of information [now]”.
In this article we explore the three biggest challenges of using big data and what to do about it: intelligent filtering, outstanding performance and good data visualization. The main question here is if the business intelligence software we are using today is capable of tackling these challenges.
It’s not news that all big organizations are changing quickly at the moment. Trends like layoff rounds, different visions and goals, changing markets, and so on, are daily occurrences. Change is the most stable factor at the moment. Innovation, speed, flexibility, and taking advantage of the changing market are crucial factors.
There’s a demand for a faster time to market, other products and services that are a better fit for this moment. But having to react quickly to new developments like social media, online sales, and the explosive growth of data has a downside: how can the organization keep everything under control, and how do you know if everything is going well?
In a recent poll among the visitors to this website, it has become clear that mobile Business Intelligence (BI) is no longer just a hype. More than 75% of those who voted, expressed that they believe that Business Intelligence will become completely mobile in the near future.
This is hardly a strange outcome since the business case for mobile Business Intelligence is very strong. In addition, managers love gadgets like smartphones and tablets. The future of Business Intelligence is mobile. We asked visitors: “Business intelligence will become completely mobile in the near future”.
The importance of business analytics (BA) was revealed in a recent Bloomberg survey, which found that BA has been effective in decision making for three out of four enterprises. Among the improvements are increased profitability, reduced cost, faster decision making, and critical performance improvements.
BA refers to technologies, applications, skills, and practices for the investigation of past business performance to improve insight into this past performance. BA is much more than merely providing simple data to a business. By using BA tools, meaning can be found in data, which results in a business improving its business intelligence. Some examples of Business Analytics include statistical analysis, decision processes, web analytics etc.
Question: Which ETL tools can support Data Vault modeling out-of-the-box? What are the challenges and issues building a data vault with ETL tools?
1. Marcel de Wit – See another discussion on LinkedIn (still active; Dutch)
2. Daan van Beek – Thanks Marcel, I did read the comments of that discussion too, it was the reason I started this discussion actually. After reading it, it was still unclear to me whether ETL tools do support Data Vault out-of-the-box like slowly changing dimensions or not. So, who knows the answer(s)? The vendors?
You can still hand-code an extract, transform and load system, but in most cases the self-documentation, structured development path and extensibility of an ETL tool is well worth the cost. Here’s a close look at the pros and cons of buying rather than building.
The Extract, Transformation, and Load (ETL) system is the most time-consuming and expensive part of building a data warehouse and delivering business intelligence to your user community.
Ab Initio has been around now for a long very time in the ETL market space to make a name for themselves. Their marketing approach appears to be one of mystique: maintain secrecy around the product while allowing some information out about the high-end customers, creating interest because of the tantalizing tidbits they provide.
Their website is more typical of an advertising firm with nothing meaningful to say than a technology company. Years later (2015) this is still the case. Their website loads very slowly and is full of flash (no HTML) and there is even no phone number or contact form. Our guess: they are finished.
For the last three months of last year Passionned Group have run a poll on their ETLtool.com site asking visitors what they thought were the most important requirements when choosing an ETL tool. The ETLtool.com site has been running since 2005 and advises visitors on the various ETL (data integration) tools available, what their strong and weak points are, how to choose an ETL tool and sells a 100 page report where popular products are analyzed and compared to facilitate choosing the right product for individual circumstances.