This is our blog about designing and implementing Intelligent organizations. Within this area of expertise we write often about the following subjects BI, analytics, the tools, decision management, data visualization, BI success, Business Intelligence concepts, data management, continuous improvement, and the organization of BI & Analytics.
Of course you can order our book here too.
Pentaho Business Analytics 5.0 greatly simplifies the entire analytics experience for everyone and delivers the industry’s first just in time big data blending ‘at the source’
Delivering the future of analytics, Pentaho Corporation announces the availability of Pentaho Business Analytics 5.0, a completely redesigned data integration and Business Analytics platform. Pentaho 5.0 provides a full spectrum of analytics for today’s big data-driven businesses regardless of data type and volume, IT architecture or analysis required. The new, modern interface simplifies the user experience for all those working to turn data into competitive advantage. Pentaho 5.0 includes more than 250 new features and improvements. Highlights include:
How to manage Business Intelligence projects? Business Intelligence projects require a specific project approach – the BI project cycle – because they significantly differ from traditional system development projects on a number of aspects.
The BI project cycle starts with awareness: we need to be aware of the nature and character of Business Intelligence. Once this is clear, we start looking for the business cases and we determine the scope. The blueprint of the BI system is created by using the basic principles that govern the BI architecture whilst defining the indicators and dimensions (information analysis) based on which we create a functional design that ultimately leads towards both a balanced architecture and a suitable data model.
Automation is always about people. This is certainly true with regard to ‘automating’ decisions and transforming data into ‘actionable intelligence’.
The point is that people will use this information and start acting differently. In order to create a proper (well-designed) Intelligent organization, we will need people in certain roles, with specific (behavioral) competencies, experience and knowledge, first on a project basis and later in the daily operations. In this article, we describe the ideal Business Intelligence project organization. Note that, in practice, we do not always require all roles or we cannot always facilitate all roles, due for example to budgetary constraints.
Business Intelligence projects are characterized by an extremely high risk factor and many obstacles. These obstacles are mostly related to the fact that Business Intelligence projects typically go beyond the boundaries of departments, processes and even business units; contain a mix of strategy, business operations and technology and are often highly political.
Also, the BI systems are derived from the operational information systems. The obstacles refer to the ten most common and most important forces and risks that can make or break a Business Intelligence project, which are divided into three main categories:
If we do not predetermine what we test and how we test the Business Intelligence system, then this testing process can be very time consuming.
Precisely because Business Intelligence allows flexible reporting, we are initially tempted to test all possible combinations of indicators and dimensions. What we forget is that, even in small Business Intelligence systems with for example ten indicators that are linked to eight dimensions, the number of possible combinations can reach ten million!
Based on the business requirements, the user groups and the architecture, we determine what tools we need in order to collect the data and to create reports and analysis.
We also determine what tools we require for the distribution of data and reports. In addition to the specific Business Intelligence tools mentioned in another post, we will also need a ‘relational database management system’ (RDBMS) for the storage and retrieval of the data.
When we have (too) low expectations of Business Intelligence and we operate in a highly political environment, this may result in a return on investment (of BI) that is far below its potential.
A year ago, a transport company with five thousand employees worldwide, implemented a well thought out and professional Business Intelligence system, which offered the possibility to not only monitor market share figures and revenues, but also to present various financial details and ratios.
It is important to determine whether and how Business Intelligence can be profitable within the organization and in which way we can achieve lasting success. In other words: what is the business case (for BI)?
The benefits of Business Intelligence can be split into four categories (Liautaud and Hammond, 2001):
not immediately apparent;
Does successful Business Intelligence start from the business, the processes and the strategy, or from the information systems and technology that support the business? Should an organization first collect the data (data-driven) or must we first map out our strategy and the associated information needs (business-driven)?
The answer to both questions – as it appears in practice – is that we need to combine the approaches: top-down and bottom-up, business-driven and data-driven. However, Business Intelligence must always begin with (or from) the business . It is, after all, called ‘business intelligence’. BI projects that are not managed by the business are unlikely to succeed, simply because on completion the project results will typically not match with the organization’s ‘company language’ and information needs.
This article discusses the required project approach to the competencies and roles needed to merge the Business Intelligence processes, the architecture, the tools, and the applications into a lasting, working entity. Business Intelligence projects differ from ‘traditional’ system development projects in several ways, namely:
They often go beyond the boundaries of departments, processes and even business units;
A large French retail organization in the food industry that operates as a franchisor was not able to measure the effects of its advertisements correctly. Due to the economic decline, the organization was forced to minimize the marketing costs and to try to maximize the impact of its marketing efforts at the same time.
A major challenge was that different advertisements in the media (TV, online, papers) were mixed up and largely overlapped.
The Intelligent Organization will uses visualization and simulation to ensure that employees can rapidly absorb information and knowledge –which encourages action. After all, a picture is worth a thousand words and “every problem can usually be solved, provided that we use an appropriate model” (Winston, 1992).
Even though people have their own specific preferences when it comes to tables and graphs, a single good quality picture can often communicate a message much faster. A picture may tell us, at a glance, whether our market share increased or decreased in relation to prior periods. Of course you can still use tables (with text), however graphs and charts offer much more variety when it comes to depicting information effectively so that we can quickly interpret that information.
We use dashboards, reports and interactive analysis to enable us to see general and simple relationships within business operations – for example, more customer visits lead to an increase in cross-selling which leads to better financial results – visible. The smaller, more specific and more complex relationships surface when we use data mining.
“Data mining is the uncovering of hidden (unknown) relationships or segments in large data collections that have a predictive value for a specific part of the business operations”.
On the path towards maturity in the field of metadata, an organization can go through four levels of ambition. These are shown in the figure below.
Figure: The four levels of ambition of metadata
Level 0: an organization is on this level if there is hardly any need for metadata, for example because all applications store data in one and the same database from which they also retrieve the data. The applications are more or less isolated, which means that hardly any data is exchanged between them. The organization is structured as such and managers primarily focus on their own departments.
A marketing employee of an importer of well-known car brands notices on her dashboard that the car sales in the Northern region are much lower than budgeted.
She switches from the dashboard to the interactive analysis mode and performs analysis per postal code region and per household type. The analysis shows that lower sales mainly occur in two-person households in rural areas. She consults (via chat) with the responsible dealer in the region to verify that the dealer knows about the drop in sales and whether he knows what the reasons might be.
From the outside a portal looks a bit like a stained glass window. The little windows within it – the so-called portlets – represent the different parts of the portal.
Portals offer direct, centralized access to relevant information and business applications through intranets or secured extranets (pull). Portals also ensure that the relevant information – or a reference to it – is sent to users (push).
In 2011, a British pension fund with assets of around 256 billion Euros realised a state-of-the-art data warehouse with an emphasis on the performance of different funds and portfolios.
All kinds of reports and analysis were used to achieve the highest possible return at an acceptable risk. For investment in shares, the front office – the people who commission stock market trading – used different operational systems that were linked to the systems of the brokers at the stock exchange. The performance was monitored daily and where necessary assets (shares, derivatives, bonds) were acquired or disposed of, via the operational systems.
A Data Vault is a modeling technique for the CDW, designed by Dan Linstedt, which chooses to store all incoming transactions regardless of whether the details are in fact trustworthy and correct: “100% of the data 100% of the time”.
For example: a sales transaction has already taken place, but the corresponding customer does not yet exist in the CRM system. The sales transaction can nonetheless be stored in the CRM system. When the customer becomes known to the system, the transaction changes from a ‘meaningless’ fact into a useful ‘truth’ because now its context is known.
Interactive analysis and OLAP is often regarded as the most important innovation in the world of Business Intelligence.
The eyes of analysts and managers often begin to sparkle when they see the possibilities of interactive analysis and OLAP for the first time: “This is what we always wanted to have!” Their enthusiasm is not without reason: this powerful tool enables us to quickly and casually ‘browse’ through large volumes of data, regardless of whether we search for specific information. This allows us to quickly approach data with a varying level of detail from different angles.
The business unit of one the three largest construction companies in Europe, which is responsible for the construction and maintenance of the railways, wishes to improve its information supply. The internal organization creates a very large amount of management reports and analysis using the popular Microsoft Office tools Access and Excel.
The set of queries and reports kept growing and growing to a respectable total of one thousand. After a few years working this way, a sudden major problem occurred for the Board of Directors to deal with.
We use reporting – just as dashboards – in order to display the known key success factors and the associated indicators. Both already prepared in the data warehouse. In this way, we can signal potential problems in business operations at an early stage.
Reports typically provide detailed information. This is why they use both data from data marts (or cubes) and detailed data from the ‘Central Data warehouse’. Reports generally display information in charts or tables, although alternative visualizations are also possible. An example of a report is shown below.
Interactive analysis or reporting? A well-executed – business-driven – information analysis ensures that many potential decision problems are already on the ‘agenda’ and that information and goals demonstrate coherence.
The indicators that are crucial for the realization of our strategy, for accomplishing our goals and for achieving our mission, are placed on the dashboard or in the reports. Interactive analysis powered by OLAP – playfully analyzing enterprise data – allows us to identify additional indicators and key success factors.
Informatica Corporation, the world’s number one independent provider of data integration software, today announced Informatica MDM 9.6, a major milestone in Informatica’s roadmap for powering universal master data management (MDM). The announcement came at Informatica’s annual user conference – Informatica World 2013.
Universal MDM depends on four essential capabilities: universal services, universal domains, universal governance, and universal solutions. Informatica delivers on all four capabilities with a truly flexible master data management technology that enables companies to start small by solving their most immediate business problem and scaling to others across the enterprise. Informatica MDM 9.6 advances this universal MDM strategy, delivering such key benefits as:
The Business Intelligence architecture will not be able to produce good results if we cannot link tools to it. This blog provides an overview of the most important Business Intelligence tools (BI Tools) in relation to the decision-making cycle. It also describes the purposes, the features and the workings of these tools.
Each Business Intelligence Tool plays a supporting role in one or more phases of the decision-making cycle. The decision-making cycle comprises of the following phases, which can be supported by the following tools (see figure below):
For a proper Business Intelligence system we need to have some sort of ETL in place. When extracting and filtering data from the source systems the following aspects are important:
Indicators and other types of management information need data from tables in the source system. If this requires one or more attributes, it is highly recommended to copy ALL attributes (from a table) to the SA. Why? Well, firstly it is simpler: instead of having to name each attribute explicitly, the table name will do*. Secondly, we can produce indicators or dimensions – which should be based on data in tables that are already in the SA – faster. After all, there is no need to first adjust the extraction. The disadvantage is that more data need to be processed, which may be a burden on the loading time of the SA.
A pharmaceutical wholesaler wanted to find out more about its market performance compared with the performance of other wholesalers in the industry. The data required to calculate the market shares were only partially available in the wholesaler’s source system, namely the order files.
In order to achieve exchange of sales data – anonymous of course – between wholesalers in the same line of business a negotiation took place at board level. As a result, the wholesalers established a joint foundation with the aim to once a month process all sales data – according to a specific format – into one large file containing all revenue data divided by product level and month level.
More and more organizations are wondering what the use is of a data warehouse, and whether or not it’s worth the investment. They also want to know what alternatives are available. A growing number of IT vendors and some “experts” claim that the end of the data warehouse is nigh. When we say vendors, we’re referring to suppliers of data warehouse appliances, data virtualization tools, and data discovery tools. We have a different opinion, though. The data warehouse still is the beating heart of the Intelligent organization and it serves different vital goals.
The aim of the collection process in the Business Intelligence cycle is to collect, filter, cleanse, combine, transform and aggregate data from different internal and external sources in order to increase the likelihood of good management information and useful knowledge during the process of analysis.
This process is usually supported by a data warehouse, which includes a specific architecture that fits in with both the information needs and the technical infrastructure. The data warehouse must ensure that data is transformed into information and knowledge that encourages action.
Oracle today announced the availability of Oracle Big Data Appliance X3-2 Starter Rack and Oracle Big Data Appliance X3-2 In-Rack Expansion. Oracle Big Data Appliance X3-2 Starter Rack enables customers to jump-start their first Big Data projects with an optimally-sized appliance. Oracle Big Data Appliance X3-2 In-Rack Expansion helps customers easily and cost-effectively scale their footprint as their data grows.
Business Intelligence only works well when we regularly retrieve data from the source systems and copy it to a separate computer and database. This means that the data from the source system are stored redundantly: in the source system and in the data warehouse.
A traditionally minded IT specialist will find this unacceptable: data should – within the company network – never be stored more than once so that when we change data we will not need to do this at several places. The fact that this principle benefits the maintainability of data is beyond dispute. Especially when we need to analyze large volumes of (unstructured) data: Big Data.
The architecture of a Business Intelligence system is guided by a good number of basic principles that specifically apply to Business Intelligence systems. A basic umbrella principle here is the well-known phenomenon of ‘structure follows function’: the indicators that are derived from either the business processes or the strategy as well as additional management information largely determine the architectural structure of the Business Intelligence system. In other words: the content of the system determines its architecture and structure.
On the website of Actuate we found a press release that stated that “Actuate is delivering more insights to more people than all Business Intelligence companies combined”. The company was removed from our 100% vendor independent Business Intelligence Tools survey in December 2011 because no single Actuate official responded to our requests any more.
In a recent survey from one of the Analyst companies the Actuate platform was awarded a high final score. The press release said: “Actuate outranked most major Business Intelligence vendors in final score, exceeding peer and overall scores for most measures”. And they received recognition for the fact that they beat all other major vendors on Big Data Analytics.
The Intelligent Organization observes well. It transforms incoming signals into information and knowledge and responds adequately. This is only possible when the different processes within the Business Intelligence cycle are supported by a good architecture.
The architecture ensures that the processes run smoothly and that they are properly aligned. The Business Intelligence architecture forms the link between the processes we described earlier and the applications and Business Intelligence tools. Without a well thought out architecture, we cannot properly organize the processes of the Intelligent Organization and consequently, we cannot apply information and knowledge in the way we should.
The last months the Passionned Group has given several training sessions in New York, Amsterdam and Singapore about Business Intelligence & Analytics. We like to share a few of the testimonials regarding our Business Intelligence training with you. Most of them are very positive and we are truly thankful for that.
“Enjoyed the interactiveness of the master class, good level for my understanding, we are just getting started”
International supermarket operator Ahold has a very professionally organized Business Intelligence system with an infrastructure that averagely handles seven million information requests from 90 different information sources annually.
Business Intelligence is used throughout the company at all management levels and by almost all humon ‘processors’. A good example: store managers check their state of affairs every single morning. Ahold’s differentiation strategy also leans heavily on Business Intelligence.