This is our blog about designing and implementing Intelligent organizations. Within this area of expertise we write often about the following subjects BI, analytics, the tools, decision management, data visualization, BI success, Business Intelligence concepts, data management, continuous improvement, and the organization of BI & Analytics.
Of course you can order our book here too.
The latest version of Oracle’s data discovery tool Endeca Information Discovery (OEID 1.3) also offers self-service data mash-up and discovery dashboards, extended support for unstructured analytics, and full integration with other Oracle BI software.
With the data mash-up capabilities business users can combine and analyze data from numerous sources; the creation of discovery dashboards is simplified due to an intuitive drag-and drop layout and a configuration wizard.
At the end of 2013, MicroStrategy have launched a new Analytics Platform, which is one of the most important new release for the company in some time. Together with the enterprise platform come two free additional self-service visualization tools.
MicroStrategy does not want to go with the popular flow of what they call ‘quick-hit data discovery solutions’; indeed, whilst these do provide some analytical satisfaction, they can lead into a BI cul-de-sac due to poor scalability, governance, trust, security and lack of native support for mobile apps. Instead, the company sticks to their full service enterprise-grade business intelligence.
From early 2013 open source BI platform Jaspersoft has been fully available for Amazon Web Services. Jaspersoft 5.5 from the cloud has the same utility-based pricing model as the on-premise versions. Within the AWS-platform Jaspersoft automatically discovers en connects to a customer’s Amazon Relational Database Service and Redshift data sources.
New in release 5.5 are enhanced capabilities for data visualization, advanced time series groups, quick links to tutorials and samples, templates for reporting and cluster-aware caching. Instead of Jaspersoft’s former reporting tool iReport designer, Eclipse-based Jaspersoft Studio is included with the BI Platform.
Within the field of Business Intelligence and Analytics, the following developments will gradually take shape and gain momentum:
In recent years, Business Intelligence has proved to be highly technology-driven. It was all about snazzy reporting tools with all sorts of ‘bells and whistles’, performance, technical debates about the most suitable data modeling method and advanced drill-down possibilities. These ‘toys’ diverted our attention from the real purpose of Business Intelligence, namely creating a truly intelligent organization. Fortunately, today we see that Business Intelligence is increasingly driven by the business and that the attention shifts to the behavioural side. Many organisations today prepare a business case in advance in order to assess whether the payback period of data warehouses and Business Intelligence applications is in line with current economic and business standards. Business Intelligence becomes increasingly business-driven and permeates through the organisation more easily. Business Intelligence gradually matures and technology becomes secondary to the processes and applications.
The Intelligent Organization will both develop and maintain a Business Intelligence strategy and will apply the principles of BI governance in order to ensure that the efforts put into Business Intelligence produce lasting results.
BI governance (compass) and BI strategy (map) are mostly about ‘alignment’ between the processes registering, processing and responding, between internal and external information and between business and technology. The map and compass enable us to make better choices when it comes to desired BI projects and their order, which leads – among other things – to a higher return on these projects.
Even though there are many standardized tools, specific solutions will always have a place. New challenges demand tailor-made solutions. A cotton manufacturer wanted to work more efficiently. Critical questions spurred that desire on. The production team wondered: why do we do things the way we do them? Why are we throwing away so much good material? Can we do better?
Most problems only become interesting once the money runs out. Or when the margins shrink, or revenue keeps decreasing. That’s when alarm bells start going off. In many cases, that’s too late to turn the tide. The solutions which were thought up under pressure are no longer executable. If only you’d taken action sooner. If only you’d listened to the data.
Pentaho Business Analytics 5.0 greatly simplifies the entire analytics experience for everyone and delivers the industry’s first just in time big data blending ‘at the source’
Delivering the future of analytics, Pentaho Corporation announces the availability of Pentaho Business Analytics 5.0, a completely redesigned data integration and Business Analytics platform. Pentaho 5.0 provides a full spectrum of analytics for today’s big data-driven businesses regardless of data type and volume, IT architecture or analysis required. The new, modern interface simplifies the user experience for all those working to turn data into competitive advantage. Pentaho 5.0 includes more than 250 new features and improvements. Highlights include:
How to manage Business Intelligence projects? Business Intelligence projects require a specific project approach – the BI project cycle – because they significantly differ from traditional system development projects on a number of aspects.
The BI project cycle starts with awareness: we need to be aware of the nature and character of Business Intelligence. Once this is clear, we start looking for the business cases and we determine the scope. The blueprint of the BI system is created by using the basic principles that govern the BI architecture whilst defining the indicators and dimensions (information analysis) based on which we create a functional design that ultimately leads towards both a balanced architecture and a suitable data model.
Automation is always about people. This is certainly true with regard to ‘automating’ decisions and transforming data into ‘actionable intelligence’.
The point is that people will use this information and start acting differently. In order to create a proper (well-designed) Intelligent organization, we will need people in certain roles, with specific (behavioral) competencies, experience and knowledge, first on a project basis and later in the daily operations. In this article, we describe the ideal Business Intelligence project organization. Note that, in practice, we do not always require all roles or we cannot always facilitate all roles, due for example to budgetary constraints.
Business Intelligence projects are characterized by an extremely high risk factor and many obstacles. These obstacles are mostly related to the fact that Business Intelligence projects typically go beyond the boundaries of departments, processes and even business units; contain a mix of strategy, business operations and technology and are often highly political.
Also, the BI systems are derived from the operational information systems. The obstacles refer to the ten most common and most important forces and risks that can make or break a Business Intelligence project, which are divided into three main categories:
If we do not predetermine what we test and how we test the Business Intelligence system, then this testing process can be very time consuming.
Precisely because Business Intelligence allows flexible reporting, we are initially tempted to test all possible combinations of indicators and dimensions. What we forget is that, even in small Business Intelligence systems with for example ten indicators that are linked to eight dimensions, the number of possible combinations can reach ten million!
Based on the business requirements, the user groups and the architecture, we determine what tools we need in order to collect the data and to create reports and analysis.
We also determine what tools we require for the distribution of data and reports. In addition to the specific Business Intelligence tools mentioned in another post, we will also need a ‘relational database management system’ (RDBMS) for the storage and retrieval of the data.
When we have (too) low expectations of Business Intelligence and we operate in a highly political environment, this may result in a return on investment (of BI) that is far below its potential.
A year ago, a transport company with five thousand employees worldwide, implemented a well thought out and professional Business Intelligence system, which offered the possibility to not only monitor market share figures and revenues, but also to present various financial details and ratios.
It is important to determine whether and how Business Intelligence can be profitable within the organization and in which way we can achieve lasting success. In other words: what is the business case (for BI)?
The benefits of Business Intelligence can be split into four categories (Liautaud and Hammond, 2001):
not immediately apparent;
Does successful Business Intelligence start from the business, the processes and the strategy, or from the information systems and technology that support the business? Should an organization first collect the data (data-driven) or must we first map out our strategy and the associated information needs (business-driven)?
The answer to both questions – as it appears in practice – is that we need to combine the approaches: top-down and bottom-up, business-driven and data-driven. However, Business Intelligence must always begin with (or from) the business . It is, after all, called ‘business intelligence’. BI projects that are not managed by the business are unlikely to succeed, simply because on completion the project results will typically not match with the organization’s ‘company language’ and information needs.
This article discusses the required project approach to the competencies and roles needed to merge the Business Intelligence processes, the architecture, the tools, and the applications into a lasting, working entity. Business Intelligence projects differ from ‘traditional’ system development projects in several ways, namely:
They often go beyond the boundaries of departments, processes and even business units;
A large French retail organization in the food industry that operates as a franchisor was not able to measure the effects of its advertisements correctly. Due to the economic decline, the organization was forced to minimize the marketing costs and to try to maximize the impact of its marketing efforts at the same time.
A major challenge was that different advertisements in the media (TV, online, papers) were mixed up and largely overlapped.
The Intelligent Organization will uses visualization and simulation to ensure that employees can rapidly absorb information and knowledge –which encourages action. After all, a picture is worth a thousand words and “every problem can usually be solved, provided that we use an appropriate model” (Winston, 1992).
Even though people have their own specific preferences when it comes to tables and graphs, a single good quality picture can often communicate a message much faster. A picture may tell us, at a glance, whether our market share increased or decreased in relation to prior periods. Of course you can still use tables (with text), however graphs and charts offer much more variety when it comes to depicting information effectively so that we can quickly interpret that information.
We use dashboards, reports and interactive analysis to enable us to see general and simple relationships within business operations – for example, more customer visits lead to an increase in cross-selling which leads to better financial results – visible. The smaller, more specific and more complex relationships surface when we use data mining.
“Data mining is the uncovering of hidden (unknown) relationships or segments in large data collections that have a predictive value for a specific part of the business operations”.
On the path towards maturity in the field of metadata, an organization can go through four levels of ambition. These are shown in the figure below.
Figure: The four levels of ambition of metadata
Level 0: an organization is on this level if there is hardly any need for metadata, for example because all applications store data in one and the same database from which they also retrieve the data. The applications are more or less isolated, which means that hardly any data is exchanged between them. The organization is structured as such and managers primarily focus on their own departments.
A marketing employee of an importer of well-known car brands notices on her dashboard that the car sales in the Northern region are much lower than budgeted.
She switches from the dashboard to the interactive analysis mode and performs analysis per postal code region and per household type. The analysis shows that lower sales mainly occur in two-person households in rural areas. She consults (via chat) with the responsible dealer in the region to verify that the dealer knows about the drop in sales and whether he knows what the reasons might be.
From the outside a portal looks a bit like a stained glass window. The little windows within it – the so-called portlets – represent the different parts of the portal.
Portals offer direct, centralized access to relevant information and business applications through intranets or secured extranets (pull). Portals also ensure that the relevant information – or a reference to it – is sent to users (push).
In 2011, a British pension fund with assets of around 256 billion Euros realised a state-of-the-art data warehouse with an emphasis on the performance of different funds and portfolios.
All kinds of reports and analysis were used to achieve the highest possible return at an acceptable risk. For investment in shares, the front office – the people who commission stock market trading – used different operational systems that were linked to the systems of the brokers at the stock exchange. The performance was monitored daily and where necessary assets (shares, derivatives, bonds) were acquired or disposed of, via the operational systems.
A Data Vault is a modeling technique for the CDW, designed by Dan Linstedt, which chooses to store all incoming transactions regardless of whether the details are in fact trustworthy and correct: “100% of the data 100% of the time”.
For example: a sales transaction has already taken place, but the corresponding customer does not yet exist in the CRM system. The sales transaction can nonetheless be stored in the CRM system. When the customer becomes known to the system, the transaction changes from a ‘meaningless’ fact into a useful ‘truth’ because now its context is known.
Interactive analysis and OLAP is often regarded as the most important innovation in the world of Business Intelligence.
The eyes of analysts and managers often begin to sparkle when they see the possibilities of interactive analysis and OLAP for the first time: “This is what we always wanted to have!” Their enthusiasm is not without reason: this powerful tool enables us to quickly and casually ‘browse’ through large volumes of data, regardless of whether we search for specific information. This allows us to quickly approach data with a varying level of detail from different angles.
The business unit of one the three largest construction companies in Europe, which is responsible for the construction and maintenance of the railways, wishes to improve its information supply. The internal organization creates a very large amount of management reports and analysis using the popular Microsoft Office tools Access and Excel.
The set of queries and reports kept growing and growing to a respectable total of one thousand. After a few years working this way, a sudden major problem occurred for the Board of Directors to deal with.
We use reporting – just as dashboards – in order to display the known key success factors and the associated indicators. Both already prepared in the data warehouse. In this way, we can signal potential problems in business operations at an early stage.
Reports typically provide detailed information. This is why they use both data from data marts (or cubes) and detailed data from the ‘Central Data warehouse’. Reports generally display information in charts or tables, although alternative visualizations are also possible. An example of a report is shown below.
Interactive analysis or reporting? A well-executed – business-driven – information analysis ensures that many potential decision problems are already on the ‘agenda’ and that information and goals demonstrate coherence.
The indicators that are crucial for the realization of our strategy, for accomplishing our goals and for achieving our mission, are placed on the dashboard or in the reports. Interactive analysis powered by OLAP – playfully analyzing enterprise data – allows us to identify additional indicators and key success factors.
Informatica Corporation, the world’s number one independent provider of data integration software, today announced Informatica MDM 9.6, a major milestone in Informatica’s roadmap for powering universal master data management (MDM). The announcement came at Informatica’s annual user conference – Informatica World 2013.
Universal MDM depends on four essential capabilities: universal services, universal domains, universal governance, and universal solutions. Informatica delivers on all four capabilities with a truly flexible master data management technology that enables companies to start small by solving their most immediate business problem and scaling to others across the enterprise. Informatica MDM 9.6 advances this universal MDM strategy, delivering such key benefits as:
The Business Intelligence architecture will not be able to produce good results if we cannot link tools to it. This blog provides an overview of the most important Business Intelligence tools (BI Tools) in relation to the decision-making cycle. It also describes the purposes, the features and the workings of these tools.
Each Business Intelligence Tool plays a supporting role in one or more phases of the decision-making cycle. The decision-making cycle comprises of the following phases, which can be supported by the following tools (see figure below):
For a proper Business Intelligence system we need to have some sort of ETL in place. When extracting and filtering data from the source systems the following aspects are important:
Indicators and other types of management information need data from tables in the source system. If this requires one or more attributes, it is highly recommended to copy ALL attributes (from a table) to the SA. Why? Well, firstly it is simpler: instead of having to name each attribute explicitly, the table name will do*. Secondly, we can produce indicators or dimensions – which should be based on data in tables that are already in the SA – faster. After all, there is no need to first adjust the extraction. The disadvantage is that more data need to be processed, which may be a burden on the loading time of the SA.
A pharmaceutical wholesaler wanted to find out more about its market performance compared with the performance of other wholesalers in the industry. The data required to calculate the market shares were only partially available in the wholesaler’s source system, namely the order files.
In order to achieve exchange of sales data – anonymous of course – between wholesalers in the same line of business a negotiation took place at board level. As a result, the wholesalers established a joint foundation with the aim to once a month process all sales data – according to a specific format – into one large file containing all revenue data divided by product level and month level.
More and more organizations are wondering what the use is of a data warehouse, and whether or not it’s worth the investment. They also want to know what alternatives are available. A growing number of IT vendors and some “experts” claim that the end of the data warehouse is nigh. When we say vendors, we’re referring to suppliers of data warehouse appliances, data virtualization tools, and data discovery tools. We have a different opinion, though. The data warehouse still is the beating heart of the Intelligent organization and it serves different vital goals.
The aim of the collection process in the Business Intelligence cycle is to collect, filter, cleanse, combine, transform and aggregate data from different internal and external sources in order to increase the likelihood of good management information and useful knowledge during the process of analysis.
This process is usually supported by a data warehouse, which includes a specific architecture that fits in with both the information needs and the technical infrastructure. The data warehouse must ensure that data is transformed into information and knowledge that encourages action.
Oracle today announced the availability of Oracle Big Data Appliance X3-2 Starter Rack and Oracle Big Data Appliance X3-2 In-Rack Expansion. Oracle Big Data Appliance X3-2 Starter Rack enables customers to jump-start their first Big Data projects with an optimally-sized appliance. Oracle Big Data Appliance X3-2 In-Rack Expansion helps customers easily and cost-effectively scale their footprint as their data grows.