Big Data: new business models, faster learning & acting
Big Data is a huge part of our society. All kinds of technologies are generating data at all times: social media, sensors in flight engines, smart pills, trucks, running shoes, pumps, etc. We call this Big Data, because we’re talking about massive volumes of (unstructured) files. This tidal wave of data is flooding the world. You get the feeling your organization can and should work with this. You might wonder the following:
- What should my organization do with Big Data? What does a successful implementation look like?
- What new business models does Big Data enable?
- Where and how should I save the data? What tools are available for Big Data?
- Which skills do my professionals need?
- How should my organization handle any possible privacy concerns?
Big Data can contain massive value for every organization, but it’s difficult to analyze and apply. Why? Because the data is fleeting, complex, huge in volume, and unstructured. The Big Data consultants of Passionned Group can assist you in obtaining clear answers. Of course, we’re also eager to help your organization implement Big Data successfully.
The Dublin Case: optimizing the flow of traffic
The Dublin city council researched the possibilities of Big Data to see if it would be possible to optimize the flow of traffic in the city and reduce the amount of congestion. They planted a large amount of sensors in the roads and GPS trackers in buses. They also put rain detectors in key places in the city. Finally, they siphoned data from all the city’s cameras.
Now that the system is live, they gather all the data in real-time and save it in a data lake. They visualize the data on a map of the city. This allows them to see exactly where the traffic jams are or are threatening to appear. Using camera feeds, the employees of the traffic control center can spot the cause of the congestion right away, and they can take appropriate actions immediately.
In the case of a serious accident they can immediately notify the police and ambulances, even before bystanders can. In other cases they can send traffic controllers to the scene to direct traffic. They also use Big Data to determine the optimal routes of buses.
First, think of an application
The Dublin Case clearly illustrates that they thought of a clear application before starting – the most important step to take before working with Big Data. Which better or faster decisions do you want to make based on the data? Too often, the focus with Big Data is on the data itself, and not what results it should lead to or what new business models it can enable. Without a clear goal, the data will not yield any significant results, and the Big Data machine will stall.
What are typical Big Data sources?
Big Data can be extracted from many possible data sources. Some examples:
- Social media like Twitter and FaceBook.
- Sensors in the human body or other organisms.
- Sensors beneath the surface of the earth measuring, for example, seismic activity.
- Sensors in space or sensors measuring events in space from Earth.
- RFID tag sensors measuring product movements.
- Logs containing the surfing behavior of your website’s visitors.
- Sensors in machines, clothes, or devices measuring, for example, the condition of the device or the temperature.
Often, you need special adapters (API’s) to be able to extract the data and do a decent data integration job.
Big Data and the five Vs
- Volume: The volume of data is so great that the data won’t fit in a traditional SQL database. Storage takes place in file systems or so-called NoSQL databases. Or extracts are saved in a data warehouse.
- Velocity: The data emerges quickly and can disappear even faster. Twitter, for example, archives older tweets. The data evaporates quickly, so you have to act fast.
- Variety: The data has great variety in structure, volume, and meaning.
- Veracity: Variable data quality and reliability makes using the data uncertain.
- Value: This is what it’s really about: which value is Big Data going to generate for your customers and your organization?
From traditional Business Intelligence to Big Data
Traditionally, Business Intelligence works with structured data that’s relatively easy to store and retrieve. Based on this data you can make cubes or dashboards. With Big Data, BI is about processing (large volumes of) unstructured data. How can you process this effectively? And what else do you need to think of?
A Hadoop computer cluster has massive processing power
Big Data versus Zero Data
Look past your own data
What Big Data ‘solutions’ are available?
This is the most difficult question to answer, because it depends on a variety of factors. It’s difficult to talk about solutions when it comes to Big Data, because the specific challenge can be different for each industry or company. It may depend on what type of application you want to build and from which source(s) the data needs to be extracted. However, there are dozens of technologies which can be helpful in exploiting Big Data. These technologies can be classified as follows:
- Software for data extraction like the Twitter API. Often these adapters are available as an add-on for ETL software;
- Software for data storage like the Hadoop Distributed File System (HDFS), but there are many alternatives;
- Techniques for data analysis & classification like machine learning, natural language processing, neural networks, pattern recognition, predictive modeling;
- Software for data analysis like MapReduce (part of Hadoop), for which there are many alternatives;
- Software for data visualization like Tableau Software and IBM OpenDX;
- Hardware (appliances) for parallel processing or Cloud computing platforms.
Depending on your type of problem you may need a specific mix of the above technologies.
A few Business Intelligence software solutions are able to read directly from social media API’s. HDFS and others use specific software for data analysis beneath the surface of their own software. Download the Business Intelligence Tools Survey for more information.
Volume makes Big Data very interesting
What if there are some gold nuggets hidden inside your pile of Big Data? These might, for example, let you know that the price of a natural resource is going to go up a month before your competitor does. Or maybe the sensor data from a plane engine shows that it glitches during a flight, at a certain height and under certain weather conditions. A malfunctioning engine usually spells disaster. It’s these kinds of critical applications that make Big Data hugely interesting.
Or consider the analysis of millions of camera feeds of patients in a psychiatric clinic. This might allow you to quickly build a model to detect deviant behavior in a patient. Noticing these patterns could inform you when a patient might experience a psychotic episode, with harm or property damage as a result. By detecting such behavior early you can perform extra check-ups. That’s why organizations want to decipher the mountain of data: to discover opportunities and manage risks. We would like to help you start working proactively instead of reactively. Read our article ‘Why every controller should know everything about Big Data.’
Six hallmarks of successful Big Data tracks
A successful Big Data track is characterized by an open, learning analytical company culture. And, of course, involvement and budget possibilities from management’s side. In addition, both the business people and the data scientist need creativity and a lot of industry knowledge. Also important:
- Synchronizing with organizational goals and mission: The goals of Big Data should match the strategic company vision, so that you can realize your goals. Building a data lake without a clear goal is pointless.
- Involve your users: User participation, and especially awareness among users about what Big Data can mean for their work, is of massive importance to the success of a Big Data track. An agile and scrum approach can help you realize this participation.
- Source and data quality: The data’s quality is of even greater importance in Big Data projects than for regular Business Intelligence programs. After all, Big Data leads to automated decision-making.
- Usability and ease of use: Usability, accessibility, and ease of use should be high.
- Solid data infrastructure: The quality and flexibility of the data infrastructure should also be high. The system should be robust and scalable.
- A well-considered team composition: The team should have plenty of data science expertise so you can match your business, IT, and BI competencies. That will allow you to play into the various information needs more quickly.
How come it can still go wrong? The answer is simple: it’s not easy to manage all of the above. They influence each other and demand a steady hand with a lot of expertise and experience in Big Data Analytics.
The Big Data & Data Science Quick Scan
How mature is your Big Data approach?
Our Big Data Quick Scan gives you a clear impression of your maturity level in Big Data and which steps you can take to improve the value of your data. Aside from content and technical aspects, we also look carefully at processes and organizational embedding. Of course, we won’t forget about your organization’s strategic course. Only then can your Big Data provide a strategic advantage, as well as a tactical and operational one.
Success stories about Big Data Analytics
Success stories about Big Data & Analytics are cropping up at record speeds. These stories are no longer going unnoticed in mainstream media either. The Amsterdam fire department already made the BBC news by using Big Data to prevent fires. The Amsterdam police department won a podium position in “The smartest organization in The Netherlands” by catching criminals before they commit a crime. Dublin’s traffic optimization using Big Data is a guiding light for all public institutions. Now they have a better understanding of how you can improve your services for citizens. In short: these stories are proof positive that Big Data and Predictive Analytics determine the difference between smart and dumb organizations. Between winners and losers.
Do you want to become a smart, date-driven organization?
Feel free to contact us for an orientation with one of our Big Data specialists. We’d love to help you make your organization work smarter and turn your Big Data into dollars.