The term business intelligence was probably first used in a 1958 research paper from IBM researcher Hans Peter Luhn. He defined intelligence as "the ability to apprehend the interrelationships
In 1989, Howard Dresner proposed business intelligence – or BI – as an umbrella term used to describe "concepts and methods to improve business decision-making by using fact-based support systems."
In short, business intelligence is about taking all of the data in your organization and presenting it in a way that connects various facts to one another. You drill through those facts, constrained by their real-world relationships, so that you can make smarter decisions about how to manage your company. There are really two main goals behind modern BI:
- Put as many facts as possible in front of decision-makers, but organize and connect those facts so that their relationships are obvious and can be explored.
- Make these facts available to as many people as possible, since ultimately every employee in a business makes decisions that affect the organization.
Let me offer an example.
Suppose you work for a company that sells golf clubs. A sales manager notices that revenue on the golf clubs is doing really well, which means the company is selling them like hotcakes. So he decides to expand the line, offer more golf club models, and stock more units in each store. Remember that he's based this decision on two things: the fact that clubs are selling well, and his own instincts and experience telling him that they should do more of it.
Within a few weeks, however, his boss is yelling at him because profit margins have dropped on the golf clubs, and the company is considering dropping them entirely. Aghast, the manager wants to know why -- but nobody can put their finger on precisely the right answer.
Now suppose a business intelligence system was in place, and it had access to all of the company's various bits of data. It would be easier to see that the shipping prices for the golf clubs increased when the number of units in each shipment went up. Turns out an extra club or two in the box put the total box weight over some threshold for the shipping carrier, bumping the costs exponentially. That's not information the sales manager ever had access to – which is why one of the goals of a BI system is to get all the facts in front of everyone, or at least as many people as possible. With better business intelligence, the manager could potentially have done a "what if" test, adding more units to each store shipment and letting the BI system show him the outcome of lowered margins.
So business intelligence is about basing business decisions on solid facts rather than instincts or guesswork, and on getting all the facts in play before decisions are made.
Data warehouses, data marts, and in-memory analytics
In order for a business intelligence system to work, it has to have rapid access to a lot of data from across a company. Typical transactional databases seek to balance both read and write performance, as well as reduce data redundancy. Combining data from dozens of these databases can be extremely time-consuming, meaning users might have to wait hours or even days for a particular report to be produced. That's not fast enough for decision-making.
Business intelligence addresses this issue through data warehousing. Essentially, a data warehouse is a really big database with a structure designed solely to speed up read operations. That structure is custom-built to contain the data you put into it.
On a regular basis (let's say nightly), automated tasks run that copy data from your production data sources into the data warehouse, restructuring the data in the process so that it can be queried very, very fast. Reports which might have taken hours now take minutes or even seconds. The entire business intelligence system is built on top of this specialized data structure.
A data mart is basically just a mini data warehouse that contains the data related to a particular portion of your company, such as the Finance department or Retail division. If you create a data mart for each division, and then link those together, you have (in theory, anyway) a data warehouse.
There are two major downsides to data warehouses:
- It can take a long time to populate them from production data sources, meaning the data warehouse is always somewhat out of date.
- Data warehouses can be huge. To properly show trends and patterns, they need to keep a lot of old data hanging around, and their structure encourages redundant and repetitious data. That's faster for querying, but takes up a ton of disk space.
With the ready availability of ridiculously fast processors and the rapidly-falling prices of server memory, a technique called in-memory analytics has become more feasible. With this approach, you skip the data warehouse and just read data straight from production data sources. An analytics server rearranges the data into a data warehouse-like structure in memory, uses that in-memory structure to support whatever analysis you're doing at the time, and then removes the structure from memory when you're done.
The goal is to provide faster, on-demand analysis capabilities that might need a structure different from that of your on-disk data warehouse. You do need fast servers and lots of memory to support this activity, but it can also offer more dynamic, ad-hoc analysis capabilities. Most modern business intelligence solutions incorporate data warehouses as well as in-memory analytics.
ABOUT THE AUTHOR
Don Jones a co-founder of Concentrated Technology LLC, the author of more than 30 IT books and a speaker at technical conferences worldwide. Contact him through his website at www.ConcentratedTech.com.
This was first published in March 2010