• Historical notes on analytics — the pre-computer era
• Historical notes on analytic terminology (in which many terms used in this post are defined)
• Historical notes on analytics — departmental adoption (this post)
  

What set off my “history of analytics” posting kick is, simply put:

• Most interesting analytic software has been adopted first and foremost at the departmental level.
• People seem to be forgetting that fact.

In particular, I would argue that the following analytic technologies started and prospered largely through departmental adoption:

• Fourth-generation languages (the analytically-focused ones, which in fact started out being consumed on a remote/time-sharing basis)
• Electronic spreadsheets
• 1990s-era business intelligence
• Dashboards
• Fancy-visualization business intelligence
• Planning/budgeting
• Predictive analytics
• Text analytics
• Rules engines

If we leave out data management/system technologies* — e.g. data warehouse appliances or Hadoop — that’s pretty much everything that succeeded (and a couple that perhaps didn’t). I don’t know what to put on an “Enterprise-wide from the get-go” list except for a couple of duds like executive information systems and balanced scorecards.

*”System software” technologies such as DBMS often do eventually fall under the purview of central IT. But even for them there’s typically a multi-year period during which departments take the initiative in bringing them in.

Of course, this should surprise nobody; information technology is almost always adopted departmentally first, with the exceptions arising mainly in cases where departmental adoption makes no sense. Reasons include:

• The problem being solved is department-specific.
• The expertise and specific techniques to solve the problem are (or seem) subject/department-specific.
• The budget to solve the problem is department-specific.
• The best reasons to centralize technology often involve integration among departments, and new technology is rarely expected to start out being all that integrated.

Departments most likely to be early adopters (relative to others) of analytic technology seem to be:

• Finance/planning, especially in the old days when analytic technology was newer (nowadays finance might be more involved in trying to push reporting/analysis discipline out to a whole company).
• Sales/marketing, because they often have more data than other departments (actual purchase transactions, other customer contacts, and also a lot of external data).
• Investment research, because financial analysis is almost literally their core product. (Ditto trading, for very similar reasons.)

Three examples, for me, serve to bring all this home.

Business PC use famously started with individuals and departments just acquiring PCs, outside of the IT department’s control or even knowledge, way back in the day of the Apple II. Most commonly, the reason to get the PC was to run an electronic spreadsheet, generally VisiCalc.

10-15 years ago, when business intelligence vendors banged the drum for enterprise-wide BI/dashboard adoption, I’d ask them “So, do you have an enterprise-wide dashboard yourselves?” Invariably, they didn’t — but they did have departmental dashboards for sales and/or marketing. It became clear that this was a general pattern in BI adoption.

Multiple generations of technologies that one might think of as having to do with artificial intelligence – e.g. expert systems, predictive analytics* and text analytics — have wound up with applications being concentrated in the same few areas:

• Marketing
• Quality/maintenance
• Scientific/engineering research
• National security/law enforcement/anti-fraud
• Underwriting/investing/risk assessment
• Publishing (for the text-oriented technologies)

Those categories comprise 90%+ of the applications I can think of for the golly-gee-whiz technologies of their day. (You could add simulation to the list as well.) And outside of the publishing and criminal-catching sectors, those apps are pretty departmental in nature.

*I think predictive analytics has evolved into a blend of statistics and (other) machine learning, and machine learning can be viewed as a kind of AI.

So why do I think you should care about all this? Two reasons:

• History is cool.
• It has relevance to current issues in analytic technology adoption, which I plan to write more about soon.

I hope you agree.

• Historical notes on analytics — the pre-computer era
• Historical notes on analytic terminology (this post)
  
• Historical notes on analytics — departmental adoption

Discussions of the history of analytic technology are complicated by the broad variety of product category names that have been used over the decades. So let me collect here in one place some notes on how (and when) various terms have been used, specifically:

• Management information systems
• Decision support (systems)
• Report writer
• Fourth-generation language
• Executive information system
• Business intelligence
• OLAP (OnLine Analytic Processing)

Obviously, I can’t cover everything in this post. Omissions include but are not limited to:

• Anything in the data warehouse/data mart area. (For one thing, I don’t want to deal with the whole Inmon/Kimball dispute.)
• Anything in the predictive analytics area (but see the first point in a 2009 SPSS post).
• Terms I’ve recently sponsored, such as investigative analytics or machine-generated data.
• Big data (analytics) — I just discussed that mess a week ago.

The first prevalent term I recall for “information technology” was management information systems (MIS). I mention that mainly to note that it actually sounded a bit analytics-oriented, and hence to point out that in the old days — 1960s and so on — it didn’t seem necessary to name a separate category that amounted to “analytics”.

Meanwhile, the first prevalent term I recall that covered much of what we’d now call “analytics” was decision support, or decision support systems (DSS). I think DSS was always ill-defined, with multiple subcategories, just as analytics is today. The heyday of this term was in the 1970s/1980s.

Report writers were around in various forms for decades; consider for example the early 1970s history of Cullinane/Cullinet. By the time I became an analyst in the early 1980s, these were mainframe tools that let you specify paper reports, and the market leader were probably Pansophic’s EASYTRIEVE and Informatics’ Mark IV. According to marketing, they could be used by non-programmers; in reality, they were a much easier way for programmers to do what end users asked. They were used both for one-shot queries and, as their main design point, repetitive reporting.

The report writer category then survived into the era of business intelligence (BI). (Indeed, Cognos’ big integrated BI tool early this century was called ReportNet.) More on that in the BI discussion below.

The term fourth-generation language (4GL) was widely used from the 1970s through the first part of the 1990s. Usually, a 4GL was:

• A vendor-specific programming language …
• … sold in connection with an interpreter …
• … that was particularly good at database manipulation.

In particular, the original 4GLs:

• Were primarily sold and used for analytics.
• Were often sold/used on a remote/timeshared basis.
• Often had some kind of (pre-relational) DBMS bundled in.

Classic examples of such 4GLs included FOCUS (the core product of Information Builders), RAMIS, and NOMAD; SAS arguably started out as a product of that kind too. Starting in the 1980s, however, 4GLs were used more generally, and indeed survived as an OLTP (OnLine Transaction Processing) technology long after they were supplanted by BI tools for most analytic purposes.

The last pre-BI term I want to mention is executive information system (EIS). EIS was in essence the 1980s term for “dashboard”, although the technology was much more primitive than it is today.

The term business intelligence was coined in the 1950s and then reinvented in the 1980s; however, it has described a major category only from the 1990s onward, specifically starting when GUIs (Graphical User Interfaces) became prevalent.. “Business intelligence” is sometimes used to comprise all of analytics; more commonly, however, it refers to tools focused on data selection and presentation.

These days, most of what we’d call BI comes in a single integrated package, focused on a dashboard; most of the exceptions are somewhat old-fashioned report writers. In the 1990s and early 2000s, however, business intelligence had several distinct subcategories …

… one of which was called OLAP (OnLine Analytic Processing). Actually,  the term “OLAP” has been confusingly been used to mean several different things, including:

• Pretty much all of analytics.
• A particular non-relational DBMS architecture that I prefer to refer to as MOLAP (Multidimensional OLAP).
• An integrated suite of DBMS, 4GL, and perhaps other tools around a MOLAP architecture. (Examples: The IRI Express and Arbor/Hyperion Essbase products Oracle bought.)
• A client-side BI tool with a little MOLAP DBMS built in. (Example: Cognos’ erstwhile flagship BI product PowerPlay.)

I hate the term “OLAP” with a passion, in part due to that confusion, and in part due to the specific way the confusion came about: Ted Codd introduced the term, allegedly objectively, but actually as a marketing shill for Arbor Software, which had an obvious business incentive to pretend that its specific technologies solved a broader class of problems than they actually did.

OK. With that too cleared away, I feel ready to write about the actual history of analytic technology.

• Historical notes on analytics — the pre-computer era (this post)
  
• Historical notes on analytic terminology
• Historical notes on analytics — departmental adoption

Sometimes, what people describe as being “New, new, new!!!” in analytics has actually been happening since before they were born, or even before their parents were. Occasionally, I point this out. I think it’s time to collect some of those observations into a short series of posts.

Before getting to the history of actual analytic software, I can’t resist racing through some really old stuff. In a 2004 white paper, I wrote:

Transactional business processes have been around literally since the beginning of recorded history. Some of the oldest known writings are clay tablets that record merchants’ tallies in Sumerian cuneiform, complete with seals to enforce transaction integrity. Analytic business processes date back nearly as long, especially in military applications; the first chapter of Sun Tzu’s The Art of War is called “Calculations,” or in some translations “Laying Plans.”*

As enterprise complexity increased, so did the sophistication of analytic business processes. Almost two centuries ago, Nathan Rothschild made an investment fortune from early news about the Battle of Waterloo, and several decades later Florence Nightingale** introduced statistics to the study of public health. With the invention of machines to tabulate information in the late 19th Century, analysis began to blossom.

*Back when I wrote that, I also considered including some of the accounting Caesar cited in his Commentaries on the Gallic Wars, but eventually decided that was a “production application” rather than anything we’d recognize as “analytics”.

**Florence Nightingale is, simply put, one of the more impressive women in history. Unfortunately, a couple of other statistical greats were associated with the deplorable subject of eugenics. I am thinking specifically of Francis Galton, who invented several really basic statistical concepts, and seems to generally have been a brilliant guy, and Karl Pearson. (Hat-tip to John Verostek for, um, tipping me off to Galton.)

Statistics also seems to have led the way in business applications of analytics. Specifically, statistical quality control dates back to the 1920s or so, pioneered by Walter Shewart and given greater visibility by his protégé W. Edwards Deming. On the monitoring side, various organizations collected industry-wide numbers by the 1930s or so. For example, movie box office receipts were reported at least as far back as 1939, perhaps by Variety; the New York Times bestseller list seems to have started slightly later, in 1942; and a rather superficial site on media history first gives hard numbers for the decade of the 1930s.

Still, to continue quoting the same white paper:

What utterly transformed both transactional and analytic business processes was the advent of electronic computing in the 20th Century. In particular, the volume of available data exploded. Even more important to analytic processes was the superhuman increase in the speed of computation. Various types of software emerged — business intelligence (BI) tools, spreadsheets, statistical packages, and the like – permitting kinds of analysis that had been infeasible without computers.

And with that out of the way, let’s return to discussing computer-era analytics.

• The brochure quote basically pitches business intelligence/ performance management, with unconstrained drilldown.
• Pitching BI/analytics benefits for what start out as being transactional applications has been going on for pretty much the whole history of the applications industry. This just happens to be a great proof point.
• The unconstrained drilldown part could almost be taken for granted today, in the relational era. In 1972, however, it was a rather bold (and for all I know exaggerated) claim.

The exact verbiage is: 

1.  The system should focus management attention on the critical variables in distribution profitability, providing measures of the productivity of all key resources of the firm.

2.  Where deviations from expected performance levels are detected, they should be highlighted on an exception basis, so that managers can allocate their most critical resource — time — to the problems or opportunities that variances in performance represent.

3. The manager should be able to pursue the analysis of an exception to whatever level of detailed information is needed to take positive action; and the system should, therefore, permit the manager to secure information at any desired level of detail without prior knowledge of his information needs.

Comshare — a BI vendor before anybody called it BI — did actually build a DBMS based on Childs’ ideas, as Ron Jeffries reminds us. It was relational. Eventually, if I recall correctly, it was swapped out for Essbase (the original MOLAP product, now owned by Oracle).

What Childs really focuses on, however, seems to be “Extended Set Theory.” (This was brought to my attention by Algebraix, even though Algebraix doesn’t actually use many of Childs’ ideas.) And he’s been doing it for a long time. Way back in 1968, Childs wrote a paper outlining how set theory, relations, and tuples could be applied to data management.

And that’s where I did a double-take, because 1968 < 1970. Sure enough, Footnote #1 in Codd’s seminal paper is to Childs’ 1968 work. Indeed, Childs’ paper is the only predecessor Codd acknowledges as having significant portions of his idea.

I’m far from convinced that “Extended set theory” has much to offer versus the standard relational model. But that debate quite aside — Childs’ original achievement doesn’t get the credit it deserves.

Either way, it’s been established that Howard and his colleagues were several decades late; the term was first coined no later than the late 1950s. Whether anybody much used it in the interim is, of course, quite a different matter: I recall terms like decision support and executive information systems (EIS), but not “business intelligence” before the time frame in which Howard claims to have (re)introduced it.

By the way — that “Monash BI” link is NOT to anything I wrote. It’s something associated with Monash University, on the other side of the planet.