• Historical notes on analytics — the pre-computer era
• Historical notes on analytic terminology (in which many terms used in this post are defined)
• Historical notes on analytics — departmental adoption (this post)
  

What set off my “history of analytics” posting kick is, simply put:

• Most interesting analytic software has been adopted first and foremost at the departmental level.
• People seem to be forgetting that fact.

In particular, I would argue that the following analytic technologies started and prospered largely through departmental adoption:

• Fourth-generation languages (the analytically-focused ones, which in fact started out being consumed on a remote/time-sharing basis)
• Electronic spreadsheets
• 1990s-era business intelligence
• Dashboards
• Fancy-visualization business intelligence
• Planning/budgeting
• Predictive analytics
• Text analytics
• Rules engines

If we leave out data management/system technologies* — e.g. data warehouse appliances or Hadoop — that’s pretty much everything that succeeded (and a couple that perhaps didn’t). I don’t know what to put on an “Enterprise-wide from the get-go” list except for a couple of duds like executive information systems and balanced scorecards.

*”System software” technologies such as DBMS often do eventually fall under the purview of central IT. But even for them there’s typically a multi-year period during which departments take the initiative in bringing them in.

Of course, this should surprise nobody; information technology is almost always adopted departmentally first, with the exceptions arising mainly in cases where departmental adoption makes no sense. Reasons include:

• The problem being solved is department-specific.
• The expertise and specific techniques to solve the problem are (or seem) subject/department-specific.
• The budget to solve the problem is department-specific.
• The best reasons to centralize technology often involve integration among departments, and new technology is rarely expected to start out being all that integrated.

Departments most likely to be early adopters (relative to others) of analytic technology seem to be:

• Finance/planning, especially in the old days when analytic technology was newer (nowadays finance might be more involved in trying to push reporting/analysis discipline out to a whole company).
• Sales/marketing, because they often have more data than other departments (actual purchase transactions, other customer contacts, and also a lot of external data).
• Investment research, because financial analysis is almost literally their core product. (Ditto trading, for very similar reasons.)

Three examples, for me, serve to bring all this home.

Business PC use famously started with individuals and departments just acquiring PCs, outside of the IT department’s control or even knowledge, way back in the day of the Apple II. Most commonly, the reason to get the PC was to run an electronic spreadsheet, generally VisiCalc.

10-15 years ago, when business intelligence vendors banged the drum for enterprise-wide BI/dashboard adoption, I’d ask them “So, do you have an enterprise-wide dashboard yourselves?” Invariably, they didn’t — but they did have departmental dashboards for sales and/or marketing. It became clear that this was a general pattern in BI adoption.

Multiple generations of technologies that one might think of as having to do with artificial intelligence – e.g. expert systems, predictive analytics* and text analytics — have wound up with applications being concentrated in the same few areas:

• Marketing
• Quality/maintenance
• Scientific/engineering research
• National security/law enforcement/anti-fraud
• Underwriting/investing/risk assessment
• Publishing (for the text-oriented technologies)

Those categories comprise 90%+ of the applications I can think of for the golly-gee-whiz technologies of their day. (You could add simulation to the list as well.) And outside of the publishing and criminal-catching sectors, those apps are pretty departmental in nature.

*I think predictive analytics has evolved into a blend of statistics and (other) machine learning, and machine learning can be viewed as a kind of AI.

So why do I think you should care about all this? Two reasons:

• History is cool.
• It has relevance to current issues in analytic technology adoption, which I plan to write more about soon.

I hope you agree.

• Historical notes on analytics — the pre-computer era
• Historical notes on analytic terminology (this post)
  
• Historical notes on analytics — departmental adoption

Discussions of the history of analytic technology are complicated by the broad variety of product category names that have been used over the decades. So let me collect here in one place some notes on how (and when) various terms have been used, specifically:

• Management information systems
• Decision support (systems)
• Report writer
• Fourth-generation language
• Executive information system
• Business intelligence
• OLAP (OnLine Analytic Processing)

Obviously, I can’t cover everything in this post. Omissions include but are not limited to:

• Anything in the data warehouse/data mart area. (For one thing, I don’t want to deal with the whole Inmon/Kimball dispute.)
• Anything in the predictive analytics area (but see the first point in a 2009 SPSS post).
• Terms I’ve recently sponsored, such as investigative analytics or machine-generated data.
• Big data (analytics) — I just discussed that mess a week ago.

The first prevalent term I recall for “information technology” was management information systems (MIS). I mention that mainly to note that it actually sounded a bit analytics-oriented, and hence to point out that in the old days — 1960s and so on — it didn’t seem necessary to name a separate category that amounted to “analytics”.

Meanwhile, the first prevalent term I recall that covered much of what we’d now call “analytics” was decision support, or decision support systems (DSS). I think DSS was always ill-defined, with multiple subcategories, just as analytics is today. The heyday of this term was in the 1970s/1980s.

Report writers were around in various forms for decades; consider for example the early 1970s history of Cullinane/Cullinet. By the time I became an analyst in the early 1980s, these were mainframe tools that let you specify paper reports, and the market leader were probably Pansophic’s EASYTRIEVE and Informatics’ Mark IV. According to marketing, they could be used by non-programmers; in reality, they were a much easier way for programmers to do what end users asked. They were used both for one-shot queries and, as their main design point, repetitive reporting.

The report writer category then survived into the era of business intelligence (BI). (Indeed, Cognos’ big integrated BI tool early this century was called ReportNet.) More on that in the BI discussion below.

The term fourth-generation language (4GL) was widely used from the 1970s through the first part of the 1990s. Usually, a 4GL was:

• A vendor-specific programming language …
• … sold in connection with an interpreter …
• … that was particularly good at database manipulation.

In particular, the original 4GLs:

• Were primarily sold and used for analytics.
• Were often sold/used on a remote/timeshared basis.
• Often had some kind of (pre-relational) DBMS bundled in.

Classic examples of such 4GLs included FOCUS (the core product of Information Builders), RAMIS, and NOMAD; SAS arguably started out as a product of that kind too. Starting in the 1980s, however, 4GLs were used more generally, and indeed survived as an OLTP (OnLine Transaction Processing) technology long after they were supplanted by BI tools for most analytic purposes.

The last pre-BI term I want to mention is executive information system (EIS). EIS was in essence the 1980s term for “dashboard”, although the technology was much more primitive than it is today.

The term business intelligence was coined in the 1950s and then reinvented in the 1980s; however, it has described a major category only from the 1990s onward, specifically starting when GUIs (Graphical User Interfaces) became prevalent.. “Business intelligence” is sometimes used to comprise all of analytics; more commonly, however, it refers to tools focused on data selection and presentation.

These days, most of what we’d call BI comes in a single integrated package, focused on a dashboard; most of the exceptions are somewhat old-fashioned report writers. In the 1990s and early 2000s, however, business intelligence had several distinct subcategories …

… one of which was called OLAP (OnLine Analytic Processing). Actually,  the term “OLAP” has been confusingly been used to mean several different things, including:

• Pretty much all of analytics.
• A particular non-relational DBMS architecture that I prefer to refer to as MOLAP (Multidimensional OLAP).
• An integrated suite of DBMS, 4GL, and perhaps other tools around a MOLAP architecture. (Examples: The IRI Express and Arbor/Hyperion Essbase products Oracle bought.)
• A client-side BI tool with a little MOLAP DBMS built in. (Example: Cognos’ erstwhile flagship BI product PowerPlay.)

I hate the term “OLAP” with a passion, in part due to that confusion, and in part due to the specific way the confusion came about: Ted Codd introduced the term, allegedly objectively, but actually as a marketing shill for Arbor Software, which had an obvious business incentive to pretend that its specific technologies solved a broader class of problems than they actually did.

OK. With that too cleared away, I feel ready to write about the actual history of analytic technology.

• Historical notes on analytics — the pre-computer era (this post)
  
• Historical notes on analytic terminology
• Historical notes on analytics — departmental adoption

Sometimes, what people describe as being “New, new, new!!!” in analytics has actually been happening since before they were born, or even before their parents were. Occasionally, I point this out. I think it’s time to collect some of those observations into a short series of posts.

Before getting to the history of actual analytic software, I can’t resist racing through some really old stuff. In a 2004 white paper, I wrote:

Transactional business processes have been around literally since the beginning of recorded history. Some of the oldest known writings are clay tablets that record merchants’ tallies in Sumerian cuneiform, complete with seals to enforce transaction integrity. Analytic business processes date back nearly as long, especially in military applications; the first chapter of Sun Tzu’s The Art of War is called “Calculations,” or in some translations “Laying Plans.”*

As enterprise complexity increased, so did the sophistication of analytic business processes. Almost two centuries ago, Nathan Rothschild made an investment fortune from early news about the Battle of Waterloo, and several decades later Florence Nightingale** introduced statistics to the study of public health. With the invention of machines to tabulate information in the late 19th Century, analysis began to blossom.

*Back when I wrote that, I also considered including some of the accounting Caesar cited in his Commentaries on the Gallic Wars, but eventually decided that was a “production application” rather than anything we’d recognize as “analytics”.

**Florence Nightingale is, simply put, one of the more impressive women in history. Unfortunately, a couple of other statistical greats were associated with the deplorable subject of eugenics. I am thinking specifically of Francis Galton, who invented several really basic statistical concepts, and seems to generally have been a brilliant guy, and Karl Pearson. (Hat-tip to John Verostek for, um, tipping me off to Galton.)

Statistics also seems to have led the way in business applications of analytics. Specifically, statistical quality control dates back to the 1920s or so, pioneered by Walter Shewart and given greater visibility by his protégé W. Edwards Deming. On the monitoring side, various organizations collected industry-wide numbers by the 1930s or so. For example, movie box office receipts were reported at least as far back as 1939, perhaps by Variety; the New York Times bestseller list seems to have started slightly later, in 1942; and a rather superficial site on media history first gives hard numbers for the decade of the 1930s.

Still, to continue quoting the same white paper:

What utterly transformed both transactional and analytic business processes was the advent of electronic computing in the 20th Century. In particular, the volume of available data exploded. Even more important to analytic processes was the superhuman increase in the speed of computation. Various types of software emerged — business intelligence (BI) tools, spreadsheets, statistical packages, and the like – permitting kinds of analysis that had been infeasible without computers.

And with that out of the way, let’s return to discussing computer-era analytics.

• Oracle started out doing what amounted to custom development for government (military/intelligence) clients.
• Even when Oracle said it had productized its software, the stuff didn’t work very well without services to get it running.
• Oracle and Ingres both got a huge fraction of their early revenue* from deals to port their software to various brands of hardware.** That’s a lot like professional services.
• Oracle’s huge Tools Group grew out of professional services, if I have the story straight. Indeed, its first product was written by later long-time group chief Sohaib Abbasi when he was a consultant.

*Revenue recognition rules were rather different back then. Multi-million payments or guarantees for ports could be recognized as lump-sum revenue up front.

**Ingres once ran on more hardware platforms than it had employees, when both numbers were somewhere in the 40s. Most of the boxes on which the porting was tested were in one small office.

The benefits for a young software company of being in the professional services business include:

• It can be high-margin, especially if it shares the cost of sales with your software offering.
• It allows you to say “yes” to whatever the customer asks for. (More precisely, it takes you closer to that goal that you’d be without a service offering.)
• It allows you to fund capable staff. Or to put it even more bluntly …
• … professional services brings in revenue that pays your bills.
• It gets you involved with customers, learning stuff about their needs, and specifically addressing them.

The disadvantages of professional services generally boil down to various forms of defocus; you can screw up your development schedule, your development priorities, your sales priorities, your partnering efforts, your market positioning, your burn rate or just about anything else.

Many software companies pursue substantial professional services when they’re young. Many don’t. Both strategic choices can make sense.

• The brochure quote basically pitches business intelligence/ performance management, with unconstrained drilldown.
• Pitching BI/analytics benefits for what start out as being transactional applications has been going on for pretty much the whole history of the applications industry. This just happens to be a great proof point.
• The unconstrained drilldown part could almost be taken for granted today, in the relational era. In 1972, however, it was a rather bold (and for all I know exaggerated) claim.

The exact verbiage is: 

1.  The system should focus management attention on the critical variables in distribution profitability, providing measures of the productivity of all key resources of the firm.

2.  Where deviations from expected performance levels are detected, they should be highlighted on an exception basis, so that managers can allocate their most critical resource — time — to the problems or opportunities that variances in performance represent.

3. The manager should be able to pursue the analysis of an exception to whatever level of detailed information is needed to take positive action; and the system should, therefore, permit the manager to secure information at any desired level of detail without prior knowledge of his information needs.

Here’s the part I know from my own memories as

• The son of a Management Horizons employee (namely my Dad).
• A software industry stock analyst (in particular, one who followed Informatics General).

My father moved to the Columbus area in 1973 to join Management Horizons, a consulting firm serving retailers. Management Horizons had its own spin-out already, a time-sharing company called Management Horizons Data Services (MHDS), with which it still shared a building on what is now Old Henderson Road in Upper Arlington. And, this being a world full of coincidences, MHDS is very on-topic for the primary focus of this blog (software industry history).

MHDS’ main business was a full suite of what we might now call ERP for distributors and/or retailers. That never amounted to much. But its secondary business was an electronic interchange for direct placement of orders, called Ordernet. Ordernet turned into Sterling Commerce, a > $1/2 billion company that has been acquired for >$1 billion more than once.

The chain of events, roughly, is: 

• Management Horizons sold MHDS to what would become Citibank.
• Citibank flipped MHDS to early software industry conglomerate Informatics General.
• Sterling Software grew itself 10X in size by a hostile takeover of Informatics. (Sam Wyly was involved in that.)
• Sterling split into Sterling Commerce and what I might call The Rest of Sterling.

At that point I forget the details, but a couple of multi-billion dollar acquisitions/divestitures have ensued.

Some day I may dig out the numbers for Informatics’ revenue breakdown in the early 1980s. Ordernet wasn’t big, but looked like a rising star. MHDS  classic wasn’t big either, and didn’t look like it was going anywhere. Both appearances were later born out.

Online research uncovered some other sources, namely:

• A two-part biography of Alton Doody’s Management Horizons co-founder William Davidson, whose main relevance to this post is that it pointed me to …
• … a 1996 history of Management Horizons written by Davidson.
• An oral history from Informatics co-founder Werner Frank.

On pages 18-19, the Werner Frank link notes that Informatics acquired MHDS in what sounds like 1974, from then-owner Citibank. MHDS was doing $7.8 million in revenue (which Frank failed to break out among its two business segments), was acquired for $3.4 million, and apparently brought a $3.3 million long-term contract along with it, as well as a cheap loan.

Unfortunately, Davidson’s history* is inaccurate in some details I can check from memory. E.g. it says the Doody Company was created in 1974, which is too early, and it omits Informatics General from the history of MH’s information processing division. It also includes Aetna in the list of outfits that owned MHDS, which is news to me and also, apparently, to Werner Frank.

*As I write this, another copy is available for sale on Amazon. Just Google on “History of Management Horizons.” Yes, we paid around $20 for my copy too, even though there’s a $0.99 price tag on it clearly crossed out.

Anyhow, Davidson wrote:

• One of Management Horizons’ founding projects was helping retrain the NCR sales force away from dumb cash registers and toward what we’d now call point-of-sale devices, a key step in the IT revolution of actually using POS information.
• MHDS started out with two leased IBM System/370-155 computers.
• MHDS was divested to Citicorp’s predecessor bank in May of 1973. “At the time of divestiture some 92 wholesale distribution sites were under contract to MHDS with about 30 or so on-line to it.”

Ingres and Oracle were developed around the same time, in rapidly-growing startup companies. Ingres generally was the better-featured product, moving a little earlier than Oracle into application development tools, distributed databases, etc., whereas Oracle seems to be ahead on the most important attributes, such as SQL compatibility — Oracle always used IBM’s suggested standard of SQL, while Ingres at first used the arguably superior Quel from the INGRES research project. Oracle eventually pulled ahead on superior/more aggressive sales and marketing.

Then in the 1990s, Ingres just missed the DBMS architecture boat. Oracle, Informix, Microsoft, and IBM all came out with completely new products, based respectively on Oracle + Rdb, Informix + a joint Ingres/Sequent research project, Sybase, and mainframe DB2. Ingres’s analogous effort basically floundered, in no small part because they made the pound-wise, penny-foolish decision to walk away from a joint venture research product they’d undertaken with innovative minicomputer vendor Sequent in the Portland, OR area.

Computer Associates bought Ingres in mid-1994, and immediately brought me in to do a detailed strategic evaluation. (Charles Wang telephoned the day the acquisition closed, in one of the more surprising phone calls I’ve ever gotten, but I digress … Anyhow, the relevant NDA agreements, legal and moral alike, have long since expired.) There was nothing terribly wrong with the product, but unfortunately there was nothing terribly right either. Aggressive investment — e.g., to get fully competitive in parallelism and object/relational functionality, the two biggest competitive differentiators in those days — would have been no guarantee of renewed market success.

Notwithstanding the economic question marks, CA surprised me with its enthusiasm for taking on these technical challenges. But another problem reared its head — almost all the core developers left the company. (If you weren’t willing to sign a noncompete agreement that was utterly ridiculous in those days, at least in the hot Northern California market, you couldn’t keep your job post-merger.) And so, like almost all CA acquisitions outside of the system management/security/data center areas, Ingres fell further and further behind the competition.

Some of the same information made it into my post here on Ingres history later the same year, but for some reason not all did.

Comshare — a BI vendor before anybody called it BI — did actually build a DBMS based on Childs’ ideas, as Ron Jeffries reminds us. It was relational. Eventually, if I recall correctly, it was swapped out for Essbase (the original MOLAP product, now owned by Oracle).

What Childs really focuses on, however, seems to be “Extended Set Theory.” (This was brought to my attention by Algebraix, even though Algebraix doesn’t actually use many of Childs’ ideas.) And he’s been doing it for a long time. Way back in 1968, Childs wrote a paper outlining how set theory, relations, and tuples could be applied to data management.

And that’s where I did a double-take, because 1968 < 1970. Sure enough, Footnote #1 in Codd’s seminal paper is to Childs’ 1968 work. Indeed, Childs’ paper is the only predecessor Codd acknowledges as having significant portions of his idea.

I’m far from convinced that “Extended set theory” has much to offer versus the standard relational model. But that debate quite aside — Childs’ original achievement doesn’t get the credit it deserves.

Folks, QUEL was the original language for Postgres. And Ingres. And, more or less, Teradata.*  I’d guess Britton-Lee too, but I don’t recall for sure.

*Once upon a distant time, when I was a cocky young stock analyst, I explained to Phil Neches, chief scientist of Teradata, just why it was a really good business idea to drop T-QUEL for SQL. I doubt he was convinced quite on that day, more’s the pity.

*A blog-tagging meme is, in essence, an internet chain letter without the noxious elements.

Without further ado, the Rules of the No-Fooling Meme are:

Rule 1: Post on your blog 1 or more surprisingly true things about you,* plus their explanations. I’m starting off with 10, but it’s OK to be a lot less wordy than I’m being. I suggest the following format:

• A noteworthy capsule sentence. (Example: “I was not of mortal woman born.”)
• A perfectly reasonable explanation. (Example: “I was untimely ripped from my mother’s womb. In modern parlance, she had a C-section.”)

*If you want to relax the “about you” part, that’s fine too.

Rule 2: Link back to this post. That explains what you’re doing.

Rule 3: Drop a link to your post into the comment thread. That will let people who check here know that you’ve contributed too.

Rule 4: Ping 1 or more other people encouraging them to join in the meme with posts of their own.

Hopefully, the end result of all this will be that we all know each other just a little bit better! And hopefully we’ll preserve some cool stories as well.

To kick it off, here are my entries. (Please pardon any implied boastfulness; a certain combustibility aside, I’ve lived a pretty fortunate life.)

I was physically evicted by hotel security from a DBMS vendor’s product announcement venue. It was the Plaza Hotel in NYC, at Cullinet’s IDMS/R announcement. Phil Cooper, then Cullinet’s marketing VP, blocked my entrance to the ballroom for the main event, and then called hotel security to have me removed from the premises.

A few years later, the same Phil Cooper stood me up for a breakfast meeting in his own house in Wellesley. When one’s around Phil Cooper, weird things just naturally happen.

I got James Marsters (“Spike” on Buffy the Vampire Slayer) to autograph a shirtless picture of himself. Linda was a very serious Buffy fan, and I was no slouch in that regard myself. So we flew out to Santa Barbara to join some acquaintances from a Buffy-centric mailing list. Except first, we went – along with the hostess for the gathering – to see James perform with his rock band Ghost of the Robot in Santa Monica. She had downloaded a photo she wanted to pass out to the other (mostly female) attendees. But neither she nor Linda wanted to actually ask him to autograph it – too fangirlish or something. So I stepped up and made the request in their place. (Technically, asking James to sign anything other than a Ghost of the Robot CD was against the rules – but he was more than gracious when I said that if he signed for me it would be a great help to my relationship. )

I had (inter)national reputations in four different fields before my 24th birthday – two academic, two non-academic. I got my PhD in game theory, independently proving a theorem that was simultaneously (and a few months earlier) proved by Mertens and Neyman. Naturally, the game theory community was quite aware of my work. Then I did a post-doc in public policy for a couple of years. Some of my work – sort of a public-sector version of real options analysis, although unfortunately I wasn’t familiar with that concept – was widely circulated among the US utility regulation community. Then, at age 21, I went to Wall Street, as a stock analyst covering the software industry for a major firm (PaineWebber). That pretty much assured me of being known in both the investment and software businesses, especially as I soon got pretty high in the third-party stock analyst rankings.

Graduating college at age 16 cost me my NCAA eligibility – and I wound up regretting that aspect. On the whole, I’m hardly an athlete. But I did take a few fencing classes, and enjoyed them. So when my grad school offered an intermediate intramural fencing class, I was psyched. But when I went the first day, they explained that they didn’t have the resources to offer the class. Instead, anybody who wanted to could join and hence work out with the Harvard fencing team, whether or not they were good enough to compete. Only they couldn’t do that for me – because my NCAA eligibility had been shot when I graduated from Ohio State.

I double-dated with Larry Ellison, twice. Part of the explanation is that when I lived in Manhattan, I had almost no friends there, but quite a few in the SF Bay area. (Seriously, which kind of work acquaintances would you expect me to have more in common with – tech entrepreneurs or Wall Street professionals?) So if I wanted to introduce a new girlfriend to my friends, we’d fly out west. And if there’s one company at which I had a lot of friends, it was Oracle, which I’d first visited in 1983, when it was still located at 3000 Sand Hill Road and had fewer than 50 employees. I was very engaged with Oracle professionally through most of the 1990s – but we also got together just for fun.

I was the top-rated chess player my age in the United States. This is when I was 13 years old, in 1973. But I wasn’t nearly as good as that factoid suggests. 1973 may have been the weakest year for 13-year-old US chess players in modern times; I made #1 with only a 1650 rating. (To put that number in context, it indicates that this 2300+ rated 13-year-old could have routinely obliterated players who could, in turn, have routinely obliterated me.) More on the story may be found at this link.

I told people at a conference “I just spent the afternoon with Bill Gates’ girlfriend — and boy is my butt sore!” This was in 1985 or so. I was a stock analyst living in NYC. The annual American Electronics Association investment conference in Monterey was a must-attend event. Flight connections, however, were imperfect. So Ann Winblad had the idea that, during my several-hour layover, she’d pick me up at SFO, and we’d go drive around the mini race track that’s still there off of 101. I got pretty sore bouncing around in the little plastic cars, perhaps because I wasn’t a very experienced driver. Indeed …

… I didn’t own (or lease) a car until I was 36 years old. (To readers from California, this one may sound the oddest of all. ) I left home at age 16 to go to school in Cambridge, MA, where one doesn’t typically have a car. I left there at age 21 to move to Manhattan. Finally, when I was 36, I left Manhattan for Lexington, MA, at which point I of course got wheels. That Toyota served me well for about a decade, but eventually …

… my first car forced a DBMS vendor to evacuate its whole office building. My Toyota Camry had an engine fire in Intersystems’ building on Memorial Drive, and the building has indoor parking. The story is here. Actually, many other companies had to evacuate the same building.

My kitchen caught on fire, just as I was Twittering with LeVar Burton of Star Trek:TNG and Roots fame. That juxtaposition was a total coincidence. LeVar had just tweeted me his vehement agreement to something I said, when Linda appeared at my office door, having noticed the sound of what turned out to be a fire on the stove. This story has been the subject of several other blog posts …

By the way, what I’d said was, after LeVar tweeted his pleasure at for once actually acting again (on stage, no less), that we choose our original professions for a reason – taking them up again, even for a little while, is like going home again. Even today, I rarely feel more right than when I’m doing mathematics.