July 2, 2009

Historical significance of TPC benchmarks

In case you missed it, I’ve had a couple of recent conversations about the TPC-H benchmark.  Some people suggest that, while almost untethered from real-world computing, TPC-Hs inspire real world product improvements.  Richard Gostanian even offered a specific example of same — Solaris-specific optimizations for the ParAccel Analytic Database.

That thrilling advance notwithstanding, I’m not aware of much practical significance to any TPC-H-related DBMS product development. But multiple people this week have reminded me this week the TPC-A and TPC-B played a much greater role spurring product development in the 1990s.  And I indeed advised clients in those days that they’d better get their TPC results up to snuff, because they’d be at severe competitive disadvantage until they did.

It’s tough to be precise about examples, because few vendors will admit they developed important features just to boost their benchmark scores. But it wasn’t just TPCs — I recall marketing wars around specific features (row-level locking, nested subquery) or trade-press benchmarks (PC World?) as much as around actual TPC benchmarks.  Indeed, Oracle had an internal policy called WAR, which stood for Win All Reviews; trade press benchmarks were just a subcase of that.

And then there’s Dave DeWitt’s take.  Dave told me yesterday at SIGMOD that it’s unfortunate Jim Gray-inspired debit/credit TPCs won out over the Wisconsin benchmarks, because that led the industry down the path of focusing on OLTP at the expense of decision support/data warehousing.  Whether or not the causality is as strict as Dave was suggesting, it’s hard to dispute that mainstream DBMS met or exceeded almost all users’ OTLP performance needs by early in his millenium. And it’s equally hard to dispute that those systems* performance on analytic workloads, as of last year, still needed a great deal of improvement.

*IBM’s DB2 perhaps excepted. And I say “last year” so as to duck the questions of whether Exadata finally solved Oracle’s problems and whether Madison will once Microsoft releases it.

Comments

4 Responses to “Historical significance of TPC benchmarks”

  1. Serge Rielau on July 3rd, 2009 9:25 am

    Curt,

    An interesting post. There is no doubt in my mind that both TPC benchmarks caused product improvements. It pays however to give developers some credit here. We are at heart lazy creatures and do hate throw-away code.
    Let’s take TPC-D and “materialized views” for example. There was a race between Oracle and IBM DB2 at the time as to who gets them out first and of course one major marketing goal was certainly to bat the TPC-D performance numbers out of the ballpark which rendered the whole benchmark obsolete within months.
    A more recent example is TPC-C. Remember those Oracle adds on “IBM won’t say”. Eventually soem IBM exec must have had it and decided that enough is enough. What followed was a flurry of activities resulting in DB2 8.1.4 and the breach of the 700k, 1M and now 6M TpmC threshholds.
    You might say that all features delivered in DB2 8.1.4 were specifically implemented for the benchmark. that is true as far as motivation is concerned (i.e. executive funding).
    The features however, such as a rewrite of DB2’s logger for ultra high performance, “alternate page cleaners”, and my personal toy the “unification of queries and updates”. Essentially we developers don’t care for the excuse to put in features which we believe will benefit the product. I’m confident this is no different for other vendors.

    Cheers
    Serge

  2. Curt Monash on July 3rd, 2009 5:12 pm

    Makes sense, Serge. Thanks.

  3. A few notes from XLDB 4 | DBMS 2 : DataBase Management System Services on October 10th, 2010 1:49 pm

    [...] their products run faster for scientific purposes, in line with the supposed salutary effects of TPC-A, TPC-B, and TPC-C. Notwithstanding that attendees included Oracle, Microsoft, EMC/Greenplum, Teradata, and [...]

  4. Confluence: Global Benchmark Center Collaboration on September 24th, 2012 1:50 pm

    Belgacom…

    09/24 A Request for either a Benchmark or POC has been received via the Global Benchmark Center website…….

Leave a Reply




Feed including blog about software history Subscribe to the Monash Research feed via RSS or email:

Login

Search our blogs and white papers

Monash Research blogs

User consulting

Building a short list? Refining your strategic plan? We can help.

Vendor advisory

We tell vendors what's happening -- and, more important, what they should do about it.

Monash Research highlights

Learn about white papers, webcasts, and blog highlights, by RSS or email.