Re: Oracle threatens to sue Standish over an article

In article <1993Jul3.021408.10121_at_qiclab.scn.rain.com>, tcox_at_qiclab.scn.rain.com (Thomas Cox) writes...
>timo_at_mits.mdata.fi (Timo Talasmaa) writes:
>2. There appears to be no doubt that all benchmarks of Oracle7 have been
> completely within the letter of the TPC (Transaction Processing
> Council) rules, and all have been audited. This is in contrast to
> DEC's RDB benchmark not long ago that was *withdrawn* for *not*
> following the rules.

Excuse me, but that's bull crap. There was one particular benchmark for Rdb where the TPC changed the rules (clarified a technicality) after the benchmark was published. The benchmark configuration was obsolete by that time, so rather then correct the technicality and spend $10s of thousands to re-run and re-audit that particular test it was simply withdrawn. This happened SEVERAL years ago by the way. Several other vendor tests have been withdrawn as a result of technical rulings, or of the vendor discovering a mistake in the benchmark results.

Challenges to TPC results, or to their misrepresentation by vendors, occur all the time. The TPC has mechanisms for resolving these disputes internally and so far, that I can remember, no public censoring or ejection from TPC has been required. Just because nothing has reached the general public, don't simply assume your favorite vendor is guilt-free.

The Oracle7 results (and all currently published results) are completely valid within TPC rules. Extensive study was done by several parties to find a violation of the rules, and the one challenge made was rejected. I don't believe that's the issue people should be focusing on.

There are two relevent issues here:

1. Should there be some mechanism in the TPC process for rejecting "benchmark special" features? On the surface the easy answer is yes, but as you study the problem you discover "benchmark specials" are in the eye of the beholder. Rdb has many tuning features that are designed to help application designers optimize for their particular environment. Some of these features apply only to high-volume update intensive processing. Are they benchmark specials? Hardly. Then why should Oracle7's "Discrete Transactions" be considered a benchmark special?

The uproar over discrete transactions is that they severely restrict the usage of SQL within the transaction and that they severely mess up Oracle's standard concurrency control mechanisms (thus making it unlikely discrete transactions can be used in any application with substantially more complex queries then TPC-A). To a large extent the objection to discrete transactions is not based on the validity of the feature, but rather on how un-Oracle-like they are and therefor how unlikely it is that an Oracle customer could or would use them!

So the problem with a "no benchmark special" rule is that determining what is or isn't a benchmark special is very subjective. How would a vendor prove that a feature wasn't a benchmark special? Would they have to produce a customer who would testify they are using the feature? Would one be enough? Two? Ten? Should the TPC provide the exact program to be run, thus forcing the use of standard SQL syntax and preventing the use of vendor-specific features? Would this really provide a useful comparison, since customers DO use many non-standard features, or would it be just another useless data point?

The TPC does have a very strict definition of what the benchmark can and can not do. That definition exists at a fairly high level (essentially a detailed functional specification) so that any software system on any hardware platform can run the benchmark. So, it is possible using TPC benchmarks to compare a TPF2 system with the application implemented in IBM 370 assembly language accessing a private file structure with application implemented recovery mechanisms to a Compaq PC running SCO UNIX and ORACLE with the application written in C calling PL/SQL procedures. You can also use TPC benchmarks to evaluate the suitability of OODB products for TP applications, which you couldn't do if the TPC provided an ISO SQL program as the only valid benchmark program.

2) The real point, the one Standish made but more or less seems to be getting lost in the noise between them and Oracle, is that the TPC-A and TPC-B benchmarks are obsolete as comparators of database and transaction processing systems. When a benchmark is created all vendors will try to optimize for that benchmark. At some point the vendors will "crack" the benchmark by tuning their systems to the point that the benchmark (a) no longer demonstrates user-useful differences between products and/or (b) no longer can be extrapolated to predict the performance of customer applications. TPC-A and TPC-B have more or less reached this point. This doesn't make them bad benchmarks nor does it make vendors who bent over backwards to "win" these benchmarks unethical. TPC-A&B have caused about an order of magnitude improvement in simple-query pathlength, recovery algorithms, and I/O since 1987. Nearly every customer application had benefited dramatically from the TPC-A&B benchmark wars. 1993 pretty much marks the end of those customer benefits from TPC-A&B. The TPC-C benchmark not only is representative of a much broader set of applications then TPC-A&B, it poses a host of new problems demanding vendor attention and optimization. TPC-A&B should be retired as obsolete and both customers and vendors should focus on TPC-C (and the soon to appear TPC-D). TPC-C&D will provide more useful information to customers, and over the next 2-4 years will drive dramatic additional improvements in database system performance.

.............................................................................

Hal Berenson

Home: 71640.3535_at_compuserve.com
Work: berenson_at_nova.enet.dec.com

• Disclaimer: Opinions expressed here are my own and should not be construed as representing the views of my employer or its employees, officers, directors, or stockholders. --