Re: Oracle threatens to sue Standish over an article

Hal Berenson (berenson_at_nova.enet.dec.com) wrote:

: In article <1993Jul3.021408.10121_at_qiclab.scn.rain.com>, tcox_at_qiclab.scn.rain.com (Thomas Cox) writes...
[ bunch o' stuff deleted ]

: So the problem with a "no benchmark special" rule is that determining
: what is or isn't a benchmark special is very subjective. How would a
: vendor prove that a feature wasn't a benchmark special? Would they have
: to produce a customer who would testify they are using the feature?
: Would one be enough? Two? Ten? Should the TPC provide the exact
: program to be run, thus forcing the use of standard SQL syntax and
: preventing the use of vendor-specific features? Would this really
: provide a useful comparison, since customers DO use many non-standard
: features, or would it be just another useless data point?

In a word, YES. When evaluating DB systems, the TPC benchmark is used *OFTEN* to compare A vs. B. Period. 'Special' features that influence a benchmard (aka propietary features) are of *NO* use, because you're not talking about inplementing everyting in a vendor-specific situation. You're talking about apples vs. apples, DB vendor vs. DB vendor.

Because Oracle uses a 'feature' that is not something that would be useful in a REAL WORLD situation (aka any real situation) to 'up' their TPC benchmarks disqualifies them, in my opinion.

They play 'dirty pool'. Period. I want to see what *REALLY* happens with a DB system, not some fake system. When TPC can do a real world test (hopefully varied), I expect it to accurately reflect what I can do with *MY* database. 'Discrete transactions' are not what I can do with my database ... and expect the database to be intact at the end of the day.

: The TPC does have a very strict definition of what the benchmark can and
: can not do. That definition exists at a fairly high level (essentially a
: detailed functional specification) so that any software system on any
: hardware platform can run the benchmark. So, it is possible using TPC
: benchmarks to compare a TPF2 system with the application implemented in
: IBM 370 assembly language accessing a private file structure with
: application implemented recovery mechanisms to a Compaq PC running SCO
: UNIX and ORACLE with the application written in C calling PL/SQL
: procedures. You can also use TPC benchmarks to evaluate the
: suitability of OODB products for TP applications, which you couldn't do
: if the TPC provided an ISO SQL program as the only valid benchmark
: program.

Again, real world vs. fantasy world.

: 2) The real point, the one Standish made but more or less seems to be
: getting lost in the noise between them and Oracle, is that the TPC-A and
: TPC-B benchmarks are obsolete as comparators of database and transaction
: processing systems. When a benchmark is created all vendors will try to
: optimize for that benchmark. At some point the vendors will "crack" the
: benchmark by tuning their systems to the point that the benchmark (a) no
: longer demonstrates user-useful differences between products and/or (b)
: no longer can be extrapolated to predict the performance of customer
: applications. TPC-A and TPC-B have more or less reached this point.
: This doesn't make them bad benchmarks nor does it make vendors who bent
: over backwards to "win" these benchmarks unethical. TPC-A&B have caused
: about an order of magnitude improvement in simple-query pathlength,
: recovery algorithms, and I/O since 1987. Nearly every customer
: application had benefited dramatically from the TPC-A&B benchmark wars.
: 1993 pretty much marks the end of those customer benefits from TPC-A&B.
: The TPC-C benchmark not only is representative of a much broader set of
: applications then TPC-A&B, it poses a host of new problems demanding
: vendor attention and optimization. TPC-A&B should be retired as obsolete
: and both customers and vendors should focus on TPC-C (and the soon to
: appear TPC-D). TPC-C&D will provide more useful information to customers,
: and over the next 2-4 years will drive dramatic additional improvements
: in database system performance.

I have to agree, with one qualifier: If the TPC-C benchmark (and TPC-X where X>C) *ACCURATELY* reflects what goes on on a typical DB machine, then what you say has relevance. If it does not, however, it means that databases are being optimized for functions that are not common.

I do feel that any TPC benchmark will test some functionality of a DB system. I contend that the system must be demonstrably close to what happens in the real world to be usable and/or believable.

: .............................................................................
 

: Hal Berenson
 

: Home: 71640.3535_at_compuserve.com
: Work: berenson_at_nova.enet.dec.com
 

: -- Disclaimer: Opinions expressed here are my own and should not be
: construed as representing the views of my employer or its employees,
: officers, directors, or stockholders. --

Greg.

-- 
_______________________________________________________________________________
You can't go against nature, because when you do, 	Greg Shaw
go against nature, it's part of nature too.		shaw_at_feanor.xel.com 
			Love & Rockets			uunet!csn!xel.com!shaw