Re: Oracle threatens to sue Standish over an article

In <CAAKsE.MDJ_at_feanor.xel.com> shaw_at_feanor.xel.com (Greg Shaw) writes:

>Hal Berenson (berenson_at_nova.enet.dec.com) wrote:
 

>: In article <1993Jul3.021408.10121_at_qiclab.scn.rain.com>, tcox_at_qiclab.scn.rain.com (Thomas Cox) writes...
>[ bunch o' stuff deleted ]
 

>: So the problem with a "no benchmark special" rule is that determining
>: what is or isn't a benchmark special is very subjective. How would a
>: vendor prove that a feature wasn't a benchmark special? Would they have
>: to produce a customer who would testify they are using the feature?
>: Would one be enough? Two? Ten? Should the TPC provide the exact
>: program to be run, thus forcing the use of standard SQL syntax and
>: preventing the use of vendor-specific features? Would this really
>: provide a useful comparison, since customers DO use many non-standard
>: features, or would it be just another useless data point?
 

>In a word, YES.

Is that YES it's a useful comparison, or YES it's a useless data point??? I've now read your post three times, and I'm still not sure!

> When evaluating DB systems, the TPC benchmark is used
>*OFTEN* to compare A vs. B. Period. 'Special' features that influence a
>benchmard (aka propietary features) are of *NO* use, because you're not
>talking about inplementing everyting in a vendor-specific situation.
>You're talking about apples vs. apples, DB vendor vs. DB vendor.

Which means that a vendor specific feature that you would choose to use (or that is automatic, for that matter) *should* be measured. Fast/ Group Commits were vendor specific, it's just that they're now specific to just about every vendor. The chances are that *your* particular database now goes faster because of that particular "benchmark special".

>Because Oracle uses a 'feature' that is not something that would be useful
>in a REAL WORLD situation (aka any real situation) to 'up' their TPC
>benchmarks disqualifies them, in my opinion.

Just because it's not useful in a particular real world situation does *NOT* mean that it is not useful in any real world situation.

>They play 'dirty pool'. Period. I want to see what *REALLY* happens with
>a DB system, not some fake system.

Don't we all?

> When TPC can do a real world test
>(hopefully varied), I expect it to accurately reflect what I can do with
>*MY* database.

But I want it to reflect *MY* database, not *YOUR* database ... and there's the rub!

> 'Discrete transactions' are not what I can do with my
>database ... and expect the database to be intact at the end of the day.

Why wouldn't you expect it to be intact at the end of the day? If it can do more than 60 million TPC-A transactions in a day and remain intact, why wouldn't it for you? It sounds as if you've misunderstood Discrete Transaction. It passes the ACID test, so where's the beef?

>: The TPC does have a very strict definition of what the benchmark can and
>: can not do. That definition exists at a fairly high level (essentially a
>: detailed functional specification) so that any software system on any
>: hardware platform can run the benchmark. So, it is possible using TPC
>: benchmarks to compare a TPF2 system with the application implemented in
>: IBM 370 assembly language accessing a private file structure with
>: application implemented recovery mechanisms to a Compaq PC running SCO
>: UNIX and ORACLE with the application written in C calling PL/SQL
>: procedures. You can also use TPC benchmarks to evaluate the
>: suitability of OODB products for TP applications, which you couldn't do
>: if the TPC provided an ISO SQL program as the only valid benchmark
>: program.
 

>Again, real world vs. fantasy world.

I'm confused again, you mean these are your fantasies???!

>: 2) The real point, the one Standish made but more or less seems to be
>: getting lost in the noise between them and Oracle, is that the TPC-A and
>: TPC-B benchmarks are obsolete as comparators of database and transaction
>: processing systems. When a benchmark is created all vendors will try to
>: optimize for that benchmark. At some point the vendors will "crack" the
>: benchmark by tuning their systems to the point that the benchmark (a) no
>: longer demonstrates user-useful differences between products and/or (b)
>: no longer can be extrapolated to predict the performance of customer
>: applications. TPC-A and TPC-B have more or less reached this point.
>: This doesn't make them bad benchmarks nor does it make vendors who bent
>: over backwards to "win" these benchmarks unethical. TPC-A&B have caused
>: about an order of magnitude improvement in simple-query pathlength,
>: recovery algorithms, and I/O since 1987. Nearly every customer
>: application had benefited dramatically from the TPC-A&B benchmark wars.
>: 1993 pretty much marks the end of those customer benefits from TPC-A&B.
>: The TPC-C benchmark not only is representative of a much broader set of
>: applications then TPC-A&B, it poses a host of new problems demanding
>: vendor attention and optimization. TPC-A&B should be retired as obsolete
>: and both customers and vendors should focus on TPC-C (and the soon to
>: appear TPC-D). TPC-C&D will provide more useful information to customers,
>: and over the next 2-4 years will drive dramatic additional improvements
>: in database system performance.
 

>I have to agree,

Great. Maybe we're *both* in the real world, after all!

> with one qualifier: If the TPC-C benchmark (and TPC-X
>where X>C) *ACCURATELY* reflects what goes on on a typical DB machine, then
>what you say has relevance. If it does not, however, it means that
>databases are being optimized for functions that are not common.

It doesn't mean that at all! Even if you were prescient enough to be able to design a benchmark which tested all the functions in common usage, you would find that the number of typical DB machines that it *ACCURATELY* reflected was < 1!

A typical DB machine actually uses only a subset of the common functions, plus a subset of the less common functions. The problem is that each one (ie each schema design/application, not each occurrence/ installation) uses a *different* pair of subsets.

>I do feel that any TPC benchmark will test some functionality of a DB
>system. I contend that the system must be demonstrably close to what
>happens in the real world to be usable and/or believable.

The closer it gets to what happens in any particular piece of the real world, the more specialised it becomes, and, as a direct result, the less useful it becomes (except to the inhabitant of that particular real-world location). Whether it is believable or not is an administrative issue, and has nothing to do with functionality.

>: .............................................................................
 

>: Hal Berenson
 

>: Home: 71640.3535_at_compuserve.com
>: Work: berenson_at_nova.enet.dec.com
 

>: -- Disclaimer: Opinions expressed here are my own and should not be
>: construed as representing the views of my employer or its employees,
>: officers, directors, or stockholders. --
 

>Greg.
>--
>_______________________________________________________________________________
>You can't go against nature, because when you do, Greg Shaw
>go against nature, it's part of nature too. shaw_at_feanor.xel.com
> Love & Rockets uunet!csn!xel.com!shaw

graeme

--
Disclaimer:	The author's opinions are his own, and not necessarily
		those of Pyramid Technology Ltd. or Pyramid Technology Inc.
---------------------------------------------------------------------------
      -m------- Graeme Sargent                 Voice: +44 (0)252 373035
    ---mmm----- Senior Database Consultant     Fax  : +44 (0)252 373135
  -----mmmmm--- Pyramid Technology Ltd.        Telex: Tell who???
-------mmmmmmm- Farnborough, Hants  GU14 7PL   Email: graeme_at_pyra.co.uk
---------------------------------------------------------------------------
    We have the technology.  The tricky bit is learning how to use it.