Re: Oracle threatens to sue Standish over an article

Graeme Sargent (graeme_at_pyra.co.uk) wrote:
: In <CAAKsE.MDJ_at_feanor.xel.com> shaw_at_feanor.xel.com (Greg Shaw) writes:
 

: >Hal Berenson (berenson_at_nova.enet.dec.com) wrote:
 

: >: In article <1993Jul3.021408.10121_at_qiclab.scn.rain.com>, tcox_at_qiclab.scn.rain.com (Thomas Cox) writes...
: >[ bunch o' stuff deleted ]
 

: >: So the problem with a "no benchmark special" rule is that determining
: >: what is or isn't a benchmark special is very subjective. How would a
: >: vendor prove that a feature wasn't a benchmark special? Would they have
: >: to produce a customer who would testify they are using the feature?
: >: Would one be enough? Two? Ten? Should the TPC provide the exact
: >: program to be run, thus forcing the use of standard SQL syntax and
: >: preventing the use of vendor-specific features? Would this really
: >: provide a useful comparison, since customers DO use many non-standard
: >: features, or would it be just another useless data point?
 

: >In a word, YES.
 

: Is that YES it's a useful comparison, or YES it's a useless data point???
: I've now read your post three times, and I'm still not sure!

YES it's a useless data point. I'm sorry for any ambiguity.

: > When evaluating DB systems, the TPC benchmark is used
: >*OFTEN* to compare A vs. B. Period. 'Special' features that influence a
: >benchmard (aka propietary features) are of *NO* use, because you're not
: >talking about inplementing everyting in a vendor-specific situation.
: >You're talking about apples vs. apples, DB vendor vs. DB vendor.
 

: Which means that a vendor specific feature that you would choose to use
: (or that is automatic, for that matter) *should* be measured. Fast/
: Group Commits were vendor specific, it's just that they're now specific
: to just about every vendor. The chances are that *your* particular
: database now goes faster because of that particular "benchmark special".

The question being, in your first sentence 'would' vs. 'could'. When I'm selecting a DB vendor, I want to see what happens in the BASE system. I can always turn some vendor-specific feature later for system performance testing. But, that will come later. In the beginning implementation phases, I want to know how the database performs on a dataset without having to specially setup the database -- e.g. put the data in and away it goes.

If something is automatic, great. I won't see it, I won't know about it -- I'll assume it's part of the base system, and it *WILL* influence my decision on which DB vendor to go with.

: >Because Oracle uses a 'feature' that is not something that would be useful
: >in a REAL WORLD situation (aka any real situation) to 'up' their TPC
: >benchmarks disqualifies them, in my opinion.
: Just because it's not useful in a particular real world situation does
: *NOT* mean that it is not useful in any real world situation.

Again, when I am doing comparisons of DB vendors, I look at their TPC numbers. They're supposed to reflect, in some meaningful way, how the database will perform on 'normal' databases.

Transactions without database locks is *NOT* part of a 'normal' database. On a single-user database, perhaps, but that is not what the TPC is measuring -- you won't get 300 transactions per second from a single user. (Well, if you do, you type *MUCH* *MUCH* faster than I do! ;-)

: > When TPC can do a real world test
: >(hopefully varied), I expect it to accurately reflect what I can do with
: >*MY* database.
: But I want it to reflect *MY* database, not *YOUR* database ... and
: there's the rub!

No, you misunderstand my context. I'm talking specifically about 'discrete transactions'. More than that, I don't care. I'm saying tha 'discrete transactions', as used by a DB vendor (specifically oracle) to measure TPC benchmarks is an invalid setup.

: > 'Discrete transactions' are not what I can do with my
: >database ... and expect the database to be intact at the end of the day.
 

: Why wouldn't you expect it to be intact at the end of the day? If it
: can do more than 60 million TPC-A transactions in a day and remain
: intact, why wouldn't it for you? It sounds as if you've misunderstood
: Discrete Transaction. It passes the ACID test, so where's the beef?

As I understand it (and correct me if I am wrong), discrete transactions require that each transaction is unique in the database. e.g. you are assured that there is not an update occuring on one record in the database by two transactions. This means that database locking (record and/or table) isn't necessary. This is what I contend is not an accurate test for a 'real world' database system. Generally, when going hundreds of transactions per second, that is doing updates AS WELL AS additions.

: >: The TPC does have a very strict definition of what the benchmark can and
: >: can not do. That definition exists at a fairly high level (essentially a
: >: detailed functional specification) so that any software system on any
: >: hardware platform can run the benchmark. So, it is possible using TPC
: >: benchmarks to compare a TPF2 system with the application implemented in
: >: IBM 370 assembly language accessing a private file structure with
: >: application implemented recovery mechanisms to a Compaq PC running SCO
: >: UNIX and ORACLE with the application written in C calling PL/SQL
: >: procedures. You can also use TPC benchmarks to evaluate the
: >: suitability of OODB products for TP applications, which you couldn't do
: >: if the TPC provided an ISO SQL program as the only valid benchmark
: >: program.
 

: >Again, real world vs. fantasy world.
: I'm confused again, you mean these are your fantasies???!

No, I feel that the flexibility in the system leaves gaping holes where DB vendors can drive through with outrageous claims of performance. My example is the TPC test using 'discrete transactions'. See above arguments as to why I consider 'discrete transactions' an invalid database testing usage parameter.

: >: 2) The real point, the one Standish made but more or less seems to be
: >: getting lost in the noise between them and Oracle, is that the TPC-A and
: >: TPC-B benchmarks are obsolete as comparators of database and transaction
: >: processing systems. When a benchmark is created all vendors will try to
: >: optimize for that benchmark. At some point the vendors will "crack" the
: >: benchmark by tuning their systems to the point that the benchmark (a) no
: >: longer demonstrates user-useful differences between products and/or (b)
: >: no longer can be extrapolated to predict the performance of customer
: >: applications. TPC-A and TPC-B have more or less reached this point.
: >: This doesn't make them bad benchmarks nor does it make vendors who bent
: >: over backwards to "win" these benchmarks unethical. TPC-A&B have caused
: >: about an order of magnitude improvement in simple-query pathlength,
: >: recovery algorithms, and I/O since 1987. Nearly every customer
: >: application had benefited dramatically from the TPC-A&B benchmark wars.
: >: 1993 pretty much marks the end of those customer benefits from TPC-A&B.
: >: The TPC-C benchmark not only is representative of a much broader set of
: >: applications then TPC-A&B, it poses a host of new problems demanding
: >: vendor attention and optimization. TPC-A&B should be retired as obsolete
: >: and both customers and vendors should focus on TPC-C (and the soon to
: >: appear TPC-D). TPC-C&D will provide more useful information to customers,
: >: and over the next 2-4 years will drive dramatic additional improvements
: >: in database system performance.
 

: >I have to agree,
 

: Great. Maybe we're *both* in the real world, after all!

Perhaps. One point should be noted: I don't feel that the TPC-A&B should be retired. The continued use of TPC-A&B (as well as newer generations of TPC benchmarks) will cause a more level measurement of database performance to be shown. By this, I mean that the vendor has to optimize for many benchmarks, hence, the database has a lot of flexibility. You're assured that the database has been optimized for many situations (as evidenced by the sheer number of TPC benchmarks it runs with). This should give the administrator confidence that the database will perform well in 'their' database application.

: > with one qualifier: If the TPC-C benchmark (and TPC-X
: >where X>C) *ACCURATELY* reflects what goes on on a typical DB machine, then
: >what you say has relevance. If it does not, however, it means that
: >databases are being optimized for functions that are not common.
 

: It doesn't mean that at all! Even if you were prescient enough to be
: able to design a benchmark which tested all the functions in common
: usage, you would find that the number of typical DB machines that it
: *ACCURATELY* reflected was < 1!
 

: A typical DB machine actually uses only a subset of the common
: functions, plus a subset of the less common functions. The problem is
: that each one (ie each schema design/application, not each occurrence/
: installation) uses a *different* pair of subsets.

I think that your second paragraph depicts what I was trying to say. I just feel that for database selection, you must look at those functions that are common to all database systems. Unusual features that may or may not be of use to you should be extraneous features when looking at benchmark tests.

: >I do feel that any TPC benchmark will test some functionality of a DB
: >system. I contend that the system must be demonstrably close to what
: >happens in the real world to be usable and/or believable.
 

: The closer it gets to what happens in any particular piece of the real
: world, the more specialised it becomes, and, as a direct result, the
: less useful it becomes (except to the inhabitant of that particular
: real-world location). Whether it is believable or not is an
: administrative issue, and has nothing to do with functionality.

My point above being that there are similar functions for every database in existence. To compare those databases for performance, there must be some common ground to test with. Discrete transactions, as explained to me, do not qualify as common, let alone 'common ground'.

There is a point, as you say, that the benchmark is no longer believable. That point is when the benchmark criteria has become too specific to a particular use to be usable as a generic benchmark figure. But, there must be some common usage of the database by a benchmark so that those benchmark figures may be used to compare database A vs. database B. The use of unusual features found only in that database by the benchmark (specifically, discrete transactions), I feel, invalidates the comparison of database A vs. database B.

: >: Hal Berenson
 

: graeme
: --
: Disclaimer: The author's opinions are his own, and not necessarily
: those of Pyramid Technology Ltd. or Pyramid Technology Inc.

Greg.

-- 
_______________________________________________________________________________
You can't go against nature, because when you do, 	Greg Shaw
go against nature, it's part of nature too.		shaw_at_feanor.xel.com 
			Love & Rockets			uunet!csn!xel.com!shaw