Re: Benchmarks was Re: Which one is better? Oracel 9i or DB2 7.2??

Serge Rielau wrote:

> > There is not a chance I am going to find a TCP report of any value in 90%
> > of these projects. To me they are just another form of marketing hype.
>
> I don't thiok they are mere marketing hype. Yes, they are used for marketing,
> but the vendors also constantly calibrate their products against them and the
> benchmarks do comprise some "real world like" queries.
>
> Lets' take a look at materialized views (or AST, indexed views...). Oracle and
> DB2 broke the TPC-D
> benchmark with those. Were they a benchmark special? No, even Microsoft has
> introduced them despite not playing in TPC-D (and now TPC-R and -H).
> Also note that often the availability of the system is 6 months after the test
> because the test uses bleeding edge technology.
> I would call TPC a sandbox - simplified? yes - but also relevant.
> Participating in the game means a vendor is serious about this kind of
> workloads. Who is on top at any given moment is not really the point as long
> as they are at the top every so often.
>
> Just my 2 cents
> Serge
> --
> Serge Rielau
> DB2 UDB SQL Compiler Development
> IBM Software Lab, Canada

I can not agree. I think the vendors tweak their products to do well on the test. Do the tests emulate systems I build? Unlikely. So if the system being built happens to be an exact simulation of the test it might have some relevance. Otherwise the test mean little or nothing. And I'm not sure what you mean by "serious about this kind of workload." What the vendors are serious about is marketshare and profits. They are, after all, for-profit businesses. And if they couldn't use these benchmarks for hype their products the benchmarks wouldn't exist.

When there is a hacking benchmark where the databases are made available to a bunch of 16-22 year olds and there is a security rating I will be interested. When the databases are hammered on for months on end until the last one breaks and there is a stability benchmark I'll be interested. When they keep adding users doing simultaneous inserts/updates/deletes/selects on the same records and the same tables and there is a scalability benchmark I'll be interested.

That some query takes 0.0214 seconds vs 0.0235 seconds is not enough to make me look up from my latte'.

Daniel Morgan Received on Wed May 08 2002 - 09:54:23 CDT