Oracle FAQ | Your Portal to the Oracle Knowledge Grid |
Home -> Community -> Usenet -> c.d.o.server -> Re: Benchmarks was Re: Which one is better? Oracel 9i or DB2 7.2??
But of course in the case of TPC-D, queries had fantastic improvements in response
times in each iteration (from dozens of minutes to a few seconds). And, the features
that accomplished are used by lots of customers with non TPC-D like workloads. Add to
that the fact that one of the TPC-D queries would cause a particular RDBMS to crash
when the workload was first published in 1995, and I think we have a useful workload.
If these benchmarks were used only for hype, you might have a point. But TPC-D, TPC-C,
TPC-H and TPC-W are all run internally by RDBMS vendors to gauge their performance
quality.
Daniel Morgan wrote:
> Serge Rielau wrote:
>
> > > There is not a chance I am going to find a TCP report of any value in 90%
> > > of these projects. To me they are just another form of marketing hype.
> >
> > I don't thiok they are mere marketing hype. Yes, they are used for marketing,
> > but the vendors also constantly calibrate their products against them and the
> > benchmarks do comprise some "real world like" queries.
> >
> > Lets' take a look at materialized views (or AST, indexed views...). Oracle and
> > DB2 broke the TPC-D
> > benchmark with those. Were they a benchmark special? No, even Microsoft has
> > introduced them despite not playing in TPC-D (and now TPC-R and -H).
> > Also note that often the availability of the system is 6 months after the test
> > because the test uses bleeding edge technology.
> > I would call TPC a sandbox - simplified? yes - but also relevant.
> > Participating in the game means a vendor is serious about this kind of
> > workloads. Who is on top at any given moment is not really the point as long
> > as they are at the top every so often.
> >
> > Just my 2 cents
> > Serge
> > --
> > Serge Rielau
> > DB2 UDB SQL Compiler Development
> > IBM Software Lab, Canada
>
> I can not agree. I think the vendors tweak their products to do well on the test.
> Do the tests emulate systems I build? Unlikely. So if the system being built
> happens to be an exact simulation of the test it might have some relevance.
> Otherwise the test mean little or nothing. And I'm not sure what you mean by
> "serious about this kind of workload." What the vendors are serious about is
> marketshare and profits. They are, after all, for-profit businesses. And if they
> couldn't use these benchmarks for hype their products the benchmarks wouldn't
> exist.
>
> When there is a hacking benchmark where the databases are made available to a
> bunch of 16-22 year olds and there is a security rating I will be interested. When
> the databases are hammered on for months on end until the last one breaks and
> there is a stability benchmark I'll be interested. When they keep adding users
> doing simultaneous inserts/updates/deletes/selects on the same records and the
> same tables and there is a scalability benchmark I'll be interested.
>
> That some query takes 0.0214 seconds vs 0.0235 seconds is not enough to make me
> look up from my latte'.
>
> Daniel Morgan
Received on Wed May 08 2002 - 10:21:58 CDT