Oracle FAQ Your Portal to the Oracle Knowledge Grid
HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US
 

Home -> Community -> Usenet -> c.d.o.server -> Re: Benchmarks was Re: Which one is better? Oracel 9i or DB2 7.2??

Re: Benchmarks was Re: Which one is better? Oracel 9i or DB2 7.2??

From: Daniel Morgan <dmorgan_at_exesolutions.com>
Date: Wed, 08 May 2002 16:28:16 GMT
Message-ID: <3CD95217.3FBCE441@exesolutions.com>


Obnoxio The Clown wrote:

> On Wed, 08 May 2002 14:54:23 GMT, Daniel Morgan
> <dmorgan_at_exesolutions.com> wrote:
>
> >Serge Rielau wrote:
> >
> >> > There is not a chance I am going to find a TCP report of any value in 90%
> >> > of these projects. To me they are just another form of marketing hype.
> >>
> >> I don't thiok they are mere marketing hype. Yes, they are used for marketing,
> >> but the vendors also constantly calibrate their products against them and the
> >> benchmarks do comprise some "real world like" queries.
> >>
> >> Lets' take a look at materialized views (or AST, indexed views...). Oracle and
> >> DB2 broke the TPC-D
> >> benchmark with those. Were they a benchmark special? No, even Microsoft has
> >> introduced them despite not playing in TPC-D (and now TPC-R and -H).
> >> Also note that often the availability of the system is 6 months after the test
> >> because the test uses bleeding edge technology.
> >> I would call TPC a sandbox - simplified? yes - but also relevant.
> >> Participating in the game means a vendor is serious about this kind of
> >> workloads. Who is on top at any given moment is not really the point as long
> >> as they are at the top every so often.
> >>
> >I can not agree. I think the vendors tweak their products to do well on the test.
> >Do the tests emulate systems I build? Unlikely. So if the system being built
> >happens to be an exact simulation of the test it might have some relevance.
> >Otherwise the test mean little or nothing. And I'm not sure what you mean by
> >"serious about this kind of workload." What the vendors are serious about is
> >marketshare and profits. They are, after all, for-profit businesses. And if they
> >couldn't use these benchmarks for hype their products the benchmarks wouldn't
> >exist.
> >
> >When there is a hacking benchmark where the databases are made available to a
> >bunch of 16-22 year olds and there is a security rating I will be interested. When
> >the databases are hammered on for months on end until the last one breaks and
> >there is a stability benchmark I'll be interested. When they keep adding users
> >doing simultaneous inserts/updates/deletes/selects on the same records and the
> >same tables and there is a scalability benchmark I'll be interested.
> >
> >That some query takes 0.0214 seconds vs 0.0235 seconds is not enough to make me
> >look up from my latte'.
>
> I beg to differ. The issue is not one of 0.0214 seconds vs 0.0235
> seconds. The issue is pushing the envelope and making stuff break.
> Running arcane and unrealistic benchmarks looking for numbers that a
> handful of sites around the world actually need makes a better product
> for us all. These benchmarks stress test the products and provoke the
> vendors into adding features that can make the products more useful to
> us lesser mortals (as well as some features that just make the
> benchmarks run well! :-)
>
> Also, while a TPC benchmark is meaningless as an indicator of absolute
> performance, the fact that a vendor is playing in the top ten means
> that performance in the area that the benchmarks cover will generally
> be acceptable and you can focus on other differentiators, unless
> perfomance is a particularly key criterion. (This is for those of us
> playing at the lower end of the market who don't have massive
> benchmarking budgets or our own R&D departments.)
>
> I would never advocate buying product X, just because it's currently
> no 1 in the TPC-C. But I would always say that X takes performance
> seriously if they play the game.

To not play the game dooms a vendor to lower marketshare. Of course they play the game.

Daniel Morgan Received on Wed May 08 2002 - 11:28:16 CDT

Original text of this message

HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US