Re: Benchmarks was Re: Which one is better? Oracel 9i or DB2 7.2??

Sailesh Krishnamurthy wrote:

> >>>>> "Daniel" == Daniel Morgan <dmorgan_at_exesolutions.com> writes:
>
> Daniel> Serge Rielau wrote:
> >> > There is not a chance I am going to find a TCP report of any
> >> value in 90% > of these projects. To me they are just another
> >> form of marketing hype.
> >>
> >> I don't thiok they are mere marketing hype. Yes, they are used
> >> for marketing, but the vendors also constantly calibrate their
> >> products against them and the benchmarks do comprise some "real
> >> world like" queries.
>
> Daniel> I can not agree. I think the vendors tweak their products
> Daniel> to do well on the test. Do the tests emulate systems I
> Daniel> build? Unlikely. So if the system being built happens to
> Daniel> be an exact simulation of the test it might have some
> Daniel> relevance. Otherwise the test mean little or nothing. And
>
> I do agree that the benchmark may not quite emulate the system that
> you specifically are trying to build. Just as SPEC-fp may not emulate
> a web client workload.
>
> However, I don't think you quite understand the sheer effort that goes
> into publishing an audited benchmark. If you really believe that
> vendors just "special case" their benchmarks with an "if" statement to
> say "ha .. this is TPC-C .. do this", then there is no point
> continuing the discussion. I think that kind of behaviour is ethically
> wrong, and I would doubt if any vendor will engage in quite that kind
> of chicanery.
>
> The thing is, vendors assemble special benchmark teams comprising of
> people who have actually designed, architected and implemented
> significant parts of the engine. Ergo, they really understand the
> product and can tune it very well - way beyond the ability of the
> average user. This is what makes a benchmark unrealistic.
>
> The value comes because of the incredible amount of learning that the
> benchmark team gains about the product. As Serge pointed out, several
> times the benchmark reveals a key feature that needs to be implemented
> - 99% of the time the team will implement the feature .. unless a hack
> will do. Often times there will be bugs that show up in that
> configuration that get fixed.
>
> The bottomline is that the benchmark proves that the system is capable
> of humming at a certain level. There is a measure of confidence this
> gives for a customer.
>
> If on the other hand you really believe that vendors are ethically
> bankrupt (some may be :-) then we can agree to disagree :-)
>
> --
> Pip-pip
> Sailesh
> http://www.cs.berkeley.edu/~sailesh

I'm not saying that the benchmarks don't stimulate vendors to improve their products. I'm not saying that they don't identify product weaknesses. But that is a far cry from making a decision to purchase Oracle versus UDB or Sybase vs Informix based on some synthetic horse race.

Daniel Morgan Received on Wed May 08 2002 - 11:26:28 CDT