Re: Benchmarks was Re: Which one is better? Oracel 9i or DB2 7.2??

>>>>> "Daniel" == Daniel Morgan <dmorgan_at_exesolutions.com> writes:

    Daniel> Serge Rielau wrote:
>> > There is not a chance I am going to find a TCP report of any
>> value in 90% > of these projects. To me they are just another
>> form of marketing hype.
>>
>> I don't thiok they are mere marketing hype. Yes, they are used
>> for marketing, but the vendors also constantly calibrate their
>> products against them and the benchmarks do comprise some "real
>> world like" queries.

    Daniel> I can not agree. I think the vendors tweak their products
    Daniel> to do well on the test.  Do the tests emulate systems I
    Daniel> build? Unlikely. So if the system being built happens to
    Daniel> be an exact simulation of the test it might have some
    Daniel> relevance.  Otherwise the test mean little or nothing. And

I do agree that the benchmark may not quite emulate the system that you specifically are trying to build. Just as SPEC-fp may not emulate a web client workload.

However, I don't think you quite understand the sheer effort that goes into publishing an audited benchmark. If you really believe that vendors just "special case" their benchmarks with an "if" statement to say "ha .. this is TPC-C .. do this", then there is no point continuing the discussion. I think that kind of behaviour is ethically wrong, and I would doubt if any vendor will engage in quite that kind of chicanery.

The thing is, vendors assemble special benchmark teams comprising of people who have actually designed, architected and implemented significant parts of the engine. Ergo, they really understand the product and can tune it very well - way beyond the ability of the average user. This is what makes a benchmark unrealistic.

The value comes because of the incredible amount of learning that the benchmark team gains about the product. As Serge pointed out, several times the benchmark reveals a key feature that needs to be implemented - 99% of the time the team will implement the feature .. unless a hack will do. Often times there will be bugs that show up in that configuration that get fixed.

The bottomline is that the benchmark proves that the system is capable of humming at a certain level. There is a measure of confidence this gives for a customer.

If on the other hand you really believe that vendors are ethically bankrupt (some may be :-) then we can agree to disagree :-)

-- 
Pip-pip
Sailesh
http://www.cs.berkeley.edu/~sailesh