Benchmarks was Re: Which one is better? Oracel 9i or DB2 7.2??

"Pablo Sanchez" <pablo_at_dev.null> wrote in message news:T3CB8.72$rR4.76730_at_news.uswest.net...
> I think you may have misunderstood my point or perhaps I wasn't clear.
> Probably the latter. Benchmarks are very time consuming and resource
> intensive, hence most shops cannot bench their application and which
> is the reason for the TPC benchmarks.

I understood that.
>
> What I was trying to state was if you want to compare vendor to vendor
> where an industry-wide benchmark is to be used, where each vendor (hw
> and sw) are going to pour their little hearts into the benchmark, seek
> the TPC. You'll get an audited, fully disclosed benchmark. The
> benchmarks themselves are created by all the vendors: hardware and
> software.

What I am saying though is that the systems that get put through TPC-C are high end systems that in themselves are unlikely to be (I would say cannot be) representative of many shops hardware and software configurations. We certainly don't have 3/4 million to spend on the hardware and software costs of our database systems. If I did want to compare across systems at that level , and I were convinced of the usefulness of benchmarking, then TPC would be a good place to look. Unfortunately neither of those two conditions is true.

The first is obviously just a function of organisation and system size.

The second however springs from what I believe to be a (common) error in creating benchmarks. The argument that shops cannot afford the time and resource to bvench their apps on a variety of hardware and software is fine. However I fail to see that it follows that a third party benchmarking what it considers to be code representative of applications in general is an equivalent replacement.If this argument holds then it would follow that faced with a choice between say db2 and oracle I could pick the hardware that most closely matched mine and pick the fastest/best tpc/$ result and know that the same would apply to my app. In fact however if I rely on the benchmark to show me which performs best I may well pick the wrong combination due to the features peculiar to my app. There is no way for an independent benchmark to avoid this. If however a benchmark cannot tell me how my apps will perform then what task can it do other than provide apparent stats for the vendors to use in marketing?

Of course if I want to run TPC-C (or H or whatever) on my setup then TPC is a useful site.

--
Niall Litchfield
Oracle DBA
Audit Commission UK
*****************************************
Please include version and platform
and SQL where applicable
It makes life easier and increases the
likelihood of a good answer

******************************************