Re: Lies, Damned Lies, Statistics, Marketing Departments

After attending the Windows 2000 DNA Readiness Conference in Denver and hearing Lon Fisher speak on the performance optimizations employed in some of the Microsoft benchmarks, I downloaded the TPC-C FDR's for SQL Server and the best Oracle results.

Looking over the code it is apparent that it was not apples- -apples. The Microsoft implementation relied on buffering "inserts" - the data was added into an array and posted later. There were a number of other hacks that could fool an end user (or auditor); you'd have to review the source at length in order to understand the test observations.

Larry Ellison's comments did not go far enough with regard to the Microsoft implementation not being "real world". Barring the 10 discrete databases, the underlying implementation of the code by Microsoft would make David Copperfield look like an amateur.

With a little experience coding database applications and the ability to read C, it should take very little time to see beyond the "smoke and mirrors". I also encourage you to look at the comparative Oracle FDR's - the differences are eye opening.

My gripe with Microsoft is that some of their technology does work, but the marketing departments publish one thing while the developers have implemented a solution using completely different technologies. I have seen a number of interviews where Microsoft spokes-persons touted the use of COM+ to manage transactions in benchmarks (the TPC-C, for example) where the implementation does not use COM for database interaction at all. (Check out the source code to the ISAPI DLL used in the benchmark.)

--
Michael D. Long
http://extremedna.homestead.com


"Tom Stearns" <tom_at_stearns.com> wrote in message
news:DCw75.785$XU2.360276_at_news.uswest.net...


> Gee David, did you wake up on the wrong side of the bed today? It is not

> surprising to me that "benchmarks" set up by organizations like TPC are

> bogus, is it to you? Wanna bet Oracle plays the same game? I suggest you

> replace that chip on your shoulder with a life. Pick your server and go to

> work.

>

> Tom Stearns

>

> "David Pomphrey - DNP" <High.Flight_at_btinternet.com> wrote in message

> news:8jlh76$bs8$1_at_plutonium.btinternet.com...

> > Two Major MS SQL Server TPC-C benchmarks are withdrawn from the TPC.

> > ###########################################################

> >

> > http://www.tpc.org/new_result/c-withdrawn-results.idc

> >

> >

> >

> > Why do withdrawls often happen?

> > ###########################################################

> >

> > Extract from the TPC Faq   (  http://www.tpc.org/faq_TPCC.html  )

> >

> > "Q: I notice that some TPC results are labelled "withdrawn." Could you

> > explain what that means?

> >

> > A: The TPC felt that users should be made aware of what results drop

> > from the TPC's official results list and why those results no longer

> > appear. Some vendors withdraw results because they feel these results no

> > longer have market relevance. Other vendors withdraw results after

> > compliance to the benchmark specification has been challenged by someone

> > within the TPC. Rather than defend their implementation (and perhaps

> > expend further resources to demonstrate compliance), the vendor chooses

> > to withdraw the result. Finally, if the Council votes that a result is

> > non-compliant, the Council will drop the r esult from the official

> > results list."

> >

> >

> >

> > Microsoft makes NO mention of this on their website.

> > ##########################################################

> >

> > http://www.microsoft.com/sql/productinfo/tpc.htm   -  1st July 2000,

> > 1921hrs G.M.T (UTC)

> >

> >

> >

> > Oracle's CEO declares the latest MS SQL Server 'PREPOSTEROUS'.

> > ##########################################################

> >

> >

> > http://www.nasdaq.com/reference/broadcast_oracle.htm

> > (you'll need real player -

> > http://www.broadcast.com/redirects/realplayer.html)

> >

> >

> > Near the 1 hour mark, an analyst from Paine Webber asked a question

> > about Microsoft SQL Server 2000. The following is Larry Ellison's

> > response:

> >

> > "In terms of microsoft.. we have no concerns at all. They still can't

> > scale. They have this benchmark that they got out which works only in

> > the laboratory.

> >

> > The only problem with microsoft's benchmark is that it has a 3-hour mean

> > time of failure. What they have done is to chop up  the database in to

> > 10 separate little databases, and if any one  of those databases fail it

> > brings down the entire system, or worse yet gives wrong results.

> >

> > So it is a completely bogus benchmark.

> >

> > I mean, it meets the letter of the benchmark rules, however by their own

> > statistics in terms of availability they have a very very short mean

> > time of failure.

> >

> > No one seriously will ever use this kind of system.

> >

> > They have 10 separate computers each with 10% of the database.

> > If you want an 11th computer you have to unload the entire database from

> > the 10 computers and then put 9.1% of the database on the 11 computers.

> > If one of the computers fail you lose 10% of the database. And that

> > means when you use your query.. you don't get the right answer back.

> >

> > If you use 10 separate systems.. if you believe Microsoft's statistics

> > on failure rates.. one failure every 30 days, you are going to get a

> > major system outage or wrong results every 3 days.

> >

> > It is a preposterous benchmark."

> >

> >

> >

> > MS SQL may be cheaper but Oracle has the highest Performance in the

> > TCP-C benchmark.

> > #########################################################

> >

> > Oracle can scale  :

> >

> > http://www.tpc.org/new_result/ttperf.idc

> >

> >

> > Microsoft can set cheap prices  :

> >

> > http://www.tpc.org/new_result/ttpp.idc

> >

> >

> >

> >

> > A monopoly company can set the price for a product any which way it

> > chooses - this is easy.

> >

> >

> > But can it make a product that truly scales?  - THAT is the question.

> > THAT takes technology.

> >

> >

> >

> >

> >

>

> --------------------------------------------------------------------------

>

>