Re: lies damn lies and benchmarks

Mike Ault <mikerault_at_earthlink.net> wrote:
>
>Most of Microsoft's TPCC numbers are meaningless since they represent
>a system no one in their correct mind would build and try to maintain,
>come on now, a 32 node 4 cpu/node federated database? Who are they
>kidding. Don't forget cost to maintain those 32 servers and their
>associated disk farms as well as the time required for the additional
>programming support to maintain all the 2PC logic not to mention
>rebuilding the whole thing when adding or removing nodes and any other
>number of maintenance tasks. Cheaper is only cheaper if it is reliable
>and requires less maintenance.

Do you seriously care about this, or is it just "my team is the best" and comparing batting averages? I do care, much of my job is getting the best possibly performance out of Oracle running on top end Unix systems. This has made me aware of how much more important the mid-range is than the top end; and why it's more concerning that Microsoft are doing well there than at the top end.

Did you actually look at tpc.org before making these claims? Of the 44 published SQL Server results for TPC-C 5.0, I count 7 clustered results. This is hardly "most".

I don't care about federated databases, ignore them, use the tpc.org option to filter out clusters. If you'd bothered to look at the results I was comparing, you would have seen that they were each a single SMP system with 4 CPUs. Very typical of a low to mid-range system.

>When comparing costs, also compare features. For the same feature set
>Oracle is comparable in price (standard Oracle is the same as SQL2000,
>not enterprise Oracle as most comparisons from MS land try to fob off
>on you.) I suggest looking at the Database shootout done by PC
>Magazine (which is notorious for being in the Microsoft camp) who did
>they choose as their editors choice? Oracle! the same article has a
>detailed feature comparison spread sheet for all of the major
>databases (as well as the public domains such as MYSQL.)

OK, first, I accept that SQL Server costs are comparable to Oracle. Most of the high costs of the Oracle/Unix solution are in the hardware.

I'd trust a PC magazine to evaluate enterprise databases about as much as I'd trust a model aircraft magazine to evaluate the purchase requirements for a space shuttle.

I'm no Microsoft fan, far from it. However simply it annoys me when Microsoft is dismissed without due consideration. That way just leaves the door open for them. To oppose Microsoft, one must recognise their strengths. In this case they're providing a competent RDBMS (if inferior to Oracle), with an feature set that's certainly acceptable to many end users. Microsoft are not stupid, they are not totally incompetent.

MySQL isn't public domain, it's GPL'd. I'm sure MySQL AB would be keen to assert their copyright on it.

>My ratio is as good as any other metric in that it tries to sift
>through the haze generated by such artificial constructs as a 32 node
>federated database. I would much rather have Oracle's TPCC performance
>on a single, maintainable 64 CPU box going against an industry
>standard SAN or disk farm than an equivilent of Microsoft's on 16-4
>cpu windows boxes each with their own disk array using some god awful
>federated architecture. Even assuming a multiplicaiton factor that is
>directly proportional to the number of nodes this means for a 64 CPU
>setup (16-4 cpu nodes) Microsoft would only achieve a TPCC of around
>206,000, 50% less than Oracle's.

Strange how SQL Server achieved a TPCC of 165218.71 on a single, SMP 32 CPU system. I fully accept that SQL Server doesn't scale as high as Oracle. Windows only supports up to 32 CPUs for a start. However, the proportion of 32 or 64 CPU systems sold is much lower than the mid-range 4 to 8 CPU systems, which is the ground Microsoft is attacking very effectively.

I recently spent five weeks running an benchmark of an Oracle based application on a 64 CPU system. If I had to spec a system of that level, SQL Server wouldn't even be a consideration. However, if I could save my company $200,000 by speccing a 4 CPU NT box with SQL Server, rather than a 4 CPU Unix box with Oracle, I'd consider that carefully.

-- 
Andrew Mobbs - http://www.chiark.greenend.org.uk/~andrewm/