Re: Nearest Common Ancestor Report (XDb1's $1000 Challenge)

On 27 May 2004 16:04:30 -0700, Neo wrote:

Hi Neo,

>> I changed my model ... (SQL and output are down below).
>
>Yes I see. The changes are a step in the right direction to make an
>equivalent comparison.

No they are not. They are a step in the direction om improving my algorithm (which incidentally already was fast enough to meet the requirements of the challenge).

Making comparisons is not my intention. Please check the subject of this thread, especially the part in parentheses.

You concocted some data. You used your own tool to generate a report from that data. Then, you set a challenge to produce the same report from the same data with the relational model, with some requirements (normalized, NULL-less, generic, not more than 2X slower).

I took on the challenge. And though hierarchies is not what the relational model is best at, I managed to meet all requirements and even outperform XDb1. I posted my solution and I claimed my price.

What followed was an increasingly embarrasing display of you trying to change the rules, throwing in additional features that XDb1 undoubtedly has but that were not mentioned in your challenge, showing your lack of knowledge about normalization in the relational model, making claims about XDb1 that I could prove untrue by running one simple test and producing version after version of XDb1 to fix those bugs and to improve performance for generating the NCA report (which is nice to your customers but of no relevance at all to the challenge). (I notice you are now on v4.5.3; the version I downloaded is v4.4.7, downloaded May 20).

To reiterate: I am NOT making comparisons. I am claiming my $1000 reward.

>> The average execution time is 11.0 ms (as you can see from the output).
>> This is based on an execution on my desktop system (1.3 GHz Athlon,
>> 256 MB RAM, 2 harddisks at 40 GB / 7200 rpm each) with several open
>> applications. XDb1 takes 16 ms for generating the report on the same
>> machine with the same applications still active.
>
>There seems to be some puzzling differences in our time measurements.
>We may need additional measurements from other users. Below are
>measurements made on my hardware.
>
>Report Generation Time Summary
> (Apple-to-Orange Comparison)
>------------------------------
> Implementation Avg
> -------------- -------
> RM Solution #2 68.9 ms
> XDb1 v4.5.3 3.4 ms

Why are you using XDb1 v4.5.3 for the comparison? That's a version you made *after* setting the challenge. The challenge was set May 16. I downloaded XDb1 from your site on May 20 and got v4.4.7; the executable has a last change date of May 2. I assume *that* is the version your challenge is based on. Performance of later versions is irrelevant to the challenge.

>
>
>Details for the above summary follows:
>
>Data Set: Small hierarchy of 8 things (as published on web page).
>Platform: Off-line, 500 Mhz, Dell PowerEdge, Dual Pentium, 512 MB,
> Ultra SCSC II, Dual 10K RPM HDs, NT 4 sp6a,
> SQL Server 7 (default setup):

I already wrote you a lengthy message, explaing that case insensitive character comparisons (SQL Server's default) are much slower than case sensitive character comparisons (my setup, but also the setup that best mimicks XDb1's behaviour accurately). You never reacted to that. Yet, you still test using the default setup.

BTW, SQL Server is outdated. I am running SQL Server 2000, SP3.

Your challenge did not include the requirement that the relational version should be able to beat even future versions of XDb1 on historic versions of the RDBMS.

I executed and timed my relational implementation #2, using the version of SQL Server that was current at the time you set your challenge. It took 110 ms for 10 consecutive executions, 11 ms average per execution.

I also executed your implementation, using the version of XDb1 that was current at the time you set your challenge. The generated report claims it took 16 ms.

The requirement in the challenge is "Report generation must not be more than 2X slower than XDb1 on equivalent hardware". I tested report generation on THE SAME hardware. It was quicker than XDb1.

(snip list of execution times - irrelevant, since XDb1 version used is too new and SQL Server version used is too old and using wrong character comparison type)

>3. IMO, it is still an apple-to-orange comparision because of
>differences in db/schema genericness and normalization. I will list
>these in another post.

Whether apple-to-orange or not, I don't care. *You* asked for this comparison. *You* wanted this report produced using the relational model and *you* included a performance criterium.

Best, Hugo

-- 

(Remove _NO_ and _SPAM_ to get my e-mail address)