Re: Datawarehouse Load Performance?

I did one like this recently on a slightly lower powered box - The prognosis before I started was 3 weeks for 63M rows, with no option for restarting.

It took a couple of days work to restructure it to complete in 27 hours, with the ability to restart self-consistently after a crash.

The main problem is more likely to be I/O than actually 'arithmetical' processing.

--

Jonathan Lewis
Yet another Oracle-related web site: http://www.jlcomp.demon.co.uk

Stephen Darlington wrote in message <87or5t$av3_at_romeo.logica.co.uk>...
>That'll teach me to rush writing a question... Thanks for your responses
one
>and all, but I think I wrote the wrong question!
>
>What we have are 1.2 million rows in the main table in the source
>(normalised) database, I suspect it's going to turn out to be four or five
>times that in the fact table. The loading, in this case, means a lot of
>PL/SQL crunching through it all doing some fairly complex stuff (tree
>traversing, date ranges). Even if it were an option, and it isn't, I don't
>think coding it in Pro*C would help since it's all data manipulation rather
>than arithmetic.
>
>I guess this most closely mirrors "Transforming into load record images"
and
>"Migrating from the legacy system to DDW system" (pg 217, The Datawarehouse
>Toolkit, Kimball). I gather that the processing we're doing is unusually
>complex. We're not able to do any transformations on the source system, in
>fact one of the criteria is that we impact them as little as possible.
>
>The main question is what level of performance can I expect in the
>transformation? Are my figures way off? But the other comments have already
>been useful.
>
>Thanks again,
>--> Stephen
>---------------------------------------------------------------
> Stephen Darlington (http://www.zx81.org.uk)
> "Never put a sock in a toaster"
>---------------------------------------------------------------
>Stephen Darlington wrote in message <87m7jn$6b7_at_romeo.logica.co.uk>...
>>I'm just starting to design a datawarehouse load and was beginning to get
a
>>little worried about the performance.
>>
>>What are other people's experience loading a datawarehouse? We're using 8i
>>on a 4 processor (Xeon, 550Mhz) NT box and are looking to process around
>1.2
>>million rows. Some very rough metrics indicated that this could take a
week
>>to load!
>>
>>I appreciate that there are too many variables here to get a very precice
>>answer, but I'd like to hear any experiences you have.
>>
>>Cheers,
>>--> Stephen
>>---------------------------------------------------------------
>> Stephen Darlington (http://www.zx81.org.uk)
>> "Never put a sock in a toaster"
>>---------------------------------------------------------------
>>
>>
>
>
Received on Tue Feb 08 2000 - 12:23:04 CST