Re: Data Warehouses

From: Karsten Schmidt <groups_at_karsten-schmidt.com>
Date: 20 Mar 2003 00:12:38 -0800
Message-ID: <c6711ac4.0303200012.2da4f915@posting.google.com>

"Joseph Wilson" <me_at_home.now> wrote in message news:<LU9ea.1636$2x2.662043_at_dca1-nnrp1.news.algx.net>...
> Does anyone have experience with large volume data warehouse? What kind of
> volumes have you seen that are feasible. The company I work for is looking
> to start a data warehouse project with data that will be coming off of the
> mainframe at about 1 million records a day at about 4k to 9k bytes per
> record(4 to 9 GB per day). They want to store this in oracle for up to a
> year to be able to trend and aggregate the data. Has anyone seen a data
> warehouse implementation this size? Any tips or recommendations?
>
> Thanks,
> Joe

Hi Joe,
this is certainly possible - I am working on a database that loads about 20 million records per day, much smaller though - just a couple of hundred bytes each. The hardware is moderate - 4 CPU HPUX box, actually shared by a few more Oracle instances.

The thing I found is that pretty much all the report generators I've seen fail miserably when confronted with a billion- row table. We resorted to hand-written - and tuned sql.

And you are most certainly looking into partitioning, and sql loader with direct path option. Deleting millions of rows is going to kill you. Drop partition is way easier.

Regards Karsten Received on Thu Mar 20 2003 - 02:12:38 CST