Re: Very Large DB

Doug Barnes wrote in message <3519046D.F85_at_rell.NOSPAM.com>...
>Jason D. Weil wrote:
>>
>> I am leading a development effort that will result in a multi-billion
>> row database, on the order of 400GB. About 80% of this database could
>> properly reside in a single partitioned table, each row being fairly
>> small. There is some contention within our organization as to whether to
>> implement this as a completely relational database (using Oracle 8), or
>> to implement the largest table as flat files and use Oracle as a front
>> end for accessing these data. Does anyone have experience with this type
>> of problem? Have you had success with either of these approaches, or is
>> there another, better, candidate?
>
>Sure would hate to do a full table scan on that thing.

Disagree. This is exactly why you need full table scans to process large volumes of data. Using index range scans for large volumes (rule of thumb according to Oracle is between 10%/15% and higher of all data in a VLT) is basically i/o suicide as it is simply impossible for a single process (taking into consideration i/o latency) to process such large volumes of data in a realistic time frame. With parallel server and parallel query, 10's if not 100's of processes can be created, each processing a chunk of the table data using a full table scan. I recall a German OPS expert telling me that they have an Oracle site running a MPP platform that processes over a billion rows in a couple of hours (for data warehousing).

I will also not recommend putting such large amounts of data into flatfile(s). To achieve the same level of advance i/o processing that Oracle (and other database systems) employ is more than just a few manmonths (if not manyears) of development. IMHO it is naive to believe that a normal application developer can put an application together that is going to rival or exceed the i/o performance of Oracle when it comes to processing such large volumes of data.

My 2'c worth. :-)

regaqrds,
Billy Received on Thu Mar 26 1998 - 00:00:00 CET