Re: Why bother with a database if ...

In article <_7Bx5.219701$i5.3017214_at_news1.frmt1.sfba.home.com>,

        "Michael J. Moore" <hicamel_at_home.com> writes:

> if for example you have a large number of records that you just need to
> store once and then read them sequentially once. We collect sessions from a
> network, aprox a million per day. These session records are the input to a
> process which looks up additional information based on the content of the
> session record. There does not seem to be any reason to dump these session
> records into a database table just so that we can sequentially processing
> them through the system. Or is there?
>
> The only possible reason I could come up with is maybe, if we had multiple
> CPU's and the table was spread over many disks, that a parallel query might
> actually be faster than a sequential read of a flat file.

[snip]

The benefits of using multiple CPUs and spreading data across disk mechanisms are different. Multiple CPUs will only deliver significant performance benefits if there is more than one reasonably CPU-intensive process or thread to make use of their cycles. Multiple disk mechanisms can allow processes running in parallel (whether time-sliced or on multiple CPUs) to make use of CPU cycles where other processes are waiting for the disk mechanism.

Reasonably active multi-user DBMSs obviously benefit from both but would your program?

Spreading data across disks is not in itself a good reason for choosing a database; you can spread data using an appropriate level of RAID or at the application level by dividing the sequential file into multiple files and spreading these across disks. You would need a multi-threaded application so that some threads work while other threads are waiting. The feasibility of this depends on whether you can access records out of order.

You say you look up additional information based on the session record? If this 'additional information' is held in a database, that would be a better reason for using a database, which is likely to be more efficient at joining to that data than a simple program?

-- 
Regards,

Richard