Re: Realtime Bulk RDBMS Inserts?

We can get around 1000 bulk-inserts per second (50 bytes/row) using Sybase's Bulk Library on relatively low-end hardware (Sun Ultra 2 into RAID 5 SCSI3) in near-realtime (we do 4-second batches). The main issue is that there is an exclusive lock on the table for the majority of that time.

To get speed beyond 1000rows/sec, I think you would have to:

1. use partitioned tables such that you can parallelize bulk-inserts across your key space
2. do the bulk-insert into one table and do your analytical queries against a "mirror" table that is inserted transactionally every N seconds, such that the mirror table is minimally locked.
3. use bigger, faster hardware.

In article <8ef0eq$t6l$1_at_nnrp1.deja.com>,   Offshore Solutions <projects_at_bluebaysoft.com> wrote:
> Looking for any white papers, discussions, links, topologies or
> descriptions of solutions that are capable of handling large-scale
> real-time insertion of data into RDBMS. Granularity would be
 sub-second
> and capable of handling 10,000 to 15,000 rows a second in total,
> round-robined from several different feeds. Cacheing is obviously
> required. Any constructive thoughts appreciated!

Sent via Deja.com http://www.deja.com/
Before you buy. Received on Mon May 01 2000 - 00:00:00 CEST