Re: Database or store to handle 30 Mb/sec and 40,000 inserts/sec

From: paul c <toledobythesea_at_oohay.ac>
Date: Wed, 08 Feb 2006 16:42:38 GMT
Message-ID: <2cpGf.465508$2k.219953@pd7tw1no>

David Cressey wrote:
> "paul c" <toledobythesea_at_oohay.ac> wrote in message
> news:Kk3Gf.580621$ki.6168_at_pd7tw2no...
>

>>uspensky_at_gmail.com wrote:

>
>

>>>Average data rate - 30 megabits/sec.
>>>Max # inserts - 40,000 records/sec.

>
>

>>are you sure those rates are right?  (i find it hard to believe that a
>>single option exchange, let alone six is doing 40,000/6*3,600 =
>>something like 21 million inserts in an hour!)

>
>
> If an infinite number of monkeys were clicking on an infinite number of
> hyperlinks, somebody would be trying to capture all the clicks in a
> database. The internet isn't quite there yet, but we're close!
>
>
>

Okay, I think I'm starting to get it - part of the angle for handling such systems is that much of the stream is redundant. If the NYSE trades a billion shares a day, the day might max-out with only 10 million trades and probably many fewer. If quotes for 1000 issues arrive every second, a five-hour trading day would produce less than 20 million quotes, the majority of them differing only in their time stamps.

pc Received on Wed Feb 08 2006 - 10:42:38 CST