Re: Theory of Timeseries extensions to SQL and database

David wrote:
>
>What I am attempting to do is understand how databases like kdb, FAME and
>IBM's IDS with Timeseries DataBlade work. These all achieve perfomance at
>least an order of magnitude better than traditional relational databases for
>specific functional analysis queries on timeseries such as market TICs or
>historic day end pricing for financial instruments.

There's no mystery. They usually load the time series data into main memory and then apply special time series analysis algorithms. A lot of statistical stuff can be done in a stream oriented fashion, so you don't even have to keep the whole thing in main memory.

>[...] I know that kdb use the basic concept of storing columns on disk
>rather than relations and are moving this column data into memory to
>operate on.

Frankly that sounds a bit like marketing speak to me. Obviously reading the whole table into memory is going to be faster if it only contains the time series data you need, but you can get that arranged in any relational database. And if I read on their site that:

  "Most relational databases are row-based, with every row stored randomly   on disk. Reading one attribute requires reading an entire record, and   scanning a column of values requires reading the entire table."

then I can only conclude that the person who wrote that doesn't know how relational databases work.

• Jan Hidders