Path: news.easynews.com!newsfeed1.easynews.com!easynews.com!easynews!newsfeed.news2me.com!newsfeed.icl.net!newsfeed.fjserv.net!syros.belnet.be!naxos.belnet.be!news.belnet.be!news.uia.ac.be!not-for-mail
From: hidders@REMOVE.THIS.uia.ua.ac.be (Jan Hidders)
Newsgroups: comp.databases.theory
Subject: Re: Theory of Timeseries extensions to SQL and database
Followup-To: comp.databases.theory
Date: 26 Oct 2002 17:05:56 +0200
Organization: University of Antwerp
Lines: 30
Message-ID: <3dbaaf54$1@news.uia.ac.be>
References: <apbre3$68e$1@venus.btinternet.com> <3db991b7$1@news.uia.ac.be> <apdqmc$j22$1@knossos.btinternet.com>
Reply-To: hidders@uia.ua.ac.be (Jan Hidders)
NNTP-Posting-Host: hnets.uia.ac.be
X-Trace: naxos.belnet.be 1035644758 5410 143.169.254.1 (26 Oct 2002 15:05:58 GMT)
X-Complaints-To: abuse@belnet.be
NNTP-Posting-Date: Sat, 26 Oct 2002 15:05:58 +0000 (UTC)
X-Newsreader: trn 4.0-test76 (Apr 2, 2001)
Originator: hidders@hcoss.uia.ac.be (Jan Hidders)
X-Original-NNTP-Posting-Host: hmacs.uia.ac.be
X-Original-Trace: 26 Oct 2002 17:05:56 +0200, hmacs.uia.ac.be
Xref: newsfeed1.easynews.com comp.databases.theory:23158
X-Received-Date: Sat, 26 Oct 2002 11:50:26 MST (news.easynews.com)

David wrote:
>
>What I am attempting to do is understand how databases like kdb, FAME and
>IBM's IDS with Timeseries DataBlade work. These all achieve perfomance at
>least an order of magnitude better than traditional relational databases for
>specific functional analysis queries on timeseries such as market TICs or
>historic day end pricing for financial instruments.

There's no mystery. They usually load the time series data into main memory
and then apply special time series analysis algorithms. A lot of statistical
stuff can be done in a stream oriented fashion, so you don't even have to
keep the whole thing in main memory.

>[...] I know that kdb use the basic concept of storing columns on disk
>rather than relations and are moving this column data into memory to
>operate on.

Frankly that sounds a bit like marketing speak to me. Obviously reading the
whole table into memory is going to be faster if it only contains the time
series data you need, but you can get that arranged in any relational
database. And if I read on their site that:

  "Most relational databases are row-based, with every row stored randomly
  on disk. Reading one attribute requires reading an entire record, and
  scanning a column of values requires reading the entire table."

then I can only conclude that the person who wrote that doesn't know how
relational databases work.

-- Jan Hidders