Searching Approach

From: spacedog <s1m0nc00k_at_hotmail.com>
Date: Tue, 04 Sep 2007 03:08:59 -0700
Message-ID: <1188900539.104563.43980@g4g2000hsf.googlegroups.com>

Hi,

I am looking for the most efficient way to perform a daily search of 15,000 customers against 60M transactions in the system, using a complicated close matching set of rules. Any matches will then be recorded for future use.

The transactions have a number of indexes which have been put on the table to help optimize the search.

The current approach suggested is to take batches of the customers and fire off multiple searches. This doesn't seem to me like a very efficient way of doing things as it looks like we'll be doing lots of repeat gets from disk. The indexes are about 15 to 20 GB in size so its doubtful that we'll be able to store them in memory.

Any ideas would be much appreciated.

Thanks
Simon Received on Tue Sep 04 2007 - 05:08:59 CDT