Re: search engines

From: tj bandrowsky <tbandrow_at_unitedsoftworks.com>
Date: 15 Feb 2002 14:54:22 -0800
Message-ID: <52e031b9.0202151454.4d9eae5d_at_posting.google.com>


It's not the search time that's an issue with most RDBMS, its the time it takes
to load up the table you want to search. If you built a table of 100,000,000 rows in Oracle, each with say a word for a key, plus a reference to a file, you could probably select all of the document ids for a given word in under a second or so. It's that fast. Unfortunately, sans bulk loading or something, normally you would get to insert around 1000-5000 rows per second into your table, which is much too slow for building up a full text index - think word == row.

Some people think that "simpler" databases are automatically faster, and that support of transactions makes a database slower. This is actually not the case. For either MySQL or Oracle, for example, I used to insert about a thousand rows per second. That was with a two field primary key plus four values for the row. You have to do more to get that speed out of Oracle, bound statements are a must, but, you can get it.

You could also theoretically write your own dedicated indexing scheme for that reason. A lot of people do. In fact, you'll see dedicated full text indeces running in parallel with a traditional RDBMS. SQL Server does this, as does Oracle.

bruces_at_real-info.com (Bruce Schreiber) wrote in message news:<f1645a5c.0202141058.64c48de2_at_posting.google.com>...
> How can search engine search through millions of records in just a few
> seconds?
Received on Fri Feb 15 2002 - 23:54:22 CET

Original text of this message