Re: Internet search engines and databases

In article <38CF8BC0.A39F60EE_at_attglobal.net>, Tony Fulford <tony_fulford_at_attglobal.net> writes
>I would like to start a new thread to discuss the future of searching
>the internet.
>
>Internet search engines utilize the accessible content of a website to
>generate their search results. More and more website content is being
>placed in databases and served dynamically to users as required. The
>information in these databases is not available to search engines
>because the databases have security mechanisms (e.g., username &
>password) to prevent access. As more and more websites become based on
>information contained in databases, the search results be come
>increasingly sparse and inaccurate.

You can only find what the web-site owners are willing to put on their sites. What you could aim to do would be to deliver the searcher to the right web-site then hand them over to the site owner.

>
>I have already found that when I want to find an item, a particular
>model computer printer (using model number for the search) for example,
>I will not get a hit from many websites via the search engines, yet if I
>go to the websites and search, I do get a search hit. So where will this
>leave us for comparison shopping for products.

That's a problem that occurs again and again in a lot of different contexts. You could have searched for the manufacturer's name, which would probably have delivered you to the right web-site.

>
>Are web search engines going to be relegated to providing search results
>from websites which only have static content?

That's not a true statement of the problem. From the label on the printer you could probably have found enough information to navigate to the right web-site. From that point onwards the process would have been under the control of the site owner. It's their data, why should anyone else have any say in how its presented or what security precautions are used?

What you are looking for seems to be a way of automating that final stage. I doubt that this is possible, it requires every web-site to record every piece of information in a standard form with no access restrictions. Essentially you want an index to the known universe.

>
>Are there ways to let search engines have access to databases without
>giving away the store (i.e., family jewels)?

If there are access restrictions the search engine could act as your agent and buy the data for you. I've been thinking of technology to do that. I could operate the agent and charge commission on the transaction. (Ooops, there goes another software patent.)

>
>If we give search engines access to the databases, how do they know how
>to search them (e.g., databases, tables, columns, etc.)?

I can imagine a standard query that causes the database to return a list of all of its tables, and then a list of all of the fields. How will it cope with a database that one of my suppliers sends to me each month. It contains twelve tables called "1", "2", "3"... etc. Each of those tables contains about 30 fields called "1", "2"....

It's a valid database and I might one day want to make the contents available through a web page. Of course they might rebuild it as an object database, and it won't have any tables at all.

>
>Is it possible to generate an XML standard which would allow general
>access to most databases?

No. This isn't just a technical issue, it depends on whether the data owners want this to be possible.

>
>Do we need some standards for databases which want to participate with
>search engines?

No we don't because this is a solved problem. If I search Yahoo and don't have any luck Yahoo will offer to pass my search on to one of the other search engines. This is a search engine interfacing to another database.

>
>The future of internet navigation and true competitive x-commerce
>depends on resolution of this problem. Let's get started!

That's hyperbole.

-- 
Bernard Peek
bap_at_shrdlu.com
bap_at_shrdlu.co.uk