Re: Evaluation metric - NLIDB

Håkan Lane wrote:
> I am a part of a university research project on Natural Language
> Interface to Databases (NLIDB:s). As a part of this research, we will
> trying to establish some kind of metric measuring how well different
> techniques and/or contemporary applications extract information from a
> given database.
>
> Please bear in mind that I'm still a novice in this area.
>
> - Are there any current contemporary "universal" statistics for this
> purpose, such as recall, rejection rate, etc.?
>
> - How are they exactly defined?
>
> - Where can I find resources on this topic?

There is an on-line book on information retrieval:

  http://www.dcs.gla.ac.uk/Keith/Preface.html

In chapter 7 you will find information on the several metrics that exist for measuring the effectiveness of an information retrieval system.

> - What are the metrics in conventional DBMS:s?

The usual metrics for information retrieval do not really apply to DBMSs. There it is usually assumed that the user knows what he or she wants, knows the contents of database and immediately can write down the query that retrieves the relevant information. In information retrieval these assumptions are not made and an attempt is made to help the user finding his or her way in the information. The effectiveness of the IR system is in fact how good it is at helping the user finding the right information.

Kind regards,

-- 
  Jan Hidders