TREC is a series of conference organized by various member of the US Defense Department around the concept of Information Retrieval. Various datasets are made available to the public to test and develop different search engines. One of the greatest advantages of these data is the availability of a relevance measure for a given query, that was computed by a number of experts that blinded reviewed the given query against the given results.

TREC uses the following working definition of relevance: If you were writing a report on the subject of the topic and would use the information contained in the document in the report, then the document is relevant. Only binary judgments (“relevant” or “not relevant”) are made, and a document is judged relevant if any piece of it is relevant (regardless of how small the piece is in relation to the rest of the document).

Judging is done using a pooling technique (described in the Overview papers in the TREC proceedings) on the set of documents used for the task that year. The relevance judgments are considered “complete” for that particular set of documents. By “complete” we mean that enough results have been assembled and judged to assume that most relevant documents have been found. When using these judgments to evaluate your own retrieval runs, it is very important to make sure the document collection and qrels match.

Tags: information retrieval