Lemur Modules and Applications (Version 4.2)

1. Parsing and Pre-processing

   ParseToFile	Parses documents compatible with Parser objects and writes output compatible with BasicDocStream
   ParseQuery	Takes a document in NIST's Web or Trec formats and creates queries
   ParseInQueryOp	Parses a file containing structured queries into BasicDocStream format

2. Building/Adding to an index

   BuildIndex	Builds an index of type Inv(FP), KeyfileInc, or Indri.
   BuildDocMgr	Builds a DocumentManager and Index for Inv(FP) or KeyfileInc indexes. (Indri has its own document manager built in)
   BuildPropIndex	Builds a positional index that can associate properties with terms, such as part of speech and named entity tags
   IndriBuildIndex	Build an IndriIndex (Indri Repository) using Indri style parameter files and parsing, not using Lemur parameters nor TextHandlers.
   PassageIndexer	Builds a positional passage index that segments documents into passage sizes
   IncIndexer	Adds documents into an existing InvFPIndex, or creates a new one
   IncPassageIndexer	Adds passages into an existing passage index

3. General Retrieval and Evaluation

   RetEval	Runs retrieval experiments (with/without feedback) to evaluate different retrieval models, such as simple TFIDF, Okapi, KL-divergence, and Indri SQL.
   RelFBEval	Runs retrieval experiments with relevance feedback
   QueryModelEval	Loads an expanded query model (e.g., one computed by GenerateQueryModel), and evaluates it with the KL-divergence retrieval model
   TwoStageRetEval	Runs retrieval experiments, using the two-stage smoothing method for the initial retrieval and the KL-divergence model for feedback
   GenL2Norm	Generates a support file for retrieval using cosine similarity
   QueryClarity	Computes clarity scores for a query model
   GenerateSmoothSupport	Generates two support files for retrieval using the language modeling approach to speed up the retrieval process
   GenerateQueryModel	Computes an expanded query model based on feedback documents and the original query model for the KL-divergence retrieval method
   EstimateDirPrior	Uses the leave-one-out method to estimate an optimal setting for the Dirichlet prior smoothing parameter
   trec_eval	The program trec_eval is available as trec_eval source distribution at NIST. Instructions for compilation and installation on unix are included. Users who want TREC style evaluation reports will be best served by using this community standard tool.

4. InQuery Structured Query Language

   ParseInQueryOp	Parses a file containing structured queries into BasicDocStream format
   StructQueryEval	Runs retrieval experiments to evaluate the performance of the structured query model using the inquery retrieval method

5. Indri Structured Query Retrieval

   RetEval	Retrieval evaluation using the IndriRetMethod (using an IndriIndex)
   IndriRunQuery	Retrieval evaluation for the Indri structured query language, directly using the Indri Repository API.

6. User Interfaces

   Lemur CGI	Cgi code for using Lemur indexes on the web
   Retrieval GUI	GUI written in java/swing for searching Lemur indexes

7. Distributed IR and Query-based Sampling

   CollSelIndex	Builds a collection selection database using either document frequency or collection term frequency for the database's term frequency count
   DistRetEval	Does distributed retrieval, using a resource selection index and individual indexes
   QryBasedSample	Performs query-based sampling on text databases

8. Summarization

   BasicSummApp	Demonstrates a simple summarizer
   MMRSummApp	A more complex summarizer which does comparisons between passages

9. Document Clustering

   Cluster	Performs the basic online clustering task over documents in an index. Can be used for TDT topic detection.
   OfflineCluster	Demonstrates the basic offline clustering task. Provides k-means and bisecting k-means partitional clustering.
   PLSA	Perform Probabilistic Latent Semantic Analysis (PLSA) on a collection, building three probability tables.