Lemur 3.0 release notes

(Older versions: 1.1, 1.9, 2.0, 2.1, 2.2 )

We have tested using gcc 3.2.2, 3.2.3, VC++ 6.0, and VC++ .NET .
This version is NOT backwards compatible. For more information see the "Code Update" section below
New Applications:
- Cluster - This application performs the basic online clustering task. In conjunction with an incremental indexer (such as KeyfileIncIndex), it could be used for TDT topic detection task. It iterates over the documents in the index, assigning each document that is not in any cluster to a cluster. The document id, cluster id, and score are printed to the standard output.
- OfflineCluster - This example application demonstrates the basic offline clustering task. Provides k-means and bisecting k-means partitional clustering. It will run each algorithm on the first 100 documents in the index (or all of them if less than 100) and print out the results.
- PDictManager - Probabilistic Dictionary Maintenance
  Provides a command line interface for interacting with probabilistic dictionaries (PDict objects). Support creating a new dictionary, merging dictionaries, pruning one dictionary from another, normalizing probabilities, outputing the dictionary to a file, and interactively looking up entries.
- PLSA - Probabilistic Latent Semantic Analysis
  This application will either perform PLSA on a collection, building three probability tables: P(z), P(d|z), and P(w|z) where z in Z are the latent variables (categories), d in D are the documents in the collection, and w in W are the terms in the vocabulary over the collection, or open those tables and read them into memory to illustrate their potential use.
- XlingRetEval - An application for running cross-lingual retrieval experiments.
Additions, Enhancements, and other changes:
- New clustering support for online and offline clustering tasks. Provides k-means and bisecting k-means partitional clustering algorithms.
- New support for cross-lingual language modeling and information retrieval.
- New support for probabilistic latent semantic analysis.
- New probabilistic dictionary class.
- Better support for passage retrieval using any retrieval method.
- New global Exception class for Lemur with macros for better exception handling. Supports nesting of messages, including file and line number information.
- New static methods in RetMethodManager for running queries with one method call (RetMethodManager::runQuery). Retrieval method parameters are taken from the parameter stack or defaults used.
- Improvements to the Lemur CGI, including showing DATE fields in results and showing document excerpts in results.
- Improvements to the Lemur Retrieval GUI, including paging and cacheing of results, showing document excerpts in results, match highlighing using MatchInfo, saving program options between sessions, exception catching, and using native system look and feel.
- New STL style external iterators for DocInfoList and TermInfoList. These iterators work for all lists currently implemented in the toolkit. No downcasting of lists returned from Index is required to use the iterator.
- Promote position specific methods into general API (ie Index::termInfoListSeq, TermInfo::position, DocInfo::positions) so that downcasting of objects is not required to access position information. Objects that don't store position information, return NULL.
- Broaden the use of different query objects with retrieval methods by creating new TermQuery class to serve as API for iterating over terms in a query. Changed TextQueryRetMethod and StructQueryRetMethod to use this in API instead of own query classes.
- Updated StringQuery class can be used with any retrieval method expecting a TermQuery. StringQuery is also a TextHandler that can be used for parsing a query or catching query terms at the end of a stopper/stemmer/etc chain. *note* - constructor parameter changed from StringQuery(queryid, query) to StringQuery(query, queryid).
- General improvements to the Lemur API by using string instead of char* and using const wherever appropriate. Other general improvements include adding of encapsulation methods to basic objects and removing many friend classes.
- General deprecations of outdated or unnecessary objects. Please see "Code Update" section below for more information.

Code Update:

This version has many API changes to comply with C++ const correctedness standards and changed most uses of char* to string objects. Don't be alarmed if you see a lot of errors when you try to compile your code against 3.0. Most of it is just simple fixes that involve adding const here and there to variables accepting return values from function calls and changing char* to string.
In general, other changes can be made to improve the performance of your programs, such as removing unnecessary downcasts and using new iterators, but they should still compile without those changes.
While you'll definitely need to recompile your code, you *should* be able to use new code with existing Inv(FP)Indexes, but we make no guarantees. KeyfileIncIndexes will definitely have to be re-indexed with the new toolkit.

Here is a list of deprecations. These are things that no longer exist in the toolkit:

Deprecated:	Replaced with:
class BasicDocInfo	class DocInfo
class BasicTermInfo	class TermInfo
class TokenTerm	class Term
class StringToken	class Term
class BasicTokenTerm	class Term
class QueryToken	class Term
class StructQuery	class TextQuery or class StringQuery
application PushIndexer	application BuildInvertedIndex (identical)
application BuildBasicIndex	the BasicIndex is being deprecated. use BuildKeyfileIncIndex to use KeyfileIncIndex or use BuildInvertedIndex for Inv(FP)Index

Here is a list of things that will be deprecated in the near future:

Will Be Deprecated:	Replaced with:
class InvDocInfo	class DocInfo
class QueryDocument	class StringQuery
class BasicIndex and its supporting classes	use Inv(FP)Index or KeyfileIncIndex
sequential iteration methods in DocInfoList and TermInfoList (startIteration, hasMore, nextEntry)	STL style external iterators

Bugs Fixed:
1. Problem: Keyfile leaks memory on open or close
  Solution: Change use of auto_ptrs
2. Problem: KeyfileIncIndex adds duplicate file and corrupts index counts
  Solution: Change to discard duplicate files (docids) with warning
  Problem: Keyfile index is ruined if a document with a long key is indexed
  Solution: Fixed to truncate long docids to maximum id length and discard terms longer than maximum length
3. Problem: BuildDocMgr crashes if KeyfileDocMgr or ElemDocMgr already exists
  Solution: On build, check for existing DocMgr and call appropriate load method for incremental adds
4. Problem: SimpleKLParameter values not set in RetEval, RelFBEval, TwoStageRetEval applications
  Solution: Call SimpleKLParameter::get in applications
5. Problem: QueryClarity per term scores are incorrect
  Solution: Correctly scale term contributions so that they sum to the total clarity score.
6. Problem: Memory leak in QueryModelEval
  Solution: Delete workSet after use.Also delete workSetStr and docPool.
7. Problem: IncIndexer fails on windows. windows stdlib does not properly handle ios::app (mode "ab") for opening file streams.
  Solution: Replace uses with work-around
8. Problem: Memory leak if Parser::setAcroList(string) is called more than once for same object.
  Solution: Delete old list before setting new one
9. Problem: KStemmer seg faults if KstemmerDir path is invalid
  Solution: Modify to emit an error message and throw an exception if failure occurs
10. Problem: Retrieval GUI crashes when using InvIndex or BasicIndex
  Solution: use Okapi as default method (no positions required) and add exception handling

The Lemur Project

Last modified: Wed Jun 30 14:21:51 EDT 2004