Lucene Change Log

Trunk (not yet released)

• Changes in runtime behavior   (2)
  1. LUCENE-1151: Fix StandardAnalyzer to not mis-identify host names (eg lucene.apache.org) as an ACRONYM. To get back to the pre-2.4 backwards compatible, but buggy, behavior, you can either call StandardAnalyzer.setDefaultReplaceInvalidAcronym(false) (static method), or, set system property org.apache.lucene.analysis.standard.StandardAnalyzer.replaceInvalidAcronym to "false" on JVM startup. All StandardAnalyzer instances created after that will then show the pre-2.4 behavior. Alternatively, you can call setReplaceInvalidAcronym(false) to change the behavior per instance of StandardAnalyzer. This backwards compatibility will be removed in 3.0 (hardwiring the value to true).
     (Mike McCandless)
  2. LUCENE-1044: IndexWriter with autoCommit=true now commits (such that a reader can see the changes) far less often than it used to. Previously, every flush was also a commit. You can always force a commit by calling IndexWriter.commit(). Furthermore, in 3.0, autoCommit will be hardwired to false (IndexWriter constructors that take an autoCommit argument have been deprecated)
     (Mike McCandless)
• API Changes   (8)
  1. LUCENE-1084: Changed all IndexWriter constructors to take an explicit parameter for maximum field size. Deprecated all the pre-existing constructors; these will be removed in release 3.0.
     (Steven Rowe via Mike McCandless)
  2. LUCENE-1150: Re-expose StandardTokenizer's constants publicly; this was accidentally lost with LUCENE-966.
     (Nicolas Lalevée via Mike McCandless)
  3. LUCENE-584: Changed Filter API to return a DocIdSet instead of a java.util.BitSet. This allows using more efficient data structures for Filters and makes them more flexible. This deprecates Filter.bits(), so all filters that implement this outside the Lucene code base will need to be adapted. See also the javadocs of the Filter class.
     (Paul Elschot, Michael Busch)
  4. LUCENE-1044: Added IndexWriter.commit() which flushes any buffered adds/deletes and then commits a new segments file so readers will see the changes. Deprecate IndexWriter.flush() in favor of IndexWriter.commit().
     (Mike McCandless)
  5. LUCENE-325: Added IndexWriter.expungeDeletes methods, which consult the MergePolicy to find merges necessary to merge away all deletes from the index. This should be a somewhat lower cost operation than optimize.
     (John Wang via Mike McCandless)
  6. LUCENE-1233: Return empty array instead of null when no fields match the specified name in these methods in Document: getFieldables, getFields, getValues, getBinaryValues.
     (Stefan Trcek vai Mike McCandless)
  7. LUCENE-1234: Make BoostingSpanScorer protected.
     (Andi Vajda via Grant Ingersoll)
  8. LUCENE-510: The index now stores strings as true UTF-8 bytes (previously it was Java's modified UTF-8). If any text, either stored fields or a token, has illegal UTF-16 surrogate characters, these characters are now silently replaced with the Unicode replacement character U+FFFD. This is a change to the index file format.
     (Marvin Humphrey via Mike McCandless)
• Bug fixes   (10)
  1. LUCENE-1134: Fixed BooleanQuery.rewrite to only optimize a single clause query if minNumShouldMatch<=0.
     (Shai Erera via Michael Busch)
  2. LUCENE-1169: Fixed bug in IndexSearcher.search(): searching with a filter might miss some hits because scorer.skipTo() is called without checking if the scorer is already at the right position. scorer.skipTo(scorer.doc()) is not a NOOP, it behaves as scorer.next().
     (Eks Dev, Michael Busch)
  3. LUCENE-1182: Added scorePayload to SimilarityDelegator
     (Andi Vajda via Grant Ingersoll)
  4. LUCENE-1213: MultiFieldQueryParser was ignoring slop in case of a single field phrase.
     (Trejkaz via Doron Cohen)
  5. LUCENE-1226: Fixed IndexWriter.addIndexes(IndexReader[]) to commit successfully created compound files.
     (Michael Busch)
  6. LUCENE-1228: IndexWriter.commit() was not updating the index version and as result IndexReader.reopen() failed to sense index changes.
     (Doron Cohen)
  7. LUCENE-1262: Fixed bug in BufferedIndexReader.refill whereby on hitting an exception in readInternal, the buffer is incorrectly filled with stale bytes such that subsequent calls to readByte() return incorrect results.
     (Trejkaz via Mike McCandless)
  8. LUCENE-1267: Added numDocs() and maxDoc() to IndexWriter; deprecated docCount().
     (Mike McCandless)
  9. LUCENE-1274: Added new prepareCommit() method to IndexWriter, which does phase 1 of a 2-phase commit (commit() does phase 2). This is needed when you want to update an index as part of a transaction involving external resources (eg a database). Also deprecated abort(), renaming it to rollback().
     (Mike McCandless)
  10. LUCENE-1003: Stop RussianAnalyzer from removing numbers.
      (TUSUR OpenTeam, Dmitry Lihachev via Otis Gospodnetic)
• New features   (13)
  1. LUCENE-1137: Added Token.set/getFlags() accessors for passing more information about a Token through the analysis process. The flag is not indexed/stored and is thus only used by analysis.
  2. LUCENE-1147: Add -segment option to CheckIndex tool so you can check only a specific segment or segments in your index.
     (Mike McCandless)
  3. LUCENE-1045: Reopened this issue to add support for short and bytes.
  4. LUCENE-584: Added new data structures to o.a.l.util, such as OpenBitSet and SortedVIntList. These extend DocIdSet and can directly be used for Filters with the new Filter API. Also changed the core Filters to use OpenBitSet instead of java.util.BitSet.
     (Paul Elschot, Michael Busch)
  5. LUCENE-494: Added QueryAutoStopWordAnalyzer to allow for the automatic removal, from a query of frequently occurring terms. This Analyzer is not intended for use during indexing.
     (Mark Harwood via Grant Ingersoll)
  6. LUCENE-1044: Change Lucene to properly "sync" files after committing, to ensure on a machine or OS crash or power cut, even with cached writes, the index remains consistent. Also added explicit commit() method to IndexWriter to force a commit without having to close.
     (Mike McCandless)
  7. LUCENE-997: Add search timeout (partial) support. A TimeLimitedCollector was added to allow limiting search time. It is a partial solution since timeout is checked only when collecting a hit, and therefore a search for rare words in a huge index might not stop within the specified time.
     (Sean Timm via Doron Cohen)
  8. LUCENE-1184: Allow SnapshotDeletionPolicy to be re-used across close/re-open of IndexWriter while still protecting an open snapshot
     (Tim Brennan via Mike McCandless)
  9. LUCENE-1194: Added IndexWriter.deleteDocuments(Query) to delete documents matching the specified query. Also added static unlock and isLocked methods (deprecating the ones in IndexReader).
     (Mike McCandless)
  10. LUCENE-1201: Add IndexReader.getIndexCommit() method.
      (Tim Brennan via Mike McCandless)
  11. LUCENE-550: Added InstantiatedIndex implementation. Experimental Index store similar to MemoryIndex but allows for multiple documents in memory.
      (Karl Wettin via Grant Ingersoll)
  12. LUCENE-400: Added word based n-gram filter (in contrib/analyzers) called ShingleFilter and an Analyzer wrapper that wraps another Analyzer's token stream with a ShingleFilter
      (Sebastian Kirsch, Steve Rowe via Grant Ingersoll)
  13. LUCENE-1166: Decomposition tokenfilter for languages like German and Swedish
      (Thomas Peuss via Grant Ingersoll)
• Optimizations   (4)
  1. LUCENE-705: When building a compound file, use RandomAccessFile.setLength() to tell the OS/filesystem to pre-allocate space for the file. This may improve fragmentation in how the CFS file is stored, and allows us to detect an upcoming disk full situation before actually filling up the disk.
     (Mike McCandless)
  2. LUCENE-1120: Speed up merging of term vectors by bulk-copying the raw bytes for each contiguous range of non-deleted documents.
     (Mike McCandless)
  3. LUCENE-1185: Avoid checking if the TermBuffer 'scratch' in SegmentTermEnum is null for every call of scanTo().
     (Christian Kohlschuetter via Michael Busch)
  4. LUCENE-1217: Internal to Field.java, use isBinary instead of runtime type checking for possible speedup of binaryValue().
     (Eks Dev via Mike McCandless)
• Documentation   (2)
  1. LUCENE-1236: Added some clarifying remarks to EdgeNGram*.java
     (Hiroaki Kawai via Grant Ingersoll)
  2. LUCENE-1157 and LUCENE-1256: HTML changes log, created automatically from CHANGES.txt. This HTML file is curretly visible only via developers page.
     (Steven Rowe via Doron Cohen)
• Build   (3)
  1. LUCENE-1153: Added JUnit JAR to new lib directory. Updated build to rely on local JUnit instead of ANT/lib.
  2. LUCENE-1230: Include *pom.xml* in source release files.
     (Michael Busch)
  3. LUCENE-1202: Small fixes to the way Clover is used to work better with contribs. Of particular note: a single clover db is used regardless of whether tests are run globally or in the specific contrib directories.
• Test Cases   (1)
  1. LUCENE-1238: Fixed intermittent failures of TestTimeLimitedCollector.testTimeoutMultiThreaded. Within this fix, "greedy" flag was added to TimeLimitedCollector, to allow the wrapped collector to collect also the last doc, after allowed-tTime passed.
     (Doron Cohen)

Release 2.3.1 [2008-02-22]

• Bug fixes   (7)
  1. LUCENE-1168: Fixed corruption cases when autoCommit=false and documents have mixed term vectors
     (Suresh Guvvala via Mike McCandless)
  2. LUCENE-1171: Fixed some cases where OOM errors could cause deadlock in IndexWriter
     (Mike McCandless)
  3. LUCENE-1173: Fixed corruption case when autoCommit=false and bulk merging of stored fields is used
     (Yonik via Mike McCandless)
  4. LUCENE-1163: Fixed bug in CharArraySet.contains(char[] buffer, int offset, int len) that was ignoring offset and thus giving the wrong answer.
     (Thomas Peuss via Mike McCandless)
  5. LUCENE-1177: Fix rare case where IndexWriter.optimize might do too many merges at the end.
     (Mike McCandless)
  6. LUCENE-1176: Fix corruption case when documents with no term vector fields are added before documents with term vector fields.
     (Mike McCandless)
  7. LUCENE-1179: Fixed assert statement that was incorrectly preventing Fields with empty-string field name from working.
     (Sergey Kabashnyuk via Mike McCandless)

Older Releases

Release 2.3.0 [2008-01-21]

• Changes in runtime behavior   (2)
  1. LUCENE-994: Defaults for IndexWriter have been changed to maximize out-of-the-box indexing speed. First, IndexWriter now flushes by RAM usage (16 MB by default) instead of a fixed doc count (call IndexWriter.setMaxBufferedDocs to get backwards compatible behavior). Second, ConcurrentMergeScheduler is used to run merges using background threads (call IndexWriter.setMergeScheduler(new SerialMergeScheduler()) to get backwards compatible behavior). Third, merges are chosen based on size in bytes of each segment rather than document count of each segment (call IndexWriter.setMergePolicy(new LogDocMergePolicy()) to get backwards compatible behavior).

     NOTE: users of ParallelReader must change back all of these defaults in order to ensure the docIDs "align" across all parallel indices.
     (Mike McCandless)

  2. LUCENE-1045: SortField.AUTO didn't work with long. When detecting the field type for sorting automatically, numbers used to be interpreted as int, then as float, if parsing the number as an int failed. Now the detection checks for int, then for long, then for float.
     (Daniel Naber)
• API Changes   (14)
  1. LUCENE-843: Added IndexWriter.setRAMBufferSizeMB(...) to have IndexWriter flush whenever the buffered documents are using more than the specified amount of RAM. Also added new APIs to Token that allow one to set a char[] plus offset and length to specify a token (to avoid creating a new String() for each Token).
     (Mike McCandless)
  2. LUCENE-963: Add setters to Field to allow for re-using a single Field instance during indexing. This is a sizable performance gain, especially for small documents.
     (Mike McCandless)
  3. LUCENE-969: Add new APIs to Token, TokenStream and Analyzer to permit re-using of Token and TokenStream instances during indexing. Changed Token to use a char[] as the store for the termText instead of String. This gives faster tokenization performance (~10-15%).
     (Mike McCandless)
  4. LUCENE-847: Factored MergePolicy, which determines which merges should take place and when, as well as MergeScheduler, which determines when the selected merges should actually run, out of IndexWriter. The default merge policy is now LogByteSizeMergePolicy (see LUCENE-845) and the default merge scheduler is now ConcurrentMergeScheduler (see LUCENE-870).
     (Steven Parkes via Mike McCandless)
  5. LUCENE-1052: Add IndexReader.setTermInfosIndexDivisor(int) method that allows you to reduce memory usage of the termInfos by further sub-sampling (over the termIndexInterval that was used during indexing) which terms are loaded into memory.
     (Chuck Williams, Doug Cutting via Mike McCandless)
  6. LUCENE-743: Add IndexReader.reopen() method that re-opens an existing IndexReader (see New features -> 8.)
     (Michael Busch)
  7. LUCENE-1062: Add setData(byte[] data), setData(byte[] data, int offset, int length), getData(), getOffset() and clone() methods to o.a.l.index.Payload. Also add the field name as arg to Similarity.scorePayload().
     (Michael Busch)
  8. LUCENE-982: Add IndexWriter.optimize(int maxNumSegments) method to "partially optimize" an index down to maxNumSegments segments.
     (Mike McCandless)
  9. LUCENE-1080: Changed Token.DEFAULT_TYPE to be public.
  10. LUCENE-1064: Changed TopDocs constructor to be public.
      (Shai Erera via Michael Busch)
  11. LUCENE-1079: DocValues cleanup: constructor now has no params, and getInnerArray() now throws UnsupportedOperationException
      (Doron Cohen)
  12. LUCENE-1089: Added PriorityQueue.insertWithOverflow, which returns the Object (if any) that was bumped from the queue to allow re-use.
      (Shai Erera via Mike McCandless)
  13. LUCENE-1101: Token reuse 'contract' (defined LUCENE-969) modified so it is token producer's responsibility to call Token.clear().
      (Doron Cohen)
  14. LUCENE-1118: Changed StandardAnalyzer to skip too-long (default > 255 characters) tokens. You can increase this limit by calling StandardAnalyzer.setMaxTokenLength(...).
      (Michael McCandless)
• Bug fixes   (28)
  1. LUCENE-933: QueryParser fixed to not produce empty sub BooleanQueries "()" even if the Analyzer produced no tokens for input.
     (Doron Cohen)
  2. LUCENE-955: Fixed SegmentTermPositions to work correctly with the first term in the dictionary.
     (Michael Busch)
  3. LUCENE-951: Fixed NullPointerException in MultiLevelSkipListReader that was thrown after a call of TermPositions.seek().
     (Rich Johnson via Michael Busch)
  4. LUCENE-938: Fixed cases where an unhandled exception in IndexWriter's methods could cause deletes to be lost.
     (Steven Parkes via Mike McCandless)
  5. LUCENE-962: Fixed case where an unhandled exception in IndexWriter.addDocument or IndexWriter.updateDocument could cause unreferenced files in the index to not be deleted
     (Steven Parkes via Mike McCandless)
  6. LUCENE-957: RAMDirectory fixed to properly handle directories larger than Integer.MAX_VALUE.
     (Doron Cohen)
  7. LUCENE-781: MultiReader fixed to not throw NPE if isCurrent(), isOptimized() or getVersion() is called. Separated MultiReader into two classes: MultiSegmentReader extends IndexReader, is package-protected and is created automatically by IndexReader.open() in case the index has multiple segments. The public MultiReader now extends MultiSegmentReader and is intended to be used by users who want to add their own subreaders.
     (Daniel Naber, Michael Busch)
  8. LUCENE-970: FilterIndexReader now implements isOptimized(). Before a call of isOptimized() would throw a NPE.
     (Michael Busch)
  9. LUCENE-832: ParallelReader fixed to not throw NPE if isCurrent(), isOptimized() or getVersion() is called.
     (Michael Busch)
  10. LUCENE-948: Fix FNFE exception caused by stale NFS client directory listing caches when writers on different machines are sharing an index over NFS and using a custom deletion policy
      (Mike McCandless)
  11. LUCENE-978: Ensure TermInfosReader, FieldsReader, and FieldsReader close any streams they had opened if an exception is hit in the constructor.
      (Ning Li via Mike McCandless)
  12. LUCENE-985: If an extremely long term is in a doc (> 16383 chars), we now throw an IllegalArgumentException saying the term is too long, instead of cryptic ArrayIndexOutOfBoundsException.
      (Karl Wettin via Mike McCandless)
  13. LUCENE-991: The explain() method of BoostingTermQuery had errors when no payloads were present on a document.
      (Peter Keegan via Grant Ingersoll)
  14. LUCENE-992: Fixed IndexWriter.updateDocument to be atomic again (this was broken by LUCENE-843).
      (Ning Li via Mike McCandless)
  15. LUCENE-1008: Fixed corruption case when document with no term vector fields is added after documents with term vector fields. This bug was introduced with LUCENE-843.
      (Grant Ingersoll via Mike McCandless)
  16. LUCENE-1006: Fixed QueryParser to accept a "" field value (zero length quoted string.)
      (yonik)
  17. LUCENE-1010: Fixed corruption case when document with no term vector fields is added after documents with term vector fields. This case is hit during merge and would cause an EOFException. This bug was introduced with LUCENE-984.
      (Andi Vajda via Mike McCandless)
  18. LUCENE-1009: Fix merge slowdown with LogByteSizeMergePolicy when autoCommit=false and documents are using stored fields and/or term vectors.
      (Mark Miller via Mike McCandless)
  19. LUCENE-1011: Fixed corruption case when two or more machines, sharing an index over NFS, can be writers in quick succession.
      (Patrick Kimber via Mike McCandless)
  20. LUCENE-1028: Fixed Weight serialization for few queries: DisjunctionMaxQuery, ValueSourceQuery, CustomScoreQuery. Serialization check added for all queries.
      (Kyle Maxwell via Doron Cohen)
  21. LUCENE-1048: Fixed incorrect behavior in Lock.obtain(...) when the timeout argument is very large (eg Long.MAX_VALUE). Also added Lock.LOCK_OBTAIN_WAIT_FOREVER constant to never timeout.
      (Nikolay Diakov via Mike McCandless)
  22. LUCENE-1050: Throw LockReleaseFailedException in Simple/NativeFSLockFactory if we fail to delete the lock file when releasing the lock.
      (Nikolay Diakov via Mike McCandless)
  23. LUCENE-1071: Fixed SegmentMerger to correctly set payload bit in the merged segment.
      (Michael Busch)
  24. LUCENE-1042: Remove throwing of IOException in getTermFreqVector(int, String, TermVectorMapper) to be consistent with other getTermFreqVector calls. Also removed the throwing of the other IOException in that method to be consistent.
      (Karl Wettin via Grant Ingersoll)
  25. LUCENE-1096: Fixed Hits behavior when hits' docs are deleted along with iterating the hits. Deleting docs already retrieved now works seamlessly. If docs not yet retrieved are deleted (e.g. from another thread), and then, relying on the initial Hits.length(), an application attempts to retrieve more hits than actually exist , a ConcurrentMidificationException is thrown.
      (Doron Cohen)
  26. LUCENE-1068: Changed StandardTokenizer to fix an issue with it marking the type of some tokens incorrectly. This is done by adding a new flag named replaceInvalidAcronym which defaults to false, the current, incorrect behavior. Setting this flag to true fixes the problem. This flag is a temporary fix and is already marked as being deprecated. 3.x will implement the correct approach. (Shai Erera via Grant Ingersoll) LUCENE-1140: Fixed NPE caused by 1068
      (Alexei Dets via Grant Ingersoll)
  27. LUCENE-749: ChainedFilter behavior fixed when logic of first filter is ANDNOT.
      (Antonio Bruno via Doron Cohen)
  28. LUCENE-508: Make sure SegmentTermEnum.prev() is accurate (= last term) after next() returns false.
      (Steven Tamm via Mike McCandless)
• New features   (13)
  1. LUCENE-906: Elision filter for French.
     (Mathieu Lecarme via Otis Gospodnetic)
  2. LUCENE-960: Added a SpanQueryFilter and related classes to allow for not only filtering, but knowing where in a Document a Filter matches
     (Grant Ingersoll)
  3. LUCENE-868: Added new Term Vector access features. New callback mechanism allows application to define how and where to read Term Vectors from disk. This implementation contains several extensions of the new abstract TermVectorMapper class. The new API should be back-compatible. No changes in the actual storage of Term Vectors has taken place.
  4. LUCENE-1038: Added setDocumentNumber() method to TermVectorMapper to provide information about what document is being accessed.
     (Karl Wettin via Grant Ingersoll)
  5. LUCENE-975: Added PositionBasedTermVectorMapper that allows for position based lookup of term vector information. See item #3 above (LUCENE-868).
  6. LUCENE-1011: Added simple tools (all in org.apache.lucene.store) to verify that locking is working properly. LockVerifyServer runs a separate server to verify locks. LockStressTest runs a simple tool that rapidly obtains and releases locks. VerifyingLockFactory is a LockFactory that wraps any other LockFactory and consults the LockVerifyServer whenever a lock is obtained or released, throwing an exception if an illegal lock obtain occurred.
     (Patrick Kimber via Mike McCandless)
  7. LUCENE-1015: Added FieldCache extension (ExtendedFieldCache) to support doubles and longs. Added support into SortField for sorting on doubles and longs as well.
     (Grant Ingersoll)
  8. LUCENE-1020: Created basic index checking & repair tool (o.a.l.index.CheckIndex). When run without -fix it does a detailed test of all segments in the index and reports summary information and any errors it hit. With -fix it will remove segments that had errors.
     (Mike McCandless)
  9. LUCENE-743: Add IndexReader.reopen() method that re-opens an existing IndexReader by only loading those portions of an index that have changed since the reader was (re)opened. reopen() can be significantly faster than open(), depending on the amount of index changes. SegmentReader, MultiSegmentReader, MultiReader, and ParallelReader implement reopen().
     (Michael Busch)
  10. LUCENE-1040: CharArraySet useful for efficiently checking set membership of text specified by char[].
      (yonik)
  11. LUCENE-1073: Created SnapshotDeletionPolicy to facilitate taking a live backup of an index without pausing indexing.
      (Mike McCandless)
  12. LUCENE-1019: CustomScoreQuery enhanced to support multiple ValueSource queries.
      (Kyle Maxwell via Doron Cohen)
  13. LUCENE-1095: Added an option to StopFilter to increase positionIncrement of the token succeeding a stopped token. Disabled by default. Similar option added to QueryParser to consider token positions when creating PhraseQuery and MultiPhraseQuery. Disabled by default (so by default the query parser ignores position increments).
      (Doron Cohen)
• Optimizations   (14)
  1. LUCENE-937: CachingTokenFilter now uses an iterator to access the Tokens that are cached in the LinkedList. This increases performance significantly, especially when the number of Tokens is large.
     (Mark Miller via Michael Busch)
  2. LUCENE-843: Substantial optimizations to improve how IndexWriter uses RAM for buffering documents and to speed up indexing (2X-8X faster). A single shared hash table now records the in-memory postings per unique term and is directly flushed into a single segment.
     (Mike McCandless)
  3. LUCENE-892: Fixed extra "buffer to buffer copy" that sometimes takes place when using compound files.
     (Mike McCandless)
  4. LUCENE-959: Remove synchronization in Document
     (yonik)
  5. LUCENE-963: Add setters to Field to allow for re-using a single Field instance during indexing. This is a sizable performance gain, especially for small documents.
     (Mike McCandless)
  6. LUCENE-939: Check explicitly for boundary conditions in FieldInfos and don't rely on exceptions.
     (Michael Busch)
  7. LUCENE-966: Very substantial speedups (~6X faster) for StandardTokenizer (StandardAnalyzer) by using JFlex instead of JavaCC to generate the tokenizer.
     (Stanislaw Osinski via Mike McCandless)
  8. LUCENE-969: Changed core tokenizers & filters to re-use Token and TokenStream instances when possible to improve tokenization performance (~10-15%).
     (Mike McCandless)
  9. LUCENE-871: Speedup ISOLatin1AccentFilter
     (Ian Boston via Mike McCandless)
  10. LUCENE-986: Refactored SegmentInfos from IndexReader into the new subclass DirectoryIndexReader. SegmentReader and MultiSegmentReader now extend DirectoryIndexReader and are the only IndexReader implementations that use SegmentInfos to access an index and acquire a write lock for index modifications.
      (Michael Busch)
  11. LUCENE-1007: Allow flushing in IndexWriter to be triggered by either RAM usage or document count or both (whichever comes first), by adding symbolic constant DISABLE_AUTO_FLUSH to disable one of the flush triggers.
      (Ning Li via Mike McCandless)
  12. LUCENE-1043: Speed up merging of stored fields by bulk-copying the raw bytes for each contiguous range of non-deleted documents.
      (Robert Engels via Mike McCandless)
  13. LUCENE-693: Speed up nested conjunctions (~2x) that match many documents, and a slight performance increase for top level conjunctions.
      (yonik)
  14. LUCENE-1098: Make inner class StandardAnalyzer.SavedStreams static and final.
      (Nathan Beyer via Michael Busch)
• Documentation   (2)
  1. LUCENE-1051: Generate separate javadocs for core, demo and contrib classes, as well as an unified view. Also add an appropriate menu structure to the website.
     (Michael Busch)
  2. LUCENE-746: Fix error message in AnalyzingQueryParser.getPrefixQuery.
     (Ronnie Kolehmainen via Michael Busch)
• Build   (8)
  1. LUCENE-908: Improvements and simplifications for how the MANIFEST file and the META-INF dir are created.
     (Michael Busch)
  2. LUCENE-935: Various improvements for the maven artifacts. Now the artifacts also include the sources as .jar files.
     (Michael Busch)
  3. Added apply-patch target to top-level build. Defaults to looking for a patch in ${basedir}/../patches with name specified by -Dpatch.name. Can also specify any location by -Dpatch.file property on the command line. This should be helpful for easy application of patches, but it is also a step towards integrating automatic patch application with JIRA and Hudson, and is thus subject to change.
     (Grant Ingersoll)
  4. LUCENE-935: Defined property "m2.repository.url" to allow setting the url to a maven remote repository to deploy to.
     (Michael Busch)
  5. LUCENE-1051: Include javadocs in the maven artifacts.
     (Michael Busch)
  6. LUCENE-1055: Remove gdata-server from build files and its sources from trunk.
     (Michael Busch)
  7. LUCENE-935: Allow to deploy maven artifacts to a remote m2 repository via scp and ssh authentication.
     (Michael Busch)
  8. LUCENE-1123: Allow overriding the specification version for MANIFEST.MF
     (Michael Busch)
• Test Cases   (1)
  1. LUCENE-766: Test adding two fields with the same name but different term vector setting.
     (Nicolas Lalevée via Doron Cohen)

Release 2.2.0 [2007-06-19]

• Changes in runtime behavior   (none)
• API Changes   (13)
  1. LUCENE-793: created new exceptions and added them to throws clause for many methods (all subclasses of IOException for backwards compatibility): index.StaleReaderException, index.CorruptIndexException, store.LockObtainFailedException. This was done to better call out the possible root causes of an IOException from these methods.
     (Mike McCandless)
  2. LUCENE-811: make SegmentInfos class, plus a few methods from related classes, package-private again (they were unnecessarily made public as part of LUCENE-701).
     (Mike McCandless)
  3. LUCENE-710: added optional autoCommit boolean to IndexWriter constructors. When this is false, index changes are not committed until the writer is closed. This gives explicit control over when a reader will see the changes. Also added optional custom deletion policy to explicitly control when prior commits are removed from the index. This is intended to allow applications to share an index over NFS by customizing when prior commits are deleted.
     (Mike McCandless)
  4. LUCENE-818: changed most public methods of IndexWriter, IndexReader (and its subclasses), FieldsReader and RAMDirectory to throw AlreadyClosedException if they are accessed after being closed.
     (Mike McCandless)
  5. LUCENE-834: Changed some access levels for certain Span classes to allow them to be overridden. They have been marked expert only and not for public consumption.
     (Grant Ingersoll)
  6. LUCENE-796: Removed calls to super.* from various get*Query methods in MultiFieldQueryParser, in order to allow sub-classes to override them.
     (Steven Parkes via Otis Gospodnetic)
  7. LUCENE-857: Removed caching from QueryFilter and deprecated QueryFilter in favour of QueryWrapperFilter or QueryWrapperFilter + CachingWrapperFilter combination when caching is desired.
     (Chris Hostetter, Otis Gospodnetic)
  8. LUCENE-869: Changed FSIndexInput and FSIndexOutput to inner classes of FSDirectory to enable extensibility of these classes.
     (Michael Busch)
  9. LUCENE-580: Added the public method reset() to TokenStream. This method does nothing by default, but may be overwritten by subclasses to support consuming the TokenStream more than once.
     (Michael Busch)
  10. LUCENE-580: Added a new constructor to Field that takes a TokenStream as argument, available as tokenStreamValue(). This is useful to avoid the need of "dummy analyzers" for pre-analyzed fields.
      (Karl Wettin, Michael Busch)
  11. LUCENE-730: Added the new methods to BooleanQuery setAllowDocsOutOfOrder() and getAllowDocsOutOfOrder(). Deprecated the methods setUseScorer14() and getUseScorer14(). The optimization patch LUCENE-730 (see Optimizations->3.) improves performance for certain queries but results in scoring out of docid order. This patch reverse this change, so now by default hit docs are scored in docid order if not setAllowDocsOutOfOrder(true) is explicitly called. This patch also enables the tests in QueryUtils again that check for docid order.
      (Paul Elschot, Doron Cohen, Michael Busch)
  12. LUCENE-888: Added Directory.openInput(File path, int bufferSize) to optionally specify the size of the read buffer. Also added BufferedIndexInput.setBufferSize(int) to change the buffer size.
      (Mike McCandless)
  13. LUCENE-923: Make SegmentTermPositionVector package-private. It does not need to be public because it implements the public interface TermPositionVector.
      (Michael Busch)
• Bug fixes   (24)
  1. LUCENE-804: Fixed build.xml to pack a fully compilable src dist.
     (Doron Cohen)
  2. LUCENE-813: Leading wildcard fixed to work with trailing wildcard. Query parser modified to create a prefix query only for the case that there is a single trailing wildcard (and no additional wildcard or '?' in the query text).
     (Doron Cohen)
  3. LUCENE-812: Add no-argument constructors to NativeFSLockFactory and SimpleFSLockFactory. This enables all 4 builtin LockFactory implementations to be specified via the System property org.apache.lucene.store.FSDirectoryLockFactoryClass.
     (Mike McCandless)
  4. LUCENE-821: The new single-norm-file introduced by LUCENE-756 failed to reduce the number of open descriptors since it was still opened once per field with norms.
     (yonik)
  5. LUCENE-823: Make sure internal file handles are closed when hitting an exception (eg disk full) while flushing deletes in IndexWriter's mergeSegments, and also during IndexWriter.addIndexes.
     (Mike McCandless)
  6. LUCENE-825: If directory is removed after FSDirectory.getDirectory() but before IndexReader.open you now get a FileNotFoundException like Lucene pre-2.1 (before this fix you got an NPE).
     (Mike McCandless)
  7. LUCENE-800: Removed backslash from the TERM_CHAR list in the queryparser, because the backslash is the escape character. Also changed the ESCAPED_CHAR list to contain all possible characters, because every character that follows a backslash should be considered as escaped.
     (Michael Busch)
  8. LUCENE-372: QueryParser.parse() now ensures that the entire input string is consumed. Now a ParseException is thrown if a query contains too many closing parentheses.
     (Andreas Neumann via Michael Busch)
  9. LUCENE-814: javacc build targets now fix line-end-style of generated files. Now also deleting all javacc generated files before calling javacc.
     (Steven Parkes, Doron Cohen)
  10. LUCENE-829: close readers in contrib/benchmark.
      (Karl Wettin, Doron Cohen)
  11. LUCENE-828: Minor fix for Term's equal().
      (Paul Cowan via Otis Gospodnetic)
  12. LUCENE-846: Fixed: if IndexWriter is opened with autoCommit=false, and you call addIndexes, and hit an exception (eg disk full) then when IndexWriter rolls back its internal state this could corrupt the instance of IndexWriter (but, not the index itself) by referencing already deleted segments. This bug was only present in 2.2 (trunk), ie was never released.
      (Mike McCandless)
  13. LUCENE-736: Sloppy phrase query with repeating terms matches wrong docs. For example query "B C B"~2 matches the doc "A B C D E".
      (Doron Cohen)
  14. LUCENE-789: Fixed: custom similarity is ignored when using MultiSearcher (problem reported by Alexey Lef). Now the similarity applied by MultiSearcer.setSimilarity(sim) is being used. Note that as before this fix, creating a multiSearcher from Searchers for whom custom similarity was set has no effect - it is masked by the similarity of the MultiSearcher. This is as designed, because MultiSearcher operates on Searchables (not Searchers).
      (Doron Cohen)
  15. LUCENE-880: Fixed DocumentWriter to close the TokenStreams after it has written the postings. Then the resources associated with the TokenStreams can safely be released.
      (Michael Busch)
  16. LUCENE-883: consecutive calls to Spellchecker.indexDictionary() won't insert terms twice anymore.
      (Daniel Naber)
  17. LUCENE-881: QueryParser.escape() now also escapes the characters '|' and '&' which are part of the queryparser syntax.
      (Michael Busch)
  18. LUCENE-886: Spellchecker clean up: exceptions aren't printed to STDERR anymore and ignored, but re-thrown. Some javadoc improvements.
      (Daniel Naber)
  19. LUCENE-698: FilteredQuery now takes the query boost into account for scoring.
      (Michael Busch)
  20. LUCENE-763: Spellchecker: LuceneDictionary used to skip first word in enumeration.
      (Christian Mallwitz via Daniel Naber)
  21. LUCENE-903: FilteredQuery explanation inaccuracy with boost. Explanation tests now "deep" check the explanation details.
      (Chris Hostetter, Doron Cohen)
  22. LUCENE-912: DisjunctionMaxScorer first skipTo(target) call ignores the skip target param and ends up at the first match.
      (Sudaakeran B. via Chris Hostetter & Doron Cohen)
  23. LUCENE-913: Two consecutive score() calls return different scores for Boolean Queries.
      (Michael Busch, Doron Cohen)
  24. LUCENE-1013: Fix IndexWriter.setMaxMergeDocs to work "out of the box", again, by moving set/getMaxMergeDocs up from LogDocMergePolicy into LogMergePolicy. This fixes the API breakage (non backwards compatible change) caused by LUCENE-994.
      (Yonik Seeley via Mike McCandless)
• New features   (9)
  1. LUCENE-759: Added two n-gram-producing TokenFilters.
     (Otis Gospodnetic)
  2. LUCENE-822: Added FieldSelector capabilities to Searchable for use with RemoteSearcher, and other Searchable implementations.
     (Mark Miller, Grant Ingersoll)
  3. LUCENE-755: Added the ability to store arbitrary binary metadata in the posting list. These metadata are called Payloads. For every position of a Token one Payload in the form of a variable length byte array can be stored in the prox file. Remark: The APIs introduced with this feature are in experimental state and thus contain appropriate warnings in the javadocs.
     (Michael Busch)
  4. LUCENE-834: Added BoostingTermQuery which can boost scores based on the values of a payload (see #3 above.)
     (Grant Ingersoll)
  5. LUCENE-834: Similarity has a new method for scoring payloads called scorePayloads that can be overridden to take advantage of payload storage (see #3 above)
  6. LUCENE-834: Added isPayloadAvailable() onto TermPositions interface and implemented it in the appropriate places
     (Grant Ingersoll)
  7. LUCENE-853: Added RemoteCachingWrapperFilter to enable caching of Filters on the remote side of the RMI connection.
     (Matt Ericson via Otis Gospodnetic)
  8. LUCENE-446: Added Solr's search.function for scores based on field values, plus CustomScoreQuery for simple score (post) customization.
     (Yonik Seeley, Doron Cohen)
  9. LUCENE-1058: Added new TeeTokenFilter (like the UNIX 'tee' command) and SinkTokenizer which can be used to share tokens between two or more Fields such that the other Fields do not have to go through the whole Analysis process over again. For instance, if you have two Fields that share all the same analysis steps except one lowercases tokens and the other does not, you can coordinate the operations between the two using the TeeTokenFilter and the SinkTokenizer. See TeeSinkTokenTest.java for examples.
     (Grant Ingersoll, Michael Busch, Yonik Seeley)
• Optimizations   (7)
  1. LUCENE-761: The proxStream is now cloned lazily in SegmentTermPositions when nextPosition() is called for the first time. This allows using instances of SegmentTermPositions instead of SegmentTermDocs without additional costs.
     (Michael Busch)
  2. LUCENE-431: RAMInputStream and RAMOutputStream extend IndexInput and IndexOutput directly now. This avoids further buffering and thus avoids unnecessary array copies.
     (Michael Busch)
  3. LUCENE-730: Updated BooleanScorer2 to make use of BooleanScorer in some cases and possibly improve scoring performance. Documents can now be delivered out-of-order as they are scored (e.g. to HitCollector). N.B. A bit of code had to be disabled in QueryUtils in order for TestBoolean2 test to keep passing.
     (Paul Elschot via Otis Gospodnetic)
  4. LUCENE-882: Spellchecker doesn't store the ngrams anymore but only indexes them to keep the spell index small.
     (Daniel Naber)
  5. LUCENE-430: Delay allocation of the buffer after a clone of BufferedIndexInput. Together with LUCENE-888 this will allow to adjust the buffer size dynamically.
     (Paul Elschot, Michael Busch)
  6. LUCENE-888: Increase buffer sizes inside CompoundFileWriter and BufferedIndexOutput. Also increase buffer size in BufferedIndexInput, but only when used during merging. Together, these increases yield 10-18% overall performance gain vs the previous 1K defaults.
     (Mike McCandless)
  7. LUCENE-866: Adds multi-level skip lists to the posting lists. This speeds up most queries that use skipTo(), especially on big indexes with large posting lists. For average AND queries the speedup is about 20%, for queries that contain very frequent and very unique terms the speedup can be over 80%.
     (Michael Busch)
• Documentation   (6)
  1. LUCENE 791 && INFRA-1173: Infrastructure moved the Wiki to http://wiki.apache.org/lucene-java/ Updated the links in the docs and wherever else I found references.
     (Grant Ingersoll, Joe Schaefer)
  2. LUCENE-807: Fixed the javadoc for ScoreDocComparator.compare() to be consistent with java.util.Comparator.compare(): Any integer is allowed to be returned instead of only -1/0/1.
     (Paul Cowan via Michael Busch)
  3. LUCENE-875: Solved javadoc warnings & errors under jdk1.4. Solved javadoc errors under jdk5 (jars in path for gdata). Made "javadocs" target depend on "build-contrib" for first downloading contrib jars configured for dynamic downloaded. (Note: when running behind firewall, a firewall prompt might pop up)
     (Doron Cohen)
  4. LUCENE-740: Added SNOWBALL-LICENSE.txt to the snowball package and a remark about the license to NOTICE.TXT.
     (Steven Parkes via Michael Busch)
  5. LUCENE-925: Added analysis package javadocs.
     (Grant Ingersoll and Doron Cohen)
  6. LUCENE-926: Added document package javadocs.
     (Grant Ingersoll)
• Build   (10)
  1. LUCENE-802: Added LICENSE.TXT and NOTICE.TXT to Lucene jars.
     (Steven Parkes via Michael Busch)
  2. LUCENE-885: "ant test" now includes all contrib tests. The new "ant test-core" target can be used to run only the Core (non contrib) tests.
     (Chris Hostetter)
  3. LUCENE-900: "ant test" now enables Java assertions (in Lucene packages).
     (Doron Cohen)
  4. LUCENE-894: Add custom build file for binary distributions that includes targets to build the demos.
     (Chris Hostetter, Michael Busch)
  5. LUCENE-904: The "package" targets in build.xml now also generate .md5 checksum files.
     (Chris Hostetter, Michael Busch)
  6. LUCENE-907: Include LICENSE.TXT and NOTICE.TXT in the META-INF dirs of demo war, demo jar, and the contrib jars.
     (Michael Busch)
  7. LUCENE-909: Demo targets for running the demo.
     (Doron Cohen)
  8. LUCENE-908: Improves content of MANIFEST file and makes it customizable for the contribs. Adds SNOWBALL-LICENSE.txt to META-INF of the snowball jar and makes sure that the lucli jar contains LICENSE.txt and NOTICE.txt.
     (Chris Hostetter, Michael Busch)
  9. LUCENE-930: Various contrib building improvements to ensure contrib dependencies are met, and test compilation errors fail the build.
     (Steven Parkes, Chris Hostetter)
  10. LUCENE-622: Add ant target and pom.xml files for building maven artifacts of the Lucene core and the contrib modules.
      (Sami Siren, Karl Wettin, Michael Busch)

Release 2.1.0 [2007-02-14]

• Changes in runtime behavior   (9)
  1. 's' and 't' have been removed from the list of default stopwords in StopAnalyzer (also used in by StandardAnalyzer). Having e.g. 's' as a stopword meant that 's-class' led to the same results as 'class'. Note that this problem still exists for 'a', e.g. in 'a-class' as 'a' continues to be a stopword.
     (Daniel Naber)
  2. LUCENE-478: Updated the list of Unicode code point ranges for CJK (now split into CJ and K) in StandardAnalyzer.
     (John Wang and Steven Rowe via Otis Gospodnetic)
  3. Modified some CJK Unicode code point ranges in StandardTokenizer.jj, and added a few more of them to increase CJK character coverage. Also documented some of the ranges.
     (Otis Gospodnetic)
  4. LUCENE-489: Add support for leading wildcard characters (*, ?) to QueryParser. Default is to disallow them, as before.
     (Steven Parkes via Otis Gospodnetic)
  5. LUCENE-703: QueryParser changed to default to use of ConstantScoreRangeQuery for range queries. Added useOldRangeQuery property to QueryParser to allow selection of old RangeQuery class if required.
     (Mark Harwood)
  6. LUCENE-543: WildcardQuery now performs a TermQuery if the provided term does not contain a wildcard character (? or *), when previously a StringIndexOutOfBoundsException was thrown.
     (Michael Busch via Erik Hatcher)
  7. LUCENE-726: Removed the use of deprecated doc.fields() method and Enumeration.
     (Michael Busch via Otis Gospodnetic)
  8. LUCENE-436: Removed finalize() in TermInfosReader and SegmentReader, and added a call to enumerators.remove() in TermInfosReader.close(). The finalize() overrides were added to help with a pre-1.4.2 JVM bug that has since been fixed, plus we no longer support pre-1.4.2 JVMs.
     (Otis Gospodnetic)
  9. LUCENE-771: The default location of the write lock is now the index directory, and is named simply "write.lock" (without a big digest prefix). The system properties "org.apache.lucene.lockDir" nor "java.io.tmpdir" are no longer used as the global directory for storing lock files, and the LOCK_DIR field of FSDirectory is now deprecated.
     (Mike McCandless)
• New features   (15)
  1. LUCENE-503: New ThaiAnalyzer and ThaiWordFilter in contrib/analyzers
     (Samphan Raruenrom via Chris Hostetter)
  2. LUCENE-545: New FieldSelector API and associated changes to IndexReader and implementations. New Fieldable interface for use with the lazy field loading mechanism.
     (Grant Ingersoll and Chuck Williams via Grant Ingersoll)
  3. LUCENE-676: Move Solr's PrefixFilter to Lucene core.
     (Yura Smolsky, Yonik Seeley)
  4. LUCENE-678: Added NativeFSLockFactory, which implements locking using OS native locking (via java.nio.*).
     (Michael McCandless via Yonik Seeley)
  5. LUCENE-544: Added the ability to specify different boosts for different fields when using MultiFieldQueryParser
     (Matt Ericson via Otis Gospodnetic)
  6. LUCENE-528: New IndexWriter.addIndexesNoOptimize() that doesn't optimize the index when adding new segments, only performing merges as needed.
     (Ning Li via Yonik Seeley)
  7. LUCENE-573: QueryParser now allows backslash escaping in quoted terms and phrases.
     (Michael Busch via Yonik Seeley)
  8. LUCENE-716: QueryParser now allows specification of Unicode characters in terms via a unicode escape of the form \uXXXX
     (Michael Busch via Yonik Seeley)
  9. LUCENE-709: Added RAMDirectory.sizeInBytes(), IndexWriter.ramSizeInBytes() and IndexWriter.flushRamSegments(), allowing applications to control the amount of memory used to buffer documents.
     (Chuck Williams via Yonik Seeley)
  10. LUCENE-723: QueryParser now parses *:* as MatchAllDocsQuery
      (Yonik Seeley)
  11. LUCENE-741: Command-line utility for modifying or removing norms on fields in an existing index. This is mostly based on LUCENE-496 and lives in contrib/miscellaneous.
      (Chris Hostetter, Otis Gospodnetic)
  12. LUCENE-759: Added NGramTokenizer and EdgeNGramTokenizer classes and their passing unit tests.
      (Otis Gospodnetic)
  13. LUCENE-565: Added methods to IndexWriter to more efficiently handle updating documents (the "delete then add" use case). This is intended to be an eventual replacement for the existing IndexModifier. Added IndexWriter.flush() (renamed from flushRamSegments()) to flush all pending updates (held in RAM), to the Directory.
      (Ning Li via Mike McCandless)
  14. LUCENE-762: Added in SIZE and SIZE_AND_BREAK FieldSelectorResult options which allow one to retrieve the size of a field without retrieving the actual field.
      (Chuck Williams via Grant Ingersoll)
  15. LUCENE-799: Properly handle lazy, compressed fields.
      (Mike Klaas via Grant Ingersoll)
• API Changes   (19)
  1. LUCENE-438: Remove "final" from Token, implement Cloneable, allow changing of termText via setTermText().
     (Yonik Seeley)
  2. org.apache.lucene.analysis.nl.WordlistLoader has been deprecated and is supposed to be replaced with the WordlistLoader class in package org.apache.lucene.analysis
     (Daniel Naber)
  3. LUCENE-609: Revert return type of Document.getField(s) to Field for backward compatibility, added new Document.getFieldable(s) for access to new lazy loaded fields.
     (Yonik Seeley)
  4. LUCENE-608: Document.fields() has been deprecated and a new method Document.getFields() has been added that returns a List instead of an Enumeration
     (Daniel Naber)
  5. LUCENE-605: New Explanation.isMatch() method and new ComplexExplanation subclass allows explain methods to produce Explanations which model "matching" independent of having a positive value.
     (Chris Hostetter)
  6. LUCENE-621: New static methods IndexWriter.setDefaultWriteLockTimeout and IndexWriter.setDefaultCommitLockTimeout for overriding default timeout values for all future instances of IndexWriter (as well as for any other classes that may reference the static values, ie: IndexReader).
     (Michael McCandless via Chris Hostetter)
  7. LUCENE-638: FSDirectory.list() now only returns the directory's Lucene-related files. Thanks to this change one can now construct a RAMDirectory from a file system directory that contains files not related to Lucene.
     (Simon Willnauer via Daniel Naber)
  8. LUCENE-635: Decoupling locking implementation from Directory implementation. Added set/getLockFactory to Directory and moved all locking code into subclasses of abstract class LockFactory. FSDirectory and RAMDirectory still default to their prior locking implementations, but now you can mix & match, for example using SingleInstanceLockFactory (ie, in memory locking) locking with an FSDirectory. Note that now you must call setDisableLocks before the instantiation a FSDirectory if you wish to disable locking for that Directory.
     (Michael McCandless, Jeff Patterson via Yonik Seeley)
  9. LUCENE-657: Made FuzzyQuery non-final and inner ScoreTerm protected.
     (Steven Parkes via Otis Gospodnetic)
  10. LUCENE-701: Lockless commits: a commit lock is no longer required when a writer commits and a reader opens the index. This includes a change to the index file format (see docs/fileformats.html for details). It also removes all APIs associated with the commit lock & its timeout. Readers are now truly read-only and do not block one another on startup. This is the first step to getting Lucene to work correctly over NFS (second step is LUCENE-710).
      (Mike McCandless)
  11. LUCENE-722: DEFAULT_MIN_DOC_FREQ was misspelled DEFALT_MIN_DOC_FREQ in Similarity's MoreLikeThis class. The misspelling has been replaced by the correct spelling.
      (Andi Vajda via Daniel Naber)
  12. LUCENE-738: Reduce the size of the file that keeps track of which documents are deleted when the number of deleted documents is small. This changes the index file format and cannot be read by previous versions of Lucene.
      (Doron Cohen via Yonik Seeley)
  13. LUCENE-756: Maintain all norms in a single .nrm file to reduce the number of open files and file descriptors for the non-compound index format. This changes the index file format, but maintains the ability to read and update older indices. The first segment merge on an older format index will create a single .nrm file for the new segment.
      (Doron Cohen via Yonik Seeley)
  14. LUCENE-732: DateTools support has been added to QueryParser, with setters for both the default Resolution, and per-field Resolution. For backwards compatibility, DateField is still used if no Resolutions are specified.
      (Michael Busch via Chris Hostetter)
  15. Added isOptimized() method to IndexReader.
      (Otis Gospodnetic)
  16. LUCENE-773: Deprecate the FSDirectory.getDirectory(*) methods that take a boolean "create" argument. Instead you should use IndexWriter's "create" argument to create a new index.
      (Mike McCandless)
  17. LUCENE-780: Add a static Directory.copy() method to copy files from one Directory to another.
      (Jiri Kuhn via Mike McCandless)
  18. LUCENE-773: Added Directory.clearLock(String name) to forcefully remove an old lock. The default implementation is to ask the lockFactory (if non null) to clear the lock.
      (Mike McCandless)
  19. LUCENE-795: Directory.renameFile() has been deprecated as it is not used anymore inside Lucene.
      (Daniel Naber)
• Bug fixes   (32)
  1. Fixed the web application demo (built with "ant war-demo") which didn't work because it used a QueryParser method that had been removed
     (Daniel Naber)
  2. LUCENE-583: ISOLatin1AccentFilter fails to preserve positionIncrement
     (Yonik Seeley)
  3. LUCENE-575: SpellChecker min score is incorrectly changed by suggestSimilar
     (Karl Wettin via Yonik Seeley)
  4. LUCENE-587: Explanation.toHtml was producing malformed HTML
     (Chris Hostetter)
  5. Fix to allow MatchAllDocsQuery to be used with RemoteSearcher
     (Yonik Seeley)
  6. LUCENE-601: RAMDirectory and RAMFile made Serializable
     (Karl Wettin via Otis Gospodnetic)
  7. LUCENE-557: Fixes to BooleanQuery and FilteredQuery so that the score Explanations match up with the real scores.
     (Chris Hostetter)
  8. LUCENE-607: ParallelReader's TermEnum fails to advance properly to new fields
     (Chuck Williams, Christian Kohlschuetter via Yonik Seeley)
  9. LUCENE-610,LUCENE-611: Simple syntax changes to allow compilation with ecj: disambiguate inner class scorer's use of doc() in BooleanScorer2, other test code changes.
     (DM Smith via Yonik Seeley)
  10. LUCENE-451: All core query types now use ComplexExplanations so that boosts of zero don't confuse the BooleanWeight explain method.
      (Chris Hostetter)
  11. LUCENE-593: Fixed LuceneDictionary's inner Iterator
      (Kåre Fiedler Christiansen via Otis Gospodnetic)
  12. LUCENE-641: fixed an off-by-one bug with IndexWriter.setMaxFieldLength()
      (Daniel Naber)
  13. LUCENE-659: Make PerFieldAnalyzerWrapper delegate getPositionIncrementGap() to the correct analyzer for the field.
      (Chuck Williams via Yonik Seeley)
  14. LUCENE-650: Fixed NPE in Locale specific String Sort when Document has no value.
      (Oliver Hutchison via Chris Hostetter)
  15. LUCENE-683: Fixed data corruption when reading lazy loaded fields.
      (Yonik Seeley)
  16. LUCENE-678: Fixed bug in NativeFSLockFactory which caused the same lock to be shared between different directories.
      (Michael McCandless via Yonik Seeley)
  17. LUCENE-690: Fixed thread unsafe use of IndexInput by lazy loaded fields.
      (Yonik Seeley)
  18. LUCENE-696: Fix bug when scorer for DisjunctionMaxQuery has skipTo() called on it before next().
      (Yonik Seeley)
  19. LUCENE-569: Fixed SpanNearQuery bug, for 'inOrder' queries it would fail to recognize ordered spans if they overlapped with unordered spans.
      (Paul Elschot via Chris Hostetter)
  20. LUCENE-706: Updated fileformats.xml|html concerning the docdelta value in the frequency file.
      (Johan Stuyts, Doron Cohen via Grant Ingersoll)
  21. LUCENE-715: Fixed private constructor in IndexWriter.java to properly release the acquired write lock if there is an IOException after acquiring the write lock but before finishing instantiation.
      (Matthew Bogosian via Mike McCandless)
  22. LUCENE-651: Multiple different threads requesting the same FieldCache entry (often for Sorting by a field) at the same time caused multiple generations of that entry, which was detrimental to performance and memory use.
      (Oliver Hutchison via Otis Gospodnetic)
  23. LUCENE-717: Fixed build.xml not to fail when there is no lib dir.
      (Doron Cohen via Otis Gospodnetic)
  24. LUCENE-728: Removed duplicate/old MoreLikeThis and SimilarityQueries classes from contrib/similarity, as their new home is under contrib/queries.
      (Otis Gospodnetic)
  25. LUCENE-669: Do not double-close the RandomAccessFile in FSIndexInput/Output during finalize(). Besides sending an IOException up to the GC, this may also be the cause intermittent "The handle is invalid" IOExceptions on Windows when trying to close readers or writers.
      (Michael Busch via Mike McCandless)
  26. LUCENE-702: Fix IndexWriter.addIndexes(*) to not corrupt the index on any exceptions (eg disk full). The semantics of these methods is now transactional: either all indices are merged or none are. Also fixed IndexWriter.mergeSegments (called outside of addIndexes(*) by addDocument, optimize, flushRamSegments) and IndexReader.commit() (called by close) to clean up and keep the instance state consistent to what's actually in the index
      (Mike McCandless)
  27. LUCENE-129: Change finalizers to do "try {...} finally {super.finalize();}" to make sure we don't miss finalizers in classes above us.
      (Esmond Pitt via Mike McCandless)
  28. LUCENE-754: Fix a problem introduced by LUCENE-651, causing IndexReaders to hang around forever, in addition to not fixing the original FieldCache performance problem.
      (Chris Hostetter, Yonik Seeley)
  29. LUCENE-140: Fix IndexReader.deleteDocument(int docNum) to correctly raise ArrayIndexOutOfBoundsException when docNum is too large. Previously, if docNum was only slightly too large (within the same multiple of 8, ie, up to 7 ints beyond maxDoc), no exception would be raised and instead the index would become silently corrupted. The corruption then only appears much later, in mergeSegments, when the corrupted segment is merged with segment(s) after it.
      (Mike McCandless)
  30. LUCENE-768: Fix case where an Exception during deleteDocument, undeleteAll or setNorm in IndexReader could leave the reader in a state where close() fails to release the write lock.
      (Mike McCandless)
  31. Remove "tvp" from known index file extensions because it is never used.
      (Nicolas Lalevée via Bernhard Messer)
  32. LUCENE-767: Change how SegmentReader.maxDoc() is computed to not rely on file length check and instead use the SegmentInfo's docCount that's already stored explicitly in the index. This is a defensive bug fix (ie, there is no known problem seen "in real life" due to this, just a possible future problem).
      (Chuck Williams via Mike McCandless)
• Optimizations   (16)
  1. LUCENE-586: TermDocs.skipTo() is now more efficient for multi-segment indexes. This will improve the performance of many types of queries against a non-optimized index.
     (Andrew Hudson via Yonik Seeley)
  2. LUCENE-623: RAMDirectory.close now nulls out its reference to all internal "files", allowing them to be GCed even if references to the RAMDirectory itself still exist.
     (Nadav Har'El via Chris Hostetter)
  3. LUCENE-629: Compressed fields are no longer uncompressed and recompressed during segment merges (e.g. during indexing or optimizing), thus improving performance .
     (Michael Busch via Otis Gospodnetic)
  4. LUCENE-388: Improve indexing performance when maxBufferedDocs is large by keeping a count of buffered documents rather than counting after each document addition.
     (Doron Cohen, Paul Smith, Yonik Seeley)
  5. Modified TermScorer.explain to use TermDocs.skipTo() instead of looping through docs.
     (Grant Ingersoll)
  6. LUCENE-672: New indexing segment merge policy flushes all buffered docs to their own segment and delays a merge until mergeFactor segments of a certain level have been accumulated. This increases indexing performance in the presence of deleted docs or partially full segments as well as enabling future optimizations.

     NOTE: this also fixes an "under-merging" bug whereby it is possible to get far too many segments in your index (which will drastically slow down search, risks exhausting file descriptor limit, etc.). This can happen when the number of buffered docs at close, plus the number of docs in the last non-ram segment is greater than mergeFactor.
     (Ning Li, Yonik Seeley)

  7. Lazy loaded fields unnecessarily retained an extra copy of loaded String data.
     (Yonik Seeley)
  8. LUCENE-443: ConjunctionScorer performance increase. Speed up any BooleanQuery with more than one mandatory clause.
     (Abdul Chaudhry, Paul Elschot via Yonik Seeley)
  9. LUCENE-365: DisjunctionSumScorer performance increase of ~30%. Speeds up queries with optional clauses.
     (Paul Elschot via Yonik Seeley)
  10. LUCENE-695: Optimized BufferedIndexInput.readBytes() for medium size buffers, which will speed up merging and retrieving binary and compressed fields.
      (Nadav Har'El via Yonik Seeley)
  11. LUCENE-687: Lazy skipping on proximity file speeds up most queries involving term positions, including phrase queries.
      (Michael Busch via Yonik Seeley)
  12. LUCENE-714: Replaced 2 cases of manual for-loop array copying with calls to System.arraycopy instead, in DocumentWriter.java.
      (Nicolas Lalevee via Mike McCandless)
  13. LUCENE-729: Non-recursive skipTo and next implementation of TermDocs for a MultiReader. The old implementation could recurse up to the number of segments in the index.
      (Yonik Seeley)
  14. LUCENE-739: Improve segment merging performance by reusing the norm array across different fields and doing bulk writes of norms of segments with no deleted docs.
      (Michael Busch via Yonik Seeley)
  15. LUCENE-745: Add BooleanQuery.clauses(), allowing direct access to the List of clauses and replaced the internal synchronized Vector with an unsynchronized List.
      (Yonik Seeley)
  16. LUCENE-750: Remove finalizers from FSIndexOutput and move the FSIndexInput finalizer to the actual file so all clones don't register a new finalizer.
      (Yonik Seeley)
• Test Cases   (3)
  1. Added TestTermScorer.java
     (Grant Ingersoll)
  2. Added TestWindowsMMap.java
     (Benson Margulies via Mike McCandless)
  3. LUCENE-744 Append the user.name property onto the temporary directory that is created so it doesn't interfere with other users.
     (Grant Ingersoll)
• Documentation   (11)
  1. Added style sheet to xdocs named lucene.css and included in the Anakia VSL descriptor.
     (Grant Ingersoll)
  2. Added scoring.xml document into xdocs. Updated Similarity.java scoring formula.(Grant Ingersoll and Steve Rowe. Updates from: Michael McCandless, Doron Cohen, Chris Hostetter, Doug Cutting). Issue 664.
  3. Added javadocs for FieldSelectorResult.java.
     (Grant Ingersoll)
  4. Moved xdocs directory to src/site/src/documentation/content/xdocs per Issue 707. Site now builds using Forrest, just like the other Lucene siblings. See http://wiki.apache.org/jakarta-lucene/HowToUpdateTheWebsite for info on updating the website.
     (Grant Ingersoll with help from Steve Rowe, Chris Hostetter, Doug Cutting, Otis Gospodnetic, Yonik Seeley)
  5. Added in Developer and System Requirements sections under Resources
     (Grant Ingersoll)
  6. LUCENE-713 Updated the Term Vector section of File Formats to include documentation on how Offset and Position info are stored in the TVF file.
     (Grant Ingersoll, Samir Abdou)
  7. Added in link to Clover Test Code Coverage Reports under the Develop section in Resources
     (Grant Ingersoll)
  8. LUCENE-748: Added details for semantics of IndexWriter.close on hitting an Exception.
     (Jed Wesley-Smith via Mike McCandless)
  9. Added some text about what is contained in releases.
     (Eric Haszlakiewicz via Grant Ingersoll)
  10. LUCENE-758: Fix javadoc to clarify that RAMDirectory(Directory) makes a full copy of the starting Directory.
      (Mike McCandless)
  11. LUCENE-764: Fix javadocs to detail temporary space requirements for IndexWriter's optimize(), addIndexes(*) and addDocument(...) methods.
      (Mike McCandless)
• Build   (3)
  1. Added in clover test code coverage per LUCENE-721 To enable clover code coverage, you must have clover.jar in the ANT classpath and specify -Drun.clover=true on the command line.
     (Michael Busch and Grant Ingersoll)
  2. Added a sysproperty in common-build.xml per Lucene 752 to map java.io.tmpdir to ${build.dir}/test just like the tempDir sysproperty.
  3. LUCENE-757 Added new target named init-dist that does setup for distribution of both binary and source distributions. Called by package and package-*-src

Release 2.0.0 [2006-05-26]

• API Changes   (4)
  1. All deprecated methods and fields have been removed, except DateField, which will still be supported for some time so Lucene can read its date fields from old indexes
     (Yonik Seeley & Grant Ingersoll)
  2. DisjunctionSumScorer is no longer public.
     (Paul Elschot via Otis Gospodnetic)
  3. Creating a Field with both an empty name and an empty value now throws an IllegalArgumentException
     (Daniel Naber)
  4. LUCENE-301: Added new IndexWriter({String,File,Directory}, Analyzer) constructors that do not take a boolean "create" argument. These new constructors will create a new index if necessary, else append to the existing one.
     (Dan Armbrust via Mike McCandless)
• New features   (2)
  1. LUCENE-496: Command line tool for modifying the field norms of an existing index; added to contrib/miscellaneous.
     (Chris Hostetter)
  2. LUCENE-577: SweetSpotSimilarity added to contrib/miscellaneous.
     (Chris Hostetter)
• Bug fixes   (16)
  1. LUCENE-330: Fix issue of FilteredQuery not working properly within BooleanQuery.
     (Paul Elschot via Erik Hatcher)
  2. LUCENE-515: Make ConstantScoreRangeQuery and ConstantScoreQuery work with RemoteSearchable.
     (Philippe Laflamme via Yonik Seeley)
  3. Added methods to get/set writeLockTimeout and commitLockTimeout in IndexWriter. These could be set in Lucene 1.4 using a system property. This feature had been removed without adding the corresponding getter/setter methods.
     (Daniel Naber)
  4. LUCENE-413: Fixed ArrayIndexOutOfBoundsException exceptions when using SpanQueries.
     (Paul Elschot via Yonik Seeley)
  5. Implemented FilterIndexReader.getVersion() and isCurrent()
     (Yonik Seeley)
  6. LUCENE-540: Fixed a bug with IndexWriter.addIndexes(Directory[]) that sometimes caused the index order of documents to change.
     (Yonik Seeley)
  7. LUCENE-526: Fixed a bug in FieldSortedHitQueue that caused subsequent String sorts with different locales to sort identically.
     (Paul Cowan via Yonik Seeley)
  8. LUCENE-541: Add missing extractTerms() to DisjunctionMaxQuery
     (Stefan Will via Yonik Seeley)
  9. LUCENE-514: Added getTermArrays() and extractTerms() to MultiPhraseQuery
     (Eric Jain & Yonik Seeley)
  10. LUCENE-512: Fixed ClassCastException in ParallelReader.getTermFreqVectors
      (frederic via Yonik)
  11. LUCENE-352: Fixed bug in SpanNotQuery that manifested as NullPointerException when "exclude" query was not a SpanTermQuery.
      (Chris Hostetter)
  12. LUCENE-572: Fixed bug in SpanNotQuery hashCode, was ignoring exclude clause
      (Chris Hostetter)
  13. LUCENE-561: Fixed some ParallelReader bugs. NullPointerException if the reader didn't know about the field yet, reader didn't keep track if it had deletions, and deleteDocument calls could circumvent synchronization on the subreaders.
      (Chuck Williams via Yonik Seeley)
  14. LUCENE-556: Added empty extractTerms() implementation to MatchAllDocsQuery and ConstantScoreQuery in order to allow their use with a MultiSearcher.
      (Yonik Seeley)
  15. LUCENE-546: Removed 2GB file size limitations for RAMDirectory.
      (Peter Royal, Michael Chan, Yonik Seeley)
  16. LUCENE-485: Don't hold commit lock while removing obsolete index files.
      (Luc Vanlerberghe via cutting)

Release 1.9.1 [2006-03-02]

• Bug fixes   (1)
  1. LUCENE-511: Fix a bug in the BufferedIndexOutput optimization introduced in 1.9-final.
     (Shay Banon & Steven Tamm via cutting)

Release 1.9 final [2006-02-27]

• Note that this release is mostly but not 100% source compatible with the previous release of Lucene (1.4.3). In other words, you should make sure your application compiles with this version of Lucene before you replace the old Lucene JAR with the new one. Many methods have been deprecated in anticipation of release 2.0, so deprecation warnings are to be expected when upgrading from 1.4.3 to 1.9.
1. The fix that made IndexWriter.setMaxBufferedDocs(1) work had negative effects on indexing performance and has thus been reverted. The argument for setMaxBufferedDocs(int) must now at least be 2, otherwise an exception is thrown.
   (Daniel Naber)
1. Optimized BufferedIndexOutput.writeBytes() to use System.arraycopy() in more cases, rather than copying byte-by-byte.
   (Lukas Zapletal via Cutting)

Release 1.9 RC1 [2006-02-21]

• Requirements   (1)
  1. To compile and use Lucene you now need Java 1.4 or later.
• Changes in runtime behavior   (9)
  1. FuzzyQuery can no longer throw a TooManyClauses exception. If a FuzzyQuery expands to more than BooleanQuery.maxClauseCount terms only the BooleanQuery.maxClauseCount most similar terms go into the rewritten query and thus the exception is avoided.
     (Christoph)
  2. Changed system property from "org.apache.lucene.lockdir" to "org.apache.lucene.lockDir", so that its casing follows the existing pattern used in other Lucene system properties.
     (Bernhard)
  3. The terms of RangeQueries and FuzzyQueries are now converted to lowercase by default (as it has been the case for PrefixQueries and WildcardQueries before). Use setLowercaseExpandedTerms(false) to disable that behavior but note that this also affects PrefixQueries and WildcardQueries.
     (Daniel Naber)
  4. Document frequency that is computed when MultiSearcher is used is now computed correctly and "globally" across subsearchers and indices, while before it used to be computed locally to each index, which caused ranking across multiple indices not to be equivalent.
     (Chuck Williams, Wolf Siberski via Otis, bug #31841 [LUCENE-295])
  5. When opening an IndexWriter with create=true, Lucene now only deletes its own files from the index directory (looking at the file name suffixes to decide if a file belongs to Lucene). The old behavior was to delete all files.
     (Daniel Naber and Bernhard Messer, bug #34695 [LUCENE-385])
  6. The version of an IndexReader, as returned by getCurrentVersion() and getVersion() doesn't start at 0 anymore for new indexes. Instead, it is now initialized by the system time in milliseconds.
     (Bernhard Messer via Daniel Naber)
  7. Several default values cannot be set via system properties anymore, as this has been considered inappropriate for a library like Lucene. For most properties there are set/get methods available in IndexWriter which you should use instead. This affects the following properties: See IndexWriter for getter/setter methods: org.apache.lucene.writeLockTimeout, org.apache.lucene.commitLockTimeout, org.apache.lucene.minMergeDocs, org.apache.lucene.maxMergeDocs, org.apache.lucene.maxFieldLength, org.apache.lucene.termIndexInterval, org.apache.lucene.mergeFactor, See BooleanQuery for getter/setter methods: org.apache.lucene.maxClauseCount See FSDirectory for getter/setter methods: disableLuceneLocks
     (Daniel Naber)
  8. Fixed FieldCacheImpl to use user-provided IntParser and FloatParser, instead of using Integer and Float classes for parsing.
     (Yonik Seeley via Otis Gospodnetic)
  9. Expert level search routines returning TopDocs and TopFieldDocs no longer normalize scores. This also fixes bugs related to MultiSearchers and score sorting/normalization.
     (Luc Vanlerberghe via Yonik Seeley, LUCENE-469)
• New features   (33)
  1. Added support for stored compressed fields (patch #31149 [LUCENE-274])
     (Bernhard Messer via Christoph)
  2. Added support for binary stored fields (patch #29370 [LUCENE-229])
     (Drew Farris and Bernhard Messer via Christoph)
  3. Added support for position and offset information in term vectors (patch #18927 [LUCENE-95]).
     (Grant Ingersoll & Christoph)
  4. A new class DateTools has been added. It allows you to format dates in a readable format adequate for indexing. Unlike the existing DateField class DateTools can cope with dates before 1970 and it forces you to specify the desired date resolution (e.g. month, day, second, ...) which can make RangeQuerys on those fields more efficient.
     (Daniel Naber)
  5. QueryParser now correctly works with Analyzers that can return more than one token per position. For example, a query "+fast +car" would be parsed as "+fast +(car automobile)" if the Analyzer returns "car" and "automobile" at the same position whenever it finds "car" (Patch #23307 [LUCENE-133]).
     (Pierrick Brihaye, Daniel Naber)
  6. Permit unbuffered Directory implementations (e.g., using mmap). InputStream is replaced by the new classes IndexInput and BufferedIndexInput. OutputStream is replaced by the new classes IndexOutput and BufferedIndexOutput. InputStream and OutputStream are now deprecated and FSDirectory is now subclassable.
     (cutting)
  7. Add native Directory and TermDocs implementations that work under GCJ. These require GCC 3.4.0 or later and have only been tested on Linux. Use 'ant gcj' to build demo applications.
     (cutting)
  8. Add MMapDirectory, which uses nio to mmap input files. This is still somewhat slower than FSDirectory. However it uses less memory per query term, since a new buffer is not allocated per term, which may help applications which use, e.g., wildcard queries. It may also someday be faster.
     (cutting & Paul Elschot)
  9. Added javadocs-internal to build.xml - bug #30360 [LUCENE-250]
     (Paul Elschot via Otis)
  10. Added RangeFilter, a more generically useful filter than DateFilter.
      (Chris M Hostetter via Erik)
  11. Added NumberTools, a utility class indexing numeric fields.
      (adapted from code contributed by Matt Quail; committed by Erik)
  12. Added public static IndexReader.main(String[] args) method. IndexReader can now be used directly at command line level to list and optionally extract the individual files from an existing compound index file.
      (adapted from code contributed by Garrett Rooney; committed by Bernhard)
  13. Add IndexWriter.setTermIndexInterval() method. See javadocs.
      (Doug Cutting)
  14. Added LucenePackage, whose static get() method returns java.util.Package, which lets the caller get the Lucene version information specified in the Lucene Jar.
      (Doug Cutting via Otis)
  15. Added Hits.iterator() method and corresponding HitIterator and Hit objects. This provides standard java.util.Iterator iteration over Hits. Each call to the iterator's next() method returns a Hit object.
      (Jeremy Rayner via Erik)
  16. Add ParallelReader, an IndexReader that combines separate indexes over different fields into a single virtual index.
      (Doug Cutting)
  17. Add IntParser and FloatParser interfaces to FieldCache, so that fields in arbitrarily formats can be cached as ints and floats.
      (Doug Cutting)
  18. Added class org.apache.lucene.index.IndexModifier which combines IndexWriter and IndexReader, so you can add and delete documents without worrying about synchronization/locking issues.
      (Daniel Naber)
  19. Lucene can now be used inside an unsigned applet, as Lucene's access to system properties will not cause a SecurityException anymore.
      (Jon Schuster via Daniel Naber, bug #34359 [LUCENE-369])
  20. Added a new class MatchAllDocsQuery that matches all documents.
      (John Wang via Daniel Naber, bug #34946 [LUCENE-389])
  21. Added ability to omit norms on a per field basis to decrease index size and memory consumption when there are many indexed fields. See Field.setOmitNorms()
      (Yonik Seeley, LUCENE-448)
  22. Added NullFragmenter to contrib/highlighter, which is useful for highlighting entire documents or fields.
      (Erik Hatcher)
  23. Added regular expression queries, RegexQuery and SpanRegexQuery. Note the same term enumeration caveats apply with these queries as apply to WildcardQuery and other term expanding queries. These two new queries are not currently supported via QueryParser.
      (Erik Hatcher)
  24. Added ConstantScoreQuery which wraps a filter and produces a score equal to the query boost for every matching document.
      (Yonik Seeley, LUCENE-383)
  25. Added ConstantScoreRangeQuery which produces a constant score for every document in the range. One advantage over a normal RangeQuery is that it doesn't expand to a BooleanQuery and thus doesn't have a maximum number of terms the range can cover. Both endpoints may also be open.
      (Yonik Seeley, LUCENE-383)
  26. Added ability to specify a minimum number of optional clauses that must match in a BooleanQuery. See BooleanQuery.setMinimumNumberShouldMatch().
      (Paul Elschot, Chris Hostetter via Yonik Seeley, LUCENE-395)
  27. Added DisjunctionMaxQuery which provides the maximum score across its clauses. It's very useful for searching across multiple fields.
      (Chuck Williams via Yonik Seeley, LUCENE-323)
  28. New class ISOLatin1AccentFilter that replaces accented characters in the ISO Latin 1 character set by their unaccented equivalent.
      (Sven Duzont via Erik Hatcher)
  29. New class KeywordAnalyzer. "Tokenizes" the entire stream as a single token. This is useful for data like zip codes, ids, and some product names.
      (Erik Hatcher)
  30. Copied LengthFilter from contrib area to core. Removes words that are too long and too short from the stream.
      (David Spencer via Otis and Daniel)
  31. Added getPositionIncrementGap(String fieldName) to Analyzer. This allows custom analyzers to put gaps between Field instances with the same field name, preventing phrase or span queries crossing these boundaries. The default implementation issues a gap of 0, allowing the default token position increment of 1 to put the next field's first token into a successive position.
      (Erik Hatcher, with advice from Yonik)
  32. StopFilter can now ignore case when checking for stop words.
      (Grant Ingersoll via Yonik, LUCENE-248)
  33. Add TopDocCollector and TopFieldDocCollector. These simplify the implementation of hit collectors that collect only the top-scoring or top-sorting hits.
• API Changes   (5)
  1. Several methods and fields have been deprecated. The API documentation contains information about the recommended replacements. It is planned that most of the deprecated methods and fields will be removed in Lucene 2.0.
     (Daniel Naber)
  2. The Russian and the German analyzers have been moved to contrib/analyzers. Also, the WordlistLoader class has been moved one level up in the hierarchy and is now org.apache.lucene.analysis.WordlistLoader
     (Daniel Naber)
  3. The API contained methods that declared to throw an IOException but that never did this. These declarations have been removed. If your code tries to catch these exceptions you might need to remove those catch clauses to avoid compile errors.
     (Daniel Naber)
  4. Add a serializable Parameter Class to standardize parameter enum classes in BooleanClause and Field.
     (Christoph)
  5. Added rewrite methods to all SpanQuery subclasses that nest other SpanQuerys. This allows custom SpanQuery subclasses that rewrite (for term expansion, for example) to nest within the built-in SpanQuery classes successfully.
• Bug fixes   (24)
  1. The JSP demo page (src/jsp/results.jsp) now properly closes the IndexSearcher it opens.
     (Daniel Naber)
  2. Fixed a bug in IndexWriter.addIndexes(IndexReader[] readers) that prevented deletion of obsolete segments.
     (Christoph Goller)
  3. Fix in FieldInfos to avoid the return of an extra blank field in IndexReader.getFieldNames() (Patch #19058 [LUCENE-102]).
     (Mark Harwood via Bernhard)
  4. Some combinations of BooleanQuery and MultiPhraseQuery (formerly PhrasePrefixQuery) could provoke UnsupportedOperationException (bug #33161 [LUCENE-337]).
     (Rhett Sutphin via Daniel Naber)
  5. Small bug in skipTo of ConjunctionScorer that caused NullPointerException if skipTo() was called without prior call to next() fixed.
     (Christoph)
  6. Disable Similiarty.coord() in the scoring of most automatically generated boolean queries. The coord() score factor is appropriate when clauses are independently specified by a user, but is usually not appropriate when clauses are generated automatically, e.g., by a fuzzy, wildcard or range query. Matches on such automatically generated queries are no longer penalized for not matching all terms.
     (Doug Cutting, Patch #33472 [LUCENE-346])
  7. Getting a lock file with Lock.obtain(long) was supposed to wait for a given amount of milliseconds, but this didn't work.
     (John Wang via Daniel Naber, Bug #33799 [LUCENE-353])
  8. Fix FSDirectory.createOutput() to always create new files. Previously, existing files were overwritten, and an index could be corrupted when the old version of a file was longer than the new. Now any existing file is first removed.
     (Doug Cutting)
  9. Fix BooleanQuery containing nested SpanTermQuery's, which previously could return an incorrect number of hits.
     (Reece Wilton via Erik Hatcher, Bug #35157 [LUCENE-393])
  10. Fix NullPointerException that could occur with a MultiPhraseQuery inside a BooleanQuery.
      (Hans Hjelm and Scotty Allen via Daniel Naber, Bug #35626 [LUCENE-404])
  11. Fixed SnowballFilter to pass through the position increment from the original token.
      (Yonik Seeley via Erik Hatcher, LUCENE-437)
  12. Added Unicode range of Korean characters to StandardTokenizer, grouping contiguous characters into a token rather than one token per character. This change also changes the token type to "<CJ>" for Chinese and Japanese character tokens (previously it was "<CJK>").
      (Cheolgoo Kang via Otis and Erik, LUCENE-444 and LUCENE-461)
  13. FieldsReader now looks at FieldInfo.storeOffsetWithTermVector and FieldInfo.storePositionWithTermVector and creates the Field with correct TermVector parameter.
      (Frank Steinmann via Bernhard, LUCENE-455)
  14. Fixed WildcardQuery to prevent "cat" matching "ca??".
      (Xiaozheng Ma via Bernhard, LUCENE-306)