<commit_before>LanguageTool Change Log

2.4-SNAPSHOT (release planned for 2013-12-30)

-Breton:
	-SRX sentence tokenization
	-added/improved a few rules
	-fixed some false alarms
	-fixed incorrect suggestions thanks to added tests on corrections

-Catalan:
	-added/improved several rules
	-fixed some false alarms
	-made some additions and fixes to the tagger dictionary
	-removed some words from synthesis dictionary (see filter-archaic.txt)

-English:
	-added/improved a few rules
	-fixed some false alarms

-French:
	-added/improved several rules
	-fixed some false alarms

-German:
	-added/improved several rules
	-added a synthesizer - the agreement rule can now make suggestions
	 for some errors (not all suggestions are correct, though)

-Polish:
	-added/improved several rules
	-fixed some false alarms

-Portuguese:
	-added/improved several rules (it now includes gender rules "a"/"o")
	-it now has 3400+ compound words

-the JAR file has been renamed to languagetool.jar, from formerly
 languagetool-standalone.jar to avoid confusion about what 'standalone'
 means in this context (github issue #29)

-for languages with many rules (like French or German) performance
 on long texts has been increased by about 20-30%

-fix for thread-safety (could cause hang in MultiWordChunker)

-fixed a bug where chunk annotations were not tested in <and> groups

-fix: \1 and <match no="..."/> had not been evaluated in <short>...</short>

-fixed a bug in the unification mechanism that discarded some of the matching
 interpretations prematurely

-added support for chunk annotations in the disambiguator, and fixed one bug
 in filtering tokens with chunk annotations

-new command line tools to export and create binary dictionaries:
	org.languagetool.dev.DictionaryExporter
	org.languagetool.dev.POSDictionaryBuilder

-LibreOffice/OpenOffice integration:
	-added a workaround for incorrect sentence detection for the case that a
		footnote appeared after a sentence full stop (Sourceforge bug #191) 

-stand-alone GUI:
	-The dialog opened by the "More..." item in the context menu of an error
	 will now also display correct and incorrect example sentences

-API:
	-SentenceTokenizer is now an interface, the implementation has been moved to
		RegexSentenceTokenizer, but this is deprecated and SRXSentenceTokenizer
		should be used instead
	-Some methods from org.languagetool.tools.StringTools have been moved to
	 the org.languagetool.gui.Tools class in the languagetool-gui-commons project
	-LanguageToolListener.languageToolEventOccured() has been renamed to
	 LanguageToolListener.languageToolEventOccurred()
	-org.languagetool.tools.SymbolLocator isn't public anymore (shouldn't affect anybody)
	-removed DanishSentenceTokenizer which had been deprecated for three years
	-Rule.getCorrectExamples() and Rule.getIncorrectExamples() don't return null anymore
	 but an empty list if there are no examples. Consequently, setCorrectExamples() and
	 setIncorrectExamples() don't accept null anymore.
	-Rule.getId() may return any string now, not just ASCII-only strings (actually this
	 has been the case before, as the ASCII-only restriction was never enforced and only
	 mentioned in the javadoc)

-languagetool-wikipedia: the command line options for checking a Wikipedia dump
	have been simplified. The command can now be called like this:
	java -jar languagetool-wikipedia.jar check-data -l en -f enwiki-20130621-pages-articles.xml
	Call just "java -jar languagetool-wikipedia.jar check-data" to get a usage message.
	More than one file can be specified with the -f option. Additionally to Wikipedia
	XML dumps, CSV files from Tatoeba (http://tatoeba.org) are now also supported,
	they need to be filtered first to contain only the relevant language.

-Updated morfologik libraries to 1.8.1 (bug fixes, stricter input sanity checking)


2.3.1 (2013-10-07, released on Maven Central only)

	-fixes for thread-safety


2.3 (2013-09-30)

-Breton:
	-added/improved a few rules
	-fixed false alarms
	-updated POS dictionary from Apertium (svn r47282)

-Catalan:
	-added support for language code ca-ES-valencia (Catalan Valencian),
	 to be used in LibreOffice 4.2.0<commit_msg>Document changes: use of frequency wordlists<commit_after>LanguageTool Change Log

2.4-SNAPSHOT (release planned for 2013-12-30)

-Breton:
	-SRX sentence tokenization
	-added/improved a few rules
	-fixed some false alarms
	-fixed incorrect suggestions thanks to added tests on corrections

-Catalan:
	-added/improved several rules
	-fixed false alarms
	-made additions and fixes to the tagger dictionary
	-removed some words from synthesis dictionary (see filter-archaic.txt)
	-added frequency data to the tagger dictionary; frequency wordlist comes from the Gaia 
	project, with Apache License, version 2.0
	(https://github.com/mozilla-b2g/gaia/tree/master/keyboard/dictionaries).

-English:
	-added/improved a few rules
	-fixed some false alarms

-French:
	-added/improved several rules
	-fixed some false alarms

-German:
	-added/improved several rules
	-added a synthesizer - the agreement rule can now make suggestions
	 for some errors (not all suggestions are correct, though)

-Polish:
	-added/improved several rules
	-fixed some false alarms

-Portuguese:
	-added/improved several rules (it now includes gender rules "a"/"o")
	-it now has 3400+ compound words

-the JAR file has been renamed to languagetool.jar, from formerly
 languagetool-standalone.jar to avoid confusion about what 'standalone'
 means in this context (github issue #29)

-for languages with many rules (like French or German) performance
 on long texts has been increased by about 20-30%

-fix for thread-safety (could cause hang in MultiWordChunker)

-fixed a bug where chunk annotations were not tested in <and> groups

-fix: \1 and <match no="..."/> had not been evaluated in <short>...</short>

-fixed a bug in the unification mechanism that discarded some of the matching
 interpretations prematurely

-added support for chunk annotations in the disambiguator, and fixed one bug
 in filtering tokens with chunk annotations

-new command line tools to export and create binary dictionaries:
	org.languagetool.dev.DictionaryExporter
	org.languagetool.dev.POSDictionaryBuilder

-LibreOffice/OpenOffice integration:
	-added a workaround for incorrect sentence detection for the case that a
		footnote appeared after a sentence full stop (Sourceforge bug #191) 

-stand-alone GUI:
	-The dialog opened by the "More..." item in the context menu of an error
	 will now also display correct and incorrect example sentences
	 
-stand-alone dev:
    -Added the option of including frequency data to taggging or spelling dictionaries. 
    The expected format of the frequency wordlists is the one in the Gaia 
	project, with Apache License, version 2.0
	(https://github.com/mozilla-b2g/gaia/tree/master/keyboard/dictionaries)

-API:
	-SentenceTokenizer is now an interface, the implementation has been moved to
		RegexSentenceTokenizer, but this is deprecated and SRXSentenceTokenizer
		should be used instead
	-Some methods from org.languagetool.tools.StringTools have been moved to
	 the org.languagetool.gui.Tools class in the languagetool-gui-commons project
	-LanguageToolListener.languageToolEventOccured() has been renamed to
	 LanguageToolListener.languageToolEventOccurred()
	-org.languagetool.tools.SymbolLocator isn't public anymore (shouldn't affect anybody)
	-removed DanishSentenceTokenizer which had been deprecated for three years
	-Rule.getCorrectExamples() and Rule.getIncorrectExamples() don't return null anymore
	 but an empty list if there are no examples. Consequently, setCorrectExamples() and
	 setIncorrectExamples() don't accept null anymore.
	-Rule.getId() may return any string now, not just ASCII-only strings (actually this
	 has been the case before, as the ASCII-only restriction was never enforced and only
	 mentioned in the javadoc)

-languagetool-wikipedia: the command line options for checking a Wikipedia dump
	have been simplified. The command can now be called like this:
	java -jar languagetool-wikipedia.jar check-data -l en -f enwiki-20130621-pages-articles.xml
	Call just "java -jar languagetool-wikipedia.jar check-data" to get a usage message.
	More than one file can be specified with the -f option. Additionally to Wikipedia
	XML dumps, CSV files from Tatoeba (http://tatoeba.org) are now also supported,
	they need to be filtered first to contain only the relevant language.

-Updated Morfologik libraries to 1.8.2 (bug fixes, stricter input sanity checking, add frequency 
    data to dictionaries)


2.3.1 (2013-10-07, released on Maven Central only)

	-fixes for thread-safety


2.3 (2013-09-30)

-Breton:
	-added/improved a few rules
	-fixed false alarms
	-updated POS dictionary from Apertium (svn r47282)

-Catalan:
	-added support for language code ca-ES-valencia (Catalan Valencian),
	 to be used in LibreOffice 4.2.0
{'fraction_non_alphanumeric': 0.06292688388758258, 'fraction_numerical': 0.015003918934049938, 'mean_word_length': 4.3325373134328355, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4422119', 'n_tokens_mistral': 2730, 'n_tokens_neox': 2478, 'n_words': 1121}