<commit_before>1.4.1-SNAPSHOT
- can specify weighting schemes per field
- added TestWeightingSchemes
- can remap the attribute numbers before generating the vector file
1.4
- added XMLCorpusReader and XMLCorpusClassifier which generate a raw file from a XML corpus or classify a XML corpus using an existing model
- added java implementation of liblinear
- added toString() method on Fields
- bugfix : tokenforms excessively normalised in class Lexicon - was losing non european characters
- TextClassifier can take unzip working directory to /tmp before using it for classifying
1.3
- added CorpusUtils class for generating vector file from raw file, generate subset of raw file, filter fields from raw file, display best attributes
- Bug fix : Lexicon differenciates between attribute number and highest attribute ID
- added object attributescorer and refactored logLikelihoodAttributeFilter to return a Scorer object. This Scorer is to filter a lexicon
- normalise attribute forms to prevent \n or \t in lexicon file
1.2
- made setlogLikelihoodratio() and keepTopNAttributesLLR() public
- uses Enums for Parameters.WeightingMethods instead of list of static ints
- Parameters.WeightingMethods  toString / methodFromString throw RunTimeException if value is not known<commit_msg>added bugfix to list of changes<commit_after>1.4.1-SNAPSHOT
- can specify weighting schemes per field
- added TestWeightingSchemes
- can remap the attribute numbers before generating the vector file
- bugfix : could not use SimpleDocuments with a file backed corpus (jiuren)
1.4
- added XMLCorpusReader and XMLCorpusClassifier which generate a raw file from a XML corpus or classify a XML corpus using an existing model
- added java implementation of liblinear
- added toString() method on Fields
- bugfix : tokenforms excessively normalised in class Lexicon - was losing non european characters
- TextClassifier can take unzip working directory to /tmp before using it for classifying
1.3
- added CorpusUtils class for generating vector file from raw file, generate subset of raw file, filter fields from raw file, display best attributes
- Bug fix : Lexicon differenciates between attribute number and highest attribute ID
- added object attributescorer and refactored logLikelihoodAttributeFilter to return a Scorer object. This Scorer is to filter a lexicon
- normalise attribute forms to prevent \n or \t in lexicon file
1.2
- made setlogLikelihoodratio() and keepTopNAttributesLLR() public
- uses Enums for Parameters.WeightingMethods instead of list of static ints
- Parameters.WeightingMethods  toString / methodFromString throw RunTimeException if value is not known
{'fraction_non_alphanumeric': 0.0338855421686747, 'fraction_numerical': 0.00677710843373494, 'mean_word_length': 4.904444444444445, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3158590', 'n_tokens_mistral': 650, 'n_tokens_neox': 635, 'n_words': 349}