<commit_before>Apache Lucene
Copyright 2006 The Apache Software Foundation

This product includes software developed by
The Apache Software Foundation (http://www.apache.org/).

The snowball stemmers in
  contrib/snowball/src/java/net/sf/snowball
were developed by <NAME> and <NAME>.
The full snowball package is available from
  http://snowball.tartarus.org/

The Arabic stemmer (contrib/analyzer) comes with a default
stopword list that is BSD-licensed created by <NAME>.  The file resides in 
contrib/analyzers/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt.
See http://members.unine.ch/jacques.savoy/clef/index.html.

Includes lib/servlet-api-2.4.jar from  Apache Tomcat

The SmartChineseAnalyzer source code (under contrib/analyzers) was
provided by Xiaop<NAME>ao and copyright 2009 by www.imdict.net.<commit_msg>add ICU4J to notice.txt<commit_after>Apache Lucene
Copyright 2006 The Apache Software Foundation

This product includes software developed by
The Apache Software Foundation (http://www.apache.org/).

The snowball stemmers in
  contrib/snowball/src/java/net/sf/snowball
were developed by <NAME> and <NAME>.
The full snowball package is available from
  http://snowball.tartarus.org/

The Arabic stemmer (contrib/analyzer) comes with a default
stopword list that is BSD-licensed created by <NAME>.  The file resides in 
contrib/analyzers/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt.
See http://members.unine.ch/jacques.savoy/clef/index.html.

Includes lib/servlet-api-2.4.jar from  Apache Tomcat

The SmartChineseAnalyzer source code (under contrib/analyzers) was
provided by <NAME> and copyright 2009 by www.imdict.net.

ICU4J, (under contrib/collation) is licensed under an MIT styles license
(contrib/collation/lib/ICU-LICENSE.txt) and Copyright (c) 1995-2008 
International Business Machines Corporation and others
{'fraction_non_alphanumeric': 0.08489054991991457, 'fraction_numerical': 0.01601708489054992, 'mean_word_length': 4.801857585139319, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1002002', 'n_tokens_mistral': 609, 'n_tokens_neox': 559, 'n_words': 185}