<commit_before>charaparser-unsupervised
========================

Java implementation of unsupervised CharaParser code

* Read Treatment
    * Preprocess
        * First replace all dot marks after abbreviation by "[DOT]". After sentence segmentation, restore them.
    * Sentence segmentation
        * Handle abbreviation. 
    * Get Lead of Sentence
    In most cases, the lead of a sentence is just the first NUM_LEAD_WORDS words in the sentence, where NUM_LEAD_WORDS has been pre-define as 3. However, there are two expections:
    1) When there is any :;.[(] in the sentence, only the words before any ,:;\.\[(] are counted. For example, for the sentence
        word1 word2 . word3
    only
        word1 word2
    are kept.
    2) Whene there is any preposition word in the sentence, only the words before the preposition word and the preposition word are counted. For example, for the sentence
        lepidotrichia of fin webs
    only the word before the preposition word "of" and the preposition word "of"
        lepidotrichia of
    are kept.
    
* Get All Unique Words




The regular expression like "blv?d" would match both "blvd" and "bld". The question mark makes the previous letter optional.


* Add Heuristics Nouns
** Learn Heuristics Nouns

For each noun, 

** Character Heuristics

* Add Stop Words

* Add Characters

* Add Numbers

* Add Cluster Strings

* Add Proper Nouns

* Learn POS By Suffix

* Markup Sentence By Pattern

* Markup Ignore
    This one is not used here.
    
* Learning rules with high certainty
    Discover with parameter "start"
    
* discover
	with parameter "start"
	
	For each sentence whose status is "start" and tag is not "ignore" and "null", 

	At the every beginning, only those sentence whose first word is a p, could have a tag of "start", see populateSentece section.
	
	* Build pattern from the lead words of the sentence
	* Find those sentences which match to this pattern
		For each of those sentences,
		* do rule based learn, all the way to the point where no further new knowledge can be learned. Then stop.

* discover
	Bootstrapping rules
	Discover with parameter "normal"  
	Same to the previous step, except that the parameter is "normal" instead of "start".  
        


* Manage Data Holder
** Update Table
If the pos is noun, find whether its is a singular or plural.

*** Mark Known Word

**** Precess New Word

Update Unknown Word
Put (word, tag) into UknownWord holder<commit_msg>Added new content into readme file.<commit_after>charaparser-unsupervised
========================

Java implementation of unsupervised CharaParser code

* Read Treatment
    * Preprocess
        * First replace all dot marks after abbreviation by "[DOT]". After sentence segmentation, restore them.
    * Sentence segmentation
        * Handle abbreviation. 
    * Get Lead of Sentence
    In most cases, the lead of a sentence is just the first NUM_LEAD_WORDS words in the sentence, where NUM_LEAD_WORDS has been pre-define as 3. However, there are two expections:
    1) When there is any :;.[(] in the sentence, only the words before any ,:;\.\[(] are counted. For example, for the sentence
        word1 word2 . word3
    only
        word1 word2
    are kept.
    2) Whene there is any preposition word in the sentence, only the words before the preposition word and the preposition word are counted. For example, for the sentence
        lepidotrichia of fin webs
    only the word before the preposition word "of" and the preposition word "of"
        lepidotrichia of
    are kept.
    
* Get All Unique Words




The regular expression like "blv?d" would match both "blvd" and "bld". The question mark makes the previous letter optional.


* Add Heuristics Nouns
** Learn Heuristics Nouns

For each noun, 

** Character Heuristics

* Add Stop Words

* Add Characters

* Add Numbers

* Add Cluster Strings

* Add Proper Nouns

* Learn POS By Suffix

* Markup Sentence By Pattern

* Markup Ignore
    This one is not used here.
    
* Learning rules with high certainty
    Discover with parameter "start"
    
* discover
	with parameter "start"
	
	For each sentence whose status is "start" and tag is not "ignore" and "null", 

	At the every beginning, only those sentence whose first word is a p, could have a tag of "start", see populateSentece section.
	
	** Build pattern from the lead words of the sentence
	** Find those sentences which match to this pattern
		For each of those sentences, do rule based learn, all the way to the point where no further new knowledge can be learned. Then stop.
        *** In rule based learn, 
        1) grow NOUNS and BDRY, and confirm tags by applying rules and clues 
        2) create and maintain decision tables
        
        for each marked sentence:
        	doit: 
        	update wordpos table (on certainty) when a sentence is tagged for the first time.
			Note: 	1) this update should not be done when a pos is looked up, because we may lookup a pos for the same example multiple times.
					2) if the tag need to be adjusted (not by doit function), also need to adjust certainty counts.
        		case 1 - case 10
        	tagit

* discover
	Bootstrapping rules
	Discover with parameter "normal"  
	Same to the previous step, except that the parameter is "normal" instead of "start".  
        


* Manage Data Holder
** Update Table
If the pos is noun, find whether its is a singular or plural.

*** Mark Known Word

**** Precess New Word

Update Unknown Word
Put (word, tag) into UknownWord holder
{'fraction_non_alphanumeric': 0.05973931933381608, 'fraction_numerical': 0.004163649529326575, 'mean_word_length': 3.1635267520723436, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3122961', 'n_tokens_mistral': 1567, 'n_tokens_neox': 1461, 'n_words': 787}