<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Jump start1.1&nbsp;&nbsp;Basic settings and imports1.2&nbsp;&nbsp;Input: "parses" ???1.3&nbsp;&nbsp;LEARN!2&nbsp;&nbsp;Response: re = learn_grammar(**kwargs)2.1&nbsp;&nbsp;Corpus statistics: re['corpus_stats']  ⇒  corpus_stats.txt file2.2&nbsp;&nbsp;Response beyond corpus_stats2.3&nbsp;&nbsp;Category tree: re['cat_tree_file']  ⇒  cat_tree.txt file2.4&nbsp;&nbsp;Link Grammar file: re['grammar_file']  ⇒  dict...dict file3&nbsp;&nbsp;More Turtle4&nbsp;&nbsp;Grammar Learner parameters: kwargs5&nbsp;&nbsp;Closer to reality5.1&nbsp;&nbsp;Basic settings5.2&nbsp;&nbsp;Test corpus settings5.3&nbsp;&nbsp;Table: pqa_table.py5.4&nbsp;&nbsp;Mutlti-line table5.5&nbsp;&nbsp;Real "Child Directed Speech" tests ~ 3 hours @ 6 cores of 12-core CPU6&nbsp;&nbsp;More Jupyter notebook samples: Github7&nbsp;&nbsp;Grammar Learner test archives  SingularityNET Unsupervised Language Learning  [GitHub.com/SingNET/Language-Learning](https://github.com/singnet/language-learning)  ⇒  [README](https://github.com/singnet/language-learning/blob/master/README.md)  Grammar Learner  [GitHub.com/SingNET/Language-Learning/src/Grammar_Learner](https://github.com/singnet/language-learning/tree/master/src/grammar_learner)  ⇒ [README](https://github.com/singnet/language-learning/blob/master/src/grammar_learner/README.md)  Jump start  Basic settings and imports<jupyter_code>import os, sys, time
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path: sys.path.append(module_path)
from src.grammar_learner.learner import learn_grammar
from src.grammar_learner.utl import UTC
out_dir = module_path + '/output/Grammar_Learner_Tutorial_' + str(UTC())[:10]
kwargs = {'output_grammar': out_dir + '/1st_test'}
print(UTC(), ':: module_path:', module_path, '\nout_dir:', out_dir)<jupyter_output>2019-02-22 08:20:48 UTC :: module_path: /home/obaskov/py/language-learning 
out_dir: /home/obaskov/py/language-learning/output/Grammar_Learner_Tutorial_2019-02-22<jupyter_text>Input: "parses" ???<jupyter_code>kwargs['input_parses'] = module_path + '/data/POC-Turtle/MST-fixed-manually'
kwargs
with open(kwargs['input_parses'] + '/poc-turtle-parses-gold.txt', 'r') as f: 
    lines = f.read().splitlines()
for line in lines[:]: print(line)<jupyter_output>tuna isa fish .
0 ###LEFT-WALL### 1 tuna
1 tuna 2 isa
2 isa 3 fish
3 fish 4 .

herring isa fish .
0 ###LEFT-WALL### 1 herring
1 herring 2 isa
2 isa 3 fish
3 fish 4 .

tuna has fin .
0 ###LEFT-WALL### 1 tuna
1 tuna 2 has
2 has 3 fin
3 fin 4 .

herring has fin .
0 ###LEFT-WALL### 1 herring
1 herring 2 has
2 has 3 fin
3 fin 4 .

parrot isa bird .
0 ###LEFT-WALL### 1 parrot
1 parrot 2 isa
2 isa 3 bird
3 bird 4 .

eagle isa bird .
0 ###LEFT-WALL### 1 eagle
1 eagle 2 isa
2 isa 3 bird
3 bird 4 .

parrot has wing .
0 ###LEFT-WALL### 1 parrot
1 parrot 2 has
2 has 3 wing
3 wing 4 .

eagle has wing .
0 ###LEFT-WALL### 1 eagle
1 eagle 2 has
2 has 3 wing
3 wing 4 .

fin isa extremity .
0 ###LEFT-WALL### 1 fin
1 fin 2 isa
2 isa 3 extremity
3 extremity 4 .

wing isa extremity .
0 ###LEFT-WALL### 1 wing
1 wing 2 isa
2 isa 3 extremity
3 extremity 4 .

fin has scale .
0 ###LEFT-WALL### 1 fin
1 fin 2 has
2 has 3 scale
3 scale 4 .

wing has feather .
0 ###LEFT-WALL### 1 wing
1 wing 2 has
2 has 3 feather
3[...]<jupyter_text>LEARN!<jupyter_code>learn_grammar(**kwargs)<jupyter_output><empty_output><jupyter_text>Response: `re = learn_grammar(**kwargs)`<jupyter_code>re = learn_grammar(**kwargs)
type(re)
# from collections import OrderedDict
re.keys()<jupyter_output><empty_output><jupyter_text>Corpus statistics: `re['corpus_stats']`  ⇒  `corpus_stats.txt` file<jupyter_code>re['corpus_stats']
re['corpus_stats_file']
with open(re['corpus_stats_file'], 'r') as f: lines = f.read().splitlines()
for line in lines[:]: print(line)<jupyter_output>Number of sentences    	12
Maximum sentence length	3
Average sentence length	3
Number of unique words in sentences	13
Number of unique parsed words      	13
Number of unique non-parsed [words]	0
Number of unique linked words      	13
Number of unique non-linked words  	0
Total  words count in sentences    	36
Parsed words count in sentences    	36
Non-parsed [words] in sentences    	0
Non-linked words (excl.non-parsed) 	0
Average word count 	3
Unique links number	19
Total  links count 	24
Average link count 	1.3
Average links per linked word	2
Unique connectors number	17
Total  connectors count 	48
Average connector count 	2.8
Unique disjuncts number	16
Total  disjuncts count 	36
Average disjunct count 	2.2
Average disjunct length	1.3
Maximum disjunct length	2
Unique seeds number	31
Total  seeds count 	36
Average seed count 	1.2<jupyter_text>Response beyond corpus_stats<jupyter_code>{k:v for k,v in re.items() if type(v) is not list}<jupyter_output><empty_output><jupyter_text>Category tree: `re['cat_tree_file']`  ⇒  `cat_tree.txt` file<jupyter_code>re['cat_tree_file']
with open(re['cat_tree_file'], 'r') as f: lines = f.read().splitlines()
for line in lines[:]: print(line)<jupyter_output>B	0	1	0	eagle fin herring parrot tuna wing	0 0 0 0 0 0
C	0	2	0	bird extremity fish	0 0 0
D	0	3	0	isa	0
E	0	4	0	feather has scale	0 0 0<jupyter_text>Link Grammar file: `re['grammar_file']`  ⇒  `dict...dict` file<jupyter_code>re['grammar_file']
with open(re['grammar_file'], 'r') as f: lines = f.read().splitlines()
for line in lines[:]: print(line)<jupyter_output>% Grammar Learner v.0.7 2019-02-22 08:23:19 UTC
<dictionary-version-number>: V0v0v7+;
<dictionary-locale>: EN4us+;

% B
"eagle" "fin" "herring" "parrot" "tuna" "wing":
(BD+) or (BE+) or (EB-);

% C
"bird" "extremity" "fish":
(DC-);

% D
"isa":
(BD- & DC+);

% E
"feather" "has" "scale":
(BE- & EB+) or (BE- & EE+) or (EE-);

<UNKNOWN-WORD>: XXX+;

% 4 word clusters, 4 Link Grammar rules.<jupyter_text>More Turtle [github.com/singnet/language-learning/notebooks/](https://github.com/singnet/language-learning/blob/master/notebooks)  ⇒  [POC-Turtle-LG-5.5.1-2018-11-08.ipynb](https://github.com/singnet/language-learning/blob/master/notebooks/POC-Turtle-LG-5.5.1-2018-11-08.ipynb)  Grammar Learner parameters: kwargs[github.com/singnet/language-learning/src/grammar_learner/README.md](https://github.com/singnet/language-learning/blob/master/src/grammar_learner/README.md)<jupyter_code>kwargs = {                              # defaults:
    # input and output files and paths:
    'input_parses'      : <input>   ,   # path to directory with input parses
    'output_grammar'    : <output>  ,   # filename or path to store Link Grammar .dict file
    # parsing:
    'max_sentence_length'   :   99  ,   # filter: max number of parsed words in sentences used for learning
    'max_unparsed_words'    :   0   ,   # filter: max number of not parsed words allowed in a sentence
    'left_wall'     :   ''          ,   # '','none' - don't use / 'LEFT-WALL' - replace ###LEFT-WALL### tag with 'LEFT-WALL'
    'period'        :   False       ,   # use full stop - end of sentence in links learning
    # word (vector) space:
    'word_space'    :   'embeddings',   # 'embeddings' / 'discrete' / sparse -- see comments below
    'context'       :   2           ,   # 1: connectors / 2: disjuncts; 
    # 'embeddings' 'word_space': 
    'dim_reduction' :   'svd'       ,   # 'svd' / 'none' for 'discrete', 'sparse' word_space
    'dim_max'       :   100         ,   # max vector space dimensionality for SVD
    'sv_min'        :   0.1         ,   # minimal singular value (fraction of the max value)
    # clustering:
    'clustering'    :   'kmeans'    ,   # 'kmeans' / 'group' / 'agglomerative'... -- see comments below
    'cluster_range' :   [2,50,1,1]  ,   # min, max, step, repeat / other options described below
    'cluster_criteria'  : 'silhouette', # optimal clustering criteria (legacy for 'kmeans' 'clustering')
    'clustering_metric' : ['silhouette', 'cosine'], # new setting (October 2018) -- comments below
    # grammar induction and generalization:
    'grammar_rules'         : 2     ,   # 1: 'connectors' / 2 - 'disjuncts'
    'rules_generalization'  : 'off' ,   # 'off' / 'hierarchical' / 'jaccard' -- see comments below 
}<jupyter_output><empty_output><jupyter_text>Closer to reality  Basic settings<jupyter_code>import os, sys, time
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path: sys.path.append(module_path)
from src.grammar_learner.learner import learn
from src.grammar_learner.utl import UTC, test_stats
from src.grammar_learner.read_files import check_dir, check_corpus
from src.grammar_learner.write_files import list2file
from src.grammar_learner.widgets import html_table
from src.grammar_learner.pqa_table import table_rows, params, wide_rows
tmpath = module_path + '/tmp/'
check_dir(tmpath, True, 'none')
start = time.time()
runs = (1,1)
print(UTC(), ':: module_path:', module_path)<jupyter_output>2019-02-22 09:36:44 UTC :: module_path: /home/obaskov/py/language-learning<jupyter_text>Test corpus settings<jupyter_code>corpus = 'CDS' # 'Child Directed Speech'
dataset = 'LG-E-clean'
kwargs = {
    'corpus'        :   corpus      ,
    'dataset'       :   dataset     ,
    'left_wall'     :   ''          ,
    'period'        :   False       ,
    'context'       :   2           ,
    'min_word_count':   1           ,
    'min_link_count':   1           ,
    'word_space'    :   'sparse'    ,
    'clustering'    :   ('mean_shift', 2),
    'clustering_metric' : ['silhouette', 'cosine'],
    'cluster_range' :   [0]         ,
    'top_level'     :   0.01        ,
    'grammar_rules' :   2           ,
    'max_disjuncts' :   1000000     ,
    'stop_words'    :   []          ,
    'tmpath'        :   ''          ,
    'verbose'       :   'log+'      ,
    'template_path' :   'poc-turtle',
    'linkage_limit' :   1000        }
print(UTC(), '\n', out_dir)<jupyter_output>2019-02-22 09:36:48 UTC 
 /home/obaskov/py/language-learning/output/Grammar_Learner_Tutorial_2019-02-22<jupyter_text>Table: `pqa_table.py`<jupyter_code>#%%capture
from src.grammar_learner.pqa_table import table_rows, params, wide_rows
corpus  = 'CDS' # 'Child Directed Speech'
dataset = 'LG-E-clean'
rp = module_path + '/data/' + corpus + '/' + dataset
cp = rp  # corpus path = reference_path
# kwargs['reference_path'] = module_path + '/data/CDS/LG-E-clean'
lines = [['1.1', corpus, dataset, 0, 0, 'none']]
a, _, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
display(html_table([header] + a)); print(test_stats(log))<jupyter_output>mean shift clustering, bandwitdth = 2
cd: [[0 0 0 ... 0 0 0]
 [1 0 1 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
mean shift labels: [  1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18
  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36
  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54
  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72
  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87   0  88  89
  90  91  92  93  94  95  96  97   0  98  99 100   0 101 102 103 104 105
 106 107 108 109 110 111 112 113 114 115 116 117 118 119   0 120 121 122
 123 124 125 126 127 128 129 130 131 132 133 134 135 136   0 137 138 139
 140 141 142 143   0 144 145 146 147 148 149 150 151 152 153 154 155   0
 156 157 158 159 160 161   0 162 163   0 164 165 166 167   0 168 169 170
 171   0 172 173   0   0   0   0 174 175 176   0 177 178 179 180 181 182
 183   0 184   0[...]<jupyter_text>Mutlti-line table<jupyter_code>%%capture
# lines = [['1.1', corpus, dataset, 0, 0, 'none']]
lines = [
    [33, 'CDS' , 'LG-E-551'          ,0,0, 'none' ], 
    [34, 'CDS' , 'LG-E-551'          ,0,0, 'rules'], 
    [35, 'CDS' , 'R=6-W=6:R-MW=+1:R' ,0,0, 'none' ], 
    [36, 'CDS' , 'R=6-W=6:R-MW=+1:R' ,0,0, 'rules']]
a, _, header, log, rules = wide_rows(lines, out_dir, cp, rp, runs, **kwargs)
display(html_table([header] + a)); print(test_stats(log))
display(html_table([header] + a)); print(test_stats(log))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13389156027192836, 'fraction_numerical': 0.0736196319018405, 'mean_word_length': 3.394535519125683, 'pattern_counts': {'":': 4, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '458997', 'license': 'MIT', 'n_tokens_mistral': 5161, 'n_tokens_neox': 4532, 'n_words': 1272, 'path': 'notebooks/Grammar_Learner_Tutorial_01_2019-02-22_.ipynb', 'repo_name': 'vsbogd/language-learning', 'size': '12062'}