<jupyter_start><jupyter_text>Czechoslovak pattern generation demo<jupyter_code>from jupyterpatgen import *
from IPython.display import display, Markdown
!cp src/csskhyphen.par out/csskhyphen.par
display(Markdown(train(".", "csskhyphen")))
display(Markdown(show_stats("csskhyphen")))<jupyter_output><empty_output><jupyter_text>We do not try to achieve 0 bad or 0 missed after the generation of the final level of patterns. The input dataset is noisy and forcing patgen to be more aggressive would be counterproductive -- it would unnecessarily learn all the mistakes in the input data. See correct optimized parameters for an example of this.The best way to evaluate hyphenation patterns is to look at the output. After a training run, the file `out/pattmp.4` contains the input word list, hyphenated by the new patterns. `*` represents a correctly learned hyphenation point, `-` represents a hyphenation point that the patterns did not find. `.` (dot) is a point where the patterns would hyphenate, but there is no hyphen in the input dataset.Note that pattmp.4 gets overwritten after each training run, so make sure to only run the "csskhyphen" cell. Comparsion of custom parameters with other variants<jupyter_code>display(Markdown(train(".", "cs-sojka-correctoptimized")))
display(Markdown(show_stats("cs-sojka-correctoptimized")))
display(Markdown(train(".", "cs-sojka-sizeoptimized")))
display(Markdown(show_stats("cs-sojka-sizeoptimized")))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08604336043360433, 'fraction_numerical': 0.0027100271002710027, 'mean_word_length': 6.422110552763819, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '137251', 'license': 'MIT', 'n_tokens_mistral': 436, 'n_tokens_neox': 425, 'n_words': 163, 'path': 'demo.ipynb', 'repo_name': 'tensojka/cshyphen', 'size': '1476'}