<jupyter_start><jupyter_text>conduct experiments of the assignment with em algorithm + more iterations + kmeans init + more data<jupyter_code># arguments for the em algorithm
emargs = {
    'n': 100,
    'init': 'kmeans',
    'check_convergence': True,
    'outliers': 0
}

show_step_plots = False
# import em implementation etc.
%run em.ipynb  
sel_plt(gen_sample_x(5000), show_step_plots, **emargs)
sel_plt(iris_x, show_step_plots, **emargs)
sel_plt(bdp_x, show_step_plots, **emargs)<jupyter_output>1 2790.4060160663075
This is the new best model according to aic (above)!
2 2203.913194081503
This is the new best model according to aic (above)!
3 2187.8701013736973
This is the new best model according to aic (above)!
4 2188.1577469879785
5 2194.152641960914
6 2199.914039691536
7 2205.936410390564
8 2212.5672282430996
9 2217.916776726349
10 2224.5560512695088
11 2230.559128063872
12 2194.2907435267625
13 2242.1370943229276
14 2252.0077965517653
15 2253.693848157818
best model:
{'fraction_non_alphanumeric': 0.083248730964467, 'fraction_numerical': 0.2812182741116751, 'mean_word_length': 5.573333333333333, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '67201', 'license': 'Apache-2.0', 'n_tokens_mistral': 568, 'n_tokens_neox': 387, 'n_words': 106, 'path': 'assignment_6_1/experiments/more_data.ipynb', 'repo_name': 'larsaars/machine_learning_tests', 'size': '985'}