<jupyter_start><jupyter_text>Classify SOC Code DirectlyThis will be worst performing because of the number of classes. One for each individual occupation for a total of 956 occupations. This will not be usable in the current state because of low accuracy.<jupyter_code>soc_code_model = SocClassifier('soc_code')
soc_code_model.accuracy()
soc_code_model.model['random_forest'].best_params_<jupyter_output><empty_output><jupyter_text>Classify SOC Major GroupsThe next grouping will be the 22 major occupation groups.<jupyter_code>soc_major_model = SocClassifier('major_group')
soc_major_model.accuracy()
import pandas as pd
import seaborn as sns
pd.options.display.max_rows = 23
matrix = soc_major_model.confusion_matrix()
matrix.style.background_gradient(cmap=sns.light_palette("#0B6DB6", as_cmap=True))
soc_major_model.model['random_forest'].best_params_<jupyter_output><empty_output><jupyter_text>Classify SOC High Level Groups<jupyter_code>soc_high_level_model = SocClassifier('high_level_groups')
soc_high_level_model.accuracy()
matrix = soc_high_level_model.confusion_matrix()
matrix.style.background_gradient(cmap=sns.light_palette("#0B6DB6", as_cmap=True))
soc_high_level_model.model['random_forest'].best_params_<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08807045636509207, 'fraction_numerical': 0.010408326661329063, 'mean_word_length': 10.26126126126126, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '615247', 'license': 'MIT', 'n_tokens_mistral': 427, 'n_tokens_neox': 406, 'n_words': 81, 'path': 'notebooks/PipelineExamples.ipynb', 'repo_name': 'caseyhartnett/SOC_classifier', 'size': '1249'}