<jupyter_start><jupyter_text>Initial set up* training.json: training data for the classifier, pulled directly from global giving's api* dictionary.json: dictionary of relevant words in each category for reference in classification* testing.json: a json of the global giving database for testingFor more information,<jupyter_code># importing libraries
from BOWClassifier import BOWClassifier, set_up_training_data
from sklearn.model_selection import train_test_split
import json

# opening json files
with open("data.json") as in_file, \
        open("dictionary.json") as dictionary_json:
    
    dataset = json.load(in_file)
    train_data, test_data = train_test_split(dataset, test_size=0.2, random_state=42)
    formatted_train_data = set_up_training_data(train_data)
    
    # loading json files
    training_data = formatted_train_data
    dictionary = json.load(dictionary_json)
    organizations = test_data

# initializing bag of words classifier
bow = BOWClassifier(training_data, dictionary)<jupyter_output><empty_output><jupyter_text>Predicting a set of organizationsGiven: A dict of organizations, each with a "text": value pairOutput: A list of lists of each organizations' scores* Example: if `targets[0][2] == 1`, then organization 0 has a positive predicted theme 2 (Education)* Example: if `targets[0][3] == 0`, then organization 0 has a negative predicted theme 3 ()<jupyter_code>print(bow.predict_set(organizations))<jupyter_output>[[1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0[...]<jupyter_text>Saving predictions (above) into a json file<jupyter_code>bow.save_predictions("predictions.json")<jupyter_output><empty_output><jupyter_text>Loading predictions from a json file<jupyter_code>with open("predictions.json") as predictions_json:
    predictions = json.load(predictions_json)
    bow.load_predictions(predictions)
    
print(bow.get_predictions())<jupyter_output>[[1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0[...]<jupyter_text>Loading targets of correct classifications to test accuracy of given dataset<jupyter_code>bow.load_targets(training_data)<jupyter_output><empty_output><jupyter_text>Calculating f1 score of predictions<jupyter_code>print(bow.get_f1_score())<jupyter_output>(0.5384124887050389, {'Economic Development': 0.527448869752422, 'Education': 0.8114682884448307, 'Microfinance': 0, 'Women and Girls': 0.6846153846153846, 'Human Rights': 0.48337028824833705, 'Climate Change': 0, 'Children': 0.743801652892562, 'Environment': 0, 'Health': 0.7033175355450237, 'Humanitarian Assistance': 0, 'Technology': 0, 'Animals': 0, 'Disaster Recovery': 0, 'Sport': 0, 'Arts and Culture': 0, 'Democracy and Governance': 0, 'Hunger': 0, 'LGBTQAI+': 0})<jupyter_text>Predicting an organization based off textGiven: textOutput: A list of the organization's predicted themes<jupyter_code>text = "Climate change is not just an environmental issue, or a social justice issue, or an economic issue — it’s all of those things at once. The only way we will be strong enough to put pressure on governments and stand up to the fossil fuel industry is if we all work together."
print(bow.predict_org(text))<jupyter_output>[1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
{'fraction_non_alphanumeric': 0.20031329547679655, 'fraction_numerical': 0.15821421578225964, 'mean_word_length': 3.8509021842355176, 'pattern_counts': {'":': 1, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '229496', 'license': 'MIT', 'n_tokens_mistral': 3026, 'n_tokens_neox': 2281, 'n_words': 942, 'path': 'classification/BOWClassifier_sample_code.ipynb', 'repo_name': 'hack4impact-uiuc/globalgiving-depth', 'size': '5107'}