<jupyter_start><jupyter_text>1. understand/compare the bounds generated between the two methods. The bounds of the bootsrap have more variance. There is more variety of accuracies from resampling than from changing the the random seed.   2A. Evaluation++: what happens to the variance if we do K bootstrap samples for each of M models?<jupyter_code>#%% DecisionTree Parameters:
params = {
    "criterion": "gini",
    "splitter": "best",
    "max_depth": 5,
}

# train 100 different models, with different sources of randomness:
N_MODELS = 100
# sample 1 of them 100 times, to get a distribution of data for that!
N_SAMPLES = 1000

seed_based_accuracies = []
for randomness in range(N_MODELS):
    f_seed = DecisionTreeClassifier(random_state=RANDOM_SEED + randomness, **params)
    f_seed.fit(X_train, y_train)
    seed_based_accuracies.append(f_seed.score(X_vali, y_vali))


bootstrap_based_accuracies = []
# single seed, bootstrap-sampling of predictions:
f_single = DecisionTreeClassifier(random_state=RANDOM_SEED, **params)
f_single.fit(X_train, y_train)
y_pred = f_single.predict(X_vali)

# do the bootstrap:
for trial in range(N_SAMPLES):
    sample_pred, sample_truth = resample(
        y_pred, y_vali, random_state=trial + RANDOM_SEED
    )
    score = accuracy_score(y_true=sample_truth, y_pred=sample_pred)
    bootstrap_based_accuracies.append(score)
boxplot_data: List[List[float]] = [seed_based_accuracies, bootstrap_based_accuracies]
plt.boxplot(boxplot_data)
plt.xticks(ticks=[1, 2], labels=["Seed-Based", "Bootstrap-Based"])
plt.xlabel("Sampling Method")
plt.ylabel("Accuracy")
plt.ylim([0.85, 0.95])
plt.show()<jupyter_output><empty_output><jupyter_text>compared to the k=100 and m = 100 (below) the k=1000 and m=100 (above) has a lot more variance which makes sense because we are taking a much larger sample<jupyter_code>#don't run this
boxplot_data: List[List[float]] = [seed_based_accuracies, bootstrap_based_accuracies]
plt.boxplot(boxplot_data)
plt.xticks(ticks=[1, 2], labels=["Seed-Based", "Bootstrap-Based"])
plt.xlabel("Sampling Method")
plt.ylabel("Accuracy")
plt.ylim([0.85, 0.95])
plt.show()<jupyter_output><empty_output><jupyter_text>2B. Return to experimenting on the decision tree: modify the plot to show ~10 max_depths of the decision tree<jupyter_code># train 100 different models, with different sources of randomness:
N_MODELS = 100
# sample 1 of them 100 times, to get a distribution of data for that!
N_SAMPLES = 100

depth_scores = {}
for i in range(1,10):
    params = {
    "criterion": "gini",
    "splitter": "best",
    "max_depth": i,
    }
    bootstrap_based_accuracies = []
    # single seed, bootstrap-sampling of predictions:
    f_single = DecisionTreeClassifier(random_state=RANDOM_SEED, **params)
    f_single.fit(X_train, y_train)
    y_pred = f_single.predict(X_vali)

    # do the bootstrap:
    for trial in range(N_SAMPLES):
        sample_pred, sample_truth = resample(
            y_pred, y_vali, random_state=trial + RANDOM_SEED
        )
        score = accuracy_score(y_true=sample_truth, y_pred=sample_pred)
        bootstrap_based_accuracies.append(score)
    depth_scores[i] = bootstrap_based_accuracies
boxplot_data: List[List[float]] = depth_scores.values()
plt.boxplot(boxplot_data)
plt.xticks(ticks=range(1,10), labels=depth_scores.keys())
plt.xlabel("Sampling Method")
plt.ylabel("Accuracy")
plt.ylim([0.7, 1.0])
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10997666277712952, 'fraction_numerical': 0.021003500583430573, 'mean_word_length': 5.385474860335195, 'pattern_counts': {'":': 6, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '560255', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1267, 'n_tokens_neox': 1168, 'n_words': 292, 'path': 'p03-bootstrap.ipynb', 'repo_name': 'Avanderheyde/cs451-practicals', 'size': '3428'}