<jupyter_start><jupyter_text>Load dataset<jupyter_code>graph_name = 'cora_ml'
graph = load_dataset(graph_name)
graph.standardize(select_lcc=True)<jupyter_output><empty_output><jupyter_text>Set up data splits First of all we need to decide whether to use the test or validation set. Be mindful that we can only look at the test set exactly _once_ and then can't change any hyperparameters oder model details, no matter what. Everything else would cause overfitting.<jupyter_code>test = True<jupyter_output><empty_output><jupyter_text>These are the seeds for the dataset splits used in the paper for test/validation.<jupyter_code>test_seeds = [
        2144199730,  794209841, 2985733717, 2282690970, 1901557222,
        2009332812, 2266730407,  635625077, 3538425002,  960893189,
        497096336, 3940842554, 3594628340,  948012117, 3305901371,
        3644534211, 2297033685, 4092258879, 2590091101, 1694925034]
val_seeds = [
        2413340114, 3258769933, 1789234713, 2222151463, 2813247115,
        1920426428, 4272044734, 2092442742, 841404887, 2188879532,
        646784207, 1633698412, 2256863076,  374355442,  289680769,
        4281139389, 4263036964,  900418539,  119332950, 1628837138]

if test:
    seeds = test_seeds
else:
    seeds = val_seeds<jupyter_output><empty_output><jupyter_text>Now we can choose the remaining settings for the training/early stopping/validation(test) split. These are the ones chosen in the paper<jupyter_code>if graph_name == 'microsoft_academic':
    nknown = 5000
else:
    nknown = 1500
    
idx_split_args = {'ntrain_per_class': 20, 'nstopping': 500, 'nknown': nknown}<jupyter_output><empty_output><jupyter_text>Set up propagationNext we need to set up the proper pmropagation scheme. In the paper we've introduced the exact PPR propagation used in PPNP and the PPR power iteration propagation used in APPNP.We use the hyperparameters from the paper.<jupyter_code>if graph_name == 'microsoft_academic':
    alpha = 0.2
else:
    alpha = 0.1

prop_ppnp = PPRExact(graph.adj_matrix, alpha=alpha)
prop_appnp = PPRPowerIteration(graph.adj_matrix, alpha=alpha, niter=10)<jupyter_output><empty_output><jupyter_text>Choose model hyperparametersNow we choose the hyperparameters. These are the ones used in the paper for all datasets.Note that we choose the propagation for APPNP.<jupyter_code>model_args = {
    'hiddenunits': [64],
    'reg_lambda': 5e-3,
    'learning_rate': 0.01,
    'keep_prob': 0.5,
    'propagation': prop_appnp}<jupyter_output><empty_output><jupyter_text>Train modelFirst we set the remaining settings for training.<jupyter_code>niter_per_seed = 5
save_result = False
print_interval = 100<jupyter_output><empty_output><jupyter_text>We use 20 different seeds for splitting and 5 iterations (different random initializations) per split, so we train 100 times altogether. This will take a while.<jupyter_code>results = []
used_seeds = []
niter_tot = niter_per_seed * len(seeds)
i_tot = 0
for seed in seeds:
    idx_split_args['seed'] = seed
    for _ in range(niter_per_seed):
        i_tot += 1
        logging_string = f"Iteration {i_tot} of {niter_tot}"
        logging.log(22,
                logging_string + "\n                     "
                + '-' * len(logging_string))
        result = train_model(
                graph_name, PPNP, graph, model_args, idx_split_args,
                stopping_args, test, save_result, None, print_interval)
        results.append({})
        results[-1]['stopping_accuracy'] = result['early_stopping']['accuracy']
        results[-1]['stopping_f1_score'] = result['early_stopping']['f1_score']
        results[-1]['valtest_accuracy'] = result['valtest']['accuracy']
        results[-1]['valtest_f1_score'] = result['valtest']['f1_score']
        results[-1]['runtime'] = result['runtime']
        results[-1]['runtime_perepoch'] = result['runtime_perepoch']
        results[-1]['split_seed'] = seed<jupyter_output>2019-03-08 10:26:36: Iteration 1 of 100
                     ------------------
2019-03-08 10:26:36: Tensorflow seed: 1119671048
2019-03-08 10:27:18: Last step: 1225, best step: 624 (40.217 sec)
2019-03-08 10:27:19: Test accuracy: 84.8%, test F1 score: 0.843
2019-03-08 10:27:19: Iteration 2 of 100
                     ------------------
2019-03-08 10:27:19: Tensorflow seed: 1361780415
2019-03-08 10:28:04: Last step: 1348, best step: 1218 (44.038 sec)
2019-03-08 10:28:04: Test accuracy: 85.3%, test F1 score: 0.846
2019-03-08 10:28:04: Iteration 3 of 100
                     ------------------
2019-03-08 10:28:04: Tensorflow seed: 3789243101
2019-03-08 10:28:49: Last step: 1361, best step: 1218 (44.529 sec)
2019-03-08 10:28:50: Test accuracy: 83.9%, test F1 score: 0.838
2019-03-08 10:28:50: Iteration 4 of 100
                     ------------------
2019-03-08 10:28:50: Tensorflow seed: 171464893
2019-03-08 10:29:36: Last step: 1398, best step: 1068 (45.566 sec)
2019-03-08 10:29:37: Test [...]<jupyter_text>EvaluationTo evaluate the data we use Pandas and Seaborn (for bootstrapping).<jupyter_code>import pandas as pd
import seaborn as sns
result_df = pd.DataFrame(results)
result_df.head()<jupyter_output><empty_output><jupyter_text>The standard deviation doesn't really say much about the uncertainty of our results and the standard error of the mean (SEM) assumes a normal distribution. So the best way to get a valid estimate for our results' uncertainty is via bootstrapping.<jupyter_code>def calc_uncertainty(values: np.ndarray, n_boot: int = 1000, ci: int = 95) -> dict:
    stats = {}
    stats['mean'] = values.mean()
    boots_series = sns.algorithms.bootstrap(values, func=np.mean, n_boot=n_boot)
    stats['CI'] = sns.utils.ci(boots_series, ci)
    stats['uncertainty'] = np.max(np.abs(stats['CI'] - stats['mean']))
    return stats
stopping_acc = calc_uncertainty(result_df['stopping_accuracy'])
stopping_f1 = calc_uncertainty(result_df['stopping_f1_score'])
valtest_acc = calc_uncertainty(result_df['valtest_accuracy'])
valtest_f1 = calc_uncertainty(result_df['valtest_f1_score'])
runtime = calc_uncertainty(result_df['runtime'])
runtime_perepoch = calc_uncertainty(result_df['runtime_perepoch'])
print("APPNP\n"
      "Early stopping: Accuracy: {:.2f} ± {:.2f}%, "
      "F1 score: {:.4f} ± {:.4f}\n"
      "{}: Accuracy: {:.2f} ± {:.2f}%, "
      "F1 score: {:.4f} ± {:.4f}\n"
      "Runtime: {:.3f} ± {:.3f} sec, per epoch: {:.2f} ± {:.2f}ms"
      .format(
          stopping_acc['mean'] * 100,
          stopping_acc['uncertainty'] * 100,
          stopping_f1['mean'],
          stopping_f1['uncertainty'],
          'Test' if test else 'Validation',
          valtest_acc['mean'] * 100,
          valtest_acc['uncertainty'] * 100,
          valtest_f1['mean'],
          valtest_f1['uncertainty'],
          runtime['mean'],
          runtime['uncertainty'],
          runtime_perepoch['mean'] * 1e3,
          runtime_perepoch['uncertainty'] * 1e3,
      ))<jupyter_output>APPNP
Early stopping: Accuracy: 84.17 ± 0.37%, F1 score: 0.8296 ± 0.0041
Test: Accuracy: 85.05 ± 0.23%, F1 score: 0.8431 ± 0.0021
Runtime: 45.661 ± 1.793 sec, per epoch: 32.72 ± 0.04ms
{'fraction_non_alphanumeric': 0.1280393534785664, 'fraction_numerical': 0.1253689388615601, 'mean_word_length': 3.782258064516129, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '268725', 'license': 'MIT', 'n_tokens_mistral': 3060, 'n_tokens_neox': 2429, 'n_words': 623, 'path': 'reproduce_results.ipynb', 'repo_name': 'kckishan/ppnp', 'size': '7115'}