<jupyter_start><jupyter_text>Evaluation of saved models  Evaluation on synthetic taskSee https://arampacha.github.io/reformer_fastai/experiment.synthetic-task.html for dataset and training details.<jupyter_code>from reformer_fastai.expscript import get_twin_sequence_dataloaders, get_synthetic_learner
import wandb
run = wandb.init()
artifact = run.use_artifact('fastai_community/reformer-fastai/run-wvb5g2re-model:v0', type='model')
artifact_dir = artifact.download(root='./models')
dls = get_twin_sequence_dataloaders()
config = SyntheticConfig(use_lsh=False)
model = LSHLM.from_config(config)
learn = get_synthetic_learner(dls, model, precision=0)
learn.add_cb(MaskTargCallback());
test_dl = learn.dls.test_dl(DeterministicTwinSequence(1024, 1000))<jupyter_output><empty_output><jupyter_text>Score of randomly initialized model as sanity check:<jupyter_code>res = learn.validate(dl=test_dl)
print_eval(learn, res)<jupyter_output><empty_output><jupyter_text>After loading pretrained weights models achieves perfect accuracy on unseen data, as expected:<jupyter_code>learn.load('run-wvb5g2re-model', with_opt=False);
res = learn.validate(dl=test_dl)
print_eval(learn, res)<jupyter_output><empty_output><jupyter_text>Evaluation on enwik8 test set<jupyter_code>from reformer_fastai.expscript import get_lm_learner
path = untar_data('http://mattmahoney.net/dc/enwik8.zip', dest='./data')
df = pd.DataFrame({'text':read_lines('data/enwik8')})

btt = ByteTextTokenizer(is_lm=True, add_bos=False, add_eos=False)
df['toks'] = df['text'].apply(btt)
df['lens'] = df['toks'].apply(len)
df['lens_cum_sum'] = df.lens.cumsum()
train_cutoff = df.lens.sum() - 10_000_000  # 10M characters for val and test
train_idxs = df.loc[df['lens_cum_sum'] < train_cutoff].index.values
train_idxs = list(range(0, max(train_idxs)))

remaining_idxs = len(df) - max(train_idxs)
validation_idxs = list(range(max(train_idxs), max(train_idxs) + int(remaining_idxs/2)))
test_idxs = list(range(max(validation_idxs), len(df)))

splits = [train_idxs, validation_idxs]

tfms = [attrgetter("text"), btt]
dsets = Datasets(df, [tfms], splits=splits, dl_type=LMDataLoader)
dl_kwargs = [{'lens':df['lens'].values[train_idxs]},
             {'val_lens':df['lens'].values[validation_idxs]}]
n_cpus = multiprocessing.cpu_count()
dls = dsets.dataloaders(bs=8, val_bs=24, seq_len=4096, dl_kwargs=dl_kwargs, shuffle_train=True, n_workers=n_cpus)
config = NHashesConfig(n_hashes=2, pad_idx=dls.byte_text_tokenizer.pad_token_id)
model = LSHLM.from_config(config)
learn = get_lm_learner(dls, model, opt_func=adafactor, precision=2)
learn.add_cb(PadBatchCallback(bucket_size=config.bucket_size, val=config.pad_idx, y_val=config.pad_idx));
test_dl = learn.dls.test_dl(df.iloc[test_idxs, :])
test_dl.show_batch(max_n=2)
res = learn.validate(dl=test_dl)
print_eval(learn, res)
# import wandb
# run = wandb.init()
artifact = run.use_artifact('fastai_community/reformer-fastai/3tbfvs77:v0', type='model')
artifact_dir = artifact.download(root='./models')
learn.load('n_hashes_n_hashes-4_enwik8_sl-4096_bs-4_n_eps-10_seed-42_grad-accum-8__24_01_2021_15:12', with_opt=False);
res = learn.validate(dl=test_dl)
print_eval(learn, res)<jupyter_output><empty_output><jupyter_text>It's possible to improve performance by increasing number of hashing rounds. In this case one can observe small improvement in perplaxity and BPC:<jupyter_code>learn.model.n_hashes = 8
res = learn.validate(dl=test_dl)
print_eval(learn, res)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13367756741250716, 'fraction_numerical': 0.019506597819850834, 'mean_word_length': 8.794943820224718, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '153531', 'license': 'MIT', 'n_tokens_mistral': 1388, 'n_tokens_neox': 1315, 'n_words': 200, 'path': 'nbs/61_evaluation.ipynb', 'repo_name': 'arampacha/reformer_fastai', 'size': '3486'}