<jupyter_start><jupyter_text>Two important submission csvs were written wrong, but in anticipation of this problem we pickled the results. Opening them now.<jupyter_code>import pickle
cd /disk/scratch/neuroglycerin/dump/
ls
with open("test.py.pkl","rb") as f:
    p = pickle.load(f)
len(p)
p[0].shape[0]*80<jupyter_output><empty_output><jupyter_text>Looks like everything should be there, just have to figure out why it didn't write these to the csv right. Next part was the stack:<jupyter_code>import numpy as np
y = np.vstack(p)
y.shape<jupyter_output><empty_output><jupyter_text>That worked, what about finding the name for the csv?<jupyter_code>import neukrill_net.utils
cd ~/repos/neukrill-net-work/
settings = neukrill_net.utils.Settings("settings.json")
import os
names = [os.path.basename(n) for n in settings.image_fnames['test']]
len(names)<jupyter_output><empty_output><jupyter_text>That also seems to be fine...Only explanation I can think of at this point is that it somehow redefined the `image_fname` dict to be over one of the splits. But that makes no sense because the `image_fname` dictionary that gets modified is a different instance to that in the `test.py` script.Looking at the submission csvs:<jupyter_code>cd /disk/scratch/neuroglycerin/submissions/
ls
!gzip -d alexnet_based_40aug.csv.gz
!wc -l alexnet_based_40aug.csv<jupyter_output>1631 alexnet_based_40aug.csv<jupyter_text>The splits would have been equal to the full dataset divided by 80:<jupyter_code>130400/80<jupyter_output><empty_output><jupyter_text>Including the header, that's exactly correct.All we can do now is rewrite the submission csv with the full names and submit it to check it's valid.<jupyter_code>neukrill_net.utils.write_predictions("alexnet_based_40aug.csv",y,names,settings.classes)<jupyter_output><empty_output><jupyter_text>And we have to do the same for 16aug predictions.<jupyter_code>cd /disk/scratch/neuroglycerin/dump/
with open("test2.py.pkl","rb") as f:
    p16aug = pickle.load(f)
y16aug = np.vstack(p16aug)
y16aug.shape
cd /disk/scratch/neuroglycerin/submissions/
neukrill_net.utils.write_predictions("alexnet_based_16aug.csv.gz",y16aug,names,settings.classes)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09614512471655329, 'fraction_numerical': 0.018594104308390022, 'mean_word_length': 6.554794520547945, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '256608', 'license': 'MIT', 'n_tokens_mistral': 812, 'n_tokens_neox': 757, 'n_words': 207, 'path': 'notebooks/troubleshooting_and_sysadmin/Opening test.py pickles.ipynb', 'repo_name': 'Neuroglycerin/neukrill-net-work', 'size': '2205'}