<jupyter_start><jupyter_text>Jupyter Notebooks for Research 2. Multi Notebook ProjectsI find that some of my notebooks get out of hand. Weeks of work leads to 100+ cells.The notebook feels encumbered (browser, kernel or server load?). A kernel restartmeans 10+ mins to get back to where I was. Its unpleasant.There is probably sub-optimal code in there but the real issue is that I've abused athe single notebook model and its time to do better. Splitting your notebook into multiple notebooksMaybe you could think of this as the chapters of the analysis. Technically, I suppose"books" might seem more natural but with a substantial dataset and/ or compute intensiveanalyses I reckon you'll be best splitting on what you'd consider chapters. Here's anexample project layout:1. Data preparation2. Exploratory analysis  1. Facet 1  2. Facet 23. Model fittingThe main technical aspect you need to consider is **exchanging data between the notebooks**.In this example you might have something like the following dependency tree:```01_01_data_prep.ipynb│└───│   │   02_01_facet_1.ipynb│   │   02_02_facet_2.ipynb│   └───└───────│   03_01_model.ipynb```There are a variety of ways you can acheive this and its going to be pretty straightforward.I would advise some form of checksumming though. Generate dataFirst we generate our raw dataset.<jupyter_code>def long_running_data_generation(n=5):
    time.sleep(n)
    np.random.seed(seed=444)  # vcs would be a pain without this
    return pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))


df = long_running_data_generation()
df.head()<jupyter_output><empty_output><jupyter_text>ProcessingWe have a few processing steps to do.<jupyter_code>df = df.assign(A2=2*df.A+df.B)
df.head()<jupyter_output><empty_output><jupyter_text>StorageLet's store that for future use.<jupyter_code># cell imports?! maybe I won't use these anywhere else...
import hashlib
import datetime
import gzip

# safely hash a dataframe
# TODO: Include reference to where I saw this
row_hashes = pd.util.hash_pandas_object(df, index=True)
df_hash = hashlib.sha256(row_hashes.values).hexdigest()
print(df_hash)
# write the file, don't clobber it if its already there, this could be slow
filename = f'data_prep_df_{df_hash[:7]}.csv.gz'
now = datetime.datetime.now()
try:
    with gzip.open(filename, 'x') as scores_file:
        scores_file.write('# Creation time: {}\n'.format(str(now)).encode())
        scores_file.write('# Table hash: {}\n'.format(df_hash).encode())
        scores_file.write(df.to_csv().encode())
    print('Saved {}'.format(filename))
except FileExistsError:
    print('{} already exists.'.format(filename))<jupyter_output>0ebc21d7983d3ce8bcc159aea7cc8127b76ef3183e260333780a54b619b129a6
Saved data_prep_df_0ebc21d.csv.gz<jupyter_text>Seems to be a problem verifying the CSV on the otherside, maybe a pickle will work.<jupyter_code>df.to_pickle(f'data_prep_df_{df_hash[:7]}.p.gz')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09651202167287504, 'fraction_numerical': 0.03047748052827633, 'mean_word_length': 5.449781659388647, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '149620', 'license': 'MIT', 'n_tokens_mistral': 1012, 'n_tokens_neox': 938, 'n_words': 312, 'path': 'multi/01_01_data_prep.ipynb', 'repo_name': 'stuartmac/sweet-jupy', 'size': '2953'}