<jupyter_start><jupyter_text>Run a Hail notebook in the backgroundTerra users frequently use notebooks interactively to test [Hail](https://hail.is/) analyses on small portions of the genome. But running that same notebook over more data may take hours. For long running Hail jobs, if the browser loses its connection, users may not have all notebook cell outputs populated even though the notebook continues to run to completion.    If you wish to capture all Hail notebook cell outputs, use this notebook to run your long-running Hail notebook (or any other long-running notebook).But also note that the cluster will autopause after 24 hours. To prevent your cluster from shutting down if your background Hail job takes longer than 24 hours, be sure to log in and start a notebook, any notebook, to reset the autopause timer.How to use this notebook:1. Copy this notebook to the workspace that contains the long-running notebook you wish to run in the background.1. Edit the filename in `NOTEBOOK_TO_RUN` to be the name of the notebook in this workspace you wish to run.1. Use menu `Cell -> Run All` to tell the kernel to run this notebook to completion.1. Close this tab and do other work. (note: when you close the tab, the outputs in *this* notebook will no longer be updated, and that is fine, because what we really want are the outputs in the Hail notebook.)1. When the Hail job is complete:    1. Your Hail cluster will pause (if you have no other notebooks open).    1. You will see a copy of your notebook in the notebooks tab with a date and timestamp suffix --> this is the output file.    1. You will also see the Hail log in `gs:///hail-logs/YYYYMMDD/hail*.log`If you check back, and you do not see the output notebook, the Hail job is still running. To confirm this, first notice whether your Cloud analysis environment is still running. If it is, from the terminal run `yarn application -list` to see that a spark context is still processing data.  Setup<jupyter_code>import nbformat
from nbconvert.preprocessors import CellExecutionError
from nbconvert.preprocessors import ExecutePreprocessor

import os
import time<jupyter_output><empty_output><jupyter_text>Change the cell below to hold the name of the notebook in this workspace that you wish to run in the background. There is no need to modify any of the other cells.<jupyter_code>#---------[ CHANGE THIS TO BE THE NAME OF THE NOTEBOOK YOU WISH TO RUN ]-----------
NOTEBOOK_TO_RUN = 'your_notebook.ipynb'<jupyter_output><empty_output><jupyter_text>Programmatically set output paths Formulate a new filename for the output notebook so that it includes a date and timestamp for when it was executed.<jupyter_code>TIMESTAMP_FILE_SUFFIX = time.strftime('_%Y%m%d_%H%M%S.ipynb')
OUTPUT_NOTEBOOK = NOTEBOOK_TO_RUN.replace('.ipynb', TIMESTAMP_FILE_SUFFIX)

print(f'Executed notebook will be written to filename "{OUTPUT_NOTEBOOK}" on the local disk and the workspace bucket.')
DATESTAMP = time.strftime('%Y%m%d')
HAIL_LOG_DIR_FOR_PROVENANCE = f'{os.getenv("WORKSPACE_BUCKET")}/hail-logs/{DATESTAMP}/'

print(f'Hail logs will be copied to {HAIL_LOG_DIR_FOR_PROVENANCE}')<jupyter_output><empty_output><jupyter_text>Execute the notebook and capture provenance The next cell will use nbconvert to run the notebook in the background until it completes (or yields an error) and will capture all notebook outputs as a separate notebook file.<jupyter_code># See also https://nbconvert.readthedocs.io/en/latest/execute_api.html
with open(NOTEBOOK_TO_RUN) as f_in:
    nb = nbformat.read(f_in, as_version=4)
    ep = ExecutePreprocessor(timeout=-1, kernel_name='python3')
    try:
        out = ep.preprocess(nb, {'metadata': {'path': ''}})
    except CellExecutionError:
        out = None
        msg = 'Error executing the notebook "%s".\n\n' % NOTEBOOK_TO_RUN
        msg += 'See notebook "%s" for the traceback.' % OUTPUT_NOTEBOOK
        print(msg)
    finally:
        with open(OUTPUT_NOTEBOOK, mode='w', encoding='utf-8') as f_out:
            nbformat.write(nb, f_out)<jupyter_output><empty_output><jupyter_text>When notebook has finished execution, the next cell will use gsutil to copy it from the local disk to the workspace bucket.<jupyter_code>%%bash -s $OUTPUT_NOTEBOOK

gsutil cp "${1}" ${WORKSPACE_BUCKET}/notebooks/<jupyter_output><empty_output><jupyter_text>When notebook has finished execution, the next cell will use gsutil to copy all hail logs from the execution directory on the local disk to the workspace bucket.<jupyter_code>%%bash -s $HAIL_LOG_DIR_FOR_PROVENANCE

gzip --keep hail*.log
gsutil -m cp hail*.log.gz ${1}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07514575685597064, 'fraction_numerical': 0.003886849492550205, 'mean_word_length': 4.69041769041769, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 27, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '73168', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1363, 'n_tokens_neox': 1333, 'n_words': 590, 'path': 'terra-notebooks-playground/Hail - How to run a notebook in the background.ipynb', 'repo_name': 'amygdala/terra-example-notebooks', 'size': '4631'}