<jupyter_start><jupyter_text>Get a list of notebooks to run<jupyter_code>import sys
import os
sys.path.append(os.environ.get('NOTEBOOK_ROOT'))

# exclude 'machine_learning' because we do not have the data 
#   to test the Uruguay Random Forest notebooks.
# exclude 'UN_SDG_15_3_1.ipynb' due to no land change data (instructions in nbk).
# exclude 'ALOS_Land_Change.ipynb' due to no ALOS data.
# exclude 'Landslide_Identification_SLIP.iypnb' due to no TERRA ASTER data.
# exclude 'Forest_Change_VNSC.ipynb' due to no TERRA ASTER data.
# exclude 'Shallow_Water_Bathymetry.ipynb' due to no Landsat 8 Level 1 data.
# exclude 'ALOS_WASARD.ipynb' due to no ALOS data.
# exclude 'water_interoperability_similarity.ipynb' due to no Sentinel-2 data.
exclude_subpaths = ['.ipynb_checkpoints', 'utils', 'test', 'IGARSS', 'legacy',
                    'machine_learning',
                    'Automated_Testing.ipynb', 'UN_SDG_15_3_1.ipynb',
                    'ALOS_Land_Change.ipynb', 'Landslide_Identification_SLIP.ipynb',
                    'Forest_Change_VNSC.ipynb', 'Shallow_Water_Bathymetry.ipynb',
                    'ALOS_WASARD.ipynb', 'water_interoperability_similarity.ipynb']
notebook_file_paths = []
for root, directories, files in os.walk('..', topdown=True):
    notebook_file_paths.extend([os.path.join(root, file) for file in files if file.endswith('.ipynb')])
notebook_file_paths = sorted(notebook_file_paths)<jupyter_output><empty_output><jupyter_text>**Filter out notebooks using `exclude_subpaths`**<jupyter_code>def is_filepath_excluded(filepath, exclude_subpaths):
    return any(list(map(lambda subpath: subpath in filepath, exclude_subpaths)))
notebook_file_paths = [filepath for filepath in notebook_file_paths if not is_filepath_excluded(filepath, exclude_subpaths)]<jupyter_output><empty_output><jupyter_text>Run the notebooks and record their status (e.g. working, error) to HTML as each completes<jupyter_code>import re
import pandas
ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
report = pandas.DataFrame(columns=['Notebook', 'Status', 'Bug Description'])
for notebook_file_path in notebook_file_paths:
    run_result = {'Notebook': notebook_file_path, 'Status': 'Working', 'Bug Description': ''}
    with open(notebook_file_path, 'r+', encoding='utf-8') as notebook_file:
        notebook = nbformat.read(notebook_file, as_version=4)
        notebook_runner = ExecutePreprocessor(timeout=None)
        try:
            notebook_runner.preprocess(notebook, {'metadata': {'path': os.path.dirname(notebook_file_path)}})
        except CellExecutionError as err:
            run_result['Status'] = 'Error'
            run_result['Bug Description'] = err
        nbformat.write(notebook, notebook_file_path)
    report = report.append(run_result, ignore_index=True)
    report.to_html('testing_report.html', escape=False, formatters={'Bug Description': lambda e: ansi_escape.sub('', str(e).replace('\n', '<br/>'))})<jupyter_output><empty_output><jupyter_text>Export the results to a CSV<jupyter_code>report['Bug Description'] = report['Bug Description'].map(lambda e: ansi_escape.sub('', str(e)))
report.to_csv('testing_report.csv')
report<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12461059190031153, 'fraction_numerical': 0.004672897196261682, 'mean_word_length': 5.116190476190476, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '571045', 'license': 'Apache-2.0', 'n_tokens_mistral': 1111, 'n_tokens_neox': 1084, 'n_words': 228, 'path': 'notebooks/test/Automated_Testing.ipynb', 'repo_name': 'AnnetteB-MQ/data_cube_notebooks', 'size': '3210'}