<jupyter_start><jupyter_text>Providing Secrets for Scale Out ExecutionCopyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License. In this notebook, we will load a package we previously saved and pass in the required secrets so we can execute the DataFlow on Spark.<jupyter_code>import azureml.dataprep as dprep

import os<jupyter_output><empty_output><jupyter_text>Let's open the previously saved package.<jupyter_code>package = dprep.Package.open(file_path='../data/secrets.dprep')
dataflow = package.dataflows[0]<jupyter_output><empty_output><jupyter_text>If we try to execute on Spark without passing in an secret, we will get an exception telling us to pass in the required secrests.<jupyter_code>try:
    dataflow.run_spark()
except Exception as e:
    print(e)<jupyter_output><empty_output><jupyter_text>Let's pass in the missing secret and execute the DataFlow on Spark.<jupyter_code>sas = os.environ['SCENARIOS_SCALE_OUT_SECRETS']
secrets = {
    'https://dpreptestfiles.blob.core.windows.net/testfiles/read_csv_duplicate_headers.csv': sas
}
dataflow.use_secrets(secrets)
dataflow.to_spark_dataframe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08561643835616438, 'fraction_numerical': 0.0008561643835616438, 'mean_word_length': 6.845637583892618, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '409187', 'license': 'MIT', 'n_tokens_mistral': 393, 'n_tokens_neox': 376, 'n_words': 101, 'path': 'API/secrets-scale_out.ipynb', 'repo_name': 'joelvisroman/MSDataPrepDocs', 'size': '1168'}