<jupyter_start><jupyter_text>Load Required Libraries**Standard Process** - ```pip install papermill``` - ```pip install jupyter```  - ```pip install notebook``` **Email Process** - Same as email_sender **SQL Log Process** - Same as da_tran_SQL  For my Safety reason, load setting file<jupyter_code>import os, re,json
import pandas as pd

with open('setting.txt','r') as f:
    setting = json.load(f)
sql_setting = setting['MSSQL']
sql_setting.keys()
mail_setting = setting['email']
mail_setting.keys()<jupyter_output><empty_output><jupyter_text>Working with run_pipeline<jupyter_code>from py_topping.run_pipeline import run_pipeline
run_pipeline(script_list = []
             , out_prefix = None
             , out_folder = ''
             , email_dict = None
             , sending = False
             , only_error = False
             , notebook_attached = False
             , attached_only_error = False
             , attached_log = False
             , log_sql = None)<jupyter_output><empty_output><jupyter_text>List of Parameter    - **script_list :** List of Script File to Run (.py or .ipynb)- **out_prefix :** Prefix of .ipynb output files- **out_folder :** Folder to contain .ipynb output files- **email_dict :**  Email config- **sending :** Send Run Result in Email or Not (True/False)- **only_error :** Only send email when there're error in result (True/False)- **notebook_attached :** Attached notebook in email or not (True/False)- **attached_only_error :** Only Attached Error Notebook in email or not (True/False)- **attached_log :** Attached Result as csv file in email or not (True/False)- **log_sql :** SQL config**Return :** Pandas DataFrame of Run Result  Simple Run All Script<jupyter_code>run_pipeline(script_list = ['SCRIPT.ipynb', 'SCRIPT.py'])<jupyter_output><empty_output><jupyter_text>Run All Script with Output of .ipynb with customize prefix in specific folder<jupyter_code>run_pipeline(script_list = ['SCRIPT.ipynb', 'SCRIPT.py']
             , out_prefix = ''
             , out_folder = 'TEMP')<jupyter_output><empty_output><jupyter_text>Run All Script and Send Run Result in Email Set Up Mail Config as a dict<jupyter_code>mail_dict = {'user' : mail_setting['user']
             , 'password' : mail_setting['password']
             , 'server' : mail_setting['server']
             ,'sendto' : mail_setting['to']
             ,'subject' : 'TEST'}
run_pipeline(['SCRIPT.ipynb', 'SCRIPT.py']
             , email_dict = mail_dict
             , sending = True
             , only_error = False
             , notebook_attached = True
             , attached_only_error = False
             , attached_log = True)<jupyter_output><empty_output><jupyter_text>Run All Script and Log Run Result in SQL Set Up SQL Config as a dict<jupyter_code>from py_topping.data_connection.database import da_tran_SQL

sql_dict = {'job_name' : 'test_pipeline'
            ,'table_name' : 'test_pipeline'
            , 'da_tran_SQL': da_tran_SQL(sql_type = sql_setting['type']
                                          , host_name = sql_setting['host']
                                          , database_name = sql_setting['database']
                                          , user = sql_setting['user']
                                          , password =  sql_setting['password'])}
run_pipeline(['SCRIPT.ipynb', 'SCRIPT.py']
             , log_sql = sql_dict)
sql_dict['da_tran_SQL'].read('test_pipeline')<jupyter_output><empty_output><jupyter_text>Combine Everything<jupyter_code>run_pipeline(['SCRIPT.ipynb', 'SCRIPT.py']
             , out_prefix = 'OUT_'
             , out_folder = 'TEMP'
             , email_dict = mail_dict
             , sending = True
             , only_error = False
             , notebook_attached = True
             , attached_only_error = False
             , attached_log = True
             , log_sql = sql_dict)
sql_dict['da_tran_SQL'].read('test_pipeline')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1138149556400507, 'fraction_numerical': 0.0, 'mean_word_length': 2.8236434108527133, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '213184', 'license': 'MIT', 'n_tokens_mistral': 1186, 'n_tokens_neox': 1174, 'n_words': 278, 'path': 'samples/run_pipeline.ipynb', 'repo_name': 'chanon-kr/Shared_Function', 'size': '3945'}