<jupyter_start><jupyter_text>Login / Switch DEV_TEST_PROD environment (1-timer)<jupyter_code>import repackage
repackage.add("../azure-enterprise-scale-ml/esml/common/")
from azureml.core.authentication import InteractiveLoginAuthentication
from esml import ESMLProject

p = ESMLProject()
p.dev_test_prod="dev"
auth = InteractiveLoginAuthentication(tenant_id = p.tenant)
ws, config_name = p.authenticate_workspace_and_write_config(auth)<jupyter_output>Using lake_settings.json with ESML version 1.4 - Models array support including LABEL<jupyter_text>1) ESML - TRAIN Classification, TITANIC model, and DEPLOY with predict_proba scoring<jupyter_code>import repackage
repackage.add("../azure-enterprise-scale-ml/esml/common/")
from esml import ESMLProject
import pandas as pd

p = ESMLProject() # Will search in ROOT for your copied SETTINGS folder '../../../settings', you should copy template settings from '../settings'
p.active_model = 10
p.inference_mode = False
p.ws = p.get_workspace_from_config() #2) Load DEV or TEST or PROD Azure ML Studio workspace
p.describe()
unregister_all_datasets=False
if(unregister_all_datasets):
    p.unregister_all_datasets(p.ws) # For DEMO purpose
def test_feature_engieering():
    # R&D purpose: Try some data wrangling here...we will later incorporate this in an Azure ML Pipeline, as "steps"
    esml_dataset = p.DatasetByName("ds01_titanic") 
    df_bronze = esml_dataset.Bronze.to_pandas_dataframe()
    df_bronze.columns = df_bronze.columns.str.replace("[/]", "_") # Rename werid column names

    df_silver = p.save_silver(esml_dataset,df_bronze) #Bronze -> Silver

    esml_dataset2 = p.DatasetByName("ds02_haircolor")
    esml_dataset3 = p.DatasetByName("ds03_housing")
    esml_dataset4 = p.DatasetByName("ds04_lightsaber")

    p.save_silver(esml_dataset2,esml_dataset2.Bronze.to_pandas_dataframe()) #Bronze -> Silver
    p.save_silver(esml_dataset3,esml_dataset3.Bronze.to_pandas_dataframe()) #Bronze -> Silver
    p.save_silver(esml_dataset4,esml_dataset4.Bronze.to_pandas_dataframe()) #Bronze -> Silver

    gold = p.save_gold(esml_dataset.Silver.to_pandas_dataframe())  #Silver -> Gold STEP
    return gold
datastore = p.init() # 3) Automapping from datalake to Azure ML datasets
gold = test_feature_engieering()
datastore = None
try:
    datastore = p.connect_to_lake() # Connects to the correct ALDS GEN 2 storage account (DEV, TEST or PROD)
    gold_train = p.GoldTrain
    gold_train.name
    print("Not 1st time. We have data mapped already...and splitted. Now connected to LAKE")
except: # If 1st time....no Gold exists, nor any mapping
    print("1st time. Lets init, map what data we have in LAKE, as Azure ML Datasets")
    datastore = p.init() # 3) Automapping from datalake to Azure ML datasets
    gold = test_feature_engieering()
gold = test_feature_engieering()
p.Gold.to_pandas_dataframe().head()<jupyter_output><empty_output><jupyter_text>SUMMARY - step 1- ESML has now `Automap` and `Autoregister` Azure ML Datasets as: `IN, SILVER, BRONZE, GOLD`- ESML has read configuration for correct environment (DEV, TEST, PROD).     - Both small customers, and large Enterprise customers often wants:  DEV, TEST, PROD in `diffferent Azure ML workspaces` (and different subscriptions)- User has done feature engineering, and saved GOLD `p.save_gold`<jupyter_code>print("rows in GOLD {}".format(p.Gold.to_pandas_dataframe().shape[0]))
M10_GOLD_TRAIN, M10_GOLD_VALIDATE, M10_GOLD_TEST = p.split_gold_3(0.6,stratified=False) # Splits and Auto-registers as AZUREM ML Datasets<jupyter_output>...<jupyter_text>IN_2_GOLD- If just wanting to refine data to GOLD, for a Power BI report (No ML involved)- Scenario: You want to refine data from "IN_2_GOLD" with an easy way to READ/WRITE data (using the enterprise datalake via ESML AutoLake and ESML SDK)<jupyter_code>p.GoldTrain.to_pandas_dataframe().head()  # Azure ML Dataset<jupyter_output><empty_output><jupyter_text>2) `ESML` Train model in `5 codelines`<jupyter_code>p.dev_test_prod
automl_performance_config = p.get_automl_performance_config() # 1)Get config, for active environment (dev,test or prod)
automl_performance_config
print('Label is: {}'.format(p.active_model["label"]))
from esml import ESMLProject
from baselayer_azure_ml import AutoMLFactory,azure_metric_regression,azure_metric_classification
from azureml.train.automl import AutoMLConfig

automl_performance_config = p.get_automl_performance_config() # 1)Get config, for active environment (dev,test or prod)
aml_compute = p.get_training_aml_compute(p.ws) # 2)Get compute, for active environment

automl_config = AutoMLConfig(task = 'classification', # 4) Override the ENV config, for model(that inhertits from enterprise DEV_TEST_PROD config baseline)
                            primary_metric = azure_metric_classification.AUC, # # Note: Regression(MAPE) are not possible in AutoML
                            compute_target = aml_compute,
                            training_data = p.GoldTrain, # is 'train_6' pandas dataframe, but as an Azure ML Dataset
                            experiment_exit_score = '0.922', # DEMO purpose (0.308 for diabetes regression, 0.6 for classification titanic)
                            label_column_name = p.active_model["label"],
                            **automl_performance_config
                        )
via_pipeline = False # Consistent/same return values from both AutoML ALTERNATIVES (run or pipeline)
best_run, fitted_model, experiment = AutoMLFactory(p).train_pipeline(automl_config) if via_pipeline else AutoMLFactory(p).train_as_run(automl_config)<jupyter_output>Loading AutoML config settings from: test
Using a model specific cluster, per configuration in project specific settings, (the integer of 'model_number' is the base for the name)
Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_settings.json' -> override_enterprise_settings_with_model_specific=False)
Creating new cluster - prj02-m10-test
InProgress.....
SucceededProvisioning operation finished, operation "Succeeded"
Succeeded
AmlCompute wait for completion finished

Minimum number of nodes requested have been provisioned
image_build_compute = prj02-m10-test
Experiment name: 10_titanic_model_clas
Azure ML Studio Workspace: msft-weu-TEST-eap-proj02_ai-amls
Start training run...
Submitting remote run.
No run_configuration provided, running on prj02-m10-test with default configuration
Running on remote compute: prj02-m10-test<jupyter_text>- ESML has now fetched `configuration & train compute` for enterprise `environment (DEV,TEST or PROD)`- ESML has `autogenerated` a AutoML-experiment, optinally as `pipline`, in correct environment.- User has overridden some AutoML settings (`label, split percentage`, `target metric`), and use the `1-liner TRAIN` code snippet   3) Production purpose: "once and only once": Wrap code- 3 Callers: MLOps, AMLPipeline, and this notebook<jupyter_code>import repackage
repackage.add("../2_A_aml_pipeline/4_inference/batch/M10/your_code/")
from your_custom_code import M01In2GoldProcessor

#p.init()
esml_dataset1 = p.DatasetByName("ds01_titanic") # Get dataset 1
df_bronze = esml_dataset1.Bronze.to_pandas_dataframe()
silver1 = p.save_silver(esml_dataset1,df_bronze) #Bronze -> Silver

esml_dataset2 = p.DatasetByName("ds02_haircolor") # Get dataset 2
df_bronze2 = esml_dataset2.Bronze.to_pandas_dataframe()
silver2 = p.save_silver(esml_dataset2,df_bronze2) #Bronze -> Silver

df1 = M01In2GoldProcessor().M01_ds01_process_in2silver(silver1.to_pandas_dataframe())  # You can then copy this statement in your pipeline-step "in2silver_ds01...py"
df2 = M01In2GoldProcessor().M01_ds02_process_in2silver(silver2.to_pandas_dataframe())  # You can then copy this statement in your pipeline-step "in2silver_ds02...py"

merged_gold = M01In2GoldProcessor().M01_merge_silvers(df1,df2) # # You can then copy this statement in your pipeline-step "silver_merged_2_gold.py"
p.save_gold(merged_gold).to_pandas_dataframe().head()<jupyter_output><empty_output><jupyter_text>2b) ESML Scoring Drift/Concept Drift: Compare with `1-codeline`: Promote model or not? If better, then `Register model`- `IF` newly trained model in `current` environment (`DEV`, `TEST` or `PROD`) scores BETTER than existing model in `target` environment, then `new model` can be registered and promoted.- Q: Do we have `SCORING DRIFT / CONCEPT DRIFT?`- Q: Is a model trained on NEW data better? IS the one in production degraded? (not fit for the data it scores - real world changed, other CONCEPT)- A: - Lets check. Instead of `DataDrift`, lets look at `actual SCORING` on new data (and/or new code, feature engineering) - See if we should PROMOTE newly trained model...<jupyter_code>print("current AI Factory environment: '{}' - AML WS: '{}'".format(p.dev_test_prod, p.ws.name))
from baselayer_azure_ml_model import ESMLModelCompare

current_env = p.dev_test_prod # dev
target_env = "dev" # Does newly trained Model v3 in DEV, score better than Model v2 in TEST?
print("promote model in DEV to TEST? (move to other Azure ML Studio Workspace)")

compare = ESMLModelCompare(p)
promote,source_model_name,new_run_id,target_model_name, target_best_run_id,target_workspace,source_model = compare.compare_scoring_current_vs_new_model(target_env) # Compare DEV to TEST (or TEST to PROD)  (1min, 17sek VS 33sec)

print("SCORING DRIFT: If new model scores better in DEV (new data, or new code), we can promote this to TEST & PROD \n")
print("New Model: {} in environment {}".format(target_model_name, p.dev_test_prod))
print("Existing Model: {} in environment {}".format(source_model_name,target_env))

if (promote): # Can register="promote" a model in same workspace (test->test), or also register in OTHER Azure ML workspace (test->prod)
    if(p.dev_test_prod == target_env):
        compare.register_active_model(target_env,source_model) # if SAME workspace this brings more "metadata" faster to the model registration
    else:
        compare.register_model_in_correct_ws(target_env) # if REMOTE target workspace we can get same metadata, BUT, just takes performancewise longer. More lookups to "source Run"<jupyter_output>promote model in DEV to TEST? (move to other Azure ML Studio Workspace)
targe=source environement. Compare model version in DEV/TEST/PROD with latest registered in same DEV/TEST/PROD workspace (same workspace & subscriptiom comparison)

TARGET is in the same Azure ML Studio workspace as SOURCE, comparing with latest registered model...
target_best_run_id AutoML_4b3c03df-eaa6-43fd-b83b-a4857db588cc
New trained model & cached RUN, has TASK_TYPE: classification and Best_Run_id: AutoML_4b3c03df-eaa6-43fd-b83b-a4857db588cc_18
Target model & RUN, in Azure ML Studio workspace to compare with, has TASK_TYPE: classification and Best_Run_id:AutoML_4b3c03df-eaa6-43fd-b83b-a4857db588cc_18 

Q: Do we have SCORING DRIFT / CONCEPT DRIFT? - Is a model trained on NEW data better? = the one in production degraded?

New trained model: 
INFO: Using ESML TEST_SET SCORING, since tagged on MODEL - using this to compare SCORING
AUC (AUC_weighted): 0.860456
Accuracy: 0.831461
Precision (precision_score_weighte[...]<jupyter_text>TEST SET SCORING  Test-set: Ensure we have a TEST_SET splitted<jupyter_code>label = p.active_model["label"]
try:
    p.GoldTest.name
except: 
    p.connect_to_lake() # p.init() + automap
    train_6, validate_set_2, test_set_2 = p.split_gold_3(0.6)<jupyter_output>...<jupyter_text>NOW we can calcualate scoring on TEST_SET<jupyter_code>from baselayer_azure_ml import ESMLTestScoringFactory

auc,accuracy,f1, precision,recall,matrix,matthews,plt = ESMLTestScoringFactory(p).get_test_scoring_7_classification()

print("AUC:")
print(auc)
print()
print("Accuracy:")
print(accuracy)
print()
print("F1 Score:")
print(f1)
print()
print("Precision:")
print(precision)
print()
print("Recall:")
print(recall)
print()
print("Mathews correlation:")
print(matthews)
print()
print("Confusion Matrix:")
print(matrix)
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
labels = p.GoldTest.to_pandas_dataframe()[p.active_model["label"]].unique()
disp = ConfusionMatrixDisplay(confusion_matrix=matrix, display_labels=labels)
p1 = disp.plot()<jupyter_output>...<jupyter_text>3) ESML `Deploy model ONLINE` in `2 lines of code` (AKS) - Deploy "offline" MODEL from old `run` in environment To →  `DEV`, `TEST` or `PROD` environment- ESML saves `API_key in Azure keyvault automatically`- ESML auto-config solves 4 common 'errors/things': `correct compute name` and `valid replicas, valid agents, valid auto scaling`    - Tip: You can adjust the number of replicas, and different CPU/memory configuration, or using a different compute target.<jupyter_code>inference_config, model, best_run = p.get_active_model_inference_config(p.ws) #  Get compute power & lib-dependecies for DOCKER...for correct (Dev,Test or Prod) environment.
service,api_uri, kv_aks_api_secret= p.deploy_automl_model_to_aks(model,inference_config,overwrite_endpoint=True) # Deploy: AKS dockerized with correct config (Dev,Test or Prod subscription & networking)<jupyter_output>Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_settings.json' -> override_enterprise_settings_with_model_specific=False)
Loading AutoML config settings from: dev
Deploying model: AutoML4b3c03dfe18 with verison: 7 to environment: dev with overwrite_endpoint=True
Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_settings.json' -> override_enterprise_settings_with_model_specific=False)
image_build_compute = prj02-m10-dev
Found existing AksWebservice endpoint, deleting it, since overwrite=True
Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_settings.json' -> override_enterprise_settings_with_model_specific=False)
Found existing cluster, esml-dev-prj02, using it.
Note: Autoscale_enabled=False, or since aks_dev_test=True in config, autoscaling is automatically shut off, e.g. overridden in config (since n[...]<jupyter_text>3b) DEPLOY TEST with ESML `2 lines of code`<jupyter_code>p.connect_to_lake()
X_test, y_test, tags = p.get_gold_validate_Xy()
print(tags)
caller_id = "10965d9c-40ca-4e47-9723-5a608a32a0e4"
df = p.call_webservice(p.ws, X_test,caller_id) 
df.head()<jupyter_output>Using GEN2 as Datastore
Searching for setting in ESML datalake...
ESML in-folder settings override = FALSE. [active_in_folder.json,active_scoring_in_folder.json] not found in LAKE. 
 - Using [active_in_folder.json,active_scoring_in_folder.json] from ArgParse or GIT. No override from datalake settings
Path for active folder (where no files exists):
AdlsGen2-ListFiles (req=1, existingItems=0)' for 'https://msftweudevcmnai2.dfs.core.windows.net/lake3?directory=projects/project002/10_titanic_model_clas/train/active&recursive=true&resource=filesystem' on storage failed with status code 'NotFound' (The specified path does not exist.), client request ID 'd11f07e2-412f-49fd-aed8-f8d07664bfb3', request ID 'a471feb8-501f-0005-1b65-d7b20b000000
M10_GOLD_VALIDATE : (89, 8)
X_test  (89, 7)
y_test  (89,)
{'split_percentage': '0.1', 'label': 'Survived', 'model': '10_titanic_model_clas'}
Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_set[...]<jupyter_text>3b) ESML `DEPLOY - custom scoring` file - predict proba<jupyter_code>import os
os.chdir(os.path.dirname(globals()['_dh'][0]))

scoring_file = 'scoring_file_dev_M01_titanic.py'
script_file_local = "./settings/project_specific/model/dev_test_prod/train/automl/"+scoring_file
script_file_abs = os.path.abspath(script_file_local)

inference_config_to_override_and_inject, model, best_run = p.get_active_model_inference_config(p.ws)
inference_config_to_override_and_inject.entry_script = script_file_abs
inference_config_to_override_and_inject.entry_script # Verify path
# DEPLOY with custom InferenceConfig (custom scoring script)
service,api_uri, kv_aks_api_secret= p.deploy_automl_model_to_aks(model,inference_config_to_override_and_inject, True) #2) (model,inference_config, overwrite_endpoint=True,deployment_config=None):<jupyter_output>Deploying model: AutoML7e3ddc93d7 with verison: 1 to environment: dev with overwrite_endpoint=True
Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_settings.json' -> override_enterprise_settings_with_model_specific=False)
image_build_compute = prj02-m10-dev
Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_settings.json' -> override_enterprise_settings_with_model_specific=False)
Found existing cluster, esml-dev-prj02, using it.
Note: Autoscale_enabled=False, or since aks_dev_test=True in config, autoscaling is automatically shut off, e.g. overridden in config (since not supported) for environment dev
Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes.
Running
2021-10-05 00:45:26+02:00 Creating Container Registry if not exists.
2021-10-05[...]<jupyter_text>INFERENCE - Scenario "Caller/Client"  1) Get MockData - Get some TEST-DATA via ESMLProject...the GoldTest<jupyter_code>import repackage
repackage.add("../azure-enterprise-scale-ml/esml/common/")
from esml import ESMLDataset, ESMLProject

p = ESMLProject() # Will search in ROOT for your copied SETTINGS folder '../../../settings', you should copy template settings from '../settings'
p.inference_mode = False # We want "TRAIN" mode
p.ws = p.get_workspace_from_config() #2) Load DEV or TEST or PROD Azure ML Studio workspace
label = p.active_model["label"]
to_score = None
try:
    X_test = p.GoldTest.to_pandas_dataframe()
    to_score = X_test.drop([label], axis=1)
    #print(to_score.head()) # gold_test_1 = Dataset.get_by_name(ws, name=p.dataset_gold_test_name_azure)
except: 
    print ("you need to have splitted GOLD dataset, GoldTest need to exist. Change next cell from MARKDOWN, to CODE, and run that. Try this again... ")
# #X_test, y_test, tags = p.get_gold_validate_Xy() # Get the X_test data, ESML knows the SPLIT and LABEL already (due to training)<jupyter_output><empty_output><jupyter_text>2) Call AKS Webservice in 3 ways (A,B,C)- A) Also let AKS save data to lake- B) Use the ESML helper method (fetched keys from vault AND joins result + features)- C) Simulate "Rest only" - No ESML dependency     - No ESML meaning: Fetch keys by your own from vault + join/format JSON yourself + save data yourself to lake)  Alt 1 - ESML.call_webservice, `get PANDAS joined` dataframe `Also saves to LAKE, automatically`<jupyter_code>#p.lakestore = p.set_lake_as_datastore(p.ws) # For AutoSave - this i NOT needed if p.init() is done...which usually is the case.
p.call_webservice(p.ws, to_score,"caller_id").head() # (X_test, firstRowOnly=True,pandas_result=True, api_uri=None,api_key="auto from keyvault")<jupyter_output>Note: OVERRIDING enterprise performance settings with project specifics. (to change, set flag in 'dev_test_prod_settings.json' -> override_enterprise_settings_with_model_specific=False)
Note: Fetching keys automatically via workspace keyvault.
Saving scoring to lake for project folder project002 and inference_model_version: 1 ...
...

Saved DATA to score successfully in LAKE, as file 'to_score_caller_id.parquet'
Saved SCORED data in LAKE, as file 'scored_caller_id.parquet'<jupyter_text>Alt 2 - use compute factory, control to `get JSON back` instead of PANDAS.  `No saving to LAKE`<jupyter_code>result, model_version_used = p.compute_factory.call_webservice(to_score,False,False) # (X_test, firstRowOnly=True,pandas_result=True, api_uri=None,api_key="auto from keyvault")
df_res = pd.read_json(result)
to_score.join(df_res) # Need to join the FEATURES yourself, post webservice call (simulate no ESML dependancy in caller)<jupyter_output>Note: Fetching keys automatically via workspace keyvault.<jupyter_text>Alt 3 - Simulate client witn no ESML SDK, just using the "scoring endpoint". - Just JSON result (No ESML dependancy `get JSON back`) `No saving to LAKE` and `no JOIN`<jupyter_code>from baselayer_azure_ml import ComputeFactory
import json
keyvault = p.ws.get_default_keyvault() # Authentica to your Azure ML workspace (ws)
api_uri = keyvault.get_secret(name='esml-dev-p02-m10-api') 
api_key = keyvault.get_secret(name='esml-dev-p02-m10-apisecret') # DEV + Titanic

#api_uri = keyvault.get_secret(name='esml-test-p02-m10-api') # TEST + Titanic
#api_key = keyvault.get_secret(name='esml-test-p02-m10-apisecret')

result_json = ComputeFactory.call_webservice_static(to_score, api_uri,api_key,firstRowOnly=False) # Simulate "REST call" (no ESML dependancy, just a wrapper for a pytnon REST call)
res_dict = json.loads(result_json.text) # json -> dictionary
df_res = pd.read_json(res_dict) # dictionary -> pandas
all_result = X_test.join(df_res) # features + result
all_result
# And....you need to save the data yourself to the lake, at this location
to_score_folder, scored_folder, date_folder = p.get_gold_scored_unique_path()
print("Save your data here, if you want to have ADF WriteBack function")
print()
print(scored_folder)
print()
print("Note: Last folder, UUID folder, should represent a 'unique scoring' for a day, but can be injected. Example: if we want a customerGUID instead ")<jupyter_output>Save your data here, if you want to have ADF WriteBack function

projects/project002/01_titanic_model_clas/inference/1/scored/dev/2021_06_21/5f410d5b1ad64e14a9613ff7a7de2e14/

Note: Last folder, UUID folder, should represent a 'unique scoring' for a day, but can be injected. Example: if we want a customerGUID instead
{'fraction_non_alphanumeric': 0.09216125932366266, 'fraction_numerical': 0.020134535304077245, 'mean_word_length': 5.801742919389978, 'pattern_counts': {'":': 1, '<': 53, '<?xml version=': 0, '>': 71, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '606281', 'license': 'MIT', 'n_tokens_mistral': 7464, 'n_tokens_neox': 7083, 'n_words': 2046, 'path': 'copy_my_subfolders_to_my_grandparent/notebook_demos/esml_classification_1_train_env_dev.ipynb', 'repo_name': 'jostrm/azure-enterprise-scale-ml', 'size': '21853'}