<jupyter_start><jupyter_text>ESML - accelerator: Batch scoring pipeline- 1) `AutoMap datalake` & init ESML project- 2) `Get earlier trained model`- 3) `Score with GOLD_TEST` and calculate ML-performance   NB! This,InteractiveLoginAuthentication, is only needed to run 1st time, then when ws_config is written, use later CELL in notebook, that just reads that fileimport repackagerepackage.add("../azure-enterprise-scale-ml/esml/common/")from azureml.core import Workspacefrom azureml.core.authentication import InteractiveLoginAuthenticationfrom esml import ESMLDataset, ESMLProjectp = ESMLProject()p.dev_test_prod="dev"auth = InteractiveLoginAuthentication(tenant_id = p.tenant)ws, config_name = p.authenticate_workspace_and_write_config(auth)  NB!<jupyter_code>unregister_all_datasets=False
import repackage
repackage.add("../azure-enterprise-scale-ml/esml/common/")
from esml import ESMLProject

p = ESMLProject() # Will search in ROOT for your copied SETTINGS folder '../../../settings', you should copy template settings from '../settings'
p.active_model = 11
p.ws = p.get_workspace_from_config() #2) Load DEV or TEST or PROD Azure ML Studio workspace
p.inference_mode = False

if(unregister_all_datasets):
    p.unregister_all_datasets(p.ws) # For DEMO purpose
    
p.describe()
datastore = p.init(p.ws) # 3) Automapping from datalake to Azure ML datasets
# Feture engineering: Bronze 2 Gold - working with Azure ML Datasets with Bronze, Silver, Gold concept
esml_dataset = p.DatasetByName("ds01_diabetes") # Get dataset
df_bronze = esml_dataset.Bronze.to_pandas_dataframe()
p.save_silver(esml_dataset,df_bronze) #Bronze -> Silver

df = esml_dataset.Silver.to_pandas_dataframe() 
df_filtered = df[df.AGE > 0.015] 
gold_train = p.save_gold(df_filtered)  #Silver -> Gold<jupyter_output><empty_output><jupyter_text>SUMMARY - step 1- ESML has now `Automap` and `Autoregister` Azure ML Datasets as: `IN, SILVER, BRONZE, GOLD`- ESML has read configuration for correct environment (DEV, TEST, PROD).     - Both small customers, and large Enterprise customers often wants:  DEV, TEST, PROD in `diffferent Azure ML workspaces` (and different subscriptions)- User has done feature engineering, and saved GOLD `p.save_gold`  2) `Regression` - Get diabetes modelhttps://docs.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-machine-learning-tutorial  TEST_SET scoring: REGRESSION - Autoregisters in Azure ML Studio, the TEST-scoring as TAGS on the GOLD_TEST dataset i - locally on build server, or via pipeline.<jupyter_code>datastore = p.connect_to_lake()
from baselayer_azure_ml import ESMLTestScoringFactory

label = p.active_model["label"]
#p.split_gold_3(0.6,label)
rmse, r2, mean_abs_percent_error,mae,spearman_corr,plt = ESMLTestScoringFactory(p).get_test_scoring_4_regression(label)
print("RMSE:")
print(rmse)
print()
print("R2:")
print(r2)
print()
print("MAPE:")
print(mean_abs_percent_error)
print()
print("MAE:")
print(mae)
print()
print("Spearman:")
print(spearman_corr)
print("From Azure ML Dataset - fetch from Azure ML Studio Dataset tags")
print("RMSE {:.5}".format(p.GoldTest.tags["RMSE"]))
print("From MODEL - fetch from Azure ML Studio MODEL tags")
print("Model name: ", p.BestModel.name)
print("RMSE: {:.5}".format(p.BestModel.tags["test_set_RMSE"]))<jupyter_output>From MODEL - fetch from Azure ML Studio MODEL tags
Loading AutoML config settings from: dev
{'fraction_non_alphanumeric': 0.10670553935860058, 'fraction_numerical': 0.007580174927113703, 'mean_word_length': 5.610789980732177, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 11, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '627990', 'license': 'MIT', 'n_tokens_mistral': 1177, 'n_tokens_neox': 1132, 'n_words': 307, 'path': 'copy_my_subfolders_to_my_grandparent/notebook_demos/esml_regression_6_score_testset_regr.ipynb', 'repo_name': 'jostrm/azure-enterprise-scale-ml', 'size': '3430'}