<jupyter_start><jupyter_text>Azure Purview for AI - Demo Initial Setup<jupyter_code>import dotenv
from pyapacheatlas.auth import ServicePrincipalAuthentication
from pyapacheatlas.core import PurviewClient

dotenv.load_dotenv()

# authenticate and instantiate client
tenant_id = os.environ["TENANT_ID"]
client_id = os.environ["SP_ID"]
client_secret = os.environ["SP_SECRET"]

auth = ServicePrincipalAuthentication(
    tenant_id=tenant_id,
    client_id=client_id,
    client_secret=client_secret
)

client = PurviewClient(account_name="ml-purview", authentication=auth)<jupyter_output><empty_output><jupyter_text>Problem settingThe vast majority of enterprise AI/ML use cases begin with raw data being copied into cloud storage. Usually this is the staging area of the enterprise data lake. In  this demo, we use the storage account `topsecretdata` as a stand-in for the enterprise data lake and the staging area is the container `data`. At the beginning of the demo, we assume that the raw data has just been copied there, and the same file will be replaced for future runs. The raw data in this case are two files: `iris-train.csv` which contains a subset of rows from iris dataset, and `iris-score.csv` which contains the remaining rows, but with the `Species` column removed. We can view this by navigating to Browse Assets > Azure Blob Storage > topsecretdata > data. Click on `iris-train.csv` and examine all the details. Click on "edit" and add/update the description.Click on lineage. If lineage is not empty:- click on the box marked TrainETL > switch to asset. - Copy the GUID from the URL- Paste it in the cell below and run.- Navigate back to the `iris-train.csv` asset lineage, hit refresh, and the lineage should be empty.![](./.img/train-lineage.png)<jupyter_code># Deletes the TrainETL entity for the demo

guid="bc8b1732-df57-4776-ac79-884784928aa2"
client.delete_entity(guid=guid)<jupyter_output><empty_output><jupyter_text>Lineage with Azure Data FactoryTypically, before data analysis, the raw data is transformed and loaded into a relational database. This could be a DataVault 2.0 model in the enterprise data warehouse, or a small database dedicated to data and ML use cases. Go to the [data factory](https://adf.azure.com/en-us/home?factory=%2Fsubscriptions%2Fd50ade7c-2587-4da8-9c63-fc828541722c%2FresourceGroups%2Frgp-show-weu-aml-databricks%2Fproviders%2FMicrosoft.DataFactory%2Ffactories%2Fadf-mlops-demo), select the pipeline `pl_train_etl` and launch a debug run. The run completes in about 15 seconds. Navigate to the `train-iris.csv` asset again, click on lineage, refresh, and the full lineage should be visible as seen below.![](./.img/train-lineage-full.png)This is en extremely powerful concept on azure since we are able to see that the `traindata` database table is loaded from the `train-iris.csv` file, and that changes in the structure of that file would cause the ETL to break, so the TrainETL Copy Activity and the traindata database table wouls need to be modified.   Customized lineage in Azure PurviewPurview is based on Apache Atlas which is a free and open source metadata management tool. As a result we can extend the functionality of Purview bu creating new entities.For example, Azure ML Pipelines are not a built-in type in Purview. Below we have some code to create a custom type definition to represent an Azure ML pipeline. We can then instantiate this type to represent specific pipeline that we are deploying. Note that since the ML pipeline inherits from the built-in `Process` type, we can specify and array of input and output `Dataset` elements to represent data flows in and out of the pipeline.  This code is used to create a custom type to represent an Azure ML Pipeline. No need to run this.```python Define ML Pipelne Process Typefrom pyapacheatlas.core import EntityTypeDef, AtlasAttributeDeffrom pyapacheatlas.core.typedef import Cardinalitypipeline_name = AtlasAttributeDef(    name="pipeline_name",    displayName="Pipeline Name",    description="Name of the Azure ML pipeline")pipeline_owner = AtlasAttributeDef(    name="pipeline_owner",    displayName="Pipeline Owner",    description="Name of the main developer of the ML pipeline")process = EntityTypeDef(    name="azureml_pipeline",    superTypes=["Process"],    attributeDefs = [pipeline_name, pipeline_owner])process.to_json()client.upload_typedefs(process)``` Navigate to the `iris-score.csv` asset in the asset browser. Verify that the schema does not contain the label.  Navigate to the lineage, copy the GUID from the url and delete the entity by running the cell below, just as we did before for the `TrainETL` Copy Activity.<jupyter_code>guid="e3848aa7-d6d8-4cc3-a139-a6cbeb382341"

client.delete_entity(guid=guid)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06675036618539443, 'fraction_numerical': 0.015693659761456372, 'mean_word_length': 5.306068601583114, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '270758', 'license': 'MIT', 'n_tokens_mistral': 1377, 'n_tokens_neox': 1315, 'n_words': 584, 'path': 'purview/custom_entity_example.ipynb', 'repo_name': 'nihil0/mlops', 'size': '4779'}