<jupyter_start><jupyter_text>Purview | Custom process> How to create a custom Purview dataset and process with python- toc: true- badges: true- comments: true- categories: [purview, python, jupyter]- hide: false- image: images/social/purview_custom_process.png  Prerequisites<jupyter_code>%pip install pyapacheatlas<jupyter_output>Collecting pyapacheatlas
  Downloading pyapacheatlas-0.10.0-py3-none-any.whl (68 kB)
     |████████████████████████████████| 68 kB 1.9 MB/s             
[?25hCollecting openpyxl>=3.0
  Downloading openpyxl-3.0.9-py2.py3-none-any.whl (242 kB)
     |████████████████████████████████| 242 kB 3.8 MB/s            
[?25hRequirement already satisfied: requests>=2.0 in /home/daforne/repos/github/davidefornelli/blog/.venv/lib/python3.7/site-packages (from pyapacheatlas) (2.26.0)
Collecting et-xmlfile
  Using cached et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)
Requirement already satisfied: certifi>=2017.4.17 in /home/daforne/repos/github/davidefornelli/blog/.venv/lib/python3.7/site-packages (from requests>=2.0->pyapacheatlas) (2021.10.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/daforne/repos/github/davidefornelli/blog/.venv/lib/python3.7/site-packages (from requests>=2.0->pyapacheatlas) (2.0.7)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/daforne/repos/g[...]<jupyter_text>Code  Import<jupyter_code>from pyapacheatlas.auth import ServicePrincipalAuthentication
from pyapacheatlas.core.client import PurviewClient
from pyapacheatlas.core.entity import AtlasEntity
from pyapacheatlas.core.typedef import EntityTypeDef
from pyapacheatlas.core.util import GuidTracker
from pyapacheatlas.core.typedef import ChildEndDef
from pyapacheatlas.core.typedef import RelationshipTypeDef
from pyapacheatlas.core.typedef import ParentEndDef
from pyapacheatlas.core.typedef import Cardinality
from pyapacheatlas.core.typedef import AtlasAttributeDef
from pyapacheatlas.core import AtlasProcess<jupyter_output><empty_output><jupyter_text>Settings<jupyter_code>tenant_id = ""
client_id = ""
client_secret = ""
purview_name = ""<jupyter_output><empty_output><jupyter_text>Clients<jupyter_code>atlas_sp = ServicePrincipalAuthentication(
    tenant_id=tenant_id,
    client_id=client_id,
    client_secret=client_secret
)

purview_client = PurviewClient(
    account_name=purview_name,
    authentication=atlas_sp
)<jupyter_output><empty_output><jupyter_text>Entities<jupyter_code>guid_tracker = GuidTracker()

custom_dataset = purview_client.upload_typedefs(
    entityDefs=[
        EntityTypeDef(
            name="myCustomDataSet",
            superTypes=["DataSet"]
        )
    ],
    force_update=True
)

myCustomDataset01 = AtlasEntity(
    name="myCustomDataset01",
    typeName="myCustomDataSet",
    qualified_name="pyapacheatlas://mycustomdataset01",
    guid=guid_tracker.get_guid()
)
myCustomDataset02 = AtlasEntity(
    name="myCustomDataset02",
    typeName="myCustomDataSet",
    qualified_name="pyapacheatlas://mycustomdataset02",
    guid=guid_tracker.get_guid()
)

myCustomProcess01 = AtlasProcess(
    name="myCustomProcess01",
    typeName="Process",
    qualified_name="pyapacheatlas://mycustomprocess01",
    inputs=[myCustomDataset01],
    outputs=[myCustomDataset02],
    guid=guid_tracker.get_guid()
)

results = purview_client.upload_entities(
    batch=[myCustomDataset01, myCustomDataset02, myCustomProcess01]
)<jupyter_output><empty_output><jupyter_text>Result![Call graph](images/purview_custom_process.png)  Utils  Delete entities<jupyter_code>purview_client.delete_typedefs(
    entityDefs=[
        {"name": "myCustomProcess"},
    ]
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12387202625102543, 'fraction_numerical': 0.029532403609515995, 'mean_word_length': 5.650909090909091, 'pattern_counts': {'":': 1, '<': 25, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '579510', 'license': 'Apache-2.0', 'n_tokens_mistral': 1359, 'n_tokens_neox': 1258, 'n_words': 191, 'path': '_notebooks/2021-12-14-nb_purview_01.ipynb', 'repo_name': 'davidefornelli/blog', 'size': '3657'}