<jupyter_start><jupyter_text>%pip install git+https://github.com/finos/legend-delta.gitsubdirectory=python<jupyter_code>%sql
DROP DATABASE IF EXISTS legend CASCADE;
CREATE DATABASE legend;<jupyter_output><empty_output><jupyter_text>Legend modelLegend project can be loaded from classpath or directory as follows<jupyter_code>from legend.delta import LegendClasspathLoader
legend = LegendClasspathLoader().loadResources()<jupyter_output><empty_output><jupyter_text>Legend schemaWe can create the spark schema for any Legend entity of type `Class`. This process will recursively loop through each of its underlying fields, enums and possibly nested properties and supertypes.<jupyter_code>schema = legend.get_schema("databricks::entity::employee")
import pandas as pd
display(pd.DataFrame(
  [[f.name, str(f.dataType), f.nullable, f.metadata['comment']] for f in schema.fields], 
  columns=['field', 'type', 'optional', 'description']
))<jupyter_output><empty_output><jupyter_text>Legend transformationsWe can transform raw entities into their desired target tables. Note that relational transformations only support direct mapping and therefore easily enforced through `.withColumnRenamed` syntax.<jupyter_code>transformations = legend.get_transformations("databricks::mapping::employee_delta")
import pandas as pd
display(pd.DataFrame(
  [[e, transformations[e]] for e in transformations], 
  columns=['from_column', 'to_column']
))<jupyter_output><empty_output><jupyter_text>Legend expectationsGiven the `multiplicity` properties, we can detect if a field is optional or not or list has the right number of elements. Given an `enumeration`, we check for value consistency. These will be considered **technical expectations** and converted into SQL constraints. In addition to the rules derived from the schema itself, we also support the conversion of **business expectations**from the PURE language to SQL expressions. We generate a legendexecution plan against a Databricks runtime, hence operating against relational legend `mapping` ratherthan pure entities of type `class`.<jupyter_code>expectations = legend.get_expectations("databricks::mapping::employee_delta")
import pandas as pd
display(pd.DataFrame(
  [[e, expectations[e]] for e in expectations], 
  columns=['expectation', 'constraint']
))<jupyter_output><empty_output><jupyter_text>Legend derivationsWe can convert Legend derived properties as SQL expressions. In the example model, the field `age` is not physically stored but can be computed at runtime.<jupyter_code>derivations = legend.get_derivations("databricks::mapping::employee_delta")
import pandas as pd
display(pd.DataFrame(
  [[e, derivations[e]] for e in derivations], 
  columns=['column', 'expression']
))<jupyter_output><empty_output><jupyter_text>Legend tablesIn order to query our validated entity from legend interface, we can easily create the target state table. This table contains a placeholder for our invalidated constraints (below field `legend`).<jupyter_code>table_name = legend.create_table("databricks::mapping::employee_delta")
display(sql("DESCRIBE EXTENDED {}".format(table_name)))<jupyter_output><empty_output><jupyter_text>Example - readFrom delta, we read objects that we transform back as a pure entity with derived properties and violated constraints. New derivations could be added from legend studio and seamlessly computed here without the need for engineering team to code. The generated dataframe would comply with business expectations and data quality, as defined from the legend studio.<jupyter_code>df = legend.query('databricks::mapping::employee_delta')
display(df.limit(10))<jupyter_output><empty_output><jupyter_text>Given the following service defined on legend studio, we generate the corresponding spark execution plan and return a dataframe with all requested attributes and calculations```|databricks::entity::employee.all()->filter(  x|$x.firstName->startsWith('G'))->project(  [    x|$x.firstName,    x|$x.lastName,    x|$x.highFives,    x|$x.age,    x|$x.sme  ],  [    'FirstName',    'LastName',    'HighFives',    'Age',    'Sme'  ])->sort(  [    desc('HighFives')  ])->take(10)```<jupyter_code>df = legend.query('databricks::service::employee')
display(df.limit(10))<jupyter_output><empty_output><jupyter_text>The same works against aggregated functions like `groupBy````|databricks::entity::employee.all()->filter(  x|!($x.gender->isEmpty()))->groupBy(  [    x|$x.gender  ],  [    agg(      x|$x.highFives,      x|$x->average()    ),    agg(      x|$x.id,      x|$x->count()    )  ],  [    'Gender',    'HighFives',    'Employees'  ])->sort(  [    desc('HighFives')  ])->take(10)```<jupyter_code>df = legend.query('databricks::service::skills')
display(df.limit(10))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11285594639865996, 'fraction_numerical': 0.0020938023450586263, 'mean_word_length': 6.6432, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '437018', 'license': 'Apache-2.0', 'n_tokens_mistral': 1430, 'n_tokens_neox': 1340, 'n_words': 415, 'path': 'databricks-python.ipynb', 'repo_name': 'aamend/legend-delta', 'size': '4776'}