<jupyter_start><jupyter_text>Introducing Databricks and Notebooks Latest version of this notebook in my DataBricks account at > https://community.cloud.databricks.com/?o=2196741575149439notebook/3862096147728353  Managing and working with Databricks notebooksBlock type page creation - Similar to Confluence, Colab, any Ipython based notebook or even the Wordpress new editor>Managing Notebooks https://docs.databricks.com/notebooks/notebooks-manage.html  Markdown SyntaxA quick intro of MD is required to write documentation in the notebook. There is no point if we do are old school and just code in Notebooks and not make Code+Documentation when it is so easy to do soHeaders are assigned by putting === or --- in the next line. Else just use  at the start of the line. Different levels are assigned by the no of .A paragraph is created by an empty lineTo get the full syntax documentation, go to <NAME>'s Markdown Syntax page - > This is where all started - http://daringfireball.net/projects/markdown/syntax.List are created with 1. or *  any number will give a serial list till a blank line 1. Text Between ** or __ is Bold2. Text Between * or _ is Italic1. Text Between *** or ___ is Bold and ItalicBackquote is inserted with a > at the start of a line which continues till a blank line>** Bold ** __Bold__ _Italic_ *Italic* ___Bold and Italic___ ***Bold and Italic***  Introducing DatabricksNow we have seen Notebooks , let us dig into a simple and quick introduction of DatabricksToday Databricks is something of a buzzword in Data Science. It is said Databricks provides quicker calculations and answers. > check for quick intro https://www.advectas.com/en/blog/what-is-databricks/This is because code execution is distributed between several computers (known as a cluster), thus offering better utilisation of available computer power in a cloud, The whole idea of a notebook in Jupyter is it has one kernel which means one primary language per notebook. DB Notebooks also has similar concept - You choose your language when you create a notebook. But you must have noticed that it is called primary language. Because you have the option to override the language for a cell/block by %sql %r %python %scala etc . Similar to %md that can be used to make a block text only and to use Markdown syntax  An example of ETL in traditional sense - We are trying to do the same in DB and SparkA standard data pipeline >source - extract - clean/merge/filter - tempload - tx - tmpload - aggregate/append/SCD - finalload>TruncateLoad to delta file/table - AppendLoad to Archive file/tableYou noticed apart from the transformations, the most important action in a datapipline is your commit designFor that lets first have a quick recap of Slowly Changing Dimension or SCDTypes of SCD* Type 0 – Fixed Dimension. No changes allowed, dimension never changes. Whole table refresh when required* Type 1 – No History. Update record directly, there is no record of historical values, only current state.> We often just want to see the current value of a dimension attribute – it could be that changes to dimensions are required only to correct mistakes - hence hitorical reporting is not expected. But i always felt even here a metadata is required for tracking changes. An audit column added which gives when last updated does the job most of the time.* Type 2 – Row Versioning.Track changes as version records with current flag & active dates and other metadata* Type 3 – Previous Value column.Track change to a specific attribute, add a column to show the previous value, which is updated as further changes occur.>But this will only provide a single historical value for the column for which we decide to maintain previous value.* Type 4 – History Table. Show current value in dimension table but track all changes in separate table>There is no change to our existing table here, we simply update the record as if a Type 1 change had occurred. However, we simultaneously maintain a type 4 history table to keep track of these changes: Whilst our Type 4 historical table is created as: Depending on your requirements, you may place both ID and Surrogate Key onto the fact record so that you can optimise performance whilst maintaining functionality. Separating the historical data makes your dimensions smaller and therefore reduces complexity and improves performance if the majority of uses only need the current value.However, if you do require historical values, this structure adds complexity and data redundancy overheads. It is generally assumed that the system will use Type 1 or Type 2 rather than Type 4.* Type 6 – Hybrid SCD.Utilise techniques from SCD Types 1, 2 and 3 to track change.>maintain a history of all changes whilst simultaneously updating a “current value” column on all records.This gives you the ability to provide an element of change comparison without additional calculation, whilst still maintaining a full, detailed history of all changes in the system.>You must have noticed that there is no SCD type 5. Anyway only type 0,1 or 2 is used. Others, it is said are there for very specific requirements. I have never used them or never felt the need to propose them in any solutions/architecture. When you mess up an architecture due to code debt - in due course of time you land up with Type 6. Or if you end up with Kimball whilst you wished for an Inmon architecture<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0368052999631947, 'fraction_numerical': 0.010305483989694516, 'mean_word_length': 4.54026503567788, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 19, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '306329', 'license': 'CC0-1.0', 'n_tokens_mistral': 1316, 'n_tokens_neox': 1231, 'n_words': 845, 'path': 'DB4ETL1.ipynb', 'repo_name': 'kpmishraindia/Databricks4ETL', 'size': '5439'}