<commit_before>// Databricks notebook source
// MAGIC %md
// MAGIC 
// MAGIC # [SDS-2.2, Scalable Data Science](https://lamastex.github.io/scalable-data-science/sds/2/2/)

// COMMAND ----------

// MAGIC %md
// MAGIC This is *slightly* outdated, but good enough for our purposes as we will do it live on the latest databricks notebooks.
// MAGIC 
// MAGIC Please go here for a relaxed and detailed-enough tour (later):
// MAGIC 
// MAGIC * [https://docs.databricks.com/index.html](https://docs.databricks.com/index.html)

// COMMAND ----------

// MAGIC %md
// MAGIC # Notebooks
// MAGIC Write Spark code for processing your data in notebooks. 
// MAGIC 
// MAGIC **NOTE**: You should have already cloned this notebook and attached it to a cluster that you started in the Community Edition of databricks by now.

// COMMAND ----------

// MAGIC %md<commit_msg>vids linked<commit_after>// Databricks notebook source
// MAGIC %md
// MAGIC 
// MAGIC # [SDS-2.2, Scalable Data Science](https://lamastex.github.io/scalable-data-science/sds/2/2/)

// COMMAND ----------

// MAGIC %md
// MAGIC Archived YouTube videos of this live unedited lab-lecture:
// MAGIC 
// MAGIC [![Archived YouTube video of this live unedited lab-lecture](http://img.youtube.com/vi/zloLA6AyNqk/0.jpg)](https://www.youtube.com/embed/zloLA6AyNqk?start=0&end=3226&autoplay=1) [![Archived YouTube video of this live unedited lab-lecture](http://img.youtube.com/vi/36tQR9UPXP4/0.jpg)](https://www.youtube.com/embed/36tQR9UPXP4?start=0&end=2341&autoplay=1)

// COMMAND ----------

// MAGIC %md
// MAGIC This is *slightly* outdated, but good enough for our purposes as we will do it live on the latest databricks notebooks.
// MAGIC 
// MAGIC Please go here for a relaxed and detailed-enough tour (later):
// MAGIC 
// MAGIC * [https://docs.databricks.com/index.html](https://docs.databricks.com/index.html)

// COMMAND ----------

// MAGIC %md
// MAGIC # Notebooks
// MAGIC Write Spark code for processing your data in notebooks. 
// MAGIC 
// MAGIC **NOTE**: You should have already cloned this notebook and attached it to a cluster that you started in the Community Edition of databricks by now.

// COMMAND ----------

// MAGIC %md
{'fraction_non_alphanumeric': 0.15787079162875342, 'fraction_numerical': 0.014558689717925387, 'mean_word_length': 3.0127737226277373, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4131010', 'n_tokens_mistral': 797, 'n_tokens_neox': 701, 'n_words': 224}