<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9857f453-47b6-488b-989a-3d68729e2fec",
   "metadata": {},
   "source": [
    "# Analysing Pipeline components\n",
    "\n",
    "In this section we will go through each components in the pipeline and anlayse its function in the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e8568f-ca84-42d1-8676-494903ac7d1a",
   "metadata": {},
   "source": [
    "## Part 1: Diving into data\n",
    "The first task in any data science or ML project is to understand and clean the data.\n",
    "\n",
    "* Understanding the data types, distributions, and other information (e.g., mean value, or number of uniques) about each feature\n",
    "* Generating a preliminary schema that describes the data\n",
    "* Identifying anomalies and missing values in the data with respect to given schema\n",
    "\n",
    "For this part the following four components are involved\n",
    "\n",
    "![analysing-pipeline-2.png](img/analysing-pipeline-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c256a0a-fd14-4dd7-a1f4-0a79e9a97003",
   "metadata": {},
   "source": [
    "[ExampleGen](https://www.tensorflow.org/tfx/guide/examplegen) ingests and splits the input dataset.\n",
    "\n",
    "```\n",
    " #airflow/dags/taxi_pipeline.py\n",
    " example_gen = CsvExampleGen(input_base=data_root)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a0c9402-3465-4f51-96b7-e157a2a8129e",
   "metadata": {},
   "source": [
    "[StatisticsGen](https://www.tensorflow.org/tfx/guide/statsgen) calculates statistics for the dataset.\n",
    "\n",
    "```\n",
    " #airflow/dags/taxi_pipeline.py\n",
    " statistics_gen = StatisticsGen(\n",
    "    examples=example_gen.outputs['examples'])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c85d7362-2333-46ef-b459-6b0c0752b58a",
   "metadata": {},
   "source": [
    "[SchemaGen](https://www.tensorflow.org/tfx/guide/schemagen) examines the statistics and creates a data schema.\n",
    "\n",
    "```\n",<commit_msg>adding spaces in notebook.ipynb<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9857f453-47b6-488b-989a-3d68729e2fec",
   "metadata": {},
   "source": [
    "# Analysing Pipeline components\n",
    "\n",
    "In this section we will go through each components in the pipeline and anlayse its function in the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e8568f-ca84-42d1-8676-494903ac7d1a",
   "metadata": {},
   "source": [
    "## Part 1: Diving into data\n",
    "The first task in any data science or ML project is to understand and clean the data.\n",
    "\n",
    "* Understanding the data types, distributions, and other information (e.g., mean value, or number of uniques) about each feature\n",
    "* Generating a preliminary schema that describes the data\n",
    "* Identifying anomalies and missing values in the data with respect to given schema\n",
    "\n",
    "For this part the following four components are involved\n",
    "\n",
    "![analysing-pipeline-2.png](img/analysing-pipeline-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c256a0a-fd14-4dd7-a1f4-0a79e9a97003",
   "metadata": {},
   "source": [
    "[ExampleGen](https://www.tensorflow.org/tfx/guide/examplegen) ingests and splits the input dataset.\n",
    "\n",
    "```\n",
    " #airflow/dags/taxi_pipeline.py\n",
    " example_gen = CsvExampleGen(input_base=data_root)\n",
    "```\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a0c9402-3465-4f51-96b7-e157a2a8129e",
   "metadata": {},
   "source": [
    "[StatisticsGen](https://www.tensorflow.org/tfx/guide/statsgen) calculates statistics for the dataset.\n",
    "\n",
    "```\n",
    " #airflow/dags/taxi_pipeline.py\n",
    " statistics_gen = StatisticsGen(\n",
    "    examples=example_gen.outputs['examples'])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c85d7362-2333-46ef-b459-6b0c0752b58a",
   "metadata": {},
   "source": [
    "[SchemaGen](https://www.tensorflow.org/tfx/guide/schemagen) examines the statistics and creates a data schema.\n",
    "\n",
    "```\n",
{'fraction_non_alphanumeric': 0.1769890424011434, 'fraction_numerical': 0.055264411624583135, 'mean_word_length': 3.1369458128078818, 'pattern_counts': {'":': 42, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '255142', 'n_tokens_mistral': 1682, 'n_tokens_neox': 1547, 'n_words': 332}