<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "8c580981-fa5a-43a9-ae99-0251cda5b7a9",
    "_uuid": "8510ac1d600e8b4c1e27a34a84fb4cc703970d37"
   },
   "source": [
    "*This tutorial is part Level 2 in the [Learn Machine Learning](https://www.kaggle.com/learn/machine-learning) curriculum. This tutorial picks up where Level 1 finished, so you will get the most out of it if you've done the exercise from Level 1.*\n",
    "\n",
    "In this step, you will learn three approaches to dealing with missing values. You will then learn to compare the effectiveness of these approaches on any given dataset.* \n",
    "\n",
    "# Introduction\n",
    "\n",
    "There are many ways data can end up with missing values. For example\n",
    "- A 2 bedroom house wouldn't include an answer for _How large is the third bedroom_\n",
    "- Someone being surveyed may choose not to share their income\n",
    "\n",
    "Python libraries represent missing numbers as **nan** which is short for \"not a number\".  You can detect which cells have missing values, and then count how many there are in each column with the command:\n",
    "```\n",
    "missing_val_count_by_column = (data.isnull().sum())\n",
    "print(missing_val_count_by_column[missing_val_count_by_column > 0\n",
    "```\n",
    "\n",
    "Most libraries (including scikit-learn) will give you an error if you try to build a model using data with missing values. So you'll need to choose one of the strategies below.\n",
    "\n",
    "---\n",
    "## Solutions\n",
    "\n",
    "\n",
    "## 1) A Simple Option: Drop Columns with Missing Values\n",
    "If your data is in a DataFrame called `original_data`, you can drop columns with missing values. One way to do that is\n",
    "```\n",
    "data_without_missing_values = original_data.dropna(axis=1)\n",
    "```\n",
    "\n",
    "In many cases, you'll have both a training dataset and a test dataset.  You will want to drop the same columns in both DataFrames. In that case, you would write\n",
    "\n",
    "```\n",
    "cols_with_missing = [col for col in original_data.columns \n",
    "                                 if original_data[col].isnull().any()]\n",
    "reduced_original_data = original_data.drop(cols_with_missing, axis=1)\n",
    "reduced_test_data = test_data.drop(cols_with_missing, axis=1)\n",
    "```\n",
    "If those columns had useful information (in the places that were not missing), your model loses access to this information when the column is dropped. Also, if your test data has missing values in places where your training data did not, this will result in an error.  \n",
    "\n",
    "So, it's somewhat usually not the best solution. However, it can be useful when most values in a column are missing.\n",
    "\n",
    "\n",
    "\n",
    "## 2) A Better Option: Imputation\n",
    "Imputation fills in the missing value with some number. The imputed value won't be exactly right in most cases, but it usually gives more accurate models than dropping the column entirely.\n",
    "\n",
    "This is done with\n",
    "```\n",
    "from sklearn.impute import SimpleImputer\n",
    "my_imputer = SimpleImputer()\n",
    "data_with_imputed_values = my_imputer.fit_transform(original_data)\n",
    "```\n",
    "The default behavior fills in the mean value for imputation.  Statisticians have researched more complex strategies, but those complex strategies typically give no benefit once you plug the results into sophisticated machine learning models.\n",
    "\n",
    "One (of many) nice things about Imputation is that it can be included in a scikit-learn Pipeline. Pipelines simplify model building, model validation and model deployment.\n",
    "\n",
    "## 3) An Extension To Imputation\n",
    "Imputation is the standard approach, and it usually works well.  However, imputed values may by systematically above or below their actual values (which weren't collected in the dataset). Or rows with missing values may be unique in some other way. In that case, your model would make better predictions by considering which values were originally missing.  Here's how it might look:\n",
    "```\n",
    "# make copy to avoid changing original data (when Imputing)\n",
    "new_data = original_data.copy()\n",
    "\n",
    "# make new columns indicating what will be imputed\n",
    "cols_with_missing = (col for col in new_data.columns \n",
    "                                 if new_data[col].isnull().any())\n",
    "for col in cols_with_missing:\n",
    "    new_data[col + '_was_missing'] = new_data[col].isnull()\n",
    "\n",
    "# Imputation\n",
    "my_imputer = SimpleImputer()\n",
    "new_data = pd.DataFrame(my_imputer.fit_transform(new_data))\n",
    "new_data.columns = original_data.columns\n",
    "```\n",
    "\n",
    "In some cases this approach will meaningfully improve results. In other cases, it doesn't help at all.\n",
    "\n",
    "---\n",
    "# Example (Comparing All Solutions)\n",
    "\n",
    "We will see am example predicting housing prices from the Melbourne Housing data.  To master missing value handling, fork this notebook and repeat the same steps with the Iowa Housing data.  Find information about both in the **Data** section of the header menu.\n",
    "\n",
    "\n",
    "### Basic Problem Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "a5e604e7-3c20-409b-ad29-5c2a4fe40738",
    "_uuid": "44b399828f0b07fe63abbdcdf74bbf3b22bb8067",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Load data\n",
    "melb_data = pd.read_csv('../input/melbourne-housing-snapshot/melb_data.csv')\n",
    "\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "melb_target = melb_data.Price\n",
    "melb_predictors = melb_data.drop(['Price'], axis=1)\n",
    "\n",
    "# For the sake of keeping the example simple, we'll use only numeric predictors. \n",
    "melb_numeric_predictors = melb_predictors.select_dtypes(exclude=['object'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "88d1c1c9-91ba-4bef-aee5-f4a19a68e61c",
    "_uuid": "de7bdb4f005022ea45742b5d25f47cba7a6d698d"
   },
   "source": [
    "### Create Function to Measure Quality of An Approach\n",
    "We divide our data into **training** and **test**. If the reason for this is unfamiliar, review [Welcome to Data Science](https://www.kaggle.com/dansbecker/welcome-to-data-science-1).\n",
    "\n",
    "We've loaded a function `score_dataset(X_train, X_test, y_train, y_test)` to compare the quality of diffrent approaches to missing values. This function reports the out-of-sample MAE score from a RandomForest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "986dbfe4-9386-4a03-b2a4-9e99bf1b08f5",
    "_kg_hide-input": true,
    "_uuid": "6088bfdac20ece9c040e83beb28ff169d17f0666",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(melb_numeric_predictors, \n",
    "                                                    melb_target,\n",
    "                                                    train_size=0.7, \n",
    "                                                    test_size=0.3, \n",
    "                                                    random_state=0)\n",
    "\n",
    "def score_dataset(X_train, X_test, y_train, y_test):\n",
    "    model = RandomForestRegressor()\n",
    "    model.fit(X_train, y_train)\n",
    "    preds = model.predict(X_test)\n",
    "    return mean_absolute_error(y_test, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "7184fc99-e266-4bd7-af0d-0da9c96887f0",
    "_uuid": "df0103056e52ffb3c500f5fc1437bd175a41adad"
   },
   "source": [
    "### Get Model Score from Dropping Columns with Missing Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "64ae7a0a-95aa-47a7-aa29-a1589ebcbd18",
    "_uuid": "2957c4a7c4e6ed990b1406d5c88a9aa4c738b28f"
   },
   "outputs": [],
   "source": [
    "cols_with_missing = [col for col in X_train.columns \n",
    "                                 if X_train[col].isnull().any()]\n",
    "reduced_X_train = X_train.drop(cols_with_missing, axis=1)\n",
    "reduced_X_test  = X_test.drop(cols_with_missing, axis=1)\n",
    "print(\"Mean Absolute Error from dropping columns with Missing Values:\")\n",
    "print(score_dataset(reduced_X_train, reduced_X_test, y_train, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "b446d3e1-0a5c-4552-8718-e07ab5f4496a",
    "_uuid": "e78f8751db60278737a388a4b85b72bed4d3f45b"
   },
   "source": [
    "### Get Model Score from Imputation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "8cb756dc-9623-43b7-92c7-dfd87c70f450",
    "_uuid": "7f1030b598d08cd586a56d4ab33d1f99f6535784"
   },
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "my_imputer = SimpleImputer()\n",
    "imputed_X_train = my_imputer.fit_transform(X_train)\n",
    "imputed_X_test = my_imputer.transform(X_test)\n",
    "print(\"Mean Absolute Error from Imputation:\")\n",
    "print(score_dataset(imputed_X_train, imputed_X_test, y_train, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "2713edc7-a4cc-42ef-8619-bf775fa85481",
    "_uuid": "5540a5c3c2dd2fd0427820a4af57baf97729c462"
   },
   "source": [
    "### Get Score from Imputation with Extra Columns Showing What Was Imputed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "39ba8166-9b03-41cb-9403-d728b342d5e7",
    "_uuid": "914b9e57b99d7964013f007537c300fe57e0bf91"
   },
   "outputs": [],
   "source": [
    "imputed_X_train_plus = X_train.copy()\n",
    "imputed_X_test_plus = X_test.copy()\n",
    "\n",
    "cols_with_missing = (col for col in X_train.columns \n",
    "                                 if X_train[col].isnull().any())\n",
    "for col in cols_with_missing:\n",
    "    imputed_X_train_plus[col + '_was_missing'] = imputed_X_train_plus[col].isnull()\n",
    "    imputed_X_test_plus[col + '_was_missing'] = imputed_X_test_plus[col].isnull()\n",
    "\n",
    "# Imputation\n",
    "my_imputer = SimpleImputer()\n",
    "imputed_X_train_plus = my_imputer.fit_transform(imputed_X_train_plus)\n",
    "imputed_X_test_plus = my_imputer.transform(imputed_X_test_plus)\n",
    "\n",
    "print(\"Mean Absolute Error from Imputation while Track What Was Imputed:\")\n",
    "print(score_dataset(imputed_X_train_plus, imputed_X_test_plus, y_train, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "bc678d8b-f4c9-464f-8fdf-0d69afaefcaa",
    "_uuid": "2336d82df8f643c42ecb1354c034f214e87d7aa4"
   },
   "source": [
    "# Conclusion\n",
    "As is common, imputing missing values allowed us to improve our model compared to dropping those columns.  We got an additional boost by tracking what values had been imputed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "a8969353-a3af-4ac3-998b-b0fb884036cb",
    "_uuid": "ede373b4ec290324b175149afca77f2634a95277"
   },
   "source": [
    "# Your Turn\n",
    "1) Find some columns with missing values in your dataset.\n",
    "\n",
    "2) Use the Imputer class so you can impute missing values\n",
    "\n",
    "3) Add columns with missing values to your predictors. \n",
    "\n",
    "If you find the right columns, you may see an improvement in model scores. That said, the Iowa data doesn't have a lot of columns with missing values.  So, whether you see an improvement at this point depends on some other details of your model.\n",
    "\n",
    "Once you've added the Imputer, keep using those columns for future steps.  In the end, it will improve your model (and in most other datasets, it is a big improvement). \n",
    "\n",
    "# Keep Going\n",
    "Once you've added the Imputer and included columns with missing values, you are ready to [add categorical variables](https://www.kaggle.com/dansbecker/using-categorical-data-with-one-hot-encoding), which is non-numeric data representing categories (like the name of the neighborhood a house is in).\n",
    "\n",
    "---\n",
    "\n",
    "Part of the **[Learn Machine Learning](https://www.kaggle.com/learn/machine-learning)** track."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}<commit_msg>Revert due to merge<commit_after>{"cells":[{"metadata":{"_uuid":"8510ac1d600e8b4c1e27a34a84fb4cc703970d37","_cell_guid":"8c580981-fa5a-43a9-ae99-0251cda5b7a9"},"cell_type":"markdown","source":"*This tutorial is part Level 2 in the [Learn Machine Learning](https://www.kaggle.com/learn/machine-learning) curriculum. This tutorial picks up where Level 1 finished, so you will get the most out of it if you've done the exercise from Level 1.*\n\nIn this step, you will learn three approaches to dealing with missing values. You will then learn to compare the effectiveness of these approaches on any given dataset.* \n\n# Introduction\n\nThere are many ways data can end up with missing values. For example\n- A 2 bedroom house wouldn't include an answer for _How large is the third bedroom_\n- Someone being surveyed may choose not to share their income\n\nPython libraries represent missing numbers as **nan** which is short for \"not a number\".  You can detect which cells have missing values, and then count how many there are in each column with the command:\n```\nmissing_val_count_by_column = (data.isnull().sum())\nprint(missing_val_count_by_column[missing_val_count_by_column > 0\n```\n\nMost libraries (including scikit-learn) will give you an error if you try to build a model using data with missing values. So you'll need to choose one of the strategies below.\n\n---\n## Solutions\n\n\n## 1) A Simple Option: Drop Columns with Missing Values\nIf your data is in a DataFrame called `original_data`, you can drop columns with missing values. One way to do that is\n```\ndata_without_missing_values = original_data.dropna(axis=1)\n```\n\nIn many cases, you'll have both a training dataset and a test dataset.  You will want to drop the same columns in both DataFrames. In that case, you would write\n\n```\ncols_with_missing = [col for col in original_data.columns \n                                 if original_data[col].isnull().any()]\nredued_original_data = original_data.drop(cols_with_missing, axis=1)\nreduced_test_data = test_data.drop(cols_with_missing, axis=1)\n```\nIf those columns had useful information (in the places that were not missing), your model loses access to this information when the column is dropped. Also, if your test data has missing values in places where your training data did not, this will result in an error.  \n\nSo, it's somewhat usually not the best solution. However, it can be useful when most values in a column are missing.\n\n\n\n## 2) A Better Option: Imputation\nImputation fills in the missing value with some number. The imputed value won't be exactly right in most cases, but it usually gives more accurate models than dropping the column entirely.\n\nThis is done with\n```\nfrom sklearn.impute import SimpleImputer\nmy_imputer = SimpleImputer()\ndata_with_imputed_values = my_imputer.fit_transform(original_data)\n```\nThe default behavior fills in the mean value for imputation.  Statisticians have researched more complex strategies, but those complex strategies typically give no benefit once you plug the results into sophisticated machine learning models.\n\nOne (of many) nice things about Imputation is that it can be included in a scikit-learn Pipeline. Pipelines simplify model building, model validation and model deployment.\n\n## 3) An Extension To Imputation\nImputation is the standard approach, and it usually works well.  However, imputed values may by systematically above or below their actual values (which weren't collected in the dataset). Or rows with missing values may be unique in some other way. In that case, your model would make better predictions by considering which values were originally missing.  Here's how it might look:\n```\n# make copy to avoid changing original data (when Imputing)\nnew_data = original_data.copy()\n\n# make new columns indicating what will be imputed\ncols_with_missing = (col for col in new_data.columns \n                                 if new_data[col].isnull().any())\nfor col in cols_with_missing:\n    new_data[col + '_was_missing'] = new_data[col].isnull()\n\n# Imputation\nmy_imputer = SimpleImputer()\nnew_data = pd.DataFrame(my_imputer.fit_transform(new_data))\nnew_data.columns = original_data.columns\n```\n\nIn some cases this approach will meaningfully improve results. In other cases, it doesn't help at all.\n\n---\n# Example (Comparing All Solutions)\n\nWe will see am example predicting housing prices from the Melbourne Housing data.  To master missing value handling, fork this notebook and repeat the same steps with the Iowa Housing data.  Find information about both in the **Data** section of the header menu.\n\n\n### Basic Problem Set-up"},{"metadata":{"collapsed":true,"_uuid":"44b399828f0b07fe63abbdcdf74bbf3b22bb8067","_cell_guid":"a5e604e7-3c20-409b-ad29-5c2a4fe40738","trusted":true},"cell_type":"code","source":"import pandas as pd\n\n# Load data\nmelb_data = pd.read_csv('../input/melbourne-housing-snapshot/melb_data.csv')\n\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.model_selection import train_test_split\n\nmelb_target = melb_data.Price\nmelb_predictors = melb_data.drop(['Price'], axis=1)\n\n# For the sake of keeping the example simple, we'll use only numeric predictors. \nmelb_numeric_predictors = melb_predictors.select_dtypes(exclude=['object'])\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"de7bdb4f005022ea45742b5d25f47cba7a6d698d","_cell_guid":"88d1c1c9-91ba-4bef-aee5-f4a19a68e61c"},"cell_type":"markdown","source":"### Create Function to Measure Quality of An Approach\nWe divide our data into **training** and **test**. If the reason for this is unfamiliar, review [Welcome to Data Science](https://www.kaggle.com/dansbecker/welcome-to-data-science-1).\n\nWe've loaded a function `score_dataset(X_train, X_test, y_train, y_test)` to compare the quality of diffrent approaches to missing values. This function reports the out-of-sample MAE score from a RandomForest."},{"metadata":{"collapsed":true,"_uuid":"6088bfdac20ece9c040e83beb28ff169d17f0666","_cell_guid":"986dbfe4-9386-4a03-b2a4-9e99bf1b08f5","_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(melb_numeric_predictors, \n                                                    melb_target,\n                                                    train_size=0.7, \n                                                    test_size=0.3, \n                                                    random_state=0)\n\ndef score_dataset(X_train, X_test, y_train, y_test):\n    model = RandomForestRegressor()\n    model.fit(X_train, y_train)\n    preds = model.predict(X_test)\n    return mean_absolute_error(y_test, preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df0103056e52ffb3c500f5fc1437bd175a41adad","_cell_guid":"7184fc99-e266-4bd7-af0d-0da9c96887f0"},"cell_type":"markdown","source":"### Get Model Score from Dropping Columns with Missing Values"},{"metadata":{"_uuid":"2957c4a7c4e6ed990b1406d5c88a9aa4c738b28f","_cell_guid":"64ae7a0a-95aa-47a7-aa29-a1589ebcbd18","trusted":true},"cell_type":"code","source":"cols_with_missing = [col for col in X_train.columns \n                                 if X_train[col].isnull().any()]\nreduced_X_train = X_train.drop(cols_with_missing, axis=1)\nreduced_X_test  = X_test.drop(cols_with_missing, axis=1)\nprint(\"Mean Absolute Error from dropping columns with Missing Values:\")\nprint(score_dataset(reduced_X_train, reduced_X_test, y_train, y_test))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e78f8751db60278737a388a4b85b72bed4d3f45b","_cell_guid":"b446d3e1-0a5c-4552-8718-e07ab5f4496a"},"cell_type":"markdown","source":"### Get Model Score from Imputation"},{"metadata":{"_uuid":"7f1030b598d08cd586a56d4ab33d1f99f6535784","_cell_guid":"8cb756dc-9623-43b7-92c7-dfd87c70f450","trusted":true},"cell_type":"code","source":"from sklearn.impute import SimpleImputer\n\nmy_imputer = SimpleImputer()\nimputed_X_train = my_imputer.fit_transform(X_train)\nimputed_X_test = my_imputer.transform(X_test)\nprint(\"Mean Absolute Error from Imputation:\")\nprint(score_dataset(imputed_X_train, imputed_X_test, y_train, y_test))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5540a5c3c2dd2fd0427820a4af57baf97729c462","_cell_guid":"2713edc7-a4cc-42ef-8619-bf775fa85481"},"cell_type":"markdown","source":"### Get Score from Imputation with Extra Columns Showing What Was Imputed"},{"metadata":{"_uuid":"914b9e57b99d7964013f007537c300fe57e0bf91","_cell_guid":"39ba8166-9b03-41cb-9403-d728b342d5e7","trusted":true},"cell_type":"code","source":"imputed_X_train_plus = X_train.copy()\nimputed_X_test_plus = X_test.copy()\n\ncols_with_missing = (col for col in X_train.columns \n                                 if X_train[col].isnull().any())\nfor col in cols_with_missing:\n    imputed_X_train_plus[col + '_was_missing'] = imputed_X_train_plus[col].isnull()\n    imputed_X_test_plus[col + '_was_missing'] = imputed_X_test_plus[col].isnull()\n\n# Imputation\nmy_imputer = SimpleImputer()\nimputed_X_train_plus = my_imputer.fit_transform(imputed_X_train_plus)\nimputed_X_test_plus = my_imputer.transform(imputed_X_test_plus)\n\nprint(\"Mean Absolute Error from Imputation while Track What Was Imputed:\")\nprint(score_dataset(imputed_X_train_plus, imputed_X_test_plus, y_train, y_test))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2336d82df8f643c42ecb1354c034f214e87d7aa4","_cell_guid":"bc678d8b-f4c9-464f-8fdf-0d69afaefcaa"},"cell_type":"markdown","source":"# Conclusion\nAs is common, imputing missing values allowed us to improve our model compared to dropping those columns.  We got an additional boost by tracking what values had been imputed."},{"metadata":{"_uuid":"ede373b4ec290324b175149afca77f2634a95277","_cell_guid":"a8969353-a3af-4ac3-998b-b0fb884036cb"},"cell_type":"markdown","source":"# Your Turn\n1) Find some columns with missing values in your dataset.\n\n2) Use the Imputer class so you can impute missing values\n\n3) Add columns with missing values to your predictors. \n\nIf you find the right columns, you may see an improvement in model scores. That said, the Iowa data doesn't have a lot of columns with missing values.  So, whether you see an improvement at this point depends on some other details of your model.\n\nOnce you've added the Imputer, keep using those columns for future steps.  In the end, it will improve your model (and in most other datasets, it is a big improvement). \n\n# Keep Going\nOnce you've added the Imputer and included columns with missing values, you are ready to [add categorical variables](https://www.kaggle.com/dansbecker/using-categorical-data-with-one-hot-encoding), which is non-numeric data representing categories (like the name of the neighborhood a house is in).\n\n---\n\nPart of the **[Learn Machine Learning](https://www.kaggle.com/learn/machine-learning)** track."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}
{'fraction_non_alphanumeric': 0.12742850247375406, 'fraction_numerical': 0.04577450625477656, 'mean_word_length': 3.9953787422141853, 'pattern_counts': {'":': 187, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 8, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5707460', 'n_tokens_mistral': 8926, 'n_tokens_neox': 8163, 'n_words': 2214}