<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Text Processing with Spacy\n",
    "    \n",
    "You're a consultant for [DelFalco's Italian Restaurant](https://defalcosdeli.com/index.html).\n",
    "The owner asked you to identify whether there are any foods on their menu that diners find disappointing. \n",
    "\n",
    "<img src=\"https://i.imgur.com/8DZunAQ.jpg\" alt=\"Meatball Sub\" width=\"250\"/>\n",
    "\n",
    "Before getting started, run the following cell to set up code checking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Set up code checking\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.nlp.ex1 import *\n",
    "print('Setup Complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The business owner suggested you use diner reviews from the Yelp website to determine which dishes people liked and disliked. You pulled the data from Yelp. Before you get to analysis, run the code cell below for a quick look at the data you have to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in the data from JSON file\n",
    "data = pd.read_json('../input/nlp-course/restaurant.json')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The owner also gave you this list of menu items and common alternate spellings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "menu = [\"Cheese Steak\", \"Cheesesteak\", \"Steak and Cheese\", \"Italian Combo\", \"Tiramisu\", \"Cannoli\",\n",
    "        \"Chicken Salad\", \"Chicken Spinach Salad\", \"Meatball\", \"Pizza\", \"Pizzas\", \"Spaghetti\",\n",
    "        \"Bruchetta\", \"Eggplant\", \"Italian Beef\", \"Purista\", \"Pasta\", \"Calzones\",  \"Calzone\",\n",
    "        \"Italian Sausage\", \"Chicken Cutlet\", \"Chicken Parm\", \"Chicken Parmesan\", \"Gnocchi\",\n",
    "        \"Chicken Pesto\", \"Turkey Sandwich\", \"Turkey Breast\", \"Ziti\", \"Portobello\", \"Reuben\",\n",
    "        \"Mozzarella Caprese\",  \"Corned Beef\", \"Garlic Bread\", \"Pastrami\", \"Roast Beef\",\n",
    "        \"Tuna Salad\", \"Lasagna\", \"Artichoke Salad\", \"Fettuccini Alfredo\", \"Chicken Parmigiana\",\n",
    "        \"Grilled Veggie\", \"Grilled Veggies\", \"Grilled Vegetable\", \"Mac and Cheese\", \"Macaroni\",  \n",
    "         \"Prosciutto\", \"Salami\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Plan Your Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the data from Yelp and the list of menu items, do you have any ideas for how you could find which menu items have disappointed diners?\n",
    "\n",
    "Think about your answer. Then run the cell below to see one approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Find items in one review\n",
    "\n",
    "You'll pursue this plan of calculating average scores of the reviews mentioning each menu item.\n",
    "\n",
    "As a first step, you'll write code to extract the foods mentioned in a single review.\n",
    "\n",
    "Since menu items are multiple tokens long, you'll use `PhraseMatcher` which can match series of tokens.\n",
    "\n",
    "Fill in the `____` values below to get a list of items matching a single menu item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "index_of_review_to_test_on = 14\n",
    "text_to_test_on = data.text.iloc[index_of_review_to_test_on]\n",
    "\n",
    "# Load the SpaCy model\n",
    "nlp = spacy.blank('en')\n",
    "\n",
    "# Create the tokenized version of text_to_test_on\n",
    "review_doc = ____\n",
    "\n",
    "# Create the PhraseMatcher object. The tokenizer is the first argument. Use attr = 'LOWER' to make consistent capitalization\n",
    "matcher = PhraseMatcher(nlp.vocab, attr='LOWER')\n",
    "\n",
    "# Create a list of tokens for each item in the menu\n",
    "menu_tokens_list = [____ for item in menu]\n",
    "\n",
    "# Add the item patterns to the matcher. \n",
    "# Look at https://spacy.io/api/phrasematcher#add in the docs for help with this step\n",
    "# Then uncomment the lines below \n",
    "\n",
    "# \n",
    "#matcher.add(\"MENU\",            # Just a name for the set of rules we're matching to\n",
    "#            None,              # Special actions to take on matched words\n",
    "#            ____  \n",
    "#           )\n",
    "\n",
    "# Find matches in the review_doc\n",
    "# matches = ____\n",
    "\n",
    "# Uncomment to check your work\n",
    "#q_2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_2.hint()\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After implementing the above cell, uncomment the following cell to print the matches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for match in matches:\n",
    "#    print(f\"Token number {match[1]}: {review_doc[match[1]:match[2]]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "import spacy\n",
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "# Load the SpaCy model\n",
    "tokenizer = spacy.blank('en')\n",
    "\n",
    "index_of_review_to_test_on = 14\n",
    "text_to_test_on = data.text.iloc[index_of_review_to_test_on]\n",
    "\n",
    "# Create the tokenized review_doc\n",
    "review_doc = tokenizer(text_to_test_on)\n",
    "\n",
    "# Create the PhraseMatcher object. The tokenizer is the first argument.\n",
    "# Reviews don't have consistent capitalization. Perform case-insensitive matching by adding argument attr='LOWER'\n",
    "matcher = PhraseMatcher(tokenizer.vocab, attr='LOWER')\n",
    "\n",
    "# Create a list of docs for each item in the menu\n",
    "menu_tokens_list = [tokenizer(item) for item in menu]\n",
    "\n",
    "# Add the item patterns to the matcher\n",
    "matcher.add(\"MENU\",            # Just a name for the set of rules we're matching to\n",
    "            None,\n",
    "            *menu_tokens_list  # Add the patterns to match to. In this case the menu_tokens_list\n",
    "           )\n",
    "\n",
    "# Find matches in the review_doc\n",
    "matches = matcher(review_doc)\n",
    "\n",
    "# Uncomment when checking code is complete\n",
    "q_2.assert_check_passed()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Matching on the whole dataset\n",
    "\n",
    "Now run this matcher over the whole dataset and collect ratings for each menu item. Each review has a rating, `review.stars`. For each item that appears in the review text (`review.text`), append the review's rating to a list of ratings for that item. The lists are kept in a dictionary `item_ratings`.\n",
    "\n",
    "To get the matched phrases, you can reference the `PhraseMatcher` documentation for the structure of each match object:\n",
    "\n",
    ">A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end]`. The `match_id` is the ID of the added match pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "# item_ratings is a dictionary of lists. If a key doesn't exist in item_ratings,\n",
    "# the key is added with an empty list as the value.\n",
    "item_ratings = defaultdict(list)\n",
    "\n",
    "for idx, review in data.iterrows():\n",
    "    doc = ____\n",
    "    # Using the matcher from the previous exercise\n",
    "    matches = ____\n",
    "    \n",
    "    # Create a set of the items found in the review text\n",
    "    found_items = ____\n",
    "    \n",
    "    # Update item_ratings with rating for each item in found_items\n",
    "    # Transform the item strings to lowercase to make it case insensitive\n",
    "    ____\n",
    "\n",
    "q_3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_3.hint()\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_3.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "\n",
    "from collections import defaultdict\n",
    "\n",
    "item_ratings = defaultdict(list)\n",
    "\n",
    "for idx, review in data.iterrows():\n",
    "    doc = tokenizer(review.text)\n",
    "    matches = matcher(doc)\n",
    "\n",
    "    found_items = set([doc[match[1]:match[2]] for match in matches])\n",
    "    \n",
    "    for item in found_items:\n",
    "        item_ratings[str(item).lower()].append(review.stars)\n",
    "        \n",
    "q_3.assert_check_passed()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: What's the worst reviewed item?\n",
    "\n",
    "Using these item ratings, find the menu item with the worst average rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the mean ratings for each menu item as a dictionary\n",
    "mean_ratings = ____\n",
    "\n",
    "# Find the worst item, and write it as a string in worst_text. This can be multiple lines of code if you want.\n",
    "worst_item = ____\n",
    "\n",
    "q_4.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_4.hint()\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_4.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After implementing the above cell, uncomment and run this to print \n",
    "# out the worst item, along with its average rating. \n",
    "\n",
    "#print(worst_item)\n",
    "#print(mean_ratings[item])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "\n",
    "mean_ratings = {item: sum(ratings)/len(ratings) for item, ratings in item_ratings.items()}\n",
    "worst_item = sorted(mean_ratings, key=mean_ratings.get)[0]\n",
    "    \n",
    "q_4.assert_check_passed()"
   ]
  },
  {<commit_msg>fixing a small bug<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Text Processing with Spacy\n",
    "    \n",
    "You're a consultant for [DelFalco's Italian Restaurant](https://defalcosdeli.com/index.html).\n",
    "The owner asked you to identify whether there are any foods on their menu that diners find disappointing. \n",
    "\n",
    "<img src=\"https://i.imgur.com/8DZunAQ.jpg\" alt=\"Meatball Sub\" width=\"250\"/>\n",
    "\n",
    "Before getting started, run the following cell to set up code checking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Set up code checking\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.nlp.ex1 import *\n",
    "print('Setup Complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The business owner suggested you use diner reviews from the Yelp website to determine which dishes people liked and disliked. You pulled the data from Yelp. Before you get to analysis, run the code cell below for a quick look at the data you have to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in the data from JSON file\n",
    "data = pd.read_json('../input/nlp-course/restaurant.json')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The owner also gave you this list of menu items and common alternate spellings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "menu = [\"Cheese Steak\", \"Cheesesteak\", \"Steak and Cheese\", \"Italian Combo\", \"Tiramisu\", \"Cannoli\",\n",
    "        \"Chicken Salad\", \"Chicken Spinach Salad\", \"Meatball\", \"Pizza\", \"Pizzas\", \"Spaghetti\",\n",
    "        \"Bruchetta\", \"Eggplant\", \"Italian Beef\", \"Purista\", \"Pasta\", \"Calzones\",  \"Calzone\",\n",
    "        \"Italian Sausage\", \"Chicken Cutlet\", \"Chicken Parm\", \"Chicken Parmesan\", \"Gnocchi\",\n",
    "        \"Chicken Pesto\", \"Turkey Sandwich\", \"Turkey Breast\", \"Ziti\", \"Portobello\", \"Reuben\",\n",
    "        \"Mozzarella Caprese\",  \"Corned Beef\", \"Garlic Bread\", \"Pastrami\", \"Roast Beef\",\n",
    "        \"Tuna Salad\", \"Lasagna\", \"Artichoke Salad\", \"Fettuccini Alfredo\", \"Chicken Parmigiana\",\n",
    "        \"Grilled Veggie\", \"Grilled Veggies\", \"Grilled Vegetable\", \"Mac and Cheese\", \"Macaroni\",  \n",
    "         \"Prosciutto\", \"Salami\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Plan Your Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the data from Yelp and the list of menu items, do you have any ideas for how you could find which menu items have disappointed diners?\n",
    "\n",
    "Think about your answer. Then run the cell below to see one approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check your answer (Run this code cell to receive credit!)\n",
    "q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Find items in one review\n",
    "\n",
    "You'll pursue this plan of calculating average scores of the reviews mentioning each menu item.\n",
    "\n",
    "As a first step, you'll write code to extract the foods mentioned in a single review.\n",
    "\n",
    "Since menu items are multiple tokens long, you'll use `PhraseMatcher` which can match series of tokens.\n",
    "\n",
    "Fill in the `____` values below to get a list of items matching a single menu item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "index_of_review_to_test_on = 14\n",
    "text_to_test_on = data.text.iloc[index_of_review_to_test_on]\n",
    "\n",
    "# Load the SpaCy model\n",
    "nlp = spacy.blank('en')\n",
    "\n",
    "# Create the tokenized version of text_to_test_on\n",
    "review_doc = ____\n",
    "\n",
    "# Create the PhraseMatcher object. The tokenizer is the first argument. Use attr = 'LOWER' to make consistent capitalization\n",
    "matcher = PhraseMatcher(nlp.vocab, attr='LOWER')\n",
    "\n",
    "# Create a list of tokens for each item in the menu\n",
    "menu_tokens_list = [____ for item in menu]\n",
    "\n",
    "# Add the item patterns to the matcher. \n",
    "# Look at https://spacy.io/api/phrasematcher#add in the docs for help with this step\n",
    "# Then uncomment the lines below \n",
    "\n",
    "# \n",
    "#matcher.add(\"MENU\",            # Just a name for the set of rules we're matching to\n",
    "#            None,              # Special actions to take on matched words\n",
    "#            ____  \n",
    "#           )\n",
    "\n",
    "# Find matches in the review_doc\n",
    "# matches = ____\n",
    "\n",
    "# Uncomment to check your work\n",
    "#q_2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_2.hint()\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After implementing the above cell, uncomment the following cell to print the matches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for match in matches:\n",
    "#    print(f\"Token number {match[1]}: {review_doc[match[1]:match[2]]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "import spacy\n",
    "from spacy.matcher import PhraseMatcher\n",
    "\n",
    "# Load the SpaCy model\n",
    "tokenizer = spacy.blank('en')\n",
    "\n",
    "index_of_review_to_test_on = 14\n",
    "text_to_test_on = data.text.iloc[index_of_review_to_test_on]\n",
    "\n",
    "# Create the tokenized review_doc\n",
    "review_doc = tokenizer(text_to_test_on)\n",
    "\n",
    "# Create the PhraseMatcher object. The tokenizer is the first argument.\n",
    "# Reviews don't have consistent capitalization. Perform case-insensitive matching by adding argument attr='LOWER'\n",
    "matcher = PhraseMatcher(tokenizer.vocab, attr='LOWER')\n",
    "\n",
    "# Create a list of docs for each item in the menu\n",
    "menu_tokens_list = [tokenizer(item) for item in menu]\n",
    "\n",
    "# Add the item patterns to the matcher\n",
    "matcher.add(\"MENU\",            # Just a name for the set of rules we're matching to\n",
    "            None,\n",
    "            *menu_tokens_list  # Add the patterns to match to. In this case the menu_tokens_list\n",
    "           )\n",
    "\n",
    "# Find matches in the review_doc\n",
    "matches = matcher(review_doc)\n",
    "\n",
    "# Uncomment when checking code is complete\n",
    "q_2.assert_check_passed()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Matching on the whole dataset\n",
    "\n",
    "Now run this matcher over the whole dataset and collect ratings for each menu item. Each review has a rating, `review.stars`. For each item that appears in the review text (`review.text`), append the review's rating to a list of ratings for that item. The lists are kept in a dictionary `item_ratings`.\n",
    "\n",
    "To get the matched phrases, you can reference the `PhraseMatcher` documentation for the structure of each match object:\n",
    "\n",
    ">A list of `(match_id, start, end)` tuples, describing the matches. A match tuple describes a span `doc[start:end]`. The `match_id` is the ID of the added match pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "# item_ratings is a dictionary of lists. If a key doesn't exist in item_ratings,\n",
    "# the key is added with an empty list as the value.\n",
    "item_ratings = defaultdict(list)\n",
    "\n",
    "for idx, review in data.iterrows():\n",
    "    doc = ____\n",
    "    # Using the matcher from the previous exercise\n",
    "    matches = ____\n",
    "    \n",
    "    # Create a set of the items found in the review text\n",
    "    found_items = ____\n",
    "    \n",
    "    # Update item_ratings with rating for each item in found_items\n",
    "    # Transform the item strings to lowercase to make it case insensitive\n",
    "    ____\n",
    "\n",
    "q_3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_3.hint()\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_3.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "\n",
    "from collections import defaultdict\n",
    "\n",
    "item_ratings = defaultdict(list)\n",
    "\n",
    "for idx, review in data.iterrows():\n",
    "    doc = tokenizer(review.text)\n",
    "    matches = matcher(doc)\n",
    "\n",
    "    found_items = set([doc[match[1]:match[2]] for match in matches])\n",
    "    \n",
    "    for item in found_items:\n",
    "        item_ratings[str(item).lower()].append(review.stars)\n",
    "        \n",
    "q_3.assert_check_passed()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: What's the worst reviewed item?\n",
    "\n",
    "Using these item ratings, find the menu item with the worst average rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the mean ratings for each menu item as a dictionary\n",
    "mean_ratings = ____\n",
    "\n",
    "# Find the worst item, and write it as a string in worst_text. This can be multiple lines of code if you want.\n",
    "worst_item = ____\n",
    "\n",
    "q_4.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_4.hint()\n",
    "#_COMMENT_IF(PROD)_\n",
    "q_4.solution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After implementing the above cell, uncomment and run this to print \n",
    "# out the worst item, along with its average rating. \n",
    "\n",
    "#print(worst_item)\n",
    "#print(mean_ratings[worst_item])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%%RM_IF(PROD)%%\n",
    "\n",
    "mean_ratings = {item: sum(ratings)/len(ratings) for item, ratings in item_ratings.items()}\n",
    "worst_item = sorted(mean_ratings, key=mean_ratings.get)[0]\n",
    "    \n",
    "q_4.assert_check_passed()"
   ]
  },
  {
{'fraction_non_alphanumeric': 0.17825618945102262, 'fraction_numerical': 0.0027556512378902045, 'mean_word_length': 2.6098849860118123, 'pattern_counts': {'":': 206, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1332614', 'n_tokens_mistral': 8190, 'n_tokens_neox': 8003, 'n_words': 2271}