<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSE 6040 Computing for Data Analysis, Fall 2015\n",
    "## Homework 1: Collecting Yelp Data Using Python\n",
    "## Due Sep 17, 2015 (Friday), 11:55  pm EDT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Submission details**: Submit a single zipped file, using the file name **“HW1-{YOUR_LAST_NAME}-{YOUR_FIRST_NAME}.zip”**. The zip file should contain this Ipython Notebook and two folders, named **“Part1”** and **“Part2”**, corresponding to the results from the two parts of this homework.\n",
    "\n",
    "If you have collaborated with other students, write down their names in the text box on the T-square submission site. Each student must write his/her own code and answers.\n",
    "\n",
    "In this homework, we will first collect Yelp data through APIs (Part 1). Due to the limitation in these APIs, we will scrape a few pages on Yelp to collect more information that is unavailable through its APIs (Part 2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1 [30 points]: Collecting Yelp Data through Yelp API\n",
    "\n",
    "For this part, you will use the Yelp API to find local restaurants information. The goal is to find 5 highly-rated restaurants in Atlanta with most reviews (largest numbers of reviews) on Yelp.\n",
    "\n",
    "### Preparation:\n",
    "\n",
    "\n",
    "1. Go to http://www.yelp.com/developers and create an account. You can use your existing Yelp account or create a new account by providing your name, email address, and zip code.\n",
    "2. Go to http://www.yelp.com/developers/manage_api_keys to generate your app key/secret and a token, by providing a website URL (can be anything, for example a dummy URL or the course page http://cse6040.gatech.edu/fa15/) and describing the purpose to use the APIs (e.g. for homework). Copy the **“Consumer Key”**, **“Consumer Secret”**, **“Token”**, and **“Token Secret”** for your own records, and we use them to fetch Yelp data in this homework.\n",
    "3. Go to http://www.yelp.com/developers/documentation, learn how to build the URLs in order to use Yelp Search API and Business API.\n",
    "4. In this homework, you will learn how to install a package in python. We will use the tool “pip”, which should already be installed with Anaconda. If not, Follow this link: https://pip.pypa.io/en/latest/installing.html, to install “pip”.\n",
    "5. For any package you want to install in python, type **“pip install {package_name}”** in the command prompt. For this part, you have to install the **oauth2** package.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instructions:\n",
    "\n",
    "Below we provide a function **yelp_req**, which you use to make requests to the Yelp API, and the return of this function a JSON object or error messages, including the information returned from Yelp API.\n",
    "\n",
    "For example, when url is 'http://api.yelp.com/v2/search?term=food&location=San+Francisco', yelp_req(url) will return a JSON object from the Search API.\n",
    "\n",
    "To use the function, first put the values of CONSUMER_KEY, CONSUMER_SECRET, TOKEN, and TOKEN_SECRET, using what you get from step 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import urllib2\n",
    "import json\n",
    "import oauth2\n",
    "\n",
    "# Please assign following values with the credentials found in your Yelp account \n",
    "CONSUMER_KEY = ''\n",
    "CONSUMER_SECRET = ''\n",
    "TOKEN = ''\n",
    "TOKEN_SECRET = ''\n",
    "\n",
    "\n",
    "def yelp_req(url):\n",
    "    \"\"\" Pass in a url that follows the format of Yelp API,\n",
    "        and this function will return either a JSON object or error messages.\n",
    "    \"\"\"\n",
    "    \n",
    "    oauth_request = oauth2.Request('GET', url, {})\n",
    "    oauth_request.update(\n",
    "        {\n",
    "            'oauth_nonce': oauth2.generate_nonce(),\n",
    "            'oauth_timestamp': oauth2.generate_timestamp(),\n",
    "            'oauth_token': TOKEN,\n",
    "            'oauth_consumer_key': CONSUMER_KEY\n",
    "        }\n",
    "    )\n",
    "    consumer = oauth2.Consumer(CONSUMER_KEY, CONSUMER_SECRET)\n",
    "    token = oauth2.Token(TOKEN, TOKEN_SECRET)\n",
    "    oauth_request.sign_request(oauth2.SignatureMethod_HMAC_SHA1(), consumer, token)\n",
    "    signed_url = oauth_request.to_url()\n",
    "\n",
    "    conn = urllib2.urlopen(signed_url, None)\n",
    "    try:\n",
    "        response = json.loads(conn.read())\n",
    "    finally:\n",
    "        conn.close()\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 1.1\n",
    "\n",
    "Your task is to find the 40 highest rated restaurants in Atlanta using the Search API. Use **“restaurants”** as the search term, and **“Atlanta, GA”** as the location parameter. After forming the URL, you need to feed it to the **yelp_req** function in the starter code to get the API response.\n",
    "\n",
    "Hints:\n",
    "1. Look at the parameters **limit**, **offset**, and **sort** in the Yelp’s API documentation to build the URL;\n",
    "2. Percent-encode the parameters using **urllib.urlencode**\n",
    "3. Many additional parameters are needed to be appended to the URL you formed for authentication’s purpose. Try printing the full URL in the HTTP request for yourself and see what parameters are included."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a. (10 points)** Save the body of the HTTP response (a JSON string) containing the restaurants ranked 1~20 in rating into a file **“first20.json”**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b. (10 points)** Save the body of the HTTP response (a JSON string) containing the restaurants ranked 21~40 in rating into a file **“next20.json”**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1.2\n",
    "\n",
    "**(5 points)** For each of the 40 highest rated restaurants you collected, get the number of reviews it has received. Create a text file named **“40restaurants.txt”**, and write in this file the restaurant names and the numbers of reviews, one line for each restaurant, higher ratings first, comma-delimited. \n",
    "\n",
    "For example:\n",
    "```\n",
    "Aviva by Kameel,138\n",
    "Purnima,43\n",
    "......\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1.3\n",
    "\n",
    "**(5 points)** From the 40 restaurants you collected, get the 5 restaurants with most reviews.\n",
    "Create a text file named **\"\"40restaurants_top_review_count.txt\"\"**, and write in this file the 5 restaurant\n",
    "names with most reviews (in descending order of their numbers of reviews) as\n",
    "well as the number of reviews, one line for each restaurant, comma-delimited. \n",
    "\n",
    "For example:\n",
    "```\n",
    "Antico Pizza,1622\n",
    "Fox Bros. Bar-B-Q,1168\n",
    "......\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Deliverables:\n",
    "\n",
    "1. Please remove your credential information (key/secret, tokens)\n",
    "2. In this part, 4 text files should be created (programmatically). Put all of them in the folder **\"Part1\"**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2 [30 points]: Scraping Yelp Pages to Collect More Data\n",
    "\n",
    "In the last part, we collected the 40 highest rated restaurants in Atlanta. What if we’d like to know more? In this part, we will collect the 100 highest rated restaurants in Atlanta by extracting information directly from the search result pages on Yelp.\n",
    "\n",
    "### Preparation:\n",
    "1. Install the BeautifulSoup package using the command **“pip install beautifulsoup4”**.\n",
    "2. Open the link: http://www.yelp.com/search?find_desc=restaurants&find_loc=Atlanta%2C+GA&sortby=rating&start=0\n",
    "\n",
    "This is the first page of search results of Atlanta restaurants on Yelp, sorted in descending order of their ratings. Browse this page to get familiar with its structure and available information, and inspect the relevant elements that render the search results.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 2.1\n",
    "\n",
    "Get the 100 highest rated restaurants in Atlanta from the search results (page 1 ~ page 10). You can figure out the URLs for the 2~10 pages of search results from the buttons on the first page or from the pattern in the above URL.\n",
    "\n",
    "Below, we show an example that reads a page into a string and calls the **preprocess_yelp_page** function to preprocess the string before proceeding to BeautifulSoup. Feel free to modify these code or write your own code, but do preprocess the page content for every web page you read. Otherwise, there might be issues when you try to find the HTML tag containing relevant information.\n",
    "\n",
    "You can consider either downloading the 10 web pages once and saving them into files for debugging your BeautifulSoup code, or downloading the web pages and analyzing them on-the-fly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "def preprocess_yelp_page(content):\n",
    "    ''' Remove extra spaces between HTML tags. '''\n",
    "    content = ''.join([line.strip() for line in content.split('\\n')])\n",
    "    return content\n",
    "\n",
    "# Example code to illustrate the use of preprocess_yelp_page\n",
    "url = 'http://www.yelp.com/search?find_desc=restaurants&find_loc=Atlanta%2C+GA&sortby=rating&start=0'\n",
    "content = urllib2.urlopen(url).read()\n",
    "content = preprocess_yelp_page(content) # Now *content* is a string containing the first page of search results, ready for processing with BeautifulSoup\n",
    "soup = BeautifulSoup(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a. (10 points)** Create a text file named **“10restaurants.txt”**, and write in this file the 10 restaurant names on the first result page, one line for each restaurant, in the original order in the search results (higher ratings first).\n",
    "\n",
    "Note that a search result page may contain advertised results on top of the actual search result. Please do a sanity check that confirms the number of restaurants in the submission file is 10, and figure out how to identify the advertised results and remove them from the list of 10 highest rated restaurants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },<commit_msg>change from urllib2 to requests<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSE 6040 Computing for Data Analysis, Fall 2015\n",
    "## Homework 1: Collecting Yelp Data Using Python\n",
    "## Due Sep 17, 2015 (Friday), 11:55  pm EDT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Submission details**: Submit a single zipped file, using the file name **“HW1-{YOUR_LAST_NAME}-{YOUR_FIRST_NAME}.zip”**. The zip file should contain this Ipython Notebook and two folders, named **“Part1”** and **“Part2”**, corresponding to the results from the two parts of this homework.\n",
    "\n",
    "If you have collaborated with other students, write down their names in the text box on the T-square submission site. Each student must write his/her own code and answers.\n",
    "\n",
    "In this homework, we will first collect Yelp data through APIs (Part 1). Due to the limitation in these APIs, we will scrape a few pages on Yelp to collect more information that is unavailable through its APIs (Part 2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1 [30 points]: Collecting Yelp Data through Yelp API\n",
    "\n",
    "For this part, you will use the Yelp API to find local restaurants information. The goal is to find 5 highly-rated restaurants in Atlanta with most reviews (largest numbers of reviews) on Yelp.\n",
    "\n",
    "### Preparation:\n",
    "\n",
    "\n",
    "1. Go to http://www.yelp.com/developers and create an account. You can use your existing Yelp account or create a new account by providing your name, email address, and zip code.\n",
    "2. Go to http://www.yelp.com/developers/manage_api_keys to generate your app key/secret and a token, by providing a website URL (can be anything, for example a dummy URL or the course page http://cse6040.gatech.edu/fa15/) and describing the purpose to use the APIs (e.g. for homework). Copy the **“Consumer Key”**, **“Consumer Secret”**, **“Token”**, and **“Token Secret”** for your own records, and we use them to fetch Yelp data in this homework.\n",
    "3. Go to http://www.yelp.com/developers/documentation, learn how to build the URLs in order to use Yelp Search API and Business API.\n",
    "4. In this homework, you will learn how to install a package in python. We will use the tool “pip”, which should already be installed with Anaconda. If not, Follow this link: https://pip.pypa.io/en/latest/installing.html, to install “pip”.\n",
    "5. For any package you want to install in python, type **“pip install {package_name}”** in the command prompt. For this part, you have to install the **oauth2** package.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instructions:\n",
    "\n",
    "Below we provide a function **yelp_req**, which you use to make requests to the Yelp API, and the return of this function a JSON object or error messages, including the information returned from Yelp API.\n",
    "\n",
    "For example, when url is 'http://api.yelp.com/v2/search?term=food&location=San+Francisco', yelp_req(url) will return a JSON object from the Search API.\n",
    "\n",
    "To use the function, first put the values of CONSUMER_KEY, CONSUMER_SECRET, TOKEN, and TOKEN_SECRET, using what you get from step 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import urllib2\n",
    "import json\n",
    "import oauth2\n",
    "\n",
    "# Please assign following values with the credentials found in your Yelp account \n",
    "CONSUMER_KEY = ''\n",
    "CONSUMER_SECRET = ''\n",
    "TOKEN = ''\n",
    "TOKEN_SECRET = ''\n",
    "\n",
    "\n",
    "def yelp_req(url):\n",
    "    \"\"\" Pass in a url that follows the format of Yelp API,\n",
    "        and this function will return either a JSON object or error messages.\n",
    "    \"\"\"\n",
    "    \n",
    "    oauth_request = oauth2.Request('GET', url, {})\n",
    "    oauth_request.update(\n",
    "        {\n",
    "            'oauth_nonce': oauth2.generate_nonce(),\n",
    "            'oauth_timestamp': oauth2.generate_timestamp(),\n",
    "            'oauth_token': TOKEN,\n",
    "            'oauth_consumer_key': CONSUMER_KEY\n",
    "        }\n",
    "    )\n",
    "    consumer = oauth2.Consumer(CONSUMER_KEY, CONSUMER_SECRET)\n",
    "    token = oauth2.Token(TOKEN, TOKEN_SECRET)\n",
    "    oauth_request.sign_request(oauth2.SignatureMethod_HMAC_SHA1(), consumer, token)\n",
    "    signed_url = oauth_request.to_url()\n",
    "\n",
    "    conn = urllib2.urlopen(signed_url, None)\n",
    "    try:\n",
    "        response = json.loads(conn.read())\n",
    "    finally:\n",
    "        conn.close()\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 1.1\n",
    "\n",
    "Your task is to find the 40 highest rated restaurants in Atlanta using the Search API. Use **“restaurants”** as the search term, and **“Atlanta, GA”** as the location parameter. After forming the URL, you need to feed it to the **yelp_req** function in the starter code to get the API response.\n",
    "\n",
    "Hints:\n",
    "1. Look at the parameters **limit**, **offset**, and **sort** in the Yelp’s API documentation to build the URL;\n",
    "2. Percent-encode the parameters using **urllib.urlencode**\n",
    "3. Many additional parameters are needed to be appended to the URL you formed for authentication’s purpose. Try printing the full URL in the HTTP request for yourself and see what parameters are included."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a. (10 points)** Save the body of the HTTP response (a JSON string) containing the restaurants ranked 1~20 in rating into a file **“first20.json”**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b. (10 points)** Save the body of the HTTP response (a JSON string) containing the restaurants ranked 21~40 in rating into a file **“next20.json”**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1.2\n",
    "\n",
    "**(5 points)** For each of the 40 highest rated restaurants you collected, get the number of reviews it has received. Create a text file named **“40restaurants.txt”**, and write in this file the restaurant names and the numbers of reviews, one line for each restaurant, higher ratings first, comma-delimited. \n",
    "\n",
    "For example:\n",
    "```\n",
    "Aviva by Kameel,138\n",
    "Purnima,43\n",
    "......\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1.3\n",
    "\n",
    "**(5 points)** From the 40 restaurants you collected, get the 5 restaurants with most reviews.\n",
    "Create a text file named **\"\"40restaurants_top_review_count.txt\"\"**, and write in this file the 5 restaurant\n",
    "names with most reviews (in descending order of their numbers of reviews) as\n",
    "well as the number of reviews, one line for each restaurant, comma-delimited. \n",
    "\n",
    "For example:\n",
    "```\n",
    "Antico Pizza,1622\n",
    "Fox Bros. Bar-B-Q,1168\n",
    "......\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Deliverables:\n",
    "\n",
    "1. Please remove your credential information (key/secret, tokens)\n",
    "2. In this part, 4 text files should be created (programmatically). Put all of them in the folder **\"Part1\"**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2 [30 points]: Scraping Yelp Pages to Collect More Data\n",
    "\n",
    "In the last part, we collected the 40 highest rated restaurants in Atlanta. What if we’d like to know more? In this part, we will collect the 100 highest rated restaurants in Atlanta by extracting information directly from the search result pages on Yelp.\n",
    "\n",
    "### Preparation:\n",
    "1. Install the BeautifulSoup package using the command **“pip install beautifulsoup4”**.\n",
    "2. Open the link: http://www.yelp.com/search?find_desc=restaurants&find_loc=Atlanta%2C+GA&sortby=rating&start=0\n",
    "\n",
    "This is the first page of search results of Atlanta restaurants on Yelp, sorted in descending order of their ratings. Browse this page to get familiar with its structure and available information, and inspect the relevant elements that render the search results.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 2.1\n",
    "\n",
    "Get the 100 highest rated restaurants in Atlanta from the search results (page 1 ~ page 10). You can figure out the URLs for the 2~10 pages of search results from the buttons on the first page or from the pattern in the above URL.\n",
    "\n",
    "Below, we show an example that reads a page into a string and calls the **preprocess_yelp_page** function to preprocess the string before proceeding to BeautifulSoup. Feel free to modify these code or write your own code, but do preprocess the page content for every web page you read. Otherwise, there might be issues when you try to find the HTML tag containing relevant information.\n",
    "\n",
    "You can consider either downloading the 10 web pages once and saving them into files for debugging your BeautifulSoup code, or downloading the web pages and analyzing them on-the-fly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "def preprocess_yelp_page(content):\n",
    "    ''' Remove extra spaces between HTML tags. '''\n",
    "    content = ''.join([line.strip() for line in content.split('\\n')])\n",
    "    return content\n",
    "\n",
    "# Example code to illustrate the use of preprocess_yelp_page\n",
    "url = 'http://www.yelp.com/search?find_desc=restaurants&find_loc=Atlanta%2C+GA&sortby=rating&start=0'\n",
    "content = requests.get(url).text\n",
    "content = preprocess_yelp_page(content) # Now *content* is a string containing the first page of search results, ready for processing with BeautifulSoup\n",
    "soup = BeautifulSoup(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a. (10 points)** Create a text file named **“10restaurants.txt”**, and write in this file the 10 restaurant names on the first result page, one line for each restaurant, in the original order in the search results (higher ratings first).\n",
    "\n",
    "Note that a search result page may contain advertised results on top of the actual search result. Please do a sanity check that confirms the number of restaurants in the submission file is 10, and figure out how to identify the advertised results and remove them from the list of 10 highest rated restaurants."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
{'fraction_non_alphanumeric': 0.14329614817933067, 'fraction_numerical': 0.012544727425805093, 'mean_word_length': 3.036017669045192, 'pattern_counts': {'":': 166, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5225439', 'n_tokens_mistral': 7805, 'n_tokens_neox': 7376, 'n_words': 2772}