<jupyter_start><jupyter_text>HW 2: Scrape Book Reviews Choose one of your favorite book at goodreads.com (e.g. for the book "The Midninght Library", the URL is: https://www.goodreads.com/book/show/52578297-the-midnight-library?from_choice=true) - Q1. Write a function to scrape all **reviews on the first page**, including,     - **reviewer's name** (see (1) in Figure)    - **rating** (see (2) in Figure)    - **date** (see (3) in Figure)    - **review content** (see (4) in Figure. For each review text, need to get the **complete text**, i.e., need to expand the `more` button. Only text is needed, pictures are not needed. (Hint: take a close look at the content of the html file. Do you need Selenium?)    - **likes** (see (5) in Figure).     - If a field, e.g. rating, is missing, use `None` to indicate it. - `Function Input`: book page URL- `Function Output`: save all reviews as a DataFrame of columns (`reviewer, rating, date, review, like`). E.g., for the given URL, you can get 30 reviews.- 10 points: 1 point for each element, 5 points for overall logic.- **Note**: GoodReads occasionaly blocks request. You may get an error that is not due to your codes. You may try to run a couple of times.         ![alt text](GoodReads.png "GoodReads")<jupyter_code>import requests
from bs4 import BeautifulSoup  
import pandas as pd
import re
 

def getReviews(page_url):
    
    # enter your codes here
    #list of column headers for scraped data
    colHeaders = ['Reviewer Name','Rating(out of 5)','Date','Review','Likes']
    #class names corrosponding to the column headers
    classNames = ["user","staticStar p10","reviewDate createdAt right","reviewText stacked","likesCount"]
    reviews = pd.DataFrame(columns = colHeaders)
    row = []

    soup = BeautifulSoup(requests.get(page_url).content, 'html.parser')
    scrapedData = soup.find_all(class_="friendReviews elementListBrown")

    for scrapedReview in scrapedData:
        for scrapedCol in range(len(classNames)):
            try:
                cell = scrapedReview.find_all(class_= classNames[scrapedCol])
                #check if data is not empty
                if cell:
                    if scrapedCol==1:#processing star count
                        #count number of stars, by counting number of times staticStar p10 appears in the scraped column
                        row.append(str(len(cell)) + " stars")
                    elif scrapedCol==3:#getting all review text, including hidden text
                        #each review has 2 spans, one contaning just the display data and other the entire review
                        #we select the span with the entire review to get all the text including hidden text
                        #since all the data is already present on the page, we can directly get it via bs4,
                        #we dont need to use selenium in this case to click (..more) to expand review text, as it will only slow down the code by adding unesseccary steps
                        #if in case the text was not in source and called via an Ajax req, or js script, then
                        #we would have to use selenium to click on the link and get the rest of the text in the page source
                        spans = scrapedReview.find_all('span',attrs = {'id' : re.compile(r'freeText')})
                        cell[0] = spans[len(spans)- 1]
                        row.append(cell[0].text.strip())
                    else:
                        row.append(cell[0].text.strip())
                else:
                    #append none if no data is found for the cell
                    row.append(None)
            except AttributeError:
                if len(row) < 5:
                    row.append(None)
                continue
        reviews.loc[len(reviews)] = row
        row = []
    #return first 30 reviews
    return reviews

#note: if we still want to use selenium ,we can simply initialize a webdriver, and use 
#webDriver.find_element_by_link_text("...more").click()
#and select span with attribute Style : Display:inline
#it will have the same result but will add extra steps, slowing execution time

# enter your url
page_url = 'https://www.goodreads.com/book/show/52578297-the-midnight-library?from_choice=true'
reviews=getReviews(page_url)
reviews<jupyter_output><empty_output><jupyter_text>- Q2 (Bonus). Modify the function you defined in Q1 to scrape **reviews on all the pages** for your url. Since a book may have multiple pages, use the **next** button at the end of each page (shown in the picture) to navigate to the next page. Continue scraping all the pages until the last page. `Please don't hardcode the pages in the URL because the number of pages varies by book`. (3 points. If URL is hardcoded, -2).![alt text](GoodReads_bonus.png "GoodReads_bonus")<jupyter_code>from selenium import webdriver
import time
import requests
from bs4 import BeautifulSoup  
import pandas as pd
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import ElementClickInterceptedException
from selenium.webdriver.common.keys import Keys

#the website uses an ajax request to get the next page data on the current page, hence we need to use selenium here
#to click on the next button and get the next page data.


#this function gets reviews on the current page
def getReviews_modified(driver,reviews):
    sourc = driver.page_source
    soup = BeautifulSoup(sourc, 'html.parser')
#---rest of the function is same as question 1----------------------------------------------#
    scrapedData = soup.find_all(class_="friendReviews elementListBrown")
    classNames = ["user","staticStar p10","reviewDate createdAt right","reviewText stacked","likesCount"]
    row = []
    for scrapedReview in scrapedData:
        for scrapedCol in range(len(classNames)):
            try:
                cell = scrapedReview.find_all(class_= classNames[scrapedCol])
                if cell:
                    if scrapedCol==1:
                        row.append(str(len(cell)) + " stars")
                    elif scrapedCol==3:
                        spans = scrapedReview.find_all('span',attrs = {'id' : re.compile(r'freeText')})
                        cell[0] = spans[len(spans)- 1]
                        row.append(cell[0].text.strip())
                    else:
                        row.append(cell[0].text.strip())
                else:
                    row.append(None)
            except AttributeError:
                if len(row) < 5:
                    row.append(None)
        reviews.loc[len(reviews)] = row
        row = []
    return reviews



#this function gets reviews from all pages
def getReviews_2(page_url):
    colHeaders = ['Reviewer Name','Rating(out of 5)','Date','Review','Likes']
    reviews = pd.DataFrame(columns = colHeaders)
    #initializing selenium webdriver
    driver = webdriver.Firefox()
    driver.maximize_window()
    driver.get(page_url)
    time.sleep(5)
    source = driver.page_source
    s = BeautifulSoup(source, 'html.parser')
    reviews = getReviews_modified(driver,reviews)
    print('scraping')
    pg_count = 2
    tryCount = 0
    while (not s.find(class_="next_page disabled")) or (pg_count<=10):
    #goodreads only shows first 10 pages of reviews hence the second condition, in case next_page disabled is not found
        try:
            element = driver.find_element_by_class_name("next_page")
            if element:
                driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + Keys.HOME)#to avoid other element obscuring next button scrolling to top of page
                time.sleep(3)
                element.click()
                time.sleep(4)
                pg_count = pg_count + 1
                source = driver.page_source
                s = BeautifulSoup(source, 'html.parser')
                reviews = getReviews_modified(driver,reviews)
                tryCount = 0
                print('.')
            else:
                return reviews
        except ElementClickInterceptedException:
            if tryCount==1:
                print('page refresh failed, might be network error')
                return reviews
            print('next button obscured,refreshing page')
            try:
                driver.get(page_url)
                time.sleep(5)
                if pg_count>2:
                    driver.find_element_by_link_text(str(pg_count-1)).click()
                time.sleep(3)
                tryCount = 1
                source = driver.page_source
                s = BeautifulSoup(source, 'html.parser')
                continue
            except:
                print('could not refresh page,try running again')
                return reviews
        except NoSuchElementException:
            if pg_count <= 10:
                try:
                    if pg_count<2:
                        driver.find_element_by_link_text(str(pg_count-1)).click()
                    else:
                        driver.find_element_by_class_name("next_page").click()
                    time.sleep(3)
                    source = driver.page_source
                    s = BeautifulSoup(source, 'html.parser')
                    continue
                except:
                    print('page load error,check network and try running again')
                    return reviews
    driver.close()
    return reviews
        
# enter your url
page_url = 'https://www.goodreads.com/book/show/52578297-the-midnight-library?from_choice=true'

reviews=getReviews_2(page_url)
reviews<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0912401165210154, 'fraction_numerical': 0.009779442363712027, 'mean_word_length': 2.194749086075108, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 10, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '421164', 'license': 'MIT', 'n_tokens_mistral': 2684, 'n_tokens_neox': 2505, 'n_words': 889, 'path': 'HW2/Homework_2.ipynb', 'repo_name': 'ipsavani/BIA-660-WebMining', 'size': '9612'}