<jupyter_start><jupyter_text>Python script used to scrape vehicle complaints information from [www.carcomplaints.com](http://www.carcomplaints.com) website.<jupyter_code>carcomplaints.com has a very easy-to-understand URL scheme.  It is basically in this format: http://www.carcomplaints.com/make/model/year/system/system.shtml<jupyter_output><empty_output><jupyter_text>from collections import OrderedDictfrom bs4 import BeautifulSoupimport urllib.request as requestimport reurl_Honda = 'http://www.carcomplaints.com/Honda/'html_Honda = request.urlopen(url_Honda)soup_Honda = BeautifulSoup(html_Honda)<jupyter_code>### Honda Models Overall Complaint Counts (http://www.carcomplaints.com/Honda/)<jupyter_output><empty_output><jupyter_text>Looking at the page source of http://www.carcomplaints.com/Honda/, it appears the data we want is embedded in the 'ul' element, with class='column bar', and id=c1 or c2 or c3<jupyter_code>ul = soup_Honda.find_all('ul', class_='column bar',id=re.compile('c*'))
ul<jupyter_output><empty_output><jupyter_text>As you can see from above, the data I want (the model name and  of complaints are in the &lt;li&gt; tags).  I will make a Python dict of this data:<jupyter_code>honda_model_counts_dict = {}
num_column_data = len(ul)  # The data is divided up in arbitrary number of columns per HTML page source
for i in range(num_column_data):  # For each column of data...
    for row in ul[i].find_all('li'):
        honda_model_counts_dict[row.a.get_text()] = int(row.span.get_text().replace(",",""))
honda_model_counts_dict<jupyter_output><empty_output><jupyter_text>Acura Models Overall Complaint Counts Using the same procedure I did for Honda, I will get the Acura models and their respective complaint counts.<jupyter_code>from collections import OrderedDict
from bs4 import BeautifulSoup
from IPython.display import HTML
import urllib.request as request
import re

url_Acura = 'http://www.carcomplaints.com/Acura/'
html_Acura = request.urlopen(url_Acura)

soup_Acura = BeautifulSoup(html_Acura)
ul = soup_Acura.find_all('ul', class_='column bar',id=re.compile('c*'))

acura_model_counts_dict = {}
num_column_data = len(ul)  # The data is divided up in arbitrary number of columns
for i in range(num_column_data):  # For each column of data...
    for row in ul[i].find_all('li'):
        acura_model_counts_dict[row.a.get_text()] = int(row.span.get_text().replace(",",""))

OD_Acura = OrderedDict(sorted(acura_model_counts_dict.items(), key=lambda t: t[1], reverse=True))

s_header = '<table border="1"><tr><th>Model Name</th><th># of Complaints</th></tr>'

s_data = ''
for key in OD_Acura.keys():
    s_data = s_data + '<tr><td align="center">' + key + '</td>' + '<td align="center">' + str(OD_Acura[key]) + '</td></tr>'

s_footer = "</table>"

h = HTML(s_header+s_data+s_footer);h<jupyter_output><empty_output><jupyter_text>Honda version executed all in one cell<jupyter_code>from collections import OrderedDict
from bs4 import BeautifulSoup
from IPython.display import HTML
import urllib.request as request
import re

url_Honda = 'http://www.carcomplaints.com/Honda/'
html_Honda = request.urlopen(url_Honda)

soup_Honda = BeautifulSoup(html_Honda)
ul = soup_Honda.find_all('ul', class_='column bar',id=re.compile('c*'))

honda_model_counts_dict = {}
num_column_data = len(ul)  # The data is divided up in arbitrary number of columns per HTML page source
for i in range(num_column_data):  # For each column of data...
    for row in ul[i].find_all('li'):
        honda_model_counts_dict[row.a.get_text()] = int(row.span.get_text().replace(",",""))
        
OD_Honda = OrderedDict(sorted(honda_model_counts_dict.items(), key=lambda t: t[1], reverse=True))

s_header = '<table border="1"><tr><th>Model Name</th><th># of Complaints</th></tr>'

s_data = ''
for key in OD_Honda.keys():
    s_data = s_data + '<tr><td align="center">' + key + '</td>' + '<td align="center">' + str(OD_Honda[key]) + '</td></tr>'

s_footer = "</table>"

h = HTML(s_header+s_data+s_footer);h<jupyter_output><empty_output><jupyter_text>OK, now that I've shown how to obtain the number of complaints for each model in a step-by-step manner, it is time to make a function out of this so we can re-use all this code<jupyter_code>from bs4 import BeautifulSoup
import urllib.request as request
import re

def getCountsByModel(make):
    """Method that returns the number of complaints for each model based on vehicle make
    Applicable make values are: 'Honda','Acura','Ford','GM',etc
    Method returns a dictionary where the key is the model, value is the qty of complaints"""
    
    url = 'http://www.carcomplaints.com/'
    url_make = url+make+'/'
    html_make = request.urlopen(url_make)
    
    soup = BeautifulSoup(html_make)
    ul = soup.find_all('ul', class_='column bar',id=re.compile('c*'))
    
    make_model_counts_dict = OrderedDict()
    num_column_data = len(ul)  # The data is divided up in arbitrary number of columns per HTML page source
    for i in range(num_column_data):  # For each column of data...
        for row in ul[i].find_all('li'):
            make_model_counts_dict[row.a.get_text()] = int(row.span.get_text().replace(",",""))
            
    return make_model_counts_dict
getCountsByModel('Honda')<jupyter_output><empty_output><jupyter_text>I also made a function to get all available makes at carcomplaints.com<jupyter_code>from bs4 import BeautifulSoup
import urllib.request as request
import re

def getMakes():
    """Function to get all the makes available at carcomplaints.com"""
    
    url = 'http://www.carcomplaints.com/'
    html = request.urlopen(url)
    
    soup = BeautifulSoup(html)
    sections = soup.find_all('section', id=re.compile('makes'))
    
    make_list = []
    for section in range(len(sections)):
        for li in sections[section].find_all('li'):
            make_list.append(li.get_text())
    
    return make_list
getMakes()<jupyter_output><empty_output><jupyter_text>Function to get available model years and complaint qty from a give model<jupyter_code>from bs4 import BeautifulSoup
import urllib.request as request
import re

def getYearCounts(make, model):
    """Function that returns a Python dict that contains model years and their complaint qty"""
    
    url = 'http://www.carcomplaints.com/'+make+'/'+model+'/'
    html = request.urlopen(url)

    soup = BeautifulSoup(html)
    li = soup.find_all('li', id=re.compile('bar*'))

    year_counts_dict = {}
    for item in li:
        year_counts_dict[int(item.find('span',class_='label').get_text())]=int(item.find('span',class_='count').get_text().replace(",",""))
    
    return year_counts_dict
getYearCounts('Honda','Accord')<jupyter_output><empty_output><jupyter_text>Function to get Top Systems by Qty<jupyter_code>from bs4 import BeautifulSoup
from collections import OrderedDict
import urllib.request as request
import re

def getTopSystemsQty(make, model, year):
    """Function that returns an OrderedDict containing system problems and their complaint qty"""
    
    url = 'http://www.carcomplaints.com/'+make+'/'+model+'/'+str(year)+'/'
    html = request.urlopen(url)

    soup = BeautifulSoup(html)
    li = soup.find_all('li', id=re.compile('bar*'))
    
    problem_counts_dict = OrderedDict()  # We want to maintain insertion order
    for item in li:
        try:
            problem_counts_dict[item.a['href'][:-1]]=int(item.span.get_text().replace(",",""))
        except:
            pass
        
    return problem_counts_dict
getTopSystemsQty(year=2012, make='Honda', model='CR-V')
getTopSystemsQty(year=2001, make='Nissan', model='Altima')<jupyter_output><empty_output><jupyter_text>Function to get number of corresponding qty of NHTSA complaints for each system category<jupyter_code>from bs4 import BeautifulSoup
from collections import OrderedDict
import urllib.request as request
import re


def getNhtsaSystemsQty(make, model, year):
    """Function that returns an OrderedDict containing qty of NHTSA complaints by system"""
    
    url = 'http://www.carcomplaints.com/'+make+'/'+model+'/'+str(year)+'/'
    html = request.urlopen(url)

    soup = BeautifulSoup(html)

    nhtsa = soup.find_all('em', class_='nhtsa')

    nhtsa_counts = []
    for item in nhtsa:
        try:
            # There are 3 string tokens separated by whitespace, i want the 3rd token which is the qty
            nhtsa_counts.append(int(item.span.get_text().split()[2]))
        except:
            # Unfortunately, some only have 2 tokens
            nhtsa_counts.append(int(item.span.get_text().split()[1]))

    systems = soup.find_all('li', id=re.compile('bar*'))

    systems_list = []
    for item in systems:
        systems_list.append(item.a['href'][:-1]) # Remove the ending forward slash

    nhtsa_systems_counts = list(zip(systems_list,nhtsa_counts))
    
    nhtsa_systems_qty_dict = OrderedDict()
    for item in nhtsa_systems_counts:
        nhtsa_systems_qty_dict[item[0]]=item[1]
    
        
    return nhtsa_systems_qty_dict
getNhtsaSystemsQty('Honda','Accord','2001')<jupyter_output><empty_output><jupyter_text>Function to get qty of complaints by sub-system<jupyter_code>from bs4 import BeautifulSoup
from collections import OrderedDict
import urllib.request as request
import re

make = 'Honda'
model = 'Civic'
year = 2001
system = 'transmission'

def getSubSystemsQty(make, model, year, system):
    """Function that will return an OrderedDict of # of complaints by sub-system"""
    
    url = 'http://www.carcomplaints.com/'+make+'/'+model+'/'+str(year)+'/'+system+'/'
    html = request.urlopen(url)
    soup = BeautifulSoup(html)

    li = soup.find_all('li', id=re.compile('bar*'))

    subsystem_counts_dict = OrderedDict()  # We want to maintain insertion order
    for item in li:
        subsystem_counts_dict[item.a['href'].split(".")[0]]=int(item.span.get_text().replace(",",""))
        
    return subsystem_counts_dict
getSubSystemsQty('Honda','Accord','2000','transmission')<jupyter_output><empty_output><jupyter_text>Function to get the review text for a specific system failure<jupyter_code>from bs4 import BeautifulSoup
import urllib.request as request
import re

def getReviews(make, model, year, system, subsystem):
    """Function that returns a list of customer reviews"""
    
    url = 'http://www.carcomplaints.com/'+make+'/'+model+'/'+str(year)+'/'+system+'/'+subsystem+'.shtml'
    html = request.urlopen(url)
    soup = BeautifulSoup(html)

    reviews = soup.find_all('div', itemprop="reviewBody")
    
    complaints = []
    for complaint in reviews:
        complaints.append(complaint.p.get_text())
        
    return complaints
for complaint in getReviews('Honda','Civic','2001','transmission','pops_out_of_gear'):
    print(complaint)
    print('*'*120)<jupyter_output>This is the second time that the speed sensor has gone out in the last nine months.
************************************************************************************************************************
Syncros are bad, pops out of gear and doesn't get back in.. throw out bearing was bad too
************************************************************************************************************************
The transmission of my 2001 civic began popping out of 4th gear on occasion. The car was my wifes and I didn't normally drive it. When I did and if popped out during acceleration she told me it had been doing it for a while. We decided to get a new car since we saw a trend since we just put in new struts and had the anti-lock break system replaced for about $1000 each . The Honda dealership test drove it and said it needed a new transmission. This had a large impact on the trade in value. I need to keep an eye on this site and when I see if our 2010 civic might have issues and [...]
{'fraction_non_alphanumeric': 0.13988978380669775, 'fraction_numerical': 0.005680373039423484, 'mean_word_length': 4.617142857142857, 'pattern_counts': {'":': 0, '<': 80, '<?xml version=': 0, '>': 80, 'https://': 0, 'lorem ipsum': 0, 'www.': 15, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '26312', 'license': 'MIT', 'n_tokens_mistral': 3889, 'n_tokens_neox': 3767, 'n_words': 1033, 'path': 'web_scraping/WebScraping_CarComplaintsDotCom.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '11795'}