<jupyter_start><jupyter_text>Web scraping> Using web scraping to get data from websites- toc: true - badges: true- comments: false- categories: [web scraping, python, BeautifulSoup, Selenium, jupyter]- author: <NAME> ![](ghtop_images/header2.png)  Overview ![](ghtop_images/ilya-pavlov-OqtafYT5kTw-unsplash.jpg)Photo by <NAME> on Unsplash  "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites." from https://en.wikipedia.org/wiki/Web_scraping> There are legal issues around web scraping, so be careful "The legality of web scraping varies across the world. In general, web scraping may be against the terms of use of some websites, but the enforceability of these terms is unclear."The first step is to go to a website and add `/robots.txt` onto the end of the web address to see what they allow. Most will disallow most areas.I am only doing this for research and learning purposes and the information is all available freely.  Indeed, BeautifulSoup and Pandas ![](ghtop_images/clem-onojeghuo-fY8Jr4iuPQM-unsplash.jpg)Photo by <NAME> on UnsplashIndeed has a wealth of job information across the UK and beyond for virtually every job. So whether you are looking for work or examining economic activity, it is a good resource. But how do you get the data from the website into a form you can work with?                By web scraping Indeed   Examine the websiteThe first step in web scraping is to examine the website. Here I’ll use Google Chrome, but the same capabilities are in most browsers. Let’s open indeed and do a search for jobs in an area. The output is below![](ghtop_images/indeed_url.webp) When searching for “electrician mate” in the location Swansea we get the url https://uk.indeed.com/jobs?q=electrician+mate&l=Swansea%2C+Swansea.Hence, in Python we can specify the url as follows, this allows us to change for job type and location later if we want:<jupyter_code>location="Swansea%2C+Swansea"
jobtype="electrician+mate"

URL="https://uk.indeed.com/jobs?q="+jobtype+"&l="+location
URL<jupyter_output><empty_output><jupyter_text>Next open developer tools on Chrome (Ctrl+Shift+I or More Tools-> developer tools) ![](ghtop_images/FindingPartsOnWebsite.png)Then click on the icon circled in the image below to examine the website. Placing the cursor over the website reveals the html structure. When this is done on one of the job blocks we can see that it is identified as:<jupyter_code><div class="job_seen_beacon">
OR
<a id="..........><jupyter_output><empty_output><jupyter_text>Examining other blocks shows this is repeated for them all. So we can get the job blocks!We’ll return to this later…  BeautifulSoupBeautiful soup (https://www.crummy.com/software/BeautifulSoup/bs4/doc/) is a Python library for pulling data out of HTML and XML files. Something similar can be done with pandas, but for this situation I’ll use the soup.<jupyter_code>#import the libraries
import requests
import bs4
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>To use: the libraries are loaded, then get get the page, then pass to Beautiful soup.Soup can be viewed. But it is still a mess so we’ll now use what we learned about the job tabs and extract these.<jupyter_code>#collapse-output
#Using beautiful soup-get page
page = requests.get(URL)
#get soup
soup = BeautifulSoup(page.text, "html.parser")
soup<jupyter_output><empty_output><jupyter_text>Using the ‘find_all’ function of soup we can find all these blocks by specifying the details of the block, i.e. ``The length of jobs_soup should be the number of job panels on the page, and each should be for a specific job. The output for the first is shown below (different search criteria).<jupyter_code>#collapse-output
jobs_soup=soup.find_all(name="div", attrs={"class":"job_seen_beacon"})

jobs_soup[0]<jupyter_output><empty_output><jupyter_text>We now need to pick out the details of the job from these blocks. To do this we go back to Chrome.![](ghtop_images/findJobtitle.png)Clicking on one of the job titles displays the job title is given as:`Data Analyst` There are a few ways we can extract this. What we want is a “span” with a feature called “title”. One way is to scroll through all “div” and find one with a title, using the except to help if it doesn’t have a title.<jupyter_code>div=jobs_soup[0]
for ab in div.find_all(name="span"):
            
   try:                            
      aJob=ab['title']   
          
   except:
      pass
        
aJob<jupyter_output><empty_output><jupyter_text>Each category we want to extract is slightly different so requires a slightly different approach. Although, the same procedure of using the developer tools in chrome and then modifying how the soup is searched is used.For the company name it is within `Company X…`So can be accessed from the string within the span with class= companyName .<jupyter_code>aaa=div.find_all(name="span", attrs={"class":"companyName"})
aaa[0].string<jupyter_output><empty_output><jupyter_text>Function to scrapeThis can be done for other details we might want from the other job details as below in a single function<jupyter_code>def extract_title(soup): 
    #initialse
    jobs = []
    company=[]
    pay=[]
    descr=[]
    
    #"job_seen_beacon" is the div for each job panel 
          # would be better to use <a> tag with 
    for div in soup.find_all(name="div", attrs={"class":"job_seen_beacon"}):

        #for the company name is in span tag with class=companyName
        aComp="-"
        for a in div.find_all(name="span", attrs={"class":"companyName"}):
            try:
                aComp=a.string
            except:
                print('exception comp')
                pass 
        company.append(a.string)
        
        aJob='-'
        for ab in div.find_all(name="span"):
            try:
                aJob=ab['title']              
            except: #Exception
                pass
        jobs.append(aJob)
        
        aPay="0"
        for abc in div.find_all(name="span", attrs={"class":"salary-snippet"}):
            try:
                aPay=abc.string
            except:
                print('exception pay')
                pass
        pay.append(aPay)

        
        aDescr=''
        abc = div.find_all(name='div',attrs={"class":"job-snippet"})
        for abcd in abc[0].find_all(name='li'):
            try:
                aDescr=aDescr+abcd.getText()+'-'
            except:
                print('exception decr')
                pass
        
        descr.append(aDescr)
        
    sa=soup.find_all(name='a',attrs={"target":"_blank"})
    urlss=[]
    
    strURLstart='https://www.indeed.co.uk'
    for aSect in sa:
        li=[]
        try:
            aCheck=aSect['id']
            li.append(aCheck)
            urlss.append(strURLstart+aSect['href'])
        except:
            pass
            
    return jobs, company, pay, descr, urlss<jupyter_output><empty_output><jupyter_text>But there may be more than the first page, so creating a function that can just take:1. Job details1. Location1. Maximum number of returned results. We get the following:<jupyter_code>## Cycle through pages and call inner function 'extract_title'
def extract_outer(jobtype,location,max_results):
    import requests
    import bs4
    from bs4 import BeautifulSoup
    import pandas as pd
    import numpy as np
    import copy as copy
    import time
    
    #Split job into two string with plus - i.e. "road sweeper"->"road + sweeper"
    jobtype=jobtype.replace(' ',' + ')
    
    #initialise values
    df=pd.DataFrame(columns=['Company','Job Title','Pay','Details','URL'])
    
    #Scroll through pages i.e. 0-9 in page1 --10-19 in page2 (in theory!)
    i=0
    for start in range(0, max_results, 10):#Add 10 on till a max
        
        #initialise values
        num = (len(df) + 1)
        jobs_, company_, pay_, descr_, urlss_ = [], [] , [], [], []
        num=str(start)
        
        #add a pause
        time.sleep(1)
        
        #get URL
        URL="https://uk.indeed.com/jobs?q="+jobtype+"&l="+location+"&start="+num#conducting a request of the stated URL above:
        page = requests.get(URL)#specifying a desired format of “page” using the html parser - this allows python to read the various components of the page, rather than treating it as one long string.
        soup = BeautifulSoup(page.text, "html.parser")#printing soup in a more structured tree format that makes for easier reading
        
        #call the inner function
        jobs_, company_, pay_, descr_, urlss_=extract_title(soup)
        
        # append the lists
        if i==0:          ## on 1st go assign df values
            df['Company']=company_
            df['Job Title']=jobs_
            df['Pay']=pay_
            df['Details']=descr_
            df['URL']=urlss_
        else:            ## on next ones concat df with a new df 
            df_=pd.DataFrame(columns=['Company','Job Title','Pay','Details','URL'])
            df_['Company']=company_
            df_['Job Title']=jobs_
            df_['Pay']=pay_
            df_['Details']=descr_
            df_['URL']=urlss_
            
            df=pd.concat(
                [df,df_ 
                
                ]
                        )
            del df_
        i=i+1
    
        df=df.reset_index(drop=True)#reset index- drop stops index becoming a column
    return df<jupyter_output><empty_output><jupyter_text>Which can be called to return a pandas DataFrame with:1. Company1. Job Title1. Pay1. Job Details1. URL for more detailsThe function is called like this:<jupyter_code>jobtitle='data analyst'
maxRes=50
locat="Swansea%2C+Swansea"
fname="indeed_"+jobtitle+'_'+locat+'_maxRes='+str(maxRes)+'.csv'
df=extract_outer(jobtitle,locat,maxRes)
df.head(10)<jupyter_output><empty_output><jupyter_text>Looking inside the individual pagesSo far the data frame just has details available on the main search page. The information is limited (see details tab below).More details can be found on the individual pages<jupyter_code>df.iloc[0].Details<jupyter_output><empty_output><jupyter_text>For each job the same webscraping procedure can be used to get information from the job page.Below is how to get the full description<jupyter_code>URL=df.URL[0]
page = requests.get(URL)#specifying a desired format of “page” using the html parser - this allows python to read the various components of the page, rather than treating it as one long string.
soup = BeautifulSoup(page.text, "html.parser")#printing soup in a more structured tree format that makes for easier reading
asoup=soup.find(name="div", attrs={"id":"jobDescriptionText"})
asoup2=asoup.text
asoup2<jupyter_output><empty_output><jupyter_text>Details of this could then be extracted to look for key words, salary etc. Using Natural Language Processing.   Indeed overviewAt this stage one might think “I can do this in my web browser“. Which is obviously true.But this data can now be processed further to find other things. Maybe we want to compare pay in different regions, look for where there are more jobs of a certain type, interrogate the jobs details for specific information.  Example using selenium ![](ghtop_images/Se.png)This uses the same methodology as above but instead uses Selenium which controls a web browser.This means links can be clicked, or other boxes can be okayed (e.g. accept cookie box).*Since this was done it appears the website has slightly changed as the code does not work.*<jupyter_code># Some imports

import requests
import bs4
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.common.keys import Keys
from selenium import webdriver
import time
import os<jupyter_output><empty_output><jupyter_text>Get web driver for chrome from https://chromedriver.chromium.org/downloads    Then use it to open the website.<jupyter_code>#hide
URLweb = "https://matmatch.com/advanced-search?categories=steel&tags=form:bar"
browser_loc='C:/Users/44781/pyproj/chromedriver.exe'
#import chrome webdriver

URL = URLweb
browser = webdriver.Chrome(browser_loc)
browser.get(URL)<jupyter_output><empty_output><jupyter_text>Chrome is now controlled from this notebook![](ghtop_images/Sel1.png)<jupyter_code>#get rid of cookie q
elem2=browser.find_element_by_tag_name('button')
browser.execute_script("arguments[0].click();", elem2)<jupyter_output><empty_output><jupyter_text>The functions 1. innerPagethis scrolls through each search pageand calls OneSearchPage 2. OneSearchPageOne of the search pagesFor each link calls gothrough_A_Link  3. gothrough_A_LinkThis looks on a materials pageThis then calls getTableData function to extract table data  4. getTableDatagetTableData<jupyter_code>def innerPage(browser):
    import time
    f=0
    ii=0
    #first time through
    dfAll=OneSearchPage(browser)
    foutname='steel_bar_'+str(ii)
    dfAll.to_csv(foutname)
    print('ii ',ii)
    while f==0:
        if ii<200:
            
            #find Next page button and click it
            elemNext=browser.find_elements_by_xpath("//button[contains(@data-test-item,'next-page')]")
            try:
                elemNext[0].click()#this should fail if last page
            except:
                f=1
                print('failed',elemNext[0].text)
                break
            time.sleep(1)
            #next timeS through
            dfAll=OneSearchPage(browser)
            
            ii=ii+1
            foutname='steel_bar_'+str(ii)
            dfAll.to_csv(foutname)
            print('ii ',ii)
            
        else:
            break
            
        
        
    return dfAll
def OneSearchPage(browser,*dfnew):
    #find each element with <a> and contain materials
    elem2=browser.find_elements_by_xpath("//a[contains(@href,'/materials/')]")

    if 'df' in locals():
        del df
    if 'df_' in locals():
        del df_


    i=0
    for eel in elem2:
        if i<200:
            print(eel.text)
            time.sleep(1)

            if len(dfnew)>0:##if we give dfnew as input
                
                df_=gothrough_A_Link(eel,browser)
                if i==0:
                    dfOne=dfnew[0].append(df_)
                else:
                    dfOne=dfOne.append(df_)
            else:#if don't give input dfnew
                if i==0:#on 1st call create dfOne
                    dfOne=gothrough_A_Link(eel,browser)
                else:#on 2nd call append
                    df_=gothrough_A_Link(eel,browser)
                    dfOne=dfOne.append(df_)

            i=i+1
            time.sleep(1)
            print('i ',i)
        else:
            break
    return dfOne
def gothrough_A_Link(pageElement,browser):
    import time
    from selenium.webdriver.common.keys import Keys 
    #give like gothrough_A_Link(elem2[ii],browser)
    time.sleep(.6)
#     print(pageElement)
    #open element- but in new tab
    try:
        pageElement.click()#send_keys(Keys.RETURN)
    except:
        time.sleep(0.9)
        time.sleep(0.9)
    #switch to new window

    #get current window
    p = browser.current_window_handle
    #get windows
    chwd=browser.window_handles
    for w in chwd:
    #switch focus to child window
        if(w!=p):
            browser.switch_to.window(w)
            break
    time.sleep(0.9)
    
    
    #get current url
    url_current=browser.current_url
    
    #get table details
    
    df=getTableData(url_current)
    
    
    #close new window
    time.sleep(0.9)
    browser.close()
    browser.switch_to.window(browser.window_handles[0])

    return df

############################################################################################

############################################################################################


    
def getTableData(URL):
    import requests
    import bs4
    from bs4 import BeautifulSoup
    import pandas as pd
    
    
    res = requests.get(URL)
    soup = BeautifulSoup(res.content,'lxml')

    # get alloy name
    titla=soup.find('h1').text

#     print(titla)
    #get alloy description
    descripta=soup.find('div',{'class':'common__SubSectionWrapper-sc-3st4qy-0 lbsGSv'}).getText()
    descripta=descripta.replace('Description','Description: ')
    descripta=descripta.split('More technical information')[0]#sometimes occurs


    #get table
    table = soup.find_all('table')

    tdAll=[]
    nomsAll=[]
    accepted_strings={'Density','Elastic modulus','Elongation','Tensile strength','Yield strength'}
    for ta in table:
    #     print('-------------------\n',ta.text[0:30])
        tatr=ta.find_all('tr')
        textdegC="°C"
        for trr in tatr:
    #         print(trr.text[0:30])
            tatrtd=trr.find_all('td')
            try:
                textaStartRow=tatrtd[0].text
                
                if textaStartRow=="Density":
                    nomsAll.append(textaStartRow)
                    
                    if 'g/cm' in tatrtd[1].p.text:
                        tdAll.append(tatrtd[1].p.text)
                    elif 'g/cm' in tatrtd[2].p.text:
                        tdAll.append(tatrtd[2].p.text)
                        
                elif textaStartRow=="Elastic modulus":
                    nomsAll.append(textaStartRow)
                    if "textdegC" in tatrtd[1].text:#if has deg C
                        tdAll.append(tatrtd[2].p.text)
                    elif not tatrtd[1].text:
                        tdAll.append(tatrtd[2].p.text)
                    else:
                        tdAll.append(tatrtd[1].p.text)
                    
                elif textaStartRow=="Elongation":
                    nomsAll.append(textaStartRow)
                    
                    if "textdegC" in tatrtd[1].text:#if has deg C
                        tdAll.append(tatrtd[2].p.text)
                    elif not tatrtd[1].text:
                        tdAll.append(tatrtd[2].p.text)
                    else:
                        tdAll.append(tatrtd[1].p.text)
                    
                elif textaStartRow=="Charpy impact energy, V-notch":
                    nomsAll.append(textaStartRow)
                    
                    if "textdegC" in tatrtd[1].text:#if has deg C
                        tdAll.append(tatrtd[2].p.text)
                    elif not tatrtd[1].text:
                        tdAll.append(tatrtd[2].p.text)
                    else:
                        tdAll.append(tatrtd[1].p.text)
                        
                elif textaStartRow=="Tensile strength":
                    nomsAll.append(textaStartRow)
                   
                    if "textdegC" in tatrtd[1].text:#if has deg C
                        tdAll.append(tatrtd[2].p.text)
                    elif not tatrtd[1].text:
                        tdAll.append(tatrtd[2].p.text)
                    else:
                        tdAll.append(tatrtd[1].p.text)
                        
                elif textaStartRow=="Yield strength":
                    nomsAll.append(textaStartRow)
                    
                    if "textdegC" in tatrtd[1].text:#if has deg C
                        tdAll.append(tatrtd[2].p.text)
                    elif not tatrtd[1].text:
                        tdAll.append(tatrtd[2].p.text)
                    else:
                        tdAll.append(tatrtd[1].p.text)
        
            except:
                
                pass
            
     #Chemical composition
    tableComp=soup.find('p',text='Chemical properties')#know it is below a paragraph with title Chemical Properties 
    try:
        taatta=tableComp.next_sibling
        trComp=taatta.find_all('tr')
        compPC=[]
        compID=[]
        for trr in trComp:
            try:
                tdd=trr.find_all('td')
                nomsAll.append(tdd[0].text)
                tdAll.append(tdd[1].p.text)
            except:
                pass
    except:
        pass
    
     #if we want to
    dff=pd.DataFrame(tdAll).transpose()
    print(nomsAll,tdAll)
    dff.columns=nomsAll
    
    dff['Description']=descripta
    dff.insert(0, 'Alloy', titla)
    
    return dff
dfOUT=innerPage(browser)

#and do some cleaning of data
df = cleanDFdatFunc(dfOUT)<jupyter_output><empty_output><jupyter_text>Look at the data<jupyter_code>#hide
df = pd.read_csv("C:/Users/44781/pyproj/JobScraping/steel_sheet_Comb_Clean")
df.drop(columns=['Unnamed: 0'],inplace=True)
df.head()
import matplotlib.pyplot as plt
coluse='Tensile strength'
coluse2='Elongation'

df_=df[df[coluse]>0]
df_=df_[df_[coluse2]>0]
plt.scatter(x=df_[coluse2],y=df_[coluse])

plt.xlabel(coluse2 +'%')
plt.ylabel(coluse +' MPa')
plt.grid('on')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10369906560061651, 'fraction_numerical': 0.007802716501300453, 'mean_word_length': 2.4449975111996016, 'pattern_counts': {'":': 14, '<': 78, '<?xml version=': 0, '>': 83, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '158304', 'license': 'Apache-2.0', 'n_tokens_mistral': 6355, 'n_tokens_neox': 6096, 'n_words': 1791, 'path': '_notebooks/2021-11-18-WebScraping.ipynb', 'repo_name': 'ThomasHSimm/THS', 'size': '20780'}