<jupyter_start><jupyter_text>Your Tasks:- Scrape the first 100 available search results- Generalize your code to allow searching for different locations/jobs- Pick out information about the URL, job title, and job location- Save the results to a file<jupyter_code>import requests
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>--- Part 1: Inspect- How do the URLs change when you navigate to the next results page?- How do the URLs change when you use a different location and/or job title search?- Which HTML elements contain the link, title, and location of each job? **Next Page**: The `start=` parameter gets added and incremented by the value of `10` for each additional page. This is because each results page displays 10 job results.E.g.:  **Different Location/Job Title**: The values for the query parameters `q` (for job title) and `l` (for location) change accordingly.<jupyter_code>page = requests.get('https://www.indeed.com/jobs?q=python&l=new+york')<jupyter_output><empty_output><jupyter_text>**HTML Elements**: A single job posting lives inside of a `div` element with the class name `result`. Inside there are other elements. You can find the specific info you're looking for here:- **Link**: In the `href` attribute of the `` Element that is a child of the title `` element- **Title**: The text of the link in the `` element which also contains the link URL mentioned above- **Location**: A `` element with the telling class name `location` --- Part 2: Scrape- Build the code to fetch the first 100 search results. This means you will need to automatically navigate to multiple results pages- Write functions that allow you to specify the job title, location, and amount of results as arguments<jupyter_code>page_2 = requests.get('https://www.indeed.com/jobs?q=python&l=new+york&start=20')<jupyter_output><empty_output><jupyter_text>Every 10 results means you're on a new page. Let's make that an argument to a function:<jupyter_code>def get_jobs(page=1):
    """Fetches the HTML from a search for Python jobs in New York on Indeed.com from a specified page."""
    base_url_indeed = 'https://www.indeed.com/jobs?q=python&l=new+york&start='
    results_start_num = page*10
    url = f'{base_url_indeed}{results_start_num}'
    page = requests.get(url)
    return page
get_jobs(3)
get_jobs(4)<jupyter_output><empty_output><jupyter_text>Great! Let's customize this function some more to allow for different search queries and search locations:<jupyter_code>def get_jobs(title, location, page=1):
    """Fetches the HTML from a search for Python jobs in New York on Indeed.com from a specified page."""
    loc = location.replace(' ', '+')  # for multi-part locations
    base_url_indeed = f'https://www.indeed.com/jobs?q={title}&l={loc}&start='
    results_start_num = page*10
    url = f'{base_url_indeed}{results_start_num}'
    page = requests.get(url)
    return page
get_jobs('python', 'new york', 3)<jupyter_output><empty_output><jupyter_text>With a generalized way of scraping the page done, you can move on to picking out the information you need by parsing the HTML. --- Part 3: Parse- Sieve through your HTML soup to pick out only the job title, link, and location- Format the results in a readable format (e.g. JSON)- Save the results to a file Let's start by getting access to all interesting search results for one page:<jupyter_code>site = get_jobs('python', 'new york')
soup = BeautifulSoup(site.content)
results = soup.find(id='resultsCol')
jobs = results.find_all('div', class_='result')<jupyter_output><empty_output><jupyter_text>**Job Titles** can be found like this:<jupyter_code>job_titles = [job.find('h2').find('a').text.strip() for job in jobs]
job_titles<jupyter_output><empty_output><jupyter_text>**Link URLs** need to be assembled, and can be found like this:<jupyter_code>base_url = 'https://www.indeed.com'
job_links = [base_url + job.find('h2').find('a')['href'] for job in jobs]
job_links<jupyter_output><empty_output><jupyter_text>**Locations** can be picked out of the soup by their class name:<jupyter_code>job_locations = [job.find(class_='location').text for job in jobs]
job_locations<jupyter_output><empty_output><jupyter_text>Let's assemble all this info into a function, so you can pick out the pieces and save them to a useful data structure:<jupyter_code>def parse_info(soup):
    """
    Parses HTML containing job postings and picks out job title, location, and link.
    
    args:
    soup (BeautifulSoup object): A parsed bs4.BeautifulSoup object of a search results page on indeed.com
    
    returns:
    job_list (list): A list of dictionaries containing the title, link, and location of each job posting
    """
    results = soup.find(id='resultsCol')
    jobs = results.find_all('div', class_='result')
    base_url = 'https://www.indeed.com'

    job_list = list()
    for job in jobs:
        title = job.find('h2').find('a').text.strip()
        link = base_url + job.find('h2').find('a')['href']
        location = job.find(class_='location').text
        job_list.append({'title': title, 'link': link, 'location': location})

    return job_list<jupyter_output><empty_output><jupyter_text>Let's give it a try:<jupyter_code>page = get_jobs('python', 'new_york')
soup = BeautifulSoup(page.content)
results = parse_info(soup)
results<jupyter_output><empty_output><jupyter_text>And let's add a final step of generalization:<jupyter_code>def get_job_listings(title, location, amount=100):
    results = list()
    for page in range(amount//10):
        site = get_jobs(title, location, page=page)
        soup = BeautifulSoup(site.content)
        page_results = parse_info(soup)
        results += page_results
    return results
r = get_job_listings('python', 'new york', 100)
len(r)
r<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11011699931176876, 'fraction_numerical': 0.0070543702684101855, 'mean_word_length': 5.011375387797311, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '462748', 'license': 'MIT', 'n_tokens_mistral': 1837, 'n_tokens_neox': 1765, 'n_words': 619, 'path': 'Web Scraping (Beautiful Soup, Scrapy, Selenium)/webScraping_Day57/Scrape-IndeedJobs/solution.ipynb', 'repo_name': 'pooja-gera/TheWireUsChallenge', 'size': '5812'}