<jupyter_start><jupyter_text>1. Collect course link texts for driver to click on<jupyter_code>page_soup = soup(driver.page_source, 'lxml')
blocks = page_soup.find("div", {"id": "concentrationstextcontainer"}).findAll("table", {"class": "sc_courselist"})
containers = [container for block in blocks for container in block.find("tbody").findAll("a")]
link_texts = [container.text.replace("\xa0", " ") for container in containers]
link_texts
len(link_texts)
link_texts = list(dict.fromkeys(link_texts))
len(link_texts)<jupyter_output><empty_output><jupyter_text>2. Automation script to scrape all courses<jupyter_code>from selenium.webdriver.common.keys import Keys

counter = 0
course_names = []
course_codes = []
course_descs = []

driver.get(url)
time.sleep(5)

for link_text in link_texts:
    
    link = driver.find_element_by_link_text(link_text)
    time.sleep(2)
    link.click()
    time.sleep(3)
    
    page_soup = soup(driver.page_source, 'lxml')
    course_block = page_soup.find("div", {"class": "lfjsbubble"}).find("div", {"class": "courseblock"})
    course_codes.append(course_block.find("div", {"class": "cols noindent"}).findAll("span")[0].text.strip())
    course_names.append(course_block.find("div", {"class": "cols noindent"}).findAll("span")[1].text.strip())
    course_descs.append(course_block.find("p").text.strip().replace("\xa0", " "))
    
    print("Scraped ", course_codes[-1])
    counter += 1
    
    link.send_keys(Keys.ESCAPE)
    time.sleep(2)
    
    
print("Successfully scraped {} courses".format(counter))<jupyter_output>Scraped  MECH 3520
Scraped  MECH 4182
Scraped  MECH 4192
Scraped  MECH 4200
Scraped  MECH 4452
Scraped  MECH 3582
Scraped  MECH 4482
Scraped  MECH 4432
Scraped  ENG 4110
Scraped  MECH 4350
Scraped  MECH 4360
Scraped  MECH 4620
Scraped  MECH 4870
Scraped  MECH 4472
Scraped  MECH 4510
Scraped  MECH 4532
Scraped  MECH 4550
Scraped  MECH 4672
Scraped  MECH 4292
Scraped  MECH 4412
Scraped  MECH 4560
Scraped  MECH 4680
Scraped  MECH 4692
Scraped  MECH 4694
Scraped  MECH 4702
Scraped  MECH 4822
Scraped  MECH 3550
Scraped  MECH 3562
Scraped  MECH 3570
Scraped  MECH 3592
Scraped  MECH 4162
Scraped  MECH 4310
Scraped  MECH 4322
Scraped  MECH 4330
Scraped  MECH 4342
Scraped  MECH 4542
Scraped  MECH 4582
Scraped  MECH 4812
Scraped  MECH 4832
Scraped  MECH 4900
Successfully scraped 40 courses<jupyter_text>3. Inspect and write to CSV<jupyter_code>course_codes
course_names
course_descs
import pandas as pd

df = pd.DataFrame({
    "Course Number": course_codes,
    "Course Name": course_names,
    "Course Description": course_descs    
})

df
df.to_csv('UManitoba_MechEng_Technical_Electives_Courses.csv', index = False)
driver.quit()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10744407774110744, 'fraction_numerical': 0.06453978731206454, 'mean_word_length': 4.956331877729258, 'pattern_counts': {'":': 9, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '98325', 'license': 'MIT', 'n_tokens_mistral': 1191, 'n_tokens_neox': 1043, 'n_words': 241, 'path': 'Web-Scraping Scripts and Data/Accredited Canadian English Undergrad MechEng Programs/UManitoba/WS_UManitoba_MechEng_Technical_Electives.ipynb', 'repo_name': 'JiaShengJerryLu/Keyword-Matching-for-Canadian-Mechanical-Engineering-Programs', 'size': '2727'}