<jupyter_start><jupyter_text>1. Collect course link texts for webdriver to click on<jupyter_code>page_soup = soup(driver.page_source, 'lxml')
#Genie Mecanique Regle 1
containers = page_soup.find("section", {"id": "section-structure"}).findAll("div", {"class": "fe-bloc-section"})[0].findAll("div", {"class": "toggle-section"})[1].find("ul", {"class": "fe--liste-cours"}).findAll("a")
len(containers)
#Cheminement sans concentration Regle 1
containers += page_soup.find("section", {"id": "section-structure"}).findAll("div", {"class": "fe-bloc-section"})[1].findAll("div", {"class": "toggle-section"})[0].findAll("a")
len(containers)
#Cheminement sans concentration Regle 2
containers += page_soup.find("section", {"id": "section-structure"}).findAll("div", {"class": "fe-bloc-section"})[1].findAll("div", {"class": "toggle-section"})[1].findAll("a")
len(containers)
#Cheminement sans concentration Regle 3
containers += page_soup.find("section", {"id": "section-structure"}).findAll("div", {"class": "fe-bloc-section"})[1].findAll("div", {"class": "toggle-section"})[2].findAll("a")
len(containers)
link_texts = [container.text.strip() for container in containers]
link_texts<jupyter_output><empty_output><jupyter_text>2. Implement automation script to scrape all courses first click open all tabs<jupyter_code>driver.find_element_by_id("genie-mecanique-regle-1").click()
from selenium.webdriver.common.keys import Keys
tab = driver.find_element_by_id("cheminement-sans-concentration-regle-1")
time.sleep(2)
tab.send_keys(Keys.DOWN)
tab.click()
tab = driver.find_element_by_id("cheminement-sans-concentration-regle-2")
time.sleep(2)
tab.send_keys(Keys.DOWN)
tab.click()
tab = driver.find_element_by_id("cheminement-sans-concentration-regle-3")
time.sleep(3)
tab.send_keys(Keys.DOWN)
tab.click()
#scroll to the top of the page
tab.send_keys(Keys.CONTROL + Keys.HOME)
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import ElementClickInterceptedException

course_codes = []
course_names = []
course_descs = []
counter = 0
    
for link_text in link_texts:
    #go to course page
    try:
        link = driver.find_element_by_link_text(link_text)
    except NoSuchElementException:
        print("no link for {}".format(link_text))
        continue

    time.sleep(2)
    try:
        link.click()
    except ElementClickInterceptedException:
        link.send_keys(Keys.DOWN)
        link.click()
        
    time.sleep(3)
    
    #scrape course info
    page_soup = soup(driver.page_source, 'lxml')
    course_title_container = page_soup.find("h1", {"class": "fe--titre"})
    course_code = course_title_container.find("span", {"class": "fe--titre-type"}).text.strip()
    course_name = course_title_container.find("span", {"class": "fe--titre-nom"}).text.strip()
    course_desc = page_soup.find("p", {"class": "fe--intro"}).text.strip()
    
    course_codes.append(course_code)
    course_names.append(course_name)
    course_descs.append(course_desc)
    
    print("Scraped ", course_codes[-1], course_names[-1], "\n", course_descs[-1])
    counter += 1    
    print("{} courses scraped".format(counter))
    print()
    
    #go to course list page
    driver.back()
    time.sleep(3)
        
print("Finished scraping {} courses".format(counter))<jupyter_output>Scraped  GMN-2902 Santé et sécurité pour ingénieur III 
 Problèmes de santé et de sécurité du travail, lois et règlements. Accidents et maladies professionnelles. Éléments d'ergonomie et postures de travail. Conséquences des accidents, incidents et maladies professionnelles. Organisation de la sécurité en entreprise. Systèmes de prévention. Évaluation en sécurité. SIMDUT. Notions du risque. Types d'expositions. Fondements de normes. Évaluation des risques associés aux poussières, aux bruits, aux vibrations, à la contrainte thermique et à l'éclairage. Méthodes de mesure, contrôle et protection, normes et règlements. Sécurité industrielle. Mesure du niveau de danger. Acceptabilité du risque. Gestion du risque. Santé publique, sécurité et environnement. Devoirs et responsabilités juridiques des ingénieurs. Lois, normes et règlements en matière de sécurité, risques et gestion de risques. Maladies infectieuses et santé des travailleurs expatriés.
1 courses scraped

Scraped  MED-1100 Santé e[...]<jupyter_text>3. Inspect and write to CSV<jupyter_code>course_codes
course_names
course_descs
len(list(dict.fromkeys(course_codes)))
course_codes1 = []
course_names1 = []
course_descs1 = []
counter = 0

for i in range(len(course_codes)):
    if course_codes[i] not in course_codes1:
        course_codes1.append(course_codes[i])
        course_names1.append(course_names[i])
        course_descs1.append(course_descs[i])
        counter += 1
    else:
        print("Duplicate: ", course_names[i])
        
print(counter)
import pandas as pd

df = pd.DataFrame({
    "Course Number": course_codes1,
    "Course Name": course_names1,
    "Course Description": course_descs1
})

df
df.to_csv('Laval_GenieMecanique_Elective_Courses.csv', index = False)
driver.quit()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.132890365448505, 'fraction_numerical': 0.010162204416650382, 'mean_word_length': 5.04250295159386, 'pattern_counts': {'":': 20, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '212097', 'license': 'MIT', 'n_tokens_mistral': 1770, 'n_tokens_neox': 1705, 'n_words': 385, 'path': 'Web-Scraping Scripts and Data/Accredited Canadian French Undergrad MechEng Programs/Laval/WS_Laval_GenieMecanique_Electives.ipynb', 'repo_name': 'JiaShengJerryLu/Keyword-Matching-for-Canadian-Mechanical-Engineering-Programs', 'size': '5117'}