<jupyter_start><jupyter_text>What I want to do:    1. Create a data frame of of parliament mebers' names and personal URLs.    2. Open each URL to find their Twitter handle.    3. Create a data frame with name, party and Twitter handle  Locating member URLs on the website<jupyter_code># Problem: without clicking on a button, the parser doesn't capture the table of politicians
# Solution: click the button and save the html file to computer and open from there

fname = 'deputes_recherche-multicritere.html' #this html file is stored on the same folder of the code file
html_file = open(fname, 'r')
source_code = html_file.read()
soup = BeautifulSoup(source_code,'html.parser')
print(soup)
# Getting the details into lists

mps = pd.DataFrame(columns=['Name', 'Surname', 'Party', 'URL'])
mpurls =[]
mpnames = []
mpsurnames = []
mpparties = []

for rows in soup.find_all('tr')[1:]: # skip the row with headers
    mpurls.append('https://www2.assemblee-nationale.fr' + rows.find_all('td')[0].find('a', href=True)['href'] + '#deputes-contact')
    mpsurnames.append(rows.find_all('td')[2].get_text().strip())
    mpnames.append(rows.find_all('td')[3].get_text().strip())
# Combining the lists into a dataframe

mps = pd.DataFrame(
    {'Name': mpnames,
     'Surname': mpsurnames,
     'URL': mpurls
    })

mps.sample(3) # does the dataframe look as it should
mpurls[4] # are the URLs functional?
mps.count() # Is the number of parliament members correct?
www = "https://www2.assemblee-nationale.fr/deputes/fiche/OMC_PA721036#deputes-contact"
soup = BeautifulSoup(requests.get(www).text)
soup.select(".twitter")<jupyter_output><empty_output><jupyter_text>The problem could be solved by downloading personal pages: https://stackoverflow.com/questions/1825438/download-html-page-and-its-contents1825465  Locating twitter handle in personal pages<jupyter_code>twitters = []
for www in mps['URL']:
    soup = BeautifulSoup(requests.get(www).text)
    
    try:
        twt = soup.find('a', attrs={'class':'twitter'}, href=True)['href']
    except:
        twt = ''
    twt = twt.replace("https://twitter.com/", "")
    twitters.append(twt)

mps['TwitterHandle'] = twitters
mps.sample(5)<jupyter_output><empty_output><jupyter_text>Cleaning up<jupyter_code># cleaning all the rubbish I found looking at the data

dirty_strings = ["/", "@"]

for dirt in dirty_strings:
    mps["TwitterHandle"] = mps["TwitterHandle"].str.replace(dirt, "")
mps.sample(5)<jupyter_output><empty_output><jupyter_text>Testing with known rubbish handles<jupyter_code>mps.to_csv("FrenchPols.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13680154142581888, 'fraction_numerical': 0.012716763005780347, 'mean_word_length': 5.093896713615023, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 4, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '297957', 'license': 'CC0-1.0', 'n_tokens_mistral': 908, 'n_tokens_neox': 876, 'n_words': 222, 'path': 'Get_FR_MPs.ipynb', 'repo_name': 'daivarep/The-Inoculation', 'size': '2595'}