<jupyter_start><jupyter_text>L'aide de Beautiful soup est ici :<jupyter_code>from bs4 import BeautifulSoup
soup = BeautifulSoup(open('data_1926-1928.html', encoding='iso-8859-1'), 'html.parser')
spacers = soup.find_all(class_='spacer')
node = spacers[4]
node
len(node)
node.contents
node.p.text<jupyter_output><empty_output><jupyter_text>Vérifions pour tous les spacers :<jupyter_code>len([node.p.text for node in spacers if node.p is not None])
3 * 12<jupyter_output><empty_output><jupyter_text>C'est suffisament régulier sur nos données de test. On a 12 valeurs par an * 3 ans.<jupyter_code>node.next_sibling.table.tr.text
[node.next_sibling.table.find(text='Maximum instantané').parent.parent.parent
 for node in spacers if node.p is not None]
node<jupyter_output><empty_output><jupyter_text>Utilisation d'une regexp<jupyter_code>node.next_sibling.table.text
t = node.next_sibling.table.text
import re
pattern_cm = re.compile('Hauteur :\n(\d+.\d*)\xa0cm.+Date :\n([\d/: ]+)', re.DOTALL)
pattern_mm = re.compile('Hauteur :\n(\d+.\d*)\xa0mm.+Date :\n([\d/: ]+)', re.DOTALL)
re.findall(pattern_cm, t)
[re.findall(p, t) for t in [node.next_sibling.table.text for node in spacers] if t is not None]
for node in spacers:
    print(node)
    if len(list(node.children)) > 0:
        if node.next_sibling.table is not None:
            t = node.next_sibling.table.text
            print(re.findall(p, t))
list(node.children)
len(node.children)<jupyter_output><empty_output><jupyter_text>On cherche les tables<jupyter_code>table = soup.find_all('table', class_='statistiques')[1]<jupyter_output><empty_output><jupyter_text>Filtrage mensuel  On prend la table si elle est liée au mois (donc dé<NAME>urnaliers) :<jupyter_code>def is_monthly(table):
    "Returns True if monthly data table."
    cap = table.find('caption')
    if cap is None:
        label = table.find(class_='procedure_choix')
        if label is not None:
            return label.text == 'Ecoulement mensuel'
        else:
            return False
    else:
        return False
table = soup.find_all('table', class_='statistiques')[1]
table
table.find(class_='procedure_choix').text
is_monthly(table)
for table in soup.find_all('table', class_='statistiques'):
    print(is_monthly(table))<jupyter_output>False
False
True
True
True
True
True
True
True
True
True
True
True
True
False
False
True
True
True
True
True
True
True
True
True
True
True
True
False
False
True
True
True
True
True
True
True
True
True
True
True
True<jupyter_text>Extraction donnée  On sait qu'on a une table mensuelle, mais on n'est pas sûr des données qu'elle contient.<jupyter_code>def extract_height_date(soup):
    "Extracts height (in cm) and date from soup."
    pattern_cm = re.compile('Hauteur :\n(\d+.\d*)\xa0cm.+Date :\n([\d/: ]+)', re.DOTALL)
    pattern_mm = re.compile('Hauteur :\n(\d+.\d*).*mm.+Date :\n([\d/: ]+)', re.DOTALL)
    res = re.findall(pattern_cm, soup.text)
    if len(res) > 0:
        # assume it was cm
        return res
    else:
        # assume it was mm
        res = re.findall(pattern_mm, soup.text)
        return res
data = []
for table in soup.find_all('table', class_='statistiques'):
    if is_monthly(table):
        data.append(*re.findall(p, table.text))
import pandas as pd
def parse_dates(df):
    df = df.copy()
    df['date'] = pd.to_datetime(df['date'])
    return df
def make_numeric(df):
    df = df.copy()
    df['hauteur'] = [float(val.replace(',', '.')) for val in df['hauteur']]
    return df
df = pd.DataFrame(data, columns=['hauteur', 'date']).pipe(parse_dates).pipe(make_numeric).set_index('date')
df
%matplotlib notebook
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
df.plot.line(ax=ax, style='-o')<jupyter_output><empty_output><jupyter_text>Tests   Fonction<jupyter_code>def extract_height_date(soup):
    "Extracts height (in cm) and date from soup."
    pattern_cm = re.compile('Hauteur :\n(\d+.\d*)[ \xa0]cm.+Date :\n([\d/: ]+)', re.DOTALL)
    pattern_mm = re.compile('Hauteur :\n(\d+.\d*)[ \xa0]mm.+Date :\n([\d/: ]+)', re.DOTALL)
    res = re.findall(pattern_cm, soup.text)
    if len(res) > 0:
        # assume it was cm
        return res
    else:
        # assume it was mm
        res = re.findall(pattern_mm, soup.text)
        if len(res) > 0:
            height, date = res[0]
            height = str(float(height) / 10)
            return [(height, date)]
        else:
            return [(None, None)]<jupyter_output><empty_output><jupyter_text>Test cases  On va tester les extractions en cm et mm et valider les résultats.<jupyter_code>test_cases = ["""<table class="statistiques" width="42%"><tbody>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Ecoulement mensuel</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Débit moyen :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">366.0 m3/s</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Débit moyen spécifique :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">8.37 l/s/km2</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Lame d'eau :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">22.4 mm</td>
</tr>
<tr><td class="left" colspan="3" id="C1"> </td></tr>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Ecoulement naturel reconstitué</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Débit moyen :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">366.0 m3/s</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Débit moyen spécifique :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">8.37 l/s/km2</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Lame d'eau :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">22.4 mm</td>
</tr>
<tr><td class="left" colspan="3" id="C1"> </td></tr>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Maximum instantané</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Hauteur :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">1570.0 mm</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Date :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">17/01/2008 19:10</td>
</tr>
</tbody></table>""",
             """<table class="statistiques" width="42%"><tbody>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Ecoulement annuel</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Débit moyen :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">441.0 m3/s</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Débit moyen spécifique :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">10.10 l/s/km2</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Lame d'eau :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">318.0 mm</td>
</tr>
<tr><td class="left" colspan="3" id="C1"> </td></tr>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Ecoulement naturel reconstitué</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Débit moyen :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">441.0 m3/s</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Débit moyen spécifique :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">10.10 l/s/km2</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Lame d'eau :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">318.0 mm</td>
</tr>
<tr><td class="left" colspan="3" id="C1"> </td></tr>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Maximum instantané</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Débit :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">1740.0 m3/s</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Validité :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">#</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Date :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">08/01/1926 07:00</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Hauteur :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">602.0 cm</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Date :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">08/01/1926 07:00</td>
</tr>
</tbody></table>""", 
             """<table class="statistiques" width="42%"><tbody>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Ecoulement mensuel</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Débit moyen :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">518.0 m3/s</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Débit moyen spécifique :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">11.80 l/s/km2</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="40%">Lame d'eau :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">28.6 mm</td>
</tr>
<tr><td class="left" colspan="3" id="C1"> </td></tr>
<tr><td class="left" colspan="3" id="C1"><span class="procedure_choix">Ecoulement naturel reconstitué</span></td></tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Débit moyen :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">518.0 m3/s</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Débit moyen spécifique :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">11.80 l/s/km2</td>
</tr>
<tr>
<td class="left" id="C1"> </td>
<td class="left" id="C2" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="30%">Lame d'eau :</td>
<td class="left" id="C3" style="text-align:left;border-color:#FFFFFF;font-size:1em;" width="50%">28.6 mm</td>
</tr>
<tr><td class="left" colspan="3" id="C1"> </td></tr>
</tbody></table>"""]
test_results = [1, 1, 0]
for tc, tr in zip(test_cases, test_results):
    test_soup = BeautifulSoup(tc, 'xml')
    print("expected: {}, actual: {}".format(tr, extract_height_date(test_soup)))<jupyter_output>expected: 1, actual: [('157.0', '17/01/2008 19:10')]
expected: 1, actual: [('602.0', '08/01/1926 07:00')]
expected: 0, actual: [(None, None)]<jupyter_text>But does it scale ?  On peut tester notre fonction sur des données un peu plus grandes.<jupyter_code>soup = BeautifulSoup(open('data_2008-2016.html', encoding='iso-8859-1'), 'html.parser')
data = []
for table in soup.find_all('table', class_='statistiques'):
    if is_monthly(table):
        data.append(*extract_height_date(table))
data
pd.DataFrame(data).dropna()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.22541314726404701, 'fraction_numerical': 0.034814542783694453, 'mean_word_length': 5.880242546740778, 'pattern_counts': {'":': 0, '<': 324, '<?xml version=': 0, '>': 328, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '603762', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 5752, 'n_tokens_neox': 5533, 'n_words': 1027, 'path': '20160606_xtract 1926.ipynb', 'repo_name': 'flothesof/crues-seine-2016', 'size': '13616'}