<jupyter_start><jupyter_text>Leeds League HistoryAfter Leeds' return to the Premier League in 2020, I decided that I wanted to plot their final league positions after each season since the formation of the Premier League. Wikipedia has a list of the league positions for each season [here](https://en.wikipedia.org/wiki/List_of_Leeds_United_F.C._seasons) so I decided to build a scraper that pulls this information and stores as a csv for me to easily read in for the visual.<jupyter_code>import requests
import pandas as pd
from bs4 import BeautifulSoup
import re 
url = "https://en.wikipedia.org/wiki/List_of_Leeds_United_F.C._seasons"
html = requests.get(url).text 
soup = BeautifulSoup(html, features="html.parser")
tables = soup.find_all(class_='wikitable')
header_rows = tables[0].find_all('tr')[:2]
data_rows = tables[0].find_all('tr')[2:]<jupyter_output><empty_output><jupyter_text>The table that we are trying to scrape here isn't the most regular and contains some rows that have 'sub-rows' and double cells, so I'm going to define some helper functions to work on being able to pull the information that we need from the rows that we have looped through and found.<jupyter_code>def find_season(data_row):
    """
    Given a data row, return the season for that row
    """
    return re.search(r'(\d\d\d\d.{1}\d\d).*', data_row.find('th').text).group(1)

def find_data_values(data_row):
    """
    Find headers to match the header in each row 
    """
    return [x.text.replace('\n', '') for x in data_row.find_all('td')]

def find_division(data_row):
    """
    For a given cleaned data row, take the first element, the division
    """
    return data_row[0]

def find_position(data_row):
    """
    For a given cleaned data row, take the ninth element, the final position for that season
    """
    return data_row[8]<jupyter_output><empty_output><jupyter_text>Due to the format of the table, we have to do some manual intervention here because `data_rows[72]` is a sub row for the same year that is started by `data_rows[71]`. This is because in that year there are 2 cells required for the 'Europe/Other' category as Leeds played in the FA Charity Shield and the Champions League. For the time being, I will manually declare the rows and as we don't need that information.<jupyter_code># Show the partial row for reference 
[x.text.replace('\n','') for x in data_rows[71:73]]
# As discussed, I have removed the sub row for easier processing of our elements 
relevant_data_rows = [data_rows[71]] + data_rows[73:]
# Pull the seasons for the ones that we are interested in
seasons = [find_season(data_rows[71])] + [find_season(x) for x in data_rows[73:]]
# Use the helper function to pull the values, and isolate the ones we are interested in
data_values = [find_data_values(x) for x in relevant_data_rows]
divisions = [find_division(x) for x in data_values]
positions = [find_position(x) for x in data_values]
# Store data as a dictionary for conversion
data_dict = {
    'season': seasons, 
    'division': divisions, 
    'position': positions
}

# Format as a data frame
pd.DataFrame(data_dict)<jupyter_output><empty_output><jupyter_text>We are almost at the stage that we have all the data that we need. We just need to do some more cleaning and add some additional columns. 1. Remove the reference brackets that provide extra information on the site2. Remove the suffix for the date and store that (in case we want to use it later)3. Add a column to indicate the number of teams in the league<jupyter_code>def remove_ref_brackets(str):
    # If a reference exists, take the part before it 
    if '[' in str:
        pattern_str = "(.*)\[[a-z]{1}\]" 
        pattern = re.compile(pattern_str)
        return re.search(pattern, str).group(1)
    else:
        return str

def separate_position_suffix(str):
    pattern_str = "(\d{1,2})([a-z]{2})"
    pattern = re.compile(pattern_str)
    return {'position': re.search(pattern, str).group(1), 
            'suffix': re.search(pattern, str).group(2)}
clean_divisions = [remove_ref_brackets(x) for x in divisions]
clean_positions = [remove_ref_brackets(x) for x in positions]
separated_positions = [separate_position_suffix(x) for x in clean_positions]
# Update dict for clean columns
data_dict = {
    'season': seasons, 
    'division': clean_divisions,
    'position_dict': separated_positions
}

# Format as a data frame
df = pd.DataFrame(data_dict)
df = pd.concat([df.loc[:,['season', 'division']], 
           df.position_dict.apply(pd.Series)], sort=False, axis = 1)

division_total_teams = {'Prem': 20, 'Champ': 24, 'Lge 1': 24}
df["division_total_teams"] = df.division.map(division_total_teams)
df
df.to_csv('../output/leeds_league_history.csv', index=False)
df
import matplotlib.pyplot as plt
df['position'] = pd.to_numeric(df["position"])
df["y_position"] = (df.division_total_teams - df.position + 1) 
df["y_base"] = df.division.map({'Prem': 48, 'Champ': 24, 'Lge 1': 0})
df["y"] = df.y_position + df.y_base
df.drop(['y_position', 'y_base'], axis=1, inplace=True)
df.sort_values('y') # Check that the logic works and that lower numbers mean lower position overall 
fig, ax = plt.subplots(figsize = (12, 4))

# Add lines and points
ax.plot(df.season, df.y, zorder=100)
ax.scatter(df.season, df.y, s=15, alpha=0.75)

# Hard code the limits 
ax.set_ylim(0, 70)

# Add lines to separate out the leagues
ax.axhline(24, zorder=-1, c='grey', alpha=0.25)
ax.axhline(48, zorder=-1, c='grey', alpha=0.25)
ax.axhline(68, zorder=-1, c='grey', alpha=0.25)

# Make the xtick labels legible
plt.xticks(rotation=50, ha='right', rotation_mode="anchor")

# Remove the spines to clear up the visual 
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.spines['left'].set_visible(False)

# Add league markers in the middle of the band for that league 
ax.set_yticks([12, 36, 58])
ax.set_yticklabels(['League 1', 'Championship', 'Premier League'])

# Remove y tick markers 
ax.tick_params(axis='y', which='both', length=0)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11715135492399206, 'fraction_numerical': 0.016192994051553205, 'mean_word_length': 4.694261523988711, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '298616', 'license': 'MIT', 'n_tokens_mistral': 2050, 'n_tokens_neox': 1940, 'n_words': 674, 'path': 'notebooks/leeds_league_history.ipynb', 'repo_name': 'willcanniford/python-web-scraping', 'size': '6052'}