<jupyter_start><jupyter_text>ESPN Cricinfo  The 'Top 4' There is a debate in cricket about who is the best batsman in the world. I think it was fairly universal that 4 batsman were part of the debate:* <NAME> (India)* <NAME> (Australia)* <NAME> (New Zealand)* <NAME> (England)However, it seems that Joe Root has fallen slightly away in this great debate. I thought it would be good to look at their test match scores spanning their entire careers and take a look at whether there is a clear fall away of England's nomination for the debate.<jupyter_code># Imports
import requests
import pandas as pd
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>We can grab the innings by innings list in test matches for each of these guys.<jupyter_code>virat_kohli = 'http://stats.espncricinfo.com/ci/engine/player/253802.html?class=1;template=results;type=allround;view=innings'
steve_smith = 'http://stats.espncricinfo.com/ci/engine/player/267192.html?class=1;template=results;type=allround;view=innings'
kane_williamson = 'http://stats.espncricinfo.com/ci/engine/player/277906.html?class=1;template=results;type=allround;view=innings'
joe_root = 'http://stats.espncricinfo.com/ci/engine/player/303669.html?class=1;template=results;type=allround;view=innings'
soup = BeautifulSoup(requests.get(joe_root).text, features="html.parser")

for caption in soup.find_all('caption'):
    if caption.get_text() == 'Innings by innings list':
        main_table = caption.find_parent('table', {'class': 'engineTable'})<jupyter_output><empty_output><jupyter_text>We should isolate the headers of the table so that we know the structure that we are aiming for.<jupyter_code>[header.get_text() for header in main_table.find('thead').find_all('tr')[0].find_all('th')]<jupyter_output><empty_output><jupyter_text>We can then take the body of the table and start working on that. We are aiming to have something that we can turn into a `pd.DataFrame` as that will make it much easier to manipulate and do plots.<jupyter_code>for innings in [row for row in main_table.find('tbody').find_all('tr')]:
    stats = [stat.get_text() for stat in innings.find_all('td')]
    print(stats)<jupyter_output>['1', '73', '-', '-', '-', '-', '-', '', 'v India', 'Nagpur', '13 Dec 2012', 'Test # 2066']
['2', '-', '1.0', '5', '0', '0', '0', '', 'v India', 'Nagpur', '13 Dec 2012', 'Test # 2066']
['3', '20*', '-', '-', '-', '-', '-', '', 'v India', 'Nagpur', '13 Dec 2012', 'Test # 2066']
['1', '4', '-', '-', '-', '-', '-', '', 'v New Zealand', 'Dunedin', '6 Mar 2013', 'Test # 2077']
['2', '-', '5.0', '8', '0', '0', '0', '', 'v New Zealand', 'Dunedin', '6 Mar 2013', 'Test # 2077']
['3', '0', '-', '-', '-', '-', '-', '', 'v New Zealand', 'Dunedin', '6 Mar 2013', 'Test # 2077']
['1', '10', '-', '-', '-', '-', '-', '', 'v New Zealand', 'Wellington', '14 Mar 2013', 'Test # 2080']
['2', '-', '1.0', '6', '0', '0', '0', '', 'v New Zealand', 'Wellington', '14 Mar 2013', 'Test # 2080']
['3', '-', '2.0', '12', '0', '0', '0', '', 'v New Zealand', 'Wellington', '14 Mar 2013', 'Test # 2080']
['1', '-', '2.0', '5', '0', '0', '0', '', 'v New Zealand', 'Auckland', '22 Mar 2013', 'Test # 2084']
['2', '45', '-', '-[...]<jupyter_text>That looks ideal for now.<jupyter_code>headers = [header.get_text() for header in main_table.find('thead').find_all('tr')[0].find_all('th')]
rows = []

for innings in [row for row in main_table.find('tbody').find_all('tr')]:
    rows.append([stat.get_text() for stat in innings.find_all('td')])
headers
rows[:2]
len(headers)
len(rows)
len(rows[0])<jupyter_output><empty_output><jupyter_text>This is encouraging that it looks like we have pulled the data in the right dimensions, so it should hopefully align.<jupyter_code>pd.DataFrame(rows, columns=headers)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.2003686150605582, 'fraction_numerical': 0.04844655081621906, 'mean_word_length': 5.41722972972973, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '387750', 'license': 'MIT', 'n_tokens_mistral': 1451, 'n_tokens_neox': 1308, 'n_words': 393, 'path': 'top_4_test_matches.ipynb', 'repo_name': 'willcanniford/python-cricketer-analysis', 'size': '3819'}