<jupyter_start><jupyter_text>Data Collection for 2018 FIFA World Cup Russia™ Prediction ProjectThe goal of this notebook is to show how I collected data from the FIFA World Cup website for the 2018 World Cup Russia - Group Stage Prediction Project. Data was collected using a web crawler (build in Python), due to lack of API. The order of this notebook is as follows:1. Information about the data source2. Step by step instruction on how the web crawler was built3. Result of data collection  Data sourceI have collected groups statistics for all FIFA World Cups between 1994 and 2014. That gives us 6 FIFA World Cups: - 2014 FIFA World Cup Brazil™ - 2010 FIFA World Cup South Africa™ - 2006 FIFA World Cup Germany™ - 2002 FIFA World Cup Korea/Japan™ - 1998 FIFA World Cup France™ - 1994 FIFA World Cup USA™The tournaments between 1998 and 2014 included 32 teams, divided into eight groups (A to H) - four teams (countries) per group. 1994 FIFA World Cup USA™ included 24 teams, divided into six groups (A to F) - four teams (countries) per group.For each World Cup edition, I collected the following groups' statistics: - Group - name of the group - Teams - name of the team (country) - Match played - how many matches did the team play - Match won - how many matches it won     - Draw - number of draws - Lost - number of loses - Goals for - total number of goals - Goals against - total number of lost goals - Goals difference - difference between goals for and against - Points - total pointsThis data is publicaly accessible on the FIFA website in the statistics and records section. According to the robots.txt, data scraping from the statistics and records page is allowed [June 10, 2018].I went through the process of building this web scraper in the [Let the robot do your work! Web scraping with Python!](https://medium.com/ub-women-data-scholars/let-the-robot-do-your-work-web-scraping-with-python-9c147fb7690f) article posted on Medium.   Web crawlerThis part of the notebook will explan how the web crawler was built.Program was built in Python 3.x The following libraries were used: - Requests - BeautifulSoup - PandasThe structure of the code:1. Libraries import2. Obtaining data from the FIFA Tournaments page - GET request to the FIFA Tournaments page - saving the page content in Beautiful Soup format  - extraction of links to individual FIFA World Cup editions - edition of links in order to generate links to the pages containing groups statistics3. Obtaining group statistics data from the individual World Cup pages - GET request to each page containing groups statistics - saving the page content in Beautiful Soup format - extraction of data from the HTML - saving data as panda's dataframe4. Export of dataframe to Excel  1. Libraries import- Request library, which allows to send HTTP requests- BeautifulSoup library for pulling data out of HTML- Pandas library to store data as dataframe<jupyter_code>import requests
from bs4 import BeautifulSoup
import pandas as pd<jupyter_output><empty_output><jupyter_text>2. Obtaining data from the FIFA Tournaments pageI set url variable to FIFA Tournaments page and then use try/except block to check if page is accessible. I set timeout to 5 seconds, and store HTTP status code in page_response variable. If page is not accessible, I print page status to console. If timeout occurred, I print notification to console.If page is accessible (200 is a standard code for successful HTTP requests) I save HTML to page_content variable using BeautifulSoup format. Then, I use BeautifulSoup lxml parser to find element, which contains links to all World Cup editions. I extract these links and edit them to generate new links, which direct to the groups statistics. As I am only interested in the last six World Cups, I keep the first 6 elements of the list.<jupyter_code>url = "http://www.fifa.com/fifa-tournaments/statistics-and-records/worldcup/index.html"

try:
    page_response = requests.get(url, timeout=5)  
    
    if page_response.status_code == 200:
        page_content = BeautifulSoup(page_response.content,'lxml')
        tournaments = page_content.find(attrs={'id':'alleditions'})

        link_list = []
        for link in tournaments.find_all('a'):
            group_link = 'http://www.fifa.com' + link.get('href').replace('index.html','groups/index.html')
            link_list.append(group_link)
        link_list = link_list[0:6]

    else:
        print(page_response.status_code)
        
except requests.Timeout as e:
        print('Timeout occurred for requested page: ' + url)
        print(str(e))<jupyter_output><empty_output><jupyter_text>As a result of these operations, I receive a list containing 6 links to group statistics, one link for each edition of the World Cup.<jupyter_code>for link in link_list:
    print(link)<jupyter_output>http://www.fifa.com/worldcup/archive/brazil2014/groups/index.html
http://www.fifa.com/worldcup/archive/southafrica2010/groups/index.html
http://www.fifa.com/worldcup/archive/germany2006/groups/index.html
http://www.fifa.com/worldcup/archive/koreajapan2002/groups/index.html
http://www.fifa.com/worldcup/archive/france1998/groups/index.html
http://www.fifa.com/worldcup/archive/usa1994/groups/index.html<jupyter_text>3. Obtaining group statistics data from the individual World Cup pagesFirst, I create an empty list to store groups' statistics and set counter variable to 0. I use while loop to iterate through the link_list, and increment counter by 1 on each iteration. While loop will not be executed if link_list is empty, and will break when counter is equal to the list's lenght. Inside the loop, I set url variable to the link from the link_list, using counter as list index. Then I use try/except block to check if page is accessible. I set timeout to 5 seconds, and store HTTP status code in page_response variable. If page is not accessible, I print page status to console. If timeout occurred, I print notification to console.If page is accessible (200 is a standard code for successful HTTP requests) I save HTML to page_content variable using BeautifulSoup format. Then, I use BeautifulSoup lxml parser to find HTML elements, which contain the data: - take the title of the webpage (e.g. 2014 FIFA World Cup Brazil™) and split it to extract World Cup year  - select HTML element that contains statistics for all groups and save it in the standings_table variable - select all HTML elements that contain group letters and use list comprehensions to extract their inner text; - repeat each group letter four times (on the FIFA website, for each group (A - H), there is only one HTML element containing the letter of the group; since there are four teams in each group, repeat the letter of the group four times) - select all HTML elements that contain relevant statistics (teams, match_played, match_won, draw, lost, goals_for, goals_against, goals_difference, points) and use list comprehensions to extract their inner text;After collecting the data, I save it in the pandas dataframe format. I append dataframe to append_data list.As a result of while loop, I receive list of dataframes. I use Pandas concat method to concatenate all dataframes into one dataframe.<jupyter_code>append_data = []
counter = 0

while counter < len(link_list) and len(link_list) > 0:
    
    url = link_list[counter]
    try:
        page_response = requests.get(url, timeout=5)

        if page_response.status_code == 200:
            page_content = BeautifulSoup(page_response.content,'lxml')

            page_title = page_content.title.text
            world_cup = page_title.split(' ')[0]

            standings_table = page_content.find('div', attrs={'id':'standings'})

            group_letters = [gl.get_text() for gl in standings_table.select('.group-wrap .caption-nolink')]
            group_letter = [gl for gl in group_letters for i in range(4)]

            teams = [tn.get_text() for tn in standings_table.select('.group-wrap .teamname-nolink span.t-nText')]
            match_played = [mp.get_text() for mp in standings_table.select('.group-wrap .tbl-matchplayed span.text')]
            match_won = [mw.get_text() for mw in standings_table.select('.group-wrap .tbl-win span.text')]
            draw = [d.get_text() for d in standings_table.select('.group-wrap .tbl-draw span.text')]
            lost = [l.get_text() for l in standings_table.select('.group-wrap .tbl-lost span.text')]
            goals_for = [gf.get_text() for gf in standings_table.select('.group-wrap .tbl-goalfor span.text')]
            goals_against = [ga.get_text() for ga in standings_table.select('.group-wrap .tbl-goalagainst span.text')]
            goals_difference = [gd.get_text() for gd in standings_table.select('.group-wrap .tbl-diffgoal span.text')]
            points = [p.get_text() for p in standings_table.select('.group-wrap .tbl-pts span.text')]

            group_tables = pd.DataFrame({
                "Fifa World Cup": world_cup,
                "Group": group_letter, 
                "Teams": teams, 
                "Match played": match_played, 
                "Match won": match_won,
                "Draw": draw,
                "Lost": lost,
                "Goals for": goals_for,
                "Goals against": goals_against,
                "Goals difference": goals_difference,
                "Points": points
            })

            append_data.append(group_tables)

        else:
            print(page_response.status_code)

    except requests.Timeout as e:
        print('Timeout occurred for requested page: ' + url)
        print(str(e))

    counter += 1
    
worldcup_data = pd.concat(append_data, axis=0, ignore_index=True)<jupyter_output><empty_output><jupyter_text>4. Export of dataframe to Excel fileAs the last step, I export dataframe to Excel.<jupyter_code>worldcup_data.to_excel('WorldCupData.xlsx')

worldcup_data.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07393976024982371, 'fraction_numerical': 0.014304422282663444, 'mean_word_length': 3.6090993500464252, 'pattern_counts': {'":': 11, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 8, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '112217', 'license': 'MIT', 'n_tokens_mistral': 2800, 'n_tokens_neox': 2610, 'n_words': 1158, 'path': 'Web Scraper - FIFA World Cup Data.ipynb', 'repo_name': 'BarbaraStempien/DA--World-Cup-Data', 'size': '9927'}