<jupyter_start><jupyter_text>Movember, moustaches, and ~~shaving~~ scraping! Movember is upon us and it's time to [support Men's health](https://us.movember.com/) by displaying a **glorious moustache**. To get inspiration, we're going to pay homage to the contestants of the 2017 World Beard and Moustache Championship, which was held in Austin, Texas USA from September 1-3, 2017. We'll do this by scraping data from the official website of the [Austin Facial Hair Club](https://www.austinfacialhairclub.com/), who hosted the event.  Connecting and collectingIf you've never webscraped before, you should know webscraping can be _messy_. It requires you to dump all the `HTML` from a site and sift through it. And websites aren't designed with webscraping in mind, so you may have to jump around to get the information you need. Fortunately, there are packages to help! The `request` module from the [urllib](https://docs.python.org/3/library/urllib.html) package lets you connect to a webpage and grab its contents for you to work with locally. You can then use [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) to help sort through all the `HTML` and extract only the items you want. In this project, we will only scrape a small amount from our chosen site, but out in the wild, you will likely need to scrape LOTS more to get the data you want.<jupyter_code># Import urlopen() and BeautifulSoup
from urllib.request import urlopen
from bs4 import BeautifulSoup

# Assign the website url to an object and connect
url = "https://www.austinfacialhairclub.com/2017-wbmc-results"
html = urlopen(url)

# Use BeautifulSoup to scrape the HTML from th url
soup = BeautifulSoup(html, 'html.parser')

# Use the .prettify() method to print the soup object in a semi-readable way
pretty_soup = soup.prettify()
print(pretty_soup)

# Check out the class of the soup objects
print('soup is a {} object and pretty_soup is a {} object.' .format(type(soup).__name__, type(pretty_soup).__name__))<jupyter_output><!DOCTYPE doctype html>
<html lang="en-US" xmlns:fb="http://www.facebook.com/2008/fbml" xmlns:og="http://opengraphprotocol.org/schema/" xmlns:website="http://ogp.me/ns/website">
 <head>
  <meta content="IE=edge,chrome=1" http-equiv="X-UA-Compatible"/>
  <meta content="initial-scale=1" name="viewport"/>
  <!-- This is Squarespace. -->
  <!-- afhc -->
  <base href=""/>
  <meta charset="utf-8">
   <title>
    Austin Facial Hair Club  — 2017 WBMC Results
   </title>
   <link href="https://static1.squarespace.com/static/54ef6bc4e4b0132962f60f94/t/590a8f8aa5790a517c906da6/favicon.ico" rel="shortcut icon" type="image/x-icon"/>
   <link href="https://www.austinfacialhairclub.com/2017-wbmc-results" rel="canonical"/>
   <meta content="ZqL9rQQihiHV75CC_mH7nzrv9r1acr1hJ-Orh718c8s" name="google-site-verification">
    <meta content="Austin Facial Hair Club" property="og:site_name"/>
    <meta content="Austin Facial Hair Club  — 2017 WBMC Results" property="og:title"/>
    <meta content="https://www[...]<jupyter_text>Inspecting `HTML` tags and exploring `BeautifulSoup` While the formatting from the `.prettify()` method was _helpful_ (print `soup` on its own without the `.prettify()` method if you don't believe me - it's terrifying), it's still quite difficult to navigate through all this `HTML` to see what we actually need to scrape. To find out how to proceed, we need to look at the website itself. Using the inspect tool on the [2017 WBMC RESULTS](https://www.austinfacialhairclub.com/2017-wbmc-results) page will reveal the site's raw `HTML`  and will allow us to look through the `HTML` tags alongside the rendered website. If you don't know anything about `HTML`, don't worry, we'll get through this together. `HTML` tags indicate the type of information contained within that tag, for example `` tags contain hyperlinks, `` tags contain an image, `` tags define a section, etc. To learn more about `HTML` and tags, [W3Schools](www.w3schools.com) provides great, succinct tutorials.[![Inspect WBMC categories](https://assets.datacamp.com/production/repositories/3081/datasets/cd5a53082794a1b7c26e960c597b012a042248fe/YouTube still2.png)](https://www.youtube.com/watch?v=EAFprPJFNBM&feature=youtu.be "Inspect WBMC categories")You can open the inspect tool using `Ctrl + Shift + i` on Windows machines or `Command + Option + i` on a Mac. You can also simply right click within a website and click on "Inspect". Clicking on the image above will send you to a YouTube video of the WBMC results page being inspecting. Something really helpful about using the inspect tool is that as you scroll over the raw `HTML`, the related sections of rendered images and text are highlighted. You can use this highlighting to find the tags you need quickly. Inspecting the WBMC results webpage reveals that all beard and moustache categories and their descriptions are located within `` tags, which stands for paragraphs. `` tags contin text elements. In the next step, we'll take this new knowledge of `HTML` and explore the various methods from `BeautifulSoup` that will ease our hairy journey into webscraping.<jupyter_code>from IPython.display import YouTubeVideo

def display_yotube_video(url, **kwargs):
    """
    Displays a Youtube video in a Jupyter notebook.
    
    Args:
        url (string): a link to a Youtube video.
        **kwargs: further arguments for IPython.display.YouTubeVideo
    
    Returns:
        YouTubeVideo: a video that is displayed in your notebook.
    """
    id_ = url.split("=")[-1]
    return YouTubeVideo(id_, **kwargs)

display_yotube_video("https://www.youtube.com/watch?v=EAFprPJFNBM", width=800, height=600)
# Import pprint module and create a PrettyPrinter instance
import pprint
pp = pprint.PrettyPrinter()

# Explore Beautiful Soup
pp.pprint(soup.title)

#Find all the paragraph tags within the HTML
categories = soup.find_all('p')

# Print out all paragraph tag contents and determine object type
pp.pprint(categories)
print('\nThe .find_all() method from Beautiful Soup returns {} objects.'.format(type(categories).__name__))<jupyter_output><title>Austin Facial Hair Club  — 2017 WBMC Results</title>
[<p>Natural Moustache</p>,
 <p>English Moustache</p>,
 <p>Dali Moustache</p>,
 <p>Imperial Moustache</p>,
 <p>Hungarian Moustache</p>,
 <p>Realistic Moustache</p>,
 <p>Creative Moustache</p>,
 <p>Realistic Beard</p>,
 <p>Creative Beard</p>,
 <p>Musketeer</p>,
 <p>Imperial Beard</p>,
 <p>Fu Manchu</p>,
 <p>Alaskan Whaler</p>,
 <p>Sideburns Natural</p>,
 <p>Natural Goatee</p>,
 <p>Verdi</p>,
 <p>Garibaldi</p>,
 <p>Full Beard with Styled Moustache</p>,
 <p>Full Beard 0.1-15cm</p>,
 <p>Full Beard 15.1-30cm</p>,
 <p>Full Beard 30.1-45cm</p>,
 <p>Full Beard 45.1-60+cm</p>,
 <p>Freestyle Moustache</p>,
 <p>Cheekbeard (Backenbart) Freestyle</p>,
 <p>Sideburns Freestyle</p>,
 <p>Partial Beard Freestyle</p>,
 <p>Full Beard Freestyle</p>,
 <p style="white-space: pre-wrap;">1514 Ed Bluestein Blvd Suite 107<br/>Austin, TX 78721<br/>email: <a href="mail to: <EMAIL>"><EMAIL></a></p>,
 <p>Thank you [...]<jupyter_text>Time to clean up! We've actually managed to get pretty far already. You now know that all the categories that competed at the last World Beard and Moustache Championships live in paragraph tags, that there are a few more paragraph tags than you need, and that using `BeautifulSoup` returns a weird object called a `ResultSet`. So, we need to do two things: convert the `ResultSet` object to something we can work with and remove the last two tags, which contain contact information and a polite "Thank you for signing up!"<jupyter_code># Create a blank list object and grab the text from within the paragraph tags
all_categories = []

for category in categories:
    all_categories.append(category.get_text("p"))

all_categories
# Remove contact info from list
all_categories = all_categories[:-2]

# Pretty print all_categories
pp.pprint(all_categories)

# Determine object type and length
print('\nall_categories is a {} object with {} items.'.format(type(all_categories).__name__, len(all_categories)))<jupyter_output>['Natural Moustache',
 'English Moustache',
 'Dali Moustache',
 'Imperial Moustache',
 'Hungarian Moustache',
 'Realistic Moustache',
 'Creative Moustache',
 'Realistic Beard',
 'Creative Beard',
 'Musketeer',
 'Imperial Beard',
 'Fu Manchu',
 'Alaskan Whaler',
 'Sideburns Natural',
 'Natural Goatee',
 'Verdi',
 'Garibaldi',
 'Full Beard with Styled Moustache',
 'Full Beard 0.1-15cm',
 'Full Beard 15.1-30cm',
 'Full Beard 30.1-45cm',
 'Full Beard 45.1-60+cm',
 'Freestyle Moustache',
 'Cheekbeard (Backenbart) Freestyle',
 'Sideburns Freestyle',
 'Partial Beard Freestyle',
 'Full Beard Freestyle']

all_categories is a list object with 27 items.<jupyter_text>Moustaches only At this point you can see that there are several moustache categories, but even more beard categories. Let's filter out the beards so you can focus on your upper lip decor. Since this data is pretty messy, we'll use a pre-processed dataset, which we can use to cleanly filter out those nasty chin and cheek hair categories. This will only require some simple `pandas` filtering, so we'll get to it and then view our remaining categories.<jupyter_code>import pandas as pd

world_beard_championship = pd.read_csv("https://raw.githubusercontent.com/beccarobins/movember/master/world_beard_championship.csv", 
                                       index_col = 0)
stache_df = world_beard_championship[world_beard_championship.index.str.contains("Moustache")==True].reset_index()
stache_df.rename(columns={'index':'category'}, inplace=True)
stache_df<jupyter_output><empty_output><jupyter_text>Bring it all together in a function Since you want your Movemember Moustache to be epic, you'll want to look at images of the winningest moustaches from the world championship held in Austin. Given that Movemenber is a month for awareness, you also want to give your friends the opportunity to participate. All your friends are `pip` savvy, so you've decided to wrap all this information into a very simple package to distribute on [PyPI](https://pypi.org/). Before all that though, you want to create a simple test function that you will use to choose your moustache by viewing each moustache and its description. The goal is for the function to print out descriptive information and then record your choice to generate the appropriate url to send you to in your next step.<jupyter_code># Create a custom function that generates a correctly formatted url
def choose_moustache(path=0):
    """This function violates every principle of a good function BUT your mates don't have time, they need to grow their mo NOW!"""
    import pandas as pd
    import webbrowser
    
    # Import dataset and clean
    world_beard_championship = pd.read_csv("https://raw.githubusercontent.com/beccarobins/movember/master/world_beard_championship.csv", 
                                           index_col = 0)
    stache_df = world_beard_championship[world_beard_championship.index.str.contains("Moustache")==True].reset_index()
    stache_df.rename(columns={'index':'category'}, inplace=True)
    
    # Choose your mo
    for i, moustache_type in stache_df.iterrows():
        print(i, moustache_type.category)
    path = int(input('\nChoose the index of your preferred moustache: '))
    
    # Clean up paths and strings
    url = "https://www.austinfacialhairclub.com/2017-wbmc-results"
    new_url = url.replace("2017-wbmc-results", "")
    test_String = stache_df.iloc[path]['category'].replace(" ", "-")
    test = new_url+'results-'+test_String
    site = test.lower()
    
    # Use webbrowser module when packaging up to send to friends
    # Use IPython in Jupypter
#     webbrowser.open(site)
    return site
# Call the choose_moustache() function with any path 0-8
my_moustache = choose_moustache()<jupyter_output>0 Natural Moustache
1 Dali Moustache
2 English Moustache
3 Imperial Moustache
4 Hungarian Moustache
5 Freestyle Moustache
6 Full Beard Styled Moustache
7 Realistic Moustache
8 Create Moustache

Choose the index of your preferred moustache: 5<jupyter_text>The unveiling Now it's finally time to check out those sweet sweet whiskers you've been dreaming of. We could do this with the `webbrowswer` module, which will open up a new tab. However, we want to stay in our notebook, so we'll use the `IPython` module to create an IFrame to view the webpage as output. Specifically, we'll be using the `.HTML()` method from the `display` module from `IPython`. First we'll need to whip up an IFrame, which stands for "inline frame." IFrames let you embed a website _within_ another website.<jupyter_code># Import the IPython module
import IPython

# Display the IFrame
IPython.display.IFrame(src=my_moustache, width=1000, height=400)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09827129730571563, 'fraction_numerical': 0.01689767948917614, 'mean_word_length': 4.7334821428571425, 'pattern_counts': {'":': 0, '<': 103, '<?xml version=': 0, '>': 102, 'https://': 16, 'lorem ipsum': 0, 'www.': 11, 'xml': 3}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '596706', 'license': 'MIT', 'n_tokens_mistral': 4136, 'n_tokens_neox': 3849, 'n_words': 1474, 'path': 'Movember Moustache Research.ipynb', 'repo_name': 'beccarobins/movember', 'size': '12886'}