<jupyter_start><jupyter_text>Using Python BeautifulSoup to scrape DataCamp Tutorials & Analyze[Source](https://www.datacamp.com/community/tutorials/tutorial-python-beautifulsoup-datacamp-tutorials)In this tutorial, we are going to scrape the tutorials section of the DataCamp website and try to get some insights.* Most contributing authors* Timeline of contributors (How it all started!)* Comparing upvotes vs. number of articles publishedBefore that, the website will be scraped using python's BeautifulSoup package.To understand the page structure, Chrome browser developer tools will need to be used. This is done to identify the Classes that will be searched to get the required information.The following information will be gathered from the page:* Author* Publish Date* Title* Description* Up Votes  Importing LibrariesWe'll start by importing the necessary libraries as follows:<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import re
import time
from datetime import datetime
import matplotlib.dates as mdates
import matplotlib.ticker as ticker
from urllib.request import urlopen
from bs4 import BeautifulSoup<jupyter_output><empty_output><jupyter_text>Determining Pages to be ScrapedA sample URL that we're going to loop and scrape is the following https://www.datacamp.com/community/tutorials?page=2. As we can see, the page=2 argument changes for each page. In order to loop through all the pages to get the necessary dataset, we need to find out the number of pages.The following lines of code do just that.<jupyter_code>url = "https://www.datacamp.com/community/tutorials"
html = urlopen(url)
soup = BeautifulSoup(html, 'html')

pages = [i.text for i in soup.find_all('a') if 'community/tutorials?page=' in str(i)]
lastpage = pages[-1]
print(lastpage)<jupyter_output>22<jupyter_text>The illustration is as follows:* Specified the url to a variable* Opened the url using urlopen which was imported earlier* Scraped the specified page and assigned it to soup variable* Identified all hyperlinks on the page using list comprehension and filtered for those having community/tutorials?page= in it* The text value of the last found url is the last page that needs to be scrapedWe proceed by declaring list variables that will hold the scraped values for the columns we intend as mentioned earlier<jupyter_code>description=[]
upvote=[]
author=[]
publishdate=[]
title=[]<jupyter_output><empty_output><jupyter_text>Doing the Actual ScrapeNow that we know how many pages we need to scrape and have declared our variables, we will now use a for loop and go through each page one by one to get our fields of interest as shown below. Note that we will end up having list of lists for each column of interest, but later we will be flattening the list further so that it can be used for Data Frames.<jupyter_code>for cp in np.arange(1,int(lastpage)+1):
    url = "https://www.datacamp.com/community/tutorials?page=" + str(cp)
    html = urlopen(url)
    soup = BeautifulSoup(html, 'html')
    description.append([i.text for i in soup.find_all(class_='jsx-379356511 blocText description')])
#     upvote.append([i.text for i in soup.find_all(class_='jsx-4192737526 voted')])
    upvote.append([i.text for i in soup.find_all(class_='jsx-1972554161 voted')])
    author.append([i.text for i in soup.find_all(class_='jsx-566588255 name')])
    publishdate.append([i.text for i in soup.find_all(class_='jsx-566588255 date')])
    title.append([i.text for i in soup.find_all(class_='jsx-379356511 blue')])
    time.sleep(3)
print ("Done!")<jupyter_output>Done!<jupyter_text>Here is what happened in the above code segment* Set the url to a variable* Opened the url using urlopen which was imported earlier* Scraped the specified page and assigned it to soup variable* Identified and extracted values for Description, Up Vote, Author, Publish Date, Title by using their relevant class names. These class names were found using Developer Tools* The time function has been used to be easy on the website this time :) 3.b Display the soup object to visually interrogate  Flattening List of ListsSince the values we got are list of lists, they will now be flatted using the following code segment:<jupyter_code>descriptionflat = [y for x in description for y in x]
len(descriptionflat)
upvoteflat = [y for x in upvote for y in x]
len(upvoteflat)
authorflat = [y for x in author for y in x]
len(authorflat)
publishdateflat = [y for x in publishdate for y in x]
len(publishdateflat)
titleflat = [y for x in title for y in x]
len(titleflat)
publishdateformatted = [datetime.strptime(re.sub('rd, ', ', ', 
                                          re.sub('st, ', ', ', 
                                          re.sub('nd, ', ', ', 
                                          re.sub('th, ',', ',a)))), "%B %d, %Y") for a in publishdateflat]
len(publishdateformatted)<jupyter_output><empty_output><jupyter_text>The last statement in the cell above converts the date values (which are currently in String Format) to DateTime.  Making a Data Frame and Saving as CSV FileThe lists will now be grouped into a dictionary, and a data frame will be created for further analysis. The last command saves the data frame to a CSV file so that it can be used later on.<jupyter_code>cdata = {"author":authorflat,
         "publishdate":publishdateformatted,
         "title":titleflat,
         "description":descriptionflat,
         "upvote":upvoteflat}

df = pd.DataFrame(data=cdata)
df.to_csv("datacamp.csv", header=True, index=False)<jupyter_output><empty_output><jupyter_text>Reading CSV FileNow we are attempting to read into the collected dataset from the CSV file we just created.<jupyter_code>data = pd.read_csv("datacamp.csv", parse_dates=["publishdate"], infer_datetime_format=True)
data.shape<jupyter_output><empty_output><jupyter_text>The above command tells us we're dealing with a dataset of 176 rows and 5 columns.<jupyter_code>data.head()<jupyter_output><empty_output><jupyter_text>Showing the first 5 rows of the dataset above using the head function.<jupyter_code>data['publishyymm'] = data['publishdate'].dt.strftime("%Y-%b")
data["posts"] = 1<jupyter_output><empty_output><jupyter_text>* The first line in the above code section creates a new column with the publish date formatted as a Year-Month format.* The second line assigns value 1 to a new column posts being used later.  Tutorials Count by TimeHere we will organize the count of Tutorials over Timeline of Year and Month:<jupyter_code>datacamp.groupby([datacamp['publishdate'].dt.year, datacamp['publishdate'].dt.month]).size().plot(kind='bar', figsize=(15,7), color='b')<jupyter_output><empty_output><jupyter_text>Since 2017 JanuarySince the duration from 2013 till 2016 represents very few posts, we will be ignoring them from now and considering posts starting Jan 2017. A filter will be applied for that as shown below<jupyter_code>data[data["publishdate"]>='2017-01-01'] \
    .sort_values(by="publishdate", ascending=True) \
    .groupby([data['publishyymm']],sort=False) \
    .size() \
    .plot(kind='bar', figsize=(15,7), color='b')<jupyter_output><empty_output><jupyter_text>Tutorials have paced up in 2018 with especially starting March onwards with a consistent upwards pace. Since this data was pulled in mid-August and almost crossing the middle line of July's tutorials count, we might be having August as the month with highest posts so far this year!  Top Authors GraphWhile we're pacing up on all these tutorials, who have been contributing on them along the way? Here we go with a simple bar chart highlighting this very fact.<jupyter_code>data[data["publishdate"]>='2017-01-01']["author"] \
    .value_counts(sort=True, ascending=False)[:10] \
    .plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Top Authors ListLet's build a list of these as well while we are at it. We'll be using this soon below:<jupyter_code>topauthors = data[data["publishdate"]>='2017-01-01']["author"] \
    .value_counts(sort=True, ascending=False)[:10] \
    .index<jupyter_output><empty_output><jupyter_text>This is what happened in the code section above.* Limited Result set to Tutorials since 2017 January* Select only Author field* Aggregated results using the value_counts function* Sorted the result set in descending order and limited it to the first 10 rows  Tutorials Paced over TimelineNow what we're going to focus on is since when and at what pace have these top 10 contributors been posting tutorials. For this, we will be using the list we just created along with some transformations to come up with a stacked bar chart that we need.<jupyter_code>dh = data[data["publishdate"]>='2017-01-01'] \
    .sort_values(by="publishdate", ascending=True) \
#     .set_index(["publishdate"])
dh.head()
dh["publishdateone"] = pd.to_datetime(dh.publishdate.astype(str).str[0:7]+'-01')<jupyter_output><empty_output><jupyter_text>This is what happened in the code section above.* Limited Result set to Tutorials since 2017 January* Sorted by Publish Date* Making Publish Date as the Index columnNow that we're going to visualize using a stacked bar chart, the data set will now be pivoted by having the date field as the Index, Posts as values which are to be aggregated, and Authors as columns.<jupyter_code>dhp = dh[dh["author"].isin(topauthors)] \
    .pivot_table(index="publishdateone",values="posts",columns="author", aggfunc=np.sum)
fig, ax = plt.subplots(figsize=(15,7))
dhp.plot(ax=ax, kind='bar', stacked=True)
ticklabels = [item.strftime('%Y %b') for item in dhp.index]
ax.xaxis.set_major_formatter(ticker.FixedFormatter(ticklabels))<jupyter_output><empty_output><jupyter_text>Here is what we possibly get out of the above Chart along with considering the earlier visualization of Tutorial Counts by Authors.  Looking at Upvotes vs. TutorialsWhen a reader likes a tutorial, an upvote is signaled respectively.Let's see who's managed to get a good amount of upvotes vs. number of tutorials they have posted! We'll be considering the top 10 contributors in this case as well. This will be done by using a scatter plot.<jupyter_code>upvotes = dh[dh["author"].isin(topauthors)] \
            .groupby(['author'], as_index=False) \
            .agg({'posts':"sum", 'upvote': "sum"})
sns.lmplot('posts',
           'upvote',
           data=upvotes,
           fit_reg=False,
           hue="author",
           scatter_kws={"marker": "D", "s": 100})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09034771043131294, 'fraction_numerical': 0.015105453163594908, 'mean_word_length': 4.940744920993228, 'pattern_counts': {'":': 7, '<': 63, '<?xml version=': 0, '>': 67, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '423774', 'license': 'MIT', 'n_tokens_mistral': 3216, 'n_tokens_neox': 3014, 'n_words': 1206, 'path': '_site/lectures/Week 06 - Web Scraping/04.b - Even More Web Scraping with BeautifulSoup.ipynb', 'repo_name': 'BrianKolowitz/data-focused-python', 'size': '10526'}