<jupyter_start><jupyter_text>_Lambda School Data Science_ Scrape and process dataObjectives- scrape and parse web pages- use list comprehensions- select rows and columns with pandasLinks-  [Automate the Boring Stuff with Python, Chapter 11](https://automatetheboringstuff.com/chapter11/)  - Requests  - Beautiful Soup- [Python List Comprehensions: Explained Visually](https://treyhunner.com/2015/12/python-list-comprehensions-now-in-color/)- [Pandas Cheat Sheet](https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf)  - Subset Observations (Rows)  - Subset Variables (Columns)- Python Data Science Handbook  - [Chapter 3.1](https://jakevdp.github.io/PythonDataScienceHandbook/03.01-introducing-pandas-objects.html), Introducing Pandas Objects  - [Chapter 3.2](https://jakevdp.github.io/PythonDataScienceHandbook/03.02-data-indexing-and-selection.html), Data Indexing and Selection  Scrape the titles of PyCon 2019 talks<jupyter_code>url = 'https://us.pycon.org/2019/schedule/talks/list/'
import bs4
import requests

result = requests.get(url)
type(result.text)
soup = bs4.BeautifulSoup(result.text)
len(soup.select('h2'))
soup.select('h2')
first = soup.select('h2')[0]
type(first.text)
first.text
text_only = first.text.strip()
text_only
last = soup.select('h2')[-1]
last.text.strip()

title_lst = []

for tag in soup.select('h2'):
  title = tag.text.strip()
  title_lst.append(title)
  
print(title_lst)
titles = [tag.text.strip()
         for tag in soup.select('h2')]
type(titles), len(titles)<jupyter_output><empty_output><jupyter_text>5 ways to look at long titlesLet's define a long title as greater than 80 characters  1. For Loop<jupyter_code>tit_lst = []

for title in titles:
  if len(title) > 80:
    tit_lst.append(title)

print(tit_lst)<jupyter_output>['¡Escuincla babosa!: Creating a telenovela script in three Python deep learning frameworks', 'Getting started with Deep Learning: Using Keras & Numpy to detect voice disorders', 'How to engage Python contributors in the long term? Tech is easy, people are hard.', 'Lessons learned from building a community of Python users among thousands of analysts', 'Life Is Better Painted Black, or: How to Stop Worrying and Embrace Auto-Formatting', 'One Engineer, an API, and an MVP: Or, how I spent one hour improving hiring data at my company.', 'Put down the deep learning: When not to use neural networks and what to do instead', 'Thinking Inside the Box: How Python Helped Us Adapt to An Existing Data Ingestion Pipeline']<jupyter_text>2. List Comprehension<jupyter_code>long_titles = [x 
              for x in titles if len(titles) > 80]<jupyter_output><empty_output><jupyter_text>3. Filter with named function<jupyter_code>def long(title):
  return len(title) > 80

long('Python is good')
    
list(filter(long, titles))
list(filter(lambda t: len(t) > 80, titles))<jupyter_output><empty_output><jupyter_text>4. Filter with anonymous function<jupyter_code>list(filter(lambda t: len(t) > 80, titles))<jupyter_output><empty_output><jupyter_text>5. Pandaspandas documentation: [Working with Text Data](https://pandas.pydata.org/pandas-docs/stable/text.html)<jupyter_code>import pandas as pd
pd.options.display.max_colwidth = 200
df = pd.DataFrame({'title': titles})
df.shape
df[ df['title'].str.len() > 80 ]
df['title'].str.len() > 80
condition = df['title'].str.len() > 80
df[condition]<jupyter_output><empty_output><jupyter_text>Make new dataframe columnspandas documentation: [apply](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.apply.html)<jupyter_code><jupyter_output><empty_output><jupyter_text>title length<jupyter_code>df['title length'] = df['title'].apply(len)
df.head()
df[ df['title length'] > 80 ]
df.loc[ df['title length'] > 80, 'title length']<jupyter_output><empty_output><jupyter_text>long title<jupyter_code>df['long title'] = df['title length'] > 80

df.shape
df.head()
df[ df['long title']==True]<jupyter_output><empty_output><jupyter_text>first letter<jupyter_code>df['first letter'] = df['title'].str[0]
df[df['first letter'] == 'P']
df[df['title'].str.startswith('P')]<jupyter_output><empty_output><jupyter_text>word countUsing [`textstat`](https://github.com/shivam5992/textstat)<jupyter_code>!pip install textstat
import textstat 
df['title word count'] = df['title'].apply(textstat.lexicon_count)
df.head()
df.shape
df[ df['title word count'] <= 3 ]<jupyter_output><empty_output><jupyter_text>Rename column`title length` --> `title character count`pandas documentation: [rename](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rename.html)<jupyter_code>df = df.rename(columns={'title length': 'title character count'})
df.head()<jupyter_output><empty_output><jupyter_text>Analyze the dataframe  Describepandas documentation: [describe](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html)<jupyter_code>df.describe(include='all')
df.describe(exclude='number')<jupyter_output><empty_output><jupyter_text>Sort valuespandas documentation: [sort_values](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort_values.html) Five shortest titles, by character count<jupyter_code>df.sort_values(by='title character count').head(5)<jupyter_output><empty_output><jupyter_text>Titles sorted reverse alphabetically<jupyter_code>df.sort_values(by='first letter', ascending=False).head()<jupyter_output><empty_output><jupyter_text>Get value countspandas documentation: [value_counts](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.value_counts.html) Frequency counts of first letters<jupyter_code>df['first letter'].value_counts()<jupyter_output><empty_output><jupyter_text>Percentage of talks with long titles<jupyter_code>df['long title'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Plotpandas documentation: [Visualization](https://pandas.pydata.org/pandas-docs/stable/visualization.html) Top 5 most frequent first letters<jupyter_code>(df['first letter']
.value_counts()
.head(5)
.plot
.barh(color='green',
     title='Top 5 most frequent first letters, PyCon 2019 talks'));<jupyter_output><empty_output><jupyter_text>Histogram of title lengths, in characters<jupyter_code>df['title character count'].plot.hist(color='orange', title = 'Distribution of Title Length in Characters');<jupyter_output><empty_output><jupyter_text>Assignment**Scrape** the talk descriptions. Hint: `soup.select('.presentation-description')`**Make** new columns in the dataframe:- description- description character count- description word count**Describe** all the dataframe's columns. What's the average description word count? The minimum? The maximum?**Answer** the question: Which descriptions could fit in a tweet? Stretch Challenge**Make** another new column in the dataframe:- description grade level (you can use [this `textstat` function](https://github.com/shivam5992/textstatthe-flesch-kincaid-grade-level) to get the Flesh-Kincaid grade level)**Answer** the question: What's the distribution of grade levels? Plot a histogram.**Be aware** that [Textstat has issues when sentences aren't separated by spaces](https://github.com/shivam5992/textstat/issues/77issuecomment-453734048). (A Lambda School Data Science student helped identify this issue, and emailed with the developer.) Also, [BeautifulSoup doesn't separate paragraph tags with spaces](https://bugs.launchpad.net/beautifulsoup/+bug/1768330).So, you may get some inaccurate or surprising grade level estimates here. Don't worry, that's ok — but optionally, can you do anything to try improving the grade level estimates?   **Objectives**1.  Scape the talk descriptions2. Make new columns in the dataframe:        a. description    b. description character count    c. description word count    3.  Describe all the dataframe's colums.    a. Average description word count    b. Minimum    c. Maximum    4.  Which descriptions could fit in a tweet?      Objective 1<jupyter_code># Isolate the html container that contains talk descriptions

first = soup.select('div.presentation-description')[0]
first
#Eliminate all characters not related to the talk description in our string
first.text.strip()
# Repeat for all descriptions in our list
talk_description = [tag.text.strip()
         for tag in soup.select('div.presentation-description')]<jupyter_output><empty_output><jupyter_text>Objective 2<jupyter_code># Create pandas database out of our cleaned list
df = pd.DataFrame({'Description': talk_description})
df.shape
# Description column created
df.head()
# Description Character Count created
df['Description Character Count'] = df['Description'].apply(len)
df.head()
# Description Word Count created
df['Description Word Count'] = df['Description'].apply(textstat.lexicon_count)
df.head()<jupyter_output><empty_output><jupyter_text>**Objective 3**The average description word count is 130.82The minimum words used is 20The maximum word used is 421<jupyter_code>df.describe(include='all')<jupyter_output><empty_output><jupyter_text>Objective 4<jupyter_code># Creating Tweet-able column with boolean values to determine whether the talk description could fit within twitters 280 character tweet limit.
df['Tweet-able?'] = df['Description Character Count'] < 280
df.head()
# Created a subset variable to determine which talk descriptions could fit in a single tweet.
df[ df['Tweet-able?']]
# Confirmed above finding
df['Tweet-able?'].value_counts()<jupyter_output><empty_output><jupyter_text>Stretch Challenge<jupyter_code># Grade level function on a string of text

def scores(x):
  
  for i in x:
    text_score = textstat.flesch_kincaid_grade(x)
    
  return text_score
# Applies score function on dataframe

df['Description Grade Level'] = df['Description'].apply(scores)
df.head()
import matplotlib.pyplot as plt

plt.figure(figsize=(20,10))

plt.xlabel('Flesch Kincaid Scores')
plt.title('Distribution of Description Grade Levels for PyCon 2019 Talks',size=15)

plt.hist(df['Description Grade Level'], bins=20);
# We will make a seperate dataframe with all descriptions above 50 to analyze

unreadable_desc = df[ df['Description Grade Level'] > 60]
unreadable_lst = list(unreadable_desc['Description'])
# Create list of three highest scores
unreadable_lst[0:3]
# I have inputted all three descriptions into 'http://www.hemingwayapp.com/' for further analysis<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1200999231360492, 'fraction_numerical': 0.01498847040737894, 'mean_word_length': 6.515523465703971, 'pattern_counts': {'":': 0, '<': 98, '<?xml version=': 0, '>': 109, 'https://': 17, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '582614', 'license': 'MIT', 'n_tokens_mistral': 3461, 'n_tokens_neox': 3246, 'n_words': 859, 'path': 'Connor_Heraty_LS_DS3_121_Scrape_and_process_data_LIVE_LESSON.ipynb', 'repo_name': 'connorpheraty/DS-Unit-1-Sprint-2-Data-Wrangling', 'size': '10408'}