<jupyter_start><jupyter_text>_Lambda School Data Science_ Scrape and process dataObjectives- scrape and parse web pages- use list comprehensions- select rows and columns with pandasLinks-  [Automate the Boring Stuff with Python, Chapter 11](https://automatetheboringstuff.com/chapter11/)  - Requests  - Beautiful Soup- [Python List Comprehensions: Explained Visually](https://treyhunner.com/2015/12/python-list-comprehensions-now-in-color/)- [Pandas Cheat Sheet](https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf)  - Subset Observations (Rows)  - Subset Variables (Columns)- Python Data Science Handbook  - [Chapter 3.1](https://jakevdp.github.io/PythonDataScienceHandbook/03.01-introducing-pandas-objects.html), Introducing Pandas Objects  - [Chapter 3.2](https://jakevdp.github.io/PythonDataScienceHandbook/03.02-data-indexing-and-selection.html), Data Indexing and Selection  Scrape the titles of PyCon 2018 talks<jupyter_code>url = 'https://us.pycon.org/2018/schedule/talks/list/'
import bs4
import requests
result = requests.get(url)
type(result.text) #confirms str
result #successful
type(result)
type(result.text) #confirms str
soup = bs4.BeautifulSoup(result.text)
soup.select('h2')
#print(soup) #dont do this
type(soup.select('h2'))
len(soup.select('h2')) #95 talks
first = soup.select('h2')[0] #first talk
first
first.text #cleaner but has problems
first.text.strip() #defaults to strip white spaces and newline parse
last = soup.select('h2')[-1] #last talk
print(type(last)) #tag
print(type(soup.select('h2')[-1:])) #list, not useful
#our complete list of titles loop style
titles = []
for tag in soup.select('h2'):
  tag.text.strip()
  titles.append(titles)
#list comp style
titles = [tag.text.strip() for tag in soup.select('h2')]
type(titles), len(titles)
titles[0], titles[-1]<jupyter_output><empty_output><jupyter_text>5 ways to look at long titlesLet's define a long title as greater than 80 characters  1. For Loop<jupyter_code>#nonfunctional compared to other methods
long_titles = []
for title in titles:
  if len(title) > 80:
    long_titles.append(title)
len(long_titles)<jupyter_output><empty_output><jupyter_text>2. List Comprehension<jupyter_code>long_titles = [title for title in titles if len(title) > 80]
len(long_titles)<jupyter_output><empty_output><jupyter_text>3. Filter with named function<jupyter_code>def long(title):
  return len(title) > 80
long('Hello') #False
list(filter(long, titles)) #filter by itself is an object<jupyter_output><empty_output><jupyter_text>4. Filter with anonymous function<jupyter_code>list(filter(lambda t: len(t)>80, titles))<jupyter_output><empty_output><jupyter_text>5. Pandaspandas documentation: [Working with Text Data](https://pandas.pydata.org/pandas-docs/stable/text.html)<jupyter_code>import pandas as pd
pd.options.display.max_colwidth = 200
df = pd.DataFrame({'title': titles})
df[df.title.str.len() > 80]
condition = df.title.str.len() > 80 #calls a Series of booleans
df[condition]
df.title.str.len() #calls a Series of ints equal to str.len<jupyter_output><empty_output><jupyter_text>Make new dataframe columnspandas documentation: [apply](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.apply.html)  title length<jupyter_code>df['title_length'] = df.title.apply(len)
df.head()
df.loc[df['title_length']>80, 'title length']]<jupyter_output><empty_output><jupyter_text>long title<jupyter_code>#alt
df[df['long_title' == True]]<jupyter_output><empty_output><jupyter_text>first letter<jupyter_code>df['first_letter'] = df.title.str[0]
df.head()

df[df['first_letter'] == 'P']<jupyter_output><empty_output><jupyter_text>word countUsing [`textstat`](https://github.com/shivam5992/textstat)<jupyter_code>!pip install textstat
import textstat
first = df.title.values[0]
last = df.title.values[-1]
first, last
textstat.lexicon_count(first), textstat.lexicon_count(last)
df['title_word_count'] = df.title.apply(textstat.lexicon_count)
df[df['title_word_count'] <= 3]<jupyter_output><empty_output><jupyter_text>Rename column`title length` --> `title character count`pandas documentation: [rename](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rename.html)<jupyter_code>df['title_character_count'] = df['title_length']
df.head()
#alt
df = df.rename(columns={'title_legth': 'title_cha'})
df.columns<jupyter_output><empty_output><jupyter_text>Analyze the dataframe  Describepandas documentation: [describe](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html)<jupyter_code>import numpy as np
df.describe(exclude=np.number)<jupyter_output><empty_output><jupyter_text>Sort valuespandas documentation: [sort_values](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort_values.html)<jupyter_code>df.sort_values(by='title_character_count').head()<jupyter_output><empty_output><jupyter_text>Five shortest titles, by character count<jupyter_code>df['first_letter'] = df['first_letter'].str.upper()
df.sort_values(by='first_letter', ascending=False).head()<jupyter_output><empty_output><jupyter_text>Titles sorted reverse alphabetically<jupyter_code><jupyter_output><empty_output><jupyter_text>Get value countspandas documentation: [value_counts](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.value_counts.html) Frequency counts of first letters<jupyter_code>df['first_letter'].value_counts().sort_index()
df[df['first_letter']=='""']<jupyter_output><empty_output><jupyter_text>Percentage of talks with long titles<jupyter_code>df['long_title'].value_counts()/len(df)
#alt
df['long_title'].value_counts(normalize=True)*100<jupyter_output><empty_output><jupyter_text>Plotpandas documentation: [Visualization](https://pandas.pydata.org/pandas-docs/stable/visualization.html) Top 5 most frequent first letters<jupyter_code>%matplotlib inline
(df['first_letter'].value_counts().head()
 .plot.barh(color='grey', title='Top 5 Most Frequent First Letters'));
#suppress subplot object with ;<jupyter_output><empty_output><jupyter_text>Histogram of title lengths, in characters<jupyter_code>title = 'Dist of Character Count'
df['title_character_count'].plot.hist(title=title);<jupyter_output><empty_output><jupyter_text>Assignment**Scrape** the talk descriptions. Hint: `soup.select('.presentation-description')`**Make** new columns in the dataframe:- description- description character count- description word count- description grade level (use [this `textstat` function](https://github.com/shivam5992/textstatthe-flesch-kincaid-grade-level) to get the Flesh-Kincaid grade level)**Describe** all the dataframe's columns. What's the average description word count? The minimum? The maximum?**Answer** these questions:- Which descriptions could fit in a tweet?- What's the distribution of grade levels? Plot a histogram.<jupyter_code>description = [tag.text.strip() for tag in soup.select('.presentation-description')]
df['description'] = description
df.head(2)
df['desc_character_count'] = df.description.apply(len)
df['desc_word_count'] = df.description.apply(textstat.lexicon_count)
df.head(2)
textlist = []
indexno = 0
for index in df['description']:
  x = textstat.flesch_kincaid_grade(df['description'][indexno])
  textlist.append(x)
  indexno = indexno+1
df['desc_gradelv'] = textlist
df.head(2)
import numpy as np
df.describe(exclude=np.number)
df.describe(include=np.number)
#min desc word count - 35
#max desc word count - 436
#mean desc word count - 134.6
df = df[df['desc_character_count'] <= 280]
df.head()
#df includes only descriptions that would be tweetable
%matplotlib inline
title = 'Distribution of Reading Levels'
ax = df['desc_gradelv'].plot.hist(title=title)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13751132979412145, 'fraction_numerical': 0.013725236307134533, 'mean_word_length': 7.698198198198198, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 89, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '652566', 'license': 'MIT', 'n_tokens_mistral': 2762, 'n_tokens_neox': 2638, 'n_words': 510, 'path': 'module1-scrape-and-process-data/LS_DS_121_Scrape_and_process_data.ipynb', 'repo_name': 'trista-paul/DS-Unit-1-Sprint-2-Data-Wrangling', 'size': '7723'}