<jupyter_start><jupyter_text>CSVs<jupyter_code>ls sources/<jupyter_output><empty_output><jupyter_text>[sources/pycon-2013-talks.csv](sources/pycon-2013-talks.csv) We could use Python's built-in [CSV module](https://pymotw.com/3/csv/).But, because we're learning [Pandas](https://pandas.pydata.org/) today anyway, let's use that.<jupyter_code>import pandas as pd
pd.read_csv('sources/pycon-2013-talks.csv')<jupyter_output><empty_output><jupyter_text>Wait a minute.  That top line is wrong.<jupyter_code>pd.read_csv?
header = ['type', 'title', 'fullname', 'firstname', 'lastname']
pd.read_csv('pycon-2013-talks.csv', header=header)
pd.read_csv('pycon-2013-talks.csv', names=header)<jupyter_output><empty_output><jupyter_text>Really hard to work with unless you assign a name to it.<jupyter_code>talks = pd.read_csv('pycon-2013-talks.csv', names=header)
talks
talks[0]
type(talks)<jupyter_output><empty_output><jupyter_text>AsideIf we want a plain-old Python list of dictionaries...<jupyter_code>talks.to_dict('records')<jupyter_output><empty_output><jupyter_text>... but we actually want to keep a Pandas DataFrame.<jupyter_code>talks.iloc[0]
talks.iloc[0].title
talks.iloc[0].title.upper()<jupyter_output><empty_output><jupyter_text>There's much more that's interesting/different about DataFrames, but more on that later.  Plain-python evaluation  Looping<jupyter_code>for row in talks.itertuples():
    print(row)
    print('\n')
for row in talks.itertuples():
    print(row.title)
mentions_python = 0
does_not_mention_python = 0
for row in talks.itertuples():
    if 'python' in row.title.lower():
        mentions_python += 1 
    else:
        does_not_mention_python += 1 
print(f'{mentions_python} titles mention Python; {does_not_mention_python} do not')<jupyter_output><empty_output><jupyter_text>Title length statistics<jupyter_code>lengths = [len(row.title) for row in talks.itertuples()]
lengths
average_len = sum(lengths) / len(lengths)
average_len
print(f'Average length {average_len}, min {min(lengths)}, max {max(lengths)}')<jupyter_output><empty_output><jupyter_text>Applying Pandas<jupyter_code>talks.columns
talks.title
talks.title.apply(len)
talks.title.apply(len).max()
pd_lengths = talks.title.apply(len)
print(f'Average length {pd_lengths.mean()}, min {pd_lengths.min()}, max {pd_lengths.max()}')
pd_lengths.v
print(f'Standard deviation {pd_lengths.std()}, variance {pd_lengths.var()}')
talks.title.value_counts()
titles = talks.title.value_counts()<jupyter_output><empty_output><jupyter_text>Filter<jupyter_code>titles[titles > 1]
## Pass your own functions
def ratio_vowels(phrase):
    "Ratio of a phrase's letters that are vowels."
    
    letters = [char for char in phrase if char.isalpha()]
    vowels = [letter for letter in letters if letter in ('a', 'e', 'i', 'o', 'u')]
    return len(vowels) / len(letters)
    
    
    
talks.title.apply(ratio_vowels)
talks['title_vowels'] = talks.title.apply(ratio_vowels)
talks['lastname_vowels'] = talks.lastname.apply(ratio_vowels)
5 / 0
import numpy as np
np.divide(5, 0)
def ratio_vowels(phrase):
    "Ratio of a phrase's letters that are vowels."
    
    letters = [char for char in phrase if char.isalpha()]
    vowels = [letter for letter in letters if letter in ('a', 'e', 'i', 'o', 'u')]
    return np.divide(len(vowels), len(letters))
    
talks['lastname_vowels'] = talks.lastname.apply(ratio_vowels)
talks[['lastname', 'lastname_vowels', 'title_vowels']]
talks.corr()<jupyter_output><empty_output><jupyter_text>Plot with [Altair](https://altair-viz.github.io/)<jupyter_code>import altair
altair.Chart(talks).mark_point().encode(
    x='lastname_vowels', y='title_vowels')<jupyter_output><empty_output><jupyter_text>Function on DataFrame<jupyter_code>talks['first_last_ratio'] = talks.apply(
    lambda talk: np.divide(len(talk.firstname), len(talk.lastname)),
    axis=1
    )

talks[['firstname', 'lastname', 'first_last_ratio']]
talks[talks.first_last_ratio == talks.first_last_ratio.max()]
talks.loc[talks.first_last_ratio.idxmax()]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1451851851851852, 'fraction_numerical': 0.00962962962962963, 'mean_word_length': 6.378870673952641, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '220588', 'license': 'MIT', 'n_tokens_mistral': 1572, 'n_tokens_neox': 1526, 'n_words': 277, 'path': 'notebooks/csv.ipynb', 'repo_name': 'catherinedevlin/harvest-wild-data-olf-2018', 'size': '4050'}