<jupyter_start><jupyter_text>Module 2 - LecturesStarting with **Loading Data** [lecture](https://courses.edx.org/courses/course-v1:Microsoft+DAT210x+4T2017/courseware/b7aa9093-0ee1-9e62-4188-6f9d25a2ce85/109598e7-56e4-2f20-99a1-f61f7c6dd93d/?child=first).<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Different read methods take different arguments and work differently.(Sample JSON [website](http://jsonstudio.com/resources/))<jupyter_code>df1 = pd.read_excel('Module2 - Lab1.xls', 'Features')
dfs = pd.read_html('https://www.textfixer.com/tutorials/css-tables.php')
len(dfs)
dfsub1 = dfs[0]
dfsub1<jupyter_output><empty_output><jupyter_text>This may not work with current setup.<jupyter_code># # Doesn't work with current setup - Not worth it to debug at this time
# from sqlalchemy import create_engine
# engine = create_engine('sqlite:///:memory:')
# df = pd.read_sql_table('my_table', engine, columns=['ColA', 'ColB'])
df = pd.read_csv('Datasets/tutorial.csv', sep=',')
df
len(df.columns)
df.columns = ['new', 'column', 'header', 'labels']
df
df1.head(7)
df1.tail()
df.describe()
df1.index
df1.dtypes<jupyter_output><empty_output><jupyter_text>Next sectionMoving on to **Slicin** section, which is about indexing, and uses a provided csv file for working on.<jupyter_code># This command resets the variable space. Uncomment, then run. Notebook will ask for confirmation.
# %reset
df = pd.read_csv('Datasets/direct_marketing.csv')
df
df.recency
df['recency']
df[['recency']]
df.loc[:, 'recency']
df.loc[:, ['recency']]
df.iloc[:, 0]
df.iloc[:, [0]]
# These return a series object (equivalent output)
df.recency
df['recency']
df.loc[:, 'recency']
df.iloc[:, 0]
# These return a dataframe (equivalent output)
df[['recency']]
df.loc[:, ['recency']]
df.iloc[:, [0]]
df[0:2]
df.iloc[0:2, :]<jupyter_output><empty_output><jupyter_text>Next sectionMoving on to **Boolean Indexing** section, which is about indexing using boolean operators and comparisons.<jupyter_code>df.recency < 7
df[df.recency < 7]
df[ (df.recency < 7) & (df.newbie == 0) ]
df[df.recency < 7] = -100
df<jupyter_output><empty_output><jupyter_text>Feature Representation([lecture](https://courses.edx.org/courses/course-v1:Microsoft+DAT210x+4T2017/courseware/b7aa9093-0ee1-9e62-4188-6f9d25a2ce85/ce1c9548-56c6-5421-5541-94aec8434947/?activate_block_id=block-v1%3AMicrosoft%2BDAT210x%2B4T2017%2Btype%40sequential%2Bblock%40ce1c9548-56c6-5421-5541-94aec8434947))<jupyter_code>import pandas as pd
ordered_satisfaction = ['Very Unhappy', 'Unhappy', 'Neutral', 'Happy', 'Very Happy']
df = pd.DataFrame({'satisfaction': ['Mad', 'Happy', 'Unhappy', 'Neutral']})
df
df['satisfaction2'] = df.satisfaction.astype(dtype='category',
                                            ordered = True,
                                            categories = ordered_satisfaction
                                            )
df<jupyter_output><empty_output><jupyter_text>Note that the undefined (in ordered_satisfaction) value, 'Mad' created a NaN in the satisfaction2 column.<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>Pandas implements a category as regular Python list.Convert to numeric fashion, with the category codes from the index in ordered_satisfaction.<jupyter_code>df.satisfaction2 = df.satisfaction2.cat.codes
df.satisfaction2
df<jupyter_output><empty_output><jupyter_text>Note the NaN value is encoded now as -1.<jupyter_code>df = pd.DataFrame({
    'vertebrates': [
        'Bird',
        'Bird',
        'Mammal',
        'Fish',
        'Amphibian',
        'Reptile',
        'Mammal'
    ]
})
df
df['vertebrates2'] = df.vertebrates.astype("category").cat.codes
df<jupyter_output><empty_output><jupyter_text>(Can erroneously/systematically introduce data relationships. Here, e.g., it seems like a Bird is 2 integer value difference from a Mammal, even though this is just related to the Alphabetic sort used to set up the typed category codes.)<jupyter_code>df2 = pd.get_dummies(df, columns=['vertebrates'])
df2<jupyter_output><empty_output><jupyter_text>(This is potentially more nuanced because it doesn't introduce the implied relationship that using `.astype("category").cat.codes` did above. It takes up more "space" or more features, but avoids introducing implied relationships. Perhaps think twice for data sets that have **many** categorical features.)  Feature Representation - Continued<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    "Authman ran faster than Harry because Authman is real and Harry is fictional.",
    "Authman and Harry ran faster then faster and faster.",
    "W00t."
]
bow = CountVectorizer()
X = bow.fit_transform(corpus) # Produces a Sparse Matrix
X
bow.get_feature_names()
X.toarray()<jupyter_output><empty_output><jupyter_text>(Bag of Words method for featurizing text counts word frequency, binning text frequency content in a chunk of text.) (Moving on to Image Features)<jupyter_code>from scipy import misc
img = misc.imread('importance.png')
type(img)
img.shape
img.dtype
img = img[::2, ::2]
img.shape
img = (img/255.).reshape(-1, 4)
red = img[:,0]
green = img[:,1]
blue = img[:,2]
gray = 0.299*red + 0.587*green + 0.114*blue
img.shape
gray.shape<jupyter_output><empty_output><jupyter_text>Audio Encoding<jupyter_code>import scipy.io.wavfile as wavfile
sample_rate, audio_data = wavfile.read('recording.wav')
sample_rate
audio_data<jupyter_output><empty_output><jupyter_text>Data Wrangling<jupyter_code>import pandas as pd
import numpy as np
df = pd.read_csv('Datasets/direct_marketing.csv', sep=',')
df
df.dtypes<jupyter_output><empty_output><jupyter_text>Fixing Dates<jupyter_code># No Date column in our example.
# This raises errors when there are conversion issues
# df.Date = pd.to_datetype(df.Date, errors='raise')
# df
# This attempts to coerce, and if fails, inserts NaT (Not a Type) into
# the data where there are conversion issues.
# df.Date = pd.to_datetime(df.Date, errors='coerce')
# df<jupyter_output><empty_output><jupyter_text>Fixing Names<jupyter_code>None == None
np.nan == None
df.zip_code == np.nan
df.zip_code.isnull()
df.zip_code.notnull()
df.zip_code.str.contains(' ')
# Using wrong dataframe/data set for this, so commenting it out but keeping for posterity.
# selector = df.FirstName.str.contains(' ') & df.LastName.isnull()
# selector
# df.FirstName[selector].apply(lambda x: x.split(' ')[1])
# This generates warnings, but works. It's not recommended, however.
# df.LastName[selector] = df.FirstName[selector].apply(lambda x: x.split(' ')[1])
# df.FirstName[selector] = df.FirstName[selector].apply(lambda x: x.split(' ')[0])

# Pandas prefers you do it this way:
# df.loc[selector, 'LastName'] = df.FirstName[selector].apply(lambda x: x.split(' ')[1])
# df.loc[selector, 'FirstName'] = df.FirstName[selector].apply(lambda x: x.split(' ')[0])
df.dtypes
df.zip_code
df[['zip_code']]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13544668587896252, 'fraction_numerical': 0.03126801152737752, 'mean_word_length': 5.432808155699722, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 65, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '27176', 'license': 'MIT', 'n_tokens_mistral': 2557, 'n_tokens_neox': 2416, 'n_words': 567, 'path': 'Module2/Module 2 - Lectures.ipynb', 'repo_name': 'tsaomao/edX--MSDS-Curriculum--DAT210x', 'size': '6940'}