<jupyter_start><jupyter_text>Introduction to Data Science with `Python` In this example, we are going cover the very basics of data science in `Python`. Throughout this course, you will be introduced to many packages used by data scientists that make it easier to manipulate, visualize and model data. Today we will be working with one of the most popular Python data science packages, **`pandas`**. Before we begin doing some of the more complicated aspects of data science, it is important to get our feet wet with manipulating data. Now, it is impossible to cover all aspects of data manipulation because the scenarios are endless. It is the job of the data scientist to figure out the most creative and effective means by which to manipulate data. The purpose of this module is to get you comfortable with the tools and provide you with the grammar of data manipulation so that you can begin to string together your own data pipeline to meet your data science needs.**Python Reference Links:**  * [`Python` Standard Library](https://docs.python.org/3.4/library/)  * [`Python` Language Reference](https://docs.python.org/3.4/reference/)    * [`pandas`](http://pandas.pydata.org/pandas-docs/version/0.19.0/)  -----  What we'll cover... - [Reading in Data](reading)- [Soft Introduction to `pandas`](pandas)- [Data Filtering](filter)- [Data Sorting](sort)- [To Come...](future)----We will begin today by reading in a dataset of baby name populariy in the U.S.   Reading In the Data<jupyter_code>import os
print(os.getcwd())
file = open('../../datasets/baby-names/NationalNames1.csv', 'r')
data = file.read()
print(repr(data[0:101]))

Above you were introduced to a function that you may be unfamiliar with, `repr` (short for "*representation*"). This returns the output in string format without escaping some of the special characters that we need to be aware of. Take for example the `\n`, known as a newline character, which deliniates between rows in the original csv file. 

Having all of your data contained in a single string is less than desirable, particularly if you want do anything meaningful with it.  Let's change this so that instead of seeing a single string of data like you did above we see something more familiar to us.
with open('../../datasets/baby-names/NationalNames1.csv', 'r') as file:
    data = file.read()

    data_lists = data.split("\n")

    list_of_lists = []
    for line in data_lists:
        row = line.split(',')
        list_of_lists.append(row)

print(list_of_lists[0:10])<jupyter_output>[['Id', 'Name', 'Year', 'Gender', 'Count'], ['2', 'Anna', '1880', 'F', '2604'], ['3', 'Emma', '1880', 'F', '2003'], ['6', 'Margaret', '1880', 'F', '1578'], ['10', 'Sarah', '1880', 'F', '1288'], ['11', 'Annie', '1880', 'F', '1258'], ['14', 'Florence', '1880', 'F', '1063'], ['26', 'Julia', '1880', 'F', '783'], ['28', 'Edith', '1880', 'F', '768'], ['30', 'Rose', '1880', 'F', '700']]<jupyter_text>So we have, yet again, run into something you may think looks strange. The `with open(...)` statement above just executes everything in the indented block. In this case, it opens the file `NationalNames1.csv` and then reads it. Since this is read in a single string, we need to `split()` the string on the `\n` character that we discussed above. This puts it in a format of different strings, or what may look like rows of data. However, we'll want to access different values within the rows, so in order to do that we need to split the rows by their delimiter. In this case, commas separate values, which is appropriate given the original file extension name `.csv`, which stands for "comma separat*ed* values". What this returns is a list of lists, which, believe it or not, makes the data easier to manipulate.But even this seems a bit complicated. If you are like me, simpler is generally preferred. Remember, packages in Python include different functions, which are made to aid you in your programming endeavors. Such a packages exists for reading in a `.csv` file, creatively named **`csv`**.<jupyter_code>import csv

data_list = list(csv.reader(open('../../datasets/baby-names/NationalNames1.csv'),  delimiter=','))<jupyter_output><empty_output><jupyter_text>See how easy that was and only one line of code! Okay, the line is jam-packed with functions inside functions, but it's nothing that we haven't seen before. The code block below breaks each part out into a format that we have seen before. ```python    with open('../../../datasets/baby-names/NationalNames1.csv', 'r') as file:        data = csv.reader(file)        data_list = list(data)```The output of the function `csv.reader()` is non-subscriptable, meaning that we can't slice our data by indexes. Now, since each line is already a list, we just have to make the data a list of lists, which is as simple as calling the `list()` function on the `data` variable.  List of ListsWhy are we so obsessed with rendering the data in a list of lists? Why can't we just split the data by rows, like the code block below, and then call it a day? ```pythonwith open('../../../datasets/baby-names/NationalNames1.csv', 'r') as file:    data = file.read()    data_lists = data.split("\n")```This is tempting, but all it gives us are separate rows of data, which are not all that interesting if you can't access their values and compare them to other rows within the dataset. Think of this list of lists as a spreadsheet where you can access different cells individually. Some of you may know this as a matrix. It's not enough to just have access to individual rows; we also need access to columns or subsets of the data in order for analysis.  A list of lists isn't the only way to manipulate data (later on we will be discussing the more friendly and flexible concept of a data frame), but it is a good way to begin understanding how to interact with the data. The variable, `data_list` has a hair over 600,000 rows at the moment. Having access to this much data is nice, but let's scale it back a bit for quicker computations.  We're are going to take a subset of the original data (only 300 rows) and remove the header, as this is really just metadata used to aid humans in keeping things orderly.From the subset, we are going to transform the column that contains the name `count`s into floats (we could also do integers if we wanted to) as they are currently strings, and you can't run mathematical operations on strings.<jupyter_code>subset = data_list[1:301]

count_floats = []         

for row in subset:
    count_flo = float(row[4])
    count_floats.append(count_flo)

count_floats[0:5]<jupyter_output><empty_output><jupyter_text>Here is what the code above looks like in plain English, step by step: 1. We created a subset of the data_list of only the first 300 rows. Remember `data_list[1:301]` is going to start at index `1` (*which is the second item*) and end at index `301`, not including index `301` itself.2. We created an empty list called, `count_floats` where we are going to store our converted `count` data.3. We performed a `for` loop which iterates through each row in the `subset` and:     1. converts the `4th` index in each row to a float.     2. appends that newly made float to `count_floats` list above.4. Return the first five items of the `count_floats` list. What if we wanted to find all of the male names in this `subset` and `print` them out? How might we interact with the rows and columns of the data to perform this action? This would require us to refer to two different indexes within each row: 1) the `3rd`\* index, which gives us the `sex` associated with the name, and 2) the `1st`\*\* index, which contains the `name` itself. We're going to use a conditional statement in order to achieve our goal.\* *when we say "3rd" index, we don't mean third chronologically. Remember, `Python` indexes its lists starting with 0, so when we say 3rd, we are saying that the index value of the list is 3.*\*\* *when we say "1st" index, we mean it has the index value of 1, not to be confused with the "0th" index, or the chronologically first index of the list. See note above.*<jupyter_code>for row in subset:
    if row[3] == 'M':
        print(row[1])<jupyter_output>George
Henry
Edward
Harry
Arthur
Fred
Louis
Charlie<jupyter_text>Once again, we are going to use a `for` loop to interate through our `subset` of data.  We are then going to use a conditional statement to check **`if`** the `3rd` index in the row is equal to `M`, meaning male. Remember, a condition will return either `True` of `False`. If that statement is `True` then it will `print` out the value of the `1st` index, which is the column containing the name; if it is `False` then it will just skip over it.**On your own:** *Think about how you might add these names to a list type variable.*  Using `pandas``pandas` is a popular data science package for `Python`. It includes different data structures and tools, which provide for flexible data manipulation and analysis. Throughout this course, we will be using what are known as data frames, a 2-dimensional table that supports different data types. This is one of the key structures that `pandas` provides, and one that will return in `R`.Here is how it works:<jupyter_code>import pandas as pd

with open('../../datasets/baby-names/NationalNames1.csv', 'r') as file:
    df = pd.read_csv(file)
    del df['Id']<jupyter_output><empty_output><jupyter_text>It's that simple! Actually, it can be even simpler. Running the one-line piece of code below would actually produce (almost) the same thing. However, I wanted to get rid of the old `Id` column in the file before outputing my data frame, so I wrapped it in the `with open()` statement that we saw above.```pythonpd.read_csv('../../../datasets/baby-names/NationalNames1.csv')```Let's see what this looks like...<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Immediately, you should notice that it is rendered in a much cleaner format than the list of lists above. It looks more familiar, like something more akin to a spreadsheet. Even if you were to execute this within your terminal application, it would be rendered in such a way that it keeps the data in rows and columns. As you can see, the top row is going to be a header row. Whereas we had to remove the header row in the list of lists in order to manipulate the data, this row serves as a reference to the columns in `pandas`. The unnamed column on the left is your index column.   Data Filtering`pandas` allows us to easily filter data. Imagine that you were only interested in tracking how many people each year named their child "James". All we have to do is filter the `Name` column by the string "James".<jupyter_code>df[df['Name']=='James'].head()<jupyter_output><empty_output><jupyter_text>That was a very simple line of code, which is one of the benefits of using `pandas`. We can break it down step by step. Let's start with the line `df['Name']=='James'`. This is going to return `True` or `False` given whether or not the condition is met. Running this line by itself would just return a boolean for each index.  For this reason, we wrap that in `df[...]`, which returns the subset of the dataframe, `df` where the index returns `True`, thus returning all rows that match the string, "James".  The final `.head()` piece is a `pandas` method that you can call on dataframes to return the first rows. You could pass a numerical argument to return the first X-number of rows. For example, if we put `df[df['Name']=='James'].head(10)`, the first 10 rows of this subset would be returned.**On your own:** *What do you think `.tail()` returns?*<jupyter_code>df[(df['Name'] == 'James') & (df['Count'] > 6000) & (df['Count'] < 10000)]<jupyter_output><empty_output><jupyter_text>`pandas` can also match on much more than strings. Take the above example, which builds on the "James" subset. What if we were interested in knowing which years had a greater than 400 *James*es named but less than 10,000? That's what we did above. `pandas` allows us to match on many conditions very easily.  Data Sorting  You can also sort the data frame by a column. This works on both numerical data types as well as strings. For example, sorting by `df['Name']` will sort the data frame alphabetically by the 'Name' column, while sorting by `df['Count']`  will sort by numerical value. Take a look at the example below.<jupyter_code>df.sort_values(by = ['Count'], ascending = True).tail(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09336432120532137, 'fraction_numerical': 0.014665811828818721, 'mean_word_length': 4.395157803718115, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '528671', 'license': 'MIT', 'n_tokens_mistral': 3571, 'n_tokens_neox': 3339, 'n_words': 1880, 'path': 'Assingments/module05/students/module1/labs/.ipynb_checkpoints/intro_data_science_python-checkpoint.ipynb', 'repo_name': 'tonysulfaro/MI-250', 'size': '12478'}