<jupyter_start><jupyter_text>Getting Started with `NumPy` [**`NumPy`**](http://www.numpy.org/) is one of the fundamental packages for conducting any type of data science or data analysis project in `Python`. It seemlessly integrates with other packages like `scikit-learn` or `pandas`, and provides an extensive mathematical library to perform analyses or build learning algorithms. Throughout this course (and the entire program) we will rely heavily on the `NumPy` package for data analysis. In this notebook we will cover some of the data types that `NumPy` affords us as well as some of the methods that are commonly used.------  What we'll cover... 1. [Creating an Array](array)2. [Array Indexing](indexing)3. [Array Math](math)4. [Array Methods](methods)5. [Subsetting](subset)------ DataIn this notebook, we are going to utilize a Cats v. Dogs dataset. We will reference this dataset by its column indexes so here is a table of them below.Index | Description------|------------`0`| Number of households (in 1000s)`1`| Percentage of households with pets`2`| Number of households with pets (in 1000s)`3`| Percentage of dog owners`4`| Dog owning households (in 1000s)`5`| Mean number of dogs per household`6`| Dog population (in 1000s)`7`| Percentage of cat owners`8`| Cat owning households (in 1000s)`9`| Mean number of catss per household`10`| Cat population (in 1000s)Keep in mind that the original first row (index 0) is actually a column of state names but we will eventually remove this column as it is the only row of strings.---  Reading in the fileBefore we begin with `NumPy`, we are first going to read in our dataset using the `csv` library. This will give us a point of comparison when we are looking at the capabilities that `Numpy` affords us.<jupyter_code>import csv

with open("../../datasets/catsvdogs.csv") as f:
    file = list(csv.reader(f, delimiter=","))<jupyter_output><empty_output><jupyter_text>For indexing purposes, we went ahead and created a list of lists of our dataset. The main list can be thought of as a collection of rows and each of these rows can be thought of as a list of values.  We can select particular rows by specifying their respective indexes. For example, if we wanted to view the first three rows we would do the following...<jupyter_code>print(file[:3])<jupyter_output>[['Location', 'Number of Households (in 1000)', 'Percentage of households with pets', 'Number of Pet Households (in 1000)', 'Percentage of Dog Owners', 'Dog Owning Households (1000s)', 'Mean Number of Dogs per household', 'Dog Population (in 1000)', 'Percentage of Cat Owners', 'Cat Owning Households', 'Mean Number of Cats', 'Cat Population'], ['Alabama', '1828', '60', '1088', '44', '807', '2', '1410', '27', '501', '3', '1252'], ['Arizona', '2515', '60', '1497', '40', '1008', '2', '1798', '30', '743', '2', '1438']]<jupyter_text>Quick reminder, 0 is the first index, not 1. Also, the notation `[:3]` means from the beginning of the subscriptable item (index 0) all the way to but not including index 3.So the first row is the header. Let's go ahead and take a look at the values in the header row.<jupyter_code>print(file[0])<jupyter_output>['Location', 'Number of Households (in 1000)', 'Percentage of households with pets', 'Number of Pet Households (in 1000)', 'Percentage of Dog Owners', 'Dog Owning Households (1000s)', 'Mean Number of Dogs per household', 'Dog Population (in 1000)', 'Percentage of Cat Owners', 'Cat Owning Households', 'Mean Number of Cats', 'Cat Population']<jupyter_text>The header row will get in our way of any analysis. Its only purpose is to serve (and this is an important purpose) as a reference information to the attributes of that column.  So, if we were interested in finding the mean number of households with pets, we would have to ensure that we are excluding the header. Speaking of finding the mean, we can go ahead and do that.<jupyter_code>pets_n = [int(item[3]) for item in file[1:]]

print(sum(pets_n)/len(pets_n))<jupyter_output>1342.591836734694<jupyter_text>We first have to grab the correct column. This isn't the easy task when using a list of lists, but create a new list using a `for`-loop to grab the fourth item from every row, We can then use some of `Python`'s built in function to find the sum of those values and the number of values in that list (the two metrics we need to calculate the mean). Also, the notation above may be unfamiliar to you, but this...```pythonpets_n = [int(item[3]) for item in file[1:]]```... is the same as...```pythonpets_n = []for item in file[1:]:    pets_n.append(int(item[3]))```Notice how the loop starts at index 1, not index 0. That is because index 0 is the header name for that column. The `item` in this loop is the row, and from each row we want the fourth item, or `item[3]`. But at the moment, `item[3]` is a string, but in order to find the mean, we need to be working with some sort of numeric type, so we change each one to an `int`.---Now this next step is just to get us started with `Numpy`. We first want to remove the header row from our data and then we want to split up the numerical columns and the state column.<jupyter_code>nums = [item[1:-1] for item in file[1:]]
states = [item[0] for item in file[1:] ]<jupyter_output><empty_output><jupyter_text>`nums` is now a list of lists of all of the numeric rows in the `file` object or, in other words, all of the columns except for the first column. The first column is the state column and we will come back to it later.  1. Creating an ArrayOne of the most popular objects that `NumPy` provides is the the `numpy.array`. These arrays come with a variety of methods that can be run on them, but before we begin to do anything interesting with arrays, we first need to learn how to create them.<jupyter_code>import numpy as np

pets = np.array(nums, dtype=np.int)
pets[:3]<jupyter_output><empty_output><jupyter_text>One of the reasons that we split the states column from the rest of the columns is that `NumPy` requires that all the values of a single array be of the same type. That being said, since you can't coerce a string to integer, but you can an integer to string, keeping the states column around would change all of the values in the `pets` array to a string. For the operations that we want to demonstrate in this notebook, some type of numeric data type is required.The `np.array()` method creates an array from our list of lists. In this case, it is a 2-dimensional array, which can be thought of as having one dimension of rows and one dimension of columns.  We can take a look at the shape of this array by doing the following...<jupyter_code>pets.shape<jupyter_output><empty_output><jupyter_text>`pets.shape` returns a tuple object, where the first value is the number of items in the first dimension (number of rows) and the second is the number of items in the second dimension (number of columns). This (row, column) notation will come up over and over in `Numpy` as well as other data science libraries such as `pandas`, so be sure to remember which is which.  Alternative ways to creating arraysWe can also create our own arrays by simply specifying a list or tuple of values. Below are some of the ways we can construct arrays.We can pass the `np.array()` method a list of values...<jupyter_code>np.array([2,3,1,0])<jupyter_output><empty_output><jupyter_text>We can also create a two-dimensional array by passing a list of lists/tuples. Take a look at this example closely...Notice how the last item is a tuple, but the first two are lists. `NumPy` can handle these together.<jupyter_code>np.array([[1,2.0],[0,0],(1+1,3.)])<jupyter_output><empty_output><jupyter_text>You can also create arrays with different data types as inputs. However, since all of the data types of a single array have to be of the same type, the output will coerce the `int`s to `strings`.<jupyter_code>np.array(['x', 'y', 0, 1])<jupyter_output><empty_output><jupyter_text>We can even create an array of 0s. The `.zeros()` method can take a tuple as an argument that specifies the number of rows and columns that we would like. Notice how the below creates an array of all zeros with 4 rows and 5 columns.<jupyter_code>np.zeros((4,5))<jupyter_output><empty_output><jupyter_text>The `np.arange()` method will return an array with a sequence of evenly spaced intervals. If you only specify one number, an array of that length will be returned starting at the index of 0.<jupyter_code>np.arange(6)<jupyter_output><empty_output><jupyter_text>If you pass two integers, then it will return an array starting at the first value and going to, but not including, the second. You can also specify the data type that you wish for it to be.<jupyter_code>np.arange(2, 10, dtype=np.float)<jupyter_output><empty_output><jupyter_text>If you add a third number, that will then become the interval spacing. In the example below, we are telling `NumPy` that we want an array that begins at 2, ends at 3 (but doesn't include it), and every value in between should be .5 greater than the value before it.<jupyter_code>np.arange(2, 3, 0.5)<jupyter_output><empty_output><jupyter_text>Finally, we can create an array of random values as well...<jupyter_code>np.random.rand(2,2)<jupyter_output><empty_output><jupyter_text>----- 2. Array IndexingOne of the major benefits of using `Numpy` is its indexing capabilities. Unlike the string of string method that we used above, we can easily grab entire rows **and** columns, or slices of them.In the next few examples, we are going to return back to our array that we created from our dataset. For reference, we called the array object, `pets` and we created it [here](array). If the array is two-dimensional, as is the case with our `pets` array, we can specify the row index as the first number and the column as the second to select a particular value. For example, if we want to select the value at row-index 4 and column-index 2, we do this...<jupyter_code>pets[4,2]<jupyter_output><empty_output><jupyter_text>Or we can take slices of the array by specifying a range of values. For example, if we wanted the first 10 rows of the fourth column, we would do this...<jupyter_code>pets[0:10, 3]<jupyter_output><empty_output><jupyter_text>And if we wanted the entire fourth row, we would do this...<jupyter_code>pets[3,]<jupyter_output><empty_output><jupyter_text>However, if you want to select an entire column, you will have to provide the `:` operator in the row-index position like so...<jupyter_code>pets[:,4]<jupyter_output><empty_output><jupyter_text>Modifying the arrayIndexing is particularly important when it comes to modifying values of the array. For these next couple of blocks, we are going to be working with a copy of the `pets` array. One might think that you could just create a copy by using the assignment operator, `=` like so:```python pets_cp = pets```However, with `NumPy` arrays, any modification you make to the `pets_cp` will also modify the `pets` array. Instead, we can create a copy of the pets `array` by using the `np.copy()` method.<jupyter_code>pets_cp = np.copy(pets)<jupyter_output><empty_output><jupyter_text>Now, if we want to change the value at any particular point we could just select the value by its indexes and assign it a new value like so...<jupyter_code>pets_cp[4,5] = 40
print("The original value of pets_cp[4,5] was {} but now it is {}."
      .format(pets[4,5],pets_cp[4,5]))<jupyter_output>The original value of pets_cp[4,5] was 2 but now it is 40.<jupyter_text>We can also change the value of an entire slice. If we wanted to change the entire 4th column to the value 400, you can do that.<jupyter_code>pets_cp[:,3]= 400

print("First few values of the 4th column of the pets array were {} and now they are {}."
      .format(pets[0:4,3],pets_cp[0:4,3]))<jupyter_output>First few values of the 4th column of the pets array were [44 40 48 33] and now they are [400 400 400 400].<jupyter_text>--- 3. Array MathOne of the major strengths of `NumPy` is its easy to use mathematical operations on arrays. Let's take a look at the column that we will be modifying in the next few exercises...<jupyter_code>pets[:,3]<jupyter_output><empty_output><jupyter_text>If we wanted to add 10 to every value in this column, all we would have to do is specify the column, the operator `+` and then `10`<jupyter_code>pets[:,3] + 10<jupyter_output><empty_output><jupyter_text>Similarly with subtraction...<jupyter_code>pets[:,3] - 10<jupyter_output><empty_output><jupyter_text>The fourth row is the "Percentage of households with dogs" attribute in the dataset. If we wanted to change this value into a proportion, all we would need to do is multiply it by 0.01.<jupyter_code>pets[:,3] * .01<jupyter_output><empty_output><jupyter_text>...or divide by 100.<jupyter_code>pets[:,3] / 100<jupyter_output><empty_output><jupyter_text>We can also perform operations between arrays. Take for example, multiplying column 4 by column 4...<jupyter_code>pets[:,3] * pets[:,3]<jupyter_output><empty_output><jupyter_text>---- 4. Array Methods`NumPy` also delivers some convenient methods to perform on arrays to make our life easier. For example, if we wanted to find the sum of all of the values in the 4th column, we can simply use the `sum()` method.<jupyter_code>pets[:,3].sum()<jupyter_output><empty_output><jupyter_text>And if we wanted to find the mean value of the column you would...<jupyter_code>pets[:,3].mean()<jupyter_output><empty_output><jupyter_text>We can check this by using the `sum()` method again and dividing by the number of values in that row.<jupyter_code>pets[:,3].sum()/len(pets[:,3])<jupyter_output><empty_output><jupyter_text>Checks out! And if we find the mean, we often want to know the standard deviation...<jupyter_code>pets[:,3].std()<jupyter_output><empty_output><jupyter_text>We may also want to know the range of values...<jupyter_code>pets[:,3].min()
pets[:,3].max()<jupyter_output><empty_output><jupyter_text>The `pets` array currently has 49 rows and 10 columns. There are situations in which you may want to transpose the array so that the rows become the columns and the columns become the rows. In order to do that, you simply call the `.transpose()` method.<jupyter_code>pets_tr = pets.transpose()
pets_tr.shape<jupyter_output><empty_output><jupyter_text>---- 5. SubsettingWe may also want to subset the array based on the values meeting  particular conditions. We can use our logical operators to return an array of `bool`s, which we can then use to subset our array.Imagine that we wanted to know those rows that had a percentage of households with dogs greater than 45. All we have to do is this...<jupyter_code>pets[:,3] > 45<jupyter_output><empty_output><jupyter_text>We can now name this array something, and then use it as our row subsetter like so...<jupyter_code>many_dogs = pets[:,3] > 45
pets[many_dogs,:]<jupyter_output><empty_output><jupyter_text>As we can see, only 5 rows are returned from the entire array. We can also specify multiple conditions.<jupyter_code>many_pets = (pets[:,3] > 40) & (pets[:,7] > 30)
pets[many_pets, :]<jupyter_output><empty_output><jupyter_text>Remember the states list that we created above? Well, we can now change that to an array of strings. We then need to reshpape it so that we can perform some subsetting operations on it. To do so, we use the `reshape()` method and specify that we want it to be 49 rows long and 1 column wide.<jupyter_code>states_arr = np.array(states)
states_resh = states_arr.reshape(49,1)<jupyter_output><empty_output><jupyter_text>We can now subset on this array using the `many_pets` condition we defined above. This is convenient as it gives us an easy way to see what states have a lot of households with cats and dogs.<jupyter_code>states_resh[many_pets, :]<jupyter_output><empty_output><jupyter_text>Joining arraysWe can also join arrays. In this case, since we want to join them horizontally, our number of rows for each array have to match. We will use the `.hpstack()` method to join the two ("h" in this case stands for horizontal). Remember, when we do this, the datatypes will have to match so the new array will be an array of strings.<jupyter_code>joined = np.hstack((pets, states_resh))<jupyter_output><empty_output><jupyter_text>Now, with this, we can subset the entire array...<jupyter_code>joined[many_pets, :]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0918572735590119, 'fraction_numerical': 0.020616041476059773, 'mean_word_length': 5.101972459992557, 'pattern_counts': {'":': 0, '<': 160, '<?xml version=': 0, '>': 164, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '41', 'dir': 'jupyter-structured-clean-dedup', 'id': '378510', 'license': 'MIT', 'n_tokens_mistral': 5099, 'n_tokens_neox': 4813, 'n_words': 2319, 'path': 'Assingments/module05/students/module2/labs/intro_numpy.ipynb', 'repo_name': 'tonysulfaro/MI-250', 'size': '16395'}