<jupyter_start><jupyter_text>DataFrame Statistical MethodsIn this chapter, we cover [statistical methods][1] for the DataFrame which are identical to those available to a Series. Again, we distinguish between methods that aggregate and those that do not. A method that performs an aggregation returns a **single** number to summarize the values. Any method that does not return a single value is not an aggregation. We begin by reading in the San Francisco employee compensation dataset.[1]: http://pandas.pydata.org/pandas-docs/stable/reference/frame.htmlcomputations-descriptive-stats<jupyter_code>import pandas as pd
sf_emp = pd.read_csv('../data/sf_employee_compensation.csv')
sf_emp.head(3)<jupyter_output><empty_output><jupyter_text>Aggregation methodsThe following are some common aggregation methods available for DataFrames:* `sum`* `min`* `max`* `mean`* `median`* `std` - standard deviation* `var` - variance* `count` - returns number of non-missing values* `describe` - returns most of the above aggregations in one Series* `quantile` - returns the given percentile of the distribution Differences between DataFrame and Series methodsWhen calling an aggregation methods on a DataFrame, it is applied to each individual column by default. For instance, calling the `sum` method sums each column individually. A single value is returned for each column. Calling the `sum` method on a Series produces a single scalar value. Select numeric columnsSome of these statistical methods above work only with numeric columns. In order to successfully call these methods, we'll select only the columns with compensation information.<jupyter_code>comp = sf_emp.iloc[:, 3:]
comp.head()<jupyter_output><empty_output><jupyter_text>Take the mean of each columnLet's demonstrate taking the mean of each column by calling the `mean` aggregation method.<jupyter_code>comp.mean()<jupyter_output><empty_output><jupyter_text>Did you notice what type of object was returned?pandas takes the mean of each column and returns a Series. The new Series uses the old column names as the index and the calculated mean as the values. Let's call a couple of different aggregation methods.<jupyter_code>comp.max()
comp.std()
comp.mean().to_frame().T<jupyter_output><empty_output><jupyter_text>Potentially confusing orientationThe above results should be fairly easy to understand. If someone asked you what the standard deviation of the `security_delay` column, you would easily be able to respond with the correct number. What is potentially confusing is the orientation of the result. We began with a DataFrame, and were returned a Series which is visually displayed in the notebook as a vertical sequence of values. The orientation of the columns changed. It might have been easier to understand the operation if the columns remained horizontal as in the following image.![](images/df_agg_keep_dim.png) DataFrames are collections of columnsIt's good to think of DataFrames as a collection of columns as opposed to a collection of rows. It is the column that is the fundamental component of the DataFrame. Each column has a data type and all values in that column are the same data type. It is the column that is acted on by default by most of the methods as demonstrated with the aggregations above.  Changing the direction of the operationSince DataFrames are two-dimensional, we might want to complete an operation horizontally across the rows instead of vertically down the columns. The `axis` parameter controls the direction of the operationMost DataFrame methods have an `axis` parameter. This is a crucial parameter to understand as it controls the direction of the operation. By default, operations happen vertically down each column. Each axis may be referenced by number or string labelDataFrames are two-dimensional and therefore have two axes. Both the rows and the columns may be referenced with a number or a string label. The rows are referenced by the number 0 and also by the label 'index'. The columns are referenced by the number 1 and also by the label 'columns'. Default value of `axis` is 0For most DataFrame methods, the default value of the `axis` parameter is 0. Technically, you will see `None` in the method signature, but if you don't explicitly set it, pandas will use 0. You can also refer to it as the string 'index'. Let's take the mean of each column again, but use the string 'index' for the value of the `axis` parameter. This produces the exact same result as calling it with the defaults.<jupyter_code>comp.mean(axis='index')<jupyter_output><empty_output><jupyter_text>We could have set `axis` to 0, which also returns the same result. Since the default behavior is to act vertically, it's not necessary to specify the axis parameter as such and most people do not do so when calculating aggregations on each column.<jupyter_code>comp.mean(axis=0)<jupyter_output><empty_output><jupyter_text>Change the direction of the operation with `axis='columns'`Let's change the direction of the operation and sum each row by setting the `axis` parameter to 'columns'. This gives us the total compensation for each employee.<jupyter_code>comp.sum(axis='columns').head(10)<jupyter_output><empty_output><jupyter_text>Instead of using the string 'columns', you can set `axis` to 1 to achieve the same result.<jupyter_code>comp.sum(axis=1).head(10)<jupyter_output><empty_output><jupyter_text>Use either `axis='columns'` or `axis=1`You are free to use either `axis='columns'` or `axis=1` as they both accomplish the same exact task. Difficult to rememberIt's definitely confusing and difficult to remember which direction the operation is going to happen. As with the examples above, using 'index' or 0 sums up each column while using 'columns' or 1 sums up each row.![][1][1]: images/df_axes_explanation.png A little trick that helps me remember is that when using `axis='columns'` the result is going to be the same length as a column in the DataFrame. Let's verify this below.<jupyter_code>comp.shape
len(comp.sum(axis='columns'))<jupyter_output><empty_output><jupyter_text>Summary of the `axis` parameter* **axis 0**    * Default axis for most DataFrame methods    * Also referenced by the string 'index'    * Operations happen vertically, up and down the columns    * Example - `df.sum()` computes the sum of each column individually* **axis 1**    * Also referenced by the string 'columns'    * Operations happen horizontally, left to right across each row    * Example - `df.sum(axis='columns')` computes the sum of each row individually Non-Aggregation methodsThe non-aggregation DataFrame methods do not return a single value for each column and instead return a DataFrame that usually has the same shape as the original. Here are some common non-aggregation methods.* `abs` - takes absolute value* `round` - round to the nearest given decimal place* `cummin` - cumulative minimum* `cummax` - cumulative maximum* `cumsum` - cumulative sumLet's use the `round` method to round each column to the nearest thousand. Remember that negative numbers round to the left of the decimal place.<jupyter_code>comp.round(-3).head(3)<jupyter_output><empty_output><jupyter_text>Some methods don't have an `axis` parameterMethods such as `round` work independently of the axis and therefore do not have an `axis` parameter. Other non-aggregation methods such as `cumsum` do have an `axis` parameter. Called with the defaults (`axis=0`), the `cumsum` method computes the cumulative sum of each column individually.<jupyter_code>comp.cumsum().head(3)<jupyter_output><empty_output><jupyter_text>Changing the direction of the operation, the `cumsum` method calculates the cumulative sum of each row individually.<jupyter_code>comp.cumsum(axis='columns').head(3)<jupyter_output><empty_output><jupyter_text>The values in the last column of the above DataFrame are equal to the sum of the entire row.<jupyter_code>comp.sum(axis=1).head(3)<jupyter_output><empty_output><jupyter_text>Summary statistics for all columns with the `describe` methodThe describe method calculates several summary statistics for each column and is a nice way to inspect all of your data at once. Notice that a DataFrame is returned with the name of each summary statistic in the index. By default, it returns the 25th, 50th, and 75th percentiles. You can customize these by passing in a list of numbers between 0 and 1 to the `percentiles` parameter.<jupyter_code>comp.describe(percentiles=[.1, .2, .4, .5, .9, .99])<jupyter_output><empty_output><jupyter_text>The `describe` method with non-numeric columnsThe `comp` DataFrame from above contains only numeric columns. If `describe` is called on a DataFrame containing a mix of numeric and non-numeric columns, then summary statistics for just the numeric columns will be returned. The others will be ignored. The original `sf_emp` DataFrame contains a mix of data types. Let's call `describe` on it. Notice how the number of columns after calling `describe` decreased from 9 to 7.<jupyter_code>sf_emp.shape
sf_emp.describe()
sf_emp.describe().shape<jupyter_output><empty_output><jupyter_text>Calling `describe` on non-numeric columnsThe `describe` method can work with non-numeric columns, but you'll need to set the `include` parameter to a string of the data type you would like to use. Below, a summary of the object columns is produced. Notice that pandas returns a completely different set of summary statistics that make more sense with strings.<jupyter_code>sf_emp.describe(include='object')<jupyter_output><empty_output><jupyter_text>Transposing a  DataFrame with the `T` attributeTransposing a DataFrame 'rotates' the data 90 degrees. The columns and the rows switch places. The first column is now the first row. The `.T` attribute transposes the DataFrame. I find this useful after running the `describe` method when there are many columns as it's easier to read many rows of data as opposed to many columns of data.<jupyter_code>sf_emp.describe().T<jupyter_output><empty_output><jupyter_text>Nuisance ColumnsAbove, we called common statistical methods from the `comp` DataFrame, which was composed of only numeric columns. It is possible to call these same methods from DataFrames composed of any combination of data types. Dropping columns that don't work with the methodpandas allows you to call these statistical methods on DataFrames containing columns with data types that don't work for that particular method. The entire `sf_emp` DataFrame contains string and numeric columns. Taking the mean of a string column does not work. Instead of raising an error, pandas **silently** drops these column. These DataFrame columns that don't compute with certain methods are sometimes referred to as **nuisance columns**.Let's show this by calling the `mean` method on the San Francisco employee compensation dataset with all of the original columns. We will work with only 100 rows of the data, which will be explained shortly.<jupyter_code>sf_emp_100 = sf_emp.head(100)
sf_emp_100.head(3)<jupyter_output><empty_output><jupyter_text>Calling the mean method drops the two columns containing strings from the result. No error is raised.<jupyter_code>sf_emp_100.mean()<jupyter_output><empty_output><jupyter_text>The `round` method doesn't drop columnsThe `round` method works in a slightly different manner. Instead of dropping the non-numeric columns, it keeps them in the resulting DataFrame as they were. This is quite nice as we can use `round` on a DataFrame with both strings and numeric columns without error and without dropping columns.<jupyter_code>sf_emp_100.round(0).head(3)<jupyter_output><empty_output><jupyter_text>Many methods do work with non-numeric data typesMany of the aggregation methods do work with string and datetime columns. Let's find the max of all the `sf_emp` columns.<jupyter_code>sf_emp_100.max()<jupyter_output><empty_output><jupyter_text>The `sum` method is valid for string (but not datetime) columns and concatenates all the values together to produce one long string. This usually isn't something you'd like to do. It's also a computationally expensive operation. The following call to `sum` took about 4 seconds on the full dataset (50k rows) on my machine.<jupyter_code>sf_emp_100.sum()<jupyter_output><empty_output><jupyter_text>Use `numeric_only=True`The `sum` method, as well as all the other aggregation methods,  provides the boolean parameter `numeric_only` that is defaulted to `False`. By setting it to `True`, pandas will only apply the method to boolean, integer, and float columns. The following operation only took 7 ms on the full dataset on my machine or more than 1,000 times faster than the previous one.<jupyter_code>sf_emp.sum(numeric_only=True)<jupyter_output><empty_output><jupyter_text>The slow  `mean` methodThe `mean` method is also extremely slow, even though it only works on numeric columns. This is because pandas takes the `sum` of all the columns first and then divides by the length. The reason pandas doesn't just skip over string columns is that they are technically object columns and an object column can hold any data type. The only way for pandas to decide whether or not the `mean` will work on an object column is to actually sum up every value first and then attempt to divide by the length. If that fails, then it will skip it. The issue with this, is that it is extremely slow for string columns since strings can be summed. pandas only fails after the string column has been concatenated together when it attempts to divide by the length. If you want to take the `mean` on a DataFrame with string columns, make sure you set `numeric_only` to `True.` Even on this small dataset of 100 rows, there is a substantial performance difference.<jupyter_code>%timeit -n 5 sf_emp_100.mean()
%timeit -n 5 sf_emp_100.mean(numeric_only=True)<jupyter_output><empty_output><jupyter_text>ExercisesExecute the following cell to read in the movie dataset with the title in the index selecting all three actor Facebook like columns.<jupyter_code>movie = pd.read_csv('../data/movie.csv', index_col='title')
cols = ['actor1_fb', 'actor2_fb', 'actor3_fb']
actor_fb = movie[cols]
actor_fb.head(3)<jupyter_output><empty_output><jupyter_text>Exercise 1Calculate the mean of each actor Facebook like column. Which actor (1, 2, or 3) has the highest mean?  Exercise 2The result of exercise 1 is a Series of three values. Can you call a method on this Series to choose the column name with the highest mean Facebook likes.  Exercise 3Calculate the total Facebook likes of all three actors for each movie  Exercise 4What percentage of movies have more than 10,000 total actor FB likes?  Exercise 5Find the median gross revenue in millions of dollars for the movies that have more than 10,000 total actor FB likes. Do the same for movies with 10,000 or less total actor FB likes.  Exercise 6From exercise 5, it appears that movies with more than 10,000 total actor FB likes gross 2.5 times as much. This may be due to the fact that newer movies have more actors that are recognized by FB users. Find the median year produced for both groups.  Exercise 7For each movie made in the year 2016, what is the median of the total actor FB likes?  Exercise 8Write a function that has a single parameter, `year`. Have it return the median of the total actor FB likes for the given year. Test your function with the year 2016 and verify the result with Exercise 6.  Exercise 9Write a loop to print out the year and median total actor FB likes for that year from 1990 to 2016 Use the college dataset with the institution name as the index for the remaining exercises.<jupyter_code>college = pd.read_csv('../data/college.csv', index_col='instnm')
college.head(3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06132943940067297, 'fraction_numerical': 0.00984064503841026, 'mean_word_length': 5.4557377049180324, 'pattern_counts': {'":': 0, '<': 105, '<?xml version=': 0, '>': 105, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '615328', 'license': 'MIT', 'n_tokens_mistral': 4236, 'n_tokens_neox': 3959, 'n_words': 2144, 'path': 'jupyter_notebooks/pandas/mastering_data_analysis/04. Essential DataFrame Commands/02. DataFrame Statistical Methods.ipynb', 'repo_name': 'manual123/Nacho-Jupyter-Notebooks', 'size': '15751'}