<jupyter_start><jupyter_text>Chapter 2: Essential DataFrame Operations Recipes* [Selecting multiple DataFrame columns](Selecting-multiple-DataFrame-columns)* [Selecting columns with methods](Selecting-columns-with-methods)* [Ordering column names sensibly](Ordering-column-names-sensibly)* [Operating on the entire DataFrame](Operating-on-the-entire-DataFrame)* [Chaining DataFrame methods together](Chaining-DataFrame-methods-together)* [Working with operators on a DataFrame](Working-with-operators-on-a-DataFrame)* [Comparing missing values](Comparing-missing-values)* [Transposing the direction of a DataFrame operation](Transposing-the-direction-of-a-DataFrame-operation)* [Determining college campus diversity](Determining-college-campus-diversity)<jupyter_code>import pandas as pd
import numpy as np
pd.options.display.max_columns = 40<jupyter_output><empty_output><jupyter_text>Selecting multiple DataFrame columns Read in the movie dataset, and pass in a list of the desired columns to the indexing operator:<jupyter_code>movie = pd.read_csv('data/movie.csv')
movie_actor_director = movie[['actor_1_name', 'actor_2_name', 'actor_3_name', 'director_name']]
movie_actor_director.head()
type(movie.columns)
movie.columns
[col for col in movie.columns if "act" in col]
actdf = movie[[col for col in movie.columns if "act" in col]]
actdf.head()
movie[['director_name']].head()
# remember that column names go inside double brackets [[column1, column2, etc]]
movie[['actor_1_name', 'actor_2_name', 'actor_3_name', 'director_name']]<jupyter_output><empty_output><jupyter_text>There's more... Passing a long list inside the indexing operator might cause readability issues. To help withthis, you may save all your column names to a list variable first. The following codeachieves the same result as step 1<jupyter_code>cols =['actor_1_name', 'actor_2_name', 'actor_3_name', 'director_name']
movie_actor_director = movie[cols]<jupyter_output><empty_output><jupyter_text>Selecting columns with methods<jupyter_code>movie = pd.read_csv('data/movie.csv', index_col='movie_title')
movie.get_dtype_counts()
movie['director_name'].dtype == 'O'
intdf = movie[[col for col in movie.columns if movie[col].dtype == 'int64']]
intdf.head()
# Use the select_dtypes method to select only the integer columns
movie.select_dtypes(include=['int64']).head()
# If you would like to select all the numeric columns, you may simply pass the 
# string number to the include parameter:
#we filter for floats AND ints
movie.select_dtypes(include=['number']).head()
# An alternative method to select columns is with the filter method. This method 
# is flexible and searches column names (or index labels) based on which
# parameter is used. Here, we use the like parameter to search for all column
# names that contain the exact string, facebook:
movie.filter(like='facebook').head()
# The filter method allows columns to be searched through regular expressions 5.
# with the regex parameter. Here, we search for all columns that have a digit
# somewhere in their name
movie.filter(regex='\d').head()
# we can filter for nonexisting column names (such as 'asdf') and KeyError will NOT be raised
movie.filter(items=['actor_1_name', 'asdf']).head()<jupyter_output><empty_output><jupyter_text>Ordering column names sensibly<jupyter_code>movie = pd.read_csv('data/movie.csv')
movie.head()
movie.columns
disc_core = ['movie_title','title_year', 'content_rating','genres']
disc_people = ['director_name','actor_1_name', 'actor_2_name','actor_3_name']
disc_other = ['color','country','language','plot_keywords','movie_imdb_link']
cont_fb = ['director_facebook_likes','actor_1_facebook_likes','actor_2_facebook_likes',
           'actor_3_facebook_likes', 'cast_total_facebook_likes', 'movie_facebook_likes']
cont_finance = ['budget','gross']
cont_num_reviews = ['num_voted_users','num_user_for_reviews', 'num_critic_for_reviews']
cont_other = ['imdb_score','duration', 'aspect_ratio', 'facenumber_in_poster']
new_col_order = disc_core + disc_people + disc_other + \
                    cont_fb + cont_finance + cont_num_reviews + cont_other
set(movie.columns) == set(new_col_order)
movie2 = movie[new_col_order]
movie2.head()<jupyter_output><empty_output><jupyter_text>Operating on the entire DataFrame<jupyter_code>pd.options.display.max_rows = 8
movie = pd.read_csv('data/movie.csv')
movie.shape
movie.size
movie.ndim
len(movie)
movie.count()
movie.min()
movie.describe()
movie.describe().T
pd.options.display.max_rows = 30
movie.describe(percentiles=[.01, .3, .99])
pd.options.display.max_rows = 8
movie.isnull().sum()<jupyter_output><empty_output><jupyter_text>There's more...<jupyter_code>movie.min(skipna=False)<jupyter_output><empty_output><jupyter_text>Chaining DataFrame methods together<jupyter_code>movie = pd.read_csv('data/movie.csv')
movie.isnull().head()
movie.isnull().sum().head()
movie.isnull().sum().sum()
movie.isnull().any().any()<jupyter_output><empty_output><jupyter_text>How it works...<jupyter_code>movie.isnull().get_dtype_counts()<jupyter_output><empty_output><jupyter_text>There's more...<jupyter_code>movie[['color', 'movie_title', 'color']].max()
# here we fill all empty object cells with some string (here it is empty '')
movie.select_dtypes(['object']).fillna('').max()
# here we fill all empty object cells with some string (here it is empty '')
movie.select_dtypes(['object']).fillna('AA').min()<jupyter_output><empty_output><jupyter_text>Working with operators on a DataFrame  Getting ready...<jupyter_code>college = pd.read_csv('data/college.csv')
college.head()
college.count()
college.shape
college.describe().T
college.describe(include='object').T
college = pd.read_csv('data/college.csv')
college + 5
college = pd.read_csv('data/college.csv', index_col='INSTNM')
college_ugds_ = college.filter(like='UGDS_')
college_ugds_.head()
college == 'asdf'
college_ugds_.head()
college_ugds_.head() + .00501
undergrads = college_ugds_
undergrads.head()
undergrads['SUM'] = 0
undergrads['SUM'] = undergrads.sum(axis = 1, skipna = True)
undergrads.head()
undergrads.head()
undergrads.cumsum(axis=1).head()
undergrads.median(axis="index")
# here we use // so we get percentages
(college_ugds_.head() + .00501) // 0.01
college_ugds_op_round = (college_ugds_ + .00501) // .01 / 100
college_ugds_op_round.head()
college_ugds_round = (college_ugds_ + .00001).round(2)
college_ugds_round.head()
.045 + .005
college_ugds_op_round.equals(college_ugds_round)<jupyter_output><empty_output><jupyter_text>There's more...<jupyter_code>college_ugds_op_round_methods = college_ugds_.add(.00501).floordiv(.01).div(100)<jupyter_output><empty_output><jupyter_text>Comparing missing values<jupyter_code>np.nan == np.nan
None == None
5 > np.nan
np.nan > 5
5 != np.nan
college = pd.read_csv('data/college.csv', index_col='INSTNM')
college_ugds_ = college.filter(like='UGDS_')
college_ugds_.head() == .0019
college_self_compare = college_ugds_ == college_ugds_
college_self_compare.head()
college_self_compare.all()
(college_ugds_ == np.nan).sum()
college_ugds_.isnull().sum()
from pandas.testing import assert_frame_equal
assert_frame_equal(college_ugds_, college_ugds_)<jupyter_output><empty_output><jupyter_text>There's more...<jupyter_code>college_ugds_.head()
college_ugds_[college_ugds_ == .0019]
college_ugds_.eq(.0019).head()<jupyter_output><empty_output><jupyter_text>Transposing the direction of a DataFrame operation<jupyter_code>college = pd.read_csv('data/college.csv', index_col='INSTNM')
college_ugds_ = college.filter(like='UGDS_')
college_ugds_.head()
college_ugds_.count()
college_ugds_.count(axis=0)
college_ugds_.count(axis='index')
college_ugds_.count(axis='columns').head()
college_ugds_.sum(axis='columns').head()
college_ugds_.median(axis='index')<jupyter_output><empty_output><jupyter_text>There's more<jupyter_code>college_ugds_cumsum = college_ugds_.cumsum(axis=1)
college_ugds_cumsum.head()
undergrads.sort_values('UGDS_ASIAN', ascending=False).head()
college_ugds_cumsum.sort_values('UGDS_WHITE', ascending=False)<jupyter_output><empty_output><jupyter_text>Determining college campus diversity<jupyter_code>pd.read_csv('data/college_diversity.csv', index_col='School')
college = pd.read_csv('data/college.csv', index_col='INSTNM')
college_ugds_ = college.filter(like='UGDS_')
college_ugds_.head()
college_ugds_.isnull().sum(axis=1).sort_values(ascending=False).head()
college_ugds_ = college_ugds_.dropna(how='all')
college_ugds_.isnull().sum()
college_ugds_.ge(.15).head()
diversity_metric = college_ugds_.ge(.15).sum(axis='columns')
diversity_metric.head()
diversity_metric.value_counts()
diversity_metric.sort_values(ascending=False).head()
college_ugds_.loc[['Regency Beauty Institute-Austin', 
                          'Central Texas Beauty College-Temple']]
us_news_top = ['Rutgers University-Newark', 
               'Andrews University', 
               'Stanford University', 
               'University of Houston',
               'University of Nevada-Las Vegas']
diversity_metric.loc[us_news_top]<jupyter_output><empty_output><jupyter_text>There's more...<jupyter_code>college_ugds_.max(axis=1).sort_values(ascending=False).head(10)
college_ugds_.loc['Talmudical Seminary Oholei Torah']
(college_ugds_ > .01).all(axis=1).any()
undergrads.head()
undergrads.iloc[:5, 1:5]
college2 = pd.read_csv('data/college.csv', index_col='INSTNM')
college2.head()
undergrads['Black_Asian'] = undergrads['UGDS_BLACK'] + college2['UGDS_ASIAN']
undergrads.head()
undergrads['BLACKnWHITE'] = undergrads['UGDS_BLACK'] + undergrads['UGDS_WHITE']
undergrads['ACROSSALL'] = undergrads.apply
undergrads.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13662790697674418, 'fraction_numerical': 0.012562292358803987, 'mean_word_length': 7.494708994708994, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '117522', 'license': 'MIT', 'n_tokens_mistral': 3516, 'n_tokens_neox': 3307, 'n_words': 619, 'path': 'Pandas-CookBook/Chapter 02 Essential DataFrame Operations.ipynb', 'repo_name': 'ValRCS/RCS_ML_01_20', 'size': '9632'}