<jupyter_start><jupyter_text>Intro to Pandas Pandas is a very commonly used package in Python. It allows us to easily work with dataframes (similar to tables in excel). As with any package, first we have to import it. It is also customary to include other packages you want to import up here.<jupyter_code>#import pandas
import pandas as pd<jupyter_output><empty_output><jupyter_text>To work with data (typically in a csv), you need to 'read' it into Python - you also want to store it as a dataframe, and name it something (typically df, or something that helps you remember what the data is)<jupyter_code>df = pd.read_csv('census.csv',index_col='FIPS') #if we don't specify index_col, Python will default to 0,1,2...n<jupyter_output><empty_output><jupyter_text>Exploring the DataFrame<jupyter_code>#.head() gives you the first 5 rows - you can also specify a different number for more or fewer rows
df.head()
#tail() gives you the last 5 rows - here we specify to only get the last 2
df.tail(2)
#tells you number of rows
len(df)
#gives you dimensions of your df
#not here that certain methods do not take () - they do not take arguments
df.shape
#some summary stats for your df
df.describe()
#tells you data types & whether rows are null
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 917 entries, 10100 to 49820
Data columns (total 12 columns):
 #   Column                                                 Non-Null Count  Dtype  
---  ------                                                 --------------  -----  
 0   Geography                                              917 non-null    object 
 1   2017 Population                                        917 non-null    int64  
 2   2017 Median Age                                        917 non-null    float64
 3   2017 Total households                                  917 non-null    int64  
 4   2017 Median HHIncome                                   917 non-null    int64  
 5   2017 Average household size                            917 non-null    float64
 6   2017 Total families                                    917 non-null    int64  
 7   2017 Average family size                               917 non-null    float64
 8   2017 Percent owner-occupied housing units   [...]<jupyter_text>Columns - Selecting & Renaming<jupyter_code>df['Geography']
#you can also do it this way, but does not work if columns have spaces
df.Geography
#2 sets of square brackets gives you data (can be multiple columns), in df format
df[['Geography', '2017 Population']] #list inside the square brackets
#similar for just one column
df[['Geography']] #this way it'll come out as a df instead of series
#inplace = True permanently renames the column in your df, the default is false, where the change would not carry over
df.rename(columns ={'2017 Population': 'Population'}, inplace=True)
#check to see it's changed
df.head(3)
#you can also rename multiple columns at once, but all columns must be in the list, in the correct order
df.columns = ['geography', 'population', 'median_age',
             'total_households', 'median_hh_income', 
              'avg_hh_size','total_families',
             'avg_family_size', 'pct_owner_occupied_housing', 
             'pct_renter', 'pop_16_plus', 'unemp_rate']
#check to see it's changed
df.head(3)
#to change the index column name, you need:
df.index.name = 'fips'
#one more time - our new df - these columns are closer to typical column naming conventions!
df.head(3)<jupyter_output><empty_output><jupyter_text>Sorting Data<jupyter_code>#we can sort data by columns
df.sort_values('population').head()
#can also sort by descending values
df.sort_values('population', ascending=False).head()<jupyter_output><empty_output><jupyter_text>Running Functions on Columns<jupyter_code>#similar to before, we can run functions on columns
df['population'].mean()
df.population.max()
#astype lets us choose the type of output - in this case this gives us integer
df['median_age'].astype(int)
#other simple functions include:
#min()
#max()
#std()
#count()
#round()
#sample()
#we can chain together multiple functions
df['geography'].str.lower().head()
#similar to before, we can define our own functions to use on columns
def add_one(number):
    return number + 1
#to use new function, we have to use "apply" function 
df['median_age'].apply(add_one).head()<jupyter_output><empty_output><jupyter_text>Adding / Removing Columns<jupyter_code>#we can perform actions to columns and save them as new ones
df['pct_pop_16_plus'] = df['pop_16_plus'] / df['population']

#check if its there
df.head(3)
#we can also delete this new column - remember if we put inplace it will disappear
df.drop(columns = ['pct_pop_16_plus'], inplace=True)

#check if it happened 
df.head(3)<jupyter_output><empty_output><jupyter_text>Working with Rows<jupyter_code># we can select rows by calling indexed numbers
df[1:] #skips the first row
#or select based on conditions
df[df.population >= 500000]
#you can also figure out how many rows fit into your conditions
age_rest = df[(df.median_age >= 20) & (df.median_age < 30)] #& means and
len(age_rest)
# you can choose and / or
df[(df.median_age < 20) | (df.median_age > 50)] #this means OR
# you can also search for text - just specify string
df[df.geography.str.contains('NY')]<jupyter_output><empty_output><jupyter_text>Copying to a New DataFrame & Exporting your Data<jupyter_code>df_ny = df[df.geography.str.contains('NY')].copy() 
df_ny.sort_values('population', ascending = False).head()
#export data to a CSV
df_ny.sort_values('population', ascending = False).to_csv('ny_cities.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09478337754199823, 'fraction_numerical': 0.02493368700265252, 'mean_word_length': 3.3440860215053765, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '182325', 'license': 'CNRI-Python', 'n_tokens_mistral': 1799, 'n_tokens_neox': 1627, 'n_words': 632, 'path': '1_PythonBasics/3_intro_to_pandas.ipynb', 'repo_name': 'clcindyli/intro_to_python', 'size': '5655'}