<jupyter_start><jupyter_text>Dataframe: Tabular dataset -with different types of data- built on numpy + matplotlib `dataframe.values` returns a 2d np array containing a list of lists (a list of "rows")* access with square brackets`[]`for columns (that returns a pandas series); double squared brackets `[[]]`returns dataframe    * to select a col name without spaces dot notation can be used.* advanced:    * loc    * iloc  Differences with basic python/other libraries: Loops* `for val in df` iterates over columns*  to iterate over rows `for i, row in df.iterrows()` Boolean conditions to subset* & instead of and* | instead of or<jupyter_code># import library
import pandas as pd<jupyter_output><empty_output><jupyter_text>Basics<jupyter_code># import dataset from external file to dataframe
df = pd.read_csv('../datasets/clean/simpledf_applevalues.csv', index_col=0)
# display the head of the data
display(df.head()) #df.tail() will print the last rows
# Print information about df
print(df.info()) #number of rows, cols, dtype in each col, number of missing values...
display(df.index, type(df.index)) # the row index (can be accessed as a list)
display(df.columns, type(df.columns)) # the column index
# Sort df by apple production in Brazil (value in column Br)
df_br = df.sort_values('Br') # by default values are sorted in ascendent order
df_br.head()
# Filter for rows where apple production in Brazil (value in column Br) is greater than 20000
br_gt_20k = df[df['Br'] > 20000]
br_gt_20k.head()
df.sum(axis=0)
# Add total col for the four countries
df['total'] = df.sum(axis=1) # axis 1: sum 'through the columns'; axis 0: through the rows
# if executing that several times the sum is performed several times
df.head()
# Add total col for the south of europe as sum 
df['seu']=df['Sp'] + df['Br'] # also df = df.assign(seu=df['Sp'] + df.Br)
df.head()
# Add prop_seu col as proportion of total
df['prop_seu'] = df['seu']/df['total']
df.head()<jupyter_output><empty_output><jupyter_text>Sumary statistics<jupyter_code>#another dataset
pop = pd.read_csv('../datasets/clean/pop.csv', index_col=0, header=0)
display(pop.head())
display(pop.shape) # (rows, columns)
# Print the mean of the value for the rural population
print(pop[pop.Element == 'Rural population']['Value'].mean(), '\n')

# Print the median of the value for the rural population
print(pop[pop.Element == 'Rural population'].median(), '\n') #sumary statistics of all columns with numeric values

# Print the maximum of the year column
print(pop.Year.max(), '\n')

# Print the minimum of the year column
print(pop.Year.min(), '\n')

# Print the cumulative sum/ the cumulative max
print(pop[pop.Element == 'Rural population']['Value'].cumsum().head(), '\n', 
     pop[pop.Element == 'Rural population']['Value'].cummax().head(), '\n')

# Print the 10th percentile and the 1st quartile of the value column
print('10th percentile:', pop.Value.quantile(0.1),'\n1st quartile:', pop.Value.quantile(0.25))
# Create custom function 
def iqr(column):
    return column.quantile(0.75) - column.quantile(0.25)

# With the first dataset, update to print IQR and median of temperature_c, fuel_price_usd_per_l, & unemployment
print(df[['Br', 'Pt', 'Sp', 'UK']].agg([iqr, 'median','mean']), '\n')
print(df[['Br', 'Pt', 'Sp', 'UK']].agg([iqr, 'median', sum, min, max]))
# Import numpy and use its function to do the above calculations
import numpy as np
print(df[['Br', 'Pt', 'Sp', 'UK']].agg([iqr, np.median, np.mean]))<jupyter_output>Br             Pt             Sp            UK
iqr     917592.000000  162755.000000  333941.500000  179750.00000
median  526904.000000  229794.000000  730000.000000  346600.00000
mean    552902.101695  193667.830508  745146.135593  346803.59322<jupyter_text>More subseting and calculations<jupyter_code># Subset for rows of rural and urban population
pop_ru = pop[pop['Element'].isin(['Rural population','Urban population'])] # isin takes a list of values
display(pop_ru.head())
display(pop_ru.shape) 
# Drop duplicate year/value combinations
pop_ru2 = pop.drop_duplicates(['Area','Element'])
display(pop_ru2.head())
#check conditions (number of rows)
display(pop_ru.shape[0] == pop_ru2.shape[0], pop.shape[0] == pop_ru2.shape[0])

# Subset the rows for Rural population and drop duplicate countries
ruralpop_per_country = pop[pop.Element =='Rural population'].drop_duplicates('Area',
                                                                             ignore_index=True)[['Area', 'Year', 'Value']]
display(ruralpop_per_country.sort_values('Year', ascending=False).head())
# Count the number of measurements per element
pop_counts = pop.Element.value_counts()
print(pop_counts)
# Get the proportion of measurements per element in the dataset
pop_props = pop.Element.value_counts()/len(pop.Element.index)
print(pop_props)
# Get the proportion of measurements per element in the dataset and sort
pop_props_sorted = pop.Element.value_counts(sort=True, normalize=True) #same calculation as above
print(pop_props_sorted)
# Calc total weekly sales
pop_all = pop.Value.sum() #used in a series instead of df, no need to specify axis
pop_all
# Subset for Rural population, calc total population value
ruralpop = pop[pop.Element =='Rural population']["Value"].sum()

# Subset for Urban population, calc total population value
urbanpop = pop[pop.Element =='Urban population']["Value"].sum()

# Subset for female population, calc total population value
fempop = pop[pop.Element =='Total Population - Female']["Value"].sum()

# Subset for male population, calc total population value
malepop = pop[pop.Element =='Total Population - Male']["Value"].sum()

# Get proportion for each type
pop_propn_by_element = [ruralpop, urbanpop, fempop, malepop] / pop_all
print(pop_propn_by_element)
#same as before with less code: groupby()
pop_by_element = pop.groupby('Element')['Value'].sum()
pop_by_element
# Get proportion for each type
pop_propn_by_element2 = pop_by_element / sum(pop.Value)
print(pop_propn_by_element2, '\n', pop_propn_by_element) # both calculations give the same results (different order)<jupyter_output>Element
Rural population                 0.194021
Total Population - Both sexes    0.333419
Total Population - Female        0.165108
Total Population - Male          0.168266
Urban population                 0.139185
Name: Value, dtype: float64 
 [0.1940213  0.13918546 0.16510831 0.16826564]<jupyter_text>Index<jupyter_code># subset by setting a range
display(df.loc['1975':'1979'])
display(df.loc['1975':'1979', 'Pt':'Sp'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12549557792009758, 'fraction_numerical': 0.04239097285757853, 'mean_word_length': 4.861483467381591, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '447350', 'license': 'CC0-1.0', 'n_tokens_mistral': 2256, 'n_tokens_neox': 2035, 'n_words': 670, 'path': 'notes_on_data_manipulation/notes_on_Pandas1.ipynb', 'repo_name': 'blazaropinto/my_notes_learning_DS', 'size': '6558'}