<jupyter_start><jupyter_text>That's not great looking.  Let's use a jupyter notebook specific way to create a better looking table.<jupyter_code>from IPython.display import display
display(data)
# note - you can also just do this by writing the df name without calling print<jupyter_output><empty_output><jupyter_text>Notice the index (0, 1).  This is, by default, how you refer to a row.  Let's say we want the row that holds information about New Zealand.  Use iloc, for integer based location.<jupyter_code>display(data.iloc[1])<jupyter_output><empty_output><jupyter_text>But suppose we want to get information by country name (ie., look up as "Australia" rather than 0.  Use index_col to specify that a column’s values should be used as row headings. * Row headings are numbers (0 and 1 in this case).* Really want to index by country.* Pass the name of the column to read_csv as its index_col parameter to do this.<jupyter_code>data = pd.read_csv('data/gapminder_gdp_oceania.csv', index_col='country')
display(data)<jupyter_output><empty_output><jupyter_text>Now we can use **named location**, loc, rather than **integer location**, iloc<jupyter_code>display(data.loc["New Zealand"])<jupyter_output><empty_output><jupyter_text>A review of some utilities on pandas dataframes<jupyter_code>print(data.shape)
print(data.columns)
print(data.T)
print(data.T.loc["gdpPercap_2002"])
data.describe()<jupyter_output><empty_output><jupyter_text>**Exercises**Read the data in gapminder_gdp_americas.csv (which should be in the same directory as gapminder_gdp_oceania.csv) into a variable called americas and display its summary statistics.After reading the data for the Americas, use help(americas.head) and help(americas.tail) to find out what DataFrame.head and DataFrame.tail do.* What method call will display the first rows of this data?* What method call will display the last columns of this data? (Hint: you may need to change your view of the data.)<jupyter_code>americas = pd.read_csv('data/gapminder_gdp_americas.csv', index_col='country')
display(americas.head())
display(americas.tail())
americas_t = americas.T
display(americas_t.tail())<jupyter_output><empty_output><jupyter_text>**Back to selection and working with dataframes**read in the data set for europe<jupyter_code>data = pd.read_csv('data/gapminder_gdp_europe.csv', index_col='country')
display(data)<jupyter_output><empty_output><jupyter_text>To get a single value<jupyter_code>print(data.loc["Albania", "gdpPercap_1952"])<jupyter_output>1601.056136<jupyter_text>Use : on its own to mean all columns or all rows.(this will do the same as getting Albania without specifying columns)<jupyter_code>print(data.loc["Albania", :])<jupyter_output>gdpPercap_1952    1601.056136
gdpPercap_1957    1942.284244
gdpPercap_1962    2312.888958
gdpPercap_1967    2760.196931
gdpPercap_1972    3313.422188
gdpPercap_1977    3533.003910
gdpPercap_1982    3630.880722
gdpPercap_1987    3738.932735
gdpPercap_1992    2497.437901
gdpPercap_1997    3193.054604
gdpPercap_2002    4604.211737
gdpPercap_2007    5937.029526
Name: Albania, dtype: float64<jupyter_text>Selecting multiple columns with named columns and rowsNote that this is inclusive of start and end parameters, unlike lists.The behavior is different for iloc and loc (why do you think this is the case?)<jupyter_code>print(data.loc['Italy':'Poland', 'gdpPercap_1962':'gdpPercap_1972'])<jupyter_output>gdpPercap_1962  gdpPercap_1967  gdpPercap_1972
country                                                    
Italy           8243.582340    10022.401310    12269.273780
Montenegro      4649.593785     5907.850937     7778.414017
Netherlands    12790.849560    15363.251360    18794.745670
Norway         13450.401510    16361.876470    18965.055510
Poland          5338.752143     6557.152776     8006.506993<jupyter_text>Result of slicing can be used in further operations.<jupyter_code>print(data.loc['Italy':'Poland', 'gdpPercap_1962':'gdpPercap_1972'].max())<jupyter_output>gdpPercap_1962    13450.40151
gdpPercap_1967    16361.87647
gdpPercap_1972    18965.05551
dtype: float64<jupyter_text>**Use comparisons to select data based on value.*** Comparison is applied element by element.* Returns a similarly-shaped dataframe of True and False.<jupyter_code>subset = data.loc['Italy':'Poland', 'gdpPercap_1962':'gdpPercap_1972']
display(subset)
display(subset > 10000)<jupyter_output><empty_output><jupyter_text>That's useful, but suppose we actually want to do an operation on the actual numbers, not just know whether they are or are not in the subset...**Select values or NaN using a Boolean mask.*** A frame full of Booleans is sometimes called a mask because of how it can be used.<jupyter_code>mask = subset > 10000
display(subset[mask])<jupyter_output><empty_output><jupyter_text>this is wordier, but may make it easier to understand<jupyter_code>subset[subset > 10000]
subset[subset > 1000].describe()<jupyter_output><empty_output><jupyter_text>Pandas **vectorizing methods and grouping operations** are features that provide users much flexibility to analyse their data.For instance, let’s say we want to have a clearer view on how the european countries split themselves according to their GDP.* We may have a glance by splitting the countries in two groups during the years surveyed, those who presented a GDP higher than the european average and those with a lower GDP.* We then estimate a wealthy score based on the historical (from 1962 to 2007) values, where we account how many times a country has participated in the groups of lower or higher GDP  Exercise* GDP per capita for all countries in 1982.* GDP per capita for Denmark for all years.* GDP per capita for all countries for years after 1985.* GDP per capita for each country in 2007 as a multiple of GDP per capita for that country in * 1952.<jupyter_code>data['gdpPercap_1982']
data.loc['Denmark',:]
# notice there is no 1985...
data.loc[:,'gdpPercap_1985':]
data['gdpPercap_2007']/data['gdpPercap_1952']
# other operations
# corr
data.loc['Italy',:].corr(data.loc['France',:])
# Exercise - Find some countries with lower correlations, see what they are. 
#https://pandas.pydata.org/pandas-docs/stable/user_guide/computation.html
# Sometimes you want to see things side by side. 
# stacking, vertical and horizontal
# stack the DataFrames on top of each other
df_IP = data.loc['Italy':'Poland', 'gdpPercap_1952':'gdpPercap_1962']
df_PS = data.loc['Portugal':'Spain':, 'gdpPercap_1952':'gdpPercap_1962']
#verticalStack = pd.concat([data.loc[:, 'gdpPercap_1952':'gdpPercap_1962'], data.loc[:, 'gdpPercap_1982':'gdpPercap_1992']], axis=0,)
df_IP
df62_72.T
pd.concat([df52_62.T, df62_72.T], sort=False)
# place the DataFrames side by side
horizontalStack = pd.concat([data.loc['Denmark',:], data.loc['Portugal',:]], axis=1)
horizontalStack<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09220892909250072, 'fraction_numerical': 0.0789320105048147, 'mean_word_length': 5.655339805825243, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 61, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '353455', 'license': 'MIT', 'n_tokens_mistral': 2630, 'n_tokens_neox': 2215, 'n_words': 691, 'path': 'PandasIntro.ipynb', 'repo_name': 'geoffswc/IntroToPythonPart2', 'size': '6854'}