<jupyter_start><jupyter_text>Import Libraries<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Load data from S3<jupyter_code># define CloudFront domain name (to access S3) and data

cloudfront = 'https://d3cu2src083cxg.cloudfront.net'
data_key = 'rfmo_query_result.csv'

data_location = '{}/{}'.format(cloudfront, data_key)

# load select columns
df_rfmo = pd.read_csv(data_location, usecols = ['year', 'name_rfmo', 'name_comm_group', 'name_fishing_entity', 'name_sector_type', 'catch_sum', 'real_value'])

# rename columns as needed
df_rfmo.rename(columns = {"name_rfmo": "rfmo", "name_comm_group": "commercial_group", "name_fishing_entity": "fishing_entity", "name_sector_type": "sector_type"}, 
               inplace = True)

# print shape (rows, columns) of dataframe
df_rfmo.shape

# sample data

df_rfmo.head(3)

# dataframe information

df_rfmo.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1027995 entries, 0 to 1027994
Data columns (total 7 columns):
 #   Column            Non-Null Count    Dtype  
---  ------            --------------    -----  
 0   rfmo              1027995 non-null  object 
 1   year              1027995 non-null  int64  
 2   commercial_group  1027995 non-null  object 
 3   fishing_entity    1027995 non-null  object 
 4   sector_type       1027995 non-null  object 
 5   catch_sum         1027995 non-null  float64
 6   real_value        1027995 non-null  float64
dtypes: float64(2), int64(1), object(4)
memory usage: 54.9+ MB<jupyter_text>Data analysis<jupyter_code># Function definitions for simple data analysis scenarios

def average_catch_and_value_by_year(start_year = 2010, end_year = 2015):
    return df_rfmo[(df_rfmo['year'] >= start_year) & (df_rfmo['year'] <= end_year)].groupby('year', as_index = False).agg({'catch_sum':'mean', 'real_value':'mean'}).round(2).copy()

def average_catch_and_value_by_region():
    return df_rfmo.groupby('fishing_entity', as_index = False)[['catch_sum', 'real_value']].mean().copy()

def average_catch_and_value_by_year_and_region(start_year = 2010, end_year = 2015, fishing_entity = 'Canada'):
    return df_rfmo[(df_rfmo['year'] >= start_year) & (df_rfmo['year'] <= end_year) & (df_rfmo['fishing_entity'] == fishing_entity)].groupby('year', as_index = False).agg({'catch_sum':'mean', 'real_value':'mean'}).round(2).copy()

def catch_and_value_by_commercial_groups():
    return df_rfmo.groupby('commercial_group', as_index = False)[['catch_sum', 'real_value']].sum().copy()

def catch_and_value_by_commercial_groups_and_year(start_year = 2010, end_year = 2015):
    return df_rfmo[(df_rfmo['year'] >= start_year) & (df_rfmo['year'] <= end_year)].groupby(['year', 'commercial_group'], as_index = False)[['catch_sum', 'real_value']].sum().copy()<jupyter_output><empty_output><jupyter_text>Average catch sum & real value by year (parameters: start year, end year)<jupyter_code># Aggregate the data by Year, and display the result dataframe

df_table1 = average_catch_and_value_by_year(2005, 2010)
df_table1<jupyter_output><empty_output><jupyter_text>Average catch sum & real value by fishing entity or region<jupyter_code># Aggregate the data by Fishing entity, and display the result dataframe

df_table2 = average_catch_and_value_by_region()
df_table2.head(10) # display only first 10 records<jupyter_output><empty_output><jupyter_text>Average catch sum & real value by year and fishing entity (parameters: start year, end year, region)<jupyter_code># Aggregate the data by Year, filter the data to a Region, and display the result dataframe

df_table3 = average_catch_and_value_by_year_and_region(2010, 2015, 'Iceland')
df_table3<jupyter_output><empty_output><jupyter_text>Total catch sum & real value of all commercial groups<jupyter_code># Aggregate the data by Commercial group, and display the result dataframe

df_table4 = catch_and_value_by_commercial_groups()
df_table4<jupyter_output><empty_output><jupyter_text>Total catch sum & real value of all commercial groups by year (parameters: start year, end year)<jupyter_code># Aggregate the data by Year and Commercial group, and display the result dataframe

df_table5 = catch_and_value_by_commercial_groups_and_year(2010, 2011)
df_table5<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11822660098522167, 'fraction_numerical': 0.037532254281022755, 'mean_word_length': 5.768253968253968, 'pattern_counts': {'":': 4, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '581776', 'license': 'MIT', 'n_tokens_mistral': 1568, 'n_tokens_neox': 1433, 'n_words': 333, 'path': 'notebooks/03 - Data_analysis_examples.ipynb', 'repo_name': 'NicBailly/Sea-Around-Us-1', 'size': '4263'}