<jupyter_start><jupyter_text>Understanding .nc datatype.nc file contains geo information including longitude, latitude, time, and the parameter of interest such as sea surface temperature, emissions, or precipitation. To plot the graph, we need to match the dimension of the longitude and latitude of the map to the matrix of the parameter of interest. In the graph, we choose to take the average of the parameter of interest with respect to time.In this section, we are using netCDF4 to retrieve information from the .nc file and using matplotlib/basemap to plot the resulting graph. Sea surface temperature EDA (sst)The following is the EDA and plot of the sea surface temperature. In general, we can see the sea surface temperature is highest near the equator and the lowest near the poles from the plot. However, some of the sea surface temperatures appear to be affected near the coastline. For example, one of the most prominent fishing grounds in Canada, the Hibernia shelf off Newfoundland, displays colder ocean temperature when compared to the same latitude in the Atlantic. Here we are using the sst mean to visualize results, but the selected years are merged into our dataset before training.<jupyter_code>nc_file = '../data/sst_noaa.nc'
nc = netCDF4.Dataset(nc_file, mode='r', compression='gzip')
print(nc.variables.keys())

sst = ma.array(nc.variables['sst'][:,:,:])
lats = nc.variables['lat'][:]
lons = nc.variables['lon'][:]
sst1 = sst.data
sst_mean = np.mean(sst1,axis=0)

m = Basemap(projection='gall',llcrnrlat=-90,urcrnrlat=90,llcrnrlon=0,urcrnrlon=360, resolution='c')

lon, lat = np.meshgrid(lons, lats)
xi, yi = m(lon, lat)

fig,ax=plt.subplots(figsize=(20,10))

cs = m.contourf(xi,yi,np.squeeze(sst_mean),  cmap='magma', levels=300)
cb = fig.colorbar(cs, ax=ax, orientation='vertical', pad=0.01)
m.fillcontinents(color='white',lake_color='white',zorder=10)
m.drawcoastlines(linewidth=0.7, zorder=10)<jupyter_output>dict_keys(['lat', 'lon', 'sst', 'time', 'time_bnds'])<jupyter_text>Add sst as featureThe following section prepares the sea surface temperature features as a reduced grid to join our training data easily. Transforming .nc data from a longitude range of 0 - 360 to -180 - 180. Round all df longitude and latitude data to nearest 0.5 degrees with a resolution of 1 degree. Adjust time to Datetime formal with date (yyyy-mm-dd) only format.One of the limitations we find in adding features is the grid of longitude and latitude. All datasets have a different set of standards (different decimal places) when matching parameters to longitude and latitude grid. As such, our data can only be accurate to the largest grid format to join the df. Another limitation is that our local computer cannot handle large join operations.<jupyter_code>nc_sst = xr.open_dataset('../data/sst_noaa.nc')
df_sst = nc_sst.to_dataframe()
df_sst = df_sst[(df_sst['time_bnds'].dt.year > 2011) & (df_sst['time_bnds'].dt.year  < 2017)]
df_sst.reset_index(level=[0,1], inplace=True)
# def custom_round(x):
#     return 0.5 * round(float(x)/0.5)

# df_winter = df[(df.index.get_level_values('time') < '2015-02-28') | \
#                (df.index.get_level_values('time') > '2015-11-01')]
# df_winter = df_winter.groupby(level=['lat', 'lon']).mean()
# df_winter

df_sst['time_bnds'] = df_sst['time_bnds'].dt.to_period('M')
df_sst['lon'] = df_sst['lon'] - 180
df_sst_gb = df_sst.groupby(['lat', 'lon', 'time_bnds']).mean()
df_sst_gb.reset_index(level=[0,1,2], inplace=True)
df_sst_gb.to_csv('../data/sst_grid.csv')
df_sst_gb<jupyter_output><empty_output><jupyter_text>Test adding SST to AIS training data as additional featureThe section attempts to join the above sst to the trollers' dataset as an extra feature. The steps including dropping data that have 'is_fishing' = -1. Adjusting the time, longitude, latitude format to match the sst dataframe. Then both df is joined together with the 'is_fishing' column as the label and other columns as features.<jupyter_code>path = '../data/trollers.csv'

fishing_df = pd.read_csv(path)
fishing_df = fishing_df[fishing_df['is_fishing'] > -0.5]
fishing_df['is_fishing'] = [0 if x < 0.3 else 1 for x in fishing_df['is_fishing']]
fishing_df = fishing_df[['is_fishing', 'lat', 'lon', 'course', 'speed', 'timestamp', 'distance_from_shore', 'distance_from_port', 'mmsi', 'source']]
def custom_round(x): 
    return 0.5 + np.floor(float(x))

fishing_df['adjust_lat'] = fishing_df['lat'].apply(lambda x: custom_round(x))
fishing_df['adjust_lon'] = fishing_df['lon'].apply(lambda x: custom_round(x))
fishing_df["adjust_time_date"] = pd.to_datetime(fishing_df['timestamp'], unit='s').dt.date
fishing_df["adjust_time"] = pd.to_datetime(fishing_df["adjust_time_date"]).dt.to_period('M')
df_all = pd.merge(fishing_df, df_sst_gb,  how='left', \
                  left_on=['adjust_lat','adjust_lon', 'adjust_time'], \
                  right_on = ['lat','lon', 'time_bnds'])
df_all = df_all.drop(columns=['lat_y', 'lon_y', 'time_bnds'])
df_all.to_csv('../data/feature_1.csv')
df_all<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1255196990694912, 'fraction_numerical': 0.017422292615323697, 'mean_word_length': 5.552529182879377, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '344142', 'license': 'MIT', 'n_tokens_mistral': 1762, 'n_tokens_neox': 1676, 'n_words': 521, 'path': 'preprocessing/feature_sst.ipynb', 'repo_name': 'WilliamZipanHe/CMPT733-Fish-Activity-Detection', 'size': '5051'}