<jupyter_start><jupyter_text>This notebook deals with the way to do away with the inclusion of land points in the horizontal interpolation, so we are looking to mask the data before we do the interpolation (also we would be doing it in 2 steps instead of one to achieve a slightly higher accuracy and smoothness of data points near the coast so we donot have a noisy signal in the NEMO output near the coastlines)  We choose to tackle the problem in 4 steps1) Mask out JP's land using JP's t mask (and u, v masks for his u and v data not in IC but in BC later)2) Do a gridata linear interpolation to all our grid points (WCVI). After this step we get NAN over a handful of WCVI points.3) Do a gridata nearest neighbour interpolation only to the places where we have NAN values right now in our WCVI domain.4) Multiply the whole thing with our tmask (and u, v masks for the u,v data) from one of our mesh_mask.nc<jupyter_code>import numpy as np
import netCDF4 as nc
import xarray as xr
from scipy.interpolate import griddata, interp1d
from salishsea_tools import nc_tools,viz_tools
import matplotlib.pyplot as plt
import matplotlib.cm as cm

from fancyimpute import KNN
#### Load JP's data 

fname = '/home/ssahu/saurav/JP_BC/cat_42_days_T.nc';

gridT = xr.open_dataset(fname);
gridT.keys
#### Load JP's mask

mask = nc.Dataset('/data/mdunphy/NEP036-N30-OUT/INV/mesh_mask.nc');

tmask_JP = mask.variables['tmask'][0,:,55:400,446:701];
umask_JP = mask.variables['umask'][0,:,55:400,446:701];
vmask_JP = mask.variables['vmask'][0,:,55:400,446:701];

print(tmask_JP.shape, umask_JP.shape, vmask_JP.shape)<jupyter_output>(50, 345, 255) (50, 345, 255) (50, 345, 255)<jupyter_text>STEP 1) Mask JP's data<jupyter_code>#### Slice out our domain of WCVI from JP and mask it at the same time saying the zero values as NAN

votemper_JP = np.array(gridT['votemper'][0,:,55:400,446:701]);
votemper_JP[...] = np.ma.masked_array(votemper_JP[...], mask = tmask_JP[...]);
votemper_JP[votemper_JP == 0] =['Nan'];

vosaline_JP = np.array(gridT['vosaline'][0,:,55:400,446:701]);
vosaline_JP[...] = np.ma.masked_array(vosaline_JP[...], mask = tmask_JP[...]);
vosaline_JP[vosaline_JP == 0] = ['Nan'];

glamt_bc_JP = np.array(gridT['nav_lon'][55:400,446:701]);
gphit_bc_JP = np.array(gridT['nav_lat'][55:400,446:701]);
deptht_JP = np.array(gridT['deptht'][:]);

vosaline_JP[:,0,0]
vosaline_JP[np.where(np.isnan(vosaline_JP))]=0;
vosaline_JP[:,0,0]<jupyter_output><empty_output><jupyter_text>STEP 2) First of the 2 step interpolation (linearly interpolate to our WCVI grid points)<jupyter_code>#### Load the WCVI points

fname_wcvi = '/ocean/ssahu/CANYONS/wcvi/grid/coordinates.nc'

with nc.Dataset(fname_wcvi, 'r') as coord:
    gphit_wcvi = coord.variables['gphit'][0,...];
    glamt_wcvi =  coord.variables['glamt'][0,...]; 
    
print((glamt_bc_JP[0,0],gphit_bc_JP[0,0]), (glamt_wcvi[0,0],gphit_wcvi[0,0]))
X = glamt_bc_JP.flatten();

Y = gphit_bc_JP.flatten();

points = (X[:],Y[:]);

xi = (glamt_wcvi.flatten(), gphit_wcvi.flatten());


votemper_ic = np.zeros((votemper_JP.shape[0], glamt_wcvi.shape[0], glamt_wcvi.shape[1]));
vosaline_ic = np.zeros((vosaline_JP.shape[0], glamt_wcvi.shape[0], glamt_wcvi.shape[1]));

for i,j in enumerate(votemper_JP[:,...]):
    votemper_ic[i,...] = np.reshape(griddata(points, votemper_JP[i,...].flatten(), xi, method= 'linear'), glamt_wcvi.shape)
    vosaline_ic[i,...] = np.reshape(griddata(points, vosaline_JP[i,...].flatten(), xi, method= 'linear'), glamt_wcvi.shape)


vosaline_ic[:,0,0]<jupyter_output><empty_output><jupyter_text>STEP 3) Fill up the Nan values in the votemper and vosaline arrays of WCVI using the nearest neighbours<jupyter_code>for i,j in enumerate(votemper_ic[:,...]):
    votemper_ic[i,...] = KNN(k=3).complete(votemper_ic[i,...]);
    vosaline_ic[i,...] = KNN(k=3).complete(vosaline_ic[i,...]);<jupyter_output>Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 4 missing, elapsed time: 0.005
Imputing row 1/100 with 4 missing, elapsed time: 0.005
Imputing row 1/100 with 4 missing, elapsed time: 0.005
Imputing row 1/100 with 4 missing, elapsed time: 0.005
Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 4 missing, elapsed time: 0.006
Imputing row 1/100 with 8 missing, elapsed time: 0.005
Imputing row 1/100 with 8 missing, elapsed time: 0.006
Imputing row 1/100 with 9 missing, elapsed time: 0.006
Imputing row 1/100 with 9 missing, elapsed time: 0.006
Imputing row 1/100 with 9 missing, elapsed time: 0.006
Imputing row 1/100 with 9 missing, elapsed time: 0.005
Imputing r[...]<jupyter_text>STEP 4) Multiply with our tmask from our mesh_mask to get the final data (This we will do after doing the vertical interpolation onto our depth levels and for now we will create the temporary nc file that we were making)<jupyter_code>i=4;

votemper_ic[np.where(np.isnan(votemper_ic[i,...]))] = np.reshape(griddata(points, votemper_ic[i,np.where(votemper_ic[~isnan(votemper_ic[i,...]))])].flatten(), xi, method= 'nearest')), \
                                                                   glamt_wcvi.shape)
votemper_ic[4,:,:]
votemper_JP.shape
glamt_wcvi.shape
votemper_ic.shape
len(Y)
np.where(np.isnan(votemper_ic_imuted[4,...]))
np.mean(votemper_ic[0,2:7,4:8])
np.mean(votemper_ic_imuted[4,...])
np.where(votemper_ic[votemper_ic == 'nan'])
%reset
np.mean(votemper_ic[0,...])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13966578715919084, 'fraction_numerical': 0.05716798592788039, 'mean_word_length': 4.985263157894737, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '83256', 'license': 'Apache-2.0', 'n_tokens_mistral': 2461, 'n_tokens_neox': 2193, 'n_words': 598, 'path': 'grid/Working_with_mask_during_HORIZONTAL_interpolation_land_points.ipynb', 'repo_name': 'UBC-MOAD/wcvi', 'size': '5685'}