<jupyter_start><jupyter_text>Exploratory Data Analysis (continued)<jupyter_code>from icepyx import icesat2data as ipd
import os
import shutil
from pathlib import Path
import h5py
import pandas as pd
import random
import seaborn as sns

%pylab inline

# Fixing figure output for all figures
rcParams['axes.labelsize'] = 9
rcParams['xtick.labelsize'] = 9
rcParams['ytick.labelsize'] = 9
rcParams['legend.fontsize'] = 9
rcParams['figure.dpi'] = 150
rcParams['savefig.dpi'] = 200

from utils_atl03 import *<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>Load the ATL06+VIIRS dataframe.<jupyter_code>atl6 = pd.read_csv("dataframe20200702_deltatime")
atl6.head()
atl6.columns<jupyter_output><empty_output><jupyter_text>1) Preliminary observations on VIIRS integration __NOTE__: A lot of missing `Integer_Cloud_Mask`.<jupyter_code>missing_mask = round(atl6.Integer_Cloud_Mask.isnull().sum()/atl6.shape[0]*100)
print('About %d%% of data do not have a VIIRS match.' % missing_mask)<jupyter_output>About 63% of data do not have a VIIRS match.<jupyter_text>We could include cloud masks from other reliable satellites to try reduce this high level of missingness (probably due to the fact that in many cases the time difference between data from VIIRS and ICESat2 is non-negligible).  For now, I focus only on data with VIIRS match.<jupyter_code>atl6.rename(columns={'Unnamed: 0':'id'}, inplace=True)
atl6 = atl6.set_index("id")
atl6 = atl6[ ~ atl6.Integer_Cloud_Mask.isnull()]
atl6.head()<jupyter_output><empty_output><jupyter_text>The variable `VIIRS_offset` describes the time difference in minutes between VIIRS and ICESat2 data. We can see that in our dataframe such variable only takes values about -20 or sligthly above 40. This can be an issue for cloud mask accuracy.<jupyter_code>time_diff=atl6.groupby('VIIRS_offset').size()
atl6.hist('VIIRS_offset',bins=200)<jupyter_output><empty_output><jupyter_text>2) Exploring relation between `Integer_Cloud_Mask` and `cloud_flg_asr` Using the graphical model with this dataframe, we obtain the following output.<jupyter_code>graphical_model=plt.imread('output_graph.png')
plt.figure()
plt.imshow(graphical_model)
plt.axis('off')<jupyter_output><empty_output><jupyter_text>First, let's focus on the relation between `Integer_Cloud_Mask` and ATL06 flags. The strongest relation is with `cloud_flg_asr`. Note that:* The middle row in this cell, corresponding to `Integer_Cloud_Mask` = 1, is greysh probably because we have really few observations with this label.* In general, the relation does not seem really strong, probably because of the non-negligible `VIIRS_offset` values.* The colours in the cell make sense:    * `Integer_Cloud_Mask` = 3 (clear) is positively correlated with small values of `cloud_flg_asr` (clear) and negatively correlated with large values of `cloud_flg_asr` (cloudy).    * `Integer_Cloud_Mask` = 0 (cloudy) is negatively correlated with small values of `cloud_flg_asr` (clear) and positively correlated with large values of `cloud_flg_asr` (cloudy). Let's explore more on the relation between `Integer_Cloud_Mask` and `cloud_flg_asr`. In the following plot we observe the histogram of `cloud_flg_asr` values grouped by `Integer_Cloud_Mask`. There is a meaningful connection between cloudy sky label in VIIRS (0.0) and in ATL06 (5), and between clear sky label in VIIRS (3.0) and in ATL06 (1, 2). There are some results that are counterintuitive but they can be partly due to the time mismatch between VIIRS and ATL06, and partly to possible wrong catches of `cloud_flg_asr` that labels as clouds other anomalies.<jupyter_code>g = sns.FacetGrid(atl6, col="Integer_Cloud_Mask")
g.map(plt.hist, "cloud_flg_asr")
atl6.groupby(['Integer_Cloud_Mask','cloud_flg_asr']).size()<jupyter_output><empty_output><jupyter_text>Let's look at the spatial distribution of the cloud labels for `Integer_Cloud_Mask` and `cloud_flg_asr`. The eastern area seems to be mostly clear and it correspond to the data with `VIIRS_offset` of about 40 minutes.<jupyter_code>ax1 = atl6.plot.scatter(x='lon', y='lat', c='Integer_Cloud_Mask', s = 2,  colormap='inferno_r')
ax2 = atl6.plot.scatter(x='lon', y='lat', c='cloud_flg_asr', s = 2,  colormap='inferno')<jupyter_output><empty_output><jupyter_text>Let's just consider the western part, where VIIRS data are 20 minutes earlier than ATL06 data. From the scatterplot, it seems like the top cloud in the VIIRS image moved a bit northwise in the 20-minute time gap, whereas what is going on in the bottom part is more complicated to understand.<jupyter_code>ax1 = atl6.loc[atl6['VIIRS_offset']<0].plot.scatter(x='lon', y='lat', c='Integer_Cloud_Mask', s = 2,  colormap='inferno_r')
ax2 = atl6.loc[atl6['VIIRS_offset']<0].plot.scatter(x='lon', y='lat', c='cloud_flg_asr', s = 2,  colormap='inferno')<jupyter_output><empty_output><jupyter_text>ATL06 provides another cloud flag, `cloud_flg_atm`, which counts the number of layers found from the backscatter profile.Something that would be interesting to understand is: what are the points that correspond to one or more layers and are located in a non-cloudy area? Blowing snow? Crevasses? See for example the light grey points in the eastern tracks.<jupyter_code>ax1 = atl6.plot.scatter(x='lon', y='lat', c='Integer_Cloud_Mask', s = 2,  colormap='inferno_r')
ax2 = atl6.plot.scatter(x='lon', y='lat', c='cloud_flg_atm', s = 2)<jupyter_output><empty_output><jupyter_text>The `layer_flag` in ATL06 combines `cloud_flag_atm`, `cloud_flag_asr`, and `bsnow_con` to flag whether cloud or blowing snow are likely (value 1) or not (value 0). We can compare it with the `Integer_Cloud_Flag` derived from VIIRS. We can see that, ignoring some mismatch probably due to the time difference, the results are really similar.<jupyter_code>ax1 = atl6.plot.scatter(x='lon', y='lat' , marker='_', c='Integer_Cloud_Mask', s = 2,  colormap='inferno_r')
ax2 = atl6.plot.scatter(x='lon', y='lat', c='layer_flag', s = 2,  colormap= 'inferno')<jupyter_output><empty_output><jupyter_text>The next plot highlights in yellow the areas where the `layer_flag` spots a cloud and the `Integer_Cloud_Mask` doesn't, and vice versa. We can see that most of such yellow points are in the areas where we conjectured a cloud movement over time. It would be interesting to check whether there are sparse yellow points in the dense violet areas. That would be unlikely due to cloud movement, but to incorrect classification of the `layer_flag`.<jupyter_code>diff = (((atl6['Integer_Cloud_Mask']==0.0) & (atl6['layer_flag']!=1.0)) | ((atl6['Integer_Cloud_Mask']==3.0) & (atl6['layer_flag']!=0.0)))
plt.scatter(atl6.lon, atl6.lat, s=2, c=diff)<jupyter_output><empty_output><jupyter_text>There are no yellow points in the eastern area with latitude lower than 67.6.<jupyter_code>east = atl6.loc[atl6['VIIRS_offset']>0]
diff_east = (east['Integer_Cloud_Mask']==0.0) & (east['layer_flag']!=1.0) | ((east['Integer_Cloud_Mask']==3.0) & (east['layer_flag']!=0.0)) 
east.loc[diff_east & (east['lat']<67.6)]
#plt.scatter(east.lon, east.lat, s=2, c=diff_east, cmap='Greys')<jupyter_output><empty_output><jupyter_text>Nothing anomalous is going on in the clear area at the center of the western area.<jupyter_code>west = atl6.loc[atl6['VIIRS_offset']<0]
diff_west = (west['Integer_Cloud_Mask']==0.0) & (west['layer_flag']!=1.0) | ((west['Integer_Cloud_Mask']==3.0) & (west['layer_flag']!=0.0)) 
plt.scatter(west.lon, west.lat, s=2, c=diff_west, cmap='Greys',alpha=0.1)
sns.catplot(x="cloud_flg_asr", hue="cloud_flg_atm", col="Integer_Cloud_Mask", kind="count", data=atl6)
sns.catplot(data = atl6.loc[atl6['Integer_Cloud_Mask'] == 0.0], hue = 'cloud_flg_atm', x = 'cloud_flg_asr', kind ='count')
sns.catplot(data = atl6.loc[atl6['Integer_Cloud_Mask'] == 3.0], hue = 'cloud_flg_atm', x = 'cloud_flg_asr', kind ='count')
atl6.groupby(['Integer_Cloud_Mask','cloud_flg_atm']).size()
atl6.groupby(['cloud_flg_asr','cloud_flg_atm']).size()
g = sns.FacetGrid(atl6, col="Integer_Cloud_Mask")
g.map(plt.hist, "layer_flag")<jupyter_output><empty_output><jupyter_text>3) Other ATL06 flags * __q_flag__: 0 corresponds to no quality issues, 1 to potential quality issues. As we could expect, no area that is labeled as clear has quality issues, whereas some areas that have `layer_flag`=1 have potential quality issues.<jupyter_code>g = sns.FacetGrid(atl6, col="layer_flag")
g.map(plt.hist, "q_flag")<jupyter_output><empty_output><jupyter_text>* __h_li__: land-ice segment height We can see that there are outliers for which `h_li` is extremely high. We expect them to be data of poor quality.<jupyter_code>atl6.groupby('layer_flag').h_li.mean()
atl6.h_li.describe()<jupyter_output><empty_output><jupyter_text>No good quality data has `h_li` above 2.5e+03.<jupyter_code>atl6.loc[(atl6['h_li']>2.5e+03) & (atl6['q_flag']<1)]<jupyter_output><empty_output><jupyter_text>On the other hand, there are many low quality data that have `h_li` equal to 3.402824e+38.<jupyter_code>print(atl6.loc[(atl6['h_li']>2.5e+03) & (atl6['q_flag']==1)].h_li.min(), atl6.loc[(atl6['h_li']>2.5e+03) & (atl6['q_flag']==1)].h_li.max())
s = np.shape(atl6.loc[ (atl6['h_li']>2.5e+03) & (atl6['q_flag']==1) ])[0] 
t = np.shape(atl6.loc[ atl6['q_flag']==1 ])[0]
print('A fraction of %f low quality data have extermely high h_li.' %(s/t))
atl6.plot.scatter(x='lon', y='lat' , c='h_li', s = 2,  colormap='inferno')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12676355150100774, 'fraction_numerical': 0.025246632014426645, 'mean_word_length': 6.850124895920066, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '215844', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 3485, 'n_tokens_neox': 3293, 'n_words': 941, 'path': 'examples/EDA2.ipynb', 'repo_name': 'espg/CloudMask', 'size': '9427'}