<jupyter_start><jupyter_text>Read Data<jupyter_code>filename = '/home/sbarc/students/nash/data/IVT_MERRA2/MERRA2_arcatalog_1980-2015.csv'
df1 = pd.read_csv(filename, header=0, parse_dates=True)
df1<jupyter_output><empty_output><jupyter_text>That's a lot of data! Let's create a function that selects only the rows from the dataframe that have central latitude and longitude values within the specified bounding box.<jupyter_code>def subset_df(df, lat_min, lat_max, lon_min, lon_max):
    new_df = df[
        (df.loc[:, 'Centroid_lat'] >= lat_min) & (df.loc[:,'Centroid_lat'] <= lat_max) 
        & (df.loc[:, 'Centroid_lon'] >= lon_min) & (df.loc[:, 'Centroid_lon'] <= lon_max)]
    
    return new_df
## subset the dataframe using the function
## Region 1
R01 = subset_df(df1, lat_min=30.0, lat_max=40.0, lon_min=65.0, lon_max=75.0)

## Region 2
R02 = subset_df(df1, lat_min=25.0, lat_max=35.0, lon_min=75.0, lon_max=85.0)
R02<jupyter_output><empty_output><jupyter_text>Notice that we only have 299 rows now within subregion R02, down from 648240. Now I want to assign each AR "event" an ID number so that I can track the number of events. We consider each AR to be a part of the same "event" if the central latitude and longitude moves less than 5 degrees in any direction between the 6 hour timesteps.<jupyter_code>def new_event_id(df):
    ## Add columns to calculate difference in time, lat, and lon
    df.loc[:,'times_list'] = pd.to_datetime(df.loc[:,('Year', 'Month', 'Day', 'Hour')])
    df.loc[:,'time_diff'] = df['times_list'] - df['times_list'].shift(1)
    df.loc[:,'diff_lat'] =  df['Centroid_lat'] - df['Centroid_lat'].shift(1)
    df.loc[:,'diff_lon'] =  df['Centroid_lon'] - df['Centroid_lon'].shift(1)
    
    ## Add column of total IVT
    df.loc[:,"total_IVT"] = np.sqrt(np.square(df['meridional_ivt']) + np.square(df['zonal_ivt']))
    
    ## Add columns of old_idx and new_idx (to track the AR event)
    sLength = len(df['Day'])
    df.loc[:,'old_idx'] = pd.Series(np.arange(1, sLength+1), index=df.index)
    df.loc[:,'new_idx'] = pd.Series(np.arange(1, sLength+1), index=df.index)
    
    ## Create mask of AR objects that meet the criteria
    mask = ((df['time_diff'] <= timedelta(0, hours=6)) & (df['diff_lat'] <= 5.0) & (df['diff_lat'] >= -5.0)
        & (df['diff_lon'] <= 5.0) & (df['diff_lon'] >= -5.0))
    
    ## Mask new_idx objects that meet the criteria with NaN
    df['new_idx'] = df['new_idx'].mask(mask, np.nan)
    ## Propagate previous value forward to fill NaN
    df['new_idx'].fillna(method='ffill', inplace=True)
    
    return df
R01 = new_event_id(R01)
R02 = new_event_id(R02)
R01<jupyter_output>/home/nash/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py:362: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[key] = _infer_fill_value(value)
/home/nash/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py:543: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item] = s
/home/nash/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:21: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveat[...]<jupyter_text>Let's get some counts to confirm our indexing worked.<jupyter_code>print('Number of AR timesteps: ',  R01.old_idx.nunique())
print( '# of unique AR events:', R01.new_idx.nunique())
print('Mean of IVT for all events:', R01.total_IVT.mean())
print('Min IVT: ', R01['total_IVT'].min(), 'Max IVT:', R01['total_IVT'].max())<jupyter_output>Number of AR timesteps:  398
# of unique AR events: 283
Mean of IVT for all events: 180.79726350563539
Min IVT:  99.67993780094368 Max IVT: 266.11674129975364<jupyter_text>Create AR Event Frequency Barplot Group AR events into categories based on IVT strength and month, and then count the number of events<jupyter_code>def ar_ivt_categories(df):
    bins = np.arange(0, 1500, 250)
    ar_category = pd.cut(df['total_IVT'], bins=bins, precision=1, 
                         labels=['<250 kg m-1 s-1', '>250-500 kg m-1 s-1', 
                                 '>500-750 kg m-1 s-1', '>750-1000 kg m-1 s-1', 
                                 '>1000-1250 kg m-1 s-1'])
    new_df = df.groupby([(ar_category), 'Month']).nunique()
    ar_ivt = new_df['new_idx']
    
    return ar_ivt, bins
ivtcat_R01, ivt_bins_R01 = ar_ivt_categories(R01)
ivtcat_R02, ivt_bins_R02 = ar_ivt_categories(R02)
ivtcat_R01
df2 = pd.concat([ivtcat_R01.rename('R01'), ivtcat_R02.rename('R02')], axis=1)
df2<jupyter_output><empty_output><jupyter_text>Make the dataframe ["tidy"](http://shzhangji.com/blog/2017/09/30/pandas-and-tidy-data/) for easy plotting with seaborn.<jupyter_code>df2 = df2.reset_index()
df2 = pd.melt(df2, id_vars=['Month', 'total_IVT'], var_name='subregion', value_name='frequency')
df2
sns.catplot(x='Month', y='frequency', hue='subregion', col='total_IVT',
            data=df2, kind='bar')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14525655223329642, 'fraction_numerical': 0.047619047619047616, 'mean_word_length': 4.397410358565737, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '442985', 'license': 'MIT', 'n_tokens_mistral': 2178, 'n_tokens_neox': 2021, 'n_words': 481, 'path': 'examples/using_pandas_for_queries.ipynb', 'repo_name': 'itbaxter/pyclivac', 'size': '5418'}