<jupyter_start><jupyter_text>1.0 Import Modules First, you need to import the following modules. Make sure the sys.path.append('') points to the location on your computer that you saved pyce_tools.<jupyter_code>import pandas as pd
import os
import datetime
sys.path.append('C:\\Users\\trueblood\\projects\\pyce_tools')
import pyce_tools.pyce_tools as pt<jupyter_output>Cool party!<jupyter_text>For this tutorial, we'll take a raw data file of aerosol sample type and preprocess it, then do some basic calculations and analysis. The first step is to pre-process all the data. 2.0 Pre-processing of INP Data 2.1 Raw Data Calculation Before we do anything, we need to calculate the raw blank data into a calculated report file.<jupyter_code># Create blank file
issues = None
type_ = 'aerosol'
process = 'uf'
collection_date = '06042020 '
sample_name = 'Bubbler 3 stage BLANK pc'
analysis_date = '01/01/21'
num_tubes = 26
vol_tube = 0.2
rinse_vol = 20
size = 'sub'
location = 'bubbler'
raw = pt.calculate_raw_blank(type_, process, location, sample_name, collection_date,
                  analysis_date, issues, num_tubes, vol_tube, rinse_vol, size)<jupyter_output>...Raw blank data calculated!<jupyter_text>Now that we've created our blank file, we can use it to calculate a raw data file from a sample. Note that we pass the file into the 'blank_source' parameter below. Pyce Tools uses the metadata that you pass when calling it to calculate the report files, so it's important you fill everything in.<jupyter_code># Calculate aerosol sample
issues = None
blank_source = '..\\data\\interim\\IN\\calculated\\blank\\bubbler_blank_uf_sub_060420_calculated.xlsx'
type_ = 'aerosol'
size = 'sub'
location = 'bubbler'
process = 'uf'
sample_name = 'Day 01 bubbler 3 stage PC'
collection_date = '17032020 11h25'
sample_stop_time = '18032020 09h45'
analysis_date = '01/01/2021'
num_tubes = 26
vol_tube = 0.2
flow_start = 10.50
flow_stop = 9.35
rinse_vol = 20

raw = pt.calculate_raw(blank_source, type_, location, process, sample_name, collection_date, analysis_date, issues, num_tubes, vol_tube, rinse_vol, size, flow_start, flow_stop, sample_stop_time)<jupyter_output>...IN data calculated!
Calculated report file saved to ..\data\interim\IN\calculated\aerosol\bubbler\aerosol_bubbler_uf_sub_170320_1125_calculated.xlsx.<jupyter_text>Running the above code will create a calculated report file in which the blank corrected values are given. Take a minute to open the report file now (file path is given above) and have a look yourself. Also be sure to check the 'tubeholderfilling' tab which gives an overview of all the cells. Sometimes unexpected things happen during analysis, like the LED turns off mid experiment. You should compare this with the final image from the LINDA to ensure the automated process correctly identified frozen vs nonfrozen cells. For example, during this experiment, the LED turned off several times. This caused the program to assume all cells froze. Go into the 'freezepointdetection' tab. Whenever you see several rows where EVERY column has indicated freezing occurred, you can assume it's because the LED turned off. To fix this, simply delete those rows. Also note a separate tab for heated vs unheated samples is given (e.g., summary_UF_UH and summary_UF_H). The code is written to assume the bottom 26 cells are unheated and the top 26 cells are heated. You can alter this as needed. Each summary tab has calculations for blank corrected concentrations and uncertainty as well as metadata. Once you're familiar with this report file you can come back here and begin the next step. 2.2 Clean Calculated Report File Cleaning the available calculated report files is as sample as calling clean_calculated_in() function and passing the parameters for INP type and location. The script will automatically find all relevant files in the folder and combine them into a single time series.<jupyter_code># Create cleaned files of INP and uncertainty
pt.clean_calculated_in(type_='aerosol',location='bubbler')
pt.calculate_wilson_errors(project='tutorial', location='bubbler', type_='aerosol', n=26)<jupyter_output><empty_output><jupyter_text>2.3 Load Cleaned Data and Final Preprocessing Load the cleaned INP data and uncertainty files. Do a final bit of preprocessing. This is unique to each experiment but I've given an example below.<jupyter_code># Load cleaned data and create time timeseries index in New Zealand Standard Time
inp_bubbler = pd.read_csv('..\\data\\interim\\IN\\cleaned\\combinedtimeseries\\aerosol\\bubbler_17032020_17032020.csv', index_col='datetime')
inp_bubbler.index = pd.to_datetime(inp_bubbler.index, format='%d%m%Y %Hh%M')
inp_bubbler= inp_bubbler.tz_localize('NZ')

bubbler_error = pd.read_csv('..\\data\\interim\\IN\\cleaned\\combinedtimeseries\\aerosol\\bubbler_170320_170320wilson_error.csv', index_col='datetime', parse_dates=True)
bubbler_error.index=pd.to_datetime(bubbler_error.index, dayfirst=True, format='%d%m%Y %Hh%M')
bubbler_error= bubbler_error.tz_localize('NZ')

inp_bubbler.head()
# Create melted dataframes with INP/m^3
inp_bubbler_melt=inp_bubbler.reset_index().melt(id_vars = ['datetime','start_date','stop_date','size','process', 'type','location','filtered'], var_name = 'temp', value_name='inp/l')
inp_bubbler_melt['inp/m^3'] = inp_bubbler_melt['inp/l']*1000

bubbler_error_melt = pd.melt(bubbler_error.reset_index(), id_vars = ['datetime','time','process','filtered','location','type','value_name', 'size'], var_name = 'temp', value_name='error (inp/l)')
bubbler_error_melt=bubbler_error_melt.drop(columns='time')
bubbler_error_melt['error (inp/m^3)'] = bubbler_error_melt['error (inp/l)']*1000
inp_bubbler_melt
# create a pivoted uway error dataframe then merge with the actual data
bubbler_error_pivot = bubbler_error_melt.pivot_table(index=['process','datetime','temp','type','location','filtered'], values=['error (inp/l)'], columns='value_name')
bubbler_df = pd.merge(inp_bubbler_melt,bubbler_error_pivot.reset_index(), on=['datetime','temp','process','type','location','filtered'])

# clean column names
bubbler_df = bubbler_df.rename(columns={(
    'error (inp/l)','IN/L_lower'):'IN/L_lower',
    ('error (inp/l)','IN/L_upper'):'IN/L_upper',
    ('error (inp/l)','error_minus_y'):'error_minus_y',
    ('error (inp/l)','error_y'):'error_y'})

bubbler_df.set_index('datetime', inplace=True)
bubbler_df<jupyter_output><empty_output><jupyter_text>3.0 Analysis Now we'll begin analysis of our INP data. To start, we'll calculate the surface area normalized INP concentrations. Some of the code below may seem complicated, but just follow along and it will make sense. In the future I'd like to make this process a bit less 'hacky'. 3.1 Calculate Surface Area Normalized INP Concentrations 3.1.1 Clean Inverted SMPS Data<jupyter_code># Define constants
instr= 'scanotron'

# Note that we have placed the inverted data in the 'interim' folder. This is because technically the raw data from the scanotron has already been preprocessed when we inverted it.
inpath= "..\\data\\interim\\"+instr+"\\inverted\\pro\\" 
outpath = '..\\data\\interim\\'+instr+'\\combinedtimeseries\\'

# Call the clean_inverted function to prepare smps data. Give the input path, nbins, and outputh path.
df, outName, dLogDp=pt.clean_inverted(inpath=inpath, nbins=26, outpath=outpath)<jupyter_output><empty_output><jupyter_text>3.1.2 Prepare Cleaned SMPS Data<jupyter_code># load scanotron data that was cleaned using pt.clean_inverted. Look inside the combinedtimeseries folder for the dates.
dates = '2020-03-16_2020-03-18'
instr = 'scanotron' 

dN, dNdLogDp_full = pt.load_scano_data(dates, instr)<jupyter_output><empty_output><jupyter_text>This next step is mandatory. We want to average SMPS across the times that we were sampling INP. To do this, we create a dictionary where the keys are the day number or analysis number and the values are the INP sampling time. For example, here we have one INP sample that we collected from 11:25 on 2020-03-17 until 09:45 on 2020-03-18.<jupyter_code>in_time_dict = {    
    '17': ['2020-03-17 11:25','2020-03-18 09:45'],
    }<jupyter_output><empty_output><jupyter_text>Next, we will loop through the dictionary we just created and extract the corresponding SMPS data.<jupyter_code># create empty smps dictionary
smps_dict = {}
# loop through each key in the in_time_dict and index those times from the dNdlogDp_full dataframe and save each of them as a dictionary in the smps dictionary
for day in in_time_dict:
    smps_dict[day] = dNdLogDp_full.loc[in_time_dict[day][0]:in_time_dict[day][1]]<jupyter_output><empty_output><jupyter_text>The following piece of code is optional. We use it to remove outliers.<jupyter_code># loop through the dictionary and remove massive outliers
from scipy import stats
for day in in_time_dict:
    z = abs(stats.zscore(smps_dict[day].sum(axis=1)))
    smps_dict[day] = smps_dict[day][(z<1)]<jupyter_output><empty_output><jupyter_text>Now we'll take our extracted SMPS data and average it and calculate standard deviation.<jupyter_code># create a daily standard deviation from the smps dictionary
smps_daily_std = {}
smps_daily_mean = {}

# create a daily mean dictionary from the smps dictionary
for day in smps_dict:
    smps_daily_mean[day] = smps_dict[day].mean()
    smps_daily_std[day] = smps_dict[day].std()

# turn them into dataframes
smps_daily_mean_df = pd.DataFrame(smps_daily_mean)
smps_daily_std_df = pd.DataFrame(smps_daily_std)

# clean it up a bit
smps_daily_mean_df.reset_index(inplace=True)
smps_daily_mean_df=smps_daily_mean_df.rename(columns={'index':'Dp'})
smps_daily_mean_df.set_index('Dp', inplace=True)

smps_daily_std_df.reset_index(inplace=True)
smps_daily_std_df=smps_daily_std_df.rename(columns={'index':'Dp'})
smps_daily_std_df.set_index('Dp', inplace=True)
smps_daily_mean_df.index=smps_daily_mean_df.index.astype(int)
smps_daily_std_df.index=smps_daily_std_df.index.astype(int)<jupyter_output><empty_output><jupyter_text>Finally, we have a cleaned dataframe of the average particle size distibution across our INP sampling time. Each column corresponds to a specific INP sample and each row is the average particle count for a given diameter.<jupyter_code>smps_daily_mean_df.head()<jupyter_output><empty_output><jupyter_text>We also have a dataframe for the standard deviation.<jupyter_code>smps_daily_std_df.head()<jupyter_output><empty_output><jupyter_text>Now we will calculate the surface area. We pass it the two dataframes we just created, along with the number of bins.<jupyter_code>dAdLogDp, dA_total, dN_total, dAdLogDp_std = pt.surface_area(smps_daily_mean_df, smps_daily_std_df, 25)
pt.plot_number_dist()<jupyter_output><empty_output><jupyter_text>Finally, we create plots by passing the data to the pt.plot_number_dist and pt.plot_surface_dist() functions. Note that the functions require the dataframes to  only contain the particle count (or surface area) data, where rows are size bins (as the index) and columns are the mean of daily (or other timespan) data. This means we have to drop a few columns. See below for explanation.<jupyter_code># Our smps dataframe has 3 columns that we don't need.
smps_daily_mean_df.head()
# This is how the dataframe will need to look. We've dropped the three extra columns
smps_daily_mean_df.drop(columns=['Dp (m)', 'Dp2 (m2)', 'factorA']).head()
fig3 = pt.plot_number_dist(smps_daily_mean_df.drop(columns=['Dp (m)', 'Dp2 (m2)', 'factorA']), smps_daily_std_df.drop(columns=['Dp (m)', 'Dp2 (m2)', 'factorA']))
fig4 = pt.plot_surface_dist(dAdLogDp, dAdLogDp_std)
fig3.update_layout(height=400, width=400)
fig4.update_layout(height=400, width=400)<jupyter_output><empty_output><jupyter_text>3.1.3 Normalize INP Data  See the Pyce Tools manual for deeper explanation of the code below. Basically, we use create an INP object which consists of some INP data and corresponding biology data. We then use the surface area distribution we calculated above to calculate surface area normalized concentration.<jupyter_code>some_bio_dataframe = pd.DataFrame(columns=['location'])
some_other_bio_dataframe = pd.DataFrame(columns=['location'])

inp_bubbler = pt.inp(inp_type='aerosol', inp_location='bubbler', cyto_location='n/a', cyto_data=some_bio_dataframe, uway_bio_data=some_other_bio_dataframe, inp_data=bubbler_df)

inp_bubbler.sa_normalize(dA_total)
# NOTE: If you want to compare to different literature values, you can alter the function in the source code.
fig2 = inp_bubbler.plot_ins_inp()
fig2.update_yaxes(type='log', title='INP per cm<sup>2</sup> of SSA Surface (D<sub>p</sub> = 10-5000nm)', exponentformat='power')<jupyter_output><empty_output><jupyter_text>3.2 Calculate Correlations  See the manual for detailed explanation on calculating correlations.<jupyter_code>inp_bubbler.inp.set_index('datetime',inplace=True)
inp_bubbler.correlations(temps=temps,process='UH', inp_units='inp_sa_normalized', size='sub')
inp_bubbler.correlations(temps=temps,process='H', inp_units='inp_sa_normalized', size='sub')
temp = '-17.5'
inp_bubbler.results['UH'][temp]['corrs'].sort_values(by='p', ascending=True)[:10].style.set_caption(f"Correlations with Unheated Bubbler INP {temp}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10022710068130204, 'fraction_numerical': 0.020136260408781227, 'mean_word_length': 6.384572386808273, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '448480', 'license': 'MIT', 'n_tokens_mistral': 4602, 'n_tokens_neox': 4315, 'n_words': 1319, 'path': 'tutorial/notebook/tutorial.ipynb', 'repo_name': 'jotruebl/Pyce_Tools', 'size': '13210'}