<jupyter_start><jupyter_text>Analysis of combined data sets Cf072115 - Cf072215b<NAME>  University of Michigan  2/9/2018  We are expecting 8.5" of snow today.   ..........I am combining four data sets:* Cf072115* Cf072115b* Cf072215a* Cf072215bI have combined the `sparse_bhm.npz`, `sparse_bhm_neg.npz`, and `singles_hist.npz` files on flux and downloaded to my local machine.Now I will revive those files and produce `bhp_nn` for positive and negative time ranges. This is so that I don't have to keep importing the entire `bhm` files each time because it takes forever and a ton of memory.I'm going to use 1 ns time binning for this to save 16x space in the time dimensions.<jupyter_code>import os
import sys
import matplotlib.pyplot as plt
import matplotlib.colors
import numpy as np
import imageio
import scipy.io as sio
sys.path.append('../../scripts/')
import bicorr as bicorr
import bicorr_plot as bicorr_plot
%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Use seaborn to make plots prettier<jupyter_code>import seaborn as sns
sns.set(style='ticks')<jupyter_output><empty_output><jupyter_text>Load `det_df`, channel lists<jupyter_code>os.listdir('../../meas_info/')
det_df = bicorr.load_det_df('../../meas_info/det_df_pairs_angles.csv',plot_flag=True)
chList, fcList, detList, num_dets, num_det_pairs = bicorr.build_ch_lists(print_flag=True)<jupyter_output>Fission chamber channels: [ 0 16 32]
Detector channels: [ 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 17 18 19 20 21 22 23 24 25
 26 27 28 29 30 31 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47]
Number of detectors: 45
Number of detector pairs: 990<jupyter_text>Calculate `num_fissions`Sometime it would be convenient to store all of the measurement in a database, and then load it according to which datasets are specified in `note`. For now, input the data manually.(Calculated in excel file `analysis_status.xlsx`)<jupyter_code>num_fissions = int(int(sio.loadmat('datap/num_fissions.mat')['num_fissions'])*float(sio.loadmat('datap/fc_efficiency.mat')['fc_efficiency']))
num_fissions<jupyter_output><empty_output><jupyter_text>Load data  Load `sparse_bhm.npz`, revive `bhm`<jupyter_code>os.listdir()
sparse_bhm, dt_bin_edges, note = bicorr.load_sparse_bhm(filepath='datap')
sparse_bhm.nbytes
bhm_pos = bicorr.revive_sparse_bhm(sparse_bhm, det_df, dt_bin_edges)
(bhm_pos.nbytes)/16 # .5 ns bins<jupyter_output><empty_output><jupyter_text>I'm going to perform the background subtraction, then store `bhp_nn_diff` for all 990 pairs to disk so I can reload it later.   Load `sparse_bhm_neg.npz`, revive `bhm_neg`<jupyter_code>sparse_bhm_neg, dt_bin_edges_neg, note_neg = bicorr.load_sparse_bhm(filename = 'sparse_bhm_neg.npz', filepath='datap')
bhm_neg = bicorr.revive_sparse_bhm(sparse_bhm_neg, det_df, dt_bin_edges_neg)<jupyter_output><empty_output><jupyter_text>Load `singles_hist.npz`<jupyter_code>singles_hist, dt_bin_edges_sh, dict_det_to_index, dict_index_to_det = bicorr.load_singles_hist(filepath='datap')
help(bicorr.load_singles_hist)
plt.figure(figsize=(4,3))
dt_bin_centers_sh = (dt_bin_edges_sh[:-1]+dt_bin_edges_sh[1:])/2
plt.plot(dt_bin_centers_sh,np.sum(singles_hist,axis=(0,1)))
plt.xlabel('Time (ns)')
plt.ylabel('Number of events')
plt.title('TOF distribution, all events')
plt.yscale('log')
sns.despine(right=False)
bicorr_plot.save_fig_to_folder('singles_hist_allt_allp',extensions=['png','pdf'])
plt.show()
plt.figure(figsize=(4,3))
plt.plot(dt_bin_centers_sh,np.sum(singles_hist[0,:,:],axis=(0)))
plt.plot(dt_bin_centers_sh,np.sum(singles_hist[1,:,:],axis=(0)))
plt.xlabel('Time (ns)')
plt.ylabel('Number of events')
plt.title('TOF distribution, all detectors')
plt.legend(['N','G'])
plt.yscale('log')
sns.despine(right=False)
bicorr_plot.save_fig_to_folder('singles_hist_ng_allp',extensions=['png','pdf'])
plt.show()
plt.figure(figsize=(4,3))
plt.plot(dt_bin_centers_sh,singles_hist[0,dict_det_to_index[2],:])
plt.plot(dt_bin_centers_sh,singles_hist[1,dict_det_to_index[2],:])
plt.xlabel('Time (ns)')
plt.ylabel('Number of events')
plt.title('TOF distribution, channel 2')
plt.legend(['N','G'])
plt.yscale('log')
sns.despine(right=False)
bicorr_plot.save_fig_to_folder('singles_hist_ng_ch2',extensions=['png','pdf'])
plt.show()<jupyter_output><empty_output><jupyter_text>Coarsen `bhm` to 1 ns. time binning.<jupyter_code>print(bhm_pos.shape)
print(bhm_neg.shape)
bhm_pos, dt_bin_edges =     bicorr.coarsen_bhm(bhm_pos,dt_bin_edges,    4,True)
bhm_neg, dt_bin_edges_neg = bicorr.coarsen_bhm(bhm_neg,dt_bin_edges_neg,4,True)
print(bhm_pos.shape)
print(bhm_neg.shape)<jupyter_output>(990, 4, 200, 200)
(990, 4, 200, 200)<jupyter_text>Produce `bhp` for $nn$ events One key piece of data that I am going to work with for producing multiple plots is the `bhp` for $nn$ events across all detector pairs. (Actually, only the pairs not next to the fission chambers)So I am going to produce that for future use. This will be copied into another notebook, but the process of loading all of the data is the same so I'm doing that here since all the data is loaded.I'm going to make this with 1 ns time binning to keep the file size manageable.  Produce `pair_is` for pairs not next to fission chamber<jupyter_code>pair_is = bicorr.generate_pair_is(det_df, ignore_fc_neighbors_flag=True)
len(pair_is)<jupyter_output><empty_output><jupyter_text>Look at this distribution.<jupyter_code>plt.figure(figsize=(6,6))
plt.plot(det_df.iloc[pair_is]['d1'],det_df.iloc[pair_is]['d2'],'sk')
for i in [1,17,33]:
    plt.axvline(i,c='r')
    plt.axhline(i,c='r')
plt.xlabel('Detector 1 channel')
plt.ylabel('Detector 2 channel')
plt.title('Included detector pairs')
sns.despine(right=False)
bicorr_plot.save_fig_to_folder(fig_filename='pair_is_without_fc_neighbors',extensions=['png','pdf'])
plt.show()<jupyter_output><empty_output><jupyter_text>Create `bhp_nn_pos`, `bhp_nn_neg`, `bhp_nn_diff`Following instructions from `bicorr > methods > nn_sum_and_br_subtraction`.I'm going to create arrays with 1 ns time binning and save them to disk, so I can easily reload them in the future.<jupyter_code>bhm_pos.shape
bhm_pos_shape = bhm_pos[pair_is,:,:,:].shape
print(bhm_pos_shape)<jupyter_output>(861, 4, 200, 200)<jupyter_text>The challenge here is that I want to preserve the dimension of `pair_is` (I don't want to sum across all pairs in `pair_is`). How can I do this without significantly modifying my code base?Set up arrays to fill<jupyter_code>bhp_nn_pos = np.zeros((bhm_pos_shape[0],bhm_pos_shape[2],bhm_pos_shape[3]))
bhp_nn_neg = np.zeros((bhm_pos_shape[0],bhm_pos_shape[2],bhm_pos_shape[3]))
bhp_nn_neg.shape
for i in np.arange(len(pair_is)):
    pair_i = pair_is[i]
    
    bhp_nn_pos[i,:,:] = bicorr.build_bhp(bhm_pos,dt_bin_edges,pair_is=[pair_i],type_is=[0])[0]
    bhp_nn_neg[i,:,:] = bicorr.build_bhp(bhm_neg,dt_bin_edges_neg,pair_is=[pair_i],type_is=[0])[0]
    
print(bhp_nn_pos.shape)
print(bhp_nn_neg.shape)<jupyter_output>(861, 200, 200)
(861, 200, 200)<jupyter_text>Plot a few to make sure they look good.<jupyter_code>i = 500
bicorr_plot.bhp_plot(bhp_nn_pos[i,:,:],dt_bin_edges,show_flag=True,title='bhp_nn_pos at i={}'.format(i))
bicorr_plot.bhp_plot(bhp_nn_neg[i,:,:],dt_bin_edges_neg,show_flag=True,title='bhp_nn_neg at i={}'.format(i))<jupyter_output><empty_output><jupyter_text>Plot them now as sums across all pairs.<jupyter_code>bicorr_plot.bhp_plot(np.sum(bhp_nn_pos,axis=0),dt_bin_edges,show_flag=True,title='bhp_nn_pos')
bicorr_plot.bhp_plot(np.sum(bhp_nn_neg,axis=0),dt_bin_edges_neg,show_flag=True,title='bhp_nn_neg')<jupyter_output><empty_output><jupyter_text>Now create `bhp_nn_diff`.Question: Should I create `bhp_nn_diff` here, or work with `bhp_nn_pos` and `bhp_nn_neg`? The data is still pretty sparse, so `bhp_nn_diff` would end up with a lot of negative values in it. Mathematically, once I start taking sums, it would be the same. But I will always have to load `bhp_nn_pos` and `bhp_nn_neg` anyway, so I could just create `bhp_nn_diff` whenever I load them. Yeah. Do that.<jupyter_code>bhp_nn_diff = np.subtract(bhp_nn_pos.astype(np.int32),bhp_nn_neg[:,::-1,::-1].astype(np.int32))
bicorr_plot.bhp_plot(np.sum(bhp_nn_diff,axis=0),dt_bin_edges,show_flag=True,title='bhp_nn_diff')
i = 4
bicorr_plot.bhp_plot(bhp_nn_diff[i,:,:],dt_bin_edges,show_flag=True,title='bhp_nn_diff')<jupyter_output><empty_output><jupyter_text>One thing to keep in mind is that `bicorr.bicorr_plot` does not show negative values, so the background subtraction makes it look "cleaner" than it is in reality.  Verify `bhp_nn_diff` Calculate `bhp_nn_diff` pair by pair and make sure it matches what I've already done.<jupyter_code>bhp_nn_diff_pair = np.zeros((861, 200, 200))
for i in np.arange(len(pair_is)):
    pair_i = pair_is[i]
    bhp_nn_diff_pair[i,:,:] = np.subtract(bhp_nn_pos[i,:,:].astype(np.int32),bhp_nn_neg[i,::-1,::-1].astype(np.int32))
bhp_nn_diff_pair.shape
np.array_equal(bhp_nn_diff,bhp_nn_diff_pair)<jupyter_output><empty_output><jupyter_text>Store `bhp_nn_pos`, `bhp_nn_neg`* `bhp_nn_neg, bhp_nn_pos`* `dt_bin_edges`* `pair_is`* `note`Save these four arrays and the note to the same `.npz` file<jupyter_code>note = 'Stored from Cf072115_to_Cf072215b with 1 ns time binning. Pairs are without fc neighbors. -PFS, 2/9/18'
save_filename = 'datap/bhp_nn_by_pair_1ns'
np.savez(save_filename, bhp_nn_neg = bhp_nn_neg, bhp_nn_pos = bhp_nn_pos,
         dt_bin_edges = dt_bin_edges, pair_is = pair_is, note = note)<jupyter_output><empty_output><jupyter_text>Reload instructionsIn my other analysis files, I'm going to load these variables from disk.Restart the notebook so I start fresh and start from here.<jupyter_code>whos<jupyter_output>Interactive namespace is empty.<jupyter_text>Go back and import all the packages.<jupyter_code>whos
load_filename = 'datap/bhp_nn_by_pair_1ns.npz'
npzfile = np.load(load_filename)
print(npzfile.files)
print(npzfile['note'])
pair_is = npzfile['pair_is']
bhp_nn_pos = npzfile['bhp_nn_pos']
bhp_nn_neg = npzfile['bhp_nn_neg']
dt_bin_edges = npzfile['dt_bin_edges']<jupyter_output><empty_output><jupyter_text>Calculate `bhp_nn_diff`.<jupyter_code>bhp_nn_diff = np.subtract(bhp_nn_pos.astype(np.int32),bhp_nn_neg[:,::-1,::-1].astype(np.int32))
bhp_nn_diff.shape<jupyter_output><empty_output><jupyter_text>Plot them to make sure they look good.<jupyter_code>bicorr.bicorr_plot(np.sum(bhp_nn_diff,axis=0),dt_bin_edges,show_flag=True,title='bhp_nn_diff')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1254311996933691, 'fraction_numerical': 0.028171713300114986, 'mean_word_length': 7.719298245614035, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 83, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '494665', 'license': 'MIT', 'n_tokens_mistral': 4459, 'n_tokens_neox': 4126, 'n_words': 858, 'path': 'analysis/Cf072115_to_Cf072215b/create_bhp_nn_1ns.ipynb', 'repo_name': 'pfschus/fission_bicorrelation', 'size': '10447'}