<jupyter_start><jupyter_text>This is a test to see if new scanner head and old one are significatly different even after normalization by using core SO202-37-2.<jupyter_code>import numpy as np 
import pandas as pd
import glob
import matplotlib.pyplot as plt

#plt.style.use('ggplot')
plt.style.use('seaborn-colorblind')
plt.style.use('dark_background')
plt.rcParams['figure.dpi'] = 300
plt.rcParams['savefig.dpi'] = 300
plt.rcParams['savefig.bbox'] = 'tight'
plt.rcParams['savefig.transparent'] = True

%matplotlib inline

import datetime
date = datetime.datetime.now().strftime('%Y%m%d')<jupyter_output><empty_output><jupyter_text>Read and build spectral datasets  Inconsistent formatThe old head has different format to the previous cores, but quite similar to the format of core LV28-44-3.<jupyter_code># test
spe = 'data/SO202-37-2/Run 3 at  10kV_937-1138cm/SO202-37-2_0937   10.0mm  10s  10kV 1000uA F1.spe'
check_depth = spe.split()[3].split('_')[-1]
if len(check_depth) >= 5:
        start_depth = int(check_depth)
    # 3 and 4 digit means in cm, needs to be multipled to be mm
elif len(check_depth) >= 3:
        start_depth = int(check_depth) * 10

with open(spe, 'r') as f:
    content = []
    lines = f.readlines()
    for line in lines[20:]:
        content = np.hstack((content, line.split()))
len(content)<jupyter_output><empty_output><jupyter_text>The parameters looks fine. the length of spe is correct.<jupyter_code>file_name = []
spe_all = []
depth_all = []
cps_all = []
head_all = []
s_depth_all = []
  
# only read the 10kV which having better signal to the light elements
spe_dir = glob.glob('data/SO202-37-2/Run 3 at  10kV_937-1138cm/*.spe')

# make sure the order follows the depthes in filename
spe_dir.sort()

for spe in spe_dir:
    check_depth = spe.split()[3].split('_')[-1]

    # there are some inconsistencies in nameing...as usual
    # 5 and 6 digis means in mm
    if len(check_depth) >= 5:
        start_depth = int(check_depth)
    # 3 and 4 digit means in cm, needs to be multipled to be mm
    elif len(check_depth) >= 3:
        start_depth = int(check_depth) * 10

    file_name.append(spe.split('/')[-1])

    with open(spe, 'r') as f:
        content = []
        lines = f.readlines()
        for line in lines[20:]:
            content = np.hstack((content, line.split()))

        section_depth = round(float(spe.split()[4][:-2]))
        spe_all.append(content.astype(int))
        cps_all.append(0)
        head_all.append('old')
        s_depth_all.append(section_depth)
        depth_all.append(section_depth + start_depth)<jupyter_output><empty_output><jupyter_text>Consistent formatThe new head has the same spe format as the previous cores.<jupyter_code># only read the 10kV which having better signal to the light elements
spe_dir = glob.glob('data/SO202-37-2_2021rescan_937-1138cm/Run 1 at  10kV_937-1138cm/*.spe')

# make sure the order follows the depthes in filename
spe_dir.sort()

for spe in spe_dir:
    check_depth = spe.split()[3].split('_')[-1]

    # there are some inconsistencies in nameing...as usual
    # 5 and 6 digis means in mm
    if len(check_depth) >= 5:
        start_depth = int(check_depth)
    # 3 and 4 digit means in cm, needs to be multipled to be mm
    elif len(check_depth) >= 3:
        start_depth = int(check_depth) * 10

    file_name.append(spe.split('/')[-1])

    with open(spe, 'r') as f:
        content = []
        lines = f.readlines()
        for line in lines[49:]:
            content = np.hstack((content, line.split()))

        section_depth = int(lines[13][:-3])
        spe_all.append(content.astype(int))
        cps_all.append(int(lines[28]))
        head_all.append('new')
        s_depth_all.append(section_depth)
        depth_all.append(section_depth + start_depth)<jupyter_output><empty_output><jupyter_text>Before normalization<jupyter_code>spe_df = pd.DataFrame(spe_all, columns = [str(_) for _ in range(2048)])
spe_df['cps'] = cps_all
spe_df['heads'] = head_all
spe_df['composite_depth_mm'] = depth_all
spe_df['section_depth_mm'] = s_depth_all
spe_df['filename'] = file_name
spe_df
X = np.array(spe_all)
X = X / X.sum(axis = 1, keepdims = True)
spe_n_df = pd.DataFrame(X, columns = [str(_) for _ in range(2048)])
spe_n_df['cps'] = cps_all
spe_n_df['heads'] = head_all
spe_n_df['composite_depth_mm'] = depth_all
spe_n_df['section_depth_mm'] = s_depth_all
spe_n_df['filename'] = file_name
spe_n_df
spe_df.isna().any(axis=None)
spe_df[spe_df.heads == 'old']<jupyter_output><empty_output><jupyter_text>Compare two heads<jupyter_code>fig, axes = plt.subplots(1, 2, sharex='row', sharey='row', figsize=(7, 5))
for ax, head in zip(axes, ['old', 'new']):
    for _, row in spe_df[spe_df.heads == head].iterrows():
        ax.plot(range(520), row[:520], lw=.7, alpha=.5)
        ax.set_xlabel('{} head'.format(head))
plt.subplots_adjust(wspace=.02, top=.95)
fig.suptitle('Core SO202-37-2')
#fig.savefig('results/test_heads_spe_{}.png'.format(date))
print(date)
fig, axes = plt.subplots(1, 2, sharex='row', sharey='row', figsize=(7, 5))
for ax, head in zip(axes, ['old', 'new']):
    for _, row in spe_n_df[spe_n_df.heads == head].iterrows():
        ax.plot(range(520), row[:520], lw=.7, alpha=.5)
        ax.set_xlabel('{} head'.format(head))
plt.subplots_adjust(wspace=.02, top=.95)
fig.suptitle('Core SO202-37-2')
fig.savefig('results/test_heads_spe_{}.png'.format(date))
print(date)<jupyter_output>20210628<jupyter_text>Unfortunately, even after normalization, the spectral behavior of the same section from core SO202-37-2 is different in two heads. It points out mixing data produced by different scaner heads should be treated with caution.  Check the whole datasetOkay, let's check the scanner information of the whole dataset. date (\\$DATE_MEA), volt (\\$ACC_VOLT), current (\\$TUBE_CUR), measring time (\\$MEAS_TIM), data amout<jupyter_code>dirs = glob.glob('data/SO*')
# remove this rescan section
dirs.remove('data/SO202-37-2_2021rescan_937-1138cm')
dirs = np.hstack((dirs, glob.glob('data/LV*')))
dirs = np.hstack((dirs, glob.glob('data/PS*')))
dirs
# test
cores = []
spe_amount = []
dates = []
times = []
volts = []
currs = []

for folder in dirs:
    spe_dirs = glob.glob('{}/*10kV/*.spe'.format(folder))
    spe_amount.append(len(spe_dirs))
    cores.append(folder[5:])
    # select only the first spe file to read informations
    with open(spe_dirs[0], 'r') as f:
        lines = f.readlines()
    for i, line in enumerate(lines):
        if line == '$ACC_VOLT:\n':
            volts.append(float(lines[i+1].split()[0]))
        elif line == '$TUBE_CUR:\n':
            currs.append(float(lines[i+1].split()[0]))
        elif line == '$DATE_MEA:\n':
            dates.append(datetime.datetime.strptime(lines[i+1].split()[0], '%m-%d-%Y'))
        elif line == '$MEAS_TIM:\n':
            times.append(lines[i+1].split())

print(len(spe_amount), len(dates), len(times), len(volts), len(currs))
info_df = pd.DataFrame({
    'core': cores,
    'date': dates,
    'time': times,
    'volt_kV': volts,
    'current_A': currs,
    'data_amount': spe_amount
})
info_df
info_df.to_csv('results/scanning_info_{}.csv'.format(date))
print(date)<jupyter_output>20210628
{'fraction_non_alphanumeric': 0.14896282890157317, 'fraction_numerical': 0.03466518167896422, 'mean_word_length': 3.995827538247566, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '405436', 'license': 'MIT', 'n_tokens_mistral': 2777, 'n_tokens_neox': 2579, 'n_words': 606, 'path': 'test_heads.ipynb', 'repo_name': 'dispink/CaCO3_NWP', 'size': '7183'}