<jupyter_start><jupyter_text>Basic RBR vs SBE CTD Comparison on AQ1801<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from matplotlib.dates import (YearLocator, WeekdayLocator, MonthLocator, 
                            MinuteLocator, DayLocator, HourLocator, DateFormatter)
import matplotlib.ticker as ticker

import cmocean
import matplotlib as mpl
import matplotlib.pyplot as plt

### specify primary bulk figure parameters
fontsize = 10
labelsize = 10
#plotstyle = 'seaborn'
max_xticks = 10
plt.style.use('seaborn-ticks')
mpl.rcParams['svg.fonttype'] = 'none'
mpl.rcParams['ps.fonttype'] = 42 #truetype/type2 fonts instead of type3
mpl.rcParams['pdf.fonttype'] = 42 #truetype/type2 fonts instead of type3
mpl.rcParams['axes.grid'] = False
mpl.rcParams['axes.edgecolor'] = 'black'
mpl.rcParams['axes.linewidth'] = 1.5
mpl.rcParams['axes.labelcolor'] = 'black'
mpl.rcParams['grid.linestyle'] = '--'
mpl.rcParams['grid.linestyle'] = '--'
mpl.rcParams['xtick.major.size'] = 4
mpl.rcParams['xtick.minor.size'] = 2
mpl.rcParams['xtick.major.width'] = 2
mpl.rcParams['xtick.minor.width'] = 0.5
mpl.rcParams['ytick.major.size'] = 4
mpl.rcParams['ytick.minor.size'] = 2
mpl.rcParams['ytick.major.width'] = 2
mpl.rcParams['ytick.minor.width'] = 0.5
mpl.rcParams['ytick.direction'] = 'out'
mpl.rcParams['xtick.direction'] = 'out'
mpl.rcParams['ytick.color'] = 'black'
mpl.rcParams['xtick.color'] = 'black'
import pandas as pd
import datetime<jupyter_output><empty_output><jupyter_text>Cast 025**Location: ** __Notes:__ __Programming Notes:__ .rsk file is a SQLite3 file and it appears that all conductivity-temperature corrections are done prior to recording conductivity values.  Can read straight from file instead of converting with RBR software__Processing Notes:__ .hex seabird files are converted to .cnv files as per usual processing (all seabird routines run).  .rsk files are not adjusted at all, but the salinity is calculated from the Temp/Cond/Press values as it is not in the raw data file.<jupyter_code>import sqlite3
import seawater
data_file = '/Users/bell/ecoraid/2018/CTDcasts/AQ1801/raw/2018_aquila_rbr/AQ1801cast025.rsk'
db = sqlite3.connect(data_file)

query = 'SELECT * FROM channels'
columns = pd.read_sql_query(query,db)
query = 'SELECT * FROM data'

df_r = pd.read_sql_query(query,db)
#time in rbr file is msec since 1970-1-1
import cftime #replaces import from netCDF4 date2num
df_r['cfdatetime'] = cftime.num2date(df_r['tstamp'],'milliseconds since 1970-1-1') - datetime.timedelta(seconds=1.75)
df_r.set_index(df_r['cfdatetime'],inplace=True)
#Using TEOS-10 calculations for salinity from conductivity, temperature, pressure
df_r['conductivity'] = df_r.channel01
df_r['temperature'] = df_r.channel02
df_r['pressure'] = df_r.channel03 - 10.75 #subtract standard atmosphere + an arbitrary offset? due to matching features up on seabird, may be due to offset on sbe? 
                                          # see 2019 cruises on dyson (although this was an AQ cruise)

df_r['salinity'] = seawater.salt(r=df_r['conductivity']/42.914,t=df_r['temperature'],p=df_r['pressure'])<jupyter_output>/Users/bell/anaconda2/envs/py37/lib/python3.7/site-packages/seawater/library.py:363: RuntimeWarning: invalid value encountered in sqrt
  Rtx = (rt) ** 0.5<jupyter_text>**For comprehensive analysis, one should follow Processing Notes Provided by RBR**Summary of Steps:1. Determine and subtract atmospheric pressure from total pressure1. Interpolate over first-order holds (what is this?)1. Smooth temperature and conductivity with a running 5 sample mean1. Lag conductivity by 1 sample1. Derive salinity, depth, and profiling speed1. Remove data occurring when profiling speed falls below 0.4 m/s, and when the CTD reverses from ship heave ("loops")1. Average all data in to 1dbar binsRBR's Matlab toolbox can do this and should be used as algorithm references.   This is not implemented below yet<jupyter_code>### atm pressure
# Basic Assumption: 1013.25 or 14.7psi
# Advance Assumption: pull from reanalysis or ship obs<jupyter_output><empty_output><jupyter_text>Ingest and compare against Seabird Data<jupyter_code>colnames = ['c0mS/cm','c1mS/cm','flECO-AFL','sbeox0V','t090C','t190C','timeS','prDM','sbeox1V','par','turbWETntu0','prDM','sal00','sal11','flag']
df_sbe = pd.read_csv('/Users/bell/ecoraid/2018/CTDcasts/AQ1801/raw/CTD/datcnv_only/ctd025.cnv',skiprows=296,names=colnames,delimiter='\s+')

#start_time
df_sbe['DateTime']=[datetime.datetime.strptime('Oct 11 2018  05:39:51','%b %d %Y %H:%M:%S')+datetime.timedelta(seconds=y['timeS']) for x,y in df_sbe.iterrows()]
df_sbe.set_index(df_sbe['DateTime'],inplace=True)
df_sbe.head()
df_r.head()<jupyter_output><empty_output><jupyter_text>There appears to be a 1.75 second offset between the actual RBR time by matching up the bottle holds.**Goal is to match up features in the profiles both in depth and in time**.  Now - if you compare this to depth profiles below: there is a half meter difference in features.  This is not due to the clock, this is due to the removal of the weight of the atmosphere by assuming a standard atmsophere 10.1325.  I've pulled deck data from earlier in the timeseries to use as the atmsopheric pressure - 9.919 dBar - however, to compare to the SBE - one would need to know what atmospheric pressure they removed.  In this case the total ATM pressure correction to the RBR to match features is ~10.6 (based on just making manual adjustments).  ___RBR has a pressure corrected field in their output.  Should see how they determine the value___Additional factor may be slight deviation in sighting/location of instrumentation (aka a small lag due to vertical differences although G.Lebon. did his best to minimize this)<jupyter_code>tlim = [datetime.datetime(2018,10,11,5,40),datetime.datetime(2018,10,11,5,44)]

fig1 = plt.figure(figsize=(12,6))
plt.subplot(3,1,1)
df_sbe['t090C'].plot()
df_sbe['t190C'].plot()
df_r['temperature'].plot()
plt.xlim(tlim)
#plt.ylim([3.3,3.9])
ax=plt.gca()
ax.xaxis.set_major_locator(MinuteLocator([0,15,30,45]))
ax.xaxis.set_minor_locator(MinuteLocator(range(0,60)))
ax.xaxis.set_major_formatter(DateFormatter(''))
plt.xlabel('')

plt.subplot(3,1,2)
df_sbe['sal00'].plot()
df_sbe['sal11'].plot()
df_r['salinity'].plot()
plt.xlim(tlim)
plt.ylim([32,32.75])
ax=plt.gca()
ax.xaxis.set_major_locator(MinuteLocator([0,15,30,45]))
ax.xaxis.set_minor_locator(MinuteLocator(range(0,60)))
ax.xaxis.set_major_formatter(DateFormatter(''))
plt.xlabel('')

plt.subplot(3,1,3)
df_sbe['prDM'].plot()
df_sbe['prDM'].plot()
df_r['pressure'].plot()
plt.xlim(tlim)
#plt.ylim([73,75.1])
ax=plt.gca()

ax.xaxis.set_major_locator(MinuteLocator([0,15,30,45]))
ax.xaxis.set_minor_locator(MinuteLocator(range(0,60)))
ax.xaxis.set_minor_formatter(DateFormatter(''))
ax.xaxis.set_major_formatter(DateFormatter('%H:%M:%S'))
#ax.xaxis.set_tick_params(which='major', pad=15)
ax.xaxis.set_tick_params(which='minor', pad=5)
start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbe['t090C'][start_time:end_time],df_sbe['prDM'][start_time:end_time])
plt.plot(df_r['temperature'][start_time:end_time],df_r['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbe['t090C'][start_time:end_time],df_sbe['prDM'][start_time:end_time])
plt.plot(df_r['temperature'][start_time:end_time],df_r['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE Temperature (blue) - RBR Temperature (orange): raw data')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbe['t090C'][start_time:end_time],df_sbe['prDM'][start_time:end_time])
plt.plot(df_r['temperature'][start_time:end_time],df_r['pressure'][start_time:end_time])
ax.invert_yaxis()
start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbe['sal00'][start_time:end_time],df_sbe['prDM'][start_time:end_time])
plt.plot(df_r['salinity'][start_time:end_time],df_r['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbe['sal00'][start_time:end_time],df_sbe['prDM'][start_time:end_time])
plt.plot(df_r['salinity'][start_time:end_time],df_r['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE Salinity (blue) - RBR Salinity (orange): raw data')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbe['sal00'][start_time:end_time],df_sbe['prDM'][start_time:end_time])
plt.plot(df_r['salinity'][start_time:end_time],df_r['pressure'][start_time:end_time])
ax.invert_yaxis()<jupyter_output><empty_output><jupyter_text>1sec averagesNow lets sample to a comparable scale... 1s data will allow a direct comparison (although we usually use 1m/1dbar bins)No additional qc has been done to the datasets<jupyter_code>df_rsec = df_r.resample('1s').mean()
df_sbesec = df_sbe.resample('1s').mean()
start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbesec['t090C'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['temperature'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbesec['t090C'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['temperature'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE Temperature (blue) - RBR Temperature (orange): 1sec ave')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbesec['t090C'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['temperature'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()<jupyter_output><empty_output><jupyter_text>1sec averages - Difference between SBE-RBR<jupyter_code>start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbesec['t090C'][start_time:end_time]-df_rsec['temperature'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbesec['t090C'][start_time:end_time]-df_rsec['temperature'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE Temperature - RBR Temperature: 1sec ave')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbesec['t090C'][start_time:end_time]-df_rsec['temperature'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()<jupyter_output><empty_output><jupyter_text>***salinity***<jupyter_code>start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbesec['sal00'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['salinity'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbesec['sal00'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['salinity'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE Salinity (blue) - RBR Salinity (orange): 1sec ave')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbesec['sal00'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['salinity'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()

start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbesec['sal00'][start_time:end_time]-df_rsec['salinity'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbesec['sal00'][start_time:end_time]-df_rsec['salinity'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE Salinity - RBR Salinity: 1sec ave')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbesec['sal00'][start_time:end_time]-df_rsec['salinity'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()<jupyter_output><empty_output><jupyter_text>***sigma-t***<jupyter_code>### calculate density (sigma-t) to compare against each other from the 1s data

import seawater

df_sbesec['sigmat00'] = seawater.dens(s=df_sbesec['sal00'],t=df_sbesec['t090C'],p=df_sbesec['prDM']) - 1000.
df_rsec['sigmat'] = seawater.dens(s=df_rsec['salinity'],t=df_rsec['temperature'],p=df_rsec['pressure']) - 1000.

start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbesec['sigmat00'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['sigmat'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbesec['sigmat00'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['sigmat'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE SigmaT (blue) - RBR SigmaT (orange): 1sec ave')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbesec['sigmat00'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
plt.plot(df_rsec['sigmat'][start_time:end_time],df_rsec['pressure'][start_time:end_time])
ax.invert_yaxis()

start_time = '2018-10-11 05:40:00' 
end_time = '2018-10-11 05:44:00'

fig1 = plt.figure(figsize=(12,6))
ax =plt.subplot(1,3,1)
plt.plot(df_sbesec['sigmat00'][start_time:end_time]-df_rsec['sigmat'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()
plt.ylabel('Depth')

end_time = '2018-10-11 05:42:00'
ax =plt.subplot(1,3,2)
plt.plot(df_sbesec['sigmat00'][start_time:end_time]-df_rsec['sigmat'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()
plt.xlabel('SBE SigmaT - RBR SigmaT: 1sec ave')

end_time = '2018-10-11 05:41:00'
ax =plt.subplot(1,3,3)
plt.plot(df_sbesec['sigmat00'][start_time:end_time]-df_rsec['sigmat'][start_time:end_time],df_sbesec['prDM'][start_time:end_time])
ax.invert_yaxis()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16464365633688208, 'fraction_numerical': 0.06191089713161999, 'mean_word_length': 7.921960072595281, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '573340', 'license': 'MIT', 'n_tokens_mistral': 6577, 'n_tokens_neox': 5928, 'n_words': 968, 'path': '2018/RBR vs SBE - AQ1801.ipynb', 'repo_name': 'shaunwbell/ipythonnb', 'size': '14747'}