<jupyter_start><jupyter_text>Read in example data from one CML<jupyter_code>cml = pycml.io.examples.read_one_cml()
# Remove artifacts and plot data
cml.process.quality_control.set_to_nan_if('tx', '>=', 100)
cml.process.quality_control.set_to_nan_if('rx', '==', -99.9)

cml.plot_data(['tx', 'rx', 'txrx']);<jupyter_output><empty_output><jupyter_text>Do a simple wet/dry classification<jupyter_code>cml.process.wet_dry.std_dev(window_length=30, threshold=0.8)
cml.plot_data(['txrx', 'wet']);<jupyter_output><empty_output><jupyter_text>Derive a constant baselineLet's just focus on the rain events on 2016-10-25<jupyter_code>cml.process.baseline.constant()
cml.process.baseline.calc_A()
ax = cml.plot_data(['txrx', 'wet', 'baseline', 'A']);
ax[0].set_xlim('2016-10-25 00:00', '2016-10-25 10:00');
# Save a copy of these results for comparing them to the linear baseline later
baseline_constant = cml.channel_1.data.baseline.copy()
A_constant = cml.channel_1.data.A.copy()<jupyter_output><empty_output><jupyter_text>Or derive a linear baseline<jupyter_code>cml.process.baseline.linear()
cml.process.baseline.calc_A()
ax = cml.plot_data(['txrx', 'wet', 'baseline', 'A']);
ax[0].set_xlim('2016-10-25 00:00', '2016-10-25 10:00');
# Save a copy of these results for comparing them to the constant baseline
baseline_linear = cml.channel_1.data.baseline.copy()
A_linear = cml.channel_1.data.A.copy()<jupyter_output><empty_output><jupyter_text>Compare the results from constant and linear baseline<jupyter_code>fig, ax = plt.subplots(2, 1, figsize=(10, 4), sharex=True)

ax[0].plot(baseline_constant, color='C3', label='constant baseline')
ax[0].plot(baseline_linear, color='C4', label='linear baseline')

ax[1].plot(A_constant, color='C3', label='constant baseline')
ax[1].plot(A_linear, color='C4', label='linear baseline')

ax[0].set_xlim('2016-10-25 00:00', '2016-10-25 10:00');

ax[0].set_ylabel('baseline')
ax[1].set_ylabel('A')

ax[0].legend();<jupyter_output><empty_output><jupyter_text>NaN handling The algorithms for constant and linear baseline handle `NaN`s differently:* constant baseline:    * For `NaN` values in `wet` the `baseline` is also set to `NaN`.    * All `baseline` values following a `NaN` during a wet event are also set to `NaN` till the next dry event starts. This has to be done, since we do not know if a new wet event started during a `NaN` period and hence we do not know at which level the constant baseline should be.* linear baseline:    * Default:      The `baseline` for a whole wet event is set to `NaN` if there is at least one `wet` `NaN` within this period. This makes sense, since for the interpolation of the linear baseline the correct end of the wet period has to be known to its `txrx` value. Since the wet event could have ended during the `NaN` period, we do not know the end of the wet period and hence cannot savely assume a `txrx` endpoint for the interpolation.    * Option to `ignore_nan`:      If you know what you are doing, e.g. because you know that your only have very few consecutive `wet` `NaN`s and hecne can assume that a wet event will not stop during your `wet` `NaN`, then you can ignore all `NaN`s. This will take the next switch from wet to dry as endpoint of the wet event and do the interpolation accordingly.<jupyter_code># Exchange the current `wet` pd.Series in `channel_1` with a different series of floats with some `NaN`s
wet_temp = cml.channel_1.data.wet.astype(float)
wet_temp['2016-10-25 04:45': '2016-10-25 05:00'] = np.NaN
cml.channel_1.data.wet = wet_temp<jupyter_output><empty_output><jupyter_text>Constant baseline<jupyter_code>cml.process.baseline.constant()
cml.process.baseline.calc_A()
ax = cml.plot_data(['txrx', 'wet', 'baseline', 'A']);
ax[0].set_xlim('2016-10-25 00:00', '2016-10-25 10:00');<jupyter_output><empty_output><jupyter_text>Linear baseline (default)default = set `baseline` for whole wet event to `NaN` if it contains at least one `wet` `NaN`<jupyter_code>cml.process.baseline.linear()
cml.process.baseline.calc_A()
ax = cml.plot_data(['txrx', 'wet', 'baseline', 'A']);
ax[0].set_xlim('2016-10-25 00:00', '2016-10-25 10:00');<jupyter_output><empty_output><jupyter_text>Linear baseline (ignoring `NaN`s)<jupyter_code>cml.process.baseline.linear(ignore_nan=True)
cml.process.baseline.calc_A()
ax = cml.plot_data(['txrx', 'wet', 'baseline', 'A']);
ax[0].set_xlim('2016-10-25 00:00', '2016-10-25 10:00');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14111261872455902, 'fraction_numerical': 0.04862053369516056, 'mean_word_length': 6.110932475884244, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '508221', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1703, 'n_tokens_neox': 1538, 'n_words': 439, 'path': 'notebooks/Baseline determination.ipynb', 'repo_name': 'jpolz/pycomlink', 'size': '4422'}