<jupyter_start><jupyter_text>1) Movement detection<jupyter_code>import pandas as pd
import numpy as np

import seaborn as sns
import matplotlib.pyplot as plt


import glob 

import statistics

from statistics import StatisticsError
#To make things easier to run on this document, I renamed the directories.

files_movement = list(glob.glob("movement_test_data/*.csv"))
files_movement.sort()<jupyter_output><empty_output><jupyter_text>The files are numbered from 1 to 5. We assume they represent 5 consecutive minutes' worth of data.<jupyter_code># Keeping the last file as a test dataset

test_file_movement = files_movement[-1]
files_movement.pop(-1)

df_movement = pd.DataFrame()

for file in files_movement:
    df_open = pd.read_csv(file, header=None)
    df_movement = pd.concat([df_movement, df_open], axis=0)
    del df_open
    
df_movement = df_movement.reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Renaming the variable to 'obs'<jupyter_code>df_movement.columns = ['obs']<jupyter_output><empty_output><jupyter_text>Each second is made up of 250 observations<jupyter_code>obs_per_sec = round(df_movement.shape[0]/(4*4*60))

# Calculating the basic metrics

print(df_movement.describe())
plt.hist(df_movement.obs);<jupyter_output><empty_output><jupyter_text>The vast majority of values are clustered between 50 and 60<jupyter_code>mean = df_movement.obs.mean()
print("Mean: {}".format(mean))

median = df_movement.obs.median()
print("Median: {}".format(median))<jupyter_output>Mean: 51.449867694551514
Median: 53.0<jupyter_text>Visualising the dataset at two cases to see if there are any unique patterns. I did this visual inspection for a few more slices in Spyder, which I didn't add here. Only showing two.  First we look at a slice where there is movement<jupyter_code>plt.plot(df_movement.iloc[20000:22000])
####### Now we look at one where there is no movement
plt.plot(df_movement.iloc[50000:52000])<jupyter_output><empty_output><jupyter_text>Notes - 1. Where there is movement, the amplitude of the observations reaches both the upper and lower extremes on a regular basis. The peaks hit 70 and the troughs go all the way to 0. 2. In the case of movement, the amplitudes tend to 'stick' to an extreme value for a few observations, which can possibly be used as an signal for movement.3. When not moving, the data appears more noisy (and even sinusoidal in a way). The peaks max out to 70, but the troughs usually do not reach 0.   Trying to generate some features we could possibly use - The idea is to see if any of the below is a signal for movement<jupyter_code># The deviation from the global mean and the individual observation
df_movement['diff_mean'] = mean - df_movement['obs']

# The rate of change of the deviation from mean w.r.t the mean
df_movement['delta_mean'] = abs(df_movement['diff_mean'] / mean)

#Lagged mean of delta_mean for 100 milliseconds window
df_movement['delta_mean_rolling_100ms'] = df_movement['delta_mean'].rolling(window=obs_per_sec//10).mean()

#Lagged mean of delta_mean for 200 milliseconds window
df_movement['delta_mean_rolling_200ms'] = df_movement['delta_mean'].rolling(window=obs_per_sec//5).mean()

# Absolute differencing of observations by 1 value behind (to see if the time series can be made stationary)
df_movement['diff_1'] = abs(df_movement['obs'].diff(1))

# Lagged mean of diff_1 for 10 observations
df_movement['diff_1_rolling_mean_abs'] = df_movement['diff_1'].rolling(window=10).mean()

# Absolute differencing of the previous value by 1
df_movement['diff_2'] = abs(df_movement['diff_1_rolling_mean_abs'].diff())<jupyter_output><empty_output><jupyter_text>After much experimentation, I found out that values for 'delta_mean_rolling_200ms' begin to spike when movement begins and go down when movement stops. The spike happens with a lag of 200 milliseconds (50 observations), since this is a rolling statistic.<jupyter_code>plt.plot(df_movement.iloc[20000:22000].obs)

plt.plot(df_movement.iloc[20000:22000].delta_mean_rolling_200ms)<jupyter_output><empty_output><jupyter_text>After looking at the data at many intervals, I set 'delta_mean_rolling_200ms' > 0.2 as the threshold for classifying an observation as belonging to movement. Here, I construct a filter which assigns 1 for movement and 0 otherwise over a window of 200 milliseconds (50 observations)<jupyter_code>def movement_detection_filter(df_movement, threshold=0.2):


    df_movement['movement'] = 0

    for index, row in df_movement.iterrows():

        df_movement.loc[index-49:index+1, 'movement'] = np.where(
                
                (df_movement.iloc[index]['delta_mean_rolling_200ms'] > threshold) & (
                    ~np.isnan(df_movement.iloc[index]['delta_mean_rolling_200ms']))
                , 1, 0)

    
    df_movement.loc[(np.isnan(df_movement['delta_mean_rolling_200ms'])), 'delta_mean_rolling_200ms'] = 0
    
    return df_movement


df_movement = movement_detection_filter(df_movement, threshold=0.2)
# Now using this filter on test data
df_movement_test = pd.read_csv(test_file_movement, header=None, names=['obs'])

df_movement_test['diff_mean'] = mean - df_movement_test['obs']

df_movement_test['delta_mean'] = abs(df_movement_test['diff_mean'] / mean)

df_movement_test['delta_mean_rolling_100ms'] = df_movement_test['delta_mean'].rolling(window=obs_per_sec//10).mean()

df_movement_test['delta_mean_rolling_200ms'] = df_movement_test['delta_mean'].rolling(window=obs_per_sec//5).mean()

df_movement_test = movement_detection_filter(df_movement_test, threshold=0.2)<jupyter_output><empty_output><jupyter_text>Wrting True/False values when movement is detected for a particular second window (each 250 obs wide)<jupyter_code>timesteps = np.arange(0, df_movement_test.shape[0], obs_per_sec)


movement_detected = pd.Series()

for sec in timesteps:
    try:
        verdict = bool(np.where(statistics.mode(df_movement_test.iloc[sec:sec+obs_per_sec]['movement']) == 1, True, False))
    except StatisticsError:
        verdict = True
        
    movement_detected[str(sec)] = verdict<jupyter_output><empty_output><jupyter_text>We visualise this<jupyter_code>plt.plot(df_movement_test.obs)

for i, sec in enumerate(timesteps):
    if movement_detected[str(sec)] == True:
        plt.axvspan(sec, sec+obs_per_sec-1, alpha=0.3, color='r', zorder=20)<jupyter_output><empty_output><jupyter_text>We see that the filter detects movement to a reasonable extent      2) Snoring detection<jupyter_code>files_snoring = list(glob.glob("snoring_test_data/*.csv"))
files_snoring.sort()

test_file_snoring = files_snoring[-1]
files_snoring.pop(-1)
df_snoring = pd.DataFrame()

for file in files_snoring:
    df_open = pd.read_csv(file, header=None)
    df_snoring = pd.concat([df_snoring, df_open], axis=0)
    del df_open
    
df_snoring = df_snoring.reset_index(drop=True)

df_snoring.columns = ['obs']
obs_per_sec = round(df_snoring.shape[0]/(4*4*60))<jupyter_output><empty_output><jupyter_text>Each second is made up of 250 observations<jupyter_code># Calculating the basic metrics

print(df_snoring.describe())
plt.hist(df_snoring.obs);
mean = df_snoring.obs.mean()
print("Mean: {}".format(mean))

median = df_snoring.obs.median()
print("Median: {}".format(median))<jupyter_output>Mean: 52.26262516824529
Median: 52.0<jupyter_text>Visualising the dataset at two cases to see if there are any unique patterns. I did this visual inspection for a few more slices in Spyder, which I didn't add here. Only showing two. First we look at a slice where there is snoring<jupyter_code>plt.plot(df_snoring.iloc[600:800].obs)<jupyter_output><empty_output><jupyter_text>Now we look at where there is no snoring<jupyter_code>plt.plot(df_snoring.iloc[:500].obs)<jupyter_output><empty_output><jupyter_text>It can be seen that snoring makes the signal more jagged and noisy. Therefore, we must find a signal that represents this noisiness. When there is no snoring, observations in the immediate neighbourhood seem the follow and steady upwards or downwards trend, except at peaks and troughs.We try to see if the time-lagged values will show any interesting patterns. Autocorrelation might help. Generating a few features based on what we see in the data -<jupyter_code># Deviation from the mean
df_snoring['diff_mean'] = mean - df_snoring['obs']

# Rate of deviation from the mean w.r.t the mean
df_snoring['delta_mean'] = (df_snoring['diff_mean'] / mean)

#Lagged mean of delta_mean for 100 milliseconds
df_snoring['delta_mean_rolling_100ms'] = df_snoring['delta_mean'].rolling(window=obs_per_sec//10).mean()

#Lagged mean of delta_mean for 200 milliseconds
df_snoring['delta_mean_rolling_200ms'] = df_snoring['delta_mean'].rolling(window=obs_per_sec//5).mean()

# Lagged standard deviation for delta_mean for 100 milliseconds
df_snoring['delta_mean_rolling_std_100ms'] = df_snoring['delta_mean'].rolling(window=25).std()

# Lagged mean for observations for 10 observations
df_snoring['roll_mean_obs_10'] = df_snoring['obs'].rolling(window=10).mean()



# Differencing of observations going back 1
df_snoring['diff_1'] = df_snoring['obs'].diff(1)

# Lagged standard deviation for observation with window of 2 observations
df_snoring['rolling_std_2'] = df_snoring['obs'].rolling(window=2).std()

# Lagged mean for rolling_std_2 with window of 10 observations
df_snoring['rolling_std_2_rolling_mean_10'] = df_snoring['rolling_std_2'].rolling(window=10).mean()

# Differencing of observations going back 5
df_snoring['diff_5_abs'] = abs(df_snoring['obs'].diff(5))

# Centering diff_5_abs around the mean
df_snoring['diff_5_abs_centred'] = abs(df_snoring['diff_5_abs'] - df_snoring['diff_5_abs'].mean())

# Lagged sum of diff_5_abs with a window of 10 observations
df_snoring['diff_5_abs_rolling_mean'] = df_snoring['diff_5_abs'].rolling(window=10).sum()

# Lagged autocorrelation of observations with a window of 10
df_snoring['autocorr_10'] = df_snoring['obs'].rolling(window=10).apply(lambda x: x.autocorr(), raw=False)

# Lagged autocorrelation of observations with a window of 5
df_snoring['autocorr_5'] = df_snoring['obs'].rolling(window=5).apply(lambda x: x.autocorr(), raw=False)

# Dividing autocorr_5 by diff_5_abs
df_snoring['autocorr_5_plus_by_diff_5_abs'] = df_snoring['autocorr_5'] / df_snoring['diff_5_abs']

# Lagged mean of diff_mean for 100 milliseconds
df_snoring['diff_mean_rolling'] = df_snoring['diff_mean'].rolling(window=obs_per_sec//10).mean()<jupyter_output><empty_output><jupyter_text>After a few trials, I found that rolling_std_2 gives a signal whenever there is snoring  This is the plot for rolling_std_2 where there is snoring<jupyter_code>plt.plot(df_snoring.iloc[600:800].rolling_std_2)<jupyter_output><empty_output><jupyter_text>And this is rolling_std_2 where there is no snoring<jupyter_code>plt.plot(df_snoring.iloc[:500].rolling_std_2)<jupyter_output><empty_output><jupyter_text>Based on this, we write a filter to detect snoring. My hypothesis is that snoring occurs when the value of mean(rolling_std_2) goes above a certain threshold. We define the threshold as 0.75<jupyter_code>def snoring_detection_filter(df_snoring):

    df_snoring['snoring'] = 0

    for index, row in df_snoring.iterrows():
    
        df_snoring.loc[index, 'snoring'] = np.where(
                df_snoring.loc[index:index+249, 'rolling_std_2'].mean() > 0.75, 1, 0)

    return df_snoring
df_snoring = snoring_detection_filter(df_snoring)<jupyter_output><empty_output><jupyter_text>Now using this filter on test data<jupyter_code>df_snoring_test = pd.read_csv(test_file_snoring, header=None)


df_snoring_test = df_snoring_test.reset_index(drop=True)

df_snoring_test.columns = ['obs']

obs_per_sec = round(df_snoring_test.shape[0]/(4*4*60))


# Deviation from the mean
df_snoring_test['diff_mean'] = mean - df_snoring_test['obs']

# Rate of deviation from the mean w.r.t the mean
df_snoring_test['delta_mean'] = (df_snoring_test['diff_mean'] / mean)

#Lagged mean of delta_mean for 100 milliseconds
df_snoring_test['delta_mean_rolling_100ms'] = df_snoring_test['delta_mean'].rolling(window=obs_per_sec//10).mean()

#Lagged mean of delta_mean for 200 milliseconds
df_snoring_test['delta_mean_rolling_200ms'] = df_snoring_test['delta_mean'].rolling(window=obs_per_sec//5).mean()

# Lagged standard deviation for delta_mean for 100 milliseconds
df_snoring_test['delta_mean_rolling_std_100ms'] = df_snoring_test['delta_mean'].rolling(window=25).std()

# Lagged mean for observations for 10 observations
df_snoring_test['roll_mean_obs_10'] = df_snoring_test['obs'].rolling(window=10).mean()



# Differencing of observations going back 1
df_snoring_test['diff_1'] = df_snoring_test['obs'].diff(1)

# Lagged standard deviation for observation with window of 2 observations
df_snoring_test['rolling_std_2'] = df_snoring_test['obs'].rolling(window=2).std()

# Lagged mean for rolling_std_2 with window of 10 observations
df_snoring_test['rolling_std_2_rolling_mean_10'] = df_snoring_test['rolling_std_2'].rolling(window=10).mean()

# Differencing of observations going back 5
df_snoring_test['diff_5_abs'] = abs(df_snoring_test['obs'].diff(5))

# Centering diff_5_abs around the mean
df_snoring_test['diff_5_abs_centred'] = abs(df_snoring_test['diff_5_abs'] - df_snoring_test['diff_5_abs'].mean())

# Lagged sum of diff_5_abs with a window of 10 observations
df_snoring_test['diff_5_abs_rolling_mean'] = df_snoring_test['diff_5_abs'].rolling(window=10).sum()

# Lagged autocorrelation of observations with a window of 10
df_snoring_test['autocorr_10'] = df_snoring_test['obs'].rolling(window=10).apply(lambda x: x.autocorr(), raw=False)

# Lagged autocorrelation of observations with a window of 5
df_snoring_test['autocorr_5'] = df_snoring_test['obs'].rolling(window=5).apply(lambda x: x.autocorr(), raw=False)

# Dividing autocorr_5 by diff_5_abs
df_snoring_test['autocorr_5_plus_by_diff_5_abs'] = df_snoring_test['autocorr_5'] / df_snoring_test['diff_5_abs']

# Lagged mean of diff_mean for 100 milliseconds
df_snoring_test['diff_mean_rolling'] = df_snoring_test['diff_mean'].rolling(window=obs_per_sec//10).mean()

df_snoring_test = snoring_detection_filter(df_snoring_test)
# Wrting True/False values when snoring is detected for a particular second window (each 250 obs wide)
obs_per_step = 10

timesteps = np.arange(600, 800, obs_per_step)


snoring_detected = pd.Series()

for sec in timesteps:
    try:
        verdict = bool(np.where(statistics.mode(df_snoring_test.iloc[sec:sec+obs_per_step]['snoring']) == 1, True, False))
    except StatisticsError:
        verdict = True
        
    snoring_detected[str(sec)] = verdict
snoring_detected
plt.plot(df_snoring_test.iloc[600:800].obs)

for i, sec in enumerate(timesteps):
    if snoring_detected[str(sec)] == True:
        plt.axvspan(sec, sec+obs_per_step-1, alpha=0.3, color='r', zorder=20)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10565275908479138, 'fraction_numerical': 0.029340511440107672, 'mean_word_length': 6.4305, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 82, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '443290', 'license': 'MIT', 'n_tokens_mistral': 5388, 'n_tokens_neox': 5010, 'n_words': 1224, 'path': 'notebook.ipynb', 'repo_name': 'stiero/movement-snoring-detection', 'size': '14860'}