<jupyter_start><jupyter_text>CUSUMThe Cumulative sum (CUSUM) algorithm is a classical technique for monitoring change detection. Change detection refers to procedures to identify abrupt changes in a phenomenon (Basseville and Nikiforov 1993, Gustafsson 2000). By abrupt change it is meant any difference in relation to previous known data faster than expected of some characteristic of the data such as amplitude, mean, variance, frequency, etc. As its name implies, CUSUM involves the calculation of a cumulative sum (which is what makes it "sequential"). Samples from a process $x_{n}$ are assigned weights $\omega _{n}$, and summed as follows:$S_0=0$$S_{n+1}=max\left\{0,S_{n}+x_{n}-\omega _{n}\right\}$When the value of S exceeds a certain threshold value, a change in value has been found. The above formula only detects changes in the positive direction. When negative changes need to be found as well, the min operation should be used instead of the max operation, and this time a change has been found when the value of S is below the (negative) value of the threshold value.<jupyter_code>import numpy as np
import pandas as pd
def cusum(x,mean=0,K=0):
    """Tabular CUSUM per Montgomery,D. 1996 "Introduction to Statistical Process Control" p318 
    x    : series to analyze
    mean : expected process mean
    K    : reference value, allowance, slack value-- suggest K=1/2 of the shift to be detected.

    Returns:
    x  Original series
    Cp positive CUSUM
    Cm negative CUSUM
    """
    Cp=(x*0).copy()
    Cm=Cp.copy()
    for ii in np.arange(len(x)):
        if ii == 0:
            Cp[ii]=x[ii]
            Cm[ii]=x[ii]
        else:
            Cp[ii]=np.max([0,x[ii]-(mean+K)+Cp[ii-1]])
            Cm[ii]=np.max([0,(mean-K)-x[ii]+Cm[ii-1]])
    return({'x':x, 'Cp': Cp, 'Cm': Cm})

dat = pd.read_csv('case1.csv').to_numpy()
data=dat[81,:]
pd.DataFrame(cusum(data,K=0.25)).plot()
# pd.DataFrame(cusum(data,K=0.25)).plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11229135053110774, 'fraction_numerical': 0.019221041982802226, 'mean_word_length': 3.836185819070905, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '71911', 'license': 'MIT', 'n_tokens_mistral': 645, 'n_tokens_neox': 608, 'n_words': 238, 'path': 'code/detection_and_segmentation/CUSUM.ipynb', 'repo_name': 'iotanalytics/IoTTutorial', 'size': '1977'}