<jupyter_start><jupyter_text>Cortex Case Example: Anomaly DetectionIt's easy to get started with analysis on data collected from mindLAMP. In this example, we'll walk through using [Cortex](https://docs.lamp.digital/cortex) with [Luminol](https://github.com/linkedin/luminol), an anomaly detection library, and [Altair](https://altair-viz.github.io), an interactive visualization library, to tag and visualize survey scores for a particular patient.<jupyter_code>import cortex
import luminol
import pandas as pd
import numpy as np
import altair as alt
from luminol.anomaly_detector import AnomalyDetector<jupyter_output><jupyter_text>Preparing the data using CortexFirst, call `cortex.run(...)` with your Participant ID of interest. Then, we'll need to rearrange the resultant data frame by setting the index to the `timestamp` and adding an `anomaly` column for later.<jupyter_code>df = cortex.run(
    'U1089294357', ['survey_scores'], 
    start=0, end=cortex.now()
)['survey_scores']
df.index = df.timestamp.astype(int) // 10**3
df['anomaly'] = 0 # default to no anomaly<jupyter_output>[INFO:feature_types:_wrapper2] Processing primary feature "cortex.survey_scores"...
[INFO:feature_types:_wrapper2] Cortex caching directory set to: /home/_data/cortex_cache
[INFO:feature_types:_wrapper2] Processing raw feature "lamp.survey"...
[INFO:feature_types:_wrapper2] No saved raw data found, getting new...
[INFO:feature_types:_wrapper2] Saving raw data as "/home/_data/cortex_cache/survey_U1089294357_0_1621449536000.cortex"...<jupyter_text>In addition to the survey `score` column, we also have a `category` column that's derived from custom survey grouping. The Cortex feature `survey_scores` automatically scores each question for you, whether it's a Likert scale, list of options, True/False, and so on. Then, it groups together questions from a single survey, such as "Weekly Survey" by predefined categories, like "Mood" and "Anxiety" to better understand symptom domains.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>Detecting anomalies using LuminolNow, we feed the Luminol detector our `score` column. It then processes the data and returns anomalous time windows tagged with an anomaly score. We'll tag the actual survey scores in our DataFrame that lie within these windows with their respective anomaly score. We need to iterate over each category and tag anomalies within the category independent of survey scores from other categories.<jupyter_code>for cat in df.category.unique():
    sub_df = df.loc[df.category == cat, 'score'].fillna(0).to_dict()
    detector = AnomalyDetector(sub_df, score_threshold=1.5)
    for a in detector.get_anomalies():
        ts = (df.index >= a.start_timestamp) & (df.index <= a.end_timestamp)
        df.loc[ts & (df.category == cat), 'anomaly'] = a.anomaly_score<jupyter_output><empty_output><jupyter_text>Visualizing the anomalies using AltairWe'll use the Altair interactive plotting library to break question categories out into their own sub-charts. We'll also bring extra attention to anomalous survey score data points by increasing their size and changing their color.<jupyter_code>alt.Chart(df).mark_point(filled=True).properties(width=500, height=50).encode(
    
    # The timestamp column was already converted by Cortex into a human-readable Date.
    x=alt.X('timestamp', title="Date"),
    
    # We know the score is clamped between [1 <= score <= 3] for this patient.
    y=alt.Y('score', title="Score", scale=alt.Scale(domain=[1, 3])),
    
    # Color anomalies non-linearly by severity (redder is worse).
    color=alt.Color('anomaly', title='Severity', scale=alt.Scale(type='sqrt', range=['#29629E', '#CA2C21'])),
    
    # Resize anomalies non-linearly by severity (larger is worse).
    size=alt.Size('anomaly', title='Severity', scale=alt.Scale(type='sqrt', range=[25, 500]))
).facet(
    
    # By 'faceting' the plot by the category column, we can split each survey category out into its own subplot.
    row='category'
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10250061896509037, 'fraction_numerical': 0.017083436494181727, 'mean_word_length': 5.3125, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 20, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '118395', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1283, 'n_tokens_neox': 1169, 'n_words': 417, 'path': 'docs/09-data_science/cortex.ipynb', 'repo_name': 'hunterlester/LAMP-platform', 'size': '4039'}