<jupyter_start><jupyter_text>DEMO: Survey module functionalitiesThis notebook provides a demo of how to utilise the survey module.<jupyter_code>import sys
sys.path.append('../')

import numpy as np
import pandas as pd
import niimpy
from niimpy.survey import *
from niimpy.EDA import EDA_categorical<jupyter_output><empty_output><jupyter_text>Load dataWe will load a mock survey data file.<jupyter_code># Load a mock dataframe
df = niimpy.read_csv('mock-survey.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Preprocessing The dataframe's columns are raw questions from a survey. Some questions belong to a specific category, so we will annotate them with ids. The id is constructed from a prefix (the questionnaire category: GAD, PHQ, PSQI etc.), followed by the question number (1,2,3). Similarly, we will also the answers to meaningful numerical values.**Note:** It's important that the dataframe follows the below schema before passing into niimpy.<jupyter_code># Convert column name to id, based on provided mappers from niimpy
col_id = {**PHQ2_MAP, **PSQI_MAP, **PSS10_MAP, **PANAS_MAP, **GAD2_MAP}
selected_cols = [col for col in df.columns if col in col_id.keys()]

# Convert from wide to long format
m_df = pd.melt(df, id_vars=['user', 'age', 'gender'], value_vars=selected_cols, var_name='question', value_name='raw_answer')

# Assign questions to codes 
m_df['id'] = m_df['question'].replace(col_id)
m_df.head()
# Transform raw answers to numerical values
m_df['answer'] = niimpy.survey.convert_to_numerical_answer(m_df, answer_col = 'raw_answer',
                                question_id = 'id', id_map=ID_MAP_PREFIX, use_prefix=True)
m_df.head()<jupyter_output><empty_output><jupyter_text>We can also make a summary of the questionaire's score. This function can describe aggregated score over the whole population, or specific subgroups.<jupyter_code>d = niimpy.survey.print_statistic(m_df, group='gender')
pd.DataFrame(d)<jupyter_output><empty_output><jupyter_text>VisualizationWe can now make some plots for the preprocessed data frame. First, we can display the summary for a specific question.<jupyter_code>fig = niimpy.EDA.EDA_categorical.questionnaire_summary(m_df, question = 'PHQ2_1', column = 'answer', 
                                                       title='PHQ2_1', xlabel='value', ylabel='count', 
                                                       width=900, height=400)
fig.show()<jupyter_output><empty_output><jupyter_text>We can also display the summary for each subgroup.<jupyter_code>fig = niimpy.EDA.EDA_categorical.questionnaire_grouped_summary(m_df, question='PSS10_9', group='gender', 
                                                               title='PSS10_9',
                                                               xlabel='score', ylabel='count',
                                                               width=800, height=600)
fig.show()<jupyter_output><empty_output><jupyter_text>With some quick preprocessing, we can display the score distribution of each questionaire.<jupyter_code>pss_sum_df = m_df[m_df['id'].str.startswith('PSS')] \
                            .groupby(['user', 'gender']) \
                            .agg({'answer':sum}) \
                            .reset_index()
pss_sum_df['id'] = 'PSS'
fig = niimpy.EDA.EDA_categorical.questionnaire_grouped_summary(pss_sum_df, question='PSS', group='gender', 
                                                               title='PSS10',
                                                               xlabel='score', ylabel='count',
                                                               width=800, height=600)
fig.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.099622030237581, 'fraction_numerical': 0.009989200863930885, 'mean_word_length': 2.815653964984552, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '144809', 'license': 'MIT', 'n_tokens_mistral': 1091, 'n_tokens_neox': 1049, 'n_words': 280, 'path': 'examples/surveys.ipynb', 'repo_name': 'CxAalto/niima', 'size': '3704'}