<jupyter_start><jupyter_text>![SegmentLocal](../../assets/images/Logo2.png)  Data Wrangling  PrerequisitesFor this chapter, you should be familiar with the following concepts and techniques:* Basic data visualizations* working with dataframes  Scope of this SheetIn this sheet we will start to work with behavioral data. We will mainly concentrate on working with the python library `pandas` in order to preprocess and manipulate the data. If you are new to Pandas, go through the `pandas` cheatsheet that gives you important functions that you could use.This data was recorded alongside EEG for a social cognition study, conducted by <NAME>, <NAME>, <NAME> and <NAME> here at the IKW. [Here](https://www.frontiersin.org/articles/10.3389/fpsyg.2019.00361/full) you can find the respective paper. In order to properly understand how the data was recorded, I highly recommend to at least read the parts where the experimental design and the experimental procedure are described. The whole data is openly distributed on the [Open Science Framework](https://osf.io/c4wkx/).<jupyter_code>import os
import glob
import numpy as np
import pandas as pd
import seaborn as sns
import statsmodels.api as sm
import matplotlib.pyplot as plt
import io
import requests
from statsmodels.stats.anova import AnovaRM<jupyter_output><empty_output><jupyter_text>1. Loading and preprocessing behavioral data  1.1 Loading the behavioral datasetThe first step is to load the data and to get an idea of how it is structured before we start doing any plotting or evaluation. It's always a good idea to spend some time familiarizing with the data, this will turn out to be very helpful when it comes to checking whether there are any mistakes in the subsequent analysis.<jupyter_code>url = 'https://raw.githubusercontent.com/ashimakeshava/ACC_SS2021/main/Tutorials/Day-02/behavior_data/social_cog_data.csv'
df = pd.read_csv(url)
# TODO: Take a look at the columns of the dataframe
#<jupyter_output><empty_output><jupyter_text>Description of columns`target_ori`: orientation of target stimuli `target_spatial`: spatial frequency of target stimuli `ori_correct_pos`: index of correct orientation`spatial_correct_pos`:index of correct spatial frequency`player1_answer`: response of player1`player2_answer`: response of player2`time_player1`: reaction time of player1`time_player2`: reaction time of player2`player1_correct`: accuracy of player1`player2_correct`: accuracy of player2`player1_reward`:reward of player1`player2_reward`:reward of player2`trial`: trial number`condition `: social condition of trial`block`: block number `group`: id of player pair`pl1_feature`: feature for player1`pl2_feature`: feature for player2<jupyter_code># TODO: Take a look at the first 5 rows of the dataframe

# TODO: Take a look at the last 10 rows of the dataframe

# TODO: number of nan in each column. 0 means no nan in the column<jupyter_output><empty_output><jupyter_text>1.2 Preprocessing and cleaning the behavioral datasetData preprocessing is a very important step to create dataframes that are easy to work with. Especially in this case we can see that each line contains the data of two subjects (player1 and player2), which is not very convenient. Thus we need to restructure the dataframe, such that we only have the data of one subject per row.The easiest way to do this, is to filter the dataframe for each feature combination. For each filtered feature combination we need to pick the corresponding columns (the column filters are provided in the code below). As a last step, it is important to standardize the column names for each filtered feature combination. That way we can concatenate the 4 subsets easily and bring everything back together into one dataframe.We have to consider 4 feature combinations that need to be filtered: - pl1_feature: player_ori ($\rightarrow$ player 1 performed an orientation discrimination task)- pl2_feature: player_ori ($\rightarrow$ player 2 performed an orientation discrimination task)- pl1_feature: player_spatial ($\rightarrow$ player 1 performed a spatial discrimination task)- pl2_feature: player_spatial ($\rightarrow$ player 2 performed a spatial discrimination task)Tipp: Take a look at the Pandas functions [filter()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.filter.html) for creating a subset of the dataframe columns according to a given filter and [concat()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html) for concatenating pandas objects.<jupyter_code># Column filters for different feature combinations
filter_pl1_ori = ['target_ori', 'ori_correct_pos', 'player1_answer', 'time_player1', 
                  'player1_correct', 'player1_reward', 'trial', 'condition', 'block', 
                  'group', 'pl1_feature']

filter_pl2_ori = ['target_ori', 'ori_correct_pos', 'player2_answer', 'time_player2', 
                  'player2_correct', 'player2_reward', 'trial', 'condition', 'block', 
                  'group', 'pl2_feature']

filter_pl1_spatial = ['target_spatial', 'spatial_correct_pos', 'player1_answer', 
                      'time_player1', 'player1_correct', 'player1_reward', 'trial', 
                      'condition', 'block', 'group', 'pl1_feature']

filter_pl2_spatial = ['target_spatial', 'spatial_correct_pos', 'player2_answer', 
                      'time_player2', 'player2_correct', 'player2_reward', 'trial', 
                      'condition', 'block', 'group', 'pl2_feature']

# Column labels
standard_labels = ['target', 'position', 'answer_position', 'time', 'answer', 'feedback', 
                   'trial', 'condition', 'block', 'group', 'task'] 
# TODO: Do the same for the remaining 3 feature combinations 
# Do not forget to select the correct column filters!

# Create a new dataframe and query rows where player1 feature is player_ori
pl1_ori = df[df['pl1_feature'] == 'player_ori']

# Subset the dataframe columns according to **filter_pl1_ori**
pl1_ori = pl1_ori.filter(items = filter_pl1_ori)

# Rename (standardize) the columns
pl1_ori.columns = standard_labels
pl2_ori = df[df['pl2_feature'] == 'player_ori']
pl2_ori = pl2_ori.filter(items = filter_pl2_ori)
pl2_ori.columns = standard_labels

pl1_spatial = df[df['pl1_feature'] == 'player_spatial']
pl1_spatial = pl1_spatial.filter(items = filter_pl1_spatial)
pl1_spatial.columns = standard_labels

pl2_spatial = df[df['pl2_feature'] == 'player_spatial']
pl2_spatial = pl2_spatial.filter(items = filter_pl2_spatial)
pl2_spatial.columns = standard_labels 

# TODO: Concatenate the 4 data subsets
df_filtered = pd.concat([pl1_ori, pl2_ori, pl1_spatial, pl2_spatial], ignore_index=True)
# TODO: Do the same for the remaining 3 feature combinations 
# Do not forget to select the correct column filters!
pl2_ori = ...
pl2_ori = ...
pl2_ori.columns = ...

pl1_spatial = ...
pl1_spatial = ...
pl1_spatial.columns = ...

pl2_spatial = ...
pl2_spatial = ...
pl2_spatial.columns = ...

# Concatenate the 4 data subsets
df_filtered = pd.concat([pl1_ori, pl2_ori, pl1_spatial, pl2_spatial], ignore_index=True)<jupyter_output><empty_output><jupyter_text>What is still missing are unique IDs for each subject and the removal of outliers:- Calculate the number of trials for one group and the number of subjects. Hint: Keep in mind that the number of trials per group in the original loaded dataframe differs from the preprocessed dataframe.- Add a column ```subject_ID``` with unique IDs for each player to the preprocessed dataframe. Tipp: Take a look at the Pandas function [insert()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.insert.html).- Finally, exclude all trials with response times faster than 50 ms, because such fast responses are likely due to premature responses.<jupyter_code># Get the number of trials for one group
count_trials = df[df['group'] == 403].shape[0]

# Get the number of subjects
count_subjects = df.group.nunique() * 2

# Add a column 'subject_ID' with unique IDs for each subject

ids = [i for i in range(count_subjects) for _ in range(count_trials)]
df_filtered.insert(0, 'subject_ID', ids) 
# TODO: Remove all trials where response time is less than 50 ms
df_cleaned = df_filtered.query('time > 0.05')
# How many trials have been removed?
trials_excluded = df_filtered.shape[0] - df_cleaned.shape[0]
print('Excluded trials: {}'.format(trials_excluded))
df_filtered.query('time <= 0.05')<jupyter_output><empty_output><jupyter_text>Now we'd like to replace some values in the dataframe so it is more readable. Using `pandas map` replace `comp` to `competition` and `coop` to `cooperation` in the condition column. Similarly, replace `player_ori` to `orientation` and `player_spatial` to `spatial` in the task column. Alternatively, try using the replace function<jupyter_code>#TODO: replace values in task column
...


#TODO: replace values in condition column
...
df_cleaned.head()<jupyter_output><empty_output><jupyter_text>Now that we've preprocessed and cleaned the data, take a look at the head of the cleaned dataframe again. If your data looks similar to this (there might be another ordering of the groups)you can now proceed to the next section. Please keep in mind that you should use this cleaned dataframe for the following exercises.  1.3 Checking data distributionFor now we've put the data in a suitable format, but we still don't know how the data is distributed. Since a large fraction of the field of statistics is concerned with data to be normally distributed, it is important to check the distribution of your data. The distribution of your data also affects the statistical tests you can use for hypothesis testing.There are a number of techniques that you can use. The easiest way to check the distribution of your data is by creating plots. These checks are qualitative, so less accurate than statistical methods. Nevertheless, they are a good way to get a quick view of the data.- Visualize the response times in form of a histrogram with a Gaussian kernel density estimate on top of it. Tipp: Take a look at Seaborn's [distplot()](https://seaborn.pydata.org/generated/seaborn.distplot.html) function.- Use a [QQ-plot](https://en.wikipedia.org/wiki/Q–Q_plot) to compare the distribution of response times against a normal distribution.  Tipp: Take a look at [statsmodel.qq-plot()](https://www.statsmodels.org/dev/generated/statsmodels.graphics.gofplots.qqplot.html).- If the data is not normally distributed, it can be useful to do a log-transformation. Add a new column ```log_time``` with the log transformed response times to the dataframe. Repeat the previous two steps for log-transformed response times (one histogram and one QQ-plot for log-transformed response times).    For the subsequent analysis we will not use the log-transformed response times, the last part of this exercise is just supposed to be an illustration of what happens to the data when performing a log-transformation.<jupyter_code># TODO: Visualize the data distribution of response times
...
# TODO: Create a QQ-Plot of response times
fig, ax = plt.subplots()

sm.qqplot(...)
ax.set(title='QQ-plot: Response times against normal distribution')

plt.show()
# Log-transform response times
df_log = df_cleaned.copy()
df_log['log_time'] = np.log(df_log['time'])

# TODO: Create Histogram and QQ-plot for log-transformed response times
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15,4))

...

plt.show()<jupyter_output>C:\Users\lonesome\anaconda3\envs\acc_conda\lib\site-packages\statsmodels\graphics\gofplots.py:993: UserWarning: marker is redundantly defined by the 'marker' keyword argument and the fmt string "bo" (-> marker='o'). The keyword argument will take precedence.
  ax.plot(x, y, fmt, **plot_style)<jupyter_text>2. Visualizing the data across experimental conditionsIn the following, we want to take a closer look at the accuracy and the response time. It is always worth plotting your data before you do further analyses, just to have a look at your data. The important thing about plots is that they give you an immediate "picture" of the data.  Scientists like to say that the independent variable (here: correctness, task or condition) goes on the x-axis and the dependent variable (here: accuracy or response time) goes on the y-axis.   2.1 AccuracyPrior to running the actual experiment, the experimenters used a QUEST staircase procedure to adjust a difficulty in each task for participants such that participants were expected to attain a 50% accuracy. Lets check if this requirement was fulfilled.- Calculate the mean accuracy per subject, task and condition. You have to group the data according to the different subjects, tasks and conditions and calculate the mean of answers. Tipp: The function [groupby()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html) can take more than one argument according to which it groups the data. [Here](https://realpython.com/pandas-groupby/) you can find a good guide for Pandas groupby, that should help you to get a better understanding of this function.- Multiply the values of column ```answer``` with 100 and rename it as ```accuracy```. Tipp: Take a look at the Pandas function [rename()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename.html).<jupyter_code># TODO: Calculate the mean of answers per subject, task and condition
...

# TODO: Multiply the values of column answer with 100 and rename the column as accuracy
...

display(df_acc.head())<jupyter_output><empty_output><jupyter_text>Now we can do the plotting:- Create [violinplots](https://seaborn.pydata.org/generated/seaborn.violinplot.html) showing the accuracy for the different tasks and conditions. Tipp: Have a closer look at the violinplot's parameters ```inner=quartile``` and ```split=True``` and use it for your plots.- Calculate the overall mean accuracy and its standard deviation. Tipp: You can use Panda's [mean()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.mean.html) and [std()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.std.html) functions.<jupyter_code># TODO: Plot the accuracy per task and condition
...
# TODO: Calculate the mean accuracy and its standard deviation
...
print('mean accuracy: {:2.2f}%'.format(...))

...
print('standard deviation: {:2.0f}%'.format(...))<jupyter_output><empty_output><jupyter_text>2.2 Response timeAs a last step, we want to take a closer look at the response time, which was analyzed by the experimenters to test whether their experimental manipulations influenced behavioral responses. Especially interesting is how much the correctness, task or condition (= independent variables) affects the response time (= dependent variable). Some quick plots can already give us an idea if the independent variables might have any significant influence on the dependent variable.- Calculate the response time per subject per answer. You have to group the data according to the different subjects and answers and calculate the mean response time.- Calculate the response time per subject per task. You have to group the data according to the different subjects and tasks and calculate the mean response time.- Calculate the response time per subject per condition. You have to group the data according to the different subjects and conditions and calculate mean of response time.- For each grouped dataset create a [boxplot](https://seaborn.pydata.org/generated/seaborn.boxplot.html) and a [swarmplot](https://seaborn.pydata.org/generated/seaborn.swarmplot.html). The plot for one grouped dataset should be similar to the example image below.<jupyter_code># TODO: Calculate the mean response times per subject per answer
df_corr = ...

# TODO: Calculate the mean response times per subject per task
df_task = ...

# TODO: Calculate the mean response times per subject per condition
df_cond = ...

# TODO: Plot the mean response times for each of the 3 grouped datasets
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15,5))
fig.suptitle('Comparison of response times', fontsize=16)
sns.set_palette('colorblind')

...

plt.show()<jupyter_output><empty_output><jupyter_text>Please note that the editing of legends in seaborn is not easy. There are several (cumbersome) ways you can use to change the legend labels. But in fact, it is easier to simply rename the column and its entries as desired, so that the correct default labels are drawn directly from your dataframe.<jupyter_code># TODO: Calculate the mean response times per subject per condition and answer
df_cond_ans = ...
fig, ax= plt.subplots(nrows=1, ncols=1, figsize=(7,7))

plt.show()<jupyter_output><empty_output><jupyter_text>Statistical Analysis with ANOVANow that we've checked the distribution of our data, we can go ahead see if the effects of `task` `answer` and `condition` on the response `time`.- Perform a two-way repeated measures ANOVA and print the ANOVA table. To do this have a closer look at the statsmodels' [AnovaRM()](https://www.statsmodels.org/stable/generated/statsmodels.stats.anova.AnovaRM.html) function and this [tutorial](https://www.marsja.se/repeated-measures-anova-in-python-using-statsmodels/). Tipp: We have a **two-way** repeated measures ANOVA which means that we have **two within-subject factors: correctness and condition**.<jupyter_code># TODO: Perform a two-way repeated measures ANOVA (Pr > F = p-value)

fit = AnovaRM(...).fit()

fit.summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08471580867681432, 'fraction_numerical': 0.01100279345533322, 'mean_word_length': 5.030250945342042, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 55, 'https://': 19, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '207216', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 5049, 'n_tokens_neox': 4772, 'n_words': 2068, 'path': 'Tutorials/Day-02/Data_Wrangling.ipynb', 'repo_name': 'ooezenoglu/ACC_SS2021', 'size': '17570'}