<jupyter_start><jupyter_text>This script assumes the following file structure:./AXB.ipynb./english/./english/datafile1.csv./english/datafile2.csv./english/datafile3.csv./english/etc./panara/datafile1.csv./panara/datafile2.csv./panara/datafile3.csv./panara/etc First, read in all of the English speakers' data.  These data were collected through OpenSesame, which gives us a ton of variables we're not really interested in, so we'll also pare the columns down to just what we need, and also rename them.Since we'll be joining this data with the Panara speakers' data, we'll want to be able to tell which subjects are Panara native speakers and which aren't, so we'll also add a new field for subject language.<jupyter_code>engdir = os.path.abspath('./english/')
files = [f for f in os.listdir(engdir) if f[-4:]=='.csv']
responsedf_eng = pd.DataFrame()
for f in files:
    tdf = pd.read_csv(os.path.join(engdir,f))
    responsedf_eng = responsedf_eng.append(tdf)
responsedf_eng = responsedf_eng[['subject_nr','count_X','flip_order','stimA','stimX','stimB','response']]
responsedf_eng = responsedf_eng.rename(columns={"subject_nr":"subject", "count_X":"order", "stimA": "A", 
                                                "stimX":"X", "stimB":"B"})
responsedf_eng['language'] = 'non-Panara'
responsedf_eng.head()<jupyter_output><empty_output><jupyter_text>The resulting dataframe has the following fields:- subject: the subject's ID- order: the order of stimulus presentation- A, X, B: the files associated with A, X, and B; during setup, the time-step of A is _always_ less than that of B, and in order to vary this, we relied on a randomly generated variable flip_order- flip_order: whether A and B were flipped during presentation; this is a fix for the problem of wanting to vary whether A or B has the higher step value, but having too many AXB sets to present subjects with the full factorial design- response: subjects' response- language: subjects' language category Next, we'll do the same for the Panara speakers' data.  Here, we also have an additional field:- repeats: the number of times the subject repeated each stimulus set before selecting their response<jupyter_code>panaradir = os.path.abspath('./panara/')
files = [f for f in os.listdir(panaradir) if f[-4:]=='.csv']
responsedf_pan = pd.DataFrame()
for f in files:
    tdf = pd.read_csv(os.path.join(panaradir,f))
    responsedf_pan = responsedf_pan.append(tdf, sort=False)
responsedf_pan = responsedf_pan[['subject','order','repeats','flip_order','A','X','B','response']]
responsedf_pan['language']='Panara'
responsedf_pan.head()<jupyter_output><empty_output><jupyter_text>We'll join the data together, but we should note from above two fields that differ in encoding between the data sets:- response: for English speakers, we expected responses of 1 or 0, corresponding to "the middle matches the first" and "the middle matches the last" respectively, but for Panara speakers we expected 'f' or 'l' instead.- flip_order: in the English data, these are "yes" and "no", but in the Panara data, these are "1" and "0" respectively.<jupyter_code>responsedf = pd.concat([responsedf_eng,responsedf_pan],sort=False)<jupyter_output><empty_output><jupyter_text>One problem we encounter early on is that not all responses match the expected responses.  So we'll repair this by making some reasonable guesses as to what the subjects _intended_ when they responded.<jupyter_code>responsedf.response.unique()
responsedf.response = responsedf.response.apply(lambda x: 'f' if x in ['e','d','ff'] else x)
# This response occurs specifically in one Panara subject's data, where '3' seems to have replaced 'l', as all other
# responses in this subject's data are 'f'
responsedf.response = responsedf.response.apply(lambda x: 'l' if x=='3' else x) 
responsedf.response = responsedf.response.apply(lambda x: 1 if x in [2,3,'1','`'] else x)
responsedf.response = responsedf.response.apply(lambda x: 0 if x in ['0','right'] else x)
responsedf = responsedf[~responsedf.response.isna()]
responsedf.response.unique()<jupyter_output><empty_output><jupyter_text>A couple more little fixes to make the data consistent.<jupyter_code>responsedf['order']=responsedf['order']+1
responsedf.response = responsedf.response.apply(lambda x: 1 if x=='f' else (0 if x=='l' else x))
responsedf.flip_order = responsedf.flip_order.apply(lambda x: 'yes' if x==1 else ('no' if x==0 else x))
responsedf.head()<jupyter_output><empty_output><jupyter_text>We know that a few Panara speakers didn't actually complete the study, so let's find those subjects and remove them from the data.<jupyter_code>incompletes = [s for s in responsedf.subject.unique() if len(responsedf[responsedf.subject==s])<198]
responsedf = responsedf[~responsedf.subject.isin(incompletes)]
incompletes<jupyter_output><empty_output><jupyter_text>We also observed that some Panara speakers didn't appear to understand the task.  In particular, we noticed that some subjects routinely responded with alternating responses, and others responded with a streak of the same response.  Both cases suggest to us that the subjects were participating in the task because they wished to be part of the study, but didn't truly understand what they were being asked to do.  The next several cells tries to locate these subjects (in both the Panara and non-Panara speaking groups) and remove them from the data. First, we'll define three functions, each of which creates a new column in a given dataframe.- streak1 and streak0: for each row, it returns the number of 0s or 1s in a row that the subject responded with.- altstreak: for each row, returns the number of alternating 0/1 responses in a row that the subject responded with.<jupyter_code>def streak1(x):
    x['streak1'] = x.groupby((x['response'] != 1).cumsum()).cumcount() + ((x['response'] != 1).cumsum() == 0).astype(int) 
    return x

def streak0(x):
    x['streak0'] = x.groupby((x['response'] != 0).cumsum()).cumcount() + ((x['response'] != 0).cumsum() == 0).astype(int) 
    return x

def altstreak(x):
    x['streakalt'] = x.groupby((x['streak'] != 1).cumsum()).cumcount() + ((x['streak'] != 1).cumsum() == 0).astype(int) 
    return x
responsedf = responsedf.groupby('subject', sort=False).apply(streak1)
responsedf = responsedf.groupby('subject', sort=False).apply(streak0)
responsedf['streak'] = responsedf.streak1+responsedf.streak0
responsedf = responsedf.groupby('subject', sort=False).apply(altstreak)
responsedf.head(10)<jupyter_output><empty_output><jupyter_text>Now, let's find the maximum streak of either 1s or 0s as well as the maximum alternating streak for each subject.When we plot these two values against each other, we see that the vast majority of our subjects cluster together, but a handful of Panara speaking subjects are clear outliers.  In particular, it appears that a cutoff of either 25 (less generous) or 50 (more generous) for both plain streaks and alternating streaks should handle all of these outliers.<jupyter_code>maxstreaks = responsedf.groupby(['subject'], as_index=False).aggregate(np.max)
ax = sns.scatterplot('streak', 'streakalt', data=maxstreaks, hue='language', alpha=.5)
fig = ax.get_figure()
fig.savefig('ABX_error1.pdf')
ax = sns.scatterplot('streak', 'streakalt', data=maxstreaks[maxstreaks.language=='non-Panara'], hue='language', alpha=.5)
fig = ax.get_figure()
fig.savefig('ABX_error2.pdf')<jupyter_output><empty_output><jupyter_text>Armed with this information, we can now restrict our data to only those subjects who we expect did attend to the task as we intended.<jupyter_code>maxstreaks['valid'] = maxstreaks.apply(lambda x: x.streak < 50 and x.streakalt < 50, axis=1)
valid = maxstreaks[maxstreaks.valid].subject.unique()
valid
responsedf = responsedf[responsedf.subject.isin(valid)]
responsedf['response_AB'] = responsedf.apply(lambda x: 'A' if (x.response==1 and x.flip_order=='no') 
                                             or (x.response==0 and x.flip_order=='yes') 
                                             else 'B', axis=1)
responsedf['change_vowel'] = responsedf.apply(lambda x: False if (x.A[0] in ['1','2'] and x.B[0] in ['1','2'])
                                              or (x.A[0] in ['3','4'] and x.B[0] in ['3','4'])
                                              or (x.A[0]==x.B[0])
                                              else True, axis=1)
responsedf['change_nsource'] = responsedf.apply(lambda x: False if (x.A[0] in ['1','3'] and x.B[0] in ['1','3'])
                                              or (x.A[0] in ['2','4'] and x.B[0] in ['2','4'])
                                              or (x.A[0]==x.B[0])
                                              else True, axis=1)
responsedf['change_burst'] = responsedf.apply(lambda x: x.A[2]!=x.B[2], axis=1)
responsedf.head()
responsedf['same_spectral']=responsedf.apply(lambda x: '' if not (x.change_vowel or x.change_nsource or x.change_burst)
                 else ('A' if x.A[0:3]==x.X[0:3] else 'B'), axis=1)
responsedf['same_temporal']=responsedf.apply(lambda x: 'A' if x.A[4]==x.X[4] else 'B', axis=1)
responsedf.head()
responsedf['match_temporal'] = responsedf.apply(lambda x: 1 if x.same_temporal==x.response_AB else 0, axis=1)
responsedf['match_spectral'] = responsedf.apply(lambda x: 1 if x.same_spectral==x.response_AB else 0, axis=1)
responsedf.head()
responsedf['A_time'] = responsedf.A.apply(lambda x: x[4])
responsedf['A_burst'] = responsedf.A.apply(lambda x: True if x[2] == '1' else False)
responsedf.head()
plaindf=responsedf[responsedf.same_spectral=='']
plaindf['series']=plaindf.A.apply(lambda x: x[0])
plain_agg = plaindf.groupby(['subject','language','series','A_time', 'A_burst'],as_index=False).aggregate(np.mean)
plain_agg.head()
plain_agg.groupby(['subject'],as_index=False).aggregate(np.mean)
plain_agg.groupby(['series','A_time'],as_index=False).aggregate(np.mean)
sns.lineplot(x='A_time',y='match_temporal',hue='language',style='series',data=plain_agg[plain_agg.A_burst])
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

sns.lineplot(x='A_time',y='match_temporal',hue='language',style='series',data=plain_agg[plain_agg.A_burst==False])
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
varydf=responsedf[responsedf.same_spectral!='']
vary_agg=varydf.groupby(['subject','language','change_vowel','change_nsource','change_burst','A_time','A_burst'],
                        as_index=False).aggregate(np.mean)
varydf.groupby(['language','change_vowel','change_nsource','change_burst','A_burst'],as_index=False).aggregate(np.mean)
vary_agg['change'] = vary_agg.apply(lambda x: 'vowel' if x.change_vowel else ('nsource' if x.change_nsource else 'burst'), axis=1)
vary_agg.groupby(['change','subject','language'],as_index=False).aggregate(np.mean)
f, axes = plt.subplots(1, 1, figsize=(5, 5))
ax = sns.lineplot(x='A_time',y='match_spectral',hue='language',style='change',data=vary_agg[vary_agg.language=='non-Panara'])
ax.axes.set(ylabel='Proportion match w/ spectral', xlabel="A step",ylim=[0,1])
ax.legend(fontsize='small')
fig = ax.get_figure()
fig.savefig('ABX2a.pdf')
f, axes = plt.subplots(1, 1, figsize=(5, 5))
ax = sns.lineplot(x='A_time',y='match_spectral',hue='language',style='change',data=vary_agg)
ax.axes.set(ylabel='Proportion match w/ spectral', xlabel="A step",ylim=[0,1])
ax.legend(fontsize='small')
fig = ax.get_figure()
fig.savefig('ABX2.pdf')
f, axes = plt.subplots(1, 1, figsize=(5, 5))
ax = sns.boxplot(x='change',y='match_spectral',hue='language',data=vary_agg)
ax.axes.set(ylabel='Proportion match w/ spectral')
ax.legend(loc='lower right')
fig = ax.get_figure()
fig.savefig('ABX2_box.pdf')
f, axes = plt.subplots(1, 1, figsize=(5, 5))
ax = sns.lineplot(x='A_time',y='match_temporal',hue='language',data=plain_agg[plain_agg.series!='5'])
# plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.axes.set(ylabel='Proportion correct responses', xlabel="A step",ylim=[0.5,1])
# sns.lineplot(x='A_time',y='match_temporal',hue='language',style='series',data=plain_agg[plain_agg.A_burst==False])
# plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
fig = ax.get_figure()
fig.savefig('ABX1.pdf') 
f, axes = plt.subplots(1, 1, figsize=(5, 5))
ax = sns.lineplot(x='A_time',y='match_temporal',style='series',hue='language',data=plain_agg[plain_agg.series!='5'])
# plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.axes.set(ylabel='Proportion correct responses', xlabel="A step",)
# sns.lineplot(x='A_time',y='match_temporal',hue='language',style='series',data=plain_agg[plain_agg.A_burst==False])
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
fig = ax.get_figure()
fig.savefig('ABX3.pdf',bbox_inches='tight') 
# f, axes = plt.subplots(1, 1, figsize=(5, 5))
# ax = sns.lineplot(x='A_time',y='match_temporal',style='series',data=plain_agg,color=sns.color_palette()[1])
# # plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
# ax.axes.set(ylabel='Proportion correct responses', xlabel="A step",)
# # sns.lineplot(x='A_time',y='match_temporal',hue='language',style='series',data=plain_agg[plain_agg.A_burst==False])
# # plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
# fig = ax.get_figure()

g = sns.FacetGrid(vary_agg, col="subject", col_wrap=8, height=2)
g.map(sns.lineplot, "A_time", "match_spectral", data=vary_agg, hue="language",style='change')
g.add_legend();
plain_agg.head()
g = sns.FacetGrid(plain_agg, col="subject", col_wrap=8, height=2)
g.map(sns.lineplot, "A_time", "match_temporal", data=plain_agg, hue="language",style='series')
g.add_legend();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15209572047272993, 'fraction_numerical': 0.015708727886662263, 'mean_word_length': 5.423385195662424, 'pattern_counts': {'":': 5, '<': 40, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '110219', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4843, 'n_tokens_neox': 4591, 'n_words': 1159, 'path': 'AXB.ipynb', 'repo_name': 'susanlinucb/panara_perception', 'size': '13623'}