<jupyter_start><jupyter_text>Active selection of source This notebook describes analyses of an experiment variant in which participants were allowed, on each trial, to select the source from which they wanted to receive a label. It was designed to assess whether (a) learners would increasingly prefer to select one source over another and (b) the category boundary would shift reliably toward the closer source.  Load data In this earlier experiment, data were analyzed before we adopted JuPyteR notebooks. I pre-processed the data in Excel and native R, then saved the data objects. This block reads in the data.<jupyter_code>allpars <- read.csv("allpars.csv", header = T) #Initial and final estimates of boundary from grid
rating <- read.csv("rating.csv", header = T)   #Ratings of faces following classification task
sourcesel <- read.csv("sourcesel.csv", header = T) #Source-selection data
head(allpars)<jupyter_output><empty_output><jupyter_text>In this early study, boundary shift was encoded so that positive numbers indicate a shift toward the closer source. In all other studies, however, we coded so that positive numbers indicate a shift toward the *far* source. So, I will flip the direction and rename the fields correspondingly:<jupyter_code>allpars$closepos <- -1 * allpars$closepos
allpars$Zclosepos <- -1 * allpars$Zclosepos
names(allpars)[11:12] <- c("farpos", "Zfarpos")

head(allpars)
#Rename sno column to have the same label as other data files (SID)
names(allpars)[3] <- "SID"<jupyter_output><empty_output><jupyter_text>Define functions<jupyter_code>zscore <- function(v) {(v-mean(v))/sqrt(var(v))}<jupyter_output><empty_output><jupyter_text>Source selection We first assessed whether people grew more likely to choose the source that agreed with them as the learnin phase of the experiment progressed. In preprocessing we divided the 301 learning trials into blocks of 50 (51 in final block), and for each block computed the proportion of trials for which the participant selected the source with the closer boundary. The questions are (1) are people more likely to choose the closer source on average across all trials, (2) do people grow more likely to choose the closer source over time and (3) do people who are more likely to choose the closer source *less* likely to shift their boundary toward the far source. First let's look at the mean proportion of "closer source" choices made by all subjects:<jupyter_code>mnprop <- rowMeans(sourcesel[,2:7]) #Vector of mean proportions
plot(mnprop, ylim = c(0,1), pch=16, col=as.numeric(abs(zscore(mnprop))>2)+1, ylab="Subject")
abline(h=.5)
sort(zscore(mnprop))<jupyter_output><empty_output><jupyter_text>Figure shows, across all trials, the proportion of times each student selected the source with the closer boundary. Red dots indicate outliers greater than 2 standard deviations from the mean. Five participants almost always selected the closer source; four of these count as outliers by our inclusion criteria.We next consider whether the mean proportion differs reliably from chance (0.5) with and without the outliers.<jupyter_code>#With all data:
t.test(mnprop-.5)<jupyter_output><empty_output><jupyter_text>On average all subjects choose the closer source 56% of the time, reliably above chance. This appears to be due solely to the fours outliers, however:<jupyter_code>mnprop.noo <- mnprop[abs(zscore(mnprop)) <= 2.0]
t.test(mnprop.noo-.5)<jupyter_output><empty_output><jupyter_text>With the four outliers removed, mean proportion selecting closer source is 51.6%, not reliably different from chance. Does this mean that 4-5 participants showed "confirmation bias" and the majority did not? To answer this question, we looked at whether the probability of selecting the closer source increased over the course of learning. We computed the probability of selecting the closer source for each subject in each of 6 successive blocks of 50 trials.<jupyter_code>plot(0,0,type ="n", ylim = c(0,1), xlim = c(1,6), ylab="p(closer)", xlab="Block")
tmp <- sourcesel[abs(zscore(mnprop)) <= 2.0,2:7]
tmpout <- sourcesel[abs(zscore(mnprop)) > 2.0,2:7]
for(i in c(1:dim(tmp)[1])) lines(c(1:6), tmp[i,], type = "o", pch=16, col=gray(.8))
for(i in c(1:dim(tmpout)[1])) lines(c(1:6), tmpout[i,], type = "o", pch=16, col=hsv(0,1,1,.35))

lines(c(1:6), colMeans(tmp), pch=16, cex=2, type="o")
abline(h=.5)<jupyter_output><empty_output><jupyter_text>This Figure shows mean probability of choosing the closer source for each subject at each block og 50 trials. The four outliers are shown in red, all other subjects in gray. Black circles/lines show the mean across subjects excluding outliers. Note:- Mean is near chance across all blocks- Mean probability does not increase over time as people learn more about the sources- The four outliers choose the closer source from the very first block--so likely just stick with their first choice.**Note:** Sources provide different labels for 65 of the 301 items, or about 21%, so in the first 50 trials the two sources differ on an expected 10 trials. Outlying subjects chose the far source on, at most, 20% of the trials in the first block--so they have contradictory evidence from the far source on about 2 trials. It seems unlikely that this is enough to generate a confirmation bias in these participants.We can check whether the probability differs from chance in the first block where the mean probability is highest:<jupyter_code>t.test(tmp[,1]-.5)<jupyter_output><empty_output><jupyter_text>For this contrast $t(33) =~ 1.87$, p = n.s. All other tests also not significant.  Relationship between source selection and boundary shift If boundary-shift data are influenced by biased information selection as suggested by classic confirmation bias, then subjects who are more likely to choose the closer source should also show less shift toward the far source. We therefore looked at the coorelation between the probability of choosing the closer source and the amount of shift.<jupyter_code>tmp <- allpars[allpars$farsource!="C",] #Discard controls since they did not get labels from sources
exp <- merge(tmp, sourcesel, by="SID") #Merge source-selection and boundary shift data
head(exp)
plot(exp$farpos, exp$All, pch=16, col = as.numeric(abs(zscore(exp$All)) > 2.0)+1,
     ylab="p(Closer)", xlab="Boundary shift", main="All trials", ylim = c(0,1))
abline(h=.5, v=0, lty=2)

plot(exp$farpos, exp$B6, pch=16, col = as.numeric(abs(zscore(exp$All)) > 2.0)+1,
     ylab="p(Closer)", xlab="Boundary shift", main="Final block", ylim = c(0,1))
abline(h=.5, v=0, lty=2)<jupyter_output><empty_output><jupyter_text>Figure shows probability of selecting the closer source plotted against the boundary shift. Positive numbers indicate a boundary shift toward the *far* source. If boundary shift was influenced by information selection as predicted by confirmation bias, these variable should show a negative correlation. Instead no correlation is observed:<jupyter_code>summary(lm(farpos~All, data=exp))
summary(lm(farpos~B6, data=exp))<jupyter_output><empty_output><jupyter_text>The outlying subjects who almost always chose the closer source did not differ from the rest of the group in their boundary shift, while the two participants who showed a strong shift toward the closer source showed approximately chance likelihood of choosing the closer versus the farther source in both the final block and across all trials. So, there is no evidence that the direction and amount of boundary shift is related to the participants' proability of choosing the closer source.  Assessing boundary shift Finally, we considered how the boundary shifted in this experiment, using the same approach as in the main paper. First we checked for outliers in the initial boundary and in the amount of boundary shift:<jupyter_code>plot(zscore(exp$t1bound), zscore(exp$farpos), xlim = c(-3.5,3.5), pch=16,
     ylim = c(-3.5,3.5), xlab = "Initial boundary", ylab="Boundary shift")
abline(h=c(-2,2), v=c(-2,2))<jupyter_output><empty_output><jupyter_text>Remove the three outliers:<jupyter_code>exp.noo <- exp[abs(zscore(exp$t1bound)) <= 2.0 & abs(zscore(exp$farpos)) <= 2.0,]
dim(exp.noo)<jupyter_output><empty_output><jupyter_text>Does the boundary shift differ from zero, and if so, in what direction?<jupyter_code>t.test(tmp.noo$farpos)<jupyter_output><empty_output><jupyter_text>There is a marginally significant shift toward the *far* source, similar to Experiment 1 but smaller in magnitude. Does the experimental condition differ from the control condition?<jupyter_code>plot(zscore(cntrl$t1bound), zscore(cntrl$farpos), xlim = c(-3.5,3.5), pch=16,
     ylim = c(-3.5,3.5), xlab = "Initial boundary", ylab="Boundary shift")
abline(h=c(-2,2), v=c(-2,2))<jupyter_output><empty_output><jupyter_text>Remove outliers from control sample:<jupyter_code>cntrl.noo <- cntrl[abs(zscore(cntrl$t1bound)) <= 2.0 & abs(zscore(cntrl$farpos)) <= 2.0,]
dim(cntrl.noo)
t.test(cntrl.noo$farpos, exp.noo$farpos)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09462007764836383, 'fraction_numerical': 0.018524681087077093, 'mean_word_length': 5.930053804765565, 'pattern_counts': {'":': 0, '<': 83, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '481942', 'license': 'MIT', 'n_tokens_mistral': 2732, 'n_tokens_neox': 2583, 'n_words': 1091, 'path': 'ExperimentSI1-ActiveSelection/ActiveSelection.ipynb', 'repo_name': 'ttrogers/frigo-chen-rogers', 'size': '9015'}