<jupyter_start><jupyter_text>Before you Start You will need to install several things before you can start  1. Installing Python dependencies with pip:    Go download R from https://www.r-project.org    Open a new R console and run:        ~ install.packages('IRkernel')        ~ install.packages('ggplot2')        Then, in your terminal, run:        ~ R        ~ IRkernel::installspec()    For more information on installing IRkernel, visit: https://irkernel.github.io/installation  2. Download a JDK    Download and follow prompts    https://www.oracle.com/technetwork/java/javase/downloads/jdk12-downloads-5295953.html  3. Set up R in Jupyter Notebooks    Follow the instructions here through step 3    https://docs.anaconda.com/anaconda/navigator/tutorials/r-lang/  Outline 1) [What is Factor Analysis](background)2) [Prep (including PCA)](prep)3) [Factor Analysis](analysis)3) [Using the Factors](use)  What is Factor Analysis Factor analysis is a dimension reduction method, combining multiple variables into a single component.  For example, say we wanted to measure a student's confidence over the course of this survey.  It is possible to include their reported confidence for each question and use them as individual predictors, but really we care more about their overall confidence.  Thus, we use the average confidence levels over the entire study.  This will mitigate issues of multicolinearity and make the conclusions more intuative and easy to understand.  Average confidence is a straightforward dimension reduction example.  Factor analysis is a slightly more involved way to do this in a more general setting.    Background We will be using data from Professor <NAME>'s QuaRCS quantitative skills assesment.  The goal of this survey is to measure the quantitative skills in STEM for non-majors courses, with the hope that an accurate and reliable metric will help professors improve their courses.  For most of this walkthrough we will assume you are familiar with the project or care only about the techniques rather than the content.    What is Factor Analysis If you are unfamiliar with factor analysis, there are a lot of great background readings.  Here are a couple:**General Background**https://en.wikipedia.org/wiki/Factor_analysishttp://ocw.jhsph.edu/courses/StatisticsPsychosocialResearch/PDFs/Lecture8.pdf**With this kind of survey data (mostly focused on exploratory)**http://www.let.rug.nl/nerbonne/teach/rema-stats-meth-seminar/Factor-Analysis-Kootstra-04.PDFhttp://pareonline.net/pdf/v10n7.pdfhttp://www.statisticssolutions.com/factor-analysis-sem-exploratory-factor-analysis/**Computing in R**https://www.statmethods.net/advstats/factor.htmlhttps://www.r-bloggers.com/factor-analysis-introduction-with-the-principal-component-method-and-r/  Typical Outline 1) Do a PCA (with Scree Plot) to determine how many factors    a) One factor before the "elbow" is often a good number2) **Exploratory Factor Analysis** - What we're mostly doing here3) Confirmatory Factor Analysis  Factor Analysis Prep We begin by loading in the required libraries and reading in the data.<jupyter_code>install.packages('mosaic')
install.packages('foreign')
install.packages('tidyverse')
install.packages('magrittr')
install.packages('psych')
install.packages('GPArotation')
suppressMessages(library(mosaic))
suppressMessages(library(foreign))
suppressMessages(library(tidyverse))
suppressMessages(library(magrittr))
suppressMessages(library(psych))
suppressMessages(library(GPArotation))<jupyter_output><empty_output><jupyter_text>------------------------------------------***Place To Change: Read in the correct data***-----------------------------------------------------<jupyter_code>suppressWarnings(df <- read.csv("QuaRCS_Lt2_Pre_scored.csv"))<jupyter_output><empty_output><jupyter_text>---------------------------------------------------------------------------------------------------------------------- Then, we need to do some data cleaning steps.  Basically, I am selecting only those variables that I care about.  I am also converting the likard questions into numerical format, as factor analysis in R requires quantitative variables.  There is a way to read in SPSS data with all numerical values, but I originally made this to work on a .csv file that didn't have those features and I was too lazy to change it.<jupyter_code># Here I'm averaging the students confidence for all the questions

# First, define the variables of interest
vars <- c("PRE_FREQEN","PRE_DAILYM","PRE_DAILYG","PRE_ATT_DL_1", "PRE_ATT_DL_2", 
          "PRE_ATT_DL_3", "PRE_ATT_DL_4", "PRE_ATT_DL_5", "PRE_ATT_SC_1", "PRE_ATT_SC_2", 
          "PRE_ATT_SC_3", "PRE_ATT_SC_4", "PRE_ATT_SC_5", "PRE_LK1", "PRE_LK2", "PRE_LK5",
          "PRE_ANX.1_1", "PRE_ANX.1_2", "PRE_ANX.1_3", "PRE_ANX.1_4","PRE_WHYMAJ_1",
          "PRE_WHYMAJ_2","PRE_WHYMAJ_3","PRE_WHYMAJ_4","PRE_WHYMAJ_5", 
          "PRE_CF_TOTAL")

#### Now select the data
d <- df[,c(vars)]


# Sometimes, SPSS recorded "No" as NA.  This changes it to 0
#d[, c("PRE_WHYMAJ_LIKE","PRE_WHYMAJ_LIKEJOB","PRE_WHYMAJ_MONEY","PRE_WHYMAJ_GOOD","PRE_WHYMAJ_NOMATH")][is.na(d[, c("PRE_WHYMAJ_LIKE","PRE_WHYMAJ_LIKEJOB","PRE_WHYMAJ_MONEY","PRE_WHYMAJ_GOOD","PRE_WHYMAJ_NOMATH")])] <- 0<jupyter_output><empty_output><jupyter_text>Once the data is cleaned and ready to use, we can start the factor analysis! ------------------------------------------***Place To Change: Select the correct vatiables***-----------------------------------------------------<jupyter_code># First, we need students to answer all the questions.  We cannot use incomplete data
d0 <- d[complete.cases(d),]

# Uncomment to see the variable names of data so that you can select the ones you want
names(d0)

# This is another round of variable selection
# For instance, for now we believe that "Why Students Choose Their Major" does not matter
d1 <- d0[,c(1:20, 26)]
d1 <- d1[,c(-11,-5,-4,-7,-8)]
colnames(d1)<jupyter_output><empty_output><jupyter_text>----------------------------------------------------------------------------------------------------------------------  How Many Factors? One we've selected the variables we want to use, we need to decide how many factors to create.  We'll start with a [Scree plot](https://www.theanalysisfactor.com/factor-analysis-how-many-factors/), which shows the amount of additional varaince each factor will explain.  Typically, we want to choose factors that explain more varaince, so when the plot levels off (often called the "elbow") we stop.  We can also use linear algebra and calculate the eigen values of the correlation matrix, with important factors having an eigen value greater than 1.  The two methods sometimes produce slightly different restults, so go with whatever makes the most sense.<jupyter_code>fit <- princomp(d1, cor=TRUE)
plot(fit,type="lines")<jupyter_output><empty_output><jupyter_text>The scree plot suggests that we may need as many as 5 or 6 factors to explain the variance in the data.  With effort counting as one on it's own, that means we need 4 or 5 combined factors.<jupyter_code>summary(fit)<jupyter_output><empty_output><jupyter_text>Again, there is an art as to when we are comfortable stopping.  This table shows the additional variance in the data explained by each new factor.  There is no hard and fast rule as to where to stop.  We chose 5 (again, effort counts for one on its own, so we generated 4 factors).  Adding too many factors minimizes the effectiveness of factor analysis and potentially leave you vulnerable to [overfitting](http://scott.fortmann-roe.com/docs/BiasVariance.html).  Factor Analysis<jupyter_code># This is the actual Factor Analysis

## We start by generating a correlation matrix
f <- cor(d1, use="complete.obs")

## Oblimin Factor Analysis to allow correlation between predictors
preK <-  fa(r=f, nfactors = 4,fm="ml", rotate ="oblimin",  oblique.scores = TRUE, max.iter = 25)
## This is a kaiser rotation.  It's standard in SPSS, but isn't used that much elsewhere
## The reason we use it here is to replicate earlier findings from this lab, which used SPSS for analysis
load4 <- kaiser(preK)


# Here I'm extracting the loadings into a nice little table so we can analyze
loadings4 <- load4$loadings[c(1:ncol(d1)),]
loadings4 <- apply(loadings4, 2, function(x) ifelse(abs(x)>0.32,x,NA))

# Look at factor loadings
loadings4<jupyter_output><empty_output><jupyter_text>Here we can see which factors load togeather and how strong they load.  I only included loadings that are greater than 0.32 (which is a common cutoff for significance).  This particular matrix is known as a [pattern matrix](https://stats.stackexchange.com/questions/166799/which-matrix-should-be-interpreted-in-factor-analysis-pattern-matrix-or-structu).  We are analyzing this rather than the structure matrix because we assume some correlation between our factors.  Read the link to understand more about the difference.  Creating the Factors Now that we've fit the factors, we can use them to generate scores for each student.  The orignal analysis had done an [Anderson-Rubin scoring mechanism](https://www.ibm.com/support/knowledgecenter/en/SSLVMB_24.0.0/spss/base/idh_fact_sco.html), which is notated in R as "tenBerge".<jupyter_code>factor <- factor.scores(d1, load4,  method = "tenBerge")
factors <- factor$scores

score <- df$PRE_TOTAL
effort <- df$PRE_EFFORT

test <- cbind(score = score, factors, effort)
test <- as.data.frame(test)<jupyter_output>Warning message in cbind(score = score, factors, effort):
“number of rows of result is not a multiple of vector length (arg 1)”<jupyter_text>Now that we have the factors, we should look at a correlation table to ensure that we're not going to run into multicollinearity issues with factors that are too highly correlated with each other.<jupyter_code>cor(test)<jupyter_output><empty_output><jupyter_text>There is not a huge correlation between the factors.  Effort is fairly highly correlated with everything except for our second factor, which will turn out to be non-informative anyways.  This isn't ideal, but was expected. Now we can see how well these factors predict assesment performance.  If we had a large sample, we would ideally train the factors on a distinct subset from the testing data.  This was not done here as our primary goal was to reproduce previous findings.  Using the Factors<jupyter_code># Factor 2 was determined to be not significent statistically or in practice and was dropped from the model
m4 <- lm(score~., data = test)

msummary(m4)<jupyter_output><empty_output><jupyter_text>From here you can add other variables into the model, such as sex, race and major.  These factors now simply serve as regular predictors.    How did the factors do? We now want to see how each additional factor did.  We'll run a linear model adding each factor one at a time and see how much of the variance in the data it explains (R^2).<jupyter_code>names(test)
m1s <- lm(score~ML4+ML1+ML3+effort, data = test)
m2s <- lm(score~effort + ML1, data = test)
m4s <- lm(score~effort + ML1 + ML4, data = test)
m5s <- lm(score~effort + ML1 + ML4 + ML3, data = test)
m6s <- lm(score~effort + ML1 + ML4 + ML3 + ML2, data = test)
msummary(m1s)
df <- data.frame(Model = seq(1,5,1), R2 = c(summary(m1s)$r.squared,summary(m2s)$r.squared,summary(m4s)$r.squared,summary(m5s)$r.squared,summary(m6s)$r.squared))
ggplot(df)+
geom_point(aes(Model, R2))+
labs(list(title = "Effectiveness of Each Additional Factor in Linear Regression Model"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12256049960967993, 'fraction_numerical': 0.012750455373406194, 'mean_word_length': 4.165770609318996, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 46, 'https://': 10, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '149487', 'license': 'MIT', 'n_tokens_mistral': 3472, 'n_tokens_neox': 3270, 'n_words': 1283, 'path': 'tutorials/factor analysis tutorial/Factor Analysis Walkthrough.ipynb', 'repo_name': 'kfollette/QuaRCS', 'size': '11535'}