<jupyter_start><jupyter_text>Predicting Quality of Teaching and Learning at CMU Author: <NAME> Click [here](http://www.hexingren.com/practical-data-science) to go back.  Introduction Faculty Course Evaluation ([FCE](http://www.cmu.edu/hub/fce/index.html)) is a survey that all CMU students are encouraged to participate in to give constructive feedback about content of the courses they've taken and the instructors' teaching by the end of each semester. Responses to the FCE provide information on students' perceptions of their engagement, learning outcomes, the instructor's behavior and course activities[1]. The feedback will help guide changes in future iterations of the course and/or the instructor's teaching[1].Overall, there are 19 indexes listed in FCE. There are 11 of them involved in our discussion. They are Response Rate(%), Workload, Insterest in Student Learning, Explain Course Requirements, Clear Learning Goals, Feedback To Students, Importance Of Subject, Explain Subject Matter, Show Respect For Students, Overall Teaching, and Overall Course. The Response Rate is presented in percentile, Workload in hours per week and the other 9 variables are numerical between 0.0 and 5.0 representing average score for a specific part about a course from the students in the course.In this report, we are intersted in 3 questions. First, how is the overall quality of teaching and course content at CMU? Second, what indexes are related to quality of teaching and course content respectively? Are these two groups of indexes different? Last but not least, we try to build normal multivariate linear models to predict quality of teaching and course content.The report is organized as follows.- [Introduction](Introduction)- [Data Collecting and Cleansing](Data-Collecting-and-Cleansing)- [Exploratory Data Analysis](Exploratory-Data-Analysis)- [Initial Modeling and Diagnostics](Initial-Modeling-and-Diagnostics)- [Results](Results)- [Discussion](Discussion)  Data Collecting and Cleansing Several Python libraries are imported to fetch and parse webpages, and save the data as csv files.<jupyter_code># setup library imports
import requests, csv
from bs4 import BeautifulSoup
# unicode
import unicodedata
def retrieve_html(url):
    res = requests.get(url)
    return res.status_code, res.text
scs2015 = open('scs2015.html').read()
scs2016 = open('scs2016.html').read()
# print type(scs2015)<jupyter_output><empty_output><jupyter_text>BeautifulSoup is used to parse webpages and the values of different features are stored in a dictionary with their corresponding feature as a key.<jupyter_code>def parse_page(html):
    def parse_record(record):
        formatted = {}
        # Semester
        formatted['SEM'] = str(record.find_all('td')[1].text)
        # Year
        formatted['YR'] = str(record.find_all('td')[2].text)
        # Instructor
        formatted['INST'] = str(unicodedata.normalize('NFKD', record.find_all('td')[3].text))
        # Department
        formatted['DEPT'] = str(record.find_all('td')[4].text)
        # Note that cid is not primary key here.
        # Course ID
        formatted['CID'] = str(record.find_all('td')[5].text)
        # Course Name
        formatted['CNAME'] = str(unicodedata.normalize('NFKD', record.find_all('td')[6].text))
        # Section
        formatted['SEC'] = str(record.find_all('td')[7].text)
        # Type
        formatted['TYPE'] = str(record.find_all('td')[8].text)
        # Response Rate
        formatted['RR'] = str(record.find_all('td')[9].text)
        # Workload(hrs/week)
        formatted['WL'] = str(unicodedata.normalize('NFKD', record.find_all('td')[10].text))
        # Interest In Student Learning
        formatted['IISL'] = str(record.find_all('td')[11].text)
        # Explain Course Requirements
        formatted['ECR'] = str(record.find_all('td')[12].text)
        # Clear Learning Goals
        formatted['CLG'] = str(record.find_all('td')[13].text)
        # Feedback To Students
        formatted['FTS'] = str(record.find_all('td')[14].text)
        # Importance Of Subject
        formatted['IOS'] = str(record.find_all('td')[15].text)
        # Explains Subject Matter
        formatted['ESM'] = str(record.find_all('td')[16].text)
        # Show Respect For Students
        formatted['SRFS'] = str(record.find_all('td')[17].text)
        # Overall Teaching
        formatted['OT'] = str(record.find_all('td')[18].text)
        # Overall Course
        formatted['OC'] = str(record.find_all('td')[19].text)
        return formatted

    soup = BeautifulSoup(html, 'html.parser')
    records = soup.find('table', summary="Survey Results").find('tbody').find_all('tr')
    structured = [parse_record(record) for record in records]
    return structured<jupyter_output><empty_output><jupyter_text>The data in the page which contains all the SCS courses in 2015 is saved as the training data set. There are roughly 800 instances.<jupyter_code>toCSV = parse_page(scs2015)
keys = toCSV[0].keys()
with open('scs2015.csv', 'wb') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(toCSV)<jupyter_output><empty_output><jupyter_text>The data in the page which contains all the SCS courses in 2016 in saved as the testing data set. There are roughly 480 instances, which is around a half of the number of the instances in the training data set.<jupyter_code>toCSV = parse_page(scs2016)
keys = toCSV[0].keys()
with open('scs2016.csv', 'wb') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(toCSV)<jupyter_output><empty_output><jupyter_text>Table 1 shows the variables in the dataset and the corresponding abbreviations we will use in the code.| Full name | Abbreviation| :- | :-| Semester | SEM| Year | YR| Instructor | INST| Department | DEPT| Course ID | CID| Course Name | CNAME| Section | SEC| Type | TYPE| Response Rate(%) | RR| Workload(hrs/week) | WL| Interest In Student Learning | IISL| Explain Course Requirements | ECR| Clear Learning Goals | CLG| Feedback To Students | FTS| Importance Of Subject | IOS| Explain Subject Matter | ESM| Show Respect For Students | SRFS| Overall Teaching | OT| Overall Course | OCTable 1. Variables in the dataset and the corresponding abbreviations The data needs to be cleaned to better fit the R environment I am going to use to conduct the analysis work.<jupyter_code># set working directory
setwd('Desktop/388project/')
getwd()
# load the data
# "more columns than column names" error
# the following steps of loading the data
# learnd from http://stackoverflow.com/questions/25771071/r-read-csv-more-columns-than-column-names-error
# turn that vector into a text connection for read.csv

scs15_1 = readLines('scs2015.csv')
summary(scs15_1)
Sys.setlocale('LC_ALL', 'C')
scs15_2 = grep('Node Name, RTC_date', scs15_1, invert=TRUE, value=TRUE)
# 'header=TRUE' is the key!!!!!!
scs15_3 = read.csv(textConnection(paste0(scs15_2, collapse='\n')), header=TRUE, stringsAsFactors=FALSE)
str(scs15_3)
# grab the headers
headers = strsplit(scs15_1[1], ',')[[1]]
# how many of them are there?
length(headers)
# limit it to the 19 columns I want (Which matches)
scs15 = scs15_3[, 1:19]
# and add the headers
colnames(scs15) = headers
# remove '%' in Response Rate
scs15$RR = gsub('%', '', paste(scs15$RR))
scs15$RR = as.numeric(scs15$RR)

scs16_1 = readLines('scs2016.csv')
summary(scs16_1)
Sys.setlocale('LC_ALL', 'C')
scs16_2 = grep('Node Name, RTC_date', scs16_1, invert=TRUE, value=TRUE)
# 'header=TRUE' is the key!!!!!!
scs16_3 = read.csv(textConnection(paste0(scs16_2, collapse='\n')), header=TRUE, stringsAsFactors=FALSE)
str(scs16_3)
# grab the headers
headers = strsplit(scs16_1[1], ',')[[1]]
# how many of them are there?
length(headers)
# limit it to the 19 columns I want (Which matches)
scs16 = scs16_3[, 1:19]
# and add the headers
colnames(scs16) = headers
# remove '%' in Response Rate
scs16$RR = gsub('%', '', paste(scs16$RR))
scs16$RR = as.numeric(scs16$RR)<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis Data framework is constrained to the 11 variables which are initially considered relevant in the problem. The few instances (only one in the scs15 file) with missing values are removed. Table 2 gives the summary statistics for Overall Teaching, Overall Course and the other 9 continuous variables that we are interested in. Figure 1 show the histograms of Overall Teaching, Overall Course and the other continous variables.<jupyter_code># EDA
vec = c('RR', 'WL', 'IISL', 'ECR', 'CLG', 'FTS', 'IOS', 'ESM', 'SRFS', 'OT', 'OC')
scs15 = scs15[vec]
scs15 = na.omit(scs15)
nrow(scs15)
summary(scs15)
sd(scs15$RR)
sd(scs15$WL)
sd(scs15$IISL)
sd(scs15$ECR)
sd(scs15$CLG)
sd(scs15$FTS)
sd(scs15$IOS)
sd(scs15$ESM)
sd(scs15$SRFS)
sd(scs15$OT)
sd(scs15$OC)<jupyter_output><empty_output><jupyter_text>| Variables | Min | Max | Mean | Median | Std. Dev| :- | :- | :- | :- | :- | :- | RR | 17.00 | 100.00 | 67.48 | 68.00 | 18.94| WL | 2.00 | 27.00 | 13.06 | 11.77 | 6.21| IISL | 1.14 | 5.00 | 4.43 | 4.50 | 0.47| ECR | 1.29 | 5.00 | 4.26 | 4.34 | 0.57| CLG | 1.57 | 5.00 | 4.31 | 4.40 | 0.53| FTS | 1.57 | 5.00 | 4.18 | 4.27 | 0.57| IOS | 1.71 | 5.00 | 4.41 | 4.50 | 0.48| ESM | 1.57 | 5.00 | 4.41 | 4.50 | 0.48| SRFS | 1.71 | 5.00 | 4.55 | 4.65 | 0.43| OT | 1.43 | 5.00 | 4.30 | 4.42 | 0.56| OC | 2.00 | 5.00 | 4.25 | 4.33 | 0.56Table 2. Summary statistics of univariate distributions<jupyter_code>pdf(file='hist1.pdf', width=11, height=12)
par(mfrow=c(3, 2))
hist(scs15$RR, breaks=30, col=3, cex=1.3,
     xlab='Response Rate (%)', main='Histogram of Response Rate')
hist(scs15$WL, breaks=30, col=3, cex=1.3,
     xlab='Workload (hrs/week)', main='Histogram of Workload')

hist(scs15$IISL, breaks=30, col=3, cex=1.3,
     xlab='Interest In Student Learning', main='Histogram of Interest In Student Learning')
hist(scs15$ECR, breaks=30, col=3, cex=1.3,
     xlab='Explain Course Requirements', main='Histogram of Explain Course Requirements')

hist(scs15$CLG, breaks=30, col=3, cex=1.3,
     xlab='Clear Learning Goals', main='Histogram of Clear Learning Goals')
hist(scs15$FTS, breaks=30, col=3, cex=1.3,
     xlab='Feedback To Students', main='Histogram of Feedback To Students')
par(mfrow=c(1, 1))
dev.off()

pdf(file='hist2.pdf', width=11, height=12)
par(mfrow=c(3, 2))
hist(scs15$IOS, breaks=30, col=3, cex=1.3,
     xlab='Importance Of Subject', main='Histogram of Importance Of Subject')
hist(scs15$ESM, breaks=30, col=3, cex=1.3,
     xlab='Explains Subject Matter', main='Histogram of Explains Subject Matter')

hist(scs15$SRFS, breaks=30, col=3, cex=1.3,
     xlab='Show Respect For Students', main='Histogram of Show Respect For Students')
hist(scs15$OT, breaks=30, col=3, cex=1.3,
     xlab='Overall Teaching', main='Histogram of Overall Teaching')

hist(scs15$OC, breaks=30, col=3, cex=1.3,
     xlab='Overall Course', main='Histogram of Overall Course')
par(mfrow=c(1, 1))
dev.off()<jupyter_output><empty_output><jupyter_text>Figure 1. Histograms of Response Rate, Workload, Interest in Student Learning, Explain Course Requirements, Clear Learning Goals, Feedback to Students, Importance of Subject, Explain Subject Matter, Show Respect For Students, Overall Teaching and Overall Course. There are 798 observations in the dataset after the only one with missing values is removed. Two points need to be stressed are, first, a relatively high bar exists in almost all the histograms, which suggests a number of observations have 'full marks'; second, almost all the distributions are skewed to the left except that of Workload.The values of Response Rate range from 17.00% to 100.00% with a mean value of 67.48% and standard deviation 18.94. The distribution of Workload is bimodal(mode=10 and 27 hours/week). The values (hours/week) range from 2.00 to 27.00 with a mean value of 13.06 and standard deviation 6.21. The values of Interest In Student Learning range from 1.14 to 5.00 with a mean value of 4.43 and standard deviation 0.47. The thin left tail of the distribution is in part due to the observation at around 1. There is a wide gap between this observation and the rest of values which makes it possible outlier. Similarly, we can find one or a couple of possible outliers in the histograms of Explian Course Requirements, Clear Learning Goals, Feedback To Students, Importance Of Subject, Explains Subject Matter, Show Respect For Students, Overall Teaching and Overall Course, which lead to the thin left tails.<jupyter_code>pdf(file='scatter_before_transformation_OT.pdf', width=11, height=12)
pairs(OT ~ RR + WL + IISL + ECR + CLG + FTS + IOS + ESM + SRFS,
      data=scs15, pch=16, col=2, cex=0.5)
dev.off()

pdf(file='scatter_before_transformation_OC.pdf', width=11, height=12)
pairs(OC ~ RR + WL + IISL + ECR + CLG + FTS + IOS + ESM + SRFS,
      data=scs15, pch=16, col=2, cex=0.5)
dev.off()<jupyter_output><empty_output><jupyter_text>Figure 2. Scatter plots comparing Overall Teaching and the predictors for the Faculty Course Evaluation dataset. Figure 2 shows the scatter plots comparing Overall Teaching and 9 possible predictors in the dataset. Possible linear relationships may exist between the response(Overall Teaching) and single predictor(IISL, ECR, CLG, FTS, IOS or ESM). Linear relationships between predictors may also exist. We should be concerned about possible multicollinearity here. There are certain possible outliers observed in the scatter plots. For example, the individual at around 1 of Overall Teaching; the individual at around 1.5 of Clear Learning Goals. These suspicious outliers should correspond to those we observed in the histograms. We should be aware that it could be influential and check if there were errors in the data. Figure 3. Scatter plots comparing Overall Course and the predictors for the Faculty Course Evaluation dataset. Figure 3 shows the scatter plots comparing Overall Course and 9 possible predictors in the dataset. Similarly, multicollinearity should be our concern due to the possible linear relationships between the predictors. Also, we should double check those possible influential observations.  Initial Modeling and Diagnostics Based on the previous exploratory data analysis, all the 9 continuous variables are intially included to build models to predict Overall Teaching scores and Overall Course scores.<jupyter_code>scs15_OT_lm0 = lm(OT ~ RR + WL + IISL + ECR + CLG + FTS + IOS + ESM + SRFS, data=scs15, x=T)
summary(scs15_OT_lm0)<jupyter_output><empty_output><jupyter_text>The output:<jupyter_code>scs15_OC_lm0 = lm(OC ~ RR + WL + IISL + ECR + CLG + FTS + IOS + ESM + SRFS, data=scs15, x=T)
summary(scs15_OC_lm0)<jupyter_output><empty_output><jupyter_text>The output: To diagnose problems with the fit of the initial model, we use residual analysis. Figure 4 shows the plots of residuals versus fitted values in Overall Teaching and Overall Course models. Figure 5 shows the normal probability plots for these two models. The conclusions we draw from the models are based on the following assumptions:1. The error ε has mean zero.In both residual plots, the residuals are symmetrically distributed around zero, so there is little reason to doubt this assumption.2. The error ε is uncorrelated between observations.No special pattern appear in both residual plots, so there is little concern about nonlinearity or heteroskedasticity.3. The error ε has equal variance across observations.Both residual plots show an equal spread around zero for all the fitted values, suggesting that this assumption is reasonable.4. The error ε is normally distributed.The distributions in both normal probability plots have a little heavy tails on both sides. This may be a concern.We may use two ways to address this problem. First, we can apply certain transformations to the responses and predictors, and build models from there. Second, we may want to remove some predictors in the model.<jupyter_code>pdf(file='res0.pdf', width=11, height=6)
par(mfrow=c(1, 2))
plot(fitted(scs15_OT_lm0), resid(scs15_OT_lm0), xlab='Fitted Values', ylab='Residuals',
     cex=1.3, pch=16)
plot(fitted(scs15_OC_lm0), resid(scs15_OC_lm0), xlab='Fitted Values', ylab='Residuals',
     cex=1.3, pch=16)
par(mfrow=c(1, 1))
dev.off()<jupyter_output><empty_output><jupyter_text>Figure 4. Residual plots for the Faculty Course Evaluation dataset(left: Overall Teaching model; right: Overall Course model) To identify the possible influential observations in the data, we calculate the Cook's Distance and compare the value to the quantiles of the F distribution with 10 and 788 degrees of freedom. We found that that even the most influential observation (8.57e-03 percentile for Overall Teaching model and 9.23e-02 percentile) lies far below the 20th percentile. There is no reason to concern.<jupyter_code>cookd0_OT = cooks.distance(scs15_OT_lm0)
sort(pf(cookd0_OT, 10, 788), decreasing=TRUE)[1:5]

cookd0_OC = cooks.distance(scs15_OC_lm0)
sort(pf(cookd0_OC, 10, 788), decreasing=TRUE)[1:5]<jupyter_output><empty_output><jupyter_text>The cook's distance for Overall Teaching model:The cook's distance for Overall Course model:<jupyter_code>pdf(file='normprob0.pdf', width=11, height=6)
par(mfrow=c(1, 2))
qqnorm(scs15_OT_lm0$residuals, xlab='Theoretical Quantiles', ylab='Sample Quantiles',
       main='Normal Probability Plot (Overall Teaching)', pch=16)
qqline(scs15_OT_lm0$residuals)
qqnorm(scs15_OC_lm0$residuals, xlab='Theoretical Quantiles', ylab='Sample Quantiles',
       main='Normal Probability Plot (Overall Course)', pch=16)
qqline(scs15_OC_lm0$residuals)
par(mfrow=c(1, 1))
dev.off()<jupyter_output><empty_output><jupyter_text>Figure 5. Normal probability plot of the original model. Up till now, the only concern that remains is the assumption of the normal distribution of the error. The heavy tails of the distributions in previous normal probability plots suggest that there are certain deviations between the distributions of the variables in the model and normal distribution.Let's look back at the histograms in Figure 1. To alleviate the effect of the high bars on the right side, I try using an inverse transformation. After this, the histograms are supposed to slightly skewed to the right rather than the left. Then, a logarithm transformation is applied to the responses and the predictors. Ideally, these variables are supposed to be more normally distributed after the transformations.<jupyter_code>scs15$RR_inv_log = log(1.0 / scs15$RR)
scs15$WL_inv_log = log(1.0 / scs15$WL)
scs15$IISL_inv_log = log(1.0 / scs15$IISL)
scs15$ECR_inv_log = log(1.0 / scs15$ECR)
scs15$CLG_inv_log = log(1.0 / scs15$CLG)
scs15$FTS_inv_log = log(1.0 / scs15$FTS)
scs15$IOS_inv_log = log(1.0 / scs15$IOS)
scs15$ESM_inv_log = log(1.0 / scs15$ESM)
scs15$SRFS_inv_log = log(1.0 / scs15$SRFS)
scs15$OT_inv_log = log(1.0 / scs15$OT)
scs15$OC_inv_log = log(1.0 / scs15$OC)<jupyter_output><empty_output><jupyter_text>Figure 6. Histograms of Response Rate, Workload, Interest in Student Learning, Explain Course Requirements, Clear Learning Goals, Feedback to Students, Importance of Subject, Explain Subject Matter, Show Respect For Students, Overall Teaching and Overall Course after the inverse and logarithm transformations. After the inverse and logarithm transformations, several narrow "bell" shapes seem to appear in the distributions of the variables. But several of them are skewed to the right with thin tails. I refit the models with all the responses and the predictors being transformed.<jupyter_code>scs15_OT_lm1 = lm(OT_inv_log ~ RR_inv_log + WL_inv_log + IISL_inv_log + ECR_inv_log + 
                    CLG_inv_log + FTS_inv_log + IOS_inv_log + ESM_inv_log + SRFS_inv_log, 
                  data=scs15, x=T)
scs15_OC_lm1 = lm(OT_inv_log ~ RR_inv_log + WL_inv_log + IISL_inv_log + ECR_inv_log + 
                    CLG_inv_log + FTS_inv_log + IOS_inv_log + ESM_inv_log + SRFS_inv_log, 
                  data=scs15, x=T)
pdf(file='res1.pdf', width=11, height=6)
par(mfrow=c(1, 2))
plot(fitted(scs15_OT_lm1), resid(scs15_OT_lm1), xlab='Fitted Values', ylab='Residuals',
     cex=1.3, pch=16)
plot(fitted(scs15_OC_lm1), resid(scs15_OC_lm1), xlab='Fitted Values', ylab='Residuals',
     cex=1.3, pch=16)
par(mfrow=c(1, 1))
dev.off()<jupyter_output><empty_output><jupyter_text>The residual plots in Figure 7 look fine and there is no concern about the first three assumptions. Figure 7. Residual plots for the Faculty Course Evaluation dataset after inverse and logarithm transformations<jupyter_code>pdf(file='normprob1.pdf', width=11, height=6)
par(mfrow=c(1, 2))
qqnorm(scs15_OT_lm1$residuals, xlab='Theoretical Quantiles', ylab='Sample Quantiles',
       main='Normal Probability Plot (Overall Teaching)', pch=16)
qqline(scs15_OT_lm1$residuals)
qqnorm(scs15_OC_lm1$residuals, xlab='Theoretical Quantiles', ylab='Sample Quantiles',
       main='Normal Probability Plot (Overall Course)', pch=16)
qqline(scs15_OC_lm1$residuals)
par(mfrow=c(1, 1))
dev.off()<jupyter_output><empty_output><jupyter_text>Figure 8. Normal probability plots of the models after inverse and logarithm transformations. Comparing the normal probability plots of models after inverse and logarithm transformations in Figure 8 and the original normal probability plots in Figure 5, we can see that, although the transformations alleviate the heavy left tails slightly, they actually make the heavy right tails heavier. I also tried the [Yeo-Johnson Power Transformation](https://www.stat.umn.edu/arc/yjpower.pdf) (see 388project.R), but the tails still exist. Besides, one of the drawbacks of applying such a sophisticated transformation to the problem is that we can hardly describe the relationships between the original varibles (in the original feature space) intuitively. Thus, I prefer a simpler model with no transformation.  Results We now perform an exhaustive variable selection procedure using Akaike's Information Criterion([AIC](https://en.wikipedia.org/wiki/Akaike_information_criterion)).<jupyter_code>install.packages('bestglm')
library(bestglm)
vec = c('RR', 'WL', 'IISL', 'ECR', 'CLG', 'FTS', 'IOS', 'ESM', 'SRFS', 'OT', 'OC')
scs15 = scs15[vec]
scs15_OT_aic = bestglm(scs15[,-11], IC='AIC')
print(scs15_OT_aic)<jupyter_output><empty_output><jupyter_text>The output of selecting the Overall Teaching models using AIC: After the model selection procedure using AIC, only Interest In Student Learning, Explain Course Requirements, Clear Learning Goals, Feedback To students, Importance of Subject, Explain Subject Matter, and Show Respect For Students are relevant to the Overall Teaching score.<jupyter_code>scs15_OC_aic = bestglm(scs15[,-10], IC='AIC')
print(scs15_OC_aic)<jupyter_output><empty_output><jupyter_text>The output of selecting the Overall Course models using AIC: After the model selection procedure using AIC, only Workload, Explain Course Requirements, Clear Learning Goals, Feedback To students, Importance of Subject,and Explains Subject Matter are relevant to the Overall Course score.Considering the number of observations in the training dataset, I also applied Leave One Out Cross Validation ([LOOCV](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) to select the models.<jupyter_code>scs15_OT_fm = data.frame(cbind(scs15_OT_lm0$x[,-1], scs15$OT))
SCS15_OT_loocv = bestglm(scs15_OT_fm, IC='LOOCV')
print(SCS15_OT_loocv)<jupyter_output><empty_output><jupyter_text>The output of selecting the Overall Teaching models using LOOCV: After the model selection procedure using LOOCV, only Interest In Student Learning, Explain Course Requirements, Clear Learning Goals, Feedback To Students, Importance of Subject, Explains Subject Matter, and Show Respect For Students are relevant to the Overall Teaching score. Here, the exact same predictors are included after the model selection procedures using both AIC and LOOCV.<jupyter_code>scs15_OC_fm = data.frame(cbind(scs15_OC_lm0$x[,-1], scs15$OC))
SCS15_OC_loocv = bestglm(scs15_OC_fm, IC='LOOCV')
print(SCS15_OC_loocv)<jupyter_output><empty_output><jupyter_text>The output of selecting the Overall Course models using LOOCV: After the model selection procedure using LOOCV, only Workload, Explain Course Requirements, Clear Learning Goals, Feedback To students, Importance of Subject,and Explains Subject Matter are relevant to the Overall Course score. Here, the exact same predictors are included after the model selection procedures using both AIC and LOOCV.If we take a closer look at the model selection outputs above, we can find that AIC and LOOCV give us the same models below.Overall Teaching = -0.619 + 0.174 × Interest In Student Learning + 0.099 × Explain Course Requirements + 0.069 × Clear Learning Goals + 0.149 × Feedback To Students + 0.113 × Importance Of Subject + 0.368 × Explains Subject Matter + 0.159 × Show Respect For StudentsOverall Course = -0.287 + 0.004 × Workload + 0.105 × Explain Course Requirements + 0.247 × Clear Learning Goals + 0.087 × Feedback To Students + 0.391 × Importance Of Subject + 0.208 × Explains Subject Matter<jupyter_code>vec16 = c('RR', 'WL', 'IISL', 'ECR', 'CLG', 'FTS', 'IOS', 'ESM', 'SRFS', 'OT', 'OC')
scs16 = scs16[vec16]
scs16 = na.omit(scs16)
nrow(scs16)
summary(scs16)
OT_lm = lm(OT ~ IISL + ECR + CLG + FTS + IOS + ESM + SRFS, data=scs15, x=T)
OC_lm = lm(OC ~ WL + ECR + CLG + FTS + IOS + ESM, data=scs15, x=T)<jupyter_output><empty_output><jupyter_text>I refit the models using the predictors selected out in the model selection procedure and apply them to the test data set(scs16), which contains all the SCS course scores in 2016.<jupyter_code>scs16_OT = scs16[c('IISL', 'ECR', 'CLG', 'FTS', 'IOS', 'ESM', 'SRFS')]
scs16_OC = scs16[c('WL', 'ECR', 'CLG', 'FTS', 'IOS', 'ESM')]
OT_pre = predict(OT_lm, newdata=scs16_OT, interval='prediction', level=0.95)
print OT_pre[1:5,]<jupyter_output><empty_output><jupyter_text>The Overall Teaching score prediction output for the first five instances in the test dataset with a 95% confidence interval:Take the first instance as an example to state the confidence interval. The best prediction of the Overall Teaching score for the first instance is 4.53. We are 95% confident that its Overall Teaching score is between 4.18 and 4.88.<jupyter_code>OC_pre = predict(OC_lm, newdata=scs16_OC, interval='prediction', level=0.95)
print OC_pre[1:5,]<jupyter_output><empty_output><jupyter_text>The Overall Course score prediction output for the first five instances in the test dataset with a 95% confidence interval: We evaluate the models by calculating the mean of squared errors.<jupyter_code>MSE_OT = mean((OT_pre[,1] - scs16$OT) ^ 2)
print(MSE_OT)<jupyter_output><empty_output><jupyter_text>[1] 0.03353827<jupyter_code>MSE_OC = mean((OC_pre[,1] - scs16$OC) ^ 2)
print(MSE_OC)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10132519356759977, 'fraction_numerical': 0.03901131625967838, 'mean_word_length': 5.188666205943331, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 102, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '82758', 'license': 'MIT', 'n_tokens_mistral': 9323, 'n_tokens_neox': 8382, 'n_words': 2995, 'path': 'Predicting Quality of Teaching and Learning at CMU.ipynb', 'repo_name': 'hexingren/practical-data-science', 'size': '26868'}