<jupyter_start><jupyter_text>Lecture 5 - EEP 118 Spring 2021 This is the notebook for Lecture 5 where we will see how coefficients and R Squared change when the sample size in the data increases when we run a linear least squares regression estimation procedure, like we did in Lecture 5 using R<jupyter_code>#-------------------------------------------
#install needed R packages
#you will learn this in Sections
#you only need to install them once then only call them using library()
#make sure the packages are loaded onto the srever for students to us them
#-------------------------------------------
#for OLS linear regression model install below
install.packages("lmSupport")
#for summary stats install below
install.packages("psych")
#for scatter plots etc install ggplot2
install.packages("ggplot2")
#library(lmSupport)
library(psych)
library(ggplot2)
library(lmSupport)
install.packages("data.table")
library(data.table)
install.packages("dplyr")
library(dplyr)
install.packages("haven")
library(haven)
install.packages("tidyverse")
library(tidyverse)<jupyter_output>also installing the dependencies ‘zip’, ‘RcppArmadillo’, ‘openxlsx’, ‘unmarked’, ‘rio’, ‘nloptr’, ‘AICcmodavg’, ‘car’, ‘lme4’, ‘pbkrtest’

Warning message in install.packages("lmSupport"):
“installation of package ‘zip’ had non-zero exit status”Warning message in install.packages("lmSupport"):
“installation of package ‘RcppArmadillo’ had non-zero exit status”Warning message in install.packages("lmSupport"):
“installation of package ‘nloptr’ had non-zero exit status”Warning message in install.packages("lmSupport"):
“installation of package ‘openxlsx’ had non-zero exit status”Warning message in install.packages("lmSupport"):
“installation of package ‘unmarked’ had non-zero exit status”Warning message in install.packages("lmSupport"):
“installation of package ‘lme4’ had non-zero exit status”Warning message in install.packages("lmSupport"):
“installation of package ‘rio’ had non-zero exit status”Warning message in install.packages("lmSupport"):
“installation of package ‘AICcmodavg’ had non[...]<jupyter_text>Read the data into R<jupyter_code>#-------------------------------------------
#1. Read in data and see the top rows to see column names etc
#-------------------------------------------
my_data <- read.csv("Lecture5.csv")
head(my_data)<jupyter_output><empty_output><jupyter_text>Keep only year 87 , save as data2 dataframe, and redo Lecture 4<jupyter_code>my_data2 <- my_data[my_data$year ==87,] 
head(my_data2)<jupyter_output><empty_output><jupyter_text>Scatter Plot of Crime Rate and Police Per Capita for Year 1987 only N=90<jupyter_code>fig1<-plot(my_data2$polpc,my_data2$crmrte, main="Scatter of Police Per Capita and Crime Rate",
                     xlab="Police Per Capita", ylab="Crime Rate")<jupyter_output><empty_output><jupyter_text>Regression of Crime Rate on Police Per Capita for Year 1987 only N=90<jupyter_code>#regression
regLecture4 <- lm(crmrte ~ polpc,my_data2)
#show output
summary(regLecture4)<jupyter_output><empty_output><jupyter_text>Generate Predicted Crime Rate using b0 and b1 estimates of the regression you estimated<jupyter_code>#predicted crime rate
my_data2$crmrte_hat <- regLecture4$fitted.values<jupyter_output><empty_output><jupyter_text>Plot Crime Rate and Predicted Crime Rate to see how well we are doing<jupyter_code>#plot(my_data2$crmrte[,"Crime Rate"],my_data2$crmrte_hat[,"Predicted Crime Rate"])
fig2<-plot(my_data2$crmrte,my_data2$crmrte_hat, main="Scatter Plot of Crime Rate and Predicted Crime Rate",
                     xlab="Crime Rate", ylab="Predicted Crime Rate")<jupyter_output><empty_output><jupyter_text>Get regression line estimates and police per capita graph<jupyter_code>fig1<-plot(my_data2$polpc,my_data2$crmrte_hat, 
    main="Fitted Values of Crime Rate as a Function of Police Per Capita",
    xlab="Police Per Capita", ylab="Predicted Crime Rate")<jupyter_output><empty_output><jupyter_text>Use now the full sample N=630<jupyter_code>#regression
regLectureN630 <- lm(crmrte ~ polpc,my_data)
#show output
summary(regLectureN630)<jupyter_output><empty_output><jupyter_text>Compare standard errors from reg using N=90 with Standard errors from N=630<jupyter_code>summary(regLecture4)<jupyter_output><empty_output><jupyter_text>Comparing the regression output of N=630 with the one for N=90 you can see, for Police Per Capita's beta_hat, that SE is 2.00318 with N=90 and is 0.259 for N=630. so SE dropped a lot when N increased from 90 to 630. Combine the fitted values crime rate with the crime rate data on a scatterplot with police per capita on the horizontal x axis, for N=630<jupyter_code>#generate fitted values
my_data$crmrte_hat<-regLectureN630$fitted.values
#make combined scatter plot of crime rate data and fitted values of crime rate given regression estimates
scatter_data_fittedVals <- ggplot(data = my_data) + geom_point(aes(x=polpc, y=crmrte, color = "data")) +
  geom_point(aes(x=polpc, y=crmrte_hat, color = "fitted")) + 
  xlab("Police Per Capita") + ylab("Crime Rate") + ggtitle("Police Per Capita and Crime Rate Fitted Values in Blue and Data in Red")

scatter_data_fittedVals<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12398765908214424, 'fraction_numerical': 0.019475510991129964, 'mean_word_length': 5.411619283065513, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '49075', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1714, 'n_tokens_neox': 1610, 'n_words': 511, 'path': 'Spring2021-J/Lectures/Lecture5_Spring2021run.ipynb', 'repo_name': 'sungsy12345/ENVECON-118', 'size': '5186'}