<jupyter_start><jupyter_text>Chemometrics**<NAME> / 2017 / Universidad del Valle****<NAME> / 2017 / ???**An up-to-date version of this notebook can be found here: https://github.com/jwist/chemometrics/<jupyter_code>options(repr.plot.width=4, repr.plot.height=4) # change these setting to plot larger figures<jupyter_output><empty_output><jupyter_text>dependance and correlation and orthogonalityWe should try to explain the difference in results we obtained by scaling the data in this simple example. The main question we want to answer with this example is precisely about the importance of the IVs (temperature, pH, concentration and the intereaction terms). The fact that such a difference, not only in magnitude but also in trends (the ranking of importance is not maintained) call for a more thorough explanation. Consider the following function: $y=x^2$<jupyter_code>x <- runif(200, min = -1, max = 1)
y <- x^2 + rnorm(200, sd = 0.002)
plot(x,y,pch=16)
rug(x,side=1,col="grey")
rug(y,side=2,col="grey")
cor(x,y)
chisq.test(x,y, simulate.p.value = TRUE)
d1 <- hist(y, breaks = 200, right=FALSE, plot = FALSE)
plot(d1)<jupyter_output><empty_output><jupyter_text>and compare it with $y=x$<jupyter_code>x <- runif(200,min=-1,max=1)
y <- x
plot(x,y,pch=16)
rug(x,side=1,col="grey")
rug(y,side=2,col="grey")
cor(x,y)
chisq.test(x,y, simulate.p.value = TRUE)<jupyter_output><empty_output><jupyter_text>Despite in both cases $x$ and $y$ are dependent, in the left case they are completely uncorrelated. So being uncorrelated is not the same as being independent, but independant variables are uncorrelated. Two variables are considered independent when their joint probability distribution is equal to the product of their individual marginal probability distributions: $$P_{X,Y}(x,y)=P_X(x)P_Y(y)$$Two variables are uncorrelated when their correlation coefficient is zero or when their covariance is equal to zero.  correlations in experimental designExperimental design is all about designing an experiment where the IV are uncorrelated so that effect of each IV can be studied separately. Then, by definition we can assume that IV are uncorrelated. You can check this for our chemical data. For example the correlation between pH and temperature is `r cor(datas$ph, datas$temp)`, while the correlation between ph and pH$\times$T is `r cor(datas$ph,datas$phTemp)`, non zero as expected.if we look at the data for example for example for temperature and concentration we might have `r d[2,2:3]` and `r d[3,2:3]` for second and third point. If we represent them as vectors, we can see that both vectors are somehow collinear and that the product of both vectors is even more collinear.<jupyter_code>d <- c(6,60,4,34.841,6,60,2,16.567,6,20,4,45.396,6,20,2,27.939,4,60,4,19.825,4,60,2,1.444,4,20,4,37.673,4,20,2,23.131,6,40,3,23.088,4,40,3,12.325,5,60,3,16.461,5,20,3,33.489,5,40,4,26.189,5,40,2,8.337,5,40,3,19.192,5,40,3,16.579,5,40,3,17.794,5,40,3,16.65,5,40,3,16.799,5,40,3,16.635)
dim(d) <- c(4,20) # we force the data into a mtrix of dimension 4 x 20
# we create the data frame

plot(d[1,],d[2,], ylim=c(0,80), xlim=c(0,120),type='n')
for (i in 2) {
  arrows(x0=0, y0=0, x1=d[2,i], y1=d[2,i+1])
  arrows(x0=0, y0=0, x1=d[3,i], y1=d[3,i+1],col=2)
  arrows(x0=0, y0=0, x1=d[3,i]*d[2,i], y1=d[3,i+1]*d[2,i+1],col=3)
}
abline(0,0)
abline(v=0)<jupyter_output><empty_output><jupyter_text>Now, if we scale / encode these data and draw again their vector we observe that they became orthogonal. So the collinearity has been reduced by proper scaling.<jupyter_code># scale with y <- (2*x - max(x) - min(x)) / (max(x)-min(x))
maxs <- apply(d[1:3,],1,max); mins <- apply(d[1:3,],1,min)
d[1:3,] <- t( scale(t(d)[,1:3],center = (mins + maxs)/2, scale = (maxs - mins)/2) )

plot(d[1,],d[2,], ylim=c(-1,1), xlim=c(-1,1),type='n')
for (i in 2) {
  arrows(x0=0, y0=0, x1=d[2,i], y1=d[2,i+1])
  arrows(x0=0, y0=0, x1=d[3,i], y1=d[3,i+1],col=2)
  arrows(x0=0, y0=0, x1=d[3,i]*d[2,i], y1=d[3,i+1]*d[2,i+1],col=3)
}
abline(0,0)
abline(v=0)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15872624043446062, 'fraction_numerical': 0.08170822019254505, 'mean_word_length': 6.022530329289428, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 23, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '621062', 'license': 'MIT', 'n_tokens_mistral': 1739, 'n_tokens_neox': 1577, 'n_words': 442, 'path': '6_chemometrics_correlationDependance.ipynb', 'repo_name': 'jwist/chemometrics', 'size': '4063'}