<jupyter_start><jupyter_text>Data PreprocessingLab Exercise 2.8Advanced Machine Learning Made EasyFrom Theory to Practice with NumPy and scikit-learnVolume 1: Generalized Linear Models  IntroductionIn this example, we take the number of transistor count in each processor by year of its introduction and check with the simple linear regression model whether actual data support the so-called "Moore's law". We also check for the presence of a possible outlier. **Note**: *The original dataset can be found at https://en.wikipedia.org/wiki/Transistor_count. Dataset was converted to csv file with LibreOffice Calc.*  Table of contents1. [Reading and converting the data](ReadingData)2. [Scatter plot of observations](ScatterPlot)3. [Identifying the outlier](OutlierIdentification)4. [Checking Moore's law](CheckMooreLaw)  1. Reading and converting the data First, import the required libraries.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from scipy import stats
import os
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read the TransistorCount.csv into pandas dataframe and remove some columns not important for our analysis. Extract the input and output values from the pandas dataframe into vector $\mathbf{x}$ and $\mathbf{y}$. Also, extract the number of observations available in the data sets.<jupyter_code>data = pd.read_csv(os.path.join('data','TransistorCount.csv'))
data=data.drop(['Designer', 'Area', 'Process'], axis=1)
print("Transistor count in processors by year:")
print("=======================================\n",data.to_string(index=False))
x_=np.array(data["Date of introduction"])[:,np.newaxis]
y_=np.array(data["Transistor count"])[:,np.newaxis]
N,_=data.shape
print("Number of observations:",N)<jupyter_output>Number of observations: 120<jupyter_text>There are 120 observations in the dataset. Although this might be considered as still a small dataset compared to the big databases available nowadays, the size is enough to be confident in our result as represents more or less the population of the microprocessors from the last five decades.  2. Scatter plot of observations Let's use the scikit-learn library directly and try to fit a simple linear regression on the dataset.<jupyter_code>linregr=LinearRegression()
result=linregr.fit(x_,y_)<jupyter_output><empty_output><jupyter_text>Make the scatter plot of the observations and plot the estimated regression line.<jupyter_code>plt.scatter(x_,y_,s=15)
plt.plot(x_,linregr.predict(x_),'--r')
plt.xlabel('Calendar year',fontsize=14)
plt.ylabel('Transistor count',fontsize=14)
plt.title('Transistor count in processors by year',fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>As can be observed, the linear regression model is not a good fit to our data. We can also calculate the coefficient of determination.<jupyter_code>print("R-squared:",linregr.score(x_,y_))<jupyter_output>R-squared: 0.27087058819019494<jupyter_text>Based on the graph, there should be no surprise that the R-squared value is so low, and only 27% of the variation in the transistor count is explained by the year of introduction. However, we can observe from the scatter plot that the data points are not following a linear trend but rather an exponential trend. What about a scatter plot with a logarithmic scale for the output $y$?<jupyter_code>plt.scatter(x_,np.log(y_),s=15)
plt.xlabel('Calendar year',fontsize=14)
plt.ylabel('Transistor count (log scale)',fontsize=14)
plt.title('Transistor count in processors by year',fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>And indeed, the data points now follow a linear trend. Thus, the first scatter plot of the observations provided a hint about the relation between $x$ and $y$, and after taking the scatter plot with the logarithm of $y$, we find out the linear relation between $x$ and $\log y$. So let's try to fit a simple linear regression after preprocessing the output $y$, i.e. taking the logarithm of $y$.<jupyter_code>ylog_=np.log(y_)
result=linregr.fit(x_,ylog_)<jupyter_output><empty_output><jupyter_text>Make the scatter plot of the scaled data set together with the estimated linear regression line.<jupyter_code>plt.scatter(x_,ylog_,s=15)
plt.plot(x_,linregr.predict(x_),'r')
plt.xlabel('Calendar year',fontsize=14)
plt.ylabel('Transistor count (log scale)',fontsize=14)
plt.title('Transistor count in processors by year',fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Let's calculate the coefficient of determination again:<jupyter_code>print("R-squared:",linregr.score(x_,ylog_))<jupyter_output>R-squared: 0.9583221406461232<jupyter_text>That is quite impressive, almost 96% percent of the variation of logarithm $y$ is explained by the variation in the year of introduction.  3. Identifying the outlier Let's make the histogram plot for the residuals, as well as the probability plot for normal distribution.<jupyter_code>resid_=ylog_-linregr.predict(x_)
fig, ax = plt.subplots(1, 2, figsize=(15, 5))
ax[0].hist(resid_)
ax[0].set_title("Histogram Plot",fontsize=14)
result=stats.probplot(np.ravel(resid_), plot=ax[1])
ax[1].set_title("Normal Probability Plot",fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>It seems that there is an outlier. Looking at the scatter plot of the preprocessed data and the estimated regression line, we suspect that the outlier might be the observation having the maximum negative residual. Let's find the index of the observation with maximum negative residual (the minimum of the residuals) and see which processor corresponds to that.<jupyter_code>idx=np.argmin(resid_)
print(data.iloc[idx])<jupyter_output>Date of introduction         1999
Transistor count           111000
Processor               ARM 9TDMI
Name: 40, dtype: object<jupyter_text>We can check if this outlier is leverage and also an influential point. First, plot the mean of the input.<jupyter_code>print("Mean of x:",x_.mean())<jupyter_output>Mean of x: 2001.5416666666667<jupyter_text>Thus, the outlier does not have high leverage. In fact, the outlier $x$ value almost matches the mean of the input. Thus, the outlier is only in $y$ direction and removing it will only increase the R-squared value, but does not influence the parameters of the linear model.Let's create a new dataset that does not contain the outlier.<jupyter_code>x1_=np.delete(x_,idx,axis=0)
ylog1_=np.delete(ylog_,idx,axis=0)
N1,_=x1_.shape
print("Number of observations in the reduced data set is:",N1)<jupyter_output>Number of observations in the reduced data set is: 119<jupyter_text>Fit the linear regression model on the reduced dataset and make the scatter plot of the reduced dataset together with the new estimated regression line.<jupyter_code>result=linregr.fit(x1_,ylog1_)
plt.scatter(x1_,ylog1_,s=15)
plt.plot(x1_,linregr.predict(x1_),'r')
plt.xlabel('Calendar year',fontsize=14)
plt.ylabel('Transistor count (log scale)',fontsize=14)
plt.title('Transistor count in processors by year',fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Finally, let's calculate the coefficient of determination with the model fitted on the reduced data set (with outlier removed).<jupyter_code>print("R-squared:",linregr.score(x1_,ylog1_))<jupyter_output>R-squared: 0.9674301717880261<jupyter_text>As we expected, the R-squared value slightly increased. Now, almost 97% of the variation in the logarithm of transistor count is explained by the change in the year of introduction. That is very impressive! The remaining 3% of unexplained variation in $\log y$ is due to engineering decisions during processor design. That also means engineers, in general, utilized almost the maximum available number of transistors provided by the actual silicon technology during processor designs.There is a problem though with the above graph. The human mind can hardly interpret the numbers like exp(1),exp(2),exp(3), etc. So, for more readability, we should use logarithm 10 instead.<jupyter_code>ylog_=np.log10(y_)
x1_=np.delete(x_,idx,axis=0)
ylog1_=np.delete(ylog_,idx,axis=0)
N1,_=x1_.shape
result=linregr.fit(x1_,ylog1_)
plt.scatter(x1_,ylog1_,s=15)
plt.plot(x1_,linregr.predict(x1_),'r')
plt.xlabel('Calendar year',fontsize=14)
plt.ylabel('Transistor count ($\log_{10}$ scale)',fontsize=14)
plt.title('Transistor count in processors by year',fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>4. Checking Moore's law Now, that we have a good simple linear regression model fitted on the (reduced) dataset, let's check whether the so-called Moore's law is supported by the actual data from the last five decades. Moore's law is the observation that the number of transistors in a dense integrated circuit (like microprocessor) doubles every two years. The observation is named after <NAME>, the co-founder of Fairchild Semiconductor and Intel, whose 1965 paper described a doubling of the number of transistors per integrated circuit every two years. Knowing that the slope $w$ defines the change in $\log y$ for a unit change in $x$ (e.i. $w=\frac{\Delta \log y}{\Delta x}$), and assuming that the number of transistors is doubled we can use the formula $T=\frac{\log 2}{w}$ to find out the required change in $x$.<jupyter_code>print("Transistor count doubled in every {0:.3f} years".format(np.log(2)/linregr.coef_[0][0]))<jupyter_output>Transistor count doubled in every 4.604 years<jupyter_text>And indeed, Moore's law is supported by the actual data based on the number of transistors in processors in the last five decades. Although we are aware of the previous example, that predicting outside of the observed data points has great uncertainty, let's see what would be the transistor count in processors by 2030.<jupyter_code>print("Transistor count in 2030 will be {0:.3f}".format(np.exp(linregr.predict(np.array([2030]).reshape(-1, 1)))[0,0]))<jupyter_output>Transistor count in 2030 will be 153012.813
{'fraction_non_alphanumeric': 0.09451493655709861, 'fraction_numerical': 0.024378059746228396, 'mean_word_length': 6.420311341734618, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '472280', 'license': 'MIT', 'n_tokens_mistral': 3219, 'n_tokens_neox': 2885, 'n_words': 1123, 'path': 'Vol1-LabExercises/Chapter2/Ch2-Ex8-DataPreprocessing.ipynb', 'repo_name': 'FerencFarkasPhD/Advanced-Machine-Learning', 'size': '10015'}