<jupyter_start><jupyter_text>Diabetes Linear RegressionThis Jupyter notebook will be used to test my skills on linear regression based on the [diabetes dataset](https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html), readily available in the Sci-kit Learn Python library.To first bring some context to the data, it was collected by <NAME>, <NAME>, <NAME> and <NAME> in 2004 as an analysis of their new 'Least Angle Regression' (LARS) algorithm designed to outcompete previous linear regression methods (such as Lasso regression). The outline of this notebook will be as follows"* Receiving the data from Scikit-Learn* Doing introductory preprocessing and visualization* Testing out various regression methods (LARS, Lasso, Linear, Polynomial, SVR)* Finding their accuracy using RMSE [Root Mean Squared Error] * Use of grid search to increase accuracy and deep learning (Tensorflow) to decrease the loss over time...<jupyter_code>import sys
import numpy as np
import matplotlib.pyplot as plt
from math import ceil
import seaborn as sns
from sklearn.datasets import load_diabetes
from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error
from sklearn.linear_model import LinearRegression, LassoCV, LassoLarsCV, RidgeCV, LassoLars
from sklearn.model_selection import cross_val_score
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_regression


%matplotlib inline<jupyter_output><empty_output><jupyter_text>PreprocessingAccording to the source page, there is a sample size of 442 individuals in the data with 10 features and 1 target value. In the numpy array that will be downloaded, these features are * Age* Sex* BMI* Average Blood Pressure* Six Blood Serum measurements* A quantitative measure of disease progression after the baselineAccording to [Scikit-learn](https://scikit-learn.org/stable/datasets/index.htmldiabetes-dataset), the downloaded data has already been scaled for us with a centered mean and with the standard deviation * 442. Therefore, we don't have to scale the data and can proceed to explore it and use it for modelling!<jupyter_code>db_data = load_diabetes()
print(f"Diabetes Features: {db_data.feature_names} \n")
print(f"Diabetes Data: \n {db_data.data[:5]} \n")
print(f"Diabetes Target: \n {db_data.target[:5]} \n")

# Let's get a sample of the data (80%) so we can visualize it and do some preprocessing on it
# I will be using cross-validation so I won't have access to a proper training set
diabetes_len = len(db_data.data)
X = db_data.data[:int(ceil(diabetes_len*.8))]
y = db_data.target[:int(ceil(diabetes_len*.8))]
np.random.seed(42)
np.random.shuffle(X)
np.random.shuffle(y)

print("Training Data Sample: ")
print(X[:5])
print()


print("Correlation Matrix: ")
corr = np.corrcoef(X, rowvar=False)
print(corr)
sns.heatmap(corr, xticklabels=db_data.feature_names, yticklabels=db_data.feature_names)
plt.xlabel("Diabetes Features")
plt.ylabel("Diabetes Features")
plt.title("Correlation Matrix on Diabetes Dataset")<jupyter_output><empty_output><jupyter_text>According to the heat map above, none of the variabes seem directly correlated together except for the first and second serum measurements. Therefore, it is hard to combine them manually.In an attempt to visualize this data for multivariate linear regression, we need to combine the most correlated parameters via a different means, this will be done via SelectKBest and the linear regression scoring method (1). From this, we can create a regression plot.<jupyter_code># Original Shapes of the Data
print(X.shape, y.shape)

# With the most correlated item of the dataset, match the shapes of the samples
X_st = SelectKBest(f_regression, k=1).fit_transform(X, y)
print(X_st.shape, y.shape)

sns.regplot(X_st, y)
plt.xlabel("Serum measurements")
plt.ylabel("Diabetes Prediction Measurement")
plt.title("Serum measurement on the prediction of diabetes")<jupyter_output>(354, 10) (354,)
(354, 1) (354,)<jupyter_text>With this regression plot as a starting point, we can try to improve upon this score using linear regression algorithms and check whether the slope of the line matches the slope found by the algorithms in the noisy data or too much data was lost (1) in the data visualization and a better slope can be found.<jupyter_code>plt.hist(y, bins=10)
print(np.min(y), np.max(y))

plt.xlabel("Diabetes Measurements")
plt.ylabel("Frequency of Diabetes Measurements")
plt.title("The Frequency of Diabetes Measurements - The Distribution of Effective Treatment")<jupyter_output>25.0 346.0<jupyter_text>Model DevelopmentLet's gather some models to predict the likelihood of diabetes given the serum measurements and most important features. Differences between the Different Regression algorithms: [see here](https://www.analyticsvidhya.com/blog/2017/06/a-comprehensive-guide-for-linear-ridge-and-lasso-regression/). Scikit-learn lists all of the available models that can be fitted to linear regression at this [link](https://scikit-learn.org/stable/modules/linear_model.html).<jupyter_code>alphas = np.logspace(-6, 6, 13) # The alphas are for the regularization of the lines that are unique to lasso and ridge regression.
models = [LinearRegression(n_jobs=-1), RidgeCV(alphas=alphas, cv=10), LassoCV(alphas=alphas, cv=10), LassoLarsCV(n_jobs=-1), LassoLars(alpha=.01)]

report = open("model_report.txt", 'w')

for model in models:
  cross_val_scores = cross_val_score(model, db_data.data, db_data.target, cv=5, scoring="neg_mean_squared_error")
  
  model_name = type(model).__name__
  accuracy = f"Accuracy: {cross_val_scores.mean():.2f} +/- {[cross_val_scores.std()*2]:.2f}"
  coeffs = f"Coefficients: {model_name}: {model.coef_}"
  intercept = f"Intercept: {model_name}: {model.intercept_}"
  
  print(model_name)
  print(model_name, file=report)
  print(accuracy)
  print(accuracy, file=report)
  print(intercept)
  print(intercept, file=report)
  print()
  print("\n", file=report)<jupyter_output><empty_output><jupyter_text>Notes and References:1.   There are errors that can made through attempting to reduce the dimensionality for visualization (Kaski and Peltonen, 2011), which is measured by calculating the precision and recall for the parameters left out of the visualization, so tests may have to be performed multiple times.2.   List item1.   <NAME> and <NAME> (2011). Dimensionality Reduction for Data Visualization. IEEE Signal Processing Magazine 28(2):101-104. doi: https://doi.org/10.1109/MSP.2010.9400032.   List item<jupyter_code># The original shapes
print(X.shape, y.shape)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09057750759878419, 'fraction_numerical': 0.01656534954407295, 'mean_word_length': 5.91281512605042, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '379465', 'license': 'MIT', 'n_tokens_mistral': 2138, 'n_tokens_neox': 1930, 'n_words': 722, 'path': 'Data Science/DiabetesRegression.ipynb', 'repo_name': 'AstroNoodles/Mini-Projects', 'size': '6617'}