<jupyter_start><jupyter_text>1. Frame the Problem1. Define the objective in business terms.  2. How will your solution be used?  3. What are the current solutions/workarounds (if any)?  4. How should you frame this problem (supervised/unsupervised, online/offline, etc.)  5. How should performance be measured?  6. Is the performance measure aligned with the business objective?  7. What would be the minimum performance needed to reach the business objective?  8. What are comparable problems? Can you reuse experience or tools?  9. Is human expertise available?  10. How would you solve the problem manually?  11. List the assumptions you or others have made so far.  12. Verify assumptions if possible.   This is a learning exercise in carrying through the entire process of a Machine Learning question. The plan is to use data from https://www.openml.org/d/1471, this is the data set from https://archive.ics.uci.edu/ml/datasets/EEG+Eye+State.  All data is from one continuous EEG measurement with the Emotiv EEG Neuroheadset. The duration of the measurement was 117 seconds. The eye state was detected via a camera during the EEG measurement and added later manually to the file after analyzing the video frames. '1' indicates the eye-closed and '0' the eye-open state. All values are in chronological order with the first measured value at the top of the data.The features correspond to 14 EEG measurements from the headset, originally labeled AF3, F7, F3, FC5, T7, P, O1, O2, P8, T8, FC6, F4, F8, AF4, in that order.  2. Get the DataNote: automate as much as possible so you can easily get fresh data.  1. List the data you need and how much you need.  2. Find and document where you can get that data.  3. Check how much space it will take.  4. Check legal obligations, and get the authorization if necessary.  5. Get access authorizations.  6. Create a workspace (with enough storage space).  7. Get the data.  8. Convert the data to a format you can easily manipulate (without changing the data itself).  9. Ensure sensitive information is deleted or protected (e.g., anonymized). 10. Check the size and type of data (time series, sample, geographical, etc.).  11. Sample a test set, put it aside, and never look at it (no data snooping!).<jupyter_code># https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_openml.html
from sklearn.datasets import fetch_openml

data = fetch_openml("eeg-eye-state")
print(data)
feature_names = data['feature_names']
X = data['data']
y = data['target']
from scipy.io import arff
data, meta = arff.loadarff("eeg-eye-state.arff")
print(data)
print(meta)
import pandas as pd
data = pd.read_csv("eeg-eye-state.csv")
print(data.head())
print(data.info())
# Split the data into  test/training
# https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
from sklearn.model_selection import train_test_split
train_set, test_set = train_test_split(data, test_size=0.2, random_state=42)
train_set.head()<jupyter_output><empty_output><jupyter_text>3. Explore the DataNote: try to get insights from a field expert for these steps.  1. Create a copy of the data for exploration (sampling it down to a manageable size if necessary).2. Create a Jupyter notebook to keep record of your data exploration.  3. Study each attribute and its characteristics:      - Name      - Type (categorical, int/float, bounded/unbounded, text, structured, etc.)    - % of missing values      - Noisiness and type of noise (stochastic, outliers, rounding errors, etc.)    - Possibly useful for the task?      - Type of distribution (Gaussian, uniform, logarithmic, etc.)4. For supervised learning tasks, identify the target attribute(s).5. Visualize the data.  6. Study the correlations between attributes.  7. Study how you would solve the problem manually.  8. Identify the promising transformations you may want to apply.  9. Identify extra data that would be useful (go back to "Get the Data" on page 502).  10. Document what you have learned.<jupyter_code>train_set_original = train_set
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
train_set.hist()
# from pandas.tools.plotting import scatter_matrix # For older versions of Pandas
from pandas.plotting import scatter_matrix
attributes = ["V10", "V2", "V3"]
scatter_matrix(train_set[attributes], figsize=(12, 8))<jupyter_output><empty_output><jupyter_text>4. Prepare the DataNotes:    - Work on copies of the data (keep the original dataset intact).  - Write functions for all data transformations you apply, for five reasons:      - So you can easily prepare the data the next time you get a fresh dataset      - So you can apply these transformations in future projects      - To clean and prepare the test set      - To clean and prepare new data instances      - To make it easy to treat your preparation choices as hyperparameters  1. Data cleaning:      - Fix or remove outliers (optional).      - Fill in missing values (e.g., with zero, mean, median...) or drop their rows (or columns).  2. Feature selection (optional):      - Drop the attributes that provide no useful information for the task.  3. Feature engineering, where appropriates:      - Discretize continuous features.      - Decompose features (e.g., categorical, date/time, etc.).      - Add promising transformations of features (e.g., log(x), sqrt(x), x^2, etc.).    - Aggregate features into promising new features.  4. Feature scaling: standardize or normalize features.    5. Explore ModelsNotes: - If the data is huge, you may want to sample smaller training sets so you can train many different models in a reasonable time (be aware that this penalizes complex models such as large neural nets or Random Forests).  - Once again, try to automate these steps as much as possible.    1. Train many quick and dirty models from different categories (e.g., linear, naive, Bayes, SVM, Random Forests, neural net, etc.) using standard parameters.  2. Measure and compare their performance.      - For each model, use N-fold cross-validation and compute the mean and standard deviation of their performance. 3. Analyze the most significant variables for each algorithm.  4. Analyze the types of errors the models make.      - What data would a human have used to avoid these errors?  5. Have a quick round of feature selection and engineering.  6. Have one or two more quick iterations of the five previous steps.  7. Short-list the top three to five most promising models, preferring models that make different types of errors.<jupyter_code>y=train_set.Class
print(y)
X = train_set.drop(columns="Class")
print(X)
from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(X, y)
from sklearn.metrics import mean_squared_error

predictions = lin_reg.predict(X)
lin_mse = mean_squared_error(y, predictions)
lin_rmse = np.sqrt(lin_mse)
lin_rmse
from sklearn.metrics import mean_absolute_error

lin_mae = mean_absolute_error(y, predictions)
lin_mae
from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor(random_state=42)
tree_reg.fit(X, y)
predictions = tree_reg.predict(X)
tree_mse = mean_squared_error(y, predictions)
tree_rmse = np.sqrt(tree_mse)
tree_rmse<jupyter_output><empty_output><jupyter_text>6. Fine Tune ModelsNotes:  - You will want to use as much data as possible for this step, especially as you move toward the end of fine-tuning.   - As always automate what you can.    1. Fine-tune the hyperparameters using cross-validation.      - Treat your data transformation choices as hyperparameters, especially when you are not sure about them (e.g., should I replace missing values with zero or the median value? Or just drop the rows?).      - Unless there are very few hyperparamter values to explore, prefer random search over grid search. If training is very long, you may prefer a Bayesian optimization approach (e.g., using a Gaussian process priors, as described by <NAME>, <NAME>, and <NAME> ([https://goo.gl/PEFfGr](https://goo.gl/PEFfGr)))  2. Try Ensemble methods. Combining your best models will often perform better than running them invdividually.  3. Once you are confident about your final model, measure its performance on the test set to estimate the generalization error.> Don't tweak your model after measuring the generalization error: you would just start overfitting the test set.<jupyter_code>from sklearn.model_selection import cross_val_score

scores = cross_val_score(tree_reg, X, y,
                         scoring="neg_mean_squared_error", cv=10)
tree_rmse_scores = np.sqrt(-scores)
def display_scores(scores):
    print("Scores:", scores)
    print("Mean:", scores.mean())
    print("Standard deviation:", scores.std())

display_scores(tree_rmse_scores)
lin_scores = cross_val_score(lin_reg, X, y,
                             scoring="neg_mean_squared_error", cv=10)
lin_rmse_scores = np.sqrt(-lin_scores)
display_scores(lin_rmse_scores)
from sklearn.ensemble import RandomForestRegressor

forest_reg = RandomForestRegressor(n_estimators=10, random_state=42)
forest_reg.fit(X, y)
predictions = forest_reg.predict(X)
forest_mse = mean_squared_error(y, predictions)
forest_rmse = np.sqrt(forest_mse)
forest_rmse
from sklearn.model_selection import cross_val_score

forest_scores = cross_val_score(forest_reg, X, y,
                                scoring="neg_mean_squared_error", cv=10)
forest_rmse_scores = np.sqrt(-forest_scores)
display_scores(forest_rmse_scores)
scores = cross_val_score(lin_reg, X, y, scoring="neg_mean_squared_error", cv=10)
pd.Series(np.sqrt(-scores)).describe()
from sklearn.svm import SVR

svm_reg = SVR(kernel="linear")
svm_reg.fit(X,y)
predictions = svm_reg.predict(X)
svm_mse = mean_squared_error(y, predictions)
svm_rmse = np.sqrt(svm_mse)
svm_rmse
from sklearn.svm import SVC

svmc_reg = SVC()
svmc_reg.fit(X,y)
predictions = svmc_reg.predict(X)
svmc_mse = mean_squared_error(y, predictions)
svmc_rmse = np.sqrt(svmc_mse)
svmc_rmse
from sklearn.neighbors import KNeighborsClassifier

knn_reg = KNeighborsClassifier(n_neighbors=3)
knn_reg.fit(X,y)
predictions = knn_reg.predict(X)
knn_mse = mean_squared_error(y, predictions)
knn_rmse = np.sqrt(knn_mse)
knn_rmse
from sklearn.model_selection import GridSearchCV

param_grid = [
    # try 15 (5×3) combinations of hyperparameters
    {'n_neighbors': [3,4,5,6,7], 'algorithm':["ball_tree", "kd_tree", "brute"]}       
  ]

knn_grid_search = GridSearchCV(knn_reg, param_grid, cv=5,
                           scoring='neg_mean_squared_error', return_train_score=True)
knn_grid_search.fit(X, y)
knn_grid_search.best_params_
knn_grid_search.best_estimator_
cvres = knn_grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
pd.DataFrame(knn_grid_search.cv_results_)
from sklearn.model_selection import GridSearchCV

param_grid = [
    # try 12 (3×4) combinations of hyperparameters
    {'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},
    # then try 6 (2×3) combinations with bootstrap set as False
    {'bootstrap': [False], 'n_estimators': [3, 10], 'max_features': [2, 3, 4]},
  ]

forest_reg = RandomForestRegressor(random_state=42)
# train across 5 folds, that's a total of (12+6)*5=90 rounds of training 
grid_search = GridSearchCV(forest_reg, param_grid, cv=5,
                           scoring='neg_mean_squared_error', return_train_score=True)
grid_search.fit(X, y)
grid_search.best_params_
grid_search.best_estimator_
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
pd.DataFrame(grid_search.cv_results_)
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_distribs = {
        'n_estimators': randint(low=1, high=200),
        'max_features': randint(low=1, high=8),
    }

forest_reg = RandomForestRegressor(random_state=42)
rnd_search = RandomizedSearchCV(forest_reg, param_distributions=param_distribs,
                                n_iter=10, cv=5, scoring='neg_mean_squared_error', random_state=42)
rnd_search.fit(X, y)
cvres = rnd_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
feature_importances = grid_search.best_estimator_.feature_importances_
feature_importances
sorted(zip(feature_importances, list(data.columns) ), reverse=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08508083880262526, 'fraction_numerical': 0.01344645429806307, 'mean_word_length': 4.615730337078651, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 21, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '458684', 'license': 'MIT', 'n_tokens_mistral': 3928, 'n_tokens_neox': 3715, 'n_words': 1375, 'path': 'testing/EndToEnd.ipynb', 'repo_name': 'ptracton/MachineLearning', 'size': '12513'}