<jupyter_start><jupyter_text>AWS-foryou   examples ---  Example 1Running sklearn_diabetes.py which contains the user's algorithm.<jupyter_code>import boto3
import numpy as np
import os
import pandas as pd
from sklearn import datasets
from sklearn import preprocessing
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split
from sklearn.svm import SVR
import time

os.chdir("..")

from awsforyou import aws_foryou
import examples.sklearn_diabetes as sdb
# writing the csv files for x and y (data_loc and target_loc respectively)
# datasize is 3 times larger than the standard diabetes dataset
x,y = sdb.get_diabetes(3)
# assign the locations of csv files
data_loc = './examples/x_diabetes.csv'
target_loc = './examples/y_diabetes.csv'<jupyter_output><empty_output><jupyter_text>For the purpose of this example, a dummy scripy is created.The script does the following:1. fits linear regression to the diabetes data2. finds the best SVR model hyperparameters through grid search3. Reduces the dimension of the dataset through PCA4. finds the best SVR model hyperparameters again through grid search5. returns the best model and the respective parameters<jupyter_code># running the script without aws_foryou

start = time.time()
best_model = sdb.run_sklearn_diabetes(data_loc, target_loc)
finish = time.time()
runtime = finish - start
print("runtime is %f seconds" % runtime)
print("best model is \n %s" % best_model)<jupyter_output>linear regression score = 0.189382
best hyperparameters estimate from grid search = 
 SVR(C=10, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='auto',
  kernel='rbf', max_iter=-1, shrinking=True, tol=0.001, verbose=False) 
score from using best hyperparameters = 0.217661
begining 6-components PCA decomposition
percentage of variance explained = 0.774004
repeat grid search with PCA-transformed data
best hyperparameters estimate from grid search = 
 SVR(C=10, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='scale',
  kernel='rbf', max_iter=-1, shrinking=True, tol=0.001, verbose=False) 
score from using best hyperparameters = 0.214275
runtime is 220.557909 seconds
best model is 
 SVR(C=10, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='auto',
  kernel='rbf', max_iter=-1, shrinking=True, tol=0.001, verbose=False)<jupyter_text>Our script, aws_foryou, takes in two arguments.  1. The string which is used to call your script. In this case:```python"run_sklearn_diabetes(data_loc='examples/x_diabetes.csv', target_loc='examples/y_diabetes.csv')"```  2. The module the function being called is found in. In this case:```"examples.sklearn_diabetes"```Putting them together, the command used to execute our script is:  ```pythonaws_foryou.aws_foryou("run_sklearn_diabetes(data_loc='examples/x_diabetes.csv', target_loc='examples/y_diabetes.csv')", "examples.sklearn_diabetes")```This will write a html file and return a dataframe. The returned dataframe is assigned to ```df``` in this case<jupyter_code># running the script with aws_foryou

df = aws_foryou.aws_foryou("run_sklearn_diabetes(data_loc='examples/x_diabetes.csv', target_loc='examples/y_diabetes.csv')", "examples.sklearn_diabetes")<jupyter_output>linear regression score = -0.146835
best hyperparameters estimate from grid search = 
 SVR(C=5, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='scale',
  kernel='linear', max_iter=-1, shrinking=True, tol=0.001, verbose=False) 
score from using best hyperparameters = 0.315438
begining 6-components PCA decomposition
percentage of variance explained = 0.830719
repeat grid search with PCA-transformed data
best hyperparameters estimate from grid search = 
 SVR(C=5, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='scale',
  kernel='linear', max_iter=-1, shrinking=True, tol=0.001, verbose=False) 
score from using best hyperparameters = 0.304803
point 1, iteration 1 complete.
linear regression score = -0.146835
best hyperparameters estimate from grid search = 
 SVR(C=5, cache_size=200, coef0=0.0, degree=3, epsilon=0.1, gamma='scale',
  kernel='linear', max_iter=-1, shrinking=True, tol=0.001, verbose=False) 
score from using best hyperparameters = 0.315438
begining 6-components[...]<jupyter_text>Opening the html file lets you browse the results of the prediction.<jupyter_code>from IPython.display import Image
Image("examples/html-table-snip.png")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09920106524633822, 'fraction_numerical': 0.04172214824678207, 'mean_word_length': 6.222756410256411, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '209400', 'license': 'MIT', 'n_tokens_mistral': 1586, 'n_tokens_neox': 1426, 'n_words': 445, 'path': 'examples/examples.ipynb', 'repo_name': 'winnawat/AWS-foryou', 'size': '4506'}