<jupyter_start><jupyter_text>Hands-on: Linear Regression with AWS MLInput Features: xOutput/Target Feature: y_noisyObjective: Simple exercise to get hands-on experience with AWSML.AWS ML Regression model to predict value of y_noisy for a given x<jupyter_code>def straight_line(x):
    return 5 * x + 8
straight_line(25)
straight_line(1.254)
np.random.seed(5)
samples = 150
x_vals = pd.Series(np.random.rand(samples) * 20)
y_vals = x_vals.map(straight_line)
# Add random noise
y_noisy_vals = y_vals + np.random.randn(samples) * 3
df = pd.DataFrame({'x': x_vals,
                   'y':y_vals, 
                   'y_noisy': y_noisy_vals})
df.head()
# Correlation will indicate how strongly features are related to the output
df.corr()
fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df.x,
            y = df.y,
            label = 'ideal fit')
plt.scatter(x = df.x,
            y = df.y_noisy, 
            color = 'r',
            marker = '+',
            label = 'Target')
plt.grid(True)
plt.xlabel('Input Feature')
plt.ylabel('Target')
plt.legend()<jupyter_output><empty_output><jupyter_text>Training and Evaluation SetExport Row Number, 'x' and 'y_noisy' columns to a CSV fileRow number will not be used for training the model. It is simply used as unique identifier<jupyter_code>data_path = '..\Data\RegressionExamples\straight_line'
df.to_csv(os.path.join(data_path, 'straight_line_example_all.csv'),
          index = True,
          index_label = 'Row')
# 130 rows for Training + Eval Set
df[df.index < 130].to_csv(os.path.join(data_path,'straight_line_noisy_example_train.csv'),
                          index = True,
                          index_label = 'Row'
                          ,columns = ['x','y_noisy'])<jupyter_output><empty_output><jupyter_text>Test SetTest set should contain the same features expect for the target.Export Row Number, 'x'.<jupyter_code># run all the samples for prediction
df.to_csv(os.path.join(data_path, 'straight_line_example_test_all.csv'),
    index = True,
    index_label = 'Row', 
    columns = 'x')<jupyter_output><empty_output><jupyter_text>Read the target predicted by AWS ML df_predicted = pd.read_csv(os.path.join(data_path,                                        'output_straight_line_noisy',                                        'bp-oDvVSUKSpPe-straight_line_example_test_all.csv'))<jupyter_code>df_predicted = pd.read_csv('./output/bp-oDvVSUKSpPe-straight_line_example_test_all.csv.gz')<jupyter_output><empty_output><jupyter_text>AWS ML Estimated/PredictedEstimated/Predicted value is reported as 'score'. Tag is the row identifier<jupyter_code>df_predicted.head()
df_predicted.columns = ["Row", "y_predicted"]
df_predicted.index = df_predicted.Row
df_predicted.head()<jupyter_output><empty_output><jupyter_text>Plot actual value and predicted value<jupyter_code>fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df.x,
            y = df.y_noisy,
            color = 'r',
            label = 'actual',)
plt.scatter(x = df.x,
            y = df_predicted.y_predicted,
            color = 'b',
            label = 'predicted')
plt.grid(True)
plt.legend()
# Training Data Residuals
residuals = (df_predicted.y_predicted - df.y_noisy)

fig = plt.figure(figsize = (12, 8))
plt.hist(residuals)
plt.grid(True)
plt.xlabel('(Predicted - Actual)')
plt.ylabel('Count')
plt.title('Residuals Distribution')
plt.axvline(color = 'g')
# left of 0 = prediction < actual
# right of 0 = prediction > actual
fig = plt.figure(figsize = (12, 8))
plt.boxplot([df.y_noisy, df_predicted.y_predicted], 
            labels = ['actual','predicted'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('Target')
plt.grid(True)<jupyter_output><empty_output><jupyter_text>Predicted value has lot more noiseDefault AWS ML setting transformed x from numeric to bins.In this case, binning caused more jitter.Let's rerun the model by updating recipe to treat x as numericModel with x binned: Training rmse=14.5376, Evaluation rmse=13.011,  Baseline rmse=30.404Model with x as numeric: Training rmse=4.4563, Evaluation rmse=2.4838, Baseline rmse=30.404<jupyter_code>df_predicted_numeric = pd.read_csv('./output-numeric/bp-VNV4qb98Jmd-straight_line_example_test_all.csv.gz')
df_predicted_numeric.columns = ["Row", "y_predicted"]
df_predicted_numeric.head()
fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df.x,
            y = df.y_noisy,
            color = 'r',
            label = 'actual',)
plt.scatter(x = df.x,
            y = df_predicted.y_predicted,
            color = 'k',
            label = 'predicted bin')
plt.scatter(x = df.x,
            y = df_predicted_numeric.y_predicted,
            color = 'b',
            label = 'predicted num')
plt.legend()
fig = plt.figure(figsize = (12, 8))
plt.boxplot([df.y_noisy, df_predicted.y_predicted, df_predicted_numeric.y_predicted], 
            labels = ['actual','predicted-bin','predicted-numeric'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('Target')
plt.grid(True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12367632367632368, 'fraction_numerical': 0.014985014985014986, 'mean_word_length': 3.5058505850585058, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '370678', 'license': 'Apache-2.0', 'n_tokens_mistral': 1812, 'n_tokens_neox': 1710, 'n_words': 373, 'path': '17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/02 - ml_linear_examples.ipynb', 'repo_name': 'maraboinavamshi/courses', 'size': '5005'}