<jupyter_start><jupyter_text>Importing the dataframe<jupyter_code>df = pd.read_csv("Salary_Data.csv")
df.info()
df.head()<jupyter_output><empty_output><jupyter_text>Checking if there are null entries in the data frame<jupyter_code>df.isna().sum()<jupyter_output><empty_output><jupyter_text>Deleting the null records in the data frame since it is insignificant<jupyter_code>df.dropna(inplace = True)
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Analysing the data<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Outliers are detected in years of experience, lets find them<jupyter_code>df[df["YearsExperience"] > 100]<jupyter_output><empty_output><jupyter_text>Removing the outliers<jupyter_code>new_df = df[df["YearsExperience"] < 100]<jupyter_output><empty_output><jupyter_text>Analysing the data again<jupyter_code>new_df.describe()<jupyter_output><empty_output><jupyter_text>Plotting the data for further verification<jupyter_code>plt.scatter(new_df["YearsExperience"], new_df["Salary"])
plt.xlabel("Years of experience")
plt.ylabel("salary")<jupyter_output><empty_output><jupyter_text>There are outliers in Salary too, lets find and remove them.<jupyter_code>df[df["Salary"] > 1.0e+06]
new_df = new_df[new_df["Salary"] < 1.0e+06]<jupyter_output><empty_output><jupyter_text>Analysing it again to see if everything is fine or not.<jupyter_code>new_df.describe()<jupyter_output><empty_output><jupyter_text>Plotting the data<jupyter_code>plt.scatter(new_df["YearsExperience"], new_df["Salary"])
plt.xlabel("Years of experience")
plt.ylabel("salary")<jupyter_output><empty_output><jupyter_text>Got a nice linear set of data points. Can proceed for linear regression now<jupyter_code>x = new_df['YearsExperience'].to_numpy()
y = new_df['Salary'].to_numpy()

model = LinearRegression().fit(x.reshape(-1, 1), y)
m = model.coef_
c = model.intercept_
print('intercept:', c)
print('slope    :', m)
y_pred = model.predict(x.reshape(-1,1))<jupyter_output><empty_output><jupyter_text>Plotting the predicted and the original data.<jupyter_code>plt.scatter(x,y, marker = 'o', color = 'blue')

plt.plot(x,y_pred, color = 'red')
plt.title("Testing Plot")
plt.xlabel("temp (C)")
plt.ylabel("temp (F)")<jupyter_output><empty_output><jupyter_text>Predicting salary for years of experience using the model we created<jupyter_code>exp = 4
approx_salary = m * exp + c
print(approx_salary)<jupyter_output>[63592.04948449]<jupyter_text>Performing error analysis Mean Squared Error(MSE)<jupyter_code>metrics.mean_squared_error(y, y_pred)<jupyter_output><empty_output><jupyter_text>Mean Absolute Error(MAE)<jupyter_code>metrics.mean_absolute_error(y, y_pred)<jupyter_output><empty_output><jupyter_text>Root Mean Squared Error (RMSE)<jupyter_code>math.sqrt(metrics.mean_squared_error(y, y_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13399153737658676, 'fraction_numerical': 0.011283497884344146, 'mean_word_length': 10.920168067226891, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '319388', 'license': 'MIT', 'n_tokens_mistral': 1137, 'n_tokens_neox': 1073, 'n_words': 167, 'path': 'linear regression/salary linear regression/Salary prediction.ipynb', 'repo_name': 'gyan-krishna/exploring-ml-and-dl', 'size': '2836'}