<jupyter_start><jupyter_text>Load and split the dataset- Load the train data and using all your knowledge of pandas try to explore the different statistical properties of the dataset.- Separate the features and target and then split the train data into train and validation set.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
# Code starts here

# read the dataset
train_data = pd.read_csv('train.csv')
train_data.head()
train_data.info()
train_data.describe()
# Store independent variable_train
X = train_data.drop('list_price',axis=1)
# Store dependent variable
y = train_data['list_price']
# Spliting the dataset
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=6)<jupyter_output><empty_output><jupyter_text>Data Visualization- All the features including target variable are continuous. - Check out the best plots for plotting between continuous features and try making some inferences from these plots.<jupyter_code># Check if the target variable is normally distributed
plt.hist(y_train, color='blue');
# Since the target variable is right skewed, we will tranform it using logarithmic transformation to normalize it
plt.hist(np.log(y_train), color='green');
# Check if there is a linear relationship between dependent and independent variable

cols = X_train.columns
fig, axes = plt.subplots(nrows = 3 , ncols = 3)

for i in range(3):
    for j in range(3):
        col = cols[i*3 + j]
        axes[i,j].scatter(X_train[col], y_train)
        axes[i,j].set_title(col)

plt.tight_layout()
plt.show()
# Check for no or less multi-collinearity between dependent variables
corr = round(X_train.corr(), 2)
print(corr)<jupyter_output>ages  num_reviews  piece_count  play_star_rating  \
ages               1.00        -0.16        -0.09              0.23   
num_reviews       -0.16         1.00         0.55              0.15   
piece_count       -0.09         0.55         1.00              0.17   
play_star_rating   0.23         0.15         0.17              1.00   
review_difficulty -0.25        -0.20        -0.31             -0.67   
star_rating        0.10         0.16         0.18              0.90   
theme_name         0.48         0.02         0.17              0.15   
val_star_rating    0.12         0.17         0.19              0.92   
country           -0.02         0.01         0.01             -0.00   
Id                 0.01        -0.01        -0.01             -0.01   

                   review_difficulty  star_rating  theme_name  \
ages                           -0.25         0.10        0.48   
num_reviews                    -0.20         0.16        0.02   
piece_count            [...]<jupyter_text>Feature Selection- Try selecting suitable threshold and accordingly drop the columns.<jupyter_code># selecting threshold  for correlation as 0.7
# drop columns from X_train
X_train.drop(['play_star_rating','val_star_rating', 'Id'],axis = 1 ,inplace=True)
X_val.drop(['play_star_rating','val_star_rating', 'Id'],axis = 1 ,inplace=True)<jupyter_output><empty_output><jupyter_text>Model building<jupyter_code># instantiate linear regression model
regressor = LinearRegression()

# fit model on training data
regressor.fit(X_train, y_train)

# predict the result
y_pred = regressor.predict(X_val)<jupyter_output><empty_output><jupyter_text>Residual check!- Check the distribution of the residual.<jupyter_code># RMSE calculation
rmse = np.sqrt(mean_squared_error(y_val, y_pred))
print("Root Mean Squared Error:", rmse)

# R-squared calculation
rsquared = r2_score(y_val, y_pred)
print("R Squared:", round(rsquared,4))

# Residual Check
residual = y_val-y_pred
plt.hist(residual)
plt.xlabel('residual error')
plt.title('Histogram of residual error')
plt.show()<jupyter_output>Root Mean Squared Error: 48.84276343718057
R Squared: 0.7328<jupyter_text>Prediction on the test data and creating the sample submission file.- Load the test data and store the `Id` column in a separate variable.- Perform the same operations on the test data that you have performed on the train data.- Create the submission file as a `csv` file consisting of the `Id` column from the test data and your prediction as the second column.<jupyter_code># read the dataset
test_data = pd.read_csv('test.csv')
test_data.head()
# Store independent variable_train
X_test = test_data

# Store dependent variable
#y_test = test_data['list_price']
ID = X_test['Id']
# drop columns from X_train
X_test.drop(['play_star_rating','val_star_rating', 'Id'],axis = 1 ,inplace=True)
X_test.columns
# predict the result
y_pred_test = regressor.predict(X_test)
y_pred_test = pd.Series(y_pred_test, name='list_price')
submission = pd.DataFrame({'Id':ID, 'list_price':y_pred_test})
submission.set_index('Id',inplace=True)
submission.to_csv('submit.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0903353057199211, 'fraction_numerical': 0.034911242603550295, 'mean_word_length': 3.261344537815126, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '450018', 'license': 'MIT', 'n_tokens_mistral': 1700, 'n_tokens_neox': 1571, 'n_words': 470, 'path': "The-Lego-Collector's-Dilemma/Lego_Dillema_student_template.ipynb", 'repo_name': 'niyati-bhayani/ga-learner-dsmp-repo', 'size': '5070'}