<jupyter_start><jupyter_text>Dragon Real State - Price Predictor<jupyter_code>import pandas as pd
housing = pd.read_csv("data.csv")
housing.head()
housing.info()
housing['CHAS'].value_counts()
housing.describe()
%matplotlib inline
# for plotting histogram

# import matplotlib.pyplot as plt
# housing.hist(bins = 50, figsize = (20, 15))<jupyter_output><empty_output><jupyter_text>Train-Test Splitting<jupyter_code>#for learning purpose

# import numpy as np
# def split_train_test(data, test_ratio):
#     np.random.seed(42) #to fix shuffled value fix, train test fix
#     shuffled = np.random.permutation(len(data))
#     test_set_size = int(len(data) * test_ratio)
#     test_indices = shuffled[:test_set_size]
#     train_indices = shuffled[test_set_size:]
#     return data.iloc[train_indices], data.iloc[test_indices]
#train_set, test_set = split_train_test(housing, 0.2)

#print(len(train_set), len(test_set))
from sklearn.model_selection import train_test_split
train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)

print(len(train_set), len(test_set))<jupyter_output>404 102<jupyter_text>Stratified Shuffling<jupyter_code>from sklearn.model_selection import StratifiedShuffleSplit
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing['CHAS']):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]
strat_test_set['CHAS'].value_counts()
strat_train_set['CHAS'].value_counts()
# 95/7
# 376/28
housing = strat_train_set
housing.describe()<jupyter_output><empty_output><jupyter_text>Looking for Correlations<jupyter_code>corr_matrix = housing.corr()
#corr_matrix
corr_matrix['MEDV'].sort_values(ascending=False)
# from pandas.plotting import scatter_matrix
# attributes = ["MEDV", "RM", "ZN", "LSTAT"]
# scatter_matrix(housing[attributes], figsize=(12, 8))
housing.plot(kind="scatter", x="RM", y="MEDV", alpha=0.8)<jupyter_output><empty_output><jupyter_text>Trying Out Attribute Combinations<jupyter_code>#housing["TAXRM"] = housing["TAX"]/housing["RM"]
#housing["TAXRM"]
# corr_matrix = housing.corr()
# corr_matrix['MEDV'].sort_values(ascending=False)
#housing.plot(kind="scatter", x="TAXRM", y="MEDV", alpha=0.8)
housing = strat_train_set.drop("MEDV", axis = 1)
housing_labels = strat_train_set["MEDV"]
housing.shape<jupyter_output><empty_output><jupyter_text>Missing Attributes<jupyter_code># To take care of missing attributes, you have 3 options:
#     1. Get rid of the the missing data points
#     2. Get rid of the whole attribute
#     3. Set the value to some value(0, mean or median)
a = housing.dropna(subset=["RM"]) #option1 *inplace= True
a.shape
# Note that the original housing dataframe will remain unchanged
housing.drop("RM", axis=1) #option2
# Note that there is no RM column
# Note that the original housing dataframe will remain unchanged
median = housing["RM"].median() # for option 3
median
housing["RM"].fillna(median) #option3
# Note that the original housing dataframe will remain unchanged
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy = "median")
imputer.fit(housing)
imputer.statistics_
X = imputer.transform(housing)
housing_tr = pd.DataFrame(X, columns=housing.columns)
housing_tr.describe()<jupyter_output><empty_output><jupyter_text>Scikit-learn Design Primarily, three types of objects1. Estimators - It estimates some parameter based on a dataset. Eg. imputer. It has a fit method and transform method. Fit method - Fits the dataset and calculates internal parameters2. Transformers - transform method takes input and returns output based on the learnings from fit(). It also has a convenience function called fit transform() which fits and then transforms.3. Predictors - LinearRegression model is an example of predictor. fit() and predict() are two common functions. It also gives score() function which will evaluate the predictions.    Feature Scaling Primarily, two types of feature scaling methods:     1. Min-max scaling (Normalization) (value min)/(max - min)    Sklearn provides a class called MinMaxScaler for this        2. Standardization    (value - mean)/std     Sklearn provides a class called StandardScaler for this  Creating a Pipeline<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
my_pipeline = Pipeline([
    ('imputer',SimpleImputer(strategy = "median")),
    #     .......... add as many as you want in your pipeline
    ('std_scaler', StandardScaler()),
])
housing_num_tr = my_pipeline.fit_transform(housing) # its a numpy array because predictors take input numpy array in sklearn
housing_num_tr.shape<jupyter_output><empty_output><jupyter_text>Selecting a desired model for Dragon Real Estates<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
#model = LinearRegression()
#model = DecisionTreeRegressor()
model = RandomForestRegressor()
model.fit(housing_num_tr, housing_labels)
some_data = housing.iloc[:5]
some_labels = housing_labels[:5]
prepared_data = my_pipeline.transform(some_data)
model.predict(prepared_data)
list(some_labels)<jupyter_output><empty_output><jupyter_text>Evaluating the model<jupyter_code>import numpy as np
from sklearn.metrics import mean_squared_error
housing_predictions = model.predict(housing_num_tr)
mse = mean_squared_error(housing_labels, housing_predictions)
rmse = np.sqrt(mse)
rmse<jupyter_output><empty_output><jupyter_text>Using better evaluation technique - Cross Validation<jupyter_code># 1 2 3 4 5 6 7 8 9 10
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, housing_num_tr, housing_labels, scoring="neg_mean_squared_error", cv = 10)
rmse_scores = np.sqrt(-scores)
rmse_scores
def print_scores(scores):
    print("Scores :", scores)
    print("Mean :", scores.mean())
    print("Standard deviation :", scores.std())
print_scores(rmse_scores)<jupyter_output>Scores : [2.88935176 2.75370448 4.4782957  2.45372192 3.46208722 2.6381126
 5.12356657 3.27489044 3.10538443 3.41248475]
Mean : 3.359159989086032
Standard deviation : 0.7985962581406212<jupyter_text>Quiz: Convert this notebook into a python file and run the pipeline using Visual Studio Code  Saving the model<jupyter_code>from joblib import dump, load
dump(model, 'Dragon.joblib')<jupyter_output><empty_output><jupyter_text>Testing the model on test data<jupyter_code>X_test = strat_test_set.drop("MEDV", axis = 1)
Y_test = strat_test_set["MEDV"]
X_test_prepared = my_pipeline.transform(X_test)
final_predictions = model.predict(X_test_prepared)
final_mse = mean_squared_error(Y_test, final_predictions)
final_rmse = np.sqrt(final_mse)
#print(final_predictions, list(Y_test))
final_rmse
prepared_data[0]<jupyter_output><empty_output><jupyter_text>Using the model<jupyter_code>from joblib import dump, load
import numpy as np
model = load('Dragon.joblib')
features = np.array([[-0.43942006,  3.12628155, -1.12165014, -0.27288841, -1.42262747,
       -0.24050736, -19.31238772,  2.61111401, -6.0016859 , -0.5778192 ,
       -0.97491834,  5.41164221, -12.86091034]])
model.predict(features)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10633608815426997, 'fraction_numerical': 0.04283746556473829, 'mean_word_length': 6.579331941544885, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '245105', 'license': 'MIT', 'n_tokens_mistral': 2678, 'n_tokens_neox': 2429, 'n_words': 587, 'path': 'Dragon_estates.ipynb', 'repo_name': 'manas149/Housing-Price-Prediction', 'size': '7260'}