<jupyter_start><jupyter_text>Dragon Real Estate - Price Predictor<jupyter_code>import pandas as pd
housing = pd.read_csv("data.csv")
 housing.head() # Gives the first 5 rows of the data.
housing.info()
housing['CHAS'].value_counts() # Counts no of rows with same value.
housing.describe()
%matplotlib inline 
# %matplotlib inline  was written to show the data here in the notebook
# For plotting histogram
import matplotlib.pyplot as plt
housing.hist(bins=50, figsize=(20, 15))<jupyter_output><empty_output><jupyter_text>Train Test Splitting<jupyter_code># For learning purpose

import numpy as np
def split_train_test(data, test_ratio):
    np.random.seed(42)
    '''
    The problem with the below line of code is that it generates shuffled randomly.
    So at one point of time, it will know all the data which is not what we want .
    So to fix this, "np.random.seed(42)" was written, which fixes the values once the method is executed.
    '''
    shuffled = np.random.permutation(len(data))  # permutation randomly arranges the data's indices
    test_set_size = int(len(data) * test_ratio) # ratio of data to be trained and tested
    test_indices = shuffled[:test_set_size] # 0 to test_set_size of the data indices are stored in test_indices
    train_indices = shuffled[test_set_size:] # test_set_size to complete end of the data indices are stored in train_indices
    '''
    iloc()  function enables us to select a particular cell of the dataset, that is,it helps us select a value
    that belongs to a particular row or column from a set of values of data frame or data set.
    '''
    return data.iloc[train_indices], data.iloc[test_indices]
train_set, test_set = split_train_test(housing, 0.2) 
# train_set and test_set are given by the split_train_set dividing it in the ratio 0.2
print(f"Rows in train set : {len(train_set)}\nRows in test set: {len(test_set)}")
# The abovesplit_train_test is available under sklearn.model_selection module with name train_test_split
from sklearn.model_selection import train_test_split
'''
var housing is the data.
var test_size is the ratio.
random_state is the np.random_seed(random_state) which fixes the data.
'''
train_set, test_set = train_test_split(housing, test_size=0.2, random_state=42)

# But here the problem is that we cannot be sure how the data has been divided.
# For example,
# take the data under CHAS row, what if data was fully utilised in the test_set ,
# then the model will predict wrong output if the it was very important.
# So we would want to divide it equally.
# To do this task we use, stratified sampling.

# To explain this :
# Consider that aleins visit the Earth and they want to know how humans look like.
# Now imagine that they saw only Indians or Americans. 
# Whats next ? 
# Then it's possible that they learn the wrong pattern. Such as they think that the humans have white hairs only which is not 
# true.
# To solve this you would like to take some population from every part of the planet and teach them with that data.

# Similarly for machine learning....

# So you will have to teach it with stratified sampling.
# Thankfully  you have a method in scikitlearn.

from sklearn.model_selection import StratifiedShuffleSplit
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
for train_index, test_index in split.split(housing, housing['CHAS']):
    strat_train_set = housing.loc[train_index]
    strat_test_set = housing.loc[test_index]
strat_test_set['CHAS'].value_counts() # Give me the numbers of the same data under CHAS row in test set.
strat_train_set['CHAS'].value_counts() # Give me the numbers of the same data under CHAS row in train set.
# if you are solving a real problem then you don't need to look for correlations.
housing = strat_train_set.copy() # Now housing is the train set.<jupyter_output><empty_output><jupyter_text>Looking for Correaltions<jupyter_code># Taking out insights
corr_matrix = housing.corr()


# Tells us that if we increase a given value, then how much MEDV will be increased
# Strong positive coorelation means that if you increase a certain value then the given value is increased
# Strong negative coorelation means that if you increase a certain value then the given value is decreased

corr_matrix['MEDV'].sort_values(ascending=False)
# Taking out insights by plotting
from pandas.plotting import scatter_matrix
attributes = ["MEDV", "RM", "ZN", "LSTAT"]
scatter_matrix(housing[attributes], figsize= (12, 8))
# Plot Scatter graph 
# Note : alpha means that places which are dense will be marked dark.
housing.plot(kind="scatter", x="RM", y="MEDV", alpha=0.8)<jupyter_output><empty_output><jupyter_text>Trying Out Attribute Combintations<jupyter_code>housing["TAXRM"] = housing["TAX"] / housing["RM"] # Adding row tax per room
housing.head()
# Taking out insights
corr_matrix = housing.corr()


# Tells us that if we increase a given value, then how much MEDV will be increased
# Strong positive coorelation means that if you increase a certain value then the given value is increased
# Strong negative coorelation means that if you increase a certain value then the given value is decreased

corr_matrix['MEDV'].sort_values(ascending=False)
# Plot Scatter graph 
# Note : alpha means that places which are dense will be marked dark.
housing.plot(kind="scatter", x="TAXRM", y="MEDV", alpha=0.8) # Plotting tax per room with respect to price
housing = strat_train_set.drop("MEDV", axis=1) # Remove the MEDV for training.
housing_labels = strat_train_set["MEDV"].copy() # get train_sets housing cost<jupyter_output><empty_output><jupyter_text>Missing attributes<jupyter_code># To take care of missing attributes, you have three options:
#     1. Get rid of missing data points
#     2. Get rid of whole attribute
#     3. Set the value to some vallue(0, mean or median)
a = housing.dropna(subset=["RM"]) # Option 1
a.shape
# Note that original housing data frame will remain unchanged
housing.drop("RM", axis=1).shape # Option 2
# Note there is no RM Column(14) and also note that original housing data frame will remain unchanged
# Option 3
median = housing["RM"].median()
housing["RM"].fillna(median) # Fill places where nothing is available with median
# Note that original housing data frame will remain unchanged
housing.shape
housing.describe() # Before we started imputing i.e, filling missing attributes
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median")
imputer.fit(housing)
imputer.statistics_ # Show me the values you calculated imputer !
# Note it calcualted median for every column
X = imputer.transform(housing)
housing_tr = pd.DataFrame(X, columns=housing.columns)
housing_tr.describe() # Filled the missing rows<jupyter_output><empty_output><jupyter_text>Scikit-learn Design Primarily three types of objects1. Estimators - It estimates some parameter based on dataset. Eg. imputer :    It has a fit method and transform method:        Fit method : Fits the dataset and calcualtes the internal parameters            2. Transformers - transform method takes input and returns output based on the learnings from fit() :    It also has a convenience method called fit_transform() which fits and transforms.        3. Predictors - LinearRegression model is an example of predictor :    fit() and predict() are two comman functions. It also gives score fuction which will evaluate the predictions.  Feature Scaling Primarily there are two methods of feature scaling :1. Min-maz scaling (Normalization)    (value-min)/(max-min)    Sklearn provides a class MinMaxScaler for this   2. Standardization    (value-mean)/standarddeviation    Sklearn provides a class called Standard Scaler for this    Note : Standardization is more appropriate since when you change data min-max value can be changed which will have a great effect but in Standardization...      Creating a Pipeline In computing, a pipeline, also known as a data pipeline, is a set of data processing elements connected in sereies,where the output of one element is the input of the next one.Basically to write code.<jupyter_code># Note a machine learning algorithm works perfectly when data is at the same scale 
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
my_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy="median")),
    # .....add as many as you want in your pipline.
    ('std_scaler', StandardScaler()),
])

housing_num_tr = my_pipeline.fit_transform(housing)
housing_num_tr.shape<jupyter_output><empty_output><jupyter_text>Selecting a desired model for Dragon Real Estates<jupyter_code>from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
# model = LinearRegression() # Changing the model - rmse : 4. ...
# model = DecisionTreeRegressor() # Changing the model But is overfitted since rmse = 0.0
# But we want the model to learn the trend not the noise
model = RandomForestRegressor()
model.fit(housing_num_tr, housing_labels)
some_data = housing.iloc[:5]
some_labels = housing_labels.iloc[0:5]
prepared_data = my_pipeline.transform(some_data)
model.predict(prepared_data)
list(some_labels)<jupyter_output><empty_output><jupyter_text>Evaluating the model<jupyter_code>from sklearn.metrics import mean_squared_error
housing_predictions = model.predict(housing_num_tr)
mse = mean_squared_error(housing_labels, housing_predictions) # mse = Mean Squared Error
rmse = np.sqrt(mse)
rmse<jupyter_output><empty_output><jupyter_text>Using better evaluation technique - Cross Validation<jupyter_code>from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, housing_num_tr, housing_labels, scoring="neg_mean_squared_error", cv=10)
rmse_scores = np.sqrt(-scores)
rmse_scores # Errors are less as compared to linear Regressor Out 43
def print_scores(scores):
    print("Scores: ", scores)
    print("Mean: ", scores.mean())
    print("Standard deviation: ", scores.std()) # std = standard deviation
print_scores(rmse_scores)<jupyter_output>Scores:  [2.77928402 2.70686428 4.37849759 2.56638927 3.31415083 2.60018854
 4.81379732 3.26577714 3.37844338 3.18135432]
Mean:  3.2984746684245634
Standard deviation:  0.7160442457581861<jupyter_text>Quiz: Convert this notebook into a python file and run the pipeline using Visual Studio Code  Saving the model<jupyter_code>from joblib import dump, load
dump(model, "Dragon.joblib")<jupyter_output><empty_output><jupyter_text>Testing the model on test data<jupyter_code>X_test = strat_test_set.drop("MEDV", axis=1)
Y_test = strat_test_set["MEDV"].copy()
X_test_prepared = my_pipeline.transform(X_test)
final_predictions = model.predict(X_test_prepared)
final_mse= mean_squared_error(Y_test, final_predictions)
final_rmse = np.sqrt(final_mse)
print("Predictions: ", final_predictions)
print("Test data : ", list(Y_test))
final_rmse
prepared_data[0] # Taking a data for usage<jupyter_output><empty_output><jupyter_text>Using the model<jupyter_code>from joblib import dump, load
import numpy as np
model = load("Dragon.joblib")
features = np.array([[-0.43942006,  3.12628155, -1.12165014, -0.27288841, -1.42262747,
       0, 91.31238772,  25.61111401, -15543.0016859 , -0.5778192 ,
       -0.97491834,  0.41164221, -0.86091034]])
model.predict(features)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07986263978163248, 'fraction_numerical': 0.02615127234304834, 'mean_word_length': 5.2303894679100384, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '640090', 'license': 'MIT', 'n_tokens_mistral': 3644, 'n_tokens_neox': 3386, 'n_words': 1267, 'path': 'DragonRealEstate/Dragon Real Estate.ipynb', 'repo_name': 'Abhishek6579/MachineLearningProjects', 'size': '11357'}