<jupyter_start><jupyter_text>Welcome Hello and welcome to Proportunity machine learning challenge.What follows is a series of questions about machine learning theory intertwined with programming tasks. The first one puts more focus on your Python skills, the second allows us to see how you interact with data and tackle an optmisation problem.Your results will not be scored against predefined answers. Rather, we are hoping to learn how you approach a problem, what tools you use and how you explain your ideas (through code and words). Use comments where appropriate, but prefer coding in a way that is self-explanatory - your code will be evaluated by someone who is proficient in Python and core ML libraries such as pandas, numpy, sklearn or matplotlib.Once done, clean up the code, restart the kernel, run all of the cells sequentially, save the result and send us the notebook.Best of luck!  Part 1: Hyperparameter optimisation*(<90 minutes)*Model evaluation and selection is essential for construction of reliable systems. The usual computation follows this structure:1. Load data2. Configure model hyperparameters3. Train the model (slow process)4. Evaluate the model on some predefined metrics5. Repeat the above and pick the best modelAn abstract example of this is the `run` function in the code below. The purpose of the function is to find the set of hyperparameters that leads to the highest score. Task 1Complete function`evaluate_all_scores`. It should compute a weighted average of particular scoring functions with weights defined in `score_params` dictionary.*Comments (if any):* Task 2Write `optimize_model` to find the optimal model (highest score) given the search space. Your algorithmshould try every possible combination of parameters $a$, $b$ and $c$ and find the optimal one.Would your solution still work if a new parameter $d$ was added to the search space?Run the code cells below after completing all `TODO` sections.*Comments (if any):*<jupyter_code># Preferably, do not modify code in this cell

class Data:
    """Placeholder class only. Imagine this substituted with a DataFrame or another structure."""
    pass


def evaluate_score1(data: Data, model_hparams: dict):
    d = model_hparams['a'] - model_hparams['b']
    return d**2


def evaluate_score2(data: Data, model_hparams: dict):
    return (model_hparams['a']**2 + model_hparams['b']**2 + model_hparams['c']**2)/3.0


def run():
    data = Data()

    example = {
        'a': 1,
        'b': 2,
        'c': 3
    }
    print('score 1 = ', evaluate_score1(data, example))
    print('score 2 = ', evaluate_score2(data, example))
    

    # TASK 1
    score_params = {
        1: 0.5,  # Weight of evaluate_score1
        2: 0.5   # Weight of evaluate_score2
    }
    print('total_score = ', evaluate_all_scores(data, example, score_params, [evaluate_score1, evaluate_score2]))

    # TASK 2
    search_space = {
        'a': list(range(100)),
        'b': [-1, 0, 1, 10] + list(range(11, 30)),
        'c': [-10, -5, 0, 5, 10],
    }
    
    optimal_model_hparams, optimal_score = optimize_model(data, search_space, score_params, [evaluate_score1, evaluate_score2])
    print('optimal_score = ', optimal_score)
    print('optimal_model_hparams = ', optimal_model_hparams)    
    
import itertools
#Both functions work for any number of params

def evaluate_all_scores(data: Data, model_hparams: dict, score_params: dict, score_functions: list):
  """ Given score_params return total score. """ 
  weighted_evals = []
  for param, func in zip(score_params.values(), score_functions):
    weighted_evals.append(func(Data, model_hparams) * param)

  return sum(weighted_evals) / len(weighted_evals)



def optimize_model(data: Data, search_space: dict, score_params: dict, score_functions: list):
    """ Explore all settings specified in search_space and return best settings and score """
    
    best_score = 0
    best_hparams = {}

    parameters = search_space.values()
    keys = search_space.keys()
    h_params = {}

    for params in list(itertools.product(*parameters)):
      params_list = list(params)
      h_params = dict(zip(keys, params_list))
      score = evaluate_all_scores(Data, h_params, score_params, score_functions)
      if score > best_score:
        best_score = score
        best_hparams = h_params

    return best_hparams, best_score
run()<jupyter_output>score 1 =  1
score 2 =  4.666666666666667
total_score =  1.4166666666666667
optimal_score =  3325.1666666666665
optimal_model_hparams =  {'a': 99, 'b': -1, 'c': -10}<jupyter_text>Task 3Answer in text below.**Question 1- How would I parallelise this algorithm** \\In short, using pyspark. I would set up a grid search algorithm, either the one above or in a real usecase using a SKlearn pipeline, read in the data using a dask dataframe and parallelise the algorithm aross multiple nodes, say 20, using the dask API.**Q2: What is the name for the hyperparameter optimisation approach you implemented? Do you know of any alternatives? What are the tradeoffs?** \\The algorithm I implimented was a simple grid search- it iterates over all of the discrete values defined in the grid, calculates the metrics for each value combination, and finds the value for the best performing instance and what values create this optimal score.  An alternative would be a multi armed bandit algorithm for maximising values, in which instead of iterating over the entire search space, bayesian methods (or conventional methods) are to reach to optimal point in a compelx search space in far fewer iterations by trying distributions with better score distributions more often. Such an algorithm is implimented in ARM's Mango libary, it performs roughly 7000x faster than grid search for optimisation, but due to the bayesian nature there is a slim chance it may accidently disguard possible values that would be **more** optimal**Q3: What is the danger of optimizing too much? How does one alleviate this danger?** \\One issue would potentially be that an algorithm that is perfectly optimisated to a dataset becomes overfitted: if over time the data starts to exibit properties of being non stationary (skew or drift), or even if new data with other distributions are inferred upon (such as if your training data isn't representative of all real world cases lacking generality), an over optimised model would perform suddenly far worse than on its training data. This can be alleviated by using cross validation evaluation, model regularisation, early stopping and more training data.**Q4: With respect to the previous question. Is this also a problem for time series data? (e.g. prices of properties over time) Are there any additional dangers in this domain?** Yes, as mentioned above, with time series data there are some additional risks of overfitting (skew, drift) along with some others. In time series data, traditional test train split's can't be used without "temporal leaks", meaning that often time series models are backtested on the same data that they are trained on, which unsuprisingly would lead to overfitting. To avoid this, models are evaluated using nested cross validation, evaluated using information criteria, or by using bayesian methods. **Q5: Q5: Name a few machine learning algorithms that are prone to overfitting and a few that are not. What drives the difference between them? Are there differences in how one would cope with overfitting in those models?** \\Models which have larger numbers of parameters are more prone to overfitting than their counterparts with less. This means that neural networks, say for a classifier, are more prone to overfitting on data than a much more simplistic model such as a SVM. The main cause of this is that overfitting in terms of inferential statistics is when one attempts to estimate too many parameters from a sample that is too small, and thus larger complexity models (or conversly smaller sample sizes) prevent adequately approximating the true model for your response variable.Some approaches to handling overfitting may vairy- in neural networks for regression for example, to avoid overfitting dropout layers may be used, while in smaller more simplistic models lasso and ridge regression could be more useful for regularisation of models. Ultimately, the complexity of neural networks means that they are best used for cases of large amounts of training data.**Q6: Give an example of an interesting research paper from recent years. Briefly, describe the domain and the problem. What is the contribution? Would you have an idea on how improve to the method further? Or apply it in a specific setting?**\\An interesting paper that recently came out in my opinion was a paper done into the time series forcasting of chaotic systems. The general concept was that in the forcasting in long time spans, or for highly stochastic processes, deterministic components become increasingly more difficult to forcast due to the propergation of randomness in the time series. The paper worked on methods of dimentionality redux and denoising of stochasticity of these chaotic systems to improve the accuracy of forcasting tools like LTSM. The paper used PCA for dimentionality redux, and wavelet transforms for denoising.  I got in contact with the writer and suggested that they use partial fourier transforms in place of the wavelet transform, as it seemed that temporal gaps created by WT allowed the model to "cheat" as its result were not reproducable, and also using variational autoencoders for dimentionality redux to preserve possible non liniarities in the input processes as PCA forces orthognality of the resultant features in the reduced vector space. An interesting case of chaotic data for this could be order book data for cryptocurrency.  Part 2: Predicting property price IntroIn this challenge we ask you to explore a dataset, build a price predictive model and argue how further accuracy gain can be enabled.This is significantly more free-form than the previous set of tasks, so we will let the code do most the talking. We offer some helper questions, but primarily proceed as you would with the goal building an accurate model that joins some ensemble. Provide explanations and key findings through text and plots. In case you are short on time: We prefer a simpler but complete pipeline, in contrast to sophisticated preprocessing and modelling but lacking evaluation. Ideally, start with a simple pipeline and add more complexity in subsequent iterations (e.g. don't worry about overfitting on first iteration). ToolsFeel free to use whatever Python libraries suit you best. We suggest you install pandas (for loading the data)   and plotly (for displaying variables on the map) - two functions you may find helpful are defined below.<jupyter_code>import pandas as pd
import plotly.express as px


def get_data(path: str):
    """ Loads a specified .csv into a DataFrame. """
    trans = pd.read_csv(
        path,
        #index_col="transaction_id",
        parse_dates=["transfer_date", "previous_transfer_date"],
    )
    return trans


def plot_df_map(df, col=None):
    """ Display selected column from a dataframe on a map. Requires plotly. 
    
    >>> plot_df_map(trans, "price")
    
    """
    assert len(df) < 100000
    
    fig = px.scatter_mapbox(
        data_frame=df,
        lat="latitude", lon="longitude",
        color=col,
        opacity=0.99,
    )

    fig.update_layout(mapbox_style="carto-positron",
                      mapbox_zoom=10, mapbox_center={"lat": df["latitude"].mean(), "lon": df["longitude"].mean()})
    fig.update_layout(margin={"r": 20, "t": 0, "l": 0, "b": 0})
    fig.update_layout(legend_orientation="h")
    return fig<jupyter_output><empty_output><jupyter_text>DataThe training dataset contains information about retail property transactions in a particular London location in years 2005 to 2015. The test dataset contains the same columns except for the price, all the transactions there are from 2015 to 2016. Proportunity builds price models in order to understand which neighbourhoods or property types are likely to be a good investment. The following variables are available:| **Variable**              | **Description**                                                     	||------------------------	|-----------------------------------------------------------------	|| price                  	| price in GBP at which property was sold                         	|| transaction_date         	| date of the transaction                                         	|| property_type          	| building type (D)etached, (F)lat, (S)emi-detached or (T)erraced 	|| is_new_property         	| describes whether the property was new at the purchase time     	|| floor_area            	| floor area in metres squared (contains noise and NaNs)          	|| floor_number            	| floor at which the property is located                          	|| rooms_count           	| number of rooms                                                 	|| construction_year_band  	| when was the building constructed (discrete bands)              	|| latitude               	| latitude in degrees                                             	|| longitude              	| longitude in degrees                                            	|| previous_price         	| price in GBP at which property was sold previously              	|| previous_transaction_date	| date of the previous transaction                                	|Data may contain outliers and erroneous entries. We ask you to predict price based on the remaining variables. Feel free to transform them or generate new features.**Question 1:** Which variables are most informative about the price? Make a guess ahead of exploration. Then provide a measure of informativeness and ranking of variables while building the model (may be heuristic). If you could use only $k$ variables, which would you use? Do some of the variables convey similar information?*point to relevant code cells**Comments (if any):*  EvaluationAt Proportunity we train models on past data but make predictions about property prices today and in the future. Build your training and evalation with this in mind. You can choose a performance metric for your system - do justify your choice. It is absolutely acceptable for your model to not be able to make a prediction for every property. If that is the case, report also the hitrate of your model (and output a NaN).**Question 2:** What performance metric did you pick and why? Does your model exhibit bias? Is it particularly good or bad on some sub-populations of the dataset? *point to relevant code cells**Comments (if any):*  ResultsMake predictions for the properties in the test file. Return only your prediction column, indexed by `transaction_id`. Send it to us along with the notebook.**Question 3:** What performance do you expect on transactions from 2015-2016?*Comments (if any):*  **Question 1** \\ I would imagine that the most informative variable would be the floor area for the housing. The measure I plan on using to measure the informativeness is the importance of the models to a tree algorithm for its regression- ultimately the most important features will be those with the highest contribution to predictions of pricing. This will allow for ranking of variables based on contribution, allowing for the use of the kth top. Additionally, I did a pairwise correlation of the numeric variables, but since this was done pre encoding and to avoid the curse of dimentionality, this was only used to rank the numeric variables.**Question 2**\\Since this amounts to a regression problem, I chose to use RMSE to evaluate the performance of the model as it measured the 'loss' from the predicted regression line. I also provided the variance for the regression of the neural network to get an idea for how far away most points are from the predicted regression line \\The model does seem to exibit bias towards numeric features, the XGboost ranking seems to almost completely ignore the contributions of the categorical data. I believe that it performs far worse on houses are higher in price as the data is not evenly distributed by price, meaning that due to the lower number of data points in the high price range, the model overfits in that reigeme, causing far larger error.**Question 3** \\ Since I split the training data into a subsection of test and train data so that I could generate metrics, I can say that the test data is likely to have the same scores as the training data shown later on, assuming no bias was added in the splitting of the data.<jupyter_code># load the data
train = get_data("/content/ml-challenge-train.csv")
test  = get_data("/content/ml-challenge-test.csv")
import matplotlib.pyplot as plt
import numpy as np 
import seaborn as sns 
import tensorflow as tf
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.layers import Dropout

from keras.callbacks import ModelCheckpoint
from keras.models import Sequential
from keras.layers import Dense, Activation, Flatten

import xgboost as xg
from xgboost import plot_importance

from sklearn import metrics
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV

from sklearn.metrics import mean_squared_error as MSE, explained_variance_score
#Test train split
X_train, X_test, y_train, y_test = train_test_split(train.drop(['price'], axis=1), train['price'], test_size=0.20, random_state=1, shuffle=True)
#Find the number of null values by column
print(X_train.isnull().sum(), X_test.isnull().sum())
def pairwise_function(important_var):
  train.dropna()
  correlations = train.corrwith(train[important_var]).iloc[:-1].to_frame()
  correlations['absolute'] = correlations[0].abs()
  sorted_correlations = correlations.sort_values('absolute', ascending=False)[0]
  fig, ax = plt.subplots(figsize=(5,10))
  return sns.heatmap(sorted_correlations.to_frame(), cmap='coolwarm', annot=True, vmin=-1, vmax=1, ax=ax);

#Find correlations to price 
pairwise_function("price")
#Get idea of distributions of numerical variables
train.describe().transpose()
sns.countplot(train['number_habitable_rooms'])
sns.countplot(train['number_habitable_rooms'])

#Outlier removal for num habitable rooms 
indexNames = train[ train['number_habitable_rooms'] >8 ].index
train.drop(indexNames , inplace=True)

indexNames_test = test[ test['number_habitable_rooms'] >8 ].index
test.drop(indexNames_test , inplace=True)
plt.figure(figsize=(12,8))
#sns.scatterplot(x='price',y='total_floor_area',data=train)

sns.distplot(train['total_floor_area'])

#Outlier removal for floor_area, houses must have floor area so these are incorrect 

#Drop zero vals
train =train[train['total_floor_area'] !=0]
test =test[test['total_floor_area'] !=0]


indexNames_floor_large = train[ train['total_floor_area'] >400 ].index
train.drop(indexNames_floor_large , inplace=True)


#Remove outliers larger than 400 
indexNames_floor_test = test[ test['total_floor_area'] >400 ].index
test.drop(indexNames_floor_test , inplace=True)

sns.distplot(train['total_floor_area'])
sns.boxplot(x= 'number_habitable_rooms',y='price',data=train)

sns.boxplot(x= 'property_type_lr',y='price',data=train)

train.dtypes.value_counts()
train.head(1)

datetime_bin_vals = train[['transfer_date', 'previous_transfer_date']]
datetime_bin_vals.shape, train['price'].shape
index_train = X_train['transaction_id']
index_test = X_test['transaction_id']
def preprocess_numeric(input):
  '''
  Takes in a dataframe, and performs all of the preprocessing to feed into model. Made for reusability on 
  test data which is pre-split

  params- input: pandas dataframe
          output: pandas dataframe of X data

  Could not for the life of me perform a join of the onehot encoded features to the numeric so I just used the numeric features in the end.
  '''

  #Fill the missing data for the data
  input['total_floor_area']= input['total_floor_area'].fillna(value = input['total_floor_area'].mean())
  input['number_habitable_rooms'] = input['number_habitable_rooms'].fillna(value = input['number_habitable_rooms'].mean())
  input['floor_level'] = input['floor_level'].fillna(0)
  input['construction_age_band'] = input['construction_age_band'].fillna(0)
  input['previous_price'] = input['previous_price'].fillna(value = input['previous_price'].mean())


  #Define the numerical variables from the dataframe to be scaled
  numerical_vars = input[['total_floor_area', 'number_habitable_rooms', 'previous_price']]
  scaled = MinMaxScaler().fit_transform(numerical_vars)
  scaled = pd.DataFrame(scaled, columns= [['total_floor_area', 'number_habitable_rooms', 'previous_price']], index = input['transaction_id'])
  
  #Perform the needed label encoding for input columns of only two types 
  new_property = LabelEncoder().fit_transform(input['new_property'])
  new_property = pd.DataFrame(new_property, columns= [['new_property']], index= input['transaction_id'])


  #Define the variables for one hot encoding 
  onehot_vars = input[['property_type_lr', 'floor_level', 'construction_age_band']] 

  #Fill the missing values for the categorical values
  input['floor_level'].fillna("missing")
  input['construction_age_band'].fillna("missing")

  #Onehot encoding of categorical vars and dropping duplicates
  onehot = pd.get_dummies(onehot_vars, drop_first=True, )
  onehot = onehot.loc[:,~onehot.columns.duplicated()]
  onehot.index = input['transaction_id']


  #Output the feature df
  df = pd.concat([new_property, scaled], axis=1)

  return df
  def preprocess_categorical(input):
    '''takes in a dataframe of categorical variables and outputs their onehot encoded df'''
    
    #Define the variables for one hot encoding 
    onehot_vars = input[['property_type_lr', 'floor_level', 'construction_age_band']] 

    #Fill the missing values for the categorical values
    input['floor_level'].fillna("missing")
    input['construction_age_band'].fillna("missing")

    #Onehot encoding of categorical vars and dropping duplicates
    onehot = pd.get_dummies(onehot_vars, drop_first=True, )
    onehot = onehot.loc[:,~onehot.columns.duplicated()]
    onehot.index = input['transaction_id']
    return onehot
#create the feature dataframes 
X_train = pd.concat([preprocess_numeric(X_train), preprocess_categorical(X_train)], axis = 1)
X_test =  pd.concat([preprocess_numeric(X_test), preprocess_categorical(X_test)], axis = 1)
#Algorithm for finding the best hyperparams to use for the XGboost algorithm, commented out because takes way too long to run 

'''
parameters_for_testing = {
    'colsample_bytree':[0.4,0.6,0.8],
    'gamma':[0,0.03,0.1,0.3],
    'min_child_weight':[1.5,6,10],
    'learning_rate':[0.1,0.07],
    'max_depth':[3,5],
    'n_estimators':[10000],
    'reg_alpha':[1e-5, 1e-2,  0.75],
    'reg_lambda':[1e-5, 1e-2, 0.45],
    'subsample':[0.6,0.95]  
}

                    
xgb_model = xg.XGBRegressor(learning_rate =0.1, n_estimators=1000, max_depth=5,
     min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8, nthread=6, scale_pos_weight=1, seed=27)

gsearch1 = GridSearchCV(estimator = xgb_model, param_grid = parameters_for_testing, n_jobs=6,iid=False, verbose=10,scoring='neg_mean_squared_error')
gsearch1.fit(X_train,y_train)
print (gsearch1.grid_scores_)
print('best params')
print (gsearch1.best_params_)
print('best score')
print (gsearch1.best_score_)
'''<jupyter_output><empty_output><jupyter_text>Models<jupyter_code>#Set up an XGboost model to get a hirarchy of features used, numeric and categorical 
xgb_r = xg.XGBRegressor(objective ='reg:linear',
                  n_estimators = 5, seed = 123, learning_rate = 0.1, booster= "dart", scale_pos_weight=0.375)

xgb_r.fit(X_train, y_train)
pred = xgb_r.predict(X_test)

#Print the root mean squared error for the XGboost model
rmse = np.sqrt(MSE(y_test, pred))
print("RMSE : % f" %(rmse))

#Find the most important features  

plot_importance(xgb_r, max_num_features= 5)
plt.figure(figsize=(16, 1800), dpi=80)
plt.show()
from tensorflow.keras.callbacks import EarlyStopping

#Define the model
NN_model = Sequential()

# The Input Layer :
NN_model.add(Dense(128, kernel_initializer='normal', activation='relu'))

# The Hidden Layers :
NN_model.add(Dense(256, kernel_initializer='normal',activation='relu'))
NN_model.add(Dense(128, kernel_initializer='normal',activation='relu'))
NN_model.add(Dense(64, kernel_initializer='normal',activation='relu'))


# The Output Layer :
NN_model.add(Dense(1, kernel_initializer='normal',activation='linear'))

# Compile the network :
NN_model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['mean_absolute_error'])

earlystop_callback = EarlyStopping(
  monitor='loss',
  patience=2)
NN_model.fit(X_train, y_train, epochs=10, batch_size=32, validation_split = 0.2, callbacks= earlystop_callback, workers = 10)
model = NN_model
y_pred = model.predict(X_test)

#Printing performance metrics 
print('MAE:', metrics.mean_absolute_error(y_test, y_pred))  
print('MSE:', metrics.mean_squared_error(y_test, y_pred))  
print('RMSE:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
print('VarScore:',metrics.explained_variance_score(y_test,y_pred))

# Visualizing Our predictions
fig = plt.figure(figsize=(10,5))
plt.scatter(y_test,y_pred)
# Perfect predictions
plt.plot(y_test,y_test,'r')
#Plot distribution of errors
predictions = model.predict(X_test)
errors = y_test.values.reshape(8869, 1) - predictions
sns.distplot(errors)<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/distributions.py:2557: FutureWarning:

`distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).<jupyter_text>**Predict on the new data**<jupyter_code>new_x_data = pd.concat([preprocess_numeric(test), preprocess_categorical(test)], axis = 1)
#Predict on the new data
y_pred_live = model.predict(new_x_data)

#Create dataframe of predictions 
final = pd.DataFrame(y_pred_live, columns= [['price']], index= test['transaction_id'])
final<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08244650634349555, 'fraction_numerical': 0.012989964021965537, 'mean_word_length': 4.490954460386774, 'pattern_counts': {'":': 6, '<': 21, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '400896', 'license': 'Apache-2.0', 'n_tokens_mistral': 7495, 'n_tokens_neox': 7035, 'n_words': 2978, 'path': 'machine_learning_challenge.ipynb', 'repo_name': 'cm2435/Coding-Interviews', 'size': '26405'}