<jupyter_start><jupyter_text>BLU15 - Model CSI<jupyter_code>import pandas as pd
import numpy as np
import hashlib
import io
import json
import pickle
import requests
import joblib
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline, Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.metrics import precision_score, recall_score, precision_recall_curve
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from lightgbm import LGBMClassifier
from sklearn.pipeline import Pipeline<jupyter_output><empty_output><jupyter_text>Alright, let's go on with the BLU and have fun doing some exercises!  As a reminder:In the learning unit we received a pretrained model and a new batch of data and analyzed whether the model performs well and what to do with it.In the end, we realized that there are some unexpected changes in the data distribution and we need to retrain the model.As the new dataset was pretty small, we have to concat the old data with the new one and train a new model on the combination of 2 datasets.  Exercise 1:- Read the .csv file with the original dataframe as **df_old.**- If you take a look on the **VehicleSearchedIndicator** column, you understand that this subset represents the searched cars only, so we can drop the **VehicleSearchedIndicator** column.- As the new dataset doesn't contain **InterventionDateTime** column, we also need to drop it from the old dataset.- Read new observations as **df_new**.- Combine both the dataframes and add a new column called **is_new** that is going to have all **False** values for the old data and all **True** values for the new observations.- Call the combined dataframe **df_combined**- Drop all **NaN** values- Apply lowercase to department names and intervention location names in the combined dataset<jupyter_code># df_old = ...
# df_new = ...
# df_combined = ...

# YOUR CODE HERE
raise NotImplementedError()
assert df_combined.shape == (78715, 14), 'combined dataframe shape is wrong'
assert 'VehicleSearchedIndicator' not in df_combined.columns, 'Did you drop the VehicleSearchedIndicator column?'
assert 'is_new' in df_combined.columns, 'Did you add is_new column?'
assert sum(df_combined['is_new']) == 2000, 'is_new column has a wrong number of True values'
assert all([name.islower() for name in df_combined['Department Name']]), 'Department name is not lowercased'
assert all([name.islower() or not name.isalpha() for name in df_combined['InterventionLocationName']]), 'InterventionLocationName is not lowercased'<jupyter_output><empty_output><jupyter_text>Exercise 2:**Split the created dataset on train and test parts in the following way:**- Firstly create train and test set. Call them **df_train** and **df_test**.> We'll need them in the future exercises. - Then, split **train** and **test** into **X_train**, **X_test**, **y_train** and **y_test**.- Test sets shape should be 25% of df_combined shape- Make sure to have 25% of new values in the test size.- Use random state 42 while splitting the datasets<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
assert df_train.shape == (59036, 14), 'df_train shape is wrong. Are you sure test size is 25%?'
assert df_test.shape == (19679, 14), 'df_test shape is wrong. Are you sure test size is 25%?'
assert X_train.shape == (59036, 13), 'X_train shape is wrong. Are you sure test size is 25%?'
assert X_test.shape == (19679, 13), 'X_test shape is wrong. Are you sure test size is 25%?'
assert y_train.shape == (59036,), 'X_train shape is wrong. Are you sure test size is 25%?'
assert y_test.shape == (19679,), 'X_train shape is wrong. Are you sure test size is 25%?'
assert sum(X_train['is_new']) == 1500, 'is_new column in Training set has a wrong number of True values. Make sure to have 25% of new values'
assert sum(X_test['is_new']) == 500, 'is_new column in Test set has a wrong number of True values. Make sure to have 25% of new values'<jupyter_output><empty_output><jupyter_text>Now we need to retrain the model.If we simply load the pipeline and retrain it, it's going to ignore our new feature.So let's create the same pipeline as in the original notebook:<jupyter_code>categorical_features = df_combined.columns.drop(['ContrabandIndicator', 'SubjectAge'])
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))])

preprocessor = ColumnTransformer(
    transformers=[('cat', categorical_transformer, categorical_features)])

pipeline = make_pipeline(
    preprocessor,
    LGBMClassifier(n_jobs=-1, random_state=42),
)
pipeline.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Exercise 3:Now let's test how the model performs:- Make model binary predictions and save them to an array **preds**. - Make model probability predictions and save them to an array called **preds_proba**. Keep only True class probabilities (by default probability prediction returns you both False and True classes probabilities)- Create a variable called **precision** with the model precision score.- Create a variable called **recall** with the model recall score<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
expected_recall = 'a2cffa866c48b997372a62104161ba89e68fb439c418fc2559e2a32c44987ce8'
hash_recall = hashlib.sha256(bytes(str(round(recall, 2)), encoding='utf8')).hexdigest()
assert hash_recall == expected_recall
assert len(preds) == 19679, 'Are you sure you made predictions for test set only?'
assert len(preds_proba) == 19679, 'Are you sure you made predictions for test set only?'
assert not isinstance(preds_proba[0], np.ndarray), 'Are you sure you kept only the True class predictions?'
assert round(sum(preds_proba)) == 6563
np.testing.assert_almost_equal(precision, 0.64966, decimal=2)
np.testing.assert_almost_equal(recall, 0.546027, decimal=2)<jupyter_output><empty_output><jupyter_text>Exercise 4: It's already not bad, but let's now try to calculate the optimal threshold.By threshold I mean the minimal probability of a prediction that we're going to call "True". By default, any prediction with probability > 0.5 is called True, but we might find a better value. The metric is the same: our success rate (precision) needs to be at least 50%, and the recall should be as big as possible. Save the result to a variable called **threshold**. Round the result to 2 decimal points.<jupyter_code># threshold = ...
# YOUR CODE HERE
raise NotImplementedError()
assert round(threshold, 2) == threshold, 'Did you round the value?'
ans_threshold = hashlib.sha256(bytes(str(threshold), encoding='utf8')).hexdigest()
assert ans_threshold == "6382e07f9de0c85293aee2a45b88c61c28589419682ecc2f8c097f750e861a24"<jupyter_output><empty_output><jupyter_text>Exercise 5:- Now create a list of predictions. > All the values from the **preds_proba** list that have a value > threshold should be True. The rest should be False.> Save the result to a variable called **best_preds**- Calculate the precision and recall and save them to variables **precision** and **recall**<jupyter_code># best_preds = ...
# precision = ...
# recall = ...
# YOUR CODE HERE
raise NotImplementedError()
np.testing.assert_almost_equal(precision, 0.50290256, decimal=2)
np.testing.assert_almost_equal(recall, 0.84422789, decimal=2)<jupyter_output><empty_output><jupyter_text>Exercise 6:Now let's find out whether removing rare values is going to help. **Filter *df_train* (the one you created in the Exercise 2) the following way:** - Remove rows with **Department Name** that appear <= 50 times- Remove rows with **InterventionLocationName** that appear <= 50 times- Remove rows with **ReportingOfficerIdentificationID** that appear <= 30 times- Remove rows with **StatuteReason** that appear <= 10 times- Note: it's better to keep the original dataframe not touched. Create a copy of the original dataframe and save the results to a variable **train_filtered**> We have to filter the values after we split the dataset into training and test, because by filtering the test set we also affect the score. If we filtered everything besides the examples that are the easiest to predict, we'd have a super nice score, but in production we're going to expect both the filtered values and unfiltered ones. > We shouldn't worry about the fact, that some values in the test set will not be present in the training set, because the pipeline is simply going to ignore them.> (you might use the logic from the original model's notebook, but I suggest trying to implement it by yourself, it's a good exercise)<jupyter_code># train_filtered = df_train.copy()
# train_filtered = ...

# YOUR CODE HERE
raise NotImplementedError()
assert train_filtered.shape == (30502, 14), 'Make sure to filter rare values. Make sure to filter only train set.'
assert 'middlebury' not in train_filtered['Department Name'], 'Did you filter department names?'
assert 'hampton' not in train_filtered['InterventionLocationName'], 'Did you filter InterventionLocationName ?'
assert 'DACYR048' not in train_filtered['ReportingOfficerIdentificationID'], 'Did you filter officer ids?'
assert 'Stop Sign ' not in train_filtered['StatuteReason'], 'Did you filter statute reasons?'<jupyter_output><empty_output><jupyter_text>Exercise 7:**Let's split *train_filtered* into *X* and *Y* parts and do the same thing once again:**- Fit the model on the training set (this time filtered one)- Predict probabilities for the test set (untouched one).- Select the best threshold for the specified requirements (precision >= 0.5, max possible recall).- Round up the threshold up to 2 decimal points.- Transform probabilities to binary answers: probability above the threshold = True, False otherwise.- Calculate the precision and recall scores for these predictions. I believe you need no exact instructions, as you did exactly same things in Exercises 2, 3 and 4.Save the score results to variables called **filtered_precision** and **filtered_recall**<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
np.testing.assert_almost_equal(filtered_precision, 0.501309, decimal=2)
np.testing.assert_almost_equal(filtered_recall, 0.83238, decimal=2)<jupyter_output><empty_output><jupyter_text>Okay, so it seems like the original notebook had a mistake of evaluating the model on filtered test set. In fact, filtering features with these frequency limits decreased the recall (0.844 -> 0.832).   Exercise 8:    Now I'll let you use your fantasy and try to filter the categorical values differently. You're free to do whatever you want, but here are a few ideas you can use:- Instead of dropping rare categories, create a new value for them- Adjust the frequency values (e.g. keep a part of departments we just filtered or filter even more). You can try to search all the possible combinations of frequency values if you want. Your task is to create a list of *True/False* predictions for the **X_test** and call them **best_preds**. These predictions have to have precision >= 0.5 and recall > 0.84422789<jupyter_code># predictions = ...
# YOUR CODE HERE
raise NotImplementedError()
precision = precision_score(y_test, best_preds)
recall = recall_score(y_test, best_preds)
assert precision >= 0.5
assert recall > 0.84422789<jupyter_output><empty_output><jupyter_text>Exercise 9:So, we got the model. It's usually a good idea to retrain the model on the whole dataset, so now I want you to:- Apply the filters that you just created in the Exercise 7 to **df_combined**- Train the same model on the whole dataset- Export the model, train columns and data types to **/tmp/**, where files are called **new_pipeline.pickle**, **new_dtypes.pickle** and **new_columns.json**.<jupyter_code># YOUR CODE HERE
raise NotImplementedError()
with open('/tmp/new_columns.json') as fh:
    columns = json.load(fh)


with open('/tmp/new_pipeline.pickle', 'rb') as fh:
    pipeline = joblib.load(fh)


with open('/tmp/new_dtypes.pickle', 'rb') as fh:
    dtypes = pickle.load(fh)

assert isinstance(columns, list), 'columns need to be a list of training features'
assert 'ContrabandIndicator' not in columns, 'there should be only training features in columns. You got target there.'
assert 'is_new' in columns, "your columns don't contain is_new feature. Are you you updated the columns file?"
assert isinstance(pipeline, Pipeline), 'new_pipeline.pickle does not seem it be an instance of Pipeline class.'
assert isinstance(dtypes, pd.core.series.Series)
assert all([column in dtypes.index for column in columns]), 'some columns from new_columns file are not in the new_dtypes file'
assert all([dtype in columns for dtype in dtypes.index]), 'some dtypes from new_dtypes file are not in the new_columns file'<jupyter_output><empty_output><jupyter_text>Exercise 10:And now it's time to change the server! I know you missed this part :) Before we do it, I want to remind you, that in this exercise we didn't cover the ethics topic.Our model is trained on sensible features like race, sex and ethnicity. In real situation you'd need to make sure that your model is not discriminating anyone. Now, go and create a copy of the **protected_server.py** file. Call it **new_server.py**In that file:- Change the **check_valid_column** function to have the new added columns> You can also automate it by reading the columns file, it's even better!- Change the **check_categorical_values** function:> We didn't really affect any of the checked columns there besides **StatuteReason** (of course, if you didn't change it in your best solution). Remove the values that should not be in this column anymore.> We also add one more categorical feature to the dataframe (**is_new**). Go and add possible values to the check.- As soon as it's done, go ahead and start the server. - Play with the predictions. Make sure that the server checks the **is_new** feature values. Try to send requests without **is_new** or with a different value (not True or False). - After you're done, change the value of **done** to **True** to pass the exercise<jupyter_code>done = False
# YOUR CODE HERE
raise NotImplementedError()
assert done == True<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08493285523447937, 'fraction_numerical': 0.02235815228854672, 'mean_word_length': 5.492012779552716, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '509600', 'license': 'MIT', 'n_tokens_mistral': 4201, 'n_tokens_neox': 3880, 'n_words': 1831, 'path': 'S06 - DS in the Real World/BLU15 - Model CSI/Exercise notebook.ipynb', 'repo_name': 'LDSSA/batch4-students', 'size': '14223'}