<jupyter_start><jupyter_text>Jupyter Refactoring for Beginners  Workshop structure- [Introduction](Introduction)- [Why Refactoring is needed](Why-Refactoring-is-needed)- [How to refactor your code (prerequisites)](How-to-refactor-your-code-(prerequisites))- [How to write cleaner and more efficient code in Pythonic way](How-to-write-cleaner-and-more-efficient-code-in-Pythonic-way)- [Bonus](Bonus)- [Q&A time](Q&A-time)  Introduction About me**<NAME>**: [Principal Data Solutions Engineer at LINKIT](https://www.linkit.nl/en) and [Organiser at PyLadies Amsterdam](https://amsterdam.pyladies.com/)     Feel free to contact me via LinkedIn: https://www.linkedin.com/in/alyonagalyeva/  Why Refactoring is needed ![Awesome script](https://chezsoi.org/lucas/blog/images/wwcb/Ill_just_write_a_quick_script...-catacrac.net.png)  Welcome to the land of Spaghetti code and Big Ball of Mud<jupyter_code>%load_ext autoreload
%autoreload 2
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import ipytest
import pytest

ipytest.autoconfig()
pd.set_option("display.max_columns", None) 
# DO NOT REPEAT THIS AT HOME !!! IT IS DANGEROUS AND CONTAGIOUS !!!
file = "/Users/alyonagalyeva/tutorials/WIP/Jupyter-refactoring-beginner-jul2020/workshop/data/Boston_housing_prices.csv"
names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']
df=pd.read_csv(file, delim_whitespace=True, names=names)
file = "data/Boston_housing_prices.csv"
names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']
df=pd.read_csv(file, delim_whitespace=True, names=names)
df.head()
df1=df[df['CHAS'] > 0]
df1.head()
df2=df[df['AGE']==45.8]
df2.head()
df3=df.rename(str.lower, axis='columns')
df3.head()
# Hoerra, we have a function !!!
def twicecol(df, dfcol):
    df['twicecol'] = df[dfcol] * df[dfcol]
    return df
df4=twicecol(df3, 'nox')
df4.head()
# Oops, we mutated original dataframe df3, how it is even possible?
df3.head()<jupyter_output><empty_output><jupyter_text>"Refactoring is an essential technique to allow a program to be changed safely. It consists of making small changes that don't alter the observable behavior of the software. " - *<NAME>* **So, why do we need Refactoring?**- to understand the code- to improve the readability of code- to make it easier to enhance the code- to reduce time of bug hunting **What is Technical Debt?**1. It's a metaphor to describe problems with code design2. It's a lot of shortcuts taken for various reasons3. We borrow against future4. It needs to be paid off with refactoring **When to stop Refactoring?**- Stop when your refactoring step becomes too big- Stop when there is no ROI  How to refactor your code (prerequisites)  The art of Recognizing Issues in your code how comments can help or badly hurt you<jupyter_code># give new names to the columns by renaming the columns and make each columns lowercase, use axis columns 
df3=df.rename(str.lower, axis='columns')
## Correlation with independent Variable (Note: Models like RF are not linear like these)
df.columns
df2 = df.drop('MEDV', axis=1)
df.corrwith(df['MEDV']).plot.bar(
        figsize = (10, 10), title = "Correlation with MEDV", fontsize = 15,
        rot = 45, grid = True)<jupyter_output><empty_output><jupyter_text>confusing names<jupyter_code># Remember our friends df, df1, df2 and this awesome function?
def twicecol(df, dfcol):
    df['twicecol'] = df[dfcol] * df[dfcol]
    return df<jupyter_output><empty_output><jupyter_text>magic numbers<jupyter_code># Somewhere in notebook
val = 6

# later somewhere at the end of the notebook
grouped = df.groupby('TAX').agg([np.sum, np.mean, np.std])

if val > 6:
    print(grouped.head())
else:
    print(df.head())<jupyter_output><empty_output><jupyter_text>duplication and large code blocks are dangerous<jupyter_code>#Define X and y
X = df2
y = df.loc[:,'MEDV']

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

#### Model Building ####
### Comparing Models

## Multiple Linear Regression Regression
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

# Predicting Test Set
y_pred = regressor.predict(X_test)
from sklearn import metrics
mae = metrics.mean_absolute_error(y_test, y_pred)
mse = metrics.mean_squared_error(y_test, y_pred)
rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))
r2 = metrics.r2_score(y_test, y_pred)

results = pd.DataFrame([['Multiple Linear Regression', mae, mse, rmse, r2]],
               columns = ['Model', 'MAE', 'MSE', 'RMSE', 'R2 Score'])

## Suport Vector Regression 
'Necessary Standard Scaler '
from sklearn.svm import SVR
regressor = SVR(kernel = 'rbf')
regressor.fit(X_train, y_train)

# Predicting Test Set
y_pred = regressor.predict(X_test)
from sklearn import metrics
mae = metrics.mean_absolute_error(y_test, y_pred)
mse = metrics.mean_squared_error(y_test, y_pred)
rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))
r2 = metrics.r2_score(y_test, y_pred)

model_results = pd.DataFrame([['Support Vector RBF', mae, mse, rmse, r2]],
               columns = ['Model', 'MAE', 'MSE', 'RMSE', 'R2 Score'])

results = results.append(model_results, ignore_index = True)

## Decision Tree Regression
from sklearn.tree import DecisionTreeRegressor
regressor = DecisionTreeRegressor(random_state=0)
regressor.fit(X_train, y_train)

# Predicting Test Set
y_pred = regressor.predict(X_test)
from sklearn import metrics
mae = metrics.mean_absolute_error(y_test, y_pred)
mse = metrics.mean_squared_error(y_test, y_pred)
rmse = np.sqrt(metrics.mean_squared_error(y_test, y_pred))
r2 = metrics.r2_score(y_test, y_pred)

model_results = pd.DataFrame([['Decision Tree Regression', mae, mse, rmse, r2]],
               columns = ['Model', 'MAE', 'MSE', 'RMSE', 'R2 Score'])

results = results.append(model_results, ignore_index = True)<jupyter_output><empty_output><jupyter_text>poor formatting<jupyter_code>fig=plt.figure( figsize=(10, 10) ); plt.suptitle( 'Histograms', fontsize=20 )
for i in range(df2.shape[1]):
    plt.subplot( 6,3,i+1);f = plt.gca();f.set_title(df2.columns.values[i])

    vals = np.size(df2.iloc[:, i].unique())
    if vals>=100: vals=100
    
    plt.hist(df2.iloc[:, i], bins=vals, color='#3F5D7D')
plt.tight_layout(rect=[0, 0.03, 1, 0.95])<jupyter_output><empty_output><jupyter_text>zombie code<jupyter_code>#grid_sv = GridSearchCV(desc_tr, cv=kf, param_grid={"max_depth" : [1, 2, 3, 4, 5, 6, 7]}, scoring='neg_mean_squared_error')
#grid_sv.fit(x_scaled, y)
#print("Best classifier :", grid_sv.best_estimator_)

#grid_sv = GridSearchCV(knn, cv=kf, param_grid={"n_neighbors" : [2, 3, 4, 5, 6, 7]}, scoring='neg_mean_squared_error')
#grid_sv.fit(x_scaled, y)
#print("Best classifier :", grid_sv.best_estimator_)<jupyter_output><empty_output><jupyter_text>How to write cleaner and more efficient code in Pythonic way  Typical refactoring workflow - answer why refactoring is needed in this particular situation- git, conda environment (or venv) are your best friends- make a copy of a notebook you are going to refactor- fix the seed- define a layout for your refactored notebook- pick a chunk of code to refactor- if you created a function, write tests for this function, run tests. If success --> git commit- else update your function to make the tests pass --> git commit- repeat **Exercise 0:**Take a look at two notebooks in notebooks_to_refactor folder.Can you recognize some of the issues in the code? Is a goal of the notebooks clear enough?Is a structure of notebooks clear as well?  Are you ready to roll up the sleeves and start hands-on part together with me?  Refactoring techniques  RenameLet's take a look at the most powerful and widely used refactoring technique: **Rename**What we can **Rename**:- a variable- a function- function parameters- a class- class attributes - class methods<jupyter_code>x = ("data/movies_metadata.csv")
df = pd.read_csv(x)
df1 = df[df["budget"] > 30000000]

#rename variables 
filepath = ("data/movies_metadata.csv")
movies = pd.read_csv(filepath, low_memory=False)
movies_big_budget = df[df["budget"] > 30000000]
def my_func(a, b):
    return a.lower() + b.upper()

#Rename function and its parameters 
def mix_lower_upper_case(first_param, second_param):
    return first_param.lower() + second_param.upper()

#We can add type hints for readability purposes
def mix_lower_upper_case(first_param: str, second_param: str) -> str:
    """ Concatenate two strings, cast the first string to lowercase and the second string to uppercase"""
    return first_param.lower() + second_param.upper()

# Add casting to a str to make sure that the function returns what expected
def mix_lower_upper_case(first_param: str, second_param: str) -> str:
    """ Concatenate two strings, cast the first string to lowercase and the second string to uppercase"""
    return str(first_param).lower() + str(second_param).upper()<jupyter_output><empty_output><jupyter_text>**Exercise 1:**Rename a function, add docstring, type hintsdef refactor_me(x, y, z):    return 2*x+y+z**2  ExtractLet's try another refactoring technique: **Extract**What we can **Extract**:- a variable- a function- function parameters- class- class methods<jupyter_code>movies_valid_budget = movies.loc[movies["budget"] != 0]

# extract a variable
valid_budget = movies["budget"] != 0
movies_valid_budget = movies.loc[valid_budget]<jupyter_output><empty_output><jupyter_text>Let's extract a function, but before this we need to add tests<jupyter_code>%%run_pytest[clean] -vv

#Our tests
def test_say_hello_success():
    assert say_hello() == "Hello Again"

def test_say_hello_failure():
    assert say_hello() == "hello Again"

    
#A function to be tested
def say_hello():
    return "Hello Again"
%%run_pytest[clean] -vv

#Our tests
#positive case
def test_multiply_column_by_2():
    input_df = pd.DataFrame([(1, 2), (2, 8)], columns=["A", "B"])
    test_df = multiply_column(input_df, "B", 2)
    expected_df = pd.Series([4,16], name="B")
    pd.testing.assert_series_equal(test_df, expected_df)

#positive case
def test_multiply_column_by_min1():
    input_df = pd.DataFrame([(1, 2), (2, 8)], columns=["A", "B"])
    test_df = multiply_column(input_df, "A", -1)
    expected_df = pd.Series([-1,-2], name="A")
    pd.testing.assert_series_equal(test_df, expected_df)
   
# negative case
def test_multiply_column_not_exist():
    with pytest.raises(KeyError):
        input_df = pd.DataFrame([(1, 2), (2, 8)], columns=["A", "B"])
        test_df = multiply_column(input_df, "C", 0)

#Extract a function
def multiply_column(df: pd.DataFrame, column_name: str, multiplier: int) -> pd.Series:
    df = df.copy()
    updated_df = df[column_name] * multiplier
    return updated_df<jupyter_output><empty_output><jupyter_text>**Exercise 2:**Extract a function and 3 tests for the following lines below<jupyter_code>#clean_movie_data_df = movie_data_df.loc[movie_data_df["budget"] != 0  ]
#clean_movie_data_df = clean_movie_data_df.loc[clean_movie_data_df["revenue"] != 0  ]<jupyter_output><empty_output><jupyter_text>InlineLet's move to another refactoring technique: **Inline**What we can **Inline**:- a variable- a function- a class- class attributes- class methods<jupyter_code>filepath = ("data/movies_metadata.csv")
movies = pd.read_csv(filepath, low_memory=False)

#Inline variable
movies = pd.read_csv("data/movies_metadata.csv", low_memory=False)<jupyter_output><empty_output><jupyter_text>MoveAnd what about this refactoring technique: **Move**What we can **Move**:- a variable- a function- a class- class methods<jupyter_code>class MyRegressModel():
    
    def do_smth(self):
        pass
    
    def do_classification(self, inputs):
        pass
    
# Move a class method
class MyRegressModel():
    
    def do_smth(self):
        pass
    
class MyClassifModel():
      
    def do_classification(self, inputs):
        pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12550707840052985, 'fraction_numerical': 0.012335458233297458, 'mean_word_length': 5.394917946003177, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 65, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '319553', 'license': 'Apache-2.0', 'n_tokens_mistral': 4318, 'n_tokens_neox': 4138, 'n_words': 1162, 'path': 'workshop/jupyter_refactoring_workshop.ipynb', 'repo_name': 'EzheZhezhe/Jupyter_refactoring_series', 'size': '12094'}