<jupyter_start><jupyter_text>OutliersWhy you should worry about outlers?They affect the power of your models of generalizing over datasets. But remember not all outliers are bad. Some outliers may be indication of a case that you are missing or a case that your model is not prepared for.Two types of outliersMild Outliers ->  Q1 - 1.5*IQR or Q3 + 1.5*IQRExtreme outliers ->  Q1 - 3*IQR or Q3 + 3*IQR Methods to deal with outliers1) Univariate MethodThe simplest of all methods. You just have to look for extreme values.For example if you have a dataset where all values are less than 100 and you observe a value of say 1000 than that is cleary a outlier.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
# Generate data
data = 10*np.random.randn(1,50)
print("Max = ", data.max())
print("Min = ",data.min())
print("Mean = ",data.mean())
print("Std = ",data.std())
# Add some external noise
data = np.append(data, [80,95,-40,-100])
print("Max = ", data.max())
print("Min = ",data.min())
print("Mean = ",data.mean())
print("Std = ",data.std())<jupyter_output>Max =  95.0
Min =  -100.0
Mean =  1.989339354114457
Std =  24.205592413236193<jupyter_text>Now you can use box plots to deal with the problem.Boxplots<jupyter_code># In this case you can clearly identify the outliers
plt.boxplot(data)
plt.show()
# But in some cases when the outliers are part of the interquartile range. This method would fail.
data1 = np.arange(1,31).reshape((30,1))
data1[16,0] = 20

plt.plot(data1, 'bo')
plt.show()
# In this case if we draw a boxplot we would not be able to identify the outlier.
plt.boxplot(data1)
plt.show()<jupyter_output><empty_output><jupyter_text>2) Multivariate MethodNow to solve the above problem you must notice a key fact that when the outlier occured it was not a extreme value. So what made this point an outlier was the fact that suddenly the data distribution did not follow the set pattern.In this method we first create a model using the entire data and then compare the actual and the preduicted values of our model and if their difference is above a threshold we treat that point as an outlier.<jupyter_code># For illlustration purpose I create a simple linear regression model
# Although you should prefer a neural network for the purpose
b = np.zeros((1, 1))
W = np.zeros((1, 1))
learning_rate = 0.0001
iterations = 1000

X = np.arange(1,31).reshape((30,1))
Y = np.copy(data1)

for _ in range(iterations):
    y_hat = b + np.dot(X, W)
    err = Y - y_hat
    error = np.sum(np.square(err), axis = 0)
    
    db = -2*np.sum(err, axis = 0)
    dW = -2*np.dot(X.T, err)

    b = b - learning_rate*db
    W = W - learning_rate*dW
# This is the model learned by our model
plt.plot(X, y_hat, 'bo')
plt.show()
# Now set some threshold error value
X = np.arange(1,31).reshape(30,1)
threshold = 2
err = np.abs(data1 - y_hat)
for a in range(data1.shape[0]):
    if err[a,0] > threshold:
        new_data = np.delete(data1, [a,0])
        X = np.delete(X, [a,0])

plt.plot(X, new_data, 'bo')
plt.plot()
data1.shape<jupyter_output><empty_output><jupyter_text>3) Minkowski errorThis method does not remove outliers but it reduces the impact that outliers have on the data.One way to implement it is by remembering that we train our models using Mean Sqaured Loss(MSE). So when we take the square of the differece between y and y_predicted we are essentially squaring the errors and increasing the errors even more.So you can reduce the power of MSE, instead of using 2 use, say 1.5. 4) Histograms<jupyter_code>data = 10*np.random.randn(1,50)
data = np.append(data, [80,95,-40,-100])
plt.hist(data, TT)
plt.show()
x = np.array([1,1,1,3,3])
plt.hist(x, bins = 40)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1072192513368984, 'fraction_numerical': 0.04090909090909091, 'mean_word_length': 4.313920454545454, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '297491', 'license': 'MIT', 'n_tokens_mistral': 1377, 'n_tokens_neox': 1233, 'n_words': 488, 'path': 'Blog Posts Notebooks/How to deal with outliers/How to deal with outliers.ipynb', 'repo_name': 'KushajveerSingh/pytorch_projects', 'size': '3740'}