<jupyter_start><jupyter_text>Anomaly Detection* What are Outliers ?* Isolation Forest* Local Outlier Factor* DBSCAN  Outliers* New data which doesn't belong to general trend (or distribution) of entire data are known as outliers.* Data belonging to general trend are known as inliners.* Learning models are impacted by presence of outliers.* Anomaly detection is another use of outlier detection in which we find out unusual behaviour.* Data which were detected outliers can be deleted from complete dataset.* Outliers can also be marked before using them in learning methods  Isolation Forest* Based on RandomForest* Useful in detecting outliers in high dimension datasets.* This algorithm randomly selects a feature & splits further.* Random partitioning produces shorter part for anomolies.* When a forest of random trees collectively produce shorter path lengths for particular samples, they are highly likely to be anomalies.<jupyter_code>import numpy as np
import pylab as plt
rng = np.random.RandomState(42)

# Generate train data
X = 0.3 * rng.randn(100, 2)
X_train = np.r_[X + 2, X - 2]
# Generate some regular novel observations
X = 0.3 * rng.randn(20, 2)
X_test = np.r_[X + 2, X - 2]
# Generate some abnormal novel observations
X_outliers = rng.uniform(low=-4, high=4, size=(20, 2))
from sklearn.ensemble import IsolationForest
data = np.r_[X_train,X_test,X_outliers]
iso = IsolationForest(behaviour='new', contamination='auto')
iso.fit(data)
pred = iso.predict(data)
plt.scatter(data[:,0], data[:,1],s=10,c=pred)<jupyter_output><empty_output><jupyter_text>Local Outlier Factor* Based on nearest neighbours* Suited for moderately high dimension datasets* LOF computes a score reflecting degree of abnormility of a data.* LOF Calculation  - Local density is calculated from k-nearest neighbors.  - LOF of each data is equal to the ratio of the average local density of his k-nearest neighbors, and its own local density.  - An abnormal data is expected to have smaller local density.* LOF tells you not only how outlier the data is but how outlier is it with respect to all data<jupyter_code>from sklearn.neighbors import LocalOutlierFactor
lof = LocalOutlierFactor(n_neighbors=25,contamination=.1)
pred = lof.fit_predict(data)
s = np.abs(lof.negative_outlier_factor_)
plt.scatter(data[:,0], data[:,1],s=s*10,c=pred)<jupyter_output><empty_output><jupyter_text>Outlier Detection using DBSCAN* DBSCAN is a clustering method based on density* Groups data which are closer to each other.* Doesn't use distance vector calculation method* Data not close enough to any cluster is not assigned any cluster & these can be anomalies<jupyter_code>from sklearn.cluster import DBSCAN
dbscan = DBSCAN(eps=.3)
dbscan.fit(data)
plt.scatter(data[:,0], data[:,1],s=s*10,c=dbscan.labels_)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08116767532929868, 'fraction_numerical': 0.013527945888216448, 'mean_word_length': 5.738609112709832, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '242839', 'license': 'MIT', 'n_tokens_mistral': 875, 'n_tokens_neox': 804, 'n_words': 336, 'path': 'notebooks/17- Anomaly Detection.ipynb', 'repo_name': 'Gamein-SUT/DataMining-With-Python---Gamein2020', 'size': '2809'}