<jupyter_start><jupyter_text>Create an Instance of the Counts Outlier Detector<jupyter_code>det = CountsOutlierDetector.CountsOutlierDetector()<jupyter_output><empty_output><jupyter_text>Simple Example with no Correlation Between Columns<jupyter_code># Create random, uniform data with 500 rows and 10 columns
a = np.random.uniform(low=0.0, high=100.0, size=(500, 10))

# Create a row with one unusual value
a[0,0] = 1000.0

# Create a row with two unusual values
a[1,0] = 1100.0
a[1,1] = 1200.0

a = pd.DataFrame(a)
flagged_rows_df, row_explanations, output_msg, run_summary_df = det.predict(a)

# The output_msg provides a string representation of the outlier detection process, indicating, for each number of 
# dimensions checked, the number of rows flagged different numbers of times. For example, when considering 1d outliers 
# (values that are rare in a single column), rows may be flagged in 0 or more columns. 

print(output_msg)
# run_summary_df gives a higher level of summary, indicating how many rows were flagged checking each dimension (regardless
# of how many times each rows was flagged -- they may have, for example, 1 unusual pair of values, or 3 unusual pairs 
# of values).

run_summary_df
# The flagged_rows dataframe returned contains statistics about each row. In most cases, these are not useful to examine
# other than for modifying the scoring system. A dataframe row_explanations is also provided, and an example is given
# here. This has a row for each row in the data that was flagged at least once, along with the explanations for each 
# unusual value or combination of values. 

row_explanations
# Example accessing one row explanation
row_explanations.iloc[0]['1d Explanations']<jupyter_output><empty_output><jupyter_text>Example with Correlation Between Columns (allowing multi-dimensional outliers)<jupyter_code>np.random.seed(0)
X, y= make_classification(n_samples=500, n_features=6, n_informative=1, n_redundant=3, n_repeated=1, n_clusters_per_class=1)
X = pd.DataFrame(X)
X['Outlier'] = False

# Set 3 random rows to random values, setting 1 to 3 columns to realistic, but different values
X.at[0,0] = X.iloc[1, 0]
X.at[0, 'Outlier'] = True

X.at[1,0] = X.iloc[10, 0]
X.at[1,1] = X.iloc[11, 1]
X.at[1, 'Outlier'] = True

X.at[2,0] = X.iloc[30, 0]
X.at[2,1] = X.iloc[31, 1]
X.at[2,2] = X.iloc[32, 1]
X.at[2, 'Outlier'] = True
import warnings
warnings.filterwarnings('ignore' )

sns.pairplot(X, hue='Outlier')

# The graphs show the outliers in a grid of axes. Each variable is shared across one row 
# of the y-axes and one column of the x-axes. The diagonal plots present 
# a univariate distribution plot, showing the marginal distribution of the data in each column.
X_test = X.drop(columns=['Outlier'])
flagged_rows_df, row_explanations, output_msg, run_summary_df = det.predict(X_test)
print(output_msg)
# This should flag rows 0, 1, and 2 as having 2d outliers, as the values were set to common values, but ones that 
# would no longer correlate as well with other values in the row. 
# This may also flag some outliers that occur naturally in the data. 

row_explanations<jupyter_output><empty_output><jupyter_text>Example with Iris Dataset<jupyter_code>def get_iris():
    iris = load_iris()
    X, y = iris.data, iris.target
    X = pd.DataFrame(X, columns=iris['feature_names'])
    return X, y

def get_breast_cancer():
    X, y = load_breast_cancer(return_X_y=True, as_frame=True)
    return X,y

def get_wine():
    X, y = load_wine(return_X_y=True, as_frame=True)
    return X,y
X,y = get_iris()
flagged_rows_df, row_explanations, output_msg, run_summary_df = det.predict(X)
display(run_summary_df)
X['Outlier'] = flagged_rows_df['Any Scored']
sns.pairplot(X, hue='Outlier')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09526343799893561, 'fraction_numerical': 0.024215007982969664, 'mean_word_length': 5.275459098497496, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '360200', 'license': 'MIT', 'n_tokens_mistral': 1359, 'n_tokens_neox': 1263, 'n_words': 436, 'path': 'examples/Examples_Counts_Outlier_Detector.ipynb', 'repo_name': 'ChristinaPierre/CountsOutlierDetector', 'size': '3758'}