<jupyter_start><jupyter_text>RahaWelcome to Raha, a configuration-free error detection system!  We demonstrate this system in this notebook. In particular, we will learn how to detect data errors in a dataset with Raha.  1. Instantiating RahaWe first load and instantiate `raha`.<jupyter_code>import raha
app = raha.Detection()

# How many tuples would you label?
app.LABELING_BUDGET = 20   

# Do you want to filter out irrelevant error detector startegies?
app.STRATEGY_FILTERING = False
app.HISTORICAL_DATASETS = [
    {
        "name": "hospital",
        "path": "/media/mohammad/C20E45C80E45B5E7/Projects/raha/datasets/hospital/dirty.csv",
        "clean_path": "/media/mohammad/C20E45C80E45B5E7/Projects/raha/datasets/hospital/clean.csv"
    },
    {
        "name": "beers",
        "path": "/media/mohammad/C20E45C80E45B5E7/Projects/raha/datasets/beers/dirty.csv",
        "clean_path": "/media/mohammad/C20E45C80E45B5E7/Projects/raha/datasets/beers/clean.csv"
    }
]<jupyter_output><empty_output><jupyter_text>2. Instantiating the DatasetWe next load and instantiate the dataset object.<jupyter_code>dataset_dictionary = {
    "name": "flights",
    "path": "/media/mohammad/C20E45C80E45B5E7/Projects/raha/datasets/flights/dirty.csv",
    "clean_path": "/media/mohammad/C20E45C80E45B5E7/Projects/raha/datasets/flights/clean.csv"
}
d = app.initialize_dataset(dataset_dictionary)
d.dataframe.head()<jupyter_output><empty_output><jupyter_text>3. Running Error Detection StrategiesRaha runs (all or the promising) error detection strategies on the dataset. This step could take a while because all the strategies should be run on the dataset.<jupyter_code>app.run_strategies(d)<jupyter_output>I just load strategies' results as they have already been run on the dataset!<jupyter_text>4. Generating FeaturesRaha then generates a feature vector for each data cell based on the output of error detection strategies.<jupyter_code>app.generate_features(d)<jupyter_output>40 Features are generated for column 0.
66 Features are generated for column 1.
62 Features are generated for column 2.
156 Features are generated for column 3.
73 Features are generated for column 4.
156 Features are generated for column 5.
88 Features are generated for column 6.<jupyter_text>5. Building ClustersRaha next builds a hierarchical clustering model for our clustering-based sampling approach.<jupyter_code>app.build_clusters(d)<jupyter_output>A hierarchical clustering model is built for column 0.
A hierarchical clustering model is built for column 1.
A hierarchical clustering model is built for column 2.
A hierarchical clustering model is built for column 3.
A hierarchical clustering model is built for column 4.
A hierarchical clustering model is built for column 5.
A hierarchical clustering model is built for column 6.<jupyter_text>6. Interactive Tuple Sampling and LabelingRaha then iteratively samples a tuple. We should label data cells of each sampled tuple.<jupyter_code>while len(d.labeled_tuples) < app.LABELING_BUDGET:
    app.sample_tuple(d)
    if d.has_ground_truth:
        app.label_with_ground_truth(d)
    else:
        import pandas
        import IPython.display
        print("Label the dirty cells in the following sampled tuple.")
        sampled_tuple = pandas.DataFrame(data=[d.dataframe.iloc[si, :]], columns=d.dataframe.columns)
        IPython.display.display(sampled_tuple)
        for j in range(d.dataframe.shape[1]):
            cell = (si, j)
            value = d.dataframe.iloc[cell]
            d.labeled_cells[cell] = int(input("Is the value '{}' dirty?\nType 1 for yes.\nType 0 for no.\n".format(value)))<jupyter_output>Tuple 297 is sampled.
Tuple 297 is labeled.
Tuple 697 is sampled.
Tuple 697 is labeled.
Tuple 1527 is sampled.
Tuple 1527 is labeled.
Tuple 1733 is sampled.
Tuple 1733 is labeled.
Tuple 1866 is sampled.
Tuple 1866 is labeled.
Tuple 810 is sampled.
Tuple 810 is labeled.
Tuple 740 is sampled.
Tuple 740 is labeled.
Tuple 1497 is sampled.
Tuple 1497 is labeled.
Tuple 1831 is sampled.
Tuple 1831 is labeled.
Tuple 1041 is sampled.
Tuple 1041 is labeled.
Tuple 1240 is sampled.
Tuple 1240 is labeled.
Tuple 2151 is sampled.
Tuple 2151 is labeled.
Tuple 681 is sampled.
Tuple 681 is labeled.
Tuple 771 is sampled.
Tuple 771 is labeled.
Tuple 1018 is sampled.
Tuple 1018 is labeled.
Tuple 566 is sampled.
Tuple 566 is labeled.
Tuple 254 is sampled.
Tuple 254 is labeled.
Tuple 1645 is sampled.
Tuple 1645 is labeled.
Tuple 1343 is sampled.
Tuple 1343 is labeled.
Tuple 319 is sampled.
Tuple 319 is labeled.<jupyter_text>7. Propagating User LabelsRaha then propagates each user label through its cluster.<jupyter_code>app.propagate_labels(d)<jupyter_output>The number of labeled data cells increased from 140 to 13185.<jupyter_text>8. Classifying CellsRaha then trains and applies one classifier per data column to predict the label of the rest of data cells.<jupyter_code>app.classify_cells(d)<jupyter_output>A classifier is trained and applied on column 0.
A classifier is trained and applied on column 1.
A classifier is trained and applied on column 2.
A classifier is trained and applied on column 3.
A classifier is trained and applied on column 4.
A classifier is trained and applied on column 5.
A classifier is trained and applied on column 6.<jupyter_text>9. Storing ResultsRaha can also store the error detection results.<jupyter_code>app.store_results(d)<jupyter_output>The results are stored in /media/mohammad/C20E45C80E45B5E7/Projects/raha/datasets/flights/raha-results-flights/error-detection/detection.dictionary.<jupyter_text>10. Evaluating Error DetectionWe can finally evaluate our error detection process.<jupyter_code>p, r, f = d.get_data_cleaning_evaluation(d.detected_cells)[:3]
print("Raha's performance on {}:\nPrecision = {:.2f}\nRecall = {:.2f}\nF1 = {:.2f}".format(d.name, p, r, f))<jupyter_output>Raha's performance on flights:
Precision = 0.83
Recall = 0.85
F1 = 0.84
{'fraction_non_alphanumeric': 0.07823014804845221, 'fraction_numerical': 0.04845222072678331, 'mean_word_length': 4.525092936802974, 'pattern_counts': {'":': 9, '<': 34, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '290027', 'license': 'Apache-2.0', 'n_tokens_mistral': 2136, 'n_tokens_neox': 1834, 'n_words': 652, 'path': 'raha/.ipynb_checkpoints/raha-checkpoint.ipynb', 'repo_name': 'AridalF/raha', 'size': '5944'}