<jupyter_start><jupyter_text>![alt text](https://datacentricai.cc/images/logos/budgetclean_hu47fef794183e03643658b6e09800de8e_91147_250x250_fit_box_2.png "Budget Clean Logo") `dcbench`: Budget Clean Task DemoThis is a simple demo to showcase the interface of the `budgetclean` task. You can read more about the task in our [documentation](https://dcbench.readthedocs.io/en/latest/tasks.htmlminimal-feature-cleaning) and on our [website](https://www.datacentricai.cc/benchmark/tasks/budgetclean/). We can start off by importing `dcbench` and listing the available tasks.<jupyter_code>import dcbench
dcbench.tasks<jupyter_output><empty_output><jupyter_text>Exploring problemsHere we want to focus on the budget clean task. We can list its problems as such:<jupyter_code>budgetclean = dcbench.tasks["budgetclean"]<jupyter_output><empty_output><jupyter_text>Solving a specific problemWe can load a specific problem instance.<jupyter_code>problem = budgetclean.problems["p001"]
problem<jupyter_output><empty_output><jupyter_text>We can view the problems artifacts:<jupyter_code>problem.artifacts<jupyter_output><empty_output><jupyter_text>Each one of these artifacts is a Pandas data frame. We can view the dirty dataset:<jupyter_code>problem["X_train_dirty"]<jupyter_output><empty_output><jupyter_text>Solving this problem requires us to select a subset of dirty data that should be cleaned. To construct a solution, we need to produce a boolean list (or single-column data frame) that is of the same size as `X_train_dirty`. For each row in the solution artifact `True` means that the corresponding row in `X_train_dirty` should be cleaned, and `False` means that the row will be left dirty. The proportion of selected rows  must be less than or equal to the `budget` attribute of the problem. For problem `p001` the budget is `0.1`. We can select the first `50` rows and produce a solution.<jupyter_code>idx_selected = [i < 50 for i in range(len(problem["X_train_dirty"]))]<jupyter_output><empty_output><jupyter_text>Then, we can produce a solution:<jupyter_code>solution = problem.solve(idx_selected=idx_selected)<jupyter_output>/home/bojan/projects/phd/dcbench/code/dcbench/dcbench/tasks/budgetclean/problem.py:106: UserWarning: The number of selected data examples is below the allowed budget. (expected: 115, found: 50)
  warnings.warn(<jupyter_text>The `solve` method performs some rudimentary checks to see if our solution is valid. We got a warning that our number of selected rows was below the budget. Since no exception was raised, we still got a valid solution.Now we can evaluate it:<jupyter_code>problem.evaluate(solution)<jupyter_output><empty_output><jupyter_text>The result contains many attributes, which are inherited from the problem itself. The most interesting ones are the actual validation and test scores: `acc_val_gapclosed` and `acc_test_gapclosed`. They represent the percentage of the accuracy gap between fully dirty and fully clean data that was closed by our cleaning procedure.Formally, the accuracy gap is defined as:$$\mathrm{acc\_gap} = \frac{\mathrm{acc\_clean} - \mathrm{acc\_budgetcleaned}}{\mathrm{acc\_clean} - \mathrm{acc\_dirty}}$$  Using baseline methodsWe aim to provide some baseline methods for all our tasks. They can be listed as such:<jupyter_code>budgetclean.baselines<jupyter_output><empty_output><jupyter_text>We can simply call the baseline method by passing a problem instance to it and it will return a solution instance:<jupyter_code>baseline = budgetclean.baselines.random_clean
solution = baseline(problem)
problem.evaluate(solution)<jupyter_output><empty_output><jupyter_text>Solving and evaluating a batch of problemsWe usually want to solve a series of problems in the benchmark. We can use the baseline to solve all of them. We can focus primarily on the `Supreme` dataset, `MAR` mode of data missingness, and `logreg` model (i.e. logistic regression).<jupyter_code>results = []
for problem_id, problem in budgetclean.problems.items():
    if problem.attributes["dataset"] == "Supreme" and \
        problem.attributes["mode"] == "MAR" and \
        problem.attributes["model"] == "logreg":
        
        solution = baseline(problem)
        results.append(problem.evaluate(solution))<jupyter_output><empty_output><jupyter_text>Now we can concatenate results into a single data frame:<jupyter_code>import pandas as pd

result = pd.concat(results, axis=1).T
result<jupyter_output><empty_output><jupyter_text>We can also easily plot the results like this. For example, we can plot the gap closed as a function of the budget:<jupyter_code>_ = result.plot(x="budget", y=["acc_test_gapclosed", "acc_val_gapclosed"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08654050605996173, 'fraction_numerical': 0.011907293217095471, 'mean_word_length': 6.281733746130031, 'pattern_counts': {'":': 1, '<': 53, '<?xml version=': 0, '>': 52, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '368358', 'license': 'Apache-2.0', 'n_tokens_mistral': 1464, 'n_tokens_neox': 1371, 'n_words': 475, 'path': 'notebooks/dcbench_budgetclean.ipynb', 'repo_name': 'data-centric-ai/dcbench', 'size': '4703'}