<jupyter_start><jupyter_text>JAI - Trust your data Sanity: checking if your data is validThis is an example of how to use the sanity check capabilities of JAI.In this notebook we will use a subset of the [PC Games 2020](https://www.kaggle.com/jesneuman/pc-games) dataset to check if the `Name`, `Genres` and `Indie` information are valid.You can install JAI in your environment using `pip install jai-sdk`.And you can read the docs [here](https://jai-sdk.readthedocs.io/en/stable/)!If you have any comments or suggestions, feel free to contact us: <EMAIL>*The object of life is not to be on the side of the majority, but to escape finding oneself in the ranks of the insane.* - <NAME><jupyter_code># JAI imports
from jai import Jai
from jai.processing import process_predict

# I/O imports
import pandas as pd<jupyter_output><empty_output><jupyter_text>Reading data<jupyter_code># it might take a few seconds to download this dataset (10MB) to your computer
DATASET_URL = "https://jaipresentation.blob.core.windows.net/data/games_jai.parquet"
df_games = pd.read_parquet(DATASET_URL)<jupyter_output><empty_output><jupyter_text>Use columns `id`, `Name`, `Genres` and `Indie`Why? As a rule of thumb, we should send data that us humans would normally use to actually check the validity of each value. In this sense, columns Name, Genres and Indie represent a fair subset that are prone to errors when filling information in.<jupyter_code>df_sanity = df_games[["id", "Name", "Genres", "Indie"]]<jupyter_output><empty_output><jupyter_text>Now we can use JAI to check the sanity of this data!<jupyter_code>j = Jai("YOUR_AUTH_KEY")<jupyter_output><empty_output><jupyter_text>We call `sanity` passing a given `name` for the database, the `data` itself and the fraction (`frac`) of the data to be appended to the dataframe as invalid values (by shuffling them).<jupyter_code>db_name = "games_sanity2"
frac = 0.25
results = j.sanity(name=db_name,
                 data=df_sanity,
                 frac=frac,
                 columns_ref=["Indie"])<jupyter_output>Insert Data: 100%|██████████| 1/1 [00:01<00:00,  1.44s/it]
Recognized setup args:
hyperparams: {'nt': 1000.0}
JAI is working: 100%|██████████| 11/11 [05:50<00:00, 31.87s/it]
Insert Data: 100%|██████████| 1/1 [00:01<00:00,  1.37s/it]
Recognized setup args:
hyperparams: {'nt': 1000.0}
JAI is working: 100%|██████████| 11/11 [00:21<00:00,  1.95s/it]
Insert Data: 100%|██████████| 1/1 [00:01<00:00,  1.74s/it]
Recognized setup args:
mycelia_bases: [{'id_name': 'id_Name', 'db_parent': 'games_sanity2_name'}, {'id_name': 'id_Name', 'db_parent': 'games_sanity2_name'}, {'id_name': 'id_Genres', 'db_parent': 'games_sanity2_genres'}, {'id_name': 'id_Genres', 'db_parent': 'games_sanity2_genres'}]
label: {'task': 'metric_classification', 'label_name': 'is_valid'}
split: {'type': 'stratified', 'split_column': 'is_valid', 'test_size': 0.2}
JAI is working: 100%|██████████| 18/18 [01:36<00:00,  5.38s/it]
Predict: 100%|██████████| 1/1 [00:06<00:00,  6.99s/it]<jupyter_text>Finally, we process the results...<jupyter_code>processed = process_predict(results)
df_result = pd.DataFrame(processed).sort_values('id')
df_result<jupyter_output><empty_output><jupyter_text>...and check how many entries were valid<jupyter_code>df_result["predict"].value_counts()<jupyter_output><empty_output><jupyter_text>This output indicates that there are a little over 3% of samples that have some kind of inconsistency. We can check which ones were flaged as Invalid:<jupyter_code>df_games.loc[df_result["predict"] == "Invalid"]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14540887524420876, 'fraction_numerical': 0.040747976555958694, 'mean_word_length': 5.972762645914397, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 34, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '104866', 'license': 'MIT', 'n_tokens_mistral': 1417, 'n_tokens_neox': 1262, 'n_words': 369, 'path': 'examples/sanity_example.ipynb', 'repo_name': 'jquant/mycelia-sdk', 'size': '3603'}