<jupyter_start><jupyter_text>Install basic requirements<jupyter_code>pip install -U whylogs pandas
import whylogs
import pandas as pd<jupyter_output><empty_output><jupyter_text>Load example data batchesThe example data is prepared from our public S3 bucket. You can use your own data if you want if you have multiple batches of data.<jupyter_code>pdfs = []
for i in range(1, 8):
    path = f"https://whylabs-public.s3.us-west-2.amazonaws.com/demo_batches/input_batch_{i}.csv"
    print(f"Loading data from {path}")
    df = pd.read_csv(path)
    pdfs.append(df)
pdfs[0].describe()<jupyter_output><empty_output><jupyter_text>Configure whylogs`whylogs`, by default, does not send statistics to WhyLabs.There are a few small steps you need to set up. If you haven't got the access key, please onboard with WhyLabs.**WhyLabs only requires whylogs API - your raw data never leaves your premise.**<jupyter_code>from whylogs.app import Session
from whylogs.app.writers import WhyLabsWriter
import os
import datetime
import getpass

# set your org-id here
print("Enter your WhyLabs Org ID")
os.environ["WHYLABS_DEFAULT_ORG_ID"] = input()
# set your API key here
print("Enter your WhyLabs API key")
os.environ["WHYLABS_API_KEY"] = getpass.getpass()
print("Using API Key ID: ", os.environ["WHYLABS_API_KEY"][0:10])<jupyter_output>Enter your WhyLabs Org ID<jupyter_text>Creating sessionOnce the environments are set, let's create a whylogs session with a WhyLabs writer.Note that you can add your local writer or S3 writer if you want here. Check out the API docs for more information.<jupyter_code># create WhyLabs session
writer = WhyLabsWriter("", formats=[])
session = Session(project="demo-project", pipeline="demo-pipeline", writers=[writer])<jupyter_output><empty_output><jupyter_text>Logging to WhyLabsEnsure you have a **model ID** (also called **dataset ID**) before you start! Dataset Timestamp* To avoid confusion, it's recommended that you use UTC* If you don't set `dataset_timestamp` parameter, it'll default to `UTC` now* WhyLabs supports real time visualization when the timestamp is **within the last 7 days**. Anything older than than will be picked up when we run our batch processing* **If you log two profiles for the same day with different timestamps (12:00 vs 12:01), they are merged to the same batch** Logging Different Batches of Data* We'll give the profiles different **dates*** Create a new logger for each date. Note that the logger needs to be `closed` to flush out the data<jupyter_code>print("Enter your model ID from WhyLabs:")
model_id = input()
for i, df in enumerate(pdfs):
    # walking backwards. Each dataset has to map to a date to show up as a different batch
    # in WhyLabs
    dt = datetime.datetime.now(tz=datetime.timezone.utc) - datetime.timedelta(days=i)
    
    # Create new logger for date
    with session.logger(tags={"datasetId": model_id}, dataset_timestamp=dt) as ylog:
        print("Log data frame for ", dt)
        ylog.log_dataframe(df)
# Ensure everything is flushed
session.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09064232148679491, 'fraction_numerical': 0.006194978806651451, 'mean_word_length': 5.299794661190965, 'pattern_counts': {'":': 1, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '32840', 'license': 'Apache-2.0', 'n_tokens_mistral': 941, 'n_tokens_neox': 926, 'n_words': 361, 'path': 'python/WhyLabs_Platform.ipynb', 'repo_name': 'FelipeAdachi/whylogs-examples', 'size': '3067'}