<jupyter_start><jupyter_text>Saving Profiles to S3 ---<jupyter_code>from whylogs import get_or_create_session
import pandas as pd

%load_ext autoreload
%autoreload 2<jupyter_output><empty_output><jupyter_text>Create a mock s3 server  For this example we will create a fake s3 server using moto lib. You should remove this section if you have you own bucket setup on aws. Make sure you have your aws configuration set.  By default this mock server creates a server in region "us-east-1"<jupyter_code>BUCKET="super_awesome_bucket"
from moto import mock_s3
from moto.s3.responses import DEFAULT_REGION_NAME
import boto3

mocks3 = mock_s3()
mocks3.start()
res = boto3.resource('s3', region_name=DEFAULT_REGION_NAME)
res.create_bucket(Bucket=BUCKET)<jupyter_output><empty_output><jupyter_text>Load Data We can go by our usual way, load a example csv data<jupyter_code>df = pd.read_csv("data/lending_club_1000.csv")<jupyter_output><empty_output><jupyter_text>Config File Example---Seting up  whylogs to  save your data on s3 can be in several ways. Simplest is to simply create a config file,where each data format can be saved to a specific location. As shown below<jupyter_code>CONFIG = """
project: s3_example_project
pipeline: latest_results
verbose: false
writers:
- formats:
  - protobuf
  output_path: s3://super_awesome_bucket/
  path_template: $name/dataset_summary
  filename_template: dataset_summary
  type: s3
- formats:
  - flat
  output_path: s3://super_awesome_bucket/
  path_template: $name/dataset_summary
  filename_template: dataset_summary
  type: s3
- formats:
  - json
  output_path: s3://super_awesome_bucket/
  path_template: $name/dataset_summary
  filename_template: dataset_summary
  type: s3
"""
config_path=".whylogs_s3.yaml"
with open(".whylogs_s3.yaml","w") as file:
    file.write(CONFIG)<jupyter_output><empty_output><jupyter_text>Checking the content:<jupyter_code>%cat .whylogs_s3.yaml<jupyter_output>project: s3_example_project
pipeline: latest_results
verbose: false
writers:
- formats:
  - protobuf
  output_path: s3://super_awesome_bucket/
  path_template: $name/dataset_summary
  filename_template: dataset_summary
  type: s3
- formats:
  - flat
  output_path: s3://super_awesome_bucket/
  path_template: $name/dataset_summary
  filename_template: dataset_summary
  type: s3
- formats:
  - json
  output_path: s3://super_awesome_bucket/
  path_template: $name/dataset_summary
  filename_template: dataset_summary
  type: s3<jupyter_text>If you have a custom name for your config file or place it in a special location you can use the helper function<jupyter_code>from whylogs.app.session import load_config, session_from_config
config = load_config(".whylogs_s3.yaml")
session = session_from_config(config)
print(session.get_config().to_yaml())<jupyter_output>metadata: null
pipeline: latest_results
project: s3_example_project
verbose: false
with_rotation_time: null
writers:
- filename_template: <string.Template object at 0x14e917fa0>
  formats:
  - OutputFormat.protobuf
  output_path: s3://super_awesome_bucket/
  path_template: <string.Template object at 0x14d5a1c70>
- filename_template: <string.Template object at 0x14e9683d0>
  formats:
  - OutputFormat.flat
  output_path: s3://super_awesome_bucket/
  path_template: <string.Template object at 0x14e968220>
- filename_template: <string.Template object at 0x14e968580>
  formats:
  - OutputFormat.json
  output_path: s3://super_awesome_bucket/
  path_template: <string.Template object at 0x14e968490><jupyter_text>Otherwise if the file is located in your home directory or current location you are running, you can simply run `get_or_create_session()`<jupyter_code>session= get_or_create_session()
print(session.get_config().to_yaml())<jupyter_output>metadata: null
pipeline: latest_results
project: s3_example_project
verbose: false
with_rotation_time: null
writers:
- filename_template: <string.Template object at 0x14e917d30>
  formats:
  - OutputFormat.protobuf
  output_path: s3://super_awesome_bucket/
  path_template: <string.Template object at 0x14e917c10>
- filename_template: <string.Template object at 0x14e968d00>
  formats:
  - OutputFormat.flat
  output_path: s3://super_awesome_bucket/
  path_template: <string.Template object at 0x14e968e20>
- filename_template: <string.Template object at 0x14e968e80>
  formats:
  - OutputFormat.json
  output_path: s3://super_awesome_bucket/
  path_template: <string.Template object at 0x14e968fa0><jupyter_text>Loggin Data --- The data can be save by simply closing a logger, or one a logger is out of scope.<jupyter_code>with session.logger("dataset_test_s3") as logger:
    logger.log_dataframe(df)

client = boto3.client('s3')
objects = client.list_objects(Bucket=BUCKET)
[obj["Key"] for obj in objects.get("Contents",[])]<jupyter_output><empty_output><jupyter_text>You can define the configure for were the data is save through a configuration file or creating a custom writer.<jupyter_code>mocks3.stop()<jupyter_output><empty_output><jupyter_text>Without Config File---<jupyter_code>mocks3.start()
res = boto3.resource('s3', region_name=DEFAULT_REGION_NAME)
res.create_bucket(Bucket=BUCKET)
from whylogs.app.session import load_config, session_from_config
from whylogs.app.config import WriterConfig, SessionConfig

s3_writer_config= WriterConfig(type="s3",formats=["json","flat","protobuf"],
             output_path="s3://super_awesome_bucket/",
             path_template="$name/dataset_summary",
             filename_template="dataset_profile")

#you can also create a local, so you have a local version of the data. 


session_config=SessionConfig(project="my_super_duper_project_name",
                             pipeline="latest_results",
                              writers=[s3_writer_config])

session = session_from_config(session_config)
print(session.get_config().to_yaml())
with session.logger("dataset_test_s3_config_as_code") as logger:
    logger.log_dataframe(df)
client = boto3.client('s3')
objects = client.list_objects(Bucket=BUCKET)
[obj["Key"] for obj in objects.get("Contents",[])]<jupyter_output><empty_output><jupyter_text>Close mock s3 server<jupyter_code>mocks3.stop()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09156859584072223, 'fraction_numerical': 0.025149121392874417, 'mean_word_length': 5.692556634304207, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '150536', 'license': 'Apache-2.0', 'n_tokens_mistral': 2111, 'n_tokens_neox': 2062, 'n_words': 482, 'path': 'examples/S3 example.ipynb', 'repo_name': 'BearerPipelineTest/whylogs', 'size': '6203'}