<jupyter_start><jupyter_text>Exploring Cape Python with Pandas: IoT DatasetThis Jupyter Notebook was presented as part of [Coiled's Science Thursdays](https://coiled.io/blog/data-privacy-distributed-compute.html). To follow along, you will need to [download the example dataset](https://capeprivacy.com/example-dataset/) and put it in a relative folder called `data` (or update the file path below). You will also need to [download the policy file]() and put it in a relative folder called `policy` or ensure you have Cape Python installed locally and change the path to use the copy in the `examples` folder.For more information on Cape Python, please check out our [documentation](https://docs.capeprivacy.com/libraries/cape-python/).<jupyter_code>import pandas as pd
import cape_privacy as cape
import cape_privacy.pandas.transformations as pandas_transformations

%matplotlib inline
df = pd.read_csv('../data/iot_example.csv')
df.head()
df.username.value_counts().hist(bins=30)<jupyter_output><empty_output><jupyter_text>Based off an initial look, the columns I am most concerned about are timestamp and username. We want to obfuscate those but leave some information / utility so we can still do our job as data scientists!<jupyter_code>pandas_transformations.
pandas_transformations.DateTruncation?
df.dtypes<jupyter_output><empty_output><jupyter_text>I need to make sure this is the proper dtype! :)<jupyter_code>df['timestamp'] = pd.to_datetime(df['timestamp'])
df['timestamp']
truncate_date = pandas_transformations.DateTruncation(frequency="day")
truncate_date(df['timestamp'])
df['timestamp'] = truncate_date(df['timestamp'])
df.head()<jupyter_output><empty_output><jupyter_text>I'm concerned that the Username column should also be protected<jupyter_code>pandas_transformations.Tokenizer?
tokenize_username = pandas_transformations.Tokenizer(key='super_secret_please_dont_use_this', max_token_len=20)
tokenize_username(df['username']).value_counts()
df['username'] = tokenize_username(df['username'])<jupyter_output><empty_output><jupyter_text>Now it's time to save this dataset and share for larger data science tasks<jupyter_code>df.to_csv('../data/caped_iot_example.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10018050541516245, 'fraction_numerical': 0.0018050541516245488, 'mean_word_length': 7.694117647058824, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '366893', 'license': 'Apache-2.0', 'n_tokens_mistral': 692, 'n_tokens_neox': 672, 'n_words': 191, 'path': 'examples/notebooks/Cape Python with Pandas - IoT Exploratory Data Analysis.ipynb', 'repo_name': 'vismaya-Kalaiselvan/cape-python', 'size': '2216'}