<jupyter_start><jupyter_text><NAME>, 04.04.22  Introduction This notebook prepares the data sets we'll use as we go through the various components of TensorFlow Extended.  Data sets* A simplified version of the PetFinder.my data set from the Kaggle competition https://www.kaggle.com/c/petfinder-adoption-prediction. The simplified version was created by the TensorFlow team and used in their tutorials (e.g., https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers). * The full PetFinder.my data set from Kaggle.* _More TBA_   Setup<jupyter_code>%matplotlib inline
import pandas as pd, numpy as np, urllib.request, shutil, os
from pathlib import Path
# Check whether we're running on Colab
try:
    import colab
    colab=True
except:
    colab=False
if not colab:
    # Store small files in the repo:
    NB_DIR = Path.cwd()
    LOCAL_DATA = NB_DIR/'..'/'data' 
    PETFINDER_MINI = LOCAL_DATA/'petfinder-mini'
    # Store larger files outside the repo:
    DATA = Path('/home/alex/data/dat255')
elif colab:
    from google.colab import drive
    drive.mount('./gdrive')
    DATA = Path('./gdrive/MyDrive/ColabData')
    LOCAL_DATA = DATA
    PETFINDER_MINI = DATA/'petfinder-mini'
    
DATA.mkdir(exist_ok=True)<jupyter_output><empty_output><jupyter_text>Download PetFinder-mini<jupyter_code>petfinder_url = 'http://storage.googleapis.com/download.tensorflow.org/data/petfinder-mini.zip'
if not os.path.isfile(PETFINDER_MINI/'petfinder-mini.csv'): 

    _ = urllib.request.urlretrieve(petfinder_url, filename=LOCAL_DATA/'petfinder-mini.zip')

    shutil.unpack_archive(LOCAL_DATA/'petfinder-mini.zip', extract_dir=LOCAL_DATA)
pd.read_csv(PETFINDER_MINI/'petfinder-mini.csv').head()<jupyter_output><empty_output><jupyter_text>Move the CSV file to a separate subdirectory:<jupyter_code>(PETFINDER_MINI/'csv').mkdir(exist_ok=True)
(PETFINDER_MINI/'petfinder-mini.csv').rename(PETFINDER_MINI/'csv'/'petfinder-mini.csv')<jupyter_output><empty_output><jupyter_text>Split the data It will be useful to have our data stored on disk across multiple CSV files. Let's split it at random in two parts and save them to disk:<jupyter_code>df = pd.read_csv(PETFINDER_MINI/'csv'/'petfinder-mini.csv')
len(df)
# We split it in two parts at random

df1 = df.sample(frac=0.8)

df2 = df.drop(df1.index)
len(df1), len(df2)<jupyter_output><empty_output><jupyter_text>Create some new data Later, we'll have use for some data that's in some meaningful way different from what's in PetFinder.<jupyter_code>df3 = df1.sample(n=6, random_state=42)
df3.reset_index(inplace=True)
df3.drop('index', axis=1, inplace=True)
len(df3)
df3
df3.at[0, "Type"] = "Bird"
df3.at[1, "Health"] = 99
df3.at[3, "Age"] = -1
df3.head()<jupyter_output><empty_output><jupyter_text>Save to disk<jupyter_code>(PETFINDER_MINI/'split_csv').mkdir(exist_ok=True)
df1.to_csv(PETFINDER_MINI/'split_csv'/'span1.csv', index=None)
df2.to_csv(PETFINDER_MINI/'split_csv'/'span2.csv', index=None)
df3.to_csv(PETFINDER_MINI/'split_csv'/'span3.csv', index=None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12442547603414314, 'fraction_numerical': 0.013788575180564675, 'mean_word_length': 5.6383442265795205, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '415024', 'license': 'MIT', 'n_tokens_mistral': 1156, 'n_tokens_neox': 1104, 'n_words': 236, 'path': 'Module-9-2-TensorFlow_Extended/nbs/0.0-prepare_data.ipynb', 'repo_name': 'alu042/DAT255-2022', 'size': '3063'}