<jupyter_start><jupyter_text>IntroductionIn this notebook, we will explore the data from Petfinder.my that we'll use to predict adoption speed for individual pet listings. The layout of the notebook is as follows:1. Data contents overview2. 3. A summary of notable EDA findings and thoughts on modeling implications.  Data overviewWe have access to four different types of data. All have been split into training and test sets.1. Information on each adoption listing (tabular, one row per listing)2. Images that were posted with each listing (JPG files)3. Corresponding image metadata from the [Google Vision API](https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate) (JSON files)4. Sentiment analysis applied to each listing's description via the [Google Natural Language API](https://cloud.google.com/natural-language/docs/basics) (JSON files)Also provided were some supplementary data mapping IDs to text names for certain fields in the tabular data - namely breed IDs to breed names, color IDs to color names, and state IDs to names of states in Malaysia.  Which data are present for which listings?The documentation tells us that not all listings will have all types of data. For example, there are some descriptions that could not be analyzed by Google's API. We'd like to use all the information available to us, so it will be useful to know how many of the listings are missing certain data types, and whether the data types are missing at random or according to some pattern.<jupyter_code>def calc_data_info(datadir, split):
    if split not in ['train', 'test']:
        raise ValueError
    
    if split == 'train':
        tab = pd.read_csv(datadir / 'train.csv', index_col='PetID')
    else:
        # test.csv comes in an extra subfolder for some reason
        tab = pd.read_csv(datadir / 'test' / 'test.csv', index_col='PetID')
    
    images = (datadir / f'{split}_images').glob('*.jpg')
    metadata = (datadir / f'{split}_metadata').glob('*.json')
    sentiment = (datadir / f'{split}_sentiment').glob('*.json')
    
    image_cts = pd.Series(Counter(f.stem.split('-')[0] for f in images), 
                          name='Image count')
    metadata_cts = pd.Series(Counter(f.stem.split('-')[0] for f in metadata), 
                             name='Metadata count')
    sentiment_cts = pd.Series(Counter(f.stem for f in sentiment),
                              name='Sentiment count')
    
    data_info = tab[['PhotoAmt']].merge(image_cts, how='left', left_index=True, right_index=True)
    data_info = data_info.merge(metadata_cts, how='left', left_index=True, right_index=True)
    data_info = data_info.merge(sentiment_cts, how='left', left_index=True, right_index=True)
    data_info = data_info.fillna(0).astype(int)
    
    data_info['Has images'] = data_info['Image count'] > 0
    data_info['Has metadata'] = data_info['Metadata count'] > 0
    data_info['Has all metadata'] = data_info['Image count'] == data_info['Metadata count']
    data_info['Has sentiment'] = data_info['Sentiment count'] > 0
    
    return data_info
datadir = Path('../data')
train_data_info = calc_data_info(datadir, 'train')
test_data_info = calc_data_info(datadir, 'test')<jupyter_output><empty_output><jupyter_text>The table output by `calc_data_info()` contains information about the presence of data for each listing.* **`PhotoAmt`**: the number of photos for a listing is given as a feature in the tabular data. We keep it here to compare to the number of photos present in the image data. It's possible that a listing has images that were not supplied with the actual image data, and we want to know if that happens and how frequent it is.* **`Image count`**: the number of image files present for each listing. We expect this to be the same as `PhotoAmt`, but we aren't sure yet if that's the case.* **`Metadata count`**: the number of image metadata files present for each listing. This ought to be the same as `Image count`, but if there are images the Google API couldn't analyze for whatever reason, it may be less.* **`Sentiment count`**: the number of sentiment analyis files present for each listing. There is only one description per listing, so this value should only ever be 1 or 0.* **`Has images`**: `True` if there are images present for the listing, `False` otherwise* **`Has metadata`**: `True` if there is image metadata present for the listing, `False` otherwise* **`Has all metadata`**: `True` if a metadata file exists for each image, `False` if there are missing files* **`Has sentiment`**: `True` if sentiment analysis is present for the listing, `False` otherwise<jupyter_code>train_data_info.head()
test_data_info.head()<jupyter_output><empty_output><jupyter_text>Training dataLet's look at the training data first. First of all, there are 14,993 total listings.<jupyter_code>len(train_data_info)<jupyter_output><empty_output><jupyter_text>We have 58,311 images, and 97% of listings (14,652 of 14,993) contain at least one image.<jupyter_code>train_data_info['Image count'].sum()
train_data_info['Has images'].sum()<jupyter_output><empty_output><jupyter_text>Every listing has image metadata if and only if it has images.<jupyter_code>pd.crosstab(train_data_info['Has images'], train_data_info['Has metadata'], dropna=False)<jupyter_output><empty_output><jupyter_text>Every listing has the same number of metadata files as it does image files.<jupyter_code>train_data_info['Has all metadata'].value_counts()<jupyter_output><empty_output><jupyter_text>94% of the listings (14,111 of 14,993) have images, metadata, and sentiment analysis. Only 10 have none of the three.<jupyter_code>pd.crosstab(train_data_info['Has metadata'], train_data_info['Has sentiment'], dropna=False)<jupyter_output><empty_output><jupyter_text>The following figure is a heatmap of the True and False values in `train_data_info`. The listings are organized along the $x$ axis. If a listing does not have one of the three non-tabular data types, a thin vertical black bar appears. Notice that once again we see listings are either missing both images and metadata, or neither. There does not appear to be a pattern between listings that are missing sentiment analysis and listings that are missing image metadata.<jupyter_code>plt.figure(figsize=(16, 6))
cols = ['Has images', 'Has metadata', 'Has sentiment']
sns.heatmap(train_data_info[cols].sort_index().values.T,
            xticklabels=False,
            yticklabels=cols,
            cbar=False)
plt.yticks(rotation=0, va='center')
plt.ylabel(None)
plt.show()<jupyter_output><empty_output><jupyter_text>Test dataNext we'll inspect the test data. First of all, there are 3,972 total listings.<jupyter_code>len(test_data_info)<jupyter_output><empty_output><jupyter_text>We have 14,465 images, and 97% of listings (3,858 of 3,972) contain at least one image.<jupyter_code>test_data_info['Image count'].sum()
test_data_info['Has images'].sum()<jupyter_output><empty_output><jupyter_text>Every listing has image metadata if and only if it has images.<jupyter_code>pd.crosstab(test_data_info['Has images'], test_data_info['Has metadata'], dropna=False)<jupyter_output><empty_output><jupyter_text>Every listing has the same number of metadata files as it does image files.<jupyter_code>test_data_info['Has all metadata'].value_counts()<jupyter_output><empty_output><jupyter_text>94% of the listings (3,755 of 3,972) have images, metadata, and sentiment analysis. Only 4 have none of the three.<jupyter_code>pd.crosstab(test_data_info['Has metadata'], test_data_info['Has sentiment'], dropna=False)<jupyter_output><empty_output><jupyter_text>The following figure is a heatmap of the True and False values in `test_data_info`. The listings are organized along the $x$ axis. If a listing does not have one of the three non-tabular data types, a thin vertical black bar appears. Notice that once again we see listings are either missing both images and metadata, or neither. There does not appear to be a pattern between listings that are missing sentiment analysis and listings that are missing image metadata.<jupyter_code>plt.figure(figsize=(16, 6))
cols = ['Has images', 'Has metadata', 'Has sentiment']
sns.heatmap(test_data_info[cols].sort_index().values.T,
            xticklabels=False,
            yticklabels=cols,
            cbar=False)
plt.yticks(rotation=0, va='center')
plt.ylabel(None)
plt.show()<jupyter_output><empty_output><jupyter_text>Overview of tabular dataThe tabular data contains `AdoptionSpeed`, the attribute we are trying to predict. It is a categorical variable with five levels relating to how long it took for the animal to be adopted once it was listed on Petfinder.**0**: Adopted in under 1 day **1**: Adopted in 1 to 7 days **2**: Adopted in 8 to 30 days **3**: Adopted in 31 to 90 days **4**: Adopted in at least 91 days There are lots of categorical variables in the data. Most are already encoded as integers - for example, the `Type` variable is 1 for a dog and 2 for a cat. This is a convenient form for training a model, but it doesn't much lend itself to EDA. So we'll make a copy of the data, replace the uninformative integers with more descriptive category labels, and use that version to make tables and plots.  Relabeling categorical features<jupyter_code>train = pd.read_csv('../data/train.csv', index_col='PetID')
breed_labels = pd.read_csv('../data/breed_labels.csv', index_col='BreedID', usecols=['BreedID', 'BreedName'])
color_labels = pd.read_csv('../data/color_labels.csv', index_col='ColorID')
state_labels = pd.read_csv('../data/state_labels.csv', index_col='StateID')
category_renaming = {
    'Type': {
        1: 'Dog',
        2: 'Cat'
    },
    'Gender': {
        1: 'Male',
        2: 'Female',
        3: 'Mixed'
    },
    'MaturitySize': {
        1: 'Small',
        2: 'Medium',
        3: 'Large',
        4: 'Extra Large',
        0: 'Not Specified'
    },
    'FurLength': {
        1: 'Short',
        2: 'Medium',
        3: 'Long',
        0: 'Not Specified'
    },
    'Vaccinated': {
        1: 'Yes',
        2: 'No',
        3: 'Not Sure'
    },
    'Dewormed': {
        1: 'Yes',
        2: 'No',
        3: 'Not Sure'
    },
    'Sterilized': {
        1: 'Yes',
        2: 'No',
        3: 'Not Sure'
    },
    'Health': {
        1: 'Healthy',
        2: 'Minor Injury',
        3: 'Serious Injury',
        0: 'Not Specified'
    },
    **state_labels.to_dict(), 
    **breed_labels.to_dict(), 
    **color_labels.to_dict()
}
# Zeros are not given in the supplementary data, so manually replace them with "none"
category_renaming['BreedName'][0] = 'None specified'
category_renaming['ColorName'][0] = 'None specified'
# Make sure the column names in the training data are reflected in the dictionary
category_renaming['Breed2'] = category_renaming['Breed1'] = category_renaming['BreedName']
category_renaming['Color3'] = category_renaming['Color2'] = category_renaming['Color1'] = category_renaming['ColorName']
# Don't need these for the actual renaming
del category_renaming['BreedName']
del category_renaming['ColorName']
train_eda = train.copy()
train_eda = train_eda.replace(category_renaming)
train_eda.head()<jupyter_output><empty_output><jupyter_text>[This Kaggle kernel](https://www.kaggle.com/artgor/exploration-of-data-step-by-step) has a nice way to visualize the response variable split out by categories, and I've used a slightly modified version here.<jupyter_code>def prepare_diff_values(data, x, hue, groupby):
    if groupby not in ['hue', 'x']:
        raise ValueError
        
    total_pcts = data[hue].value_counts(normalize=True).sort_index()
    unique_levels = data[x].unique()
    diff_series = []
    
    flatten_order = 'C' if groupby == 'hue' else 'F'
    
    for level in unique_levels:
        level_pcts = data.loc[data[x] == level,hue].value_counts(normalize=True).sort_index()
        level_diffs = (level_pcts / total_pcts - 1) * 100
        level_diffs.name = level
        diff_series.append(level_diffs)
    
    diffs_df = pd.concat(diff_series, axis=1)
    diffs = diffs_df.values.flatten(flatten_order)
    
    return diffs

def format_text(d):
    if np.isnan(d):
        return '---%'
    if d <= 0:
        return f'{d:.1f}%'
    else:
        return f'+{d:.1f}%'

def text_color(d):
    if d <= 0:
        return 'red'
    else:
        return 'green'

def countplot(data, x, hue, groupby='hue', figsize=None, **kwargs):
    if groupby not in ['hue', 'x']:
        raise ValueError
        
    plt.figure(figsize=figsize)
    
    diff_values = prepare_diff_values(data, x, hue, groupby)
    
    if groupby == 'hue':
        g = sns.countplot(x=x, data=data, hue=hue, **kwargs)
        plt.title(f'Counts of {x} by {hue}')
    else:
        g = sns.countplot(x=hue, data=data, hue=x, **kwargs)
        plt.title(f'Counts of {hue} by {x}')
    
    ax = g.axes
    
    for p, d in zip(ax.patches, diff_values):
        height = p.get_height() if str(p.get_height()) != 'nan' else 0
        text = format_text(d)
        color = text_color(d)
        ax.annotate(text,
                    (p.get_x() + p.get_width() / 2, height),
                    ha='center',
                    va='center',
                    fontsize=10,
                    color=color,
                    xytext=(0, 8),
                    textcoords='offset points')<jupyter_output><empty_output><jupyter_text>Essentially a wrapper around [Seaborn's `countplot`](https://seaborn.pydata.org/generated/seaborn.countplot.html), this version of `countplot()` annotates the figure with the percent change in adoption speed rates by category versus the overall rate.If we were to pick a listing entirely at random, there would be a 2.7% chance that pet was adopted on the same day (410 listings with `AdoptionSpeed` 0 out of 14,993 total listings). If we instead picked only from the cat listings, that chance jumps to 3.5% (240 listings with `AdoptionSpeed` 0 out of 6,861 total cat listings). That's about a 28% increase. By calculating those increases and annotating each bar with them, we can quickly tell what features of the data appear to influence `AdoptionSpeed`. In the plot below, we see that while there are fewer cats in the data set overall, they tend to be adopted faster than dogs.<jupyter_code>countplot(train_eda, 'Type', 'AdoptionSpeed', groupby='hue', figsize=(16, 8), color='steelblue')<jupyter_output><empty_output><jupyter_text>Missing valuesThe data are remarkably complete. The only features with missing data are `Name` and `Description`. `Name` is unlikely to be helpful for prediction at all, and `Description` is already being taken into account with the sentiment analysis, so it seems safe to drop these features.<jupyter_code>train_eda.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 14993 entries, 86e1089a3 to a83d95ead
Data columns (total 23 columns):
Type             14993 non-null object
Name             13736 non-null object
Age              14993 non-null int64
Breed1           14993 non-null object
Breed2           14993 non-null object
Gender           14993 non-null object
Color1           14993 non-null object
Color2           14993 non-null object
Color3           14993 non-null object
MaturitySize     14993 non-null object
FurLength        14993 non-null object
Vaccinated       14993 non-null object
Dewormed         14993 non-null object
Sterilized       14993 non-null object
Health           14993 non-null object
Quantity         14993 non-null int64
Fee              14993 non-null int64
State            14993 non-null int64
RescuerID        14993 non-null object
VideoAmt         14993 non-null int64
Description      14981 non-null object
PhotoAmt         14993 non-null float64
AdoptionSpeed    14993 non-nul[...]<jupyter_text>Overview of sentiment analysis<jupyter_code>import json
from tqdm import tqdm_notebook as tqdm
sentiment_files = list((datadir / 'train_sentiment').glob('*.json'))
sentiments = []
for sentiment_file in tqdm(sentiment_files):
    with open(sentiment_file) as f:
        data = json.load(f)
        sentiments.append(data)
pd.DataFrame(sentiments)
sentiments[0]['sentences']
sentiments[0]['entities']
pd.DataFrame(sentiments[0]['entities'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10368366029149499, 'fraction_numerical': 0.020724432691716376, 'mean_word_length': 3.8834834834834835, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 80, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '399520', 'license': 'MIT', 'n_tokens_mistral': 5163, 'n_tokens_neox': 4856, 'n_words': 1706, 'path': 'petfinder-adoption-prediction/notebooks/EDA.ipynb', 'repo_name': 'ensley/kaggle', 'size': '16261'}