<jupyter_start><jupyter_text>Step 5: Data Wrangling & Exploration Dataset: DeepFashion MLE Capstone: Outfit Recommender - Spring 2021 By: <NAME>  GH: [@mmbazel](https://github.com/MMBazel)  In this notebook, I'll be exploring one of three datasets (DeepFashion, Paperdoll, and iMaterialist). Specifically in this notebook we'll: ☑️ Cleaning the data☑️ Dealing with missing	values☑️ Handling outliers☑️ Dealing with class imbalance  Explanation of the data, according to the dataset page here: http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion.html Files/Folders:* Anno_coarse (16.6 MB)/        * list_attr_cloth.txt        * list_attr_img.txt        * list_bbox.txt        * list_category_cloth.txt        * list_category_img.txt        * list_landmarks.txt        * Anno_fine/        * (see all files above)        * train_attr.txt, train_bbox.txt, train_cate.txt, train_landmarks.txt, train.txt        * test_attr.txt, test_bbox.txt, test_cate.txt, test_landmarks.txt, test.txt        * val_attr.txt, val_bbox.txt, val_cate.txt, val_landmarks.txt, val.txt* Eval/        * list_eval_partition.txt* Img/        * img_highres.zip (28 GB)        * img.zip (2 GB) ⚠️📝 Notes (About the Notebooks) ⚠️📝 My guiding principles:* ➡️ Be overly communicative = While that leads to verbose commenting, I hope that means I catch a bunch of questions early)  * ➡️ Human-readable over witty-optimization = For the most part I try to make everything I'm doing obvious* ➡️ Write as much code as needed, and no more = There's a time and place for error-catching & object-oriented code & there are ways to make the notebook reproducible. That's not quite the goal for this notebook (or any of the other notebooks in the early stages of the project) and my goal was to write just the code needed to get this step done.     Step 1: Proper set-up & installation of necessary libraries & packages<jupyter_code>#!pip3 install lmdb
#!pip3 install ipython-sql
#!brew install wget
##################### [TODO] SETUP #####################
# [TODO] Import any utilities functions


import json
import os
import sys
from sqlalchemy import create_engine

import io
import lmdb
import sqlite3
import pandas as pd
from PIL import Image

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

%matplotlib inline
%load_ext sql

print('Packages Imported')

modules = dir()

print(modules)
#print(os.environ)

# [TODO] Package install/load
# Confirm all the right libraries are present
# This is an important step because there's a good chance
# that for some pckaes where you use pip or pip3 install
# they could download to the wrong directory if you're not
# using the right pip executable

!conda list
# Confirm path of working directory
!pwd<jupyter_output>/Users/mikikobazeley/Github/personal/MMBazel/Wardrobe-Recommender/notebooks/Step2_EDA<jupyter_text>Step 2: Download the DeepFashion Dataset from GDrive  ⚠️ Best way is to download the Google Drive folder you want (in this case, "Category and Attribute Prediction Benchmark") as a zipped folder and then unzip to an external (as opposed to locally). The size will kill Jupyter locally and massively slow down you computer if you don't have enough memory.💡 Make sure to download the files manually -- Google Drive doesn't allow programmatically accessing the download link anymore: https://drive.google.com/drive/folders/0B7EVK8r0v71pWGplNFhjc01NbzQ?usp=sharing   Check Datasets Downloaded Properly<jupyter_code># Note: I chose to download everything to an external
# First check that we have a folder for the Category & Attribute datasets
# If not create using:

#!mkdir <path-to-DeepFashion-Folder-on-your-external>/Category_and_Attribute_Prediction_Benchmark
!ls /Volumes/MiniGator/Projects/Datasets/DeepFashion
# List files, marking whic ones are files vs folders

!ls -F /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark
# Check in the images folder

!ls -F /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Images<jupyter_output>[30m[43mimg[m[m/             [31mimg.zip[m[m*         [30m[43mimg_highres[m[m/     [31mimg_highres.zip[m[m*<jupyter_text>Unzip files<jupyter_code># Unzip all files ending in .zip 
# to DeepFashion/Category_and_Attribute_Prediction_Benchmark folder
# on external drive

#<Uncomment below>
#!unzip /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/\*.zip -d /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark
# Remove the .zip files

#<Uncomment below>
#!rm /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/*.zip
# Check existing files

!ls -F /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark
# Now we'll unzip the images

#<Uncomment below>
#!unzip /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Images/img.zip -d /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Images/
# Now we'll unzip the images

#<Uncomment below>
#!unzip /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Images/img_highres.zip -d /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Images/<jupyter_output><empty_output><jupyter_text>Let's confirm number of images 🧙‍♀️ 📸<jupyter_code># Let's get count number of folders of high res images
!ls /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Images/img_highres | wc -l
# Let's get count number of folders of regular res images
!ls /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Images/img | wc -l<jupyter_output>4807<jupyter_text>Step 3: Use CLI to start initial exploration 🔬    We want to understand:☑️ The data repository structure☑️ Number of files, types of files☑️ Type of information being captured & how it's represented  💡 Given the size of the dataset, doing initial inspection with shell is fast & easy.No need to break Jupyter before even getting started.      Let's get into the text files 📒 for the Anno_coarse repo<jupyter_code>!ls -F /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/<jupyter_output>[31mlist_attr_cloth.txt[m[m*     [31mlist_bbox.txt[m[m*           [31mlist_category_img.txt[m[m*
[31mlist_attr_img.txt[m[m*       [31mlist_category_cloth.txt[m[m* [31mlist_landmarks.txt[m[m*<jupyter_text>📙  File no.1: list_attr_cloth.txt   ---------------------------------------------------📝 Notes according to dataset README.txt:--------------- list_attr_cloth.txt --------------* First Row: number of attributes* Second Row: entry namesRest of the Rows: [attribute name] [attribute type]    Notes:1. In attribute type, "1" represents texture-related attributes, "2" represents fabric-related attributes, "3" represents shape-related attributes, "4" represents part-related attributes, "5" represents style-related attributes;2. The order of attribute labels accords with the order of attribute names;3. In attribute labels, "1" represents positive while "-1" represents negative, '0' represents unknown;4. Attribute prediction is treated as a multi-label tagging problem.<jupyter_code># Let preview file: list_attr_cloth.txt
!head -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_attr_cloth.txt
!tail -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_attr_cloth.txt<jupyter_output>zeppelin                     5
zig                          1
zigzag                       1
zip                          4
zip-front                    4
zip-pocket                   4
zip-up                       4
zipped                       4
zipper                       4
zippered                     4<jupyter_text>📙  File no.2: list_attr_img.txt   ---------------------------------------------------📝 Notes according to dataset README.txt:--------------- list_attr_img.txt --------------* First Row: number of images* Second Row: entry namesRest of the Rows: [image name] [attribute labels]    Notes:1. In attribute type, "1" represents texture-related attributes, "2" represents fabric-related attributes, "3" represents shape-related attributes, "4" represents part-related attributes, "5" represents style-related attributes;2. The order of attribute labels accords with the order of attribute names;3. In attribute labels, "1" represents positive while "-1" represents negative, '0' represents unknown;4. Attribute prediction is treated as a multi-label tagging problem.<jupyter_code># Let preview file: list_attr_img.tx
!head -n 3 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_attr_img.txt
!tail -n 3 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_attr_img.txt<jupyter_output>img/Paisley_Print_Babydoll_Dress/img_00000052.jpg                      -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1  1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1[...]<jupyter_text>📙  File no.3: list_category_cloth.txt  ---------------------------------------------------📝 Notes according to dataset README.txt:--------------- list_category_cloth.txt --------------* First Row: number of categories* Second Row: entry namesRest of the Rows: [category name] [category type]    Notes:1. In category type, "1" represents upper-body clothes, "2" represents lower-body clothes, "3" represents full-body clothes;2. The order of category labels accords with the order of category names;3. In category labels, the number represents the category id in category names;4. For the clothing categories, "Cape", "Nightdress", "Shirtdress" and "Sundress" have been merged into "Dress";5. Category prediction is treated as a 1-of-K classification problem.<jupyter_code># Let preview file: list_attr_cloth.txt
!head -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_category_cloth.txt
!tail -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_category_cloth.txt<jupyter_output>Dress          3
Jumpsuit       3
Kaftan         3
Kimono         3
Nightdress     3
Onesie         3
Robe           3
Romper         3
Shirtdress     3
Sundress       3<jupyter_text>📙  File no.4: list_category_img.txt  ---------------------------------------------------📝 Notes according to dataset README.txt:--------------- list_category_img.txt --------------* First Row: number of images* Second Row: entry namesRest of the Rows: [image name] [category label]    Notes:1. In category type, "1" represents upper-body clothes, "2" represents lower-body clothes, "3" represents full-body clothes;2. The order of category labels accords with the order of category names;3. In category labels, the number represents the category id in category names;4. For the clothing categories, "Cape", "Nightdress", "Shirtdress" and "Sundress" have been merged into "Dress";5. Category prediction is treated as a 1-of-K classification problem.<jupyter_code># Let preview file: list_attr_cloth.txt
!head -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_category_img.txt
!tail -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_category_img.txt<jupyter_output>img/Paisley_Print_Babydoll_Dress/img_00000045.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000046.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000047.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000048.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000049.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000050.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000051.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000052.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000053.jpg                      41
img/Paisley_Print_Babydoll_Dress/img_00000054.jpg                      41<jupyter_text>📙  File no.5: list_landmarks.txt  ---------------------------------------------------📝 Notes according to dataset README.txt:------------ list_landmarks.txt ------------First Row: number of imagesSecond Row: entry namesRest of the Rows: [image name] [clothes type] [variation type] [landmark visibility 1] [landmark location x_1] [landmark location y_1], ... [landmark visibility 8] [landmark location x_8] [landmark location y_8>]---------------------------------------------------Notes:1. The order of landmark labels accords with the order of entry names;2. In clothes type, "1" represents upper-body clothes, "2" represents lower-body clothes, "3" represents full-body clothes. Upper-body clothes possess six fahsion landmarks, lower-body clothes possess four fashion landmarks, full-body clothes possess eight fashion landmarks;3. In variation type, "1" represents normal pose, "2" represents medium pose, "3" represents large pose, "4" represents medium zoom-in, "5" represents large zoom-in;4. In landmark visibility state, "0" represents visible, "1" represents invisible/occluded, "2" represents truncated/cut-off;5. For upper-body clothes, landmark annotations are listed in the order of ["left collar", "right collar", "left sleeve", "right sleeve", "left hem", "right hem"]; For lower-body clothes, landmark annotations are listed in the order of ["left waistline", "right waistline", "left hem", "right hem"]; For upper-body clothes, landmark annotations are listed in the order of ["left collar", "right collar", "left sleeve", "right sleeve", "left waistline", "right waistline", "left hem", "right hem"].<jupyter_code># Let preview file: list_attr_cloth.txt
!head -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_landmarks.txt
!tail -n 10 /Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_landmarks.txt<jupyter_output>img/Paisley_Print_Babydoll_Dress/img_00000045.jpg                       3  1 061 051  1 095 054  0 029 116  0 127 119  1 042 115  0 101 116  0 072 186  0 108 189 
img/Paisley_Print_Babydoll_Dress/img_00000046.jpg                       3  1 118 073  0 180 074  1 118 112  0 177 109  0 107 165  0 189 161  0 098 278  0 222 264 
img/Paisley_Print_Babydoll_Dress/img_00000047.jpg                       3  0 083 030  0 125 043  0 070 028  0 135 048  0 054 110  0 134 119  0 032 225  0 169 227 
img/Paisley_Print_Babydoll_Dress/img_00000048.jpg                       3  0 099 102  0 130 105  0 055 135  0 159 147  0 079 181  0 151 175  0 088 293  0 148 292 
img/Paisley_Print_Babydoll_Dress/img_00000049.jpg                       3  1 079 073  0 131 077  0 046 183  0 166 185  0 066 166  0 138 164  0 051 268  0 161 261 
img/Paisley_Print_Babydoll_Dress/img_00000050.jpg                       3  0 101 026  1 185 028  0 062 123  0 223 104  0 098 150  0 197 144  2 000 000  2 000 000 
img/Paisley_Prin[...]<jupyter_text>Let's create 🐼 dataframes & start merging & analyzing     📈 DF no.1: Images + Attributes Dataframe<jupyter_code>#create & preview df of attrbiute names & type label
list_attr_cloth_df = pd.read_table('/Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_attr_cloth.txt',
                                   sep='\s{2,}', 
                                   header=0, 
                                   skiprows=[0], 
                                   engine='python')
list_attr_cloth_df
list_attr_img_df = pd.read_csv(
    '/Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_attr_img.txt', 
    delim_whitespace=True, 
    header=0,
    skiprows=[0],
    names=(['image_name'] + list_attr_cloth_df['attribute_name'].tolist()))
list_attr_img_df
# In attribute labels, "1" represents positive while "-1" represents negative, '0' represents unknown
# We'll replace 
list_attr_img_df.replace([-1,0,1],[False,False,True], inplace=True)
list_attr_img_df<jupyter_output><empty_output><jupyter_text>📈 DF no.2: Images + Categories Dataframe  ⚠️ One big issues: We're missing some categories of clothing.Clothing categories file contains 50 entries, whereas the image_category_cloth file is only showing 48. We're specifically missing 'shitdress' and 'sundress'. 📝 This can posible be explained by the note:"For the clothing categories, "Cape", "Nightdress", "Shirtdress" and "Sundress" have been merged into "Dress""<jupyter_code>#create & preview df of attrbiute names & type label
list_category_cloth_df = pd.read_table('/Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_category_cloth.txt',
                                   sep='\s{2,}', 
                                   header=0, 
                                   skiprows=[0], 
                                   engine='python')
list_category_cloth_df.head(10)
list_category_cloth_df.describe()
#create & preview df of attrbiute names & type label
list_category_img_df = pd.read_table('/Volumes/MiniGator/Projects/Datasets/DeepFashion/Category_and_Attribute_Prediction_Benchmark/Anno_coarse/list_category_img.txt',
                                   sep='\s{2,}', 
                                   header=0, 
                                   skiprows=[0], 
                                   engine='python')
list_category_img_df.head(10)
# Notice that the highest category goes to 48 only.

list_category_img_df['category_label'].describe()
list_category_img_df.loc[list_category_img_df['category_label']==1.0]
list_category_img_df.loc[list_category_img_df['category_label']==48.0]
(list_category_img_df
 .groupby('category_label')
 .count()
 .reset_index()
 .set_index('category_label')
 .plot(kind='barh', y='image_name',use_index=True)
)
# Create a column that has the actual category labels
list_category_cloth_df['actual_category_number'] = np.arange(1,len(list_category_cloth_df)+1)
img_categories_combined_df = list_category_img_df.merge(list_category_cloth_df, how='left',left_on='category_label',right_on='actual_category_number')
img_categories_combined_df
# Top 15 categories by images
(img_categories_combined_df
 .groupby('category_name')
 .count()
 .sort_values('image_name',ascending=False)
 .iloc[:15]
 .plot(kind='barh',y='image_name',use_index=True)
)
# Next 15 clothing categories
(img_categories_combined_df
 .groupby('category_name')
 .count()
 .sort_values('image_name',ascending=False)
 .iloc[15:30]
 .plot(kind='barh',y='image_name',use_index=True)
)
# Last set of clothing categories
(img_categories_combined_df
 .groupby('category_name')
 .count()
 .sort_values('image_name',ascending=False)
 .iloc[30:]
 .plot(kind='barh',y='image_name',use_index=True)
)<jupyter_output><empty_output><jupyter_text>Step 4A: 🔬 Deeper EDA & Exploration of the Attributes Data 📚    Let's get a high-level view of the attributes 👗 data<jupyter_code># In attribute type, 
#"1" represents texture-related attributes, 
#"2" represents fabric-related attributes, 
#"3" represents shape-related attributes, 
#"4" represents part-related attributes, 
#"5" represents style-related attributes;

list_attr_cloth_df.groupby('attribute_type').count()<jupyter_output><empty_output><jupyter_text>🎽 Exploring Texture Attributes     ➡️ Getting Counts & Distributions<jupyter_code>texture_list = list_attr_cloth_df.loc[list_attr_cloth_df['attribute_type']==1.0]['attribute_name'].to_list()
texture_dict = {}
for i in texture_list:
    count_ = list_attr_img_df[i].sum()
    texture_dict[i] = count_
texture_df_count = (pd.DataFrame.from_dict(texture_dict,orient='index')
                    .rename(columns={0:'count_'})
                   )
texture_df_count<jupyter_output><empty_output><jupyter_text>➡️ All by img count<jupyter_code>(texture_df_count
 .sort_values('count_',ascending=False)
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Top 20 by img count<jupyter_code>(texture_df_count
 .sort_values('count_',ascending=False)
 .iloc[:20]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Bottom 20 by img count<jupyter_code>(texture_df_count
 .sort_values('count_',ascending=False)
 .iloc[-20:]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>🎽 Exploring Fabric Attributes     ➡️ Getting Counts & Distributions<jupyter_code>fabric_list = list_attr_cloth_df.loc[list_attr_cloth_df['attribute_type']==2.0]['attribute_name'].to_list()
fabric_dict = {}
for i in fabric_list:
    count_ = list_attr_img_df[i].sum()
    fabric_dict[i] = count_
fabric_df_count = (pd.DataFrame.from_dict(fabric_dict,orient='index')
                    .rename(columns={0:'count_'})
                   )
fabric_df_count<jupyter_output><empty_output><jupyter_text>➡️ All by img count<jupyter_code>(fabric_df_count
 .sort_values('count_',ascending=False)
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Top 20 by img count<jupyter_code>(fabric_df_count
 .sort_values('count_',ascending=False)
 .iloc[:20]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Bottom 20 by img count<jupyter_code>(fabric_df_count
 .sort_values('count_',ascending=False)
 .iloc[-20:]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>🎽 Exploring Shape Attributes     ➡️ Getting Counts & Distributions<jupyter_code>shape_list = list_attr_cloth_df.loc[list_attr_cloth_df['attribute_type']==3.0]['attribute_name'].to_list()
shape_dict = {}
for i in shape_list:
    count_ = list_attr_img_df[i].sum()
    shape_dict[i] = count_
shape_df_count = (pd.DataFrame.from_dict(shape_dict,orient='index')
                    .rename(columns={0:'count_'})
                   )
shape_df_count<jupyter_output><empty_output><jupyter_text>➡️ All by img count<jupyter_code>(shape_df_count
 .sort_values('count_',ascending=False)
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Top 20 by img count<jupyter_code>(shape_df_count
 .sort_values('count_',ascending=False)
 .iloc[:20]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Bottom 20 by img count<jupyter_code>(shape_df_count
 .sort_values('count_',ascending=False)
 .iloc[-20:]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>🎽 Exploring Parts Attributes     ➡️ Getting Counts & Distributions<jupyter_code>part_list = list_attr_cloth_df.loc[list_attr_cloth_df['attribute_type']==4.0]['attribute_name'].to_list()
part_dict = {}
for i in part_list:
    count_ = list_attr_img_df[i].sum()
    part_dict[i] = count_
part_df_count = (pd.DataFrame.from_dict(part_dict,orient='index')
                    .rename(columns={0:'count_'})
                   )
part_df_count<jupyter_output><empty_output><jupyter_text>➡️ All by img count<jupyter_code>(part_df_count
 .sort_values('count_',ascending=False)
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Top 20 by img count<jupyter_code>(part_df_count
 .sort_values('count_',ascending=False)
 .iloc[:20]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Bottom 20 by img count<jupyter_code>(part_df_count
 .sort_values('count_',ascending=False)
 .iloc[-20:]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>🎽 Exploring Style Attributes     ➡️ Getting Counts & Distributions<jupyter_code>style_list = list_attr_cloth_df.loc[list_attr_cloth_df['attribute_type']==5.0]['attribute_name'].to_list()
style_dict = {}
for i in style_list:
    count_ = list_attr_img_df[i].sum()
    style_dict[i] = count_
style_df_count = (pd.DataFrame.from_dict(style_dict,orient='index')
                    .rename(columns={0:'count_'})
                   )
style_df_count<jupyter_output><empty_output><jupyter_text>➡️ All by img count<jupyter_code>(style_df_count
 .sort_values('count_',ascending=False)
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Top 20 by img count<jupyter_code>(style_df_count
 .sort_values('count_',ascending=False)
 .iloc[:20]
 .plot(kind='barh'))<jupyter_output><empty_output><jupyter_text>➡️ Bottom 20 by img count<jupyter_code>(style_df_count
 .sort_values('count_',ascending=False)
 .iloc[-20:]
 .plot(kind='barh'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12410501193317422, 'fraction_numerical': 0.04221604914120271, 'mean_word_length': 3.6821762227514196, 'pattern_counts': {'":': 0, '<': 130, '<?xml version=': 0, '>': 131, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '33', 'dir': 'jupyter-structured-clean-dedup', 'id': '352900', 'license': 'MIT', 'n_tokens_mistral': 9364, 'n_tokens_neox': 8695, 'n_words': 2215, 'path': 'notebooks/Step05_DataWrangling/Step5_DataWrangling.ipynb', 'repo_name': 'StyleVerse/Outfit-Recommender', 'size': '25568'}