<jupyter_start><jupyter_text>snkrfinder.data version 1.0 Dec 2020 (refactor using pytorch/nbdev/fastai framework) OVERVIEWThis is a project initiated while an Insight Data Science fellow.  It grew out of thinking about how to make data-driven tools that could impact the fashion industry which I had been working working in.   The original over-scoped idea was to make a shoe desighn tool which could quickly develop some sneaker designs based on choosing some examples, and some text descriptors.  Designs are constrained by the "latent space" defined (discovered?) by a database of shoe images.  However, given the 3 week sprint allowed for development, I pared the tool down to a simple "aesthetic" recommender for sneakers, using the same idea of utilizing an embedding space defined by the database fo shoe images.NOTE:  symbolic link in the nbs directory to enable the module loads in these notebooks.  i.e. `ln -s ../snkrfinder/ snkrfinder`> Used to be called data.munge  but its all munging... This is a cool reference for a fastai VAEhttps://medium.com/@dhuynh95/an-introduction-to-unsupervised-learning-with-fastai-a6dbd78eca2b First we need to get a database of sneaker images.   The sneaker images are only 136 pixels wide, and organized by 'Category', 'SubCategory', and 'Brand'.   Additional semantic information is also available from Mechanical Turk labeling, and is available packed into matlab files.<jupyter_code>#hide

print(Path().cwd())
#hide
if Path().cwd().parts[-1] == 'nbs': 
    os.chdir('..')
    print('moved out of `nbs` directory')
    
print(Path().cwd())
D_ROOT<jupyter_output><empty_output><jupyter_text>Part 0: DATA Part 1: import the UT-Zappos50k databaseUsing fastai `untar_data`.<jupyter_code>#export
def get_zappos_db():
    "import the UT zappos 50k database from vision.cs.utexas.edu"
    # the images are wider than tall with the product already taking up aproximately the whole vertical dimension
    url_images = "http://vision.cs.utexas.edu/projects/finegrained/utzap50k/ut-zap50k-images.zip"
    url_meta = "http://vision.cs.utexas.edu/projects/finegrained/utzap50k/ut-zap50k-data.zip"

    DATA_path = D_ROOT
    meta_path = untar_data(url_meta, dest=DATA_path)
    im_path = untar_data(url_images,dest=DATA_path)
    
    return meta_path, im_path
#export

def read_zappos_meta(path_meta):
    """read the metadata from UT zappos 50k database.

    The data is split between a  matlabe format, and a csv file.  A helper,
    `_path_from_mat` reads the Matlab, and the two are combined into a 
    Pandas dataframe.   Some light munging to clean up some awkward strings and
    prep the info for easy access follows.

    Args:
        path_meta (str): path where the meta-data files live

    Returns:
        df (pd.DataFrame): dataframe having all the sneaker info and and image name

    """
    
    
    def _path_from_mat(fname):
        """ reads zappos imagepath from matlab file"""
        data = sio.loadmat(fname)['imagepath']
        return [i[0][0] for i in data]

    image_path = _path_from_mat(path_meta/'image-path.mat')
    df = pd.read_csv(path_meta/'meta-data.csv')

    df["path"]=image_path
    
    # ad sub-categories (one-hot)
    categories=pd.read_csv(path_meta/'meta-data-bin.csv')
    df = pd.merge(df, categories,  how='left', on='CID')


    # fix the path by remove trailing periods in folder names
    to_fix = df.path.str.contains("./",regex=False)
    df.loc[to_fix,"path"] = [i.replace("./","/") 
                                for i in 
                                    df.loc[to_fix,"path"]]
    
    to_fix = df.path.str.contains("Levi\'s ",regex=False)
    df.loc[to_fix,"path"] = [i.replace("Levi\'s ","Levis ") 
                               for i in 
                                  df.loc[to_fix,"path"]] 

    # create brands and category stubs...
    df['path_and_file'] = df.path.apply(lambda path: (os.path.normpath(path)).split(os.sep) ) 
    df_to_add = pd.DataFrame(df['path_and_file'].tolist(), columns=['Category1','Category2','Brand','Filename'])

    #prepend the root-directory of for the images 'ut-zap50k-images/'
    df.loc[:,"path"]=df.path.apply(lambda x: 'ut-zap50k-images/'+x)

    df = df.merge(df_to_add, left_index=True, right_index=True)
    return df

meta_path, im_path = get_zappos_db()

assert (meta_path==ZAPPOS_META_DIR)
assert(im_path==D_ROOT/DBS["zappos"])

df = read_zappos_meta(meta_path)

assert (df['Category'] == df['Category1']).all()
assert (df['SubCategory'] == df['Category2']).all()<jupyter_output><empty_output><jupyter_text>data set cleaningThe dataset is very diverse and not constructed in order to think about how popular footwear is aesthetically related.  I'm simplifying the database to include 3 main categories which seem to split aesthetics for general footwear:    1. BOOTS - weatherized and/or protected footwear for work or outdoor activity    2. SHOES - generic non "boot" footwear.  includes heels and formalwear    3. SNEAKERS - atheletic / comfort inspired I will also make a fourth category which might be useful:    4. SLIPPERS - charachterized by a sole and straps<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>I've taken several steps to try and simplify the "asethetic" space the modeling will need to capture is to exclude some particular sub-categories which represent extremes.  E.g. High cut boots, high-heels, and kids sub-categores. Indeed, I will only include "adult" shoes at this time but excluding "kids" and some edge cases because the overall size scale distorts the overall "aesthetics" I'd like to get at.<jupyter_code>#export

def simplify_zappos_db(df):
    """simlify the dataframe derived from the UT zappos 50k database.

    Args:
        df (pd.DataFrame): dataframe containing all 50k sneakers / images 

    Returns:
        df (pd.DataFrame): dataframe limited to Sneakers, Boots, Shoes for Adults

    """
        
    # add our "sneaker category"
    df.loc[:,'Sneakers'] = (df['Category2'] == 'Sneakers and Athletic Shoes')

    # refine boot
    df.loc[:,'Boots'] = (  (df.Category1 == 'Boots') 
                         & (df.Category2 != 'Knee High')
                         & (df.Category2 != 'Over the Knee')
                         & (df.Category2 != 'Prewalker Boots') )

    # refine shoes
    df.loc[:,'Shoes'] = (  (df.Category1 == 'Shoes') 
                         & (df.Category2 != 'Sneakers and Athletic Shoes')
                         & (df.Category2 != 'Heels')                         
                         & (df.Category2 != 'Crib Shoes')
                         & (df.Category2 != 'Firstwalker')
                         & (df.Category2 != 'Prewalker') )

    # refine slippers
    df.loc[:,'Slippers'] = (  (df.Category1 == 'Slippers')
                         & (df.Category2 != 'Boot') )

    #remove ([ 'Boys',  'Boys;Girls', 'Girls','Women;Girls', nan
    mens =  df['Gender'] == 'Men'       
    womens =  df['Gender'] == 'Women' 
    etc =  df['Gender'].str.contains('Men;', na=False)
    df.loc[:,'Adult'] = mens | womens | etc
    df.loc[:,'Mens'] = mens 
    df.loc[:,'Womens'] = womens

    df.loc[:,'OGcategory'] = df.Category
    df.loc[:,'Category'] = pd.NA

    df.loc[(df.Shoes==1),'Category'] = 'Shoes'
    df.loc[(df.Boots==1),'Category'] = 'Boots'
    df.loc[(df.Sneakers==1),'Category'] = 'Sneakers'
    df.loc[(df.Slippers==1),'Category'] = 'Slippers'
    
    # make some expository columns 
    keep_columns = ['CID','Category',
                     'path','path_and_file',
                     'Category1', 'Category2','OGcategory'
                     'Brand','Filename',
                     'Sneakers','Boots',
                     'Shoes', 'Slippers','Adult',
                     'Gender']

    df = df.filter(items=keep_columns)
    #keep Adult, Sneakers, Boots, Shoes, Slippers
    keep_rows = (df.Sneakers | df.Boots | df.Shoes| df.Slippers) & (df.Adult)
    df = df[keep_rows.values]
    
    return df
#hide
df.Category.unique()

df = simplify_zappos_db(df)
#hide
df.head()<jupyter_output><empty_output><jupyter_text>------------  Prepare the simplified dataframe for further analysisAlso use sklearn `train_test_split` to create category stratified train/test/validate groups.  We'll keep 15 percent of our data to truly test.<jupyter_code>#export

def skl_tt_split(df,strat_cat):
    "adds stratified train-validate-test via sklearn"

    X = df.index
    y = strat_cat

    train_ratio = 0.70
    validation_ratio = 0.15

    # keep
    test_ratio = 0.15

    # train is now 70% of the entire data set
    x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=1-train_ratio, stratify=y, random_state=666)

    # test is now 15% of the initial data set
    # validation is now 15% of the initial data set
    x_val, x_test, y_val, y_test = train_test_split(x_test, y_test, test_size=test_ratio/(test_ratio + validation_ratio), stratify=y_test, random_state=666) 
    # pack into the dataframe
    df.loc[:,'train'] = False
    df.loc[:,'test'] = False
    df.loc[:,'validate'] = False
    df.loc[:,'t_t_v'] = 'train'
    df.loc[x_train,'train'] = True
    df.loc[x_test,'test'] = True
    df.loc[x_val,'validate'] = True
    df.loc[x_test,'t_t_v'] = 'test'
    df.loc[x_val,'t_t_v'] = 'valid'
    return df

    
#hide
df = skl_tt_split(df,df.Category)
df.head()<jupyter_output><empty_output><jupyter_text>------------  Create a unfied database of "sneakers"- images (need to access) path to fit beta-VAE - descriptions for fitting text autoencoder- skip "meta" data (will come back to this later for further analysis.<jupyter_code>filename = f"data/{ZAPPOS_DF_SIMPLIFIED}.pkl" #"zappos-50k-simplified_sort"

if os.path.exists(filename):
    df_zappos = pd.read_pickle(filename)
SCRAPED_META_DIR/SCRAPED_DF

filename = f"{SCRAPED_META_DIR/SCRAPED_DF}.pkl" 
filename = f"data/{SCRAPED_DF}.pkl"

if os.path.exists(filename):
    df_scraped = pd.read_pickle(filename)
#hide 
# attributes = df_scraped.attributes.values
# description = df_scraped.description.values
df_scraped[['url','hero_fullpath']].head(2)<jupyter_output><empty_output><jupyter_text>I need to put the scraped databases from SNS (sneakersnstuff.com) and GOAT (goat.com) into a dataframe with columns suitable merge with the UTZappos database I originally started with.   I'll spend some time infering the `Category`, `Brand`, `Gender` (and `model`?) so I can pay attention to class balance .  From the transfer learning work, the `Slippers` category is pretty poor, but I should be able to use that net to label the scraped data... and validate based on the `attributes` and `see_also` fields.First things first:  fix the "hero_fullpath" so its agnostic of OS (Mac or Linux).<jupyter_code># this cooresponds to the relative path in DBS_REL['goat'] and DBS_REL['sns']
df_scraped.loc[:,"path"]=df_scraped.hero_fullpath.str.split('/').apply(lambda x: 'scraped/'+x[-3]+'/'+x[-1])
#df_scraped.loc[:,"path"]=df_scraped.hero_fullpath.str.split('/').apply(lambda x: x[-3]+'/'+x[-1])
df_scraped["path"].head(),df_scraped["path"].tail()
#export
def extract_cat(lst):
    for l in lst:
        if l.startswith("CATEGORY"):
            return l.split("\n")[-1]
    return "na" #np.nan

def extract_brand_goat(lst):
    for l in lst:
        if l.startswith("BRAND"):
            return l.split("\n")[-1]
    return "na"

def extract_brand_sns(lst):
    return lst[1]

def extract_db_nm(pathn):
    return pathn.split('/')[0]<jupyter_output><empty_output><jupyter_text>Now we just need to wrap some cleaning into a function for exporting...<jupyter_code># export
def get_scraped_db():
    "collect meta data fromscraped databases"
    
    def _extract_cat(lst):
        for l in lst:
            if l.startswith("CATEGORY"):
                return l.split("\n")[-1]
        return "na" #np.nan

    def _extract_brand_goat(lst):
        for l in lst:
            if l.startswith("BRAND"):
                return l.split("\n")[-1]
        return "na"

    def _extract_brand_sns(lst):
        return lst[1]

    def _extract_db_nm(pathn):
        return pathn.split('/')[0]
    
    filename = f"data/{SCRAPED_DF}.pkl"
    df_scraped = pd.read_pickle(filename)
    df_scraped.loc[:,"path"]=df_scraped.hero_fullpath.str.split('/').apply(lambda x: 'scraped/'+x[-3]+'/'+x[-1])
    df_scraped.loc[:,"brand"]=df_scraped.attributes.apply(_extract_brand_sns)
    df_scraped.loc[:,"cat"]=df_scraped.attributes.apply(_extract_cat)
    df_scraped.loc[:,"db_name"]=df_scraped["path"].apply(_extract_db_nm)
    df_scraped.loc[df_scraped['db_name']=='goat',"brand"]=df_scraped.loc[df_scraped['db_name']=='goat',"attributes"].apply(_extract_brand_goat)
    
    
    
    return df_scraped

# export
def merge_dbs(df_zappos,df_scraped):
    "could be any dfs with 'path','train','test','validate','t_t_v'"
    # TODO:  add "is_valid" wiich is test and (so validate are part of test)?)
    return pd.merge(df_zappos,df_scraped,how='outer',on=['path','train','test','validate','t_t_v'])
df = get_scraped_db()
# stratify across db_name
df_scraped = skl_tt_split(df,df.db_name)
df_zappos['Sneakers']
#export

def extract_zap_sneakers(df_zappos):
    return df_zappos[df_zappos['Sneakers']]
df_zap = extract_zap_sneakers(df_zappos)
df_zap.shape
df_test = merge_dbs(df_zap,df_scraped)

df_scraped.columns,df_zappos.columns,df_test.columns

df_scraped.shape,df_zappos.shape,df_test.shape

# TODO:  check for duplicate paths before saving...<jupyter_output><empty_output><jupyter_text>------------<jupyter_code>#hide

from nbdev.export import notebook2script
notebook2script()<jupyter_output>Converted 00_core.ipynb.
Converted 01a_data.ipynb.
Converted 02a_model.core.ipynb.
Converted index.ipynb.
{'fraction_non_alphanumeric': 0.12788917243893547, 'fraction_numerical': 0.008749544294567991, 'mean_word_length': 3.4188144329896906, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '343751', 'license': 'Apache-2.0', 'n_tokens_mistral': 4823, 'n_tokens_neox': 4628, 'n_words': 1188, 'path': 'nbs/01a_data.ipynb', 'repo_name': 'ergonyc/snkrfinder', 'size': '13715'}