<jupyter_start><jupyter_text>Am I using a model trained on imagenet to classify imagenet images?  seems like that is unfair and would tell me that my deep learning setup is better than it really isFor kaggle, go back and retrain on entire dataset to get better results<jupyter_code># Put these at the top of every notebook, to get automatic reloading and inline plotting
%reload_ext autoreload
%autoreload 2
%matplotlib inline
# This file contains all the main external libs we'll use
from fastai.imports import *
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *
PATH = "data/breed/"
sz =224 #size of image.  also the size of imagenet immages and is typical to use
bs = 24 #batch size i belive
!ls {PATH}
train_files_head = !ls {PATH}/train | head
train_files_head
img = plt.imread(f'{PATH}train/{train_files_head[1]}')
plt.imshow(img)
img.shape
label_csv = f'{PATH}labels.csv'
!ls
label_df = pd.read_csv(f"{PATH}/labels.csv")
label_df.head()<jupyter_output><empty_output><jupyter_text>Now I need to figure out how to split into training and testing set.  Previously I just had my training and testing data in separate folders.  This is another common way to do it having labels for each file.  70 train 30 test split seems appropriate.<jupyter_code>#list of rows, minues 1, equals the number of images
n = len(list(open(f"{PATH}/labels.csv")))-1
n
label_df.pivot_table(index='breed', aggfunc=len).sort_values('id', ascending=False)
#get crossvalidation indexes custom FASTAI
val_idxs = get_cv_idxs(n)
print(val_idxs.shape)
val_idxs
??get_cv_idxs
arch=resnet34
tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
data = ImageClassifierData.from_csv(PATH, 'train' ,f"{PATH}/labels.csv", test_name='test', val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)

fn = PATH+data.trn_ds.fnames[0]; fn
img = PIL.Image.open(fn); img
#maps name of file to the size of file for analysis
size_d = {k: PIL.Image.open(PATH+k).size for k in data.trn_ds.fnames} 
size_d
row_sz, col_sz = list(zip(*size_d.values()))
row_sz=np.array(row_sz); col_sz=np.array(col_sz)
row_sz[:5]
plt.hist(row_sz) #uses matplotlib
plt.hist(row_sz[row_sz<1000])<jupyter_output><empty_output><jupyter_text>Initial model<jupyter_code>def get_data(sz, bs):
    tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
    data = ImageClassifierData.from_csv(PATH, 'train', f'{PATH}labels.csv', test_name='test', val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)
    return data if sz > 300 else data.resize(340, 'tmp')<jupyter_output><empty_output><jupyter_text>3.2 Augment looks like he jumped to 90% in the augment secion with precompute still true<jupyter_code>data = get_data(sz,bs)
learn = ConvLearner.pretrained(arch, data, precompute=True, ps=0.5)

learn.fit(1e-2, 2) #learning rate and epochs
learn.precompute=False
learn.fit(1e-2, 5, cycle_len=1)
learn.save('breed_1')
#increase size
learn.set_data(get_data(299,bs))
learn.freeze()
learn.fit(1e-2, 3, cycle_len=1)
learn.save('breed_2_299')
from sklearn.metrics import log_loss
log_preds,y = learn.TTA()
probs = np.mean(np.exp(log_preds),0) 
accuracy(probs,y)
log_loss(y, probs)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11566484517304189, 'fraction_numerical': 0.017304189435336976, 'mean_word_length': 6.225877192982456, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '575239', 'license': 'Apache-2.0', 'n_tokens_mistral': 1262, 'n_tokens_neox': 1214, 'n_words': 333, 'path': 'courses/dl1/breeds.ipynb', 'repo_name': 'c1505/fastai', 'size': '3294'}