<jupyter_start><jupyter_text>"fastai snippets"> "Useful fastai code snippets"- toc: False- branch: master- categories: [code snippets, jupyter, python] Currently using fastai2.  Installation via `pip install fastai2`.  Oversampling<jupyter_code>from fastai2.tabular.all import *
titanic = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv')
cat_vars = ['survived', 'pclass', 'sex', 'embarked', 'class', 'who', 'adult_male', 'deck', 'embark_town', 'alone']
cont_vars = ['age', 'sibsp', 'parch', 'fare']
dep_var = 'alive'
procs = [Categorify, FillMissing, Normalize]
block_y = CategoryBlock()
splits = RandomSplitter()(range_of(titanic))
to = TabularPandas(titanic, procs=procs, cat_names=cat_vars, cont_names=cont_vars,
                   y_names=dep_var, block_y=block_y, splits=splits)
to.items
to.items.alive.value_counts().plot.bar();
dls = to.dataloaders()
learn = tabular_learner(dls, [200,100], metrics=accuracy)<jupyter_output><empty_output><jupyter_text>The `get_idxs` method of DataLoaders samples the `idxs` to be used in an epoch of training:<jupyter_code>learn.dls.get_idxs()[:10]
len(learn.dls.get_idxs())
df = pd.DataFrame(1 / learn.dls.items.alive.value_counts())
weights = learn.dls.items.alive.apply(lambda x: df.to_dict()['alive'][x])
learn.dls.train.items.sample(n=700, weights=weights, replace=True).alive.value_counts()
def sample_epoch(self):
    df = pd.DataFrame(1 / self.items.alive.value_counts())
    weights = self.items.alive.apply(lambda x: df.to_dict()['alive'][x])
    self.n = 700 # going for bigger epochs as this will require quite a bit of training
    oversampled_idxs = self.items.sample(n=self.n, weights=weights, replace=True).index
    return [np.where(self.items.index == i)[0][0] for i in oversampled_idxs]
learn.dls.train.get_idxs = types.MethodType(sample_epoch, learn.dls.train)
idxs = learn.dls.train.get_idxs()
learn.dls.train.items.iloc[idxs].alive.value_counts()
learn.fit(10, 1e-3)
learn = tabular_learner(dls, [200,100], metrics=accuracy)
learn.fit(10, 1e-3)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1555127587867116, 'fraction_numerical': 0.016851227732306212, 'mean_word_length': 6.116438356164384, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '201604', 'license': 'Apache-2.0', 'n_tokens_mistral': 821, 'n_tokens_neox': 795, 'n_words': 145, 'path': '_notebooks/2020-03-18-fastai.ipynb', 'repo_name': 'fabge/snippets', 'size': '2077'}