<jupyter_start><jupyter_text>MIMIC III (sample) Code Prediction<jupyter_code>import fastbook
fastbook.setup_book()

from fastbook import *<jupyter_output><empty_output><jupyter_text>Application specific imports from fastai:<jupyter_code>from fastai.text.all import *
from fastai.distributed import *<jupyter_output><empty_output><jupyter_text>xcube specific imports:<jupyter_code>from xcube.text.learner import text_classifier_learner
from xcube.metrics import PrecisionK<jupyter_output><empty_output><jupyter_text>Setting the base path of the repository<jupyter_code>path = Path.cwd()
path_data = path/'data'
path_model = path/'models'

path, path_data, path_model<jupyter_output><empty_output><jupyter_text>`DataLoaders` for the Multi-Label Classifier (frozen)<jupyter_code>!ls -lh {path_data/"*caml*"}
!ls -lh {path_model/"caml*recent*"}
data_file = path_data/'caml_notes_labelled_sample_10percent.csv'
lm_vocab_file = path_model/'caml_dls_lm_vocab.pkl'
df = pd.read_csv(data_file, dtype={'text': str, 'labels': str})
df[['text', 'labels']] = df[['text', 'labels']].astype('str')
len(df)
# df.head(3)<jupyter_output><empty_output><jupyter_text>---<jupyter_code>df['length'] = df.apply(lambda row: len(row.text), axis=1)
df
np.percentile(df.length, 50)
np.mean(df.length)
df['text'] = df.apply(lambda row: row.text[10000:], axis=1)
df
print(df.text[678])<jupyter_output><empty_output><jupyter_text>---<jupyter_code>lm_vocab = torch.load(lm_vocab_file)
L(lm_vocab)
label_freq = Counter()
for labels in df.labels: label_freq.update(labels.split(';'))
lbls = list(label_freq.keys())
L(lbls)
def splitter(df):
    train = df.index[~df['is_valid']].tolist()
    valid = df.index[df['is_valid']].tolist()
    return train, valid
train, valid = splitter(df)
L(train), L(valid)
dblock = DataBlock(
        blocks = (TextBlock.from_df('text', seq_len=72, vocab=lm_vocab), MultiCategoryBlock(vocab=lbls)),
        get_x = ColReader('text'),
        get_y = ColReader('labels', label_delim=';'),
        splitter = splitter,
)
dls = dblock.dataloaders(df, bs=32)
torch.save(dls, path_model/'caml_dls_clas_sample_10percent_32.pkl')<jupyter_output><empty_output><jupyter_text>---<jupyter_code>dls_clas = torch.load(path_model/'caml_dls_clas_sample_10percent_32.pkl')
dls_clas.show_batch(max_n=1)
coll_repr(dls_clas.vocab[0], 30)
coll_repr(dls_clas.vocab[1], 30)
# x, y = dls_clas.one_batch()
# x.shape, y.shape
# x, y = first(dls_clas.train)
dls_iter = iter(dls_clas.train)
x,y = next(dls_iter)
x.shape, y.shape
coll_repr(lm_vocab, 30)
' '.join(lm_vocab[o] for o in x[0][:1000])<jupyter_output><empty_output><jupyter_text>`DataLoaders` for the Multi-Label Classifier (using fastai's Mid-Level Data API):  Loading Raw Data<jupyter_code>data_file = path_data/'caml_notes_labelled_sample_10percent.csv'
lm_vocab_file = path_model/'caml_dls_lm_vocab.pkl'
df = pd.read_csv(data_file, dtype={'text': str, 'labels': str})
df[['text', 'labels']] = df[['text', 'labels']].astype('str')
df.head(3)<jupyter_output><empty_output><jupyter_text>let's now gather the labels from the 'labels' column of the df:<jupyter_code>label_freq = Counter()
for labels in df.labels: label_freq.update(labels.split(';'))<jupyter_output><empty_output><jupyter_text>The total number of labels are:<jupyter_code>len(label_freq)<jupyter_output><empty_output><jupyter_text>Let's make a list of all labels (We will use it later while creating the `DataLoader`)<jupyter_code>lbls = list(label_freq.keys())<jupyter_output><empty_output><jupyter_text>Steps for creating the classifier `DataLoaders` using fastai `Transforms`:  1. train/valid `splitter`: Okay, based on the `is_valid` column of our Dataframe, let's create a splitter:<jupyter_code>def splitter(df):
    train = df.index[~df['is_valid']].tolist()
    valid = df.index[df['is_valid']].to_list()
    return train, valid<jupyter_output><empty_output><jupyter_text>Let's check the train/valid split<jupyter_code>splits = [train, valid] = splitter(df)
L(splits[0]), L(splits[1])<jupyter_output><empty_output><jupyter_text>2. Making the `Datasets` object: **Crucial:** We need the vocab of the language model so that we can make sure we use the same correspondence of token to index. Otherwise, the embeddings we learned in our fine-tuned language model won't make any sense to our classifier model, and the fine-tuning won't be of any use. So we need to pass the `lm_vocab` to the `Numericalize` transform: So let's load the vocab of the language model:<jupyter_code>lm_vocab = torch.load(lm_vocab_file)
L(lm_vocab)
@Transform
def Shrink(num:TensorText): return num[:2500] if len(num)>2500 else num
x_tfms = [Tokenizer.from_df('text'), attrgetter("text"), Numericalize(vocab=lm_vocab)] 
y_tfms = [ColReader('labels', label_delim=';'), MultiCategorize(vocab=lbls), OneHotEncode()]
tfms = [x_tfms, y_tfms]<jupyter_output><empty_output><jupyter_text>---<jupyter_code># class Shrink(Transform):
#     def encodes(self, num):
#         return num[:3500] if len(num)>3500 else num
# tls_x = TfmdLists(df, x_tfms, splits=splits)<jupyter_output><empty_output><jupyter_text>---<jupyter_code>dsets = Datasets(df, tfms, splits=splits)<jupyter_output><empty_output><jupyter_text>Let's now check if our `Datasets` got created alright:<jupyter_code>len(dsets.train), len(dsets.valid)
dsets.train[0]<jupyter_output><empty_output><jupyter_text>---<jupyter_code>type(dsets.train[0][0])
len(dsets.train[0][0])
torch.arange(10)
dsets.train[0][0][:] = TensorText(torch.arange(361))
dsets.train[0][0]<jupyter_output><empty_output><jupyter_text>---<jupyter_code>dsets.decode(dsets.train[0])<jupyter_output><empty_output><jupyter_text>Looks fine!  3. Making the `DataLoaders` object: We need to pick the sequence length and the batch size:<jupyter_code>bs, sl = 16, 72<jupyter_output><empty_output><jupyter_text>We will use the `dl_type` argument of the `DataLoaders`. The purpose is to tell `DataLoaders` to use `SortedDL` class of the `DataLoader`, and not the usual one. `SortedDL` constructs batches by putting samples of roughly the same lengths into batches.<jupyter_code>dl_type = partial(SortedDL, shuffle=True)<jupyter_output><empty_output><jupyter_text>**Crucial:** - We will use **`pad_input_chunk`** because our encoder `AWD_LSTM` will be wrapped inside `SentenceEncoder`. - A `SenetenceEncoder` expects that all the documents are padded, - with most of the padding at the beginning of the document, with each sequence beginning at a round multiple of bptt- and the rest of the padding at the end.<jupyter_code>dls_clas = dsets.dataloaders(bs=bs, seq_len=sl, 
                        dl_type=dl_type,
                       before_batch=pad_input_chunk)<jupyter_output><empty_output><jupyter_text>Creating the `DataLoaders` object takes considerable amount of time, so let's save it:<jupyter_code>torch.save(dls_clas, path_model/'caml_dls_clas_sample_10percent_16.pkl')<jupyter_output><empty_output><jupyter_text>Let's take a look at the data:<jupyter_code>dls_clas.show_batch(max_n=3)
dls_clas = torch.load(path_model/'caml_dls_clas_sample_10percent_32.pkl')<jupyter_output><empty_output><jupyter_text>---<jupyter_code>dls_iter = iter(dls_clas.train)
x, y = next(dls_iter)
x[23].shape<jupyter_output><empty_output><jupyter_text>---  `Learner` for Multi-Label Classifier Fine-Tuning<jupyter_code>dls_clas = torch.load(path_model/'caml_dls_clas_sample_10percent_16.pkl')
learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.1, max_len=72*40, metrics=PrecisionK).to_fp16()
learn = learn.load_encoder(path_model/'caml_lm_finetuned')
decoder = (learn.model[1].__class__)
decoder??<jupyter_output><empty_output><jupyter_text>Fine-Tuning the Classifier<jupyter_code># lr_min, lr_steep, lr_valley, lr_slide = learn.lr_find(suggest_funcs=(minimum, steep, valley, slide))
# lr_min, lr_steep, lr_valley, lr_slide
learn.fit_one_cycle(1, lr_max=1e-1, moms=(0.8,0.7,0.8), wd=0.1)
learn.freeze_to(-2)
learn.fit_one_cycle(8, lr_max=1e-2, moms=(0.8,0.7,0.8), wd=0.1, cbs=SaveModelCallback(fname=path_model/'caml_clas_sample_10percent'))
learn.freeze_to(-3)
learn.fit_one_cycle(10, lr_max=1e-3, moms=(0.8,0.7,0.8), wd=0.4, cbs=SaveModelCallback(fname=path_model/'caml_clas_sample_10percent'))
learn.unfreeze()
learn.fit_one_cycle(5, lr_max=1e-4, moms=(0.8,0.7,0.8), wd=0.7)
learn.recorder.plot_loss()
plt.plot(L(learn.recorder.values).itemgot(2));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14166268514094602, 'fraction_numerical': 0.019111323459149548, 'mean_word_length': 7.596509240246407, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 115, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '645805', 'license': 'Apache-2.0', 'n_tokens_mistral': 3404, 'n_tokens_neox': 3298, 'n_words': 589, 'path': 'nbs/examples/mimic/mimic3_sample_prediction.ipynb', 'repo_name': 'debjyotiSRoy/xcube', 'size': '8372'}