<jupyter_start><jupyter_text>Data[MIMIC III dataset](https://mimic.physionet.org/gettingstarted/overview/) is a dataset developed by the MIT Lab for Computational Physiology, it contains de-identified health records from about 53,000 patients, who stayed in critical care units between 2001 and 2012. MIMIC-III includes several types of clinical notes, including discharge summaries (n = 52,746) labelled by ICD-9 codes (International Classification of Diseases).  The database has 26 tables, we donwloaded them and created a local Postgres database for further exploration. We will be mostly using two tables: clinical notes (NOTEEVENTS) and ICD9 codes assigned to diagnoses (DIAGNOSES_ICD).   Related previous research didn't work with all ICD9-codes but only with the most used in diagnoses reports. We identified the top 20 icd9 codes based on number of patients with that label (DIAGNOSES_ICD) and filtered diagnoses discharge summaries reports that have those codes assigned (via a JOIN with NOTEEVENTS and further filtering). This pre-processing was done in the Postgres database (for more details please see the project report) and the results exported to the **dis_notes_icd9.csv** file.<jupyter_code>with open('psql_files/dis_notes_icd9.csv', 'rb') as csvfile:
    discharge_notes_reader = csv.reader(csvfile)
    discharge_notes_list = list(discharge_notes_reader)    
random.shuffle(discharge_notes_list)
print "Number of records in the dataset: ", len (discharge_notes_list)<jupyter_output>Number of records in the dataset:  45837<jupyter_text>Sample of a discharge summary and the ICD( codes)<jupyter_code>print 'Sample of a discharge note:'
print "-" *100
admission_id, subject_id, discharge_date, note_text, icd9_codes = discharge_notes_list[1]
print "Admission id: ", admission_id
print "Subject id:", subject_id
print "Discharge date:", discharge_date
print "ICD9 codes assigned to this discharge summary: ", icd9_codes
print "-" *100
print "Discharge Summary Clinical Note: "
print "-" *100
#print note_text 
print "MIMIC data is visible to only authorized users"<jupyter_output>Sample of a discharge note:
----------------------------------------------------------------------------------------------------
Admission id:  105471
Subject id: 409
Discharge date: 2159-09-25 00:00:00
ICD9 codes assigned to this discharge summary:  2720
----------------------------------------------------------------------------------------------------
Discharge Summary Clinical Note: 
----------------------------------------------------------------------------------------------------
MIMIC data is visible to only authorized users<jupyter_text>Some Data explorations<jupyter_code>notes= [row[3] for row in discharge_notes_list]
labels = [row[4] for row in discharge_notes_list]
#counts by icd9_codes
icd9_codes = Counter()
for label in labels:
    for icd9_code in label.split():
        icd9_codes[icd9_code] += 1
print icd9_codes
codes_counts =icd9_codes.items()
codes_counts.sort(key=itemgetter(1), reverse=True)
icd9_labels, values = zip(*codes_counts)
indexes = np.arange(len(icd9_labels))
plt.rcdefaults()
fig,ax = plt.subplots()
ax.barh(indexes, values,  align='center', color='green', ecolor='black')
ax.set_yticks(indexes)
ax.set_yticklabels(icd9_labels)
ax.invert_yaxis()  # labels read top-to-bottom
ax.set_xlabel('counts')
ax.set_ylabel('ICD9 code')
ax.set_title('Top 20 ICD9 codes and its counts')
plt.show()
notes_length = [len(note) for note in notes]
print "Discharge Summary Clinical Notes median: %10.2f" %(np.median(notes_length))
print "Discharge Summary Clinical Notes mean: %10.2f " % (np.mean(notes_length))

n, bins, patches = plt.hist(notes_length, 50, normed=1, facecolor='green', alpha=0.75)
plt.xlabel('Discharge Summary Clinical Note length')
plt.title('Histogram of Discharge Summary Clinical Note lengths')
plt.grid(True)

plt.show()<jupyter_output><empty_output><jupyter_text>Simple BaselineThis is a multilabel classification since each discharge summary has several icd9 codes assigned. The most simple baseline will be to predict the top N icd9 codes for each discharge summary.  Spliting files in training, dev and test<jupyter_code>def split_file(data, train_frac = 0.7, dev_frac = 0.15):   
    train_split_idx = int(train_frac * len(data))
    dev_split_idx = int ((train_frac + dev_frac)* len(data))
    train_data = data[:train_split_idx]
    dev_data = data[train_split_idx:dev_split_idx]
    test_data = data[dev_split_idx:]
    return train_data, dev_data, test_data

train_data_notes, dev_data_notes, test_data_notes = split_file (notes)
train_data_labels, dev_data_labels, test_data_labels = split_file (labels)
print 'Training set samples:', len (train_data_notes)
print 'Dev set samples:', len (dev_data_notes)
print 'Test set samples:', len (test_data_notes)<jupyter_output>Training set samples: 32085
Dev set samples: 6876
Test set samples: 6876<jupyter_text>Finding out list of unique icd9 codes and the top 4<jupyter_code># finding out the top icd9 codes

top_4_icd9 = icd9_codes.most_common(4)
print "most common 4 icd9_codes: ", top_4_icd9

top_4_icd9_label = ' '.join(code for code,count in top_4_icd9 )
print 'label for the top 4 icd9 codes: ', top_4_icd9_label
# list of unique icd9_codes and lookups for its index in the vector
unique_icd9_codes = list (icd9_codes)
index_to_icd9 = dict(enumerate(unique_icd9_codes))
icd9_to_id = {v:k for k,v in index_to_icd9.iteritems()}
print 'List of unique icd9 codes from all labels: ', unique_icd9_codes<jupyter_output>List of unique icd9 codes from all labels:  ['2859', '99592', '4019', '2724', '25000', '2720', '2851', '2762', '2449', '4280', '0389', '41401', '53081', '51881', '5990', '42731', '486', '496', '5070', '5849']<jupyter_text>Converting icd9 labels to vectors Each discharge note label is a list of icd9 codes, for example: ```4019 25000 496 4280   ```we will represent it as a vector for the multilabel classification process, the vector would look like:```[0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0]```<jupyter_code>#transforming list of icd_codes into a vector
def get_icd9_array(icd9_codes):
    icd9_index_array = [0]*len(unique_icd9_codes)
    for icd9_code in icd9_codes.split():
        index = icd9_to_id [icd9_code]
        icd9_index_array[index] = 1
    return icd9_index_array
#top 4 common icd9 to vector
icd9_prediction_vector = get_icd9_array(top_4_icd9_label)
print 'icd9 prediction vector: ', icd9_prediction_vector
# true icd9 codes to vector
train_data_labels_vector= list(map(get_icd9_array, train_data_labels))
dev_data_labels_vector = list(map(get_icd9_array, dev_data_labels))
print 'example of a training label vector: ', train_data_labels_vector[0]
## assign icd9_prediction_vector to every discharge
train_y_hat_baseline = [icd9_prediction_vector]* len (train_data_labels_vector)
dev_y_hat_baseline = [icd9_prediction_vector]* len (dev_data_labels_vector)<jupyter_output><empty_output><jupyter_text>Performance evaluation There are different type of metric to evaluate a multi-label classification model.  Here are some of them:    http://scikit-learn.org/stable/modules/model_evaluation.htmllabel-ranking-loss        We will use ranking loss to compared with the NN Baseline which also will be compared with the NN build by this paper: "ICD-9 Coding of Discharge Summaries"<jupyter_code>training_ranking_loss = label_ranking_loss(train_data_labels_vector, train_y_hat_baseline)
print "Training ranking loss: ", training_ranking_loss
dev_ranking_loss = label_ranking_loss(dev_data_labels_vector, dev_y_hat_baseline)
print "Development ranking loss: ", dev_ranking_loss
flat_bb_y_hat = [item for sublist in train_y_hat_baseline for item in sublist]
flat_bb_label = [item for sublist in train_data_labels_vector for item in sublist]
bb_f1_score = f1_score(flat_bb_label, flat_bb_y_hat)
print "Train F1 score: ",bb_f1_score
flat_bb_y_hat = [item for sublist in dev_y_hat_baseline for item in sublist]
flat_bb_label = [item for sublist in dev_data_labels_vector for item in sublist]
bb_f1_score = f1_score(flat_bb_label, flat_bb_y_hat)
print "Dev F1 score: ",bb_f1_score<jupyter_output>Dev F1 score:  0.354999503525<jupyter_text>NN Baseline This baseline is based on  the Neural Network model implemented in this research paper:[ICD-9 Coding of Discharge Summaries](https://www.google.com/url?q=https%3A%2F%2Fcs224d.stanford.edu%2Freports%2Flukelefebure.pdf). We will use the same input dataset than the one used by the  Basic Baseline, but only taking 10,000 records to be able to compare performance results with the NN implemented in the referenced paper.<jupyter_code># the full file needs about 8GB of memory, let's work with the first 5000
#discharge_notes= np.asarray(discharge_notes_list)
discharge_notes_nparray= np.asarray(discharge_notes_list[0:10000])
print 'Number of discharge clinical notes: ', len(discharge_notes_nparray)
discharge_notes= discharge_notes_nparray[:,3]
discharge_labels = discharge_notes_nparray[:,4]
train_notes, dev_notes, test_notes = split_file (discharge_notes)
train_labels, dev_labels, test_labels = split_file (discharge_labels)
print 'Training set samples:', len (train_notes)
print 'Dev set samples:', len (dev_notes)
print 'Test set samples:', len (test_notes)<jupyter_output>Training set samples: 7000
Dev set samples: 1500
Test set samples: 1500<jupyter_text>TF-IDF Representation of discharge clinical notesPrevious research represents this documents/notes as bag-of-words vectors [1].    In particular, it takes the 10,000 tokens with the largest tf-idf scores from the training. [1] Diagnosis code assignment: models and evaluation metrics. Journal of the American Medical Informatics<jupyter_code>max_number_features = 10000
# TfidfVectorizer
# Convert all characters to lowercase before tokenizing (by default)
# tokenization (by default)
# max_features: consider the top max_features ordered by term frequency across the corpus
vectorizer = TfidfVectorizer(max_features=max_number_features,stop_words='english',max_df=0.9 )  
train_notes_vector = vectorizer.fit_transform(train_notes)
dev_notes_vector = vectorizer.transform(dev_notes)<jupyter_output><empty_output><jupyter_text>Transforming list of ICD codes to vectors<jupyter_code>train_labels_vector= list(map(get_icd9_array, train_labels))
dev_labels_vector = list(map(get_icd9_array, dev_labels))
test_labels_vector = list(map(get_icd9_array, test_labels))<jupyter_output><empty_output><jupyter_text>Neural Network for Multilabel classification<jupyter_code>def run_epoch(lm, session, X, y, batch_size):
    for batch in xrange(0, X.shape[0], batch_size):
        X_batch = X[batch : batch + batch_size]
        y_batch = y[batch : batch + batch_size]
        feed_dict = {lm.x:X_batch,lm.target_y:y_batch}
        loss, train_op_value =  session.run( [lm.loss,lm.train],feed_dict=feed_dict ) 
    
def predict_icd9_codes(lm, session, x_data, y_dim):
    total_y_hat = []
    for batch in xrange(0, x_data.shape[0], batch_size):
        X_batch = x_data[batch : batch + batch_size]
        y_hat_out = session.run(lm.y_hat, feed_dict={lm.x:X_batch})
        total_y_hat.extend(y_hat_out)
    return  total_y_hat
    
#build tensorflow graphs
reload(nn_model)

# Model parameters
Hidden_dims = [100]
learning_rate = 0.01
y_dim = len(unique_icd9_codes)
model_params = dict(Hidden_dims=Hidden_dims, 
                    learning_rate = learning_rate, vocabulary_size =max_number_features , y_dim=y_dim)

lm = nn_model.NNLM(**model_params)
lm.BuildCoreGraph()
lm.BuildTrainGraph()
    
X = train_notes_vector.todense()
y = train_labels_vector
batch_size = 50
num_epochs = 50

with lm.graph.as_default():
    initializer = tf.global_variables_initializer()

with tf.Session(graph=lm.graph) as session:
    session.run(initializer)
    #training
    for epoch_num in xrange(num_epochs):
        run_epoch(lm, session, X, y, batch_size)
    #prediction using training and dev data 
    train_y_hat = predict_icd9_codes(lm, session, train_notes_vector.todense(), y_dim)
    dev_y_hat = predict_icd9_codes(lm, session, dev_notes_vector.todense(), y_dim)<jupyter_output><empty_output><jupyter_text>Performance EvaluationWe are using first the [ranking loss metric](http://scikit-learn.org/stable/modules/model_evaluation.htmllabel-ranking-loss) to compare resutls with the referenced paper below. We did match! same ranking loss values for a file of 10,000 records and using only top 20 icd9-codes.<jupyter_code># ranking loss
training_ranking_loss = label_ranking_loss(train_labels_vector, train_y_hat)
print "Training ranking loss: ", training_ranking_loss
dev_ranking_loss = label_ranking_loss(dev_labels_vector, dev_y_hat)
print "Development ranking loss: ", dev_ranking_loss<jupyter_output>Training ranking loss:  0.324229648916
Development ranking loss:  0.321523707628<jupyter_text>The following paper "ICD-9 Coding of Discharge Summaries"  worked with the MIMIC II database to classify ICD9-codes, the ranking loss metrics reported are:<jupyter_code>#choosing a threshold 
def get_hot_vector (probs_list, threshold):
    vector  = []
    for prob in probs_list:
        train_y_hat_hot = [ 1 if  p > threshold else 0 for p in prob]
        vector.append(train_y_hat_hot)
    return vector

# f1-score
threshold= 0.25
hot_y_hat = get_hot_vector(train_y_hat, threshold)
# flat predictions
flat_y_hat = [item for sublist in hot_y_hat for item in sublist]
flat_train_label = [item for sublist in train_labels_vector for item in sublist]
training_f1_score = f1_score(flat_train_label, flat_y_hat)
print "Training F1 score: ", training_f1_score 



hot_dev_y_hat = get_hot_vector(dev_y_hat, threshold)
flat_dev_y_hat = [item for sublist in hot_dev_y_hat for item in sublist]
flat_dev_label = [item for sublist in dev_labels_vector for item in sublist]
dev_f1_score = f1_score(flat_dev_label, flat_dev_y_hat)
print "Dev F1 score: ", dev_f1_score<jupyter_output>Dev F1 score:  0.354666425274
{'fraction_non_alphanumeric': 0.10367821036782103, 'fraction_numerical': 0.03319710331971033, 'mean_word_length': 5.265947888589398, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 46, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '12063', 'license': 'Apache-2.0', 'n_tokens_mistral': 4912, 'n_tokens_neox': 4523, 'n_words': 1337, 'path': 'baseline/.ipynb_checkpoints/mimic_icd9_baseline-checkpoint.ipynb', 'repo_name': 'letslego/W266-ProjectMIMIC', 'size': '13947'}