<jupyter_start><jupyter_text>Download Dataset: link<jupyter_code>!curl --output dataset.zip https://archive.ics.uci.edu/ml/machine-learning-databases/00462/drugsCom_raw.zip
!unzip dataset.zip
!rm dataset.zip<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 40.9M  100 40.9M    0     0  12.1M      0  0:00:03  0:00:03 --:--:-- 12.1M
Archive:  dataset.zip
replace drugsComTest_raw.tsv? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
  inflating: drugsComTest_raw.tsv    
replace drugsComTrain_raw.tsv? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
  inflating: drugsComTrain_raw.tsv<jupyter_text>Imports<jupyter_code>import pandas as pd
import numpy as np
import html<jupyter_output><empty_output><jupyter_text>Processing to Tokens  Read dataset<jupyter_code>import pandas as pd
import html

def load_dataset(filename):
    df = pd.read_csv(filename, sep='\t')
    df.drop(df.columns[0], axis=1, inplace=True)
    df.drop(['drugName', 'condition', 'date', 'usefulCount'], axis=1, inplace=True)
    df['review'] = df['review'].apply(lambda x: x[1:-1])
    df['review'] = df['review'].apply(html.unescape)
    df['rating'] = df['rating'].astype('int64')
    df['label'] = df['rating']
    df.label[df['rating'] <= 4] = 0
    df.label[(df['rating'] <= 6) & (df['rating'] >= 5)] = 1
    df.label[df['rating'] >= 7] = 2
    df.drop(['rating'], axis=1, inplace=True)
    return df

df_train = load_dataset("drugsComTrain_raw.tsv")
df_test = load_dataset("drugsComTest_raw.tsv")<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:12: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  if sys.path[0] == '':
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:13: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  del sys.path[0]
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:14: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy<jupyter_text>Removing Stopwords and Tokenizing the reviews<jupyter_code>import nltk
from nltk.tokenize import word_tokenize,RegexpTokenizer
from nltk.corpus import stopwords 

nltk.download('stopwords')
stop_words = set(stopwords.words('english'))
tokenizer = RegexpTokenizer(r'\w+')
def func(row):
    l = tokenizer.tokenize(row['review'])
    s = [i.lower() for i in l if i not in stop_words]
    return s
df_train['processed'] = df_train.apply (lambda row: func(row), axis=1)
df_test['processed'] = df_test.apply (lambda row: func(row), axis=1)
df_train[1:6]<jupyter_output><empty_output><jupyter_text>Remove very large sentences if needed<jupyter_code>df_train['processed_len'] = df_train.processed.apply(len)
df_test['processed_len'] = df_test.processed.apply(len)
df_train = df_train[df_train['processed_len'] <= 100].reset_index(drop=True)
df_test = df_test[df_train['processed_len'] <= 100].reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Embedding the Tokens  Glove embedings  Download Glove embeddings<jupyter_code>!wget "http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip"
!unzip glove.6B.zip
!rm glove.6B.zip<jupyter_output>--2019-11-01 19:40:02--  http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip
Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 172.16.31.10
Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|172.16.31.10|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 862182613 (822M) [application/zip]
Saving to: ‘glove.6B.zip’

glove.6B.zip        100%[===================>] 822.24M  2.10MB/s    in 6m 29s  

2019-11-01 19:46:31 (2.11 MB/s) - ‘glove.6B.zip’ saved [862182613/862182613]

Archive:  glove.6B.zip
  inflating: glove.6B.50d.txt        
  inflating: glove.6B.100d.txt       
  inflating: glove.6B.200d.txt       
  inflating: glove.6B.300d.txt<jupyter_text>Util functions<jupyter_code>def embed(processed, w2v_dict, default_vec):
    vectors = []
    for w in processed:
        if(w in w2v_dict):
            vectors.append(w2v_dict[w])
        else:
            vectors.append(default_vec)
    vectors = np.array(vectors)
    return vectors

def embed_using_glove(df, embed_dim=50):
    # Load Glove embeddings.
    GLOVE_FILE = 'glove.6B.' + str(embed_dim) + 'd.txt'
    # Get number of vectors and hidden dimensions
    with open(GLOVE_FILE, 'r') as f:
        for i, line in enumerate(f):
            pass
    n_vec = i + 1
    hidden_dim = len(line.split(' ')) - 1
    # Find the average of all embeddings to be assigned to tokens not in the embeddings.
    # Create a word to vector dictionary for embedding.
    avg_vec = np.zeros((hidden_dim), dtype=np.float32)
    w2v_dict = {}
    count = 0
    with open(GLOVE_FILE, 'r') as f:
        for line in f:
            w2v_dict[line.split(' ')[:1][0]] = np.array([float(n) for n in line.split(' ')[1:]], dtype=np.float32)
            avg_vec += np.array([float(n) for n in line.split(' ')[1:]], dtype=np.float32)
            count+=1
    avg_vec/=count

    df['vectors'] = df.apply(lambda row: embed(row['processed'], w2v_dict, avg_vec), axis=1)<jupyter_output><empty_output><jupyter_text>Convert tokens in to Glove embeddings and save them to file.<jupyter_code>embed_using_glove(df_train, 50)
df_train.to_pickle("train_glove_50.pckl")
embed_using_glove(df_test, 50)
df_test.to_pickle("test_glove_50.pckl")<jupyter_output><empty_output><jupyter_text>Word2Vec embeddings  Download the Google word2vec embeddings<jupyter_code>!gdown --id 0B7XkCwpI5KDYNlNUTTlSS21pQmM -O word2vec.bin.gz
!gunzip word2vec.bin.gz<jupyter_output>Downloading...
From: https://drive.google.com/uc?id=0B7XkCwpI5KDYNlNUTTlSS21pQmM
To: /content/word2vec.bin.gz
1.65GB [00:10, 162MB/s]<jupyter_text>Load the word2vec embeddings into memory<jupyter_code>from gensim.models import KeyedVectors
    
    w2v_model = KeyedVectors.load_word2vec_format('word2vec.bin', binary=True)<jupyter_output>/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL<jupyter_text>Util funcitons<jupyter_code>def embed(processed, w2v_dict, default_vec):
    vectors = []
    for w in processed:
        if(w in w2v_dict):
            vectors.append(w2v_dict[w])
        else:
            vectors.append(default_vec)
    vectors = np.array(vectors)
    return vectors<jupyter_output><empty_output><jupyter_text>Get smaller versions of train and test dataset<jupyter_code>df_train_w2v = df_train.sample(n=int(df_train.shape[0]/6)).reset_index(drop=True)
df_test_w2v = df_test.sample(n=int(df_test.shape[0]/6)).reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Convert the tokens into Word2Vec embeddings and save them to file<jupyter_code>w2v_avg = np.average(w2v_model.vectors, axis=0)
df_train_w2v['vectors'] = df_train_w2v.apply(lambda row: embed(row['processed'], w2v_model, w2v_avg), axis=1)
df_test_w2v['vectors'] = df_test_w2v.apply(lambda row: embed(row['processed'], w2v_model, w2v_avg), axis=1)
df_train_w2v.memory_usage(deep=True)<jupyter_output><empty_output><jupyter_text>Elmo embeddings  Install allennlp<jupyter_code>!pip install allennlp<jupyter_output><empty_output><jupyter_text>Imports<jupyter_code>from allennlp.commands.elmo import ElmoEmbedder
import sys<jupyter_output><empty_output><jupyter_text>Util functions<jupyter_code>def create_elmo_embeddings(elmo, documents, max_sentences = 1000):
    num_sentences = min(max_sentences, len(documents)) if max_sentences > 0 else len(documents)
    print("\n\n:: Lookup of " + str(num_sentences) + " ELMo representations. This takes a while ::")
    embeddings = []
    tokens = documents['processed'].to_numpy()
    
    documentIdx = 0
    for elmo_embedding in elmo.embed_sentences(tokens):  
        document = documents.iloc[documentIdx]
        # Average the 3 layers returned from ELMo
        avg_elmo_embedding = np.average(elmo_embedding, axis=0)             
        embeddings.append(avg_elmo_embedding)        
        
        # Some progress info
        documentIdx += 1
        percent = 100.0 * documentIdx / num_sentences
        line = '[{0}{1}]'.format('=' * int(percent / 2), ' ' * (50 - int(percent / 2)))
        status = '\r{0:3.0f}%{1} {2:3d}/{3:3d} sentences'
        sys.stdout.write(status.format(percent, line, documentIdx, num_sentences))
        
        if max_sentences > 0 and documentIdx >= max_sentences:
            break
            
    return embeddings<jupyter_output><empty_output><jupyter_text>Get smaller versions of train and test dataset<jupyter_code>df_train_elmo = df_train.sample(n=int(df_train.shape[0]/20)).reset_index(drop=True)
df_test_elmo = df_test.sample(n=int(df_test.shape[0]/20)).reset_index(drop=True)<jupyter_output>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  after removing the cwd from sys.path.<jupyter_text>Get ELMo embeddings<jupyter_code>elmo = ElmoEmbedder(cuda_device=0) #Set cuda_device to the ID of your GPU if you have one
df_train_elmo['elmo_embeddings'] = create_elmo_embeddings(elmo, df_train_elmo, -1)
df_test_elmo['elmo_embeddings'] = create_elmo_embeddings(elmo, df_test_elmo, -1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13060409385921118, 'fraction_numerical': 0.03255117324013979, 'mean_word_length': 4.542888765910349, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 73, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '569911', 'license': 'MIT', 'n_tokens_mistral': 3922, 'n_tokens_neox': 3685, 'n_words': 701, 'path': 'Pre_Processing.ipynb', 'repo_name': 'Rajesh-Khanna/NLP-Drug-review-classification', 'size': '10015'}