<jupyter_start><jupyter_text>Deep Learning based Pipeline with Multichannel Inputs for Multi-class Patent ClassificationThis notebook describes a deep learning pipeline for automatic patent classification with multichannel inputs.  A neural network model is trained with multichannel inputs namely embeddings of different segments of patent texts, and sparse linearinput of different metadata.  In this notebook the classification task is a multi-class classification. The basic outline is:    - load the patent dataset  - apply preprocessing tasks  - apply Tokenization process  - Load a pretrained word embeddings model  - prepare the embedding matrix for patent texts   - concatenated deep layers - train a deep neural network on the data  - Fit the model and show the results     Loading patent datasetWe established the training, and test datasets that are related to the information technology domain as well as life science domain.The total number of extracted records in the datasets is 1,915,308  patents filed between 1978 and 2016. Each patent document contains a  patent number, issued date, patent type, and list of citations, classification codes, a list of inventors, a list of assignees, title, abstract, technical field, background,  summary of invention, and independent claim. The data det is availabe on request.<jupyter_code>import pandas as pd

df1 = pd.read_csv("../datasets/allITPatTextWith_Metadata_simple_preprocessed_last.csv",  encoding = "ISO-8859-1", error_bad_lines=False)
df1.columns =['ID','TI','AB','TECHF','BACKG','SUMM','CLMS','ICM','AY','IPC','REF','PA','INV']

df1.dropna(subset=['ICM'], inplace=True)


df1.fillna(value='', inplace=True)
print(df1.count())


df2 = pd.read_csv("../datasets/allLSPatTextWith_Metadata_preprocessed.csv",  encoding = "utf-8", error_bad_lines=False)
df2.columns =['ID','TI','AB','TECHF','BACKG','SUMM','CLMS','ICM','AY','IPC','REF','PA','INV']

df2.dropna(subset=['ICM'], inplace=True)

 

df2.fillna(value='', inplace=True)
print(df2.count())

df = pd.concat([df1, df2], ignore_index=True)
print(df.count())
df.head()
import matplotlib.pyplot as plt

df_AY = df.groupby(['AY'])
df_AY = df_AY.size().reset_index(name='Docs')


df_AY.plot(x='AY', y='Docs', kind='bar', legend=False, grid=True, figsize=(15, 6))
plt.title("Number of Docs per Year")
plt.ylabel('# of Patents', fontsize=13)
plt.xlabel('YEAR', fontsize=12)<jupyter_output><empty_output><jupyter_text>Applying preprocessing tasks on metadat of patentConverting the metatadata such as inventors and assignees of each patent into a python list, then apply preprocessing task on each element in the list in order to remove undesired tokens.<jupyter_code>%%time
#preprocess of list fields
#convert all IPCs in df into one list
def toList(s):
    """
    this method is to convert the list of IPCs in each row from a string to a python List
    """
    s  = s.translate ({ord(c): " " for c in "[]"})
    ss= []
    for cls in s.strip().split(','):
        ss.append(cls.strip())
    return ss

#apply toList method on all rows in the DF
df['PA'] = df['PA'].map(lambda pa :   toList(pa))
df['INV'] = df['INV'].map(lambda inv :   toList(inv))

df.head()  
%%time

def metadataPreprocessing(input):
    newInput=' '
    for item in input:
        item = item.translate ({ord(c): " " for c in "!@#$%^&*()'[]{};:,./<>?\|`~°=\"+"})
        itms=' '
        for itm in item.split():
            itms= itms +' '+itm.strip()
        newInput = newInput + ' '+ itms.strip().replace(' ','_')
    return newInput.strip()

df['PA'] = df['PA'].map(lambda pa :   metadataPreprocessing(pa))
df['INV'] = df['INV'].map(lambda inv :   metadataPreprocessing(inv))

df.head()<jupyter_output>CPU times: user 29.8 s, sys: 349 ms, total: 30.2 s
Wall time: 30.2 s<jupyter_text>Applying preprocessing tasks on texts of patentA simple preprocessing tasks such as tokenization, stopword removal, lemmatization, and converting letters into lower case are performed on each text section of each patent document.<jupyter_code>#preprocessing 
standardStopwordFile = "sources/stopwords/stopwords-all.txt"

#loading terms from a file to a set
def get_terms_from_file(filePath):
    terms = set(line.strip() for line in open(filePath))
    return terms

#remove undiserd terms
def remove_terms(termSet, phrase):
    newPhrase = ""
    for term in phrase.split():
        if term.strip() not in termSet and len(term.strip())>2:
            newPhrase = newPhrase + " " + term.strip()



def clean_texts(doc):
    #Remove punctuation from texts
    doc = doc.translate ({ord(c): ' ' for c in "0123456789!@#$%^&*()'/[]{};:,./<>?\|`~°=\"+"})
    # split into tokens by white space
    tokens = doc.lower().strip().split()
    
    # filter out stop words
    stop_words = get_terms_from_file(standardStopwordFile)
    #generalStopwords = get_terms_from_file(generalWordsFile)

    
    tokens = [w.strip('-')  for w in tokens if  w not in stop_words ]
    # filter out short and long  tokens
    output = [word for word in tokens if len(word.strip()) > 2 and len(word) < 30 ]
    output = " ".join(output)
    #apply stemming
    #output = stem_text(output)
    return output


%%time
#apply simple preprocessing on text
df['TI'] = df['TI'].map(lambda line : clean_texts(line))
df['AB'] = df['AB'].map(lambda line : clean_texts(line))
df['TECHF'] = df['TECHF'].map(lambda line : clean_texts(line))
df['BACKG'] = df['BACKG'].map(lambda line : clean_texts(line))
df['SUMM'] = df['SUMM'].map(lambda line : clean_texts(line))
df['CLMS'] = df['CLMS'].map(lambda line : clean_texts(line))


df.head()<jupyter_output>CPU times: user 253 µs, sys: 25 µs, total: 278 µs
Wall time: 276 µs<jupyter_text>Applying preprocessing tasks for patent labels (main IPC codes)The main IPC codes is considered to be the labels for the patent documents. we only consider the subclass level of the IPC code. Each label/class has at least more than 500 documents.<jupyter_code>#process the ICM codes and #related-patents
df['ICM'] = df['ICM'].map(lambda icmCode : icmCode[:4])  

df = df.groupby(['ICM']).head(20000)


df_ICMs = df.groupby(['ICM'])
df_ICMs = df_ICMs.size().reset_index(name='Docs')

print(len(df_ICMs.ICM.unique()))
#filter out the rows with #docs less than N documents
df_ICMOut =  df_ICMs[df_ICMs['Docs'] >= 20000]

#filter out rows of the original dataframe df accordding to df_ICMOut
ICMList = df_ICMOut['ICM'].tolist()
df = df[df.ICM.isin(ICMList)]

icmCount = df_ICMs.count().tolist()[0]

print( 'number of remaining documents in the dataset is: ',len(df))

print('Number of unique labels is: ', len(df.ICM.unique()))<jupyter_output>592
number of remaining documents in the dataset is:  440000
Number of unique labels is:  22<jupyter_text>Randomly reorder a dataset by rows<jupyter_code>from sklearn.utils import shuffle

df = shuffle(df)

df.head()<jupyter_output><empty_output><jupyter_text>split the dataset into train and test datasets<jupyter_code># lets take n% data as training and remaining m% for test.
train_size = int(len(df) * .97)

train_TI = df['TI'][:train_size]
train_AB = df['AB'][:train_size]
train_TECHF = df['TECHF'][:train_size]
train_BACKG = df['BACKG'][:train_size]
train_SUMM = df['SUMM'][:train_size]
train_CLMS = df['CLMS'][:train_size]
train_ICM= df['ICM'][:train_size]
train_ID= df['ID'][:train_size]

test_TI = df['TI'][train_size:]
test_AB = df['AB'][train_size:]
test_TECHF = df['TECHF'][train_size:]
test_BACKG = df['BACKG'][train_size:]
test_SUMM = df['SUMM'][train_size:]
test_CLMS = df['CLMS'][train_size:]
test_ICM = df['ICM'][train_size:]
test_ID = df['ID'][train_size:]


#metadata
train_pa_series = df['PA'][:train_size]
test_pa_series = df['PA'][train_size:]

train_inv_series = df['INV'][:train_size]
test_inv_series = df['INV'][train_size:]


print(train_AB.shape)
print(test_AB.shape)

#free up some memory space
#df.iloc[0:0]<jupyter_output>(426800,)
(13200,)<jupyter_text>Applying tokenization process For texts of each segment, a Keras tokenization process is used for breaking the text into individual words, and  set the sequence length of each segment according to the length of each.<jupyter_code>from keras.preprocessing.text import Tokenizer
from keras.utils import to_categorical
from keras.preprocessing.sequence import pad_sequences
from keras.preprocessing.text import one_hot
from sklearn.preprocessing import LabelBinarizer<jupyter_output>Using TensorFlow backend.<jupyter_text>Applying Keras tokenization on Metadata of patent(Inventors, Assignees), and convert the related text into One-hot that encodes a text into a list of word indexes of size n.<jupyter_code>#PA
pa_inv_vocab_size = 5000
pa_tokenizer = Tokenizer(num_words=pa_inv_vocab_size,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~', lower=True, split=' ', char_level=False, oov_token=None)
pa_tokenizer.fit_on_texts(train_pa_series)
train_pa_one_hot =pa_tokenizer.texts_to_matrix(train_pa_series)
test_pa_one_hot =pa_tokenizer.texts_to_matrix(test_pa_series)


#INV
inv_tokenizer = Tokenizer(num_words=pa_inv_vocab_size,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~', lower=True, split=' ', char_level=False, oov_token=None)
inv_tokenizer.fit_on_texts(train_inv_series)
train_inv_one_hot =inv_tokenizer.texts_to_matrix(train_inv_series)
test_inv_one_hot =inv_tokenizer.texts_to_matrix(test_inv_series)


print('Found %s words in PA' % len(pa_tokenizer.word_index))
print('Found %s words in INV' % len(inv_tokenizer.word_index))<jupyter_output>Found 136600 words in PA
Found 355629 words in INV<jupyter_text>-  Transform each text in Titles (train and test datasets) into a sequence of integers. -  set the sequence length.-  Pads sequences to the same length.<jupyter_code>%%time

#Title
TI_tokenizer = Tokenizer(num_words=5000,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)
TI_tokenizer.fit_on_texts(train_TI)
encoded_train_TI = TI_tokenizer.texts_to_sequences(train_TI)
encoded_test_TI = TI_tokenizer.texts_to_sequences(test_TI)
#convert all sequences in a list into the same length
TI_train = pad_sequences(encoded_train_TI,  maxlen=20, padding='post')
TI_test = pad_sequences(encoded_test_TI,  maxlen=20, padding='post')<jupyter_output>CPU times: user 14.5 s, sys: 174 ms, total: 14.6 s
Wall time: 14.6 s<jupyter_text>Transform each text in Abstrcat (train and test datasets) into a sequence of integers. set the sequence length.Pads sequences to the same length.<jupyter_code>%%time
#Abstract
AB_tokenizer = Tokenizer(num_words=20000,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)
AB_tokenizer.fit_on_texts(train_AB)
encoded_train_AB = AB_tokenizer.texts_to_sequences(train_AB)
encoded_test_AB = AB_tokenizer.texts_to_sequences(test_AB)
#convert all sequences in a list into the same length
AB_train = pad_sequences(encoded_train_AB,  maxlen=100, padding='post')
AB_test = pad_sequences(encoded_test_AB,  maxlen=100, padding='post')<jupyter_output>CPU times: user 39.9 s, sys: 515 ms, total: 40.4 s
Wall time: 40.5 s<jupyter_text>Transform each text in Technical Field (train and test datasets) into a sequence of integers. set the sequence length.Pads sequences to the same length.<jupyter_code>%%time
#TECHNICAL_FIELD
TECHF_tokenizer = Tokenizer(num_words=10000,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)
TECHF_tokenizer.fit_on_texts(train_TECHF)
encoded_train_TECHF = TECHF_tokenizer.texts_to_sequences(train_TECHF)
encoded_test_TECHF = TECHF_tokenizer.texts_to_sequences(test_TECHF)
#convert all sequences in a list into the same length
TECHF_train = pad_sequences(encoded_train_TECHF,  maxlen=30, padding='post')
TECHF_test = pad_sequences(encoded_test_TECHF,  maxlen=30, padding='post')<jupyter_output>CPU times: user 34.9 s, sys: 296 ms, total: 35.2 s
Wall time: 35.3 s<jupyter_text>Transform each text in Background (train and test datasets) into a sequence of integers. set the sequence length.Pads sequences to the same length.<jupyter_code>%%time
#BACKGROUND
BACKG_tokenizer = Tokenizer(num_words=20000,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)
BACKG_tokenizer.fit_on_texts(train_BACKG)
encoded_train_BACKG = BACKG_tokenizer.texts_to_sequences(train_BACKG)
encoded_test_BACKG = BACKG_tokenizer.texts_to_sequences(test_BACKG)
#convert all sequences in a list into the same length
BACKG_train = pad_sequences(encoded_train_BACKG,  maxlen=100, padding='post')
BACKG_test = pad_sequences(encoded_test_BACKG,  maxlen=100, padding='post')<jupyter_output>CPU times: user 4min 23s, sys: 3.54 s, total: 4min 27s
Wall time: 4min 28s<jupyter_text>Transform each text in Summary (train and test datasets) into a sequence of integers. set the sequence length.Pads sequences to the same length.<jupyter_code>%%time
#SUMMARY
SUMM_tokenizer = Tokenizer(num_words=20000,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)
SUMM_tokenizer.fit_on_texts(train_SUMM)
encoded_train_SUMM = SUMM_tokenizer.texts_to_sequences(train_SUMM)
encoded_test_SUMM = SUMM_tokenizer.texts_to_sequences(test_SUMM)
#convert all sequences in a list into the same length
SUMM_train = pad_sequences(encoded_train_SUMM,  maxlen=100, padding='post')
SUMM_test = pad_sequences(encoded_test_SUMM,  maxlen=100, padding='post')<jupyter_output>CPU times: user 7min 21s, sys: 5.4 s, total: 7min 26s
Wall time: 7min 27s<jupyter_text>Transform each text in Independent Claim (train and test datasets) into a sequence of integers. set the sequence length.Pads sequences to the same length.<jupyter_code>%%time
#CLAIMS
CLMS_tokenizer = Tokenizer(num_words=20000,  filters='!"#$%&()*+,./:;<=>?@[\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)
CLMS_tokenizer.fit_on_texts(train_CLMS)
encoded_train_CLMS = CLMS_tokenizer.texts_to_sequences(train_CLMS)
encoded_test_CLMS = CLMS_tokenizer.texts_to_sequences(test_CLMS)
#convert all sequences in a list into the same length
CLMS_train = pad_sequences(encoded_train_CLMS,  maxlen=100, padding='post')
CLMS_test = pad_sequences(encoded_test_CLMS,  maxlen=100, padding='post')<jupyter_output>CPU times: user 48.4 s, sys: 1.36 s, total: 49.7 s
Wall time: 49.8 s<jupyter_text>Pre-processing Output Labels / Classes we  need to make sure our labels are represented in the numeric format accepted by neural network model.  We need to convert our labels to one hot vector. scikit-learn has a LabelBinarizer class which makes it easy to build these one-hot vectors.<jupyter_code>%%time
# 
encoder = LabelBinarizer()
encoder.fit(train_ICM)
y_train = encoder.transform(train_ICM)
y_test = encoder.transform(test_ICM)

#get the unique number of labels in the training set
classesList = train_ICM.tolist()
classesList =set(classesList)
num_classes = len(classesList)<jupyter_output>CPU times: user 3.77 s, sys: 57 ms, total: 3.82 s
Wall time: 3.82 s<jupyter_text>load the whole embeddings model into memory and get matrixWe load a pre-trained word2vec word embedding model that was trained on five million patents (Titles and abstracts)The Embeddings model is availabel  here.<jupyter_code>import numpy as np

def load_embedding_model(filePath):
    embeddings_index = dict()
    f = open(filePath, encoding='utf8')
    for line in f:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        embeddings_index[word] = coefs
        
    return embeddings_index

def create_embedding_matrix(tokenizer, embeddings_index, vocab_size_embbs, dim_size):
    embeddings_matrix = np.zeros((vocab_size_embbs, dim_size))
    for word, i in tokenizer.word_index.items():
        embedding_vector = embeddings_index.get(word)
        if embedding_vector is not None:
            embeddings_matrix[i] = embedding_vector[0:dim_size]
    
    return embeddings_matrix<jupyter_output><empty_output><jupyter_text>Loading the whole embedding into memory and get matrix<jupyter_code>%%time

embedding_index = load_embedding_model('../models/w2v/phrase/patWordPhrase2VecModel.txt')<jupyter_output>CPU times: user 1min 43s, sys: 3.46 s, total: 1min 46s
Wall time: 1min 47s<jupyter_text>Creating TITLE embeddings Matrix<jupyter_code>%%time

#vocab_size for embedding
vocab_size_embb = len(TI_tokenizer.word_index) + 1

TI_embeddings_matrix = create_embedding_matrix(TI_tokenizer,
                                              embedding_index,
                                              vocab_size_embb,
                                              20)<jupyter_output>CPU times: user 183 ms, sys: 144 ms, total: 327 ms
Wall time: 327 ms<jupyter_text>Creating ABSTRACT embeddings Matrix<jupyter_code>%%time

#vocab_size for embedding
vocab_size_embb = len(AB_tokenizer.word_index) + 1
AB_embeddings_matrix = create_embedding_matrix(AB_tokenizer,
                                              embedding_index,
                                              vocab_size_embb,
                                              100)<jupyter_output>CPU times: user 450 ms, sys: 221 ms, total: 670 ms
Wall time: 671 ms<jupyter_text>Creating TECHNICAL_FIELD embeddings Matrix<jupyter_code>%%time

#vocab_size for embedding
vocab_size_embb = len(TECHF_tokenizer.word_index) + 1
TECHF_embeddings_matrix = create_embedding_matrix(TECHF_tokenizer,
                                              embedding_index,
                                              vocab_size_embb,
                                              30)<jupyter_output>CPU times: user 363 ms, sys: 71.9 ms, total: 435 ms
Wall time: 435 ms<jupyter_text>Creating BACKGROUND embeddings Matrix<jupyter_code>%%time

#vocab_size for embedding
vocab_size_embb = len(BACKG_tokenizer.word_index) + 1
BACKG_embeddings_matrix = create_embedding_matrix(BACKG_tokenizer,
                                              embedding_index,
                                              vocab_size_embb,
                                              100)<jupyter_output>CPU times: user 1.25 s, sys: 361 ms, total: 1.61 s
Wall time: 1.61 s<jupyter_text>Creating SUMMARY embeddings Matrix<jupyter_code>%%time

#vocab_size for embedding
vocab_size_embb = len(SUMM_tokenizer.word_index) + 1
SUMM_embeddings_matrix = create_embedding_matrix(SUMM_tokenizer,
                                              embedding_index,
                                              vocab_size_embb,
                                              100)<jupyter_output>CPU times: user 1.39 s, sys: 343 ms, total: 1.74 s
Wall time: 1.74 s<jupyter_text>Creating INDEPENDENT CLAIMS embeddings Matrix<jupyter_code>%%time

#vocab_size for embedding
vocab_size_embb = len(CLMS_tokenizer.word_index) + 1
CLMS_embeddings_matrix = create_embedding_matrix(CLMS_tokenizer,
                                              embedding_index,
                                              vocab_size_embb,
                                              100)<jupyter_output>CPU times: user 371 ms, sys: 81.8 ms, total: 453 ms
Wall time: 453 ms<jupyter_text>Creating a Deep Layer for each Input<jupyter_code>import keras
from keras.models import Sequential, Model
from keras.layers import Dense, Input, Embedding, BatchNormalization, ELU, Concatenate
from keras.layers import LSTM, Conv1D, MaxPooling1D
from keras.layers.merge import concatenate
from keras.layers.core import Dropout<jupyter_output><empty_output><jupyter_text>Creating LSTM deep layer for Title Embeddings<jupyter_code>%%time
#TITLE 
sequence_len =20
dropout_pct =  0.3

TI_embedding_layer_input = Input(shape=(sequence_len,), name='TI_embed_input')
TI_embedding_layer = Embedding(input_dim=len(TI_tokenizer.word_index) + 1,
                        output_dim=20, # Dimension of the dense embedding
                        weights=[TI_embeddings_matrix],
                        input_length=20)(TI_embedding_layer_input)

lstm_size = 64
TI_deep = LSTM(lstm_size,
            dropout=dropout_pct,
            recurrent_dropout=dropout_pct,
            return_sequences=False,
            name='LSTM_TI')(TI_embedding_layer)

TI_deep = Dense(300, activation=None)(TI_deep)
TI_deep = Dropout(dropout_pct)(TI_deep)
TI_deep = BatchNormalization()(TI_deep)
TI_deep = ELU()(TI_deep)<jupyter_output>CPU times: user 4.71 s, sys: 9.21 s, total: 13.9 s
Wall time: 4.76 s<jupyter_text>Creating LSTM deep layer for Abstract Embeddings<jupyter_code>%%time
#Abstract 
sequence_len =100
dropout_pct =  0.30

AB_embedding_layer_input = Input(shape=(sequence_len,), name='AB_embed_input')
AB_embedding_layer = Embedding(input_dim=len(AB_tokenizer.word_index) + 1,
                        output_dim=100, # Dimension of the dense embedding
                        weights=[AB_embeddings_matrix],
                        input_length=100)(AB_embedding_layer_input)

lstm_size = 64
AB_deep = LSTM(lstm_size,
            dropout=dropout_pct,
            recurrent_dropout=dropout_pct,
            return_sequences=False,
            name='LSTM_AB')(AB_embedding_layer)

AB_deep = Dense(300, activation=None)(AB_deep)
AB_deep = Dropout(dropout_pct)(AB_deep)
AB_deep = BatchNormalization()(AB_deep)
AB_deep = ELU()(AB_deep)<jupyter_output>CPU times: user 3.19 s, sys: 6.95 s, total: 10.1 s
Wall time: 1.06 s<jupyter_text>Creating LSTM deep layer for TECHNICAL-Field Embeddings<jupyter_code>%%time
#TECHNICAL-Field 
sequence_len =30
dropout_pct =  0.30

TECHF_embedding_layer_input = Input(shape=(sequence_len,), name='TECHF_embed_input')
TECHF_embedding_layer = Embedding(input_dim=len(TECHF_tokenizer.word_index) + 1,
                        output_dim=30, # Dimension of the dense embedding
                        weights=[TECHF_embeddings_matrix],
                        input_length=30)(TECHF_embedding_layer_input)

lstm_size = 64
TECHF_deep = LSTM(lstm_size,
            dropout=dropout_pct,
            recurrent_dropout=dropout_pct,
            return_sequences=False,
            name='LSTM_TECHF')(TECHF_embedding_layer)

TECHF_deep = Dense(300, activation=None)(TECHF_deep)
TECHF_deep = Dropout(dropout_pct)(TECHF_deep)
TECHF_deep = BatchNormalization()(TECHF_deep)
TECHF_deep = ELU()(TECHF_deep)<jupyter_output>CPU times: user 3.25 s, sys: 6.64 s, total: 9.88 s
Wall time: 886 ms<jupyter_text>Creating LSTM deep layer for BACKGROUND Embeddings<jupyter_code>%%time
#BACKGROUND 
sequence_len =100
dropout_pct =  0.40

BACKG_embedding_layer_input = Input(shape=(sequence_len,), name='BACKG_embed_input')
BACKG_embedding_layer = Embedding(input_dim=len(BACKG_tokenizer.word_index) + 1,
                        output_dim=100, # Dimension of the dense embedding
                        weights=[BACKG_embeddings_matrix],
                        input_length=100)(BACKG_embedding_layer_input)

lstm_size = 64
BACKG_deep = LSTM(lstm_size,
            dropout=dropout_pct,
            recurrent_dropout=dropout_pct,
            return_sequences=False,
            name='LSTM_BACK')(BACKG_embedding_layer)

BACKG_deep = Dense(300, activation=None)(BACKG_deep)
BACKG_deep = Dropout(dropout_pct)(BACKG_deep)
BACKG_deep = BatchNormalization()(BACKG_deep)
BACKG_deep = ELU()(BACKG_deep)<jupyter_output>CPU times: user 4.05 s, sys: 8.78 s, total: 12.8 s
Wall time: 3.65 s<jupyter_text>Creating LSTM deep layer for Summary Embeddings<jupyter_code>%%time
#SUMMARY
sequence_len =100
dropout_pct =  0.30

SUMM_embedding_layer_input = Input(shape=(sequence_len,), name='SUMM_embed_input')
SUMM_embedding_layer = Embedding(input_dim=len(SUMM_tokenizer.word_index) + 1,
                        output_dim=100, # Dimension of the dense embedding
                        weights=[SUMM_embeddings_matrix],
                        input_length=100)(SUMM_embedding_layer_input)

lstm_size = 64
SUMM_deep = LSTM(lstm_size,
            dropout=dropout_pct,
            recurrent_dropout=dropout_pct,
            return_sequences=False,
            name='LSTM_SUMM')(SUMM_embedding_layer)

SUMM_deep = Dense(300, activation=None)(SUMM_deep)
SUMM_deep = Dropout(dropout_pct)(SUMM_deep)
SUMM_deep = BatchNormalization()(SUMM_deep)
SUMM_deep = ELU()(SUMM_deep)<jupyter_output>CPU times: user 4.14 s, sys: 7.87 s, total: 12 s
Wall time: 2.87 s<jupyter_text>Creating LSTM deep layer for Independent Claim Embeddings<jupyter_code>%%time
#CLAIMS 
sequence_len =100
dropout_pct =  0.30


CLMS_embedding_layer_input = Input(shape=(sequence_len,), name='CLMS_embed_input')
CLMS_embedding_layer = Embedding(input_dim=len(CLMS_tokenizer.word_index) + 1,
                        output_dim=100, # Dimension of the dense embedding
                        weights=[CLMS_embeddings_matrix],
                        input_length=100)(CLMS_embedding_layer_input)

lstm_size = 64
CLMS_deep = LSTM(lstm_size,
            dropout=dropout_pct,
            recurrent_dropout=dropout_pct,
            return_sequences=False,
            name='LSTM_CLMS')(CLMS_embedding_layer)

CLMS_deep = Dense(300, activation=None)(CLMS_deep)
CLMS_deep = Dropout(dropout_pct)(CLMS_deep)
CLMS_deep = BatchNormalization()(CLMS_deep)
CLMS_deep = ELU()(CLMS_deep)<jupyter_output>CPU times: user 3.58 s, sys: 6.73 s, total: 10.3 s
Wall time: 1.28 s<jupyter_text>Creating LSTM deep layers for one-hot vectors of Inventors and Assignees<jupyter_code>dropout_pct =  0.3
pa_input = Input(shape=(train_pa_one_hot.shape[1],), name='pa_input') 
pas = Dense(32,input_dim=train_pa_one_hot.shape[1], activation=None)(pa_input) 
pas = Dropout(dropout_pct)(pas)
pas = BatchNormalization()(pas)
pas = ELU()(pas)

#inv
inv_input = Input(shape=(train_inv_one_hot.shape[1],), name='inv_input') 
invs = Dense(32,input_dim=train_inv_one_hot.shape[1], activation=None)(pa_input) 
invs = Dropout(dropout_pct)(invs)
invs = BatchNormalization()(invs)

print('pa_input and inv_input layers are finished')<jupyter_output>pa_input and inv_input layers are finished<jupyter_text>Train ModelThe following cells specify the neural network architecture and hyperparameters. The model is generally composed of:- contacting sequential word embeddings ofpatent text segments into a fully-connected layer-  categorical_accuracy is used for calculatinges the mean accuracy rate across all predictions for multiclass classification problems - Compile the  the Network.<jupyter_code>import keras_metrics as km
#contacting 
model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep, BACKG_deep, SUMM_deep, CLMS_deep] 
final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)

output = Dense(128, activation=None)(final_layer)
output = Dropout(dropout_pct)(output)
output = BatchNormalization()(output)
output = ELU()(output)
output = Dense(num_classes, activation='softmax')(output)

model = Model(inputs=[TI_embedding_layer_input,
                      AB_embedding_layer_input,
                      TECHF_embedding_layer_input,
                      BACKG_embedding_layer_input,
                     SUMM_embedding_layer_input,
                     CLMS_embedding_layer_input,
                     ],
              outputs=output, name='model')

from keras.utils import multi_gpu_model
parallel_model = multi_gpu_model(model, gpus=3)

parallel_model.compile(loss='categorical_crossentropy',
                      optimizer='adam',
                      metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])
model.summary()<jupyter_output>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
TI_embed_input (InputLayer)     (None, 20)           0                                            
__________________________________________________________________________________________________
AB_embed_input (InputLayer)     (None, 100)          0                                            
__________________________________________________________________________________________________
TECHF_embed_input (InputLayer)  (None, 30)           0                                            
__________________________________________________________________________________________________
BACKG_embed_input (InputLayer)  (None, 100)          0                                            
__________[...]<jupyter_text>Train / Fit the Network<jupyter_code>%%time
from keras.utils import multi_gpu_model

batch_size= 1000 
num_epochs = 20

history = parallel_model.fit(x={'TI_embed_input': TI_train,
                       'AB_embed_input': AB_train,
             'TECHF_embed_input': TECHF_train,
             'BACKG_embed_input': BACKG_train,
             'SUMM_embed_input': SUMM_train,
             'CLMS_embed_input': CLMS_train
             
            },
          y=y_train,
          batch_size=batch_size,
          epochs=num_epochs,
          validation_data=
          ({'TI_embed_input': TI_test,
            'AB_embed_input': AB_test,
            'TECHF_embed_input': TECHF_test,
             'BACKG_embed_input': BACKG_test,
             'SUMM_embed_input': SUMM_test,
            'CLMS_embed_input': CLMS_test
            },
           y_test))

import matplotlib.pyplot as plt
def plot_history(history):
    loss_list = [s for s in history.history.keys() if 'loss' in s and 'val' not in s]
    val_loss_list = [s for s in history.history.keys() if 'loss' in s and 'val' in s]
    acc_list = [s for s in history.history.keys() if 'acc' in s and 'val' not in s]
    val_acc_list = [s for s in history.history.keys() if 'acc' in s and 'val' in s]
    
    if len(loss_list) == 0:
        print('Loss is missing in history')
        return 
    
    ## As loss always exists
    epochs = range(1,len(history.history[loss_list[0]]) + 1)
    
    ## Loss
    plt.figure(1)
    for l in loss_list:
        plt.plot(epochs, history.history[l], 'b', label='Training loss (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    for l in val_loss_list:
        plt.plot(epochs, history.history[l], 'g', label='Validation loss (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    
    plt.title('Loss')
    plt.xlabel('Epochs')
    plt.ylabel('Loss')
    plt.legend()
    
    ## Accuracy
    plt.figure(2)
    for l in acc_list:
        plt.plot(epochs, history.history[l], 'b', label='Training accuracy (' + str(format(history.history[l][-1],'.5f'))+')')
    for l in val_acc_list:    
        plt.plot(epochs, history.history[l], 'g', label='Validation accuracy (' + str(format(history.history[l][-1],'.5f'))+')')

    plt.title('Accuracy')
    plt.xlabel('Epochs')
    plt.ylabel('Accuracy')
    plt.legend()
    plt.show()

plot_history(history)
parallel_model.evaluate(
            x={'TI_embed_input': TI_test,
            'AB_embed_input': AB_test,
            'TECHF_embed_input': TECHF_test,
             'BACKG_embed_input': BACKG_test,
             'SUMM_embed_input': SUMM_test,
            'CLMS_embed_input': CLMS_test
            },
            y=y_test,
            batch_size=batch_size)

import matplotlib.pyplot as plt
def plot_history(history):
    loss_list = [s for s in history.history.keys() if 'precision' in s and 'val' not in s]
    val_loss_list = [s for s in history.history.keys() if 'precision' in s and 'val' in s]
    acc_list = [s for s in history.history.keys() if 'recall' in s and 'val' not in s]
    val_acc_list = [s for s in history.history.keys() if 'recall' in s and 'val' in s]
    
    if len(loss_list) == 0:
        print('precision is missing in history')
        return 
    
    ## As loss always exists
    epochs = range(1,len(history.history[loss_list[0]]) + 1)
    
    ## Loss
    plt.figure(1)
    for l in loss_list:
        plt.plot(epochs, history.history[l], 'b', label='Training precision (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    for l in val_loss_list:
        plt.plot(epochs, history.history[l], 'g', label='Validation precision (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    
    plt.title('precision')
    plt.xlabel('Epochs')
    plt.ylabel('precision')
    plt.legend()
    
    ## Accuracy
    plt.figure(2)
    for l in acc_list:
        plt.plot(epochs, history.history[l], 'b', label='Training recall (' + str(format(history.history[l][-1],'.5f'))+')')
    for l in val_acc_list:    
        plt.plot(epochs, history.history[l], 'g', label='Validation recall (' + str(format(history.history[l][-1],'.5f'))+')')

    plt.title('recall')
    plt.xlabel('Epochs')
    plt.ylabel('recall')
    plt.legend()
    plt.show()
    
    
plot_history(history)

from sklearn.datasets import make_circles
from keras.models import Sequential
from keras.layers import Dense
from matplotlib import pyplot<jupyter_output><empty_output><jupyter_text>Train the neural network on  multichannel inputs namely deep layers of patent text segments  and deep layers of patent metadata. Fit the model/network Train the neural network on  multichannel inputs namely deep layers of Title and Technical Field.<jupyter_code>#contacting two input models
model_inputs_to_concat = [TI_deep,  TECHF_deep] #invs , pas, invs
final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)

output = Dense(64, activation=None)(final_layer)
output = Dropout(dropout_pct)(output)
output = BatchNormalization()(output)
output = ELU()(output)
output = Dense(num_classes, activation='softmax')(output)

model3 =Model(inputs=[ TI_embedding_layer_input,
                      TECHF_embedding_layer_input
                      ],
              outputs=output, name='model')
model3.compile(loss='categorical_crossentropy',
                      optimizer='adam',
                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])
model3.summary()<jupyter_output>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
TI_embed_input (InputLayer)     (None, 20)           0                                            
__________________________________________________________________________________________________
TECHF_embed_input (InputLayer)  (None, 30)           0                                            
__________________________________________________________________________________________________
embedding_1 (Embedding)         (None, 20, 20)       1897380     TI_embed_input[0][0]             
__________________________________________________________________________________________________
embedding_3 (Embedding)         (None, 30, 30)       11125050    TECHF_embed_input[0][0]          
__________[...]<jupyter_text>Fit the model<jupyter_code>%%time
batch_size= 500 
num_epochs = 20


history3 = model3.fit(x={'TI_embed_input': TI_train,
             'TECHF_embed_input': TECHF_train
            },
          y=y_train,
          batch_size=batch_size,
          epochs=num_epochs,
          validation_data=
          ({'TI_embed_input': TI_test,
            'TECHF_embed_input': TECHF_test
            },
           y_test))
model3.evaluate(
            x={'TI_embed_input': TI_test,
            'TECHF_embed_input': TECHF_test
            },
            y=y_test,
            batch_size=batch_size)<jupyter_output>13200/13200 [==============================] - 1s 53us/step<jupyter_text>Train the neural network on  multichannel inputs namely deep layers of Title, abstrcat and Technical Field.<jupyter_code>import keras_metrics as km
#contacting two input models
model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep] #invs , pas, invs
final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)

output = Dense(128, activation=None)(final_layer)
output = Dropout(dropout_pct)(output)
output = BatchNormalization()(output)
output = ELU()(output)
output = Dense(num_classes, activation='softmax')(output)

model4 = Model(inputs=[TI_embedding_layer_input,
                      AB_embedding_layer_input,
                      TECHF_embedding_layer_input
                     ],
              outputs=output, name='model')
model4.compile(loss='categorical_crossentropy',
                      optimizer='adam',
                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])
model4.summary()<jupyter_output>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
TI_embed_input (InputLayer)     (None, 20)           0                                            
__________________________________________________________________________________________________
AB_embed_input (InputLayer)     (None, 100)          0                                            
__________________________________________________________________________________________________
TECHF_embed_input (InputLayer)  (None, 30)           0                                            
__________________________________________________________________________________________________
embedding_1 (Embedding)         (None, 20, 20)       1897380     TI_embed_input[0][0]             
__________[...]<jupyter_text>Fit the model<jupyter_code>%%time
batch_size= 500 
num_epochs = 20


history4 = model4.fit(x={'TI_embed_input': TI_train,
                         'AB_embed_input': AB_train,
             'TECHF_embed_input': TECHF_train
            },
          y=y_train,
          batch_size=batch_size,
          epochs=num_epochs,
          validation_data=
          ({'TI_embed_input': TI_test,
            'AB_embed_input': AB_test,
            'TECHF_embed_input': TECHF_test
            },
           y_test))
model4.evaluate(
            x={'TI_embed_input': TI_test,
            'AB_embed_input': AB_test,
            'TECHF_embed_input': TECHF_test
            },
            y=y_test,
            batch_size=batch_size)<jupyter_output>13200/13200 [==============================] - 2s 153us/step<jupyter_text>Train the neural network on  multichannel inputs namely deep layers of Title, Technical Field, Inventors, and Assignees.<jupyter_code>import keras_metrics as km

#contacting two input models
model_inputs_to_concat = [TI_deep, TECHF_deep,  pas, invs] #invs , pas, invs
final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)

output = Dense(128, activation=None)(final_layer)
output = Dropout(dropout_pct)(output)
output = BatchNormalization()(output)
output = ELU()(output)
output = Dense(num_classes, activation='softmax')(output)

model5 =Model(inputs=[ TI_embedding_layer_input,
                      TECHF_embedding_layer_input,
                     pa_input,
                      inv_input],
              outputs=output, name='model')
model5.compile(loss='categorical_crossentropy',
                      optimizer='adam',
                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])
model5.summary()<jupyter_output>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
TI_embed_input (InputLayer)     (None, 20)           0                                            
__________________________________________________________________________________________________
TECHF_embed_input (InputLayer)  (None, 30)           0                                            
__________________________________________________________________________________________________
embedding_1 (Embedding)         (None, 20, 20)       1897380     TI_embed_input[0][0]             
__________________________________________________________________________________________________
embedding_3 (Embedding)         (None, 30, 30)       11125050    TECHF_embed_input[0][0]          
__________[...]<jupyter_text>Fit the mode.<jupyter_code>%%time
batch_size= 500 
num_epochs = 20


history5 = model5.fit(x={'TI_embed_input': TI_train,
             'TECHF_embed_input': TECHF_train,
             'pa_input': train_pa_one_hot,
             'inv_input': train_inv_one_hot
            },
          y=y_train,
          batch_size=batch_size,
          epochs=num_epochs,
          validation_data=
          ({'TI_embed_input': TI_test,
            'TECHF_embed_input': TECHF_test,
            'pa_input': test_pa_one_hot,
            'inv_input': test_inv_one_hot
            },
           y_test))
model5.evaluate(
            x={'TI_embed_input': TI_test,
            'TECHF_embed_input': TECHF_test,
            'pa_input': test_pa_one_hot,
            'inv_input': test_inv_one_hot
            },
            y=y_test,
            batch_size=batch_size)

model5.metrics_names
import matplotlib.pyplot as plt
def plot_history(history):
    loss_list = [s for s in history.history.keys() if 'precision' in s and 'val' not in s]
    val_loss_list = [s for s in history.history.keys() if 'precision' in s and 'val' in s]
    acc_list = [s for s in history.history.keys() if 'recall' in s and 'val' not in s]
    val_acc_list = [s for s in history.history.keys() if 'recall' in s and 'val' in s]
    
    if len(loss_list) == 0:
        print('precision is missing in history')
        return 
    
    ## As loss always exists
    epochs = range(1,len(history.history[loss_list[0]]) + 1)
    
    ## Loss
    plt.figure(1)
    for l in loss_list:
        plt.plot(epochs, history.history[l], 'b', label='Training precision (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    for l in val_loss_list:
        plt.plot(epochs, history.history[l], 'g', label='Validation precision (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    
    plt.title('precision')
    plt.xlabel('Epochs')
    plt.ylabel('precision')
    plt.legend()
    
    ## Accuracy
    plt.figure(2)
    for l in acc_list:
        plt.plot(epochs, history.history[l], 'b', label='Training recall (' + str(format(history.history[l][-1],'.5f'))+')')
    for l in val_acc_list:    
        plt.plot(epochs, history.history[l], 'g', label='Validation recall (' + str(format(history.history[l][-1],'.5f'))+')')

    plt.title('recall')
    plt.xlabel('Epochs')
    plt.ylabel('recall')
    plt.legend()
    plt.show()

plot_history(history5)
import matplotlib.pyplot as plt
def plot_history(history):
    loss_list = [s for s in history.history.keys() if 'precision' in s and 'val' not in s]
    val_loss_list = [s for s in history.history.keys() if 'precision' in s and 'val' in s]
    acc_list = [s for s in history.history.keys() if 'recall' in s and 'val' not in s]
    val_acc_list = [s for s in history.history.keys() if 'recall' in s and 'val' in s]
    
    if len(loss_list) == 0:
        print('precision is missing in history')
        return 
    
    ## As loss always exists
    epochs = range(1,len(history.history[loss_list[0]]) + 1)
    
    ## Loss
    plt.figure(1)
    for l in loss_list:
        plt.plot(epochs, history.history[l], 'b', label='Training precision (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    for l in val_loss_list:
        plt.plot(epochs, history.history[l], 'g', label='Validation precision (' + str(str(format(history.history[l][-1],'.5f'))+')'))
    
    plt.title('precision')
    plt.xlabel('Epochs')
    plt.ylabel('precision')
    plt.legend()
    
    ## Accuracy
    plt.figure(2)
    for l in acc_list:
        plt.plot(epochs, history.history[l], 'b', label='Training recall (' + str(format(history.history[l][-1],'.5f'))+')')
    for l in val_acc_list:    
        plt.plot(epochs, history.history[l], 'g', label='Validation recall (' + str(format(history.history[l][-1],'.5f'))+')')

    plt.title('recall')
    plt.xlabel('Epochs')
    plt.ylabel('recall')
    plt.legend()
    plt.show()
    
    
plot_history(history5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11867352927897422, 'fraction_numerical': 0.018533568106912274, 'mean_word_length': 3.6926906779661017, 'pattern_counts': {'":': 0, '<': 135, '<?xml version=': 0, '>': 137, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 58, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '444387', 'license': 'Apache-2.0', 'n_tokens_mistral': 14932, 'n_tokens_neox': 14007, 'n_words': 3199, 'path': 'Second_Experiment/Multiclass Patent Classification with multichannel inputs on General Domain.ipynb', 'repo_name': 'sofean-mso/Deep4Patent', 'size': '44298'}