<jupyter_start><jupyter_text>Tensorflow  Framework for Multi-class Text Classification  Sentiment Classification - 3 Class Problem* Financial Phrasebank Dataset     * 0 - Positive     * 1 - Negative     * 2 - Neutral    Reading Data<jupyter_code>import sys
import numpy as np
import pandas as pd

sys.path.append("..")
pd.set_option('display.max_colwidth', -1)
df_train = pd.read_csv('../datasets/financial_phrasebank/training.csv', sep='\t')
df_test = pd.read_csv('../datasets/financial_phrasebank/testing.csv', sep='\t')
df_val = pd.read_csv('../datasets/financial_phrasebank/validation.csv', sep='\t')

df_train.shape, df_test.shape, df_val.shape
df_train.label.value_counts()
df_val.label.value_counts()<jupyter_output><empty_output><jupyter_text>Processing Data in Input format<jupyter_code>data_classes = ['positive', 'negative', 'neutral']

df_train['label'] = df_train['label'].apply(data_classes.index)
df_test['label'] = df_test['label'].apply(data_classes.index)
df_val['label'] = df_val['label'].apply(data_classes.index)
df_test.head()
df_bert_train = pd.DataFrame({'id':df_train['Unnamed: 0'],
            'text':df_train['text'],
            'label':df_train['label']})

df_bert_test = pd.DataFrame({'id':df_test['Unnamed: 0'],
            'text':df_test['text']})

df_bert_val = pd.DataFrame({'id':df_val['Unnamed: 0'],
            'text':df_val['text'],
            'label':df_val['label']})
df_bert_train.to_csv('../datasets/financial_phrasebank/train.csv',index=False)
df_bert_test.to_csv('../datasets/financial_phrasebank/test.csv',index=False)
df_bert_val.to_csv('../datasets/financial_phrasebank/dev.csv',index=False)<jupyter_output><empty_output><jupyter_text>Training (Bert | Electra) classifier with FrameWork<jupyter_code># from classitransformers.tfelectra import ElectraClassification

from classitransformers.report import metrics
from classitransformers.configs import Configs

from classitransformers.tfbert import BertClassification<jupyter_output>WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/__init__.py:1467: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.

WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/bert/optimization.py:87: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.<jupyter_text>Set Configurations* Pre-trained config filepath, vocab filepath and model checkpoint filepath* labels list* train, test and eval batch size* max_seq_len - how long is the each text input sentence !!* data_dir - data path folder (optional)* output_dir - output path folder (optional)<jupyter_code># config = Configs(pretrained_model_dir = '../models/Electra_base/', 
#                  model_name = 'electra'
#                  num_train_epochs = 3,
#                  train_batch_size = 24, 
#                  eval_batch_size = 8, 
#                  predict_batch_size = 8, 
#                  do_train = True, 
#                  do_eval = True, 
#                  label_list = ["0", "1", "2"], 
#                  max_seq_length = 64,
#                  data_dir='../datasets/financial_phrasebank/',
#                  output_dir = '../electra_output_sentiment/')

# model = ElectraClassification(config)


config = Configs(pretrained_model_dir = '../models/Bert_base/',
                 model_name = 'bert',
                 num_train_epochs = 3,
                 train_batch_size = 24, 
                 eval_batch_size = 8, 
                 predict_batch_size = 8, 
                 do_train = True, 
                 do_eval = True, 
                 label_list = ["0", "1", "2"], 
                 max_seq_length = 64,
                 data_dir='../datasets/financial_phrasebank/',
                 output_dir = '../bert_output_sentiment/')

model = BertClassification(config)<jupyter_output><empty_output><jupyter_text>Train Method<jupyter_code>model.train()<jupyter_output>WARNING:tensorflow:From ../classitransformers/tfbert.py:374: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

WARNING:tensorflow:From ../classitransformers/tfbert.py:378: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:Writing example 0 of 3100
INFO:tensorflow:***** Running training *****
INFO:tensorflow:  Num examples = 3100
INFO:tensorflow:  Batch size = 24
INFO:tensorflow:  Num steps = 387
WARNING:tensorflow:From ../classitransformers/tfbert.py:404: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

INFO:tensorflow:Skipping training since max_steps has already saved.
INFO:tensorflow:training_loop marked as finished
INFO:tensorflow:Writing example 0 of 776
INFO:tensorflow:***** Running evaluation *****
INFO:tensorflow:  Num examples = 776 (776 actual, 0 padding)
INFO:tensorflow:  Batch size = 8
WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/ten[...]<jupyter_text>Test Method<jupyter_code>prediction = model.test()<jupyter_output>INFO:tensorflow:Writing example 0 of 970
INFO:tensorflow:***** Running prediction*****
INFO:tensorflow:  Num examples = 970 (970 actual, 0 padding)
INFO:tensorflow:  Batch size = 8
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Running infer on CPU
INFO:tensorflow:*** Features ***
INFO:tensorflow:  name = input_ids, shape = (?, 64)
INFO:tensorflow:  name = input_mask, shape = (?, 64)
INFO:tensorflow:  name = is_real_example, shape = (?,)
INFO:tensorflow:  name = label_ids, shape = (?,)
INFO:tensorflow:  name = segment_ids, shape = (?, 64)
INFO:tensorflow:**** Trainable Variables ****
INFO:tensorflow:  name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*
INFO:tensorflow:  name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*
INFO:tensorflow:  name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*
INFO:tensorflow:  name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*
INFO:tensorfl[...]<jupyter_text>Export the checkpoint model to pb (protobuf) format**In TensorFlow, the protbuf file contains the graph definition as well as the weights of the model. Thus, a pb file is all you need to be able to run a given trained model.**<jupyter_code># Currently does not have support for Electra

model.export_model()<jupyter_output>INFO:tensorflow:Using config: {'_model_dir': '../bert_output_sentiment/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f2666b0a668>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:From ../classitransformers/tfbert.py:739: The name tf.placeholder [...]<jupyter_text>Inferencing from Trained Pb Model* This tensorflow format is used for deployment purpose.<jupyter_code># Currently does not have support for Electra

predictions = model.inference(test_filename = '../datasets/financial_phrasebank/test.csv', batch_size = 256)<jupyter_output>WARNING:tensorflow:From ../classitransformers/tfbert.py:790: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From ../classitransformers/tfbert.py:792: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
INFO:tensorflow:Restoring parameters from ./exported_bert_model/1594469887/variables/variables
Processing batch no.  0
Processing batch no.  1
Processing batch no.  2
Processing batch no.  3<jupyter_text>**Testing a sentence, takes a list of sentences.**<jupyter_code># Only be used after .pb model has been exported for BERT

model.text_inference(['Trading gives the best returns among all the options of investemnts'])
y_pred = [np.argmax(tup) for tup in prediction]
y = df_test.label<jupyter_output><empty_output><jupyter_text>Accuracy and Report<jupyter_code>metrics(y, y_pred)<jupyter_output>Accuracy: 0.862
Precision: 0.833
Recall: 0.848
F1-Score: 0.840
{'fraction_non_alphanumeric': 0.1210496929089894, 'fraction_numerical': 0.023562255723059745, 'mean_word_length': 4.770618556701031, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '465228', 'license': 'Apache-2.0', 'n_tokens_mistral': 3074, 'n_tokens_neox': 2858, 'n_words': 642, 'path': 'sample_notebooks/TF_classi_transformers_sentiment_example.ipynb', 'repo_name': 'mgroovyank/classitransformers', 'size': '8955'}