<jupyter_start><jupyter_text>DS8008 Final Project: Pairwise Multi-Class Document Classification for Semantic Relations Members Names: <NAME>, <NAME> Members Emails: <EMAIL>, <EMAIL>  Introduction: Problem Description:To cope with the ever-emerging information overload, digital libraries employ literature recommender systems. Thesesystems recommend related documents with the help of similarity measures, which often only distinguish between similar and dissimilar documents.The objective is to design the model for finding the relationship between two documents as a pairwise document classification task. To find the semantic relation between documents, a series of techniques, such as GloVe, Paragraph-Vectors, BERT, and XLNet under different configurations (e.g., sequence length, vector concatenation scheme), including a Siamese architecture for the Transformer-based systems. The experiments performed on a newly proposed dataset of 32,168 Wikipedia article pairs and Wikidata properties that define the semantic document relations. Context of the Problem:Many digital libraries recommend literature to their users considering the similarity between a query document and their repository. However, they often fail to distinguish what is the relationship that makes two documents alike. It remains unclear to which of the many facets the similarity relates.Based on the the findings we can deduce that classifying semantic relations between documents is a solvable task and motivates the development of recommender systems based on the evaluated techniques. The discussions in the paper serves as first steps in the exploration of documents through SPARQL-like queries such that one could find documents that are similar in one aspect but dissimilar in another. Limitation About other Approaches:While other NLP tasks, like relation extraction, deal with relations, they are not concerned with semantic relations between documents.The choice of similarity measures is widely used and works well on a wide range of documents, but has no theoretical basis. A similarity measure is a function which computes the degree of similarity between a pair of vectors or documents. Since queries and documents are both vectors, a similarity measure can represent the similarity between two documents, two queries, or one document and one query. There are a large number of similarity measures proposed in the past, because the best similarity measure doesn't exist yet. The other limitations that exists at present are:Synonyms: separate words that have the same meaning. (E.g. ‘car’ & ‘automobile’). They tend to reduce recall.Polysems: words with multiple meanings (E.g. ‘Java’). They tend to reduce precision. The problem is more general: there is a disconnect between topicsand wordsTF-IDF is based on the bag-of-words (BoW) model, therefore it does not capture position in text, semantics, co-occurrences in different documents, etc. It cannont capture semantics (e.g. as compared to topic models, word embeddings). Solution:The task of finding semantic document relations is implemented as a multi-class classification of document pairs. The paper demonstrated the viability of this approach with a new proposed dataset of 32,168 Wikipedia article pairs and Wikidata properties that define semantic relations among these articles. In an empirical study, implemented six different models AvgGloVe, Doc2vec, Siamese BERT, Siamese XLNet, vanilla BERT and vanilla XLNet, and evaluate them underdifferent settings regarding the concatenation method and sequence length. The evaluation indicates a sequence length of512 tokens as the best performing sequence limit for the Siamese and vanilla Transformer models.   BackgroundIn this project we tried to mimic the implementation based on the paper https://arxiv.org/pdf/2003.09881Bär et al. [7] discuss the notion of similarity between texts in the context of NLP. They express that while text similarity is present in many NLP tasks, the similarity is often ill-defined and used as an “umbrella term covering quite different phenomena”. Bär et al. [7] formalize text similarity and suggest content, structure, and style as the major dimensions inherent to texts. Structure and style are not actively being accounted for. Therefore, we focused only on the content. With the classification of semantic document relations, we intend to tailor recommendations depending on specific information needs. For example, we could provide either recommendations focusing on a particular relation class or diverse recommendations from multiple relation classes.The Transformer architecture allowed the efficient unsupervised pretraining of language models and led to significant improvements in many NLP benchmarks. Akkalyoncu Yilmaz et al. [3] applied BERT to an information retrieval system foran end-to-end search over large document collections. Despite their success in NLP, Transformers have gained little attention in the recommender system community so far.  In our experiments, we utilize the article text and learn the document relation using a multilayer perceptron (MLP).| Reference |Explanation |  Dataset/Input |Weakness| --- | --- | --- | --- || Malte O [1] | find the semantic relation between documents by treating the two as a pairwise document classification | digital library from Wikipedia through Wikidata | Doc2Vec model with the lowest F1 score of 0.83| <NAME>, <NAME>, <NAME>, and <NAME> [2] | Here computational expense associated with BERT inference is addressed by leveraging distill knowledge from BERT_large to small bidirectional LSTMs| Reuters-21578 , arXiv Academic Paper dataset , IMDB reviews, and Yelp 2014 reviews | Implementaion of the same model using Neural Network Architecture| <NAME>, <NAME>, <NAME>, <NAME>, and <NAME> [3] | Here BIRCH, a system that applies BERT to document retrieval via integration with the open-source Anserini information retrieval toolkit is implemented to demonstrate end-to-end search over large document collections | Shot Social Media Posts | precision at rank 30 is 70.06%| power, jchen, trishank, lakshmi [4] | This is a simple feature extraction algorithm that can achieve high document classification accuracy in the context of development-centric topics| 10000 random documents from the English wikipedia corpus| when classification tasks are restricted to relatively narrow topics of interest, we can achieve near perfect precision and recall|<NAME>, <NAME>, Md <NAME>, <NAME> [5]|Multiclass patent document classification by using array of methods like kNN, SVM and two tree based classification algorithm: Random Forest and J48|NASA’s predefined patent documents| fusion technique could further be implemented to improve the accuracy of the multiclass classification problem|<NAME>, <NAME>, and <NAME> [6]|Methods to improve the performance of text classification on data that is difficult to classify due to a large number of unbalanced classes with noisy examples | songtexts from Kaggle |Implement CNNs and RNNs for the same set for better result|<NAME>, <NAME>, and <NAME> [7]|text similarity with respect to its definition and the datasets used for evaluation |Microsoft Paraphrase Corpus |Recognizing paraphrases is a very hard task that cannot simply be tackled by computing text similarity, as sharing similar content is a necessary, but not a sufficient condition for detecting paraphrases  MethodologyHere, we combine the ideas of relation extraction, document classification, and document similarity to classify the semanticrelation of document pairs. Given a seed document ds , we are interested in finding a target document dt that shares the semantic relation r(i) with d(s) . The task is to model to find the relation r of a document pair (ds ,dt ) as a pairwise multi-class document classification problem.To evaluate the presented techniques, we build a dataset using Wikipedia and Wikidata repositories to illustrate our problem. Wikipedia articles are the seed and target documents, while Wikidata properties provide the semantic relations between a document pair. The following figure shows one example from the dataset:![Alternate text ](wikipedia_example.png "Semantic Relation Explained, location is simply the directory of the notebook")To classify the semantic relations of document pairs we are implementing deep contextual language models from BERT. In the process a novel dataset composed of 32,168 Wikipedia article pairs and Wikidata properties that define the semantic relation of these articles are shared.![Alternate text ](example_relation.png "Example Relation Explained, location is simply the directory of the notebook")As one of the main goal of this paper is to explore the multi-class classification problem, so we ensured that the same pair of documents did not share different labels.  Implementation<jupyter_code># Code cells
from models.transformers import JointBERT
from sklearn.preprocessing import LabelEncoder, OrdinalEncoder
from wiki.utils import get_text_from_wikipedia
import torch
from torch.nn.utils.rnn import pad_sequence
from transformers import BertModel, BertTokenizer
from IPython.core.display import display, HTML
import pandas as pd
import pickle
import os
from experiments import Experiment
from wiki.data_helpers import JointBERTWikiDataHelper
# Code cells
if not torch.cuda.is_available():
  raise ValueError('CUDA is not available. Please enable GPU support under: Runtime > Change runtime')

# Load experimental settings
exp_dir = '4fold_results/1/wiki.bert_base__joint__seq512'

with open(os.path.join(exp_dir, 'experiment.pickle'), 'rb') as f:
  _exp = pickle.load(f)

le = LabelEncoder()
le.fit_transform(_exp['data_helper_params']['labels'] + ['none'])

# Init model
model = JointBERT(bert_model_path='bert-base-cased', bert_cls=BertModel, prob='sigmoid', labels_count=_exp['model_params']['labels_count'])
model.load_state_dict(torch.load(os.path.join(exp_dir, 'model_weights')))

#model = BertModel.from_pretrained('bert-base-cased')
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)
# Code cells
# Retrieve article text from Wikipedia API
seed_text = get_text_from_wikipedia(seed_title)
target_text = get_text_from_wikipedia(target_title)

# Convert to model input
seed_ids = tokenizer.convert_tokens_to_ids((tokenizer.tokenize(seed_title) + tokenizer.tokenize(seed_text))[:250])
target_ids = tokenizer.convert_tokens_to_ids((tokenizer.tokenize(target_title) + tokenizer.tokenize(target_text))[:250])

pair_ids = [torch.tensor(tokenizer.build_inputs_with_special_tokens(seed_ids, target_ids))]
token_type_ids = [torch.tensor(tokenizer.create_token_type_ids_from_sequences(seed_ids, target_ids))]

pair_ids = pad_sequence(pair_ids, batch_first=True, padding_value=tokenizer.pad_token_id)
token_type_ids = pad_sequence(token_type_ids, batch_first=True, padding_value=1)

masks = torch.tensor([[float(i > 0) for i in ii] for ii in pair_ids])

# Prediction
with torch.no_grad():
  ys_pred = model(pair_ids, masks, token_type_ids).numpy()
  label_pred = le.inverse_transform([ys_pred.argmax()])[0]
    

# Probabilites for other classes
df = pd.DataFrame(dict(label=le.classes_.tolist(), probability=ys_pred.flatten().round(3).tolist()))
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05406842758376174, 'fraction_numerical': 0.007445488388583585, 'mean_word_length': 5.854799513973268, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 30, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '449896', 'license': 'MIT', 'n_tokens_mistral': 2800, 'n_tokens_neox': 2609, 'n_words': 1407, 'path': 'Final_project_Group-7.ipynb', 'repo_name': 'masukislam/DS8008_NLP', 'size': '11460'}