<jupyter_start><jupyter_text>Hep Recommender> A recommender system for scientific articles in the field of High Energy Physics.- toc: true - badges: true- comments: true- categories: [jupyter]- image: images/hep_recommender.png  IntroductionIn this note I want to discuss [hep-recommender](https://hep-recommender.herokuapp.com/), a recommender system for scientific papers in the field of High Energy Physics (HEP) aimed to help researchers in the exploration of relevant literature.  This is a personal project I have been working recently in collaboration with [<NAME>](https://github.com/JoseEliel).    A [submission](https://devpost.com/software/hep-recommender) of this project to the [Facebook Artificial Intelligence Hackathon (2020)](https://fbai1.devpost.com) won the third place.There are different approaches to determine the degree of similarity between articles in order to identify related work. Some works have used text-mining and natural language processing methods. Another popular approach is based on citation analysis, where the similarity between two articles is estimated based on bibliographic information [1,2,4,5]. Finally, other works have used a combination of the previous approaches, see for instance [3].    In this project, we develop a recommender system based on citation analysis. More specifically, we will develop a recommender system that elaborates on the ideas of co-citation analysis [1,4] and co-citation proximity analysis (CPA) [2]. Co-citation analysis is based on the premise that articles which are frequently cited together (by the same papers) should be related to each other. CPA extends this idea by incorporating the notion that the closer the citations are to each other within the article text, the more likely it is that they are related. While these methods are relatively simple they provide a high quality of related article recommendations.   Our recommender system relies on a distributed representation of articles obtained by training a Skip-Gram model on reference lists.  This model also captures the notion that articles cited close to each other on the text are similar. In this note I want to go to some detail about the model powering our recommender system, how it compares to other approaches, and how we made it available via a web application.    Related workI would like to start discussing some of the standard methods in the literature to quantify the similarity between two articles.  I will define the following concepts: * bibliometric coupling* co-citation count * co-citation proximity analysis (CPA)Bibliometric couplign reflects the idea that two papers sharing a large portion of their references should be similar [5].  Co-citation count assigns a high degree of similarity to articles which are frequenly cited together (regardless of where the citations occcured within the text) [1].   CPA extends co-citation count by assigning more weigth to cases where the articles are cited close to each other within the text [2].  In general CPA is expected to give better results than co-citation count, though it requires some work in order to parse the references of each article keeping information about the location of the references within the text. Lets formalize these concepts.  We consider a set of papers $\{  \omega_1, \omega_2, \ldots \omega_n  \}$. We define the quantities $c_{i,j}$, where $c_{i,j}=1$ if $\omega_i$ is cited by $\omega_j$ and $c_{i,j}=0$ otherwise.  Then we are ready to introduce our three measures of similarity:* bibliographic coupling is defined as $\sum_{k=1}^{n}   c_{k,i}   c_{k,j}$* co-citation count is defined as $\sum_{k=1}^{n}   c_{i,k}   c_{j,k}$* CPA can be used to define a measure $\sum_{k=1}^{n} \eta_{i,j}^{k}  c_{i,k}   c_{j,k}$, where $n_{ij}^{k}$ are coefficients that penalize cases where the citations occur far from each other within the text.    In the original CPA article [2], $\eta$ was fixed to $1$ if the citations occur in the same sentence, $1/2$ if they only occur in the same paragraph, $1/4$ if they only occur in the same chapter, and $1/8$ if they only appear on the same article. Parametric representations of $\eta$ have also been proposed [7].   Distributed representation on co-citationsWe propose a model that relies on the same idea behind CPA, namely, that the proximity of references within an article provide valuable information regarding the similarity of two articles.    However, we wanted to simplify the process of extracting the necessary data as much as possible.  We assume that it is possible to extract the list of references of each article in the order of appearance within the text. For instance the following article  > Higgs boson pair production in gluon fusion is the most promising process to find out whether the Higgs boson self-coupling is Standard-Model-like. Early studies of Higgs boson pair production within an EFT framework can be found in Refs. [1–3]. Many phenomenological investigations about the potential of this process to reveal New Physics have been performed since, see e.g. Refs. [4–8]...would give rise to a reference list [1,2,3,4,5,6,7,8,...].   In this way, articles which are frequently close to each other on these reference lists should be very similar, as the community is citing them close to each other within the text of the articles.   We will use an approach that has proven to be very fruitful in Natural Language Processing (NLP).  We will take the lists of references and train a Skip-Gram model, such that articles which tend to be cited close to each other will have similar embeddings.    Our dataset consists of ordered lists of references.  Suppose we have a total of $n$ unique articles in our dataset and for each of these articles we have its reference list.  The Skip-gram model maximizes the following objective function:\begin{align}\frac{1}{n} \sum_{q=1}^{n}    \left[  \frac{1}{n_q}   \sum_{i=1}^{n_q}  \sum_{ -c \leq j  \leq c ,\, j\neq 0  } \log p(w_{i+j}^{q}| w_i^{q}) \right]\end{align} Here $\omega_{i}^{q}$ represents the embedding of the $i$-th article appearing on the reference list of article $q$; $n_q$ represents the size of the reference list for article $q$, and $c$ represents the context-window size of the Skip-Gram model.  The probability is modelled using  a softmax function \begin{align}p(    \omega_{i}  |    \omega_j  )   = \frac{   \exp(   \omega_{i} \cdot \omega_{j}  )}{     \sum_{l=1}^{n}   \exp(   \omega_{l}  \cdot \omega_j   )   }\end{align}Just like in the Skip-Gram model used in NLP, in order to make the model training more efficient we need to implement negative sampling or hierarchical softmax.      A good implementation of the Skip-Gram model is provided by the [Gensim](https://radimrehurek.com/gensim/auto_examples/index.html) library for instance.In order to explore the structure of the trained embeddings, I took a sample of articles and made a t-SNE visualization of their embeddings in two dimensions:For this, I picked a set of articles that were published on the [arXiv](https://arxiv.org).  Each arXiv category was displayed with a different color.  On this plot, some of the arXiv categories that appear are: astro-ph (red), hep-ex (blue), hep-th (black) and hep-ph (green), among others.  As expected, the embeddings form cluster around their categories.  Data Regarding open acces digital libraries, the research community in the field of High Energy Physics mainly uses the [INSPIRE-HEP](https://inspirehep.net) and the [arXiv](https://arxiv.org). INSPIRE-HEP provides an API from which data for articles in the field of HEP can be retrieved, we use this API to extract the data we need.  Below I show some exploratory example.  Lets first define some classes that talk to the API and handle the response.<jupyter_code>import requests
from typing import List, Dict


class InspireAPI:
    """
    Simple wrapper class around the INSPIRE API

    https://inspirehep.net

    methods:
       literature: gives access to the literature endpoint
    """

    LITERATURE = "https://inspirehep.net/api/literature/"

    def __init__(self):
        pass

    def literature(self, record_id: str):
        """
        Returns api response for a given record_id
        """
        url = self.LITERATURE + record_id
        return LiteratureRecord(requests.get(url).json())


class LiteratureRecord:
    """
    Datamodel class for handling literature record data,
    implementing basic methods to access the properties
    """

    def __init__(self, data: Dict):
        self.data = data
        
    @property 
    def record_id(self) -> str:
        """
        Returns the INSPIRE id of the article
        """
        return self.data['id']
        
    @property
    def metadata(self) -> Dict:
        """
        Returns article metadata
        """
        return self.data['metadata']

    @property
    def references(self) -> List[str]:
        """
        Returns reference list of the article as a List of INSPIRE article ids
        """
        if self.metadata.get("references"):
            return [
                element["record"]["$ref"].split("/")[-1]
                for element in self.metadata["references"]
                if element.get("record")
            ]<jupyter_output><empty_output><jupyter_text>We can now look at one example. All the bibliographic data is contained on the *metadata* property of our LiteratureRecord class.  I implemented a *references* property in order to retrieve the list of references (only the article identifiers).   Lets extract data for the article with the identifier ['11883'](https://labs.inspirehep.net/literature/11883),<jupyter_code>inspireapi = InspireAPI()

record = inspireapi.literature('11883')

record.references<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07366915295800552, 'fraction_numerical': 0.006743639521814652, 'mean_word_length': 3.896448224112056, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 15, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '466827', 'license': 'Apache-2.0', 'n_tokens_mistral': 2573, 'n_tokens_neox': 2434, 'n_words': 1303, 'path': '_notebooks/2020-02-20-hep-recommender.ipynb', 'repo_name': 'celis/personal', 'size': '9806'}