TabLeX: A Benchmark Dataset for Structure and Content Information Extraction from Scientific Tables

Abstract: Information Extraction (IE) from the tables present in scientific articles is challenging due to complicated tabular representations and complex embedded text. This paper presents TabLeX, a large-scale benchmark dataset comprising table images generated from scientific articles. TabLeX consists of two subsets, one for table structure extraction and the other for table content extraction. Each table image is accompanied by its corresponding LATEX source code. To facilitate the development of robust table IE tools, TabLeX contains images in different aspect ratios and in a variety of fonts. Our analysis sheds light on the shortcomings of current state-of-the-art table extraction models and shows that they fail on even simple table images. Towards the end, we experiment with a transformer-based existing baseline to report performance scores. In contrast to the static benchmarks, we plan to augment this dataset with more complex and diverse tables at regular intervals.

\section{Introduction}
\label{section1}

Tables are compact and convenient means of representing relational information present in diverse documents such as scientific papers, newspapers, invoices, product descriptions, and financial statements~\cite{tkasar}. Tables embedded in the scientific articles provide a natural way to present data in a structured manner~\cite{douglas1995using}. They occur in numerous variations, especially visually, such as with or without horizontal and vertical lines, spanning multiple columns or rows, non-standard spacing, alignment, and text formatting~\cite{embley}. Besides, we also witness diverse semantic structures and presentation formats, dense embedded text, and formatting complexity of the typesetting tools~\cite{singh2019automated}. These complex representations and formatting options lead to numerous challenges in automatic tabular information extraction (hereafter, \textit{`TIE'}).

\noindent \textbf{Table detection vs. extraction:}
In contrast to table detection task (hereafter, \textit{`TD'}) that refers to identifying tabular region (e.g., finding a bounding box that encloses the table) in an document, TIE refers to two post identification tasks: (i) table structure recognition (hereafter, \textit{`TSR'}) and (ii) table content recognition (hereafter, \textit{`TCR'}). TSR refers to the extraction of structural information like rows and columns from the table, and TCR refers to content extraction that is embedded inside the tables. Figure 1 shows an example table image with its corresponding structure and content information in the \TeX~language. Note that, in this paper, we only focus on the two TIE tasks.

\noindent \textbf{Limitations in existing state-of-the-art TIE systems:} There are several state-of-the-art tools (Camelot\footnote{\url{https://github.com/camelot-dev/camelot}},Tabula\footnote{\url{https://github.com/chezou/tabula-py}}, PDFPlumber\footnote{\url{https://github.com/jsvine/pdfplumber}}, and Adobe Acrobat SDK\footnote{\url{https://www.adobe.com/devnet/acrobat/overview.html}}) for text-based TIE. On the contrary, Tesseract-based OCR~\cite{tesseract} is commercially available tool which can be used for image-based TIE. However, these tools perform poorly on the tables embedded in the scientific papers due to the complexity of tables in terms of spanning cells and presence of mathematical content.
The recent advancements in deep learning architectures (Graph Neural Networks~\cite{gnn} and Transformers~\cite{vaswani2017attention}) have played a pivotal role in developing the majority of the image-based TIE tools.
We attribute the limitations in the current image-based TIE primarily due to the training datasets' insufficiency. Some of the critical issues with the training datasets can be (i) the size, (ii) diversity in the fonts, (iii) image resolution (in dpi), (iv) aspect ratios, and (v) image quality parameters (blur and contrast).

\noindent \textbf{Our proposed benchmark dataset:} In this paper, we introduce \textsc{TabLeX},  a benchmark dataset for information extraction from tables embedded inside scientific documents compiled using \LaTeX-based typesetting tools. \textsc{TabLeX} is composed of two subsets---a table structure subset and a table content subset---to extract the structure and content information. The table structure subset contains more than three million images, whereas the table content subset contains over one million images.  Each tabular image accompanies its corresponding ground-truth program in \TeX~macro language. In contrast to the existing datasets~\cite{table2latex,pubtabnet,publaynet}, \textsc{TabLeX} comprises images with 12 different fonts and multiple aspect ratios.

\noindent \textbf{Main contributions:} The main contributions of the paper are:
\begin{enumerate}[nosep,noitemsep]
    \item Robust preprocessing pipeline to process the scientific documents (created in \TeX~language) and extract the tabular spans.
    \item A large-scale benchmark dataset, \textsc{TabLeX}, comprising more than three million images for structure recognition task and over one million images for the content recognition task.
    \item Inclusion of twelve font types and multiple aspect ratios during the dataset generation process.
    \item Evaluation of state-of-the-art computer vision based baseline~\cite{feng2020scene} on \textsc{TabLeX}.
\end{enumerate}

\noindent \textbf{The paper outline :} We organize the paper into several sections. Section~\ref{sec:rel_work} reviews existing datasets and corresponding extraction methodologies. Section \ref{sec:dataset} describes the preprocessing pipeline and presents  \textsc{TabLeX} dataset statistics. Section~\ref{sec:metrics} details  three evaluation metrics. Section \ref{sec:exp} presents the the baseline and discusses the experimental results and insights. Finally, we conclude and identify the future scope in Section \ref{section6}.

\begin{figure}[h]
\centering
    \subfloat[\centering Table Image]{{\includegraphics[width=0.5\textwidth]{figures/table_example.jpg} }}%
    \qquad
    \subfloat[\centering Structure Information ]{{\includegraphics[width=0.5\textwidth]{figures/structure.png} }}%
    \qquad
    \subfloat[\centering Content Information ]{{\includegraphics[width=0.5\textwidth]{figures/content.png} }}%
    \caption{Example of table image along with its structure and content information from the dataset. Tokens in content information are character-based, and the `¦' token acts as a delimiter to identify words out of a continuous stream of characters.}
    \label{fig:table_example}%
\end{figure}

\section{The Current State Of The Research}
\label{sec:rel_work}
In recent years, we witness a surge in the digital documents available online due to the high availability of communication facilities and large-scale internet infrastructure. In particular, the availability of scientific research papers that contain complex tabular structures has grown exponentially. However, we witness fewer research efforts to extract embedded tabular information automatically. Table~\ref{datasets} lists some of the popular datasets for TD and TIE tasks. Among the two tasks, there are very few datasets available for the TIE, specifically from scientific domains containing complex mathematical formulas and symbols. As the current paper primarily focuses on the TIE from scientific tables, we discuss some of the popular datasets and their limitations in this domain.

\begin{table}[!t]
\caption{Datasets and methods used for Table Detection (TD), Table Structure
Recognition (TSR) and Table Recognition (TR). * represents scientific paper datasets.}
\label{datasets}
\centering
\resizebox{\hsize}{!}{
\begin{tabular}{|c|c|c|c|c|c|c|}
		\hline
        \multicolumn{1}{|c|}{\textbf{Datasets}} &
		\multicolumn{1}{|c}{\textbf{TD}} &
		\multicolumn{1}{|c}{\textbf{TSR}} &
        \multicolumn{1}{|c}{\textbf{TR}} &
        \multicolumn{1}{|c}{\textbf{Format}} &
        \multicolumn{1}{|c|}{\textbf{\# Tables}} &
        \multicolumn{1}{|c|}{\textbf{Methods}}\\ \hline
        Marmot\cite{marmot}* & \checkmark & $\times$ & $\times$ & PDF & 958 & Pdf2Table\cite{pdf2table},TableSeer\cite{tableseer},\cite{hao2016a}\\ \hline
        PubLayNet\cite{publaynet}* & \checkmark & $\times$ & $\times$ & PDF & 113k & F-RCNN\cite{frcnn}, M-RCNN\cite{maskrcnn}\\ \hline
        DeepFigures\cite{deepfigures}* & \checkmark & $\times$ & $\times$ & PDF & 1.4m & Deepfigures\cite{deepfigures}\\ \hline
        ICDAR2013\cite{icdar2013} & \checkmark & \checkmark &\checkmark &
        PDF & 156 & Heuristics+ML\\ \hline
        ICDAR2019\cite{icdar2019} & \checkmark & \checkmark & $\times$ &
        Images & 3.6k & Heuristics+ML\\ \hline
        UNLV\cite{unlv} & \checkmark & \checkmark & $\times$ & Images & 558 & T-Recs\cite{kieninger1998paper} \\ \hline
        TableBank\cite{tablebank}* & \checkmark&\checkmark & $\times$& Images & 417k (TD) & F-RCNN\cite{frcnn} \\ \hline
        TableBank\cite{tablebank}* & \checkmark&\checkmark & $\times$& Images & 145k (TSR) & WYGIWYS\cite{Deng2016WhatYG}\\ \hline
        SciTSR\cite{scitsr}* & $\times$ & \checkmark & $\checkmark$ & PDF & 15k & GraphTSR\cite{scitsr} \\ \hline
        Table2Latex\cite{table2latex}* & $\times$ & \checkmark & \checkmark & Images & 450k & IM2Tex\cite{im2tex}\\ \hline
        Synthetic data~\cite{graphnn} & $\times$ & \checkmark & \checkmark & Images & Unbounded & DCGNN\cite{graphnn}\\ \hline
        PubTabNet\cite{pubtabnet}* & $\times$ & \checkmark & \checkmark & Images & 568k & EDD\cite{pubtabnet}\\ \hline \hline
         \textsc{TabLeX} (ours)* & $\times$ & \checkmark & \checkmark & Images & 1m\texttt{+} & TRT~\cite{feng2020scene}
        \\ \hline
	\end{tabular}
	}
\end{table}

\noindent \textbf{Scientific tabular datasets:}
Table2Latex~\cite{table2latex} dataset contains 450k scientific table images and its corresponding ground-truth in \LaTeX. It is curated from the \textit{arXiv}\footnote{\label{arxivfootnote}\url{http://arxiv.org/}} repository. To the best of our knowledge, Table2Latex is the only dataset that contains ground truth in \TeX~language but is not publicly available. TableBank~\cite{tablebank} contains 145k table images along with its corresponding ground truth in the HTML representation. TableBank~\cite{tablebank} contains table images from both Word and \LaTeX~documents curated from the internet and \textit{arXiv}\footref{arxivfootnote}, respectively. However, it does not contain content information to perform the TCR task. PubTabNet~\cite{pubtabnet} contains over 568k table images and corresponding ground truth in the HTML language. PubTabNet is curated from the PubMed Central repository\footnote{\label{pubmed}\url{https://pubmed.ncbi.nlm.nih.gov/}}. However, it is only limited to the biomedical and life sciences domain. SciTSR~\cite{scitsr} contains 15k tables in the PDF format with table cell content and the coordinate information in the JSON format. SciTSR has been curated from the \textit{arXiv}\footref{arxivfootnote} repository. However, manual analysis tells that 62 examples out of 1000 randomly sampled examples are incorrect~\cite{TableRobot}.

\noindent \textbf{TIE methodologies:}
The majority of the TIE methods employ encoder-decoder architecture~\cite{table2latex,tablebank,pubtabnet}.  Table2Latex~\cite{table2latex} uses IM2Tex~\cite{im2tex} model where encoder consists of convolutional neural network (CNN) followed by bidirectional LSTM and decoder consists of standard LSTM. TableBank~\cite{tablebank} uses WYGIWS~\cite{Deng2016WhatYG} model, an encoder-decoder architecture, where encoder consists of CNN followed by a recurrent neural network (RNN) and decoder consists of standard RNN. PubTabNet~\cite{pubtabnet} uses a proposed encoder-dual-decoder (EDD)~\cite{pubtabnet} model which consists of CNN encoder and two RNN decoders called structure and cell decoder, respectively. In contrast to the above works, SciTSR~\cite{scitsr} proposed a graph neural network-based extraction methodology. The network takes vertex and edge features as input and computes their representations using graph attention blocks, and performs classification over these edges.

\noindent \textbf{TIE metrics:}
Table2Latex~\cite{table2latex} used BLEU~\cite{bleu} score (a text-based metric) and exact match accuracy (a vision-based metric) for evaluation.
TableBank~\cite{tablebank} also conducted BLEU~\cite{bleu} metric for evaluation.  Tesseract OCR~\cite{smith2007overview} uses the Word Error Rate (WER) metric for evaluation of the output. SciTSR~\cite{scitsr} uses micro- and macro-averaged precision, recall, and F1-score to compare the output against the ground truth, respectively. In contrast to the above standard evaluation metrics in NLP literature,  PubTabNet~\cite{pubtabnet} proposed a new metric called Tree-Edit-Distance-based Similarity (TEDS) for evaluation of the output HTML representation.

\noindent \textbf{The challenges:} Table~\ref{datasets} shows that there are only two datasets for Image-based TCR from scientific tables, that is, Table2Latex~\cite{table2latex} and PubTabNet~\cite{pubtabnet}. We address some of the challenges from previous works with \textsc{TabLeX} which includes (i) large-size for training (ii) Font-agnostic learning (iii) Image-based scientific TCR (iv) domain independent

\section{The \textsc{TabLeX} Dataset}
\label{sec:dataset}
This section presents the detailed curation pipeline to create the \textsc{TabLeX} dataset. Next, we discuss the data acquisition strategy.

\subsection{Data Acquisition}
We curate data from popular preprint repository \textit{arXiv}\footref{arxivfootnote}.
We downloaded paper (uploaded between Jan 2019--Sept 2020) source code and corresponding compiled PDF. These articles belong to eight subject categories. Figure~\ref{fig:arxiv_freq} illustrates category-wise paper distribution. As illustrated, the majority of the papers belong to three subject categories, physics (33.93\%), computer science (25.79\%), and mathematics (23.27\%). Overall, we downloaded 347,502 papers and processed them using the proposed data processing pipeline (described in the next section).

\begin{figure}[!t]
\centering
\resizebox{\hsize}{!}{
\includegraphics[]{figures/arxiv_paper_freq.pdf}
}
\caption{Total number of papers in arXiv's subject categories. Here SS denotes Systems Science.}
\label{fig:arxiv_freq}
\end{figure}

\subsection{Data Processing Pipeline}
The following steps present a detailed description of the data processing pipeline.
\begin{figure}[!tbh]
\setlength{\tabcolsep}{1pt}
\begin{tabular}{cc}
    \includegraphics[scale=0.3]{figures/font1.jpg} & \includegraphics[scale=0.52]{figures/LATEX_SOURCE_CODE.png} \\
    (a) A real table image & (b) Corresponding \LaTeX~source code\\
    \includegraphics[scale=0.55]{figures/PROCESSED_TABLE_SNIPPET.png} & \includegraphics[scale=0.65]{figures/latex_template.PNG}\\
     (c) Extracted table snippet & (d) \LaTeX~code template\\
     \includegraphics[scale=0.55]{figures/CHARTER_SOURCE_CODE_1.png} & \includegraphics[scale=0.3]{figures/font3.jpg}\\
      (e) Code with \textsc{Charter} font package& (f) The final generated table image \\
\end{tabular}
    \caption{Data processing pipeline.}
    \label{fig:pipeline}
\end{figure}

\subsubsection{\LaTeX~Code Pre-Processing Steps}
\begin{enumerate}
    \item \textbf{Table Snippets Extraction:} A table snippet is a part of \LaTeX~code that begins with `\verb+\begin{tabular}+' and ends with `\verb+\end{tabular}+' command. During its extraction, we removed citation command `\verb+\cite{}+', reference command `\verb+\ref{}+', label command `\verb+\label{}+', and graphics command `\verb+\includegraphics[]{}+', along with $\sim$ symbol (preceding these commands). Also, we remove  command pairs (along with the content between them) like  `\verb+\begin{figure}+' and `\verb+\end{figure}+', and `\verb+\begin{subfigure}+' and `\verb+\end{subfigure}+', as they cannot be predicted from the tabular images. Furthermore, we also remove the nested table environments. Figure~\ref{fig:pipeline}a and \ref{fig:pipeline}b show an example table and its corresponding \LaTeX~source code, respectively.
    \item \textbf{Comments Removal:} Comments are removed by removing the characters between `\verb
    \item \textbf{Column Alignment and Rows Identification:} We keep all possible alignment tokens (`\verb+l+', `\verb+r+', and `\verb+c+') specified for column styling. The token `\verb+|+' is also kept to identify vertical lines between the columns, if present. The rows are identified by keeping the `\verb+\\+' and `\verb+\tabularnewline+' tokens. These tokens signify the end of each row in the table. For example, Figure~\ref{fig:pipeline}c shows extracted table snippet from \LaTeX~source code (see Figure~\ref{fig:pipeline}b) containing the column and rows tokens with comment statements removed.
    \item \textbf{Font Variation:} In this step, the extracted \LaTeX~code is augmented with different font styles. We experiment with a total 12 different \LaTeX~font packages\footnote{\url{https://www.overleaf.com/learn/latex/font_typefaces}}. We use popular font packages from PostScript family which includes `\verb+courier+', `\verb+helvet+', `\verb+palatino+', `\verb+bookman+', `\verb+mathptmx+', `\verb+utopia+' and also other font packages such as `\verb+tgbonum+', `\verb+tgtermes+', `\verb+tgpagella+', `\verb+tgschola+', `\verb+charter+ and `\verb+tgcursor+'. For each curated image, we create 12 variations, one in each of the font style.
    \item \textbf{Image Rendering:} Each table variant is compiled into a PDF using a \LaTeX~code template (described in Figure~\ref{fig:pipeline}d). Here, \textit{`table snippet'} represents the extracted tabular \LaTeX~code and \textit{`font package'} represents the \LaTeX~font package used to generated the PDF. The corresponding table PDF files are then converted into JPG images using the Wand library\footnote{\url{https://github.com/emcconville/wand}} which uses ImageMagick~\cite{imagemagick} API to convert PDF into images. Figure~\ref{fig:pipeline}e shows the embedded code and font information within the template. Figure~\ref{fig:pipeline}f shows the final table image with \textsc{Charter} font. During conversion, we kept the image's resolution as 300 dpi, set the background color of the image to white, and removed the transparency of the alpha channel by replacing it with the background color of the image. We use two types of aspect ratio variations during the conversion, described as follows:
    \begin{enumerate}
    \item \textit{Conserved Aspect Ratio:}
    In this case, the bigger dimension (height or width) of the image is resized to 400 pixels\footnote{We use `400' pixels as an experimental number.}. We then resize the smaller dimension (width or height) by keeping the original aspect ratio conserved. During resizing, we use a blur factor of 0.8 to keep images sharp.
    \item \textit{Fixed Aspect Ratio:}
    The images are resized to a fixed size of 400$\times$400 pixels using a blur factor of 0.8. Note that this resizing scheme can lead to extreme levels of image distortions.
\end{enumerate}

\end{enumerate}

\iffalse{}
\begin{figure}[h]
\centering
    \subfloat[\centering ]{{\includegraphics[width=5cm]{figures/font1.jpg} }}%
    \qquad
    \subfloat[\centering  ]{{\includegraphics[width=5cm]{figures/font3.jpg} }}%
    \caption{A curated table image with different font styles. (a) \textsc{Times} font and (b) \textsc{Charter} font.}
    \label{fig:table_fonts}%
\end{figure}

\begin{figure}[h]
\centering
    \subfloat[]{{\includegraphics[scale=0.6]{figures/TIMES_SOURCE_CODE_1.png} }}%
    \qquad
    \subfloat[]{{\includegraphics[scale=0.6]{figures/CHARTER_SOURCE_CODE_1.png} }}%
    \caption{LaTeX source codes for table images in Figure~\ref{fig:table_fonts} with (a)\textsc{mathptmx} font package for \textsc{TIMES} font and (b).}
    \label{fig:font_source_codes}%
\end{figure}

\begin{figure}[h]
    \small
    \centering
    \includegraphics[scale=0.8]{figures/latex_template.PNG}
    \caption{LaTeX code template for compilation with different fonts.}
    \label{fig:latex_template}
\end{figure}

\fi

\subsubsection{\LaTeX~Code Post-Processing Steps for Ground-Truth Preparation}
\begin{enumerate}
    \item \textbf{Filtering Noisy \LaTeX~tokens:} We filter out \LaTeX~environment tokens (tokens starting with `\verb+\+') having very less frequency (less than 5000) in the overall corpus compared to other \LaTeX~environment tokens and replaced it with `\verb+\LATEX_TOKEN+' as these tokens will have a little contribution. This frequency-based filtering reduced the corpus's overall vocabulary size, which helps in training models on computation or memory-constrained environments.
    \item \textbf{Structure Identification:} In this post-processing step, we create Table Structure Dataset (TSD). It consists of structural information corresponding to the table images. We keep tabular environment parameters as part of structure information and replace tokens representing table cells' content with a placeholder token `CELL'. This post-processing step creates ground truth for table images in TSD. Specifically, the vocabulary comprises digits (0-9), `\&',  `CELL', `\verb+\\+', `\verb+\hline+', `\verb+\multicolumn+',  `\verb+\multirow+', `\verb+\bottomrule+', `\verb+\midrule+', `\verb+\toprule+', `\verb+|+', `l', `r', `c', `\{', and `\}'. A sample of structure information is shown in Figure~\ref{fig:table_example}b, where the `CELL' token represents a cell structure in the table. Structural information can be used to identify the number of cells, rows, and columns in the table using `CELL', \verb+`\\'+ and alignment tokens (`c', `l', and `r'), respectively. Based on output sequence length, we divided this dataset further into two variants TSD-250 and TSD-500, where the maximum length of the output sequence is 250 and 500 tokens, respectively.
    \item \textbf{Content Identification:} Similar to TSD, we also create a Table Content Dataset (TCD). This dataset consists of content information including alphanumeric characters, mathematical symbols, and other \LaTeX~environment tokens. Content tokens are identified by removing tabular environment parameters and keeping only the tokens that identify table content. Specifically, the vocabulary includes all the alphabets (a-z and A-Z), digits (0-9), \LaTeX~environment tokens (`\verb+\textbf+', \verb+\hspace+', etc.), brackets (`\verb+(+', `\verb+)+', `\verb+{+', `\verb+}+', etc.) and all other possible symbols (`\verb+$+', `\verb+&+', etc.). In this dataset, based on output sequence length, we divided it further into two variants TCD-250 and TCD-500, where the maximum length of the output sequence is 250 and 500 tokens, respectively. A sample of content information is shown in Figure~\ref{fig:table_example}c.
\end{enumerate}

\subsection{Dataset Statistics}
We further partition each of the four dataset variants TSD-250, TSD-500, TCD-25, and TCD-500, into training, validation, and test sets in a ratio of 80:10:10. Table~\ref{tab_datasets} shows the summary of the dataset. Note that the number of samples present in the TSD and the corresponding TCD can differ due to variation in the output sequence's length.  Also, the average number of tokens per sample in TCD is significantly higher than the TSD due to more information in the tables' content than the corresponding structure. Figure~\ref{histograms} demonstrates histograms representing the token distribution for the TSD and TCD. In the case of TSD, the majority of tables contain less than 25 tokens. Overall the token distribution shows a long-tail behavior. In the case of TCD, we witness a significant proportion between 100--250 tokens. The dataset is licensed under \href{https://creativecommons.org/licenses/by-nc-sa/4.0/}{Creative Commons Attribution 4.0 International License} and available for download at \url{https://www.tinyurl.com/tablatex}.

\begin{table}[!t]
\caption{\textsc{TabLeX} Statistics. ML denotes the maximum length of output sequence. Samples denotes the total number of table images. T/S denotes the average number of tokens per sample. VS denotes the vocabulary size. AR and AC denote the average number of rows and columns among all the samples in the four dataset variants, respectively.}
\label{tab_datasets}
\centering
\begin{tabular}{|l|c|c|c|c|c|c|c|}
		\hline
        \multicolumn{1}{|l|}{\textbf{Dataset}} &
		\multicolumn{1}{|c}{\textbf{ML}} &
		\multicolumn{1}{|c}{\textbf{Samples}} &
        \multicolumn{1}{|c}{\textbf{Train/Val/Test}} &
        \multicolumn{1}{|c|}{\textbf{T/S}} &
        \multicolumn{1}{|c|}{\textbf{VS}} &
        \multicolumn{1}{|c|}{\textbf{AR}} &
        \multicolumn{1}{|c|}{\textbf{AC}} \\ \hline
        TSD-250 & 250 & \num{2938392}  & \num{2350713} / \num{293839} / \num{293840} & \num{74.09} & 25 & 6 & 4\\ \hline
        TSD-500 & 500 & \num{3191891}  & \num{2553512} / \num{319189} / \num{319190} & \num{95.39} & 25 & 8 & 5\\ \hline
  		TCD-250 & 250 & \num{1105636} & \num{884508} / \num{110564} / \num{110564} & 131.96 & 718 & 4 & 4\\ \hline
  		TCD-500 & 500 & \num{1937686} & \num{1550148} / \num{193769} / \num{193769} & \num{229.06} & 737 & 5 & 4\\ \hline
	\end{tabular}
\end{table}

\begin{figure}[h]
\begin{subfigure}[h]{.5\textwidth}
\centering
\resizebox{\textwidth}{!}{
\includegraphics{figures/TSD-250-freq_.pdf}
}
\caption{TSD-250}
\end{subfigure}
\hfill
\begin{subfigure}[h]{.5\textwidth}
\centering
\resizebox{\textwidth}{!}{
\includegraphics{figures/TSD-500-freq_.pdf}
}
\caption{TSD-500}
\end{subfigure}
\hfill
\begin{subfigure}[h]{.5\textwidth}
\centering
\resizebox{\textwidth}{!}{
\includegraphics{figures/TCD-250-freq_.pdf}
}
\caption{TCD-250}
\end{subfigure}
\begin{subfigure}[h]{.5\textwidth}
\centering
\resizebox{\textwidth}{!}{
\includegraphics[scale=0.65]{figures/TCD-500-freq_.pdf}
}
\caption{TCD-500}
\end{subfigure}
\caption{Histograms representing the number of tokens distribution for the tables present in the dataset.}
\label{histograms}
\end{figure}

\section{Evaluation Metrics}
\label{sec:metrics}
We experiment with three evaluation metrics to compare the predicted sequence of tokens ($T_{PT}$) against the sequence of tokens in the ground truth ($T_{GT}$). Even though the proposed metrics are heavily used in NLP research, few TIE works have leveraged them in the past. Next, we discuss these metrics and illustrate the implementation details using a toy example described in Figure~\ref{fig:dummy_table}.

\begin{figure}
    \centering
    \subfloat[\centering A Toy Table ]{{\includegraphics[scale=0.95]{figures/dummy_table.pdf} }}%
    \label{a}
    \qquad
    \subfloat[\centering Ground Truth ]{{\includegraphics[scale=0.7]{figures/dummy_latex_code.png} }}%
    \qquad
    \subfloat[\centering Table Content Output ]{{\includegraphics[scale=0.7]{figures/tcd_out.png} }}%
    \qquad
    \subfloat[\centering Table Structure Output ]{{\includegraphics[scale=0.7]{figures/tsd_out.png} }}%
    \caption{(a) A toy table, (b) corresponding ground truth token sequence for TSR and TCR tasks, (c) model output sequence for TCR task, and (d) model output sequence for TSR task.}
    \label{fig:dummy_table}%
\end{figure}

\begin{enumerate}
    \item \noindent\textbf{Exact Match Accuracy (EMA):} EMA outputs the fraction of predictions with exact string match of $T_{PT}$ against $T_{GT}$. A model having a higher value of EMA is considered a good model. In Figure~\ref{fig:dummy_table}, for the TSR task, $T_{PT}$ misses `\verb+\hline+' token compared to the corresponding $T_{GT}$, resulting in EMA = 0. Similarly, the $T_{PT}$ exactly matches the $T_{GT}$ for the TCR task, resulting in EMA = 1.

    \item \noindent\textbf{Bilingual Evaluation Understudy Score (BLEU):}
  The BLEU~\cite{bleu} score is primarily used in Machine Translation literature to compare the quality of the translated sentence against the expected translation.  Recently, several TIE works~\cite{Deng2016WhatYG,im2tex} have adapted BLEU for evaluation. It counts the matching n-grams in $T_{PT}$  against the n-grams in $T_{GT}$.  The comparison is made regardless of word order. The higher the BLEU score, the better is the model. We use SacreBLEU~\cite{sacrebleu} implementation for calculating the BLEU score. We report scores for the most popular variant, BLEU-4. BLEU-4 refers to the product of brevity penalty (BP) and a harmonic mean of precisions for unigrams, bigrams, 3-grams, and 4-grams (for more details see~\cite{bleu}). Figure~\ref{fig:dummy_table}, there is an exact match between $T_{PT}$ and $T_{GT}$ for TCR task yielding BLEU = 100.00. In the case of TSR, the missing `\verb+\hline+' token in $T_{PT}$ yields a lower BLEU = 89.66.

    \item \noindent\textbf{Word Error Rate (WER):} WER is defined as a ratio of the Levenshtein distance between the $T_{PT}$ and $T_{GT}$ to the length of $T_{GT}$.  It is a standard evaluation metric for several OCR tasks. Since it measures the rate of error, models with lower WER are better. We use jiwer\footnote{\url{https://github.com/jitsi/jiwer}} Python library for WER computation. In Figure~\ref{fig:dummy_table}, for the TCR task, the WER between $T_{PT}$ and $T_{GT}$ is 0. Whereas in the case of TSR, the Levenshtein distance between $T_{PT}$ and $T_{GT}$ is one due to the missing `\verb+\hline+' token. Since the length of $T_{GT}$ is 35, WER comes out to be 0.02.

\end{enumerate}

\section{Experiments}
\label{sec:exp}
In this section, we experiment with a deep learning-based model for TSR and TCR tasks. We adapt an existing model~\cite{feng2020scene} architecture proposed for the scene text recognition task and train it from scratch on the \textsc{TabLeX} dataset. It uses partial ResNet-101 along with a fully connected layer as a feature extractor module for generating feature embeddings with a cascaded Transformer~\cite{vaswani2017attention} module for encoding the features and generating the output. Figure~\ref{fig:baseline} describes the detailed architecture of the model. Note that in contrast to the scene image as an input in ~\cite{feng2020scene}, we input a tabular image and predict the \LaTeX~token sequence. We term it as the TIE-ResNet-Transformer model (\textit{TRT}).

\begin{figure}[h!]
\centering
\includegraphics[scale=0.6]{figures/baseline_diagram.pdf}
\caption{TIE-ResNet-Transformer model architecture.}
\label{fig:baseline}
\end{figure}

\subsection{Implementation Details}
For both TSR and TCR tasks, we train a similar model. We use the third intermediate hidden layer of partial ResNet-101 and an FC layer as a feature extractor module to generate a feature map. The generated feature map (size = $625\times1024$) is further embedded into 256 dimensions, resulting in reduced size of $625\times256$. We experiment with four encoders and eight decoders with learnable positional embeddings. The models are trained for ten epochs with a batch size of 32 using cross-entropy loss function and Adam Optimizer with an initial learning rate of $0.1$ and 2000 warmup training steps, decreasing with Noam's learning rate decay scheme. A dropout rate of 0.1 and $\epsilon_{ls}=0.1$ label smoothing parameter is used for regularization. For decoding, we use the greedy decoding technique in which the model generates an output sequence of tokens in an auto-regressive manner, consuming the previously generated tokens to generate the next token. All the experiments were performed on 4$\times$NVIDIA V100 graphics card.
\subsection{Results}

\begin{table}[h]
    \caption{TRT results on the \textsc{TabLeX} dataset.}
	\label{results}
	\centering
	\resizebox{.7\hsize}{!}{
    \begin{tabular}{| c | c || c | c | c | }
        \hline
        Dataset & Aspect Ratio & EMA(\%) & BLEU & WER(\%) \\ \hline

    \multirow{2}{*}{\makecell{TCD \\ 250}} & Conserved & 21.19 & 95.18 & 15.56  \\
         \cline{2-5}

     & Fixed & 20.46 & 96.75 & 14.05
     \\ \hline

    \multirow{2}{*}{\makecell{TCD\\ 500}} & Conserved  & 11.01 & 91.13 & 13.78 \\
            \cline{2-5}

    & Fixed & 11.23 & 94.34 & 11.12  \\
        \hline\hline

    \multirow{2}{*}{\makecell{TSD \\ 250}}  & Conserved  & 70.54 & 74.75 & 3.81 \\
         \cline{2-5}

    & Fixed & 74.02 & 70.59 & 4.98  \\
    \hline

    \multirow{2}{*}{\makecell{TSD\\ 500}}  & Conserved  & 70.91 & 82.72 & 2.78 \\
            \cline{2-5}

    & Fixed & 71.16 & 61.84 & 9.34  \\
         \hline
    \end{tabular}
    }
\end{table}

Table~\ref{results} illustrates that higher sequence length significantly degrades the EMA score for both the TSR and TCR tasks. For TCD-250 and TCD-500, high BLEU scores ($>$90) suggest that the model can predict a large chunk of \LaTeX~content information correctly. However, the errors are higher for images with a conserved aspect ratio than with a fixed aspect ratio, which is confirmed by comparing their BLEU score and WER metrics. Similarly, for TSD-250 and TSD-500, high EMA and lower WER suggest that the model can correctly identify structure information for most of the tables.
TCD yields a lower EMA score than TSD. We attribute this to several reasons. One of the reasons is that the TCR model fails to predict some of the curly braces (`\{' and `\}') and dollar (`\$') tokens in the predictions. After removing curly braces and dollar tokens from the ground truth and predictions, EMA scores for conserved and fixed aspect ratio images in TCD-250 increased to 68.78\% and 75.33\%, respectively. Similarly, for TCD-500, EMA for conserved and fixed aspect ratio images increases to 49.23\% and 59.94\%, respectively. In contrast, TSD do not contain dollar tokens, and curly braces are only present at the beginning of the column labels, leading to higher EMA scores. In the future, we believe the results can be improved significantly by proposing better vision-based DL architectures for TSR and TCR tasks.

\section{Conclusion}
\label{section6}
This paper presents a benchmark dataset \textsc{TabLeX}  for structure and content information extraction from scientific tables. It contains tabular images in 12 different fonts with varied visual complexities. We also proposed a novel preprocessing pipeline for dataset generation. We evaluate the existing state-of-the-art transformer-based model and show excellent future opportunities in developing algorithms for IE from scientific tables. In the future, we plan to continuously augment the dataset size and add more complex tabular structures for training and prediction.

\section*{Acknowledgment}
This work was supported by The Science and
Engineering Research Board (SERB), under sanction number ECR/2018/000087.

\bibliographystyle{splncs04}

Information Retrieval (cs.IR); Machine Learning (cs.LG)


Reconstruct the Directories for In-Memory File Systems

Abstract: Existing path lookup routines in file systems need to construct an auxiliary index in memory or traverse the dentries of the directory file sequentially, which brings either heavy writes or large timing cost. This paper designs a novel path lookup mechanism, Content-Indexed Browsing (CIB), for file systems on persistent memory, in which the structure of directory files is an exclusive index that can be searched in $O(log(n))$ time. We implement CIB in a real persistent memory file system, PMFS, denoted by CIB-PMFS. Comprehensive evaluations show that CIB can achieve times of performance improvement over the conventional lookup schemes in PMFS, and brings 20.4\% improvement on the overall performance of PMFS. Furthermore, CIB reduces the writes on persistent memory by orders of magnitude comparing with existing extra index schemes.

\section{Introduction}

Emerging persistent memory (PM) technologies \cite{3DX,burr2010phase,PCMs} provide persistent storage to complement DRAM in two aspects.
First, PMs offer significant speedup over disks and flashes.
Second, as illustrated in Figure \ref{fig:overview}, PMs can be directly accessed via load/store instructions and thus bypass the deep software stack in OS.
In recent years, many novel designs of data storage infrastructures (e.g., database and file systems) achieve higher performance and less write amplification over traditional designs by leveraging the byte-addressability of PM
 \cite{condit2009better,hu2013software,coburn2012nv,chen2011rethinking,chen2015persistent,yang2015nv,wu2011scmfs,xu2016nova,dulloor2014system}.

In traditional file systems, as the number of files growing in a certain directory, the latency of path lookup routine linearly scales up.
This is because the dentries in directory files are stored out-of-order and sequential lookup is required for searching a certain dentry.
To accelerate the lookup routine, Virtual File System (VFS) builds a hashing cache for the recently visited path.
However, for streaming applications, where most files are accessed only once, the cache mechanism fails to improve performance.
Moreover, since page cache is bypassed in PM access, such hashing cache is obsoleted.

Nevertheless, we observe that the designs for directory files of PM-oriented file systems mainly follow to that of traditional file systems on block devices.
Due to the inefficient lookup routine, these schemes are incompetent to fully exploit the potential of PMs.
To address this issue, SanGuo \cite{zeng2017efficient} devises extra indexes in DRAM for each directory file.
Indexes with tree-based structure provide steady performance for directory operations, but it needs reconstruction after power failures.
On the other hand, maintaining tree indexes persistently (like BtrFS \cite{rodeh2013btrfs}) brings extra overhead and performance penalties.
To make matters worse, modifying tree structure imports a large number of write activities on PM.
This observation implicates that the conventional directory designs degrade either performance or PM-endurance.

\begin{figure}[ht]
    \centering
    \includegraphics[width=2.8 in,height = 2.1 in]{Figure/Intro_overview.pdf}
    \caption{Direct access on PMs.}
    \label{fig:overview}
\end{figure}
In this paper, we propose a novel path lookup mechanism, Content-Indexed Browsing (CIB).
To speed up the dentry queries, we replace the conventional structure of directory files by a novel content aware index.
We also reconstruct the dentries and redefine the whole lookup routine.
With the consideration of diverse access pattern between directory and data files, we redesign the operations for directory files.
By incorporating the characteristics of indexes and PM-based file systems, we can achieve optimal performance with minimum PM writes.
Specifically, the dentries in directory files can be searched in $O(log(n))$ time without building extra indexes.

We implement the proposed CIB scheme in PMFS \cite{dulloor2014system} (denoted by CIB-PMFS), a real persistent memory file system.
Then, we evaluate the performance and write activities of CIB via the widely-used benchmark Filebench \cite{tarasov2016filebench}.
Extensive experimental results show that CIB achieves 5.45$\times$ to 176.08$\times$ performance improvement over the lookup scheme in PMFS for different types of workloads, which brings 8\% to 20.4\% improvement on the overall performance.
Furthermore, CIB reduces PM writes by orders of magnitude compared with extra index schemes.

Our main contributions are listed as follows.
\begin{itemize}[noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt]
  \item We conduct an in-depth investigation of the drawbacks in structure and lookup routine of existing file systems.
  \item We propose a novel Content-Indexed Browsing (CIB) scheme and redefine the structures and routines in terms of directory files.
  \item We implement our scheme in PMFS, denoted by CIB-PMFS. Substantial experiments show that CIB-PMFS can improve performance and reduce writes on PM in comparison with the original PMFS.
\end{itemize}

The rest of this paper is organized as follows.
In Section~\ref{Sec:Motivation}, we introduce the existing lookup schemes of existing file systems and show a motivational example with experiments.
We present the design and implementation of the CIB scheme in Section~\ref{Sec:DesignFramework} and Section~\ref{Sec:implementation}, respectively.
We show the evaluation results of CIB in Section~\ref{Sec:Exp}.
Section~\ref{Sec:Con} concludes the paper.

\section{Background and Motivation} \label{Sec:Motivation}

With the continuous progress of researches in Persistent Memory (PM) technologies, advanced in-memory systems are developed in succession \cite{tan2015memory,arulraj2018bztree,burr2008overview}.
Compared with block devices, there are several advantages of PMs.
First, PMs have orders of magnitude less read/write latency than block devices;
Second, PMs connect to CPU using load/store instructions rather than deep I/O stack, reducing overhead caused by storage hierarchy \cite{wang2018lawn}.
Third, the byte-addressability of PMs facilitates the design of file systems. Specifically, PM-based systems can easily achieve lower write amplification facing small-grained write activities.
However, some challenges stay in the design of PM-based systems.
PMs have asymmetric read and write latencies, e.g., the writing activities in PCM can be 20$\times$ slower than reading.
Moreover, write activities shorten the lifetime of PMs.
Therefore, designs for PM-based systems should not impose overmuch writing activities on PMs.
Based on the above considerations, we review the existing lookup schemes for in-memory systems and propose our motivation.

\begin{figure}[tb]
    \centering
    \includegraphics[width=2.6 in,height = 2 in]{Figure/Moti_Sequential.pdf}
    \caption{Conventional structure of directory files.}
    \label{fig:sequential}
\end{figure}

\subsection{Problems of Conventional Lookup Scheme}
Traditional structure of directory files is illustrated as Figure \ref{fig:sequential}.
When looking for a certain filename, the file system needs to sequentially seek dentries in a directory file.
This process takes $O(n)$ time to complete.
For block-based file systems, the performance of sequential reading is acceptable.

\begin{figure}[ht]
    \centering
\setlength{\abovecaptionskip}{0.2cm}
\setlength{\belowcaptionskip}{-0.2cm}
    \includegraphics[width=3.2 in,height = 1.1 in]{Figure/Moti_DLength.pdf}
    \caption{Conventional structure of dentries.}
    \label{fig:dlen}
\end{figure}

However, in PMFS, such conventional lookup schemes degrade the performance of file systems.
In particular, create operations magnify the defect in two ways.
First, it needs to traverse the whole directory to confirm if a file with the same name has been created.
Second, it needs to scan the \emph{d\_len} and \emph{name\_len} of each dentry to find a free space to insert the new one, as shown in Figure \ref{fig:dlen}.
Moreover, as for open operation, such one-by-one searching routines will be iteratively invoked during path lookup.
Above observation reveals that utilizing conventional lookup scheme cannot fully exploit the advantages of random read/write performance in the PM-based scenario.

\begin{figure}[ht]
    \centering
\setlength{\abovecaptionskip}{0.2cm}
\setlength{\belowcaptionskip}{-0.1cm}
    \includegraphics[width=2.8 in,height = 1.3 in]{Figure/Moti_ratio.pdf}
    \caption{The proportion of time spent on directory operations.}
    \label{fig:ratio}
\end{figure}

We give a motivational example to demonstrate the inefficiency of conventional lookup scheme.
In our example, we first create 100 to 10K files in a directory, while writing 400K data for each file.
Then, we record the time of creating files $C$ and the total elapsed time $T$, respectively.
To put it plainly, $C$ represents the time spent on directory operations, while $T$ represents the total time employed by the file system.
Finally, we compare the ratio of $C$ to $T$.
The time $C$ scales exponentially with the increasing number of files.
This motivational example evidently illustrates the inefficiency of conventional lookup schemes.

\subsection{Problems of Extra Tree Indexes}
To improve the time performance of directory operations, recent schemes propose extra tree-based structures in DRAM as the index of directory files \cite{xu2016nova,rodeh2013btrfs}.
However, after power failures, they need to rebuild these extra structures to reboot the system.
As PMs provide persistent massive storage with DRAM-class speed, it is possible to put the auxiliary structures into PMs to accelerate the boot routine.

 \begin{figure}[t]
    \centering
\setlength{\abovecaptionskip}{0.3cm}
\setlength{\belowcaptionskip}{-0.1cm}
    \includegraphics[width=3.3 in,height=2.1 in]{Figure/Moti_ExtraIndex-eps-converted-to2.pdf}
    \caption{Extra tree index of dentries.}
    \label{fig:tree}
\end{figure}
However, such extra indexes are inadequate to be directly deployed in PMs.
Specifically, tree structures incur a large quantity of PM writes and massive memory consumption in three aspects.
\emph{First,} maintaining tree structure incurs a large quantity of PM writes.
When building tree indexes, filenames and other content of dentries will be inserted to leaf nodes.
This operation brings sorting of entries in traditional tree structures, which imports significant PM writes.
Insertions also trigger split operation when a leaf node is full, as shown in Figure \ref{fig:tree}.
Moreover, such split operation may be propagated to the root when its parent nodes are also full.
This chain effect imports considerable PM writes and therefore incurs degradation on performance and depresses the endurance of PMs.

\emph{Second,} for large-scale storages, maintaining tree indexes suffers from large memory consumption.
As mentioned above, content of dentries will be copied to leaf nodes, which nearly doubles the size of directory files.
What's more, massive internal nodes in tree structure exacerbate the consumption.
This problem lies in both DRAM and PM based systems.

\emph{Third,} on the consideration of consistency, tree structures usually adopt logging or shadowing mechanism, either of which aggravates the number of PM writes.
In another word, it sacrifices the performance of file system to maintain the validity of extra indexes.

Based on above analysis, it can be concluded that a novel lookup scheme is demanded to accelerate the directory operations with minimum overhead for PM-based file systems.
Our design principles of the lookup scheme for in-memory file systems are given as follows.
\begin{itemize}[noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt]
  \item Operations on directory files should not traverse the whole field of dentries.
  \item On the consideration of memory consumption, the size of the extra structure should be minimized.
  \item The novel scheme should avoid too much consistency and maintenance overhead that will severely degrade system performance.
\end{itemize}

In the following two sections, we propose a novel lookup scheme CIB for in-memory file systems.

\section{CIB Design}\label{Sec:DesignFramework}

In the design of the lookup scheme, we need to resolve several problems.
First, how to redesign an elegant lookup routine to prevent sequential lookups.
Second, how to reconstruct directory files and reorganize dentries to facilitate the acceleration method.
Third, how to continually improve performance rather than introducing extra overhead.
This section will introduce the novel lookup scheme, ``Content-Indexed Browsing'' (CIB) to address the above problems.

\subsection{Regularize Dentries into fixed size}
Motivated by section \ref{Sec:Motivation}, sequential lookup performs poor with the increasing data size of directory files.
Inspired by indexing schemes, we decide to aggregate dentries in terms of uniform keys rather than irregular filenames.
To make this possible, we redesign the structure of dentries.
As illustrated in Figure \ref{fig:dentry}, besides filename and inode number, we also store the hash key of filename.

\begin{figure}[thb]
    \centering
\setlength{\abovecaptionskip}{0.2cm}
\setlength{\belowcaptionskip}{-0.1cm}
    \includegraphics[width=2.3 in,height=0.6 in]{Figure/Frame_dentry-eps-converted-to2.pdf}
    \caption{The structure of a dentry.}
    \label{fig:dentry}
\end{figure}

To avoid modifying \emph{d\_len} and \emph{name\_len} in traditional schemes, we regularize the size of dentries to be fixed.
As the length of a filename is variable, we design a \emph{nameptr} to point to the real filename, so that we can use a pointer to represent a filename.
This structure of dentry fully leverages the byte-addressability of PMs.
Different from the traditional dentry searching, we first use a hash key to find the possible dentry, then we check its corresponding filename.
This mechanism utilizes uniform hash keys to reduce the overhead of comparing filenames.
Since comparing strings is much inefficient than comparing integer keys, CIB can significantly promote dentry lookups.

\subsection{Reconstruct Directory Files}
In PMFS, blocks of data are indexed by pointers stored in inodes.
This facilitates the random access of blocks for data files.
However, sequential search in directory files makes the index redundant.
Now, as we aggregate the dentries in terms of hashing keys, the structure of directory files needs to be totally redesigned.
In our scheme, the blocks of directory files are linked as a sorted list.
As shown in Figure \ref{fig:datapage}, each block has a minimum and maximum hash key to identify the range of keys stored in the block.
Each block contains dentries whose keys are within the range and the non-overlapping ranges of blocks in the list are relatively ordered.
Since the size of dentries is designed to be fixed, we can utilize a bitmap to indicate the state of each dentry.
Finally, the field \emph{next page} points to the following block in the linked list.

\begin{figure}[tb]
    \centering
\setlength{\abovecaptionskip}{0.1cm}
\setlength{\belowcaptionskip}{-0.1cm}
    \includegraphics[width=3.5 in,height = 1.1 in]{Figure/Frame_datapage-eps-converted-to2.pdf}
    \caption{Reconstruction of directory blocks.}
    \label{fig:datapage}
\end{figure}

Based on the design, we mitigate the overhead for identifying dentries and maintaining the index.
Furthermore, we only need to store a single pointer in inode to access the head of the list, rather than store the whole index like traditional schemes.
And it is rather easy to guarantee the consistency of such linked lists without modifying inodes.

\subsection{Adaptive Acceleration Mechanism}
The directory files are frequently accessed and modified in file systems.
Accelerating the operations in terms of dentries is quite crucial for sustaining great performance of file systems.
Thus, we propose an adaptive acceleration mechanism to facilitate the access of blocks.
For directory files in small sizes, e.g., all dentries are stored in one block, sequential lookup for dentries has no side effect on performance.
As for directory files with a lengthy linked list of blocks, we build an auxiliary structure to improve its performance.

Inspired by high-performance binary search upon nodes in {B$^+$-tree}, we intend to make the scattered blocks ``gathered and flat''.
First of all, we apply an array of contiguous virtual addresses.
Then, we build a linear mapping between the array and the blocks.
As shown in Figure \ref{fig:blocklist}, we store the head address of the array into inode.
By this way, we can access the linked list of blocks randomly.
Thus, efficient searching algorithms such as binary search can be conducted upon our blocks.

It is worth noting that for the existing extra-indexing scheme, both the free space and inserted dentries in directory files need building exclusive indexes to acquire efficient management.
Otherwise, create operations still require sequential searches to find a segment of proper free space to insert new dentries.
On the contrary, CIB can efficiently manage the whole space of directory files by leveraging only a small array of addresses.
This is because CIB makes full use of the futures provided by reconstructed dentries and blocks.
Therefore, CIB significantly reduces the PM writes and memory consumption over extra index schemes.

\begin{figure}[thb]
    \centering
\setlength{\abovecaptionskip}{0.2cm}
\setlength{\belowcaptionskip}{-0.1cm}
    \includegraphics[width=3 in, height=1.2 in]{Figure/Frame_blocklist-eps-converted-to2.pdf}
    \caption{The overview layout of CIB.}
    \label{fig:blocklist}
\end{figure}

By incorporating the futures of index and PMs, we reconstruct the dentries and directory files.
We abandon extra indexing schemes to avoid massive PM writes and design a new scheme to accelerate accessing the new structure.
Thus, the dentries are indexed with minimum overhead, which means, the directory files are \textbf{Content-Indexed}.
In the next section, we will detail the browsing routines of CIB.

\section{CIB implementation}\label{Sec:implementation}
This section first introduces the building routine of auxiliary structure.
Then we give the detailed basic routines for file operations based on our CIB.

\textbf{Building auxiliary structures.}
As the number of blocks in certain directory files grow up, we build an auxiliary array to accelerate locating blocks.
We first allocate a new array in virtual address space.
Then, we put the addresses of blocks into the array in order.
When accessing the cell in this array, we obtain the corresponding block.
By this way, we can randomly access dentry blocks through the array of continuous addresses.

\textbf{Open files.}
Open is the fundamental operation in file systems, since it is the pre-work of other operations, such as read and write operations.
Given the path of a certain file, the open operation is to lookup the path and find the inode number of the file.
As described in section \ref{Sec:DesignFramework}, we use \textbf{hash key} to filter out mismatched dentries with high efficiency, and use \textbf{filename} to confirm the dentry.
We conduct open operation as the following steps.
First, we hash the filename to obtain the key as the marker of the file.
Then, we conduct a binary search on the continuous virtual address using the hash key.
After locating the block that may contain the target dentry, we start searching inside the block.
When encountering a dentry with the same key, we will check its corresponding bit in the bitmap of the block to identify the state of dentry.
If the bit equals 1, indicating that this dentry is valid, we continue to check the filename of this dentry to confirm whether it is the target dentry.
Finally, we get the inode number of the target file or return an error code if the file doesn't exist.

\textbf{Create files.}
To create a new file, its parent directory file needs to insert a new dentry.
First of all, we hash the filename and make the new dentry.
Based on the open operation, we can easily check whether there is another file with a duplicate filename.
If not, we find a free position in the block according to the bitmap.
Then we insert the new dentry into this position and set the corresponding bitmap to 1.

Note that once the block is full of dentries, we will create a new block to amortize the dentries.
First, we divide dentries into two sets in terms of their hash keys.
Then we allocate a new block and modify the ''min'' and ''max'' hash keys.
We copy (not move, for consideration of arbitrary system failures) the set of dentries with larger keys into the new block.
Finally, we link the new bucket into the block-list just after the original one.

\textbf{Delete files.}
Different from the complex deletion routine of traditional schemes that will modify several variables in inodes, delete operation in CIB is rather concise.
Same as creation operation, we first locate the target dentry in terms of the key and filename.
After that, we atomically modify the corresponding bit in the bitmap to 0 to make this dentry invalid.
That is to say, we complete deletion of dentry with only 1 bit PM writes.

\textbf{Consistency and recovery.}
In CIB, most operations are atomically conducted without recording logs.
For building operation, the head of contiguous addresses is stored in the inode as a pointer.
Since PMs can provide atomic 8-byte writes, the modification of this pointer will not introduce inconsistency.
Similarly, we utilize this future to guarantee the consistency of creation and deletion operations.
Specifically, we manage the space of blocks and identify the validation of dentries using bitmaps, which can be flushed into PMs atomically.

As for applying a new block to amortize dentries, we set a flag in the bitmap to mark the status of the two affected blocks.
When a system failure occurs, there may be two inconsistent situations.
If the original one fails to link the newly created block, the whole list is still available after reboot.
If the new block has been inserted into the list while duplicate dentries in the original one are not invalidated yet, the flag in the bitmap will remind the system fixing this problem.
By this way, CIB costs minimum overhead for consistency and requires miniature recovery routines after system failures.

\section{Experimental Results}\label{Sec:Exp}

We implement the ``Content-Indexed Browsing'' ({CIB}) mechanism and the redesigned directory structure in PMFS \cite{dulloor2014system}, denoted as CIB-PMFS.
Evaluation experiments are conducted on a workstation equipped with 256GB main memory and two Intel(R) Xeon(R) E5-2640 processors.
We partition 10 GB DRAM to stand for the PM device used for PMFS.
We compare {CIB-PMFS} with PMFS in terms of time performance for directory-related operations.
The operating system is Ubuntu 16.04, with Linux kernel 4.4.4.
We evaluate the performance and writes of CIB via the widely-used benchmark Filebench \cite{tarasov2016filebench}.

\subsection{Performance Evaluation}
This subsection first evaluates the time performance of CIB and conventional lookup scheme of PMFS, then demonstrates the overall performance of CIB-PMFS and original PMFS.
We vary the number of files manipulated in each workload and record the elapsed time of directory operations.
Specifically, we first create $10^{3}$ to $10^{5}$ files to perform the loading phase of directory files.
Then we utilize other workloads to further demonstrate the efficiency of the CIB lookup scheme.
After that, we integrate CIB into PMFS to evaluate the overall performance of the file system.
Finally, we give a comparison between CIB and extra tree structure to prove the PM-friendliness of CIB.

\begin{table}[htbp]
  \centering
\setlength{\abovecaptionskip}{0.2cm}
\setlength{\belowcaptionskip}{-0.1cm}
  \caption{Elapsed time for inserting dentries (sec).}
    \begin{tabular}{|l|r |r |r |r |}
    \hline
    \centering
    \# of files   & 1K  & 10K & 50K & 100K\\
    \hline
    Trad.   & 0.017209 & 0.946182 &16.462971& 45.54016\\
    \hline
    CIB    & 0.003566 & 0.036723 & 0.16732&0.258627\\
    \hline
    \textbf{speedup}   & \textbf{4.83$\times$}  & \textbf{25.77$\times$}& \textbf{98.39$\times$}  & \textbf{176.08$\times$}  \\
    \hline
    \end{tabular}%
  \label{tab:Exp_loading}%
\end{table}%
Table \ref{tab:Exp_loading} reports the elapsed time of two schemes in creating phase.
In this table, line \emph{\# of files} represents the number of created files, while lines \emph{Trad.} and \emph{CIB} give the elapsed time of the traditional lookup scheme in PMFS and the proposed CIB respectively.
We have three observations for this result:
(1) CIB speedups over the conventional lookup scheme from 1.07$\times$ to 176.08$\times$.
(2) The elapsed time of the conventional lookup scheme scales up quickly as analyzed in section \ref{Sec:Motivation}.
Especially, when the number of files increases from $10^{4}$ to $10^{5}$, the elapsed time increases more than 48$\times$.
There are two reasons that cause performance degradation.
First, PMFS adopts inefficient sequential lookup scheme.
Second, PMFS compares every filename in the lookup routine, while string comparison costs a lot.
(3) CIB achieves stable performance in any magnitudes of files.
When the number of files is small, the increases of elapsed time are roughly the same with that of files.
However, the elapsed time for inserting $10^{5}$ dentries is only 1.65$\times$ to that for inserting $10^{6}$ dentries.
This is because CIB utilizes uniform hash keys to conduct binary search and thus dramatically reduce lookup overhead.

Based on various numbers of inserted dentries, we further evaluate CIB with traces generated by Filebench.
We choose two workloads \emph{Webproxy} and \emph{Varmail} to perform operations on two lookup schemes.
It can be observed that CIB significantly outperforms the traditional lookup scheme in dir-sensitive workloads.

\begin{figure}[htbp]
    \centering
\setlength{\abovecaptionskip}{0.3cm}
\setlength{\belowcaptionskip}{-0.1cm}
    \includegraphics[width=3 in,height=1.35 in]{Figure/Exp_create.pdf}
    \caption{Comparison of overall elapsed time.}
    \label{fig:Exp_create}
\end{figure}
Moreover, we evaluate the \textbf{overall} performance of the file system with various directory-related operations generated by Filebench Create workload.
It is noteworthy that, although overall evaluations create massive tiny-scale directory files and suffer transactional overheads, CIB-PMFS still improves the overall performance steadily.
The proposed CIB improves the overall performance of the file system by 8\% to 20.4\%.
This is because the CIB mechanism accelerates path lookup routine and thus facilitates all other file operations.
These experiments highlight the effectiveness of CIB with real-world traces.

\subsection{Comparison with Extra Index}
As BtrFS \cite{rodeh2013btrfs} and NOVA \cite{xu2016nova} utilize tree structures for lookup routines, we compare CIB with {B$^+$-tree} to prove the PM-friendliness of CIB.
We insert various numbers of dentries into each of them and record the number of PM writes.
Comparing with {extra tree indexes}, {CIB} mechanism reduces PM writes by orders of magnitude.
This is because CIB takes full advantage of the directory structure without building massive extra index structures.

Additionally, we compare the timing performance of inserting 50M, 60M, and 100M dentries, respectively.
CIB steadily outperforms extra index schemes in this evaluation.
Specifically, the elapsed times of CIB achieve 14.1$\times$, 15.6$\times$, and 20.2$\times$ reduction over the original {B$^+$-tree} for each workload.
It can be attributed to two reasons: 1) CIB conducts less string comparisons and structural changes; 2) CIB leverages a concise structure and thus mitigates consistent overhead.
This experiment proves that CIB is an efficient PM-friendly lookup scheme.

\section{Conclusion}\label{Sec:Con}
This paper investigates the existing lookup schemes from the perspective of PM-based in-memory systems.
We reconstruct the dentries and data blocks to mitigate redundant overhead.
Then we present a novel lookup scheme, Content-Indexed Browsing (CIB) to achieve significant performance improvement.
This scheme helps in-memory file systems further exploit the potential of PMs and prevent numerous PM writes.
We implement {CIB} in PMFS and conduct extensive evaluations to illustrate the efficiency of it.
Results show that {CIB} achieves times of speedup over the PMFS and significantly reduces the number of PM writes over extra index schemes.

The major point of this paper is to incorporate the merits of indexes into directory files to accelerate path lookups for PM-based file systems.
By reconstructing the dentries and directory blocks, it can retrieve the content of directory files efficiently without maintaining exclusive indexing structures.
In other words, the auxiliary structure implemented in this paper is not the unique and fixed matching for CIB.
Thus, the CIB provides the opportunity to be customized with other types of indexes (such as trees and hashing tables) orienting to application requirements.

Distributed, Parallel, and Cluster Computing (cs.DC)