Classroom Slide Narration System

Abstract: Slide presentations are an effective and efficient tool used by the teaching community for classroom communication. However, this teaching model can be challenging for blind and visually impaired (VI) students. The VI student required personal human assistance for understand the presented slide. This shortcoming motivates us to design a Classroom Slide Narration System (CSNS) that generates audio descriptions corresponding to the slide content. This problem poses as an image-to-markup language generation task. The initial step is to extract logical regions such as title, text, equation, figure, and table from the slide image. In the classroom slide images, the logical regions are distributed based on the location of the image. To utilize the location of the logical regions for slide image segmentation, we propose the architecture, Classroom Slide Segmentation Network (CSSN). The unique attributes of this architecture differs from most other semantic segmentation networks. Publicly available benchmark datasets such as WiSe and SPaSe are used to validate the performance of our segmentation architecture. We obtained 9.54 segmentation accuracy improvement in WiSe dataset. We extract content (information) from the slide using four well-established modules such as optical character recognition (OCR), figure classification, equation description, and table structure recognizer. With this information, we build a Classroom Slide Narration System (CSNS) to help VI students understand the slide content. The users have given better feedback on the quality output of the proposed CSNS in comparison to existing systems like Facebooks Automatic Alt-Text (AAT) and Tesseract.

\section{Introduction} \label{sec:intro}

Slide presentations play an essential role in classroom teaching, technical lectures, massive open online courses ({\sc mooc}), and many formal meetings. Presentations are an efficient and effective tool for the students to understand the subject better. However, the visually impaired ({\sc vi}) students do not benefit from this type of delivery due to their limitations in reading the slides. Human assistants could help {\sc vi} students to access and interpret the slide's content. However, it is impossible and impractical for the {\sc vi} students to have a human reader every time; because of the cost and the limited availability of trained personnel. From this perspective, this would be a primary issue in modern classrooms.

The automatic extraction of logical regions from these slide images is the initial step for understanding the slide's content. This can be defined as, segmentation of semantically similar regions, closely related to natural image semantic segmentation in computer vision~\cite{long2015fully,chen2017deeplab,zhao2017pyramid}. The goal is to assign logical labels like title, paragraph, list, equation, table, and figure to each pixel in the slide images. Due to the large variability in theme (i.e., the layout), style, and slide content, extracting meaningful regions becomes a challenging task. The text extraction from these regions is beneficial for the \textsc{vi} student only if the text is tagged with its logical function. The left image of Fig.~\ref{fig:teaser}, the plain text ``\textit{Discovering Attention Patterns}'' is more meaningful to the \textsc{vi} student only if he knows the text is the heading of the slide. Similarly, detecting figures, equations, and table regions is also essential for understanding the slides. Our goal is to develop a system that meaningfully describes the classroom slides with a markup text, as shown in Fig.~\ref{fig:teaser}.

\vspace{-1cm}
\begin{figure}[!ht]
\includegraphics[width=12cm,trim={0.4cm
18.3cm
7.6cm
3cm
},clip]{./images/teserImg.pdf}
\caption{Illustrate qualitative comparison between outputs obtained by the proposed {\sc csns} and the existing assistive systems --- Automatic Alt-Text ({\sc aat})~\cite{wu2017automatic} and Tesseract {\sc ocr}~\cite{smith2007overview}. The proposed system generates a markup text, where each logical content is tagged with its appropriate logical function. The audio file generated using this markup text gives a clear picture of the classroom slide's content to the blind or \textsc{vi} students. \label{fig:teaser}}
\end{figure}

The classroom slides have a complex design. The exact text could appear in various labels such as ``heading'', ``paragraph'', and ``list''. Hence, only the visual features learned by the existing semantic segmentation networks (e.g., {\sc fcn}~\cite{long2015fully}, Deeplab~\cite{chen2017deeplab}, and {\sc pspn}et~\cite{zhao2017pyramid}) are not sufficient to distinguish various logical regions. Limited works~\cite{haurilet2019spase,haurilet2019wise} on classroom slide segmentation utilize $(x,\ y)$ coordinate values to enhance the quality of segmentation results. However, all these works~\cite{haurilet2019spase,haurilet2019wise} do not utilize location information efficiently for segmentation. In contrast, Choi \emph{et al.}~\cite{Choi_2020_CVPR} utilize the position encoding~\cite{vaswani2017attention} to impose the height information in {\sc han}et for semantic segmentation of urban-scene images. In the case of slide images, the height and width information of each logical region are equally crucial for segmenting regions accurately. In this work, we propose a Classroom Slide Segmentation Network, called {\sc cssn}et to segment slide images accurately. The proposed network consists of (i) Attention Module, which utilizes the location encoding of the logical region (height and width of a region), and (ii) Atrous Spatial Pyramid Pooling ({\sc aspp}) module, which extracts multi-scale contextual features. The experiments on publicly available benchmark datasets WiSe~\cite{haurilet2019wise} and SpaSe~\cite{haurilet2019spase} establish the effectiveness of the proposed {\sc lean}et over state-of-the-art techniques --- Haurilet \emph{et al.}~\cite{haurilet2019wise}, Haurilet \emph{et al.}~\cite{haurilet2019spase}, {\sc han}et~\cite{Choi_2020_CVPR}, {\sc dan}et~\cite{fu2019dual}, and {\sc dran}et~\cite{fu2020scene}.

The proposed Classroom Slide Narration System ({\sc csns}) creates audio content for helping {\sc vi} students to read and understand the slide’s content. The audio created corresponds to the segmented region's content in reading order. We use four existing recognizers ---{\sc ocr}~\cite{smith2007overview}, equation descriptor~\cite{mondal2019textual}, table recognizer~\cite{sachin2020table}, and figure classifier~\cite{jobin2019docfigure} to recognize content from the segmented regions. We evaluate the effectiveness of the developed system on the WiSe dataset~\cite{haurilet2019wise} and compare the user experiences with the existing assistive systems \textsc{aat}~\cite{wu2017automatic} and Tesseract {\sc ocr}~\cite{smith2007overview}. The proposed system attains a better user experience than the existing assistive systems.

The contributions of this work are as follows:
\begin{itemize}
\item Proposes a {\sc cssn}et to efficiently segment the classroom slide images by utilizing the location encoding.
\item Presents a {\sc csns} as an use case of slide segmentation task to narrate slide's content in reading order. It helps the {\sc vi} students to read and understand the slide's content.
\item Perform experiments on publicly available benchmark datasets WiSe and SPaSe establish the effectiveness of the proposed {\sc cssn}et and {\sc csns} over existing slide segmentation and assistive techniques.
\end{itemize}

\section{Related Work}

\paragraph{{\bf Visual Assistance System:}}

According to the World Health Organization (\textsc{who}), at least $2.2$ billion people have a vision impairment or blindness worldwide. The computer vision community has undertaken significant efforts to deploy automated assistive systems for such people. Some of the unique assistive systems based on computer vision are --- (i) v{\sc oic}e\footnote{https://www.seeingwithsound.com}
~\cite{auvray2007learning} vision technology is a sensory substitution system for blind people which gives visual experience through live camera views by image-to-sound renderings, (ii) Electro-Neural Vision System ({\sc envs})~\cite{meers2004vision} provides a virtual perception of the three-dimensional profile and color of the surroundings through
pulses, (iii) a wearable travel aid for environment perception and navigation~\cite{bai2019wearable}, (iv) a smartphone-based image captioning for the visually and hearing impaired people~\cite{makav2019smartphone}. (v) Currency Recognition using Mobile Phones~\cite{singh2014currency} (vi) Facebook’s Automatic Alt-Text (\textsc{aat})~\cite{wu2017automatic} system is one of the most widely used assistive features, which automatically recognizes a set of $97$ objects present in Facebook photos, (vii) FingerReader~\cite{shilkrot2015fingerreader} is a wearable text reading device to assist blind people and dyslexic readers. All these existing assistive systems fail to properly narrate the classroom slides for a {\sc vi} student due to various logical regions present in the slides.

\paragraph{{\bf Document Image Segmentation:}}

Textual regions and graphics regions dominate document images. Graphics include natural images, various types of graphs, plots, and tables. Historically, the document image segmentation task is considered as dividing a document into various regions. Existing techniques~\cite{amin2001page,ha1995recursive} are either bottom-up or top-down. In the case of bottom-up~\cite{amin2001page}, individual words are first detected based on hand-crafted features. Therefore, detected words are grouped to form lines and paragraphs. Segmenting textual regions (like paragraphs) into lines and words helps to find semantic regions present in the document in the case of top-down~\cite{ha1995recursive}. These methods mostly use feature engineering and do not necessitate any training data while applying clustering or thresholding.

With the recent advancement in deep convolutional neural networks ({\sc cnn}s), several neural-based models~\cite{chen2015page,vo2016dense,renton2017handwritten,wick2018fully,yang2017learning} have been proposed to segment document images. Chen {\em et al.}~\cite{chen2015page} considered a convolutional auto-encoder to learn features from cropped document image patches. Thereafter, those features are used to train an {\sc svm} classifier~\cite{cortes1995support} to segment document patches.
Vo {\em et al.}~\cite{vo2016dense} and Renton {\em et al.}~\cite{renton2017handwritten} proposed lines detection algorithm in the handwritten document using {\sc fcn}. In~\cite{yang2017learning}, Yang {\em et al.} proposed an end-to-end, multi-modal, fully convolutional network for extracting semantic structures from document images. Inspired by document image segmentation and natural image segmentation using deep {\sc cnn}s, Haurilet \emph{et al.}~\cite{haurilet2019wise,haurilet2019spase} explore DeepLab~\cite{chen2017deeplab} architecture on classroom slide segmentation task. More recently, \textsc{dan}et~\cite{fu2019dual}, \textsc{han}et~\cite{Choi_2020_CVPR}, and \textsc{dran}et~\cite{fu2020scene} capture long-range dependency to improve performance by extending the self-attention mechanism. The proposed {\sc lean}et is strongly influenced by \textsc{dan}et~\cite{fu2019dual}, \textsc{han}et~\cite{Choi_2020_CVPR}, and \textsc{dran}et~\cite{fu2020scene}.

\section{Classroom Slide Narration System} \label{system}

We develop a {\sc csns} as a use case of slide segmentation to help {\sc vi} students to read and understand slide's content without human assistance. The proposed {\sc csns} consists of three modules --- (i) \textbf{Slide Segmentation Module:} to identify logical regions present in the slide, (ii) \textbf{Information Extraction Module:} to extract meaningful information from each segmented region, and (iii) \textbf{Audio Creation Module:} to create an audio narration of extracted information in proper order. We discuss each of these modules in detail.

\begin{figure*}[!ht]
\includegraphics[width=12cm,trim={
0.25cm
 1cm
 0.51cm
 1cm
 },clip]{./images/SegmentationArchitecture.pdf}
\caption{Illustrate the architecture of the proposed {\sc cssn}et for classroom slide segmentation. The network consists of three modules --- (i) attention module (upper dotted region), (ii) multi-scale feature extraction module (lower  region), (iii) feature concatenation module. Here, $\oplus$ and $\otimes$ represent the element-wise summation and multiplication of features, respectively.\label{ProposedArch}}
\end{figure*}

\subsection{Slide Segmentation Module}\label{slide_segmentation}

{\sc cssn}et is the segmentation network utilized in this work to segment classroom slide images accurately. The {\sc cssn}et aims to learn more discriminating features for semantic associations among pixels through the location encoding attention mechanism. We propose a location encoded attention module that utilizes the location encoding of logical regions of slide image. To enhance the segmentation accuracy, we concatenate the attention module output feature with the multi-scale features through an Atrous Spatial Pyramid Pooling ({\sc aspp}) layer~\cite{chen2017deeplab} as shown in the Fig~\ref{ProposedArch}. Finally, a simple decoder network predicts the high-resolution, multi-label, dense segmentation output.

\subsubsection{Attention Module:}

The previous attempts~\cite{chen2017deeplab,zhao2017pyramid} suggest that the local features generated by traditional Fully Convolutional Networks ({\sc fcn}s) could not capture rich context, leading to erroneous object detection. While Choi \emph{et al.}~\cite{Choi_2020_CVPR} discussed that urban-scene images have common structural priors depending on a spatial position. The authors also discussed height-wise contextual information to estimate channels weights during pixel-level classification for urban-scene segmentation. Fu \emph{et al.}~\cite{fu2019dual} also discussed the self-attention mechanism to capture the spatial dependencies between any two positions of the feature maps. Inspired by the works~\cite{Choi_2020_CVPR,fu2019dual}, we introduce an attention module that utilizes location encoding to enhance the local feature's contextual information and relative position. The attention module selectively encodes a broader range of contextual and location information into each pixel according to semantic-correlated relations, thus enhancing the discrimination power for dense predictions.

As illustrated in Fig.~\ref{ProposedArch} (inside the dotted box),  we use a convolution layer to obtain features of reduced dimension $\mathbf{A} \in$ $\mathbb{R}^{C \times H \times W}$. We feed $\mathbf{A}$ into a convolution layer to generate two new feature maps $\mathbf{B}$ and $\mathbf{C}$, respectively. $\{\mathbf{B}, \mathbf{C}\} \in \mathbb{R}^{C \times H \times W}$. We add the location encoding $\mathbf{L}$ (refer Section~\ref{loc_enc}) to the $\mathbf{B}$ and obtain $\mathbf{L}+\mathbf{B}$. Then we reshape it to $\mathbb{R}^{C \times N}$, where $N=H \times W$ is the number of pixels. We perform a matrix multiplication between the transpose of $\mathbf{C}$ and $\mathbf{B}+\mathbf{L}$, and apply a soft-max layer to obtain spatial attention map $\mathbf{S} \in \mathbb{R}^{N \times N}$:
\begin{equation}
s_{j i}=\frac{\exp \left((B_{i}+L_{i}) \cdot C_{j}\right)}{\sum_{i=1}^{N} \exp \left((B_{i}+L_i) \cdot C_{j}\right)},
\end{equation}
where $s_{j i}$ measures the impact of $i^{th}$ position on $j^{t h}$ position. A more similar feature representation of two positions contributes greater correlation between them. Meanwhile, we feed feature $\mathbf{A}$ into a convolution layer to generate a new feature map $\mathbf{D} \in \mathbb{R}^{C \times H \times W}$ and reshape it to $\mathbb{R}^{C \times N}$. Then we perform a matrix multiplication between $\mathbf{D}$ and transpose of $\mathbf{S}$ and reshape the result to $\mathbb{R}^{C \times H \times W}$. Finally, we multiply it by a scale parameter $\alpha$ and perform an element-wise sum with the features $\mathbf{A}$ to obtain the final output $\mathbf{E} \in \mathbb{R}^{C \times H \times W}$ using:
\begin{equation}
E_{j}=\alpha \sum_{i=1}^{N}\left(s_{j i} D_{i}\right)+(1-\alpha)\times A_{j},
\end{equation}
where $\alpha$ is initialized as 0 and learned gradually.

\paragraph{\textbf{Location Encoding}}\label{loc_enc}

We follow strategy proposed by~\cite{vaswani2017attention,Choi_2020_CVPR} for location encoding and use sinusoidal position encoding~\cite{vaswani2017attention} as $\mathbf{L}$ matrix. In our approach, we give importance to both height and width location encoding. The dimension of the positional encoding is the same as the channel dimension $C$ of the intermediate feature map that combines with $\mathbf{L}$. The location encoding is defined as
$$
\begin{aligned}
P E_{v}(v, 2 i) &=\sin \left(v / 100^{2 i / C}\right);
\ \ P E_{v}(v, 2 i+1)&=\cos \left(v / 100^{2 i / C}\right), \\
P E_{h}(h, 2 i) &=\sin \left(h / 100^{2 i / C}\right);
\ \ P E_{h}(h, 2 i+1)&=\cos \left(h / 100^{2 i / C}\right),
\end{aligned}
$$
where $v$ and $h$ denote the vertical and horizontal location index in the image ranging from $0$ to ${H/r}-1$ and $0$ to ${W/r}-1$, and $i$ is the channel location index ranging from $0$ to ${C}-1$. The $r$ is called the reduction ratio, which determines the frequency of $\sin$ or $\cos$ wave. Here, we apply $\sin$ and $\cos$ location encoding for intermediate layers of the features. The new representation $\mathbf{L}$ incorporating location encoding is formulated as
$$
\mathbf{L}= \beta P E_{h} \oplus (1-\beta)P E_{v},
$$
where $\oplus$ is an element-wise sum, $\beta $ is initialized with $0$ and learned gradually. Adding both the horizontal and vertical position encoding enables the attention module to learn the location information in both directions. The $\beta$ value is learned based on the importance of width and height locations. Fig.~\ref{locationEnc} shows the location encoding value of a layer in $\mathbf{L}$ matrix. Finally, we add random jitters to the location encoding $PE_v$ and $PE_h$ for the better generalization of location encoding. The height and width of the location encoding matrix $\mathbf{L}$ are adjusted to the dimension of the feature vector.

\begin{figure}[!htp]
\includegraphics[width=12cm,trim={
 0cm
 9cm
 0cm
 2cm
 },clip]{./images/locationEncodings.pdf}
\caption{Presents visualization of various location encoding with frequency $2$, $1$, and $0.5$ with $\beta$ values $0$, $0.5$, and $1$.} \label{locationEnc}
\end{figure}

\subsection{Information Extraction Module}\label{ie_block}

We coarsely group the classroom slide image into four logical regions --- text, figure, equation, and table and use existing recognizers to recognize the content of the regions.

\paragraph{\textbf{OCR:}}
We use well-known Optical Character Recognition ({\sc ocr}), Tesseract~\cite{smith2007overview} to recognize the content of the text regions present in slide images.

\paragraph{\textbf{Figure Classification:}} Figures mainly different types of plots, sketches, block diagrams,maps, and others. appear in slides. We use DocFigure~\cite{jobin2019docfigure} to find the category of each figure in slide images.

\paragraph{\textbf{Equation Description:}}
The equation frequently occurs in the classroom slides. We follow {\sc med} model~\cite{mondal2019textual} for describing equation in natural language to interpret the logical meaning of equation for {\sc vi} students.

\paragraph{\textbf{Table Structure Recognition:}}
Recognition of the table present in the slide includes detecting cells, finding an association between cells, and recognizing cell's content. We use TabStruct-Net~\cite{sachin2020table} to recognize the physical structure of the table and then Tesseract~\cite{smith2007overview} to recognize content.

The information from the four modules is combined and saved as a {\sc json} file. These processes are done on the server and then sent to the {\sc json} file to the mobile app, and the {\sc vi} student understands the slide's content by interacting with the app.

\subsection{Audio Creation Module}

Based on a study that empirically investigates the difficulties experienced by {\sc vi} internet users~\cite{murphy2008empirical}, a mobile application is developed for {\sc vi} students to read and understand the classroom slide's content. For this, we place a camera in the classroom connected to the server for capturing slide images. The {\sc vi} students use this mobile app to connect to the server through a WiFi connection. Using the mobile application (\textbf{CAPTURE} button), the student can request the server to get information on the current slide. The server captures the current slide image through the camera, extracts the information as {\sc json} file, and sends it back to the mobile application. Fig.~\ref{fig_mobile_application}(a) shows a screenshot of the developed mobile app with one example. The app has two modes --- (i) interactive and (ii) non-interactive.  In the interactive mode, the user can navigate various logical regions of slides by themselves. The app shows the captured slide image with all extracted information. When the user touches one particular area, it automatically plays the audio sound using android {\sc tts} library of that region's content along with the category label. In the non-interactive mode, the app automatically plays the audio corresponding to the slide's content. In this case, the screen contains the \textbf{READ ALL} button; when the user touches this particular area, the app automatically plays the audio sound of the full slide's content. These features are demonstrated in the demo video\footnote{\url{https://youtu.be/PnPYrA8ykF0}}

\section{Experiments}

\subsection{Experimental Setup}

We use the residual network (e.g., ResNet-101) pre-trained on ImageNet~\cite{krizhevsky2012imagenet} as the backbone network for all the experiments. We replace a single $7\times 7$ convolution with three of $3\times 3$ convolutions in the first layer of ResNet-101. We use We also adopt an auxiliary cross-entropy loss in the intermediate feature map. Since for the multi-label segmentation task we need to predict multiple classes per pixel, we replace the softmax output layer in the previous deep models with a sigmoid activation function and train these models using binary cross entropy loss. We train the network using \textsc{sgd} optimizer with an initial learning rate of $2e-2$ and momentum of $0.9$. We set the weight decays of $5e-4$ and $1e-4$ for backbone networks and segmentation networks, respectively. The learning rate schedule follows the polynomial learning rate policy~\cite{liu2015parsenet}. The initial learning rate is multiplied by $(1-\frac{iteration}{max\ iteration})^{0.9}$. We use typical data augmentations such as random scaling in the range of $[0.5,2]$, Gaussian blur, color jittering, and random cropping to avoid over-fitting. We set crop size and batch size of $540\times 720$ and $2$, respectively. We use mean Intersection over Union (mIoU)~\cite{chen2017deeplab} and Pixel Accuracy ({\sc pa})~\cite{haurilet2019wise,haurilet2019spase} for evaluation purpose.

\subsection{Dataset}

\paragraph{\textbf{WiSe:}}

The WiSe~\cite{haurilet2019wise} dataset consists of $1300$ slide images. It is divided into a training set, validation set, and test set consisting of $900$, $100$, and $300$ slide images annotated with $25$ overlapping region categories.

\paragraph{\textbf{SPaSe:}}

The SpaSe~\cite{haurilet2019spase} dataset contains dense, pixel-wise annotations of $25$ classes for $2000$ slide images~\cite{haurilet2019spase}. It is split into training, validation, and test sets containing $1400$, $100$, and $500$ images, respectively.

\subsection{Ablation Study}

In the ablation study, we use the ResNet-101 backbone with an output stride of $16$ and evaluate the WiSe dataset. First, we examine the best feature vector for adding location encoding in the \textsc{lean}et architecture. Based on Fig.~\ref{ProposedArch} of {\sc cssn}et architecture, the location encoding can be added to any of the feature vectors including, $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, and $\mathbf{D}$. We empirically find the best feature for adding location encoding. In addition to it, we also conduct experiments by changing the frequency of the $\sin$ wave encoding in the $\mathbf{L}$ matrix. Finally, we also compare with CoordConv~\cite{liu2018intriguing} approach.

\begin{table}[htb]
\centering
\begin{tabular}{lr}
\begin{minipage}{.6\linewidth}
\begin{tabular}{@{}p{0.7cm}p{0.7cm}p{0.7cm}p{0.7cm}|c|c@{}}
\toprule
\multicolumn{4}{c|}{Features} &  \multirow{2}{*}{Frequency} & \multirow{2}{*}{mIoU} \\
$\mathbf{A}$ & $\mathbf{B}$ & $\mathbf{C}$ & $\mathbf{D}$ &   &  \\ \hline
&   &   &  &   1 & 51.07 \\
\checkmark   &   &   &  &   1 & 51.44 \\
&  \checkmark  &   &   &   1 & 52.26 \\
&   &  \checkmark  &   &   1 & 52.12 \\
&   &   &  \checkmark  &   1 & 50.04 \\ \hline
&  \checkmark  &   &   &   2 & 51.14 \\
&  \checkmark  &   &     & 0.5 & 50.32 \\ \hline
\multicolumn{5}{c|}{ $\mathbf{A}$ + CoordConv~\cite{liu2018intriguing}(width+ height)}   & 51.12
\\ \bottomrule
\multicolumn{6}{c}{(a)}
\end{tabular}
\end{minipage} &
\begin{minipage}{.3\linewidth}
\begin{tabular}{cc}
\toprule
$\beta$ & mIoU \\ \hline
0&51.73\\
1&50.25 \\
0.5&51.83 \\
learning & 52.26
\\ \bottomrule
\multicolumn{2}{c}{(b)}
\end{tabular}
\end{minipage}
\end{tabular}
\caption{Shows ablation studies and the impact of hyper-parameters on a validation set of the WiSe dataset. Table (a) shows the result of segmentation by adding location encoding to the features $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, and $\mathbf{D}$. Table (b) shows the effect of manual $\beta$ values and as a learnable parameter.\label{ablationStudy}}
\end{table}

Table~\ref{ablationStudy} shows the ablation studies conducted in the proposed network architecture by changing the various hyper-parameters. The Table~\ref{ablationStudy}(a) shows that the location encoding on the feature vector outperformed the traditional CoordConv~\cite{liu2018intriguing} approach. The experiment also shows that adding location encoding to the intermediate feature $\mathbf{B}$ and $\mathbf{C}$ is comparatively better than adding it to $\mathbf{A}$ and $\mathbf{D}$. The best frequency of $\sin$ wave in the $\mathbf{L}$ is found as 1. While Table~\ref{ablationStudy} (b) shows the learnable $\beta$ value outperforms the manually assigned $\beta$ value.

\subsection{Comparison with State-of-the-Art Techniques}

Comparison results of the proposed network with state-of-the-art techniques are presented in Table~\ref{result1}. From the table, we observe that none of the attention mechanisms (e.g., {\sc han}et~\cite{Choi_2020_CVPR}, {\sc dan}et~\cite{fu2019dual}, and {\sc dran}et~\cite{fu2020scene}) is better than Haurilet \emph{et al.}~\cite{haurilet2019spase} on SPaSe dataset. While in the case of WiSe dataset, all attention architectures {\sc han}et~\cite{Choi_2020_CVPR}, {\sc dan}et~\cite{fu2019dual}, and {\sc dran}et~\cite{fu2020scene} are better than Haurilet \emph{et al.}~\cite{haurilet2019wise}. {\sc dran}et~\cite{fu2020scene} obtains the best performance ($35.77$\% mean IoU and $78.64$\% PA) among all existing techniques on SPaSe dataset. While {\sc dan}et~\cite{fu2019dual} obtains the best performance ($44.85$\% mean IoU and $88.80$\% PA) among all existing techniques on WiSe dataset. The proposed network obtains the best results ($46.74$\% mean IoU and $89.80$\% PA) on Wise and ($36.17$\% mean IoU and $79.11$\% PA) on SPaSe dataset compared to the existing techniques.

\begin{table}[ht!]
\begin{center}
\begin{tabular}{||l||c|c||c|c||} \hline
\textbf{Methods} &\multicolumn{4}{|c||}{\textbf{Datasets}} \\ \cline{2-5}
&\multicolumn{2}{|c||}{\textbf{SPaSe}} &\multicolumn{2}{|c||}{\textbf{WiSe}} \\ \cline{2-5}
&\textbf{mean IoU} &\textbf{PA}&\textbf{mean IoU} &\textbf{PA} \\ \hline \hline
Haurilet \emph{et al.}~\cite{haurilet2019spase} &35.80  &77.40 &- &- \\
Haurilet \emph{et al.}~\cite{haurilet2019wise}  &-  &-  &37.20 &88.50 \\
HANet~\cite{Choi_2020_CVPR}                    &32.37  &76.54  &39.35 &88.72 \\
DANet~\cite{fu2019dual}                        &33.12  &77.39  &44.85 &88.80 \\
RANet~\cite{fu2020scene}                      &35.77  &78.64  &43.31 &88.77 \\
Ours                                          &36.17   &79.11         &46.74 &89.80 \\ \hline \hline
\end{tabular}
\end{center}
\caption{illustrates the comparison of the proposed method with state-of-the-art techniques on benchmark datasets WiSe and SPaSe datasets. \label{result1}}
\end{table}

\begin{figure}
\centering
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=\linewidth, height = 0.4\linewidth]{images/userInterface.jpg
\subcaption{}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.5\textwidth}
\centering
\includegraphics[width=\textwidth, trim={0cm
0cm
0cm
0cm
},clip]{./images/userExp.pdf
\subcaption{}
\end{subfigure}
\caption{\textbf{(a):} presents user interface of the developed mobile application. The mobile screen displays the camera-captured slide image. Various colors highlight the slide's logical regions for better visualization. The {\sc Read All} button plays generated audio sound corresponding to the slide's content. \textbf{(b):} presents user rating on performance of various approaches. \textbf{I} and \textbf{N} indicate \textbf{interactive} and \textbf{non-interactive} modes, respectively.\label{fig_mobile_application}}
\end{figure}
\vspace{-0.08\textwidth}
\subsection{Narration System Evaluation}

We establish the effectiveness of the developed slide narration system by comparing the performance with Facebook’s Automatic Alt-Text (\textsc{aat})~\cite{wu2017automatic} and Tesseract \textsc{ocr}~\cite{smith2007overview}. We use $300$ test images of the WiSe dataset for evaluating the system. $30$ volunteers who can read and write English are selected. Each volunteer takes $300$ slide images and evaluates corresponding audio outputs generated by various techniques. The volunteer gives a grade (value in the range $[0,\ 10]$) to the generated audio files by various techniques by comparing them with the original slide content. We collect all these grades and plot them as a bar chart (shown in Fig.~\ref{fig_mobile_application}(b)). The user gives the best grade for the audio file corresponding to the ground truth segmentation of the slide. However, the proposed {\sc csns} also performs comparatively better than the existing techniques.

\section{Summary}

This paper presents a classroom slide segmentation network to segment classroom slide images more accurately. The attention module in the proposed network enriches the learned contextual features by utilizing the location encoding of the logical region. We develop a Classroom Slide Narration System as an application of classroom slide segmentation for helping {\sc vi} students to read and understand the slide's content. Experiments on benchmark datasets WiSe and SpaSe conclude that the proposed {\sc lean}et achieves the best slide segmentation results as compared to the state-of-the-art methods. We also demonstrate the effectiveness of our developed Classroom Slide Narration System over existing assistive tools in terms of user experiences.

\bibliographystyle{splncs04}

\section{Introduction} \label{sec:intro}

Slide presentations play an essential role in classroom teaching, technical lectures, massive open online courses ({\sc mooc}), and many formal meetings.Presentations are an efficient and effective tool for the students to understand the subject better. However, the visually impaired ({\sc vi}) students do not benefit from this type of delivery due to their limitation in reading the slides. Human assistants could help {\sc vi} students to access and interpret the slide's content. However, at every time, it is impossible and impractical for the {\sc vi} students to have a human reader; because of the cost and the limited availability of the trained personnel. From this perspective, this would be an important issue in modern classrooms.
\par
The automatic extraction of logical regions from these slide images is the initial step for understanding the slide's content. We define automatic extraction of logical parts of slide images as segmentation of semantically similar regions, closely related to natural image semantic segmentation in computer vision~\cite{long2015fully,chen2017deeplab,zhao2017pyramid}. The goal is to assign logical labels like, title, paragraph, list, equation, table, figure to each pixel in the slide images. Due to the large variability in the theme (i.e., the layout of the slide), style, and content of the slide shown in Figure~\ref{fig:real_slide}, the extraction of meaningful regions is a challenging task.

The text extraction from these regions is useful to the \textsc{vi} student only if the text is tagged with its logical function. The left image of Figure~\ref{fig:teaser}, the plain text ``\textit{Discovering Attention Patterns}'' is more meaningful to the \textsc{vi} student only if he knows the text is the heading of the slide. Similarly, the detection of figures, equations, and table regions are also essential for understanding the slides. Our goal is to develop a system that meaningfully describes the classroom slides with a markup text, as shown in Figure~\ref{fig:teaser}.

\begin{figure}[!ht]
\includegraphics[width=12cm,trim={0.5cm
 18.3cm
 .5cm
 3cm
 },clip]{./images/teserImg.pdf}
\caption{Illustrate qualitative comparison between outputs obtained by the proposed {\sc csns} and the existing assistive systems such as Automatic Alt-Text ({\sc aat})~\cite{wu2017automatic} and Tesseract {\sc ocr}~\cite{smith2007overview}. The proposed system generates a markup text, where each logical content is tagged with its appropriate logical function. The audio file generated using this markup text gives a clear picture of the classroom slide's content to the blind or \textsc{vi} students.  \label{fig:teaser}}
\end{figure}

The classroom slides have a complex design, as shown in Figure~\ref{fig:real_slide}. The same text could appear in various labels such as ``heading'', ``paragraph'', and ``list''. Hence, only the visual features learned by the existing semantic segmentation networks (e.g., {\sc fcn}~\cite{long2015fully}, Deeplab~\cite{chen2017deeplab}, {\sc pspn}et~\cite{zhao2017pyramid}, etc.) are not sufficient to distinguish various logical regions. Limited works~\cite{haurilet2019spase,haurilet2019wise} on classroom slide segmentation utilize $(x,\ y)$ coordinate values to enhance the quality of segmentation results. However, all these works~\cite{haurilet2019spase,haurilet2019wise} do not utilize location information efficiently for segmentation. In contrast, Choi \emph{et al.}~\cite{Choi_2020_CVPR} utilize the position encoding~\cite{vaswani2017attention} to impose the height information in {\sc han}et for semantic segmentation of urban-scene images. In the case of slide images, height and width information of each logical region are equally important for segmenting regions accurately. In this work, we propose a Classroom Slide Segmentation Network, called Location Encoding Attention Network ({\sc lean}et) to segment slide images accurately. The proposed network consists of (i) Attention Module which utilizes the location encoding of the logical region (height and width of a region) and (ii) Atrous Spatial Pyramid Pooling ({\sc aspp}) module which extracts multi-scale contextual features. The experiments on publicly available benchmark datasets WiSe~\cite{haurilet2019wise} and SpaSe~\cite{haurilet2019spase} establish the effectiveness of the proposed {\sc lean}et over state-of-the-art techniques --- Haurilet \emph{et al.}~\cite{haurilet2019wise}, Haurilet \emph{et al.}~\cite{haurilet2019spase}, {\sc han}et~\cite{Choi_2020_CVPR}, {\sc dan}et~\cite{fu2019dual}, and {\sc dran}et~\cite{fu2020scene}.

We propose a Classroom Slide Narration System ({\sc csns}) for the proposed slide segmentation for helping {\sc vi} students for reading and understanding the slide’s content. The proposed {\sc csns} creates audio corresponding to the segmented region's content in reading order. We use four existing recognizers ---{\sc ocr}~\cite{smith2007overview}, equation descriptor~\cite{mondal2019textual}, table recognizer~\cite{sachin2020table}, and figure classifier~\cite{jobin2019docfigure} to recognize content corresponds to segmented region. The proposed {\sc csns} system  has two modes to present the slide’s content in audio format --- (i) \textbf{Interactive Mode:} the user can select a specific region on the slide for reading the slide's content, and (ii) \textbf{Non-interactive Mode:} the system narrates all extracted information in reading order. We evaluate the effectiveness of the developed system on WiSe dataset~\cite{haurilet2019wise} and compare the user experiences with the existing assistive systems \textsc{aat}~\cite{wu2017automatic} and Tesseract {\sc ocr}~\cite{smith2007overview}. The proposed system attains a better user experience than the existing assistive systems.

The contributions of this work are as follows:
\begin{itemize}
\item We proposed a {\sc lea}net segmentation architecture to efficiently segment the classroom slide images with utilizing the location encoding.

\item We present a {\sc csns} as use of slide segmentation to narrate slide's content in reading order. It helps the {\sc vi} students to read and understand the slide's content.

\item Experiments on publicly available benchmark datasets WiSe and SPaSe establish the effectiveness of the proposed {\sc lean}et and {\sc csns} over existing slide segmentation and assistive techniques.

\end{itemize}

\section{Related Work}

\paragraph{{\bf Visual Assistance System:}}

According to the World Health Organization (\textsc{who}), at least $2.2$ billion people have a vision impairment or blindness worldwide. The computer vision community has undertaken significant efforts to deploy automated assistive systems for such people. Some of the unique assistive systems based on computer vision are --- (i) v{\sc oic}e\footnote{https://www.seeingwithsound.com}
~\cite{auvray2007learning} vision technology is a sensory substitution system for blind people which gives visual experience through live camera views by image-to-sound renderings, (ii) Electro-Neural Vision System ({\sc envs})~\cite{meers2004vision} provides a virtual perception of the three-dimensional profile and color of the surroundings through electrical pulses, (iii) a wearable travel aid for environment perception and navigation~\cite{bai2019wearable}, (iv) a smartphone-based image captioning for visually and hearing impaired person~\cite{makav2019smartphone}. (v) Currency Recognition using Mobile Phones~\cite{singh2014currency} (v) Facebook’s Automatic Alt-Text (\textsc{aat})~\cite{wu2017automatic} system is one of the most widely used assistive features, which automatically recognizes a set of $97$ objects present in Facebook photos, (vi) FingerReader~\cite{shilkrot2015fingerreader} is a wearable text reading device to assist blind people and dyslexic readers. All these existing assistive systems fail to properly narrate the classroom slides for a {\sc vi} student due to various logical regions present in the slides.

\paragraph{{\bf Document Image Segmentation:}}
Document images are dominated by textual regions and graphics regions. Graphics include natural images, various types of graphs, plots, and tables. Historically, the document image segmentation task is considered as dividing a document into various regions. Existing techniques~\cite{amin2001page,ha1995recursive} are either bottom-up or top-down. In the case of bottom-up~\cite{amin2001page}, individual words are first detected based on hand-crafted features. Therefore, detected words are grouped to form lines and paragraphs. Segmenting textual regions (like paragraphs) into lines and words helps to find semantic regions present in the document in the case of top-down~\cite{ha1995recursive}. These methods mostly use feature engineering and do not necessitate any training data while applying clustering or thresholding.

With the recent advancement in deep convolutional neural networks ({\sc cnn}s), several neural-based models~\cite{chen2015page,vo2016dense,renton2017handwritten,wick2018fully,yang2017learning} have been proposed to segment document images. Chen {\em et al.}~\cite{chen2015page} considered a convolutional auto-encoder to learn features from cropped document image patches. Thereafter, those features are used to train an {\sc svm} classifier~\cite{cortes1995support} to segment document patches.
Vo {\em et al.}~\cite{vo2016dense} and Renton {\em et al.}~\cite{renton2017handwritten} proposed lines detection algorithm in the handwritten document using {\sc fcn}. In~\cite{yang2017learning}, Yang {\em et al.} proposed an end-to-end, multi-modal, fully convolutional network for extracting semantic structures from document images. Inspired by document image segmentation and natural image segmentation using deep {\sc cnn}s, Haurilet \emph{et al.}~\cite{haurilet2019wise,haurilet2019spase} explore DeepLab~\cite{chen2017deeplab} architecture on classroom slide segmentation task. More recently, \textsc{dan}et~\cite{fu2019dual}, \textsc{han}et~\cite{Choi_2020_CVPR}, and \textsc{dran}et~\cite{fu2020scene} capture long-range dependency to improve performance by extending self-attention mechanism. Our proposed {\sc lean}et is strongly influenced by \textsc{dan}et~\cite{fu2019dual}, \textsc{han}et~\cite{Choi_2020_CVPR}, and \textsc{dran}et~\cite{fu2020scene}.

\section{Classroom Slide Narration System} \label{system}

Classroom Slide Narration System ({\sc csns}) is developed as a use case of slide segmentation to help {\sc vi} students to read and understand slide's content without human assistance. The proposed {\sc csns} consists of three modules --- (i) identification of logical regions present in the slide (i.e, slide segmentation), (ii) extraction of meaningful information from each segmented region, and (iii) create an audio narration of extracted information to help {\sc vi} students read the slides in a proper order. We discuss each of these modules in detail.

\subsection{Slide Segmentation}\label{slide_segmentation}

We propose a Classroom Slide Segmentation Network, called {\sc lean}et, to segment classroom slide images accurately. The proposed network aims to learn more discriminating features for semantic associations through the location encoding attention mechanism. Fig.~\ref{ProposedArch} shows the proposed network  with its three modules --- (i) an attention module that utilizes the location encoding of logical regions of slide image, (ii) a multi-scale feature extraction module to extract multi-scale features through an Atrous Spatial Pyramid Pooling ({\sc aspp}) layer~\cite{chen2017deeplab}, and (iii) feature concatenation module which combines low-level and high-level features with the guidance of encoding location information. Finally, a simple decoder network predicts the high-resolution, multi-label, dense segmentation output. We follow the existing works~\cite{chen2017deeplab,zhao2017pyramid,fu2020scene} and employ a pre-trained residual network (e.g., ResNet-$101$) with dilated strategy~\cite{zhang2018context} as the backbone. Note that, we remove down-sampling operations and employ dilated convolutions in the last two ResNet blocks, thus enlarging the final feature map size to $1/8$ of the input image. It retains more detailed features without adding extra parameters. The features from the dilated residual network are fed into the \textsc{aspp} layer to extract multi-scale features. While in the attention module, we build a relationship between any two pixels with and without location encoding, and this results in an attention map scaled with location encoding. Low-level and high-level features are concatenated in the feature concatenation module to enrich the object representation. Finally, a decoder takes enrich features and produces dense segmentation output.

\begin{figure*}[!htp]
\includegraphics[width=12cm,trim={
0.25cm
 1cm
 0.51cm
 1cm
 },clip]{./images/SegmentationArchitecture.pdf}
\caption{Illustrate the architecture of the proposed {\sc lean}et for classroom slide segmentation. The network consists of three modules (i) attention module (upper dotted region), (ii) multi-scale feature extraction module (Atrous Spatial Pyramid Pooling layer, lower dotted region), (iii) feature concatenation module. Here, $\oplus$ and $\otimes$ represent the element-wise sum and multiplication of features, respectively.\label{ProposedArch}}
\end{figure*}

\subsubsection{Attention Module}

The previous attempts~\cite{chen2017deeplab,zhao2017pyramid} suggest that the local features generated by traditional fully Convolutional Networks ({\sc fcn}s) could not capture rich context, leading to misclassification of objects and stuff. While Choi \emph{et al.}~\cite{Choi_2020_CVPR} discussed that urban-scene images have common structural priors depending on a spatial position. The authors also discuss using height-wise contextual information to estimate channels weights during pixel-level classification for urban-scene segmentation. Fu \emph{et al.}~\cite{fu2019dual} also discuss the self-attention mechanism to capture the spatial dependencies between any two positions of the feature maps. Inspired by these works~\cite{Choi_2020_CVPR,fu2019dual}, we introduce an attention module that utilizes location encoding to enhance the contextual information and relative position to the local feature. It enriches the contextual relationship between local features and location encoding. The attention module selectively encodes a broader range of contextual information and location into each pixel according to semantic-correlated relations, thus enhancing discrimination power for dense predictions.

As illustrated in Fig.~\ref{ProposedArch} (inside the dotted box),  we use a convolution layer to obtain features of dimension reduction $\mathbf{A} \in$ $\mathbb{R}^{C \times H \times W}$. We first feed it into a convolution layer to generate two new feature maps $\mathbf{B}$ and $\mathbf{C}$, respectively, where $\{\mathbf{B}, \mathbf{C}\} \in \mathbb{R}^{C \times H \times W}$. We add the location encoding $\mathbf{L}$ (refer sec~\ref{loc_enc}) to the $\mathbf{B}$. Then we reshape them to $\mathbb{R}^{C \times N}$, where $N=H \times W$ is the number of pixels. After that, we perform a matrix multiplication between the transpose of $\mathbf{C}$ and $\mathbf{B}+\mathbf{L}$, and apply a soft-max layer to calculate the spatial attention map $\mathbf{S} \in \mathbb{R}^{N \times N}$ :
\begin{equation}
s_{j i}=\frac{\exp \left((B_{i}+L_{i}) \cdot C_{j}\right)}{\sum_{i=1}^{N} \exp \left((B_{i}+L_i) \cdot C_{j}\right)},
\end{equation}
where $s_{j i}$ measures the $i^{th}$ position's impact on $j^{t h}$ position. The more similar feature representations of the two position contributes to greater correlation between them. Meanwhile, we feed feature $\mathbf{A}$ into a convolution layer to generate a new feature map $\mathbf{D} \in \mathbb{R}^{C \times H \times W}$ and reshape it to $\mathbb{R}^{C \times N}$. Then we perform a matrix multiplication between $\mathbf{D}$ and the transpose of $\mathbf{S}$ and reshape the result to $\mathbb{R}^{C \times H \times W}$. Finally, we multiply it by a scale parameter $\alpha$ and perform a element-wise sum operation with the features $\mathbf{A}$ to obtain the final output $\mathbf{E} \in \mathbb{R}^{C \times H \times W}$ as follows:
\begin{equation}
E_{j}=\alpha \sum_{i=1}^{N}\left(s_{j i} D_{i}\right)+(1-\alpha)\times A_{j},
\end{equation}
where $\alpha$ is initialized as 0 and leaned gradually.

\subsubsection{Location Encoding}
\label{loc_enc}

We use the sinusoidal position encoding~\cite{vaswani2017attention} as $\mathbf{L}$ matrix. For injecting positional encoding, we follow the strategy proposed in Transformer~\cite{vaswani2017attention,Choi_2020_CVPR}. In our approach, we give importance to both height and width location encoding. The dimension of the positional encoding is the same as the channel dimension $C$ of the intermediate feature map that combines with $\mathbf{L}$. The positional encoding is defined as

$$
\begin{aligned}
P E_{v}(v, 2 i) &=\sin \left(v / 100^{2 i / C}\right);
\ \ P E_{v}(v, 2 i+1)&=\cos \left(v / 100^{2 i / C}\right), \\
P E_{h}(h, 2 i) &=\sin \left(h / 100^{2 i / C}\right);
\ \ P E_{h}(h, 2 i+1)&=\cos \left(h / 100^{2 i / C}\right),
\end{aligned}
$$
where $v$ and $h$ denote the vertical and horizontal location index in the entire image ranging from $0$ to ${H/r}-1$ and $0$ to ${W/r}-1$, and $i$ is the channel location index ranging from $0$ to ${C}-1$. The $r$ is called the reduction ratio, which determines the frequency of $\sin$ or $\cos$ wave. Here, we apply $\sin$ and $\cos$ location encoding for intermediate layers of the features. The new representation $\mathbf{L}$ incorporating positional encoding is formulated as
$$
\mathbf{L}= \beta P E_{h} \oplus (1-\beta)P E_{v},
$$
where $\oplus$ is an element-wise sum and $\beta $ is initialized as $0$ and learned gradually. Adding both the horizontal and vertical position encoding enables the attention module to learn the location information in both direction. The $\beta$ value is learned based on the importance of width and height locations. Figure~\ref{locationEnc} shows the location encoding value of a layer in the $\mathbf{L}$ matrix. Finally, we add random jitters to the location encoding $PE_v$ and $PE_h$ for the better generalization of location encoding. The height and width of the location encoding matrix $\mathbf{L}$ adjusted to the dimension of the feature vector.

\begin{figure}[!htp]
\includegraphics[width=12cm,trim={
 0cm
 9cm
 0cm
 2cm
 },clip]{./images/locationEncodings.pdf}
\caption{Presents visualization of various location encoding with frequency $2$, $1$, and $0.5$ and the $\beta$ value $0$, $0.5$, and $1$.} \label{locationEnc}
\end{figure}

\subsection{Information Extraction}\label{ie_block}

We group the classroom slide image into four logical regions - text, figure, equation, and table and explore existing recognizers to recognize the content of these four regions.

\paragraph*{\textbf{OCR:}}

Optical Character Recognition ({\sc ocr}) module is used to recognize the content of text regions present in the slide image. The logical regions such as heading, paragraph, and enumeration are grouped into a text region. The curved or skewed text does not commonly appear in the slide images. The only complexities are text with various fonts and colors. Tesseract \textsc{ocr}~\cite{smith2007overview} recognizes text with various fonts and colors of the slide images.

\paragraph*{\textbf{Figure Classification Module:}} Most figures appearing on the slides are belongs to document figures such as different types of graphs, sketches, block diagram , maps, etc. The WiSe dataset has limited figure labels include Diagram, Legend, Logos, Maps, Realistic, and other images. To overcome this drawback, we use ResNet~\cite{he2016deep} trained on DocFigure~\cite{jobin2019docfigure} to recognize the category label of each figure present in slides. In addition to it, we further classify the realistic image using ResNet~\cite{he2016deep} trained on ImageNet~\cite{krizhevsky2012imagenet}.
\paragraph*{\textbf{Equation Description:}}
The equation frequently occurs in the classroom slides. Describing the mathematical equation to a {\sc vi} student is a challenging task due to the complex mathematical notations. Various approaches~\cite{mondal2019textual,medjkoune2017combining} exist to describe equations. We follow {\sc med} model~\cite{mondal2019textual} for describing equation in natural language to interpret the logical meaning of equation to {\sc vi} students.
\paragraph*{\textbf{Table Structure Recognition:}}
Recognition of the table present in the slide includes detecting cells, identifying cells' associations, and recognizing its content. We use TabStruct-Net~\cite{sachin2020table} to recognize the physical structure of the table and then Tesseract \textsc{ocr}~\cite{smith2007overview} to recognize content.
\begin{figure}[!htp]
\includegraphics[width=10cm, height=4cm,trim={
0.15cm
 0.25cm
 0.71cm
 0.15cm
 },clip]{./images/info_extract.pdf}
\caption{Illustrate the information extraction pipeline for the proposed {\sc csns} solution from the classroom slide image. The four modules extracts slide information and save as individual Json object. The x, y, w, and h represents the bounding rectangle of the region. \label{Fullpipeline}}
\end{figure}
\par
The classroom slide image information extraction pipeline is visualy described in the figure~\ref{Fullpipeline}. All the information from the four module are combined and save to a {\sc json} file. All these processes are done on the server and then send the {\sc json} file to a mobile app, and the {\sc vi} student understand the slide's content by interacting with the app.

\subsection{Application}

We develop a mobile application for {\sc vi} student to read and understand the classroom slide's content inspired by the study on an empirical investigation into difficulties experienced by {\sc vi} internet users~\cite{murphy2008empirical}. For this purpose, we place a camera in the classroom, which is connected to the server for capturing slide images. Each blind student has a mobile app, which is connected to the server through a WiFi connection. Using the mobile application (\textbf{CAPTURE} button), the student can request the server to get information on the current slide. The server captures the current slide image through the camera, extracts the information as {\sc json} file, and sends it back to the mobile application. Figure~\ref{fig_mobile_application} shows a screenshot of the developed mobile app with one example. The app has two modes --- (i) interactive and (ii) non-interactive.  In the interactive mode, the user can navigate various logical regions of slides by themselves.  The app shows the captured slide image with all extracted information. When the user touches one particular area, it automatically plays the audio sound using android {\sc tts} library of that region's content along with category label. In the non-interactive mode, the app automatically plays the audio corresponding to the slide's content. In this case, the screen contains the \textbf{READ ALL} button; when the user touches this particular area, the app automatically plays the audio sound of the full slide's content. These features are demonstrated in the demo video\footnote{\url{https://youtu.be/PnPYrA8ykF0}}

\section{Experiments}

In this section, we first describe the implementation details of proposed \textsc{lean}et architecture. Then, we demonstrate the effectiveness of the \textsc{lean}et architecture by extensive quantitative analysis including ablation studies. We evaluate the approach with two different classroom slide segmentation datasets including WiSe~\cite{haurilet2019wise} and SpaSe~\cite{haurilet2019spase}. For all the quantitative experiments, we measure the segmentation performance in terms of mean Intersection over Union (mIoU) and pixel accuracy matrix. Finally, we demonstrate the {\sc csns} and compare with Facebook’s Automatic Alt-Text (\textsc{aat})~\cite{wu2017automatic} and Tesseract \textsc{ocr}~\cite{smith2007overview}.

\subsection{Experimental Setup}

We use the residual network (e.g., ResNet-101) pre-trained on ImageNet~\cite{krizhevsky2012imagenet} as the backbone network for all the experiments. We replace a single $7\times 7$ convolution with three of $3\times 3$ convolutions in the first layer of ResNet-101. We also adopt an auxiliary cross-entropy loss in the intermediate feature map. Since for the multi-label segmentation task we need to predict multiple classes per pixel, we replace the softmax output layer in the previous deep models with a sigmoid activation function and train these models
using binary cross entropy loss. We train the network using \textsc{sgd} optimizer with an initial learning rate of $2e-2$ and momentum of $0.9$. We set the weight decays of $5e-4$ and $1e-4$ for backbone networks and segmentation networks, respectively. The learning rate schedule follows the polynomial learning rate policy~\cite{liu2015parsenet}. The initial learning rate is multiplied by$(1-\frac{iteration}{max\ iteration})^{0.9}$. We use typical data augmentations such as random scaling in the range of $[0.5,2]$, Gaussian blur, color jittering, and random cropping to avoid over-fitting. We set crop size and batch size of $540\times 720$ and $2$, respectively. We use mean Intersection over Union (mIoU)~\cite{chen2017deeplab} and Pixel Accuracy ({\sc pa})~\cite{haurilet2019wise,haurilet2019spase} for evaluation purpose.

\subsection{Dataset}

\paragraph{\textbf{WiSe:}}

The WiSe~\cite{haurilet2019wise} dataset consists of $1300$ slide images. It is divided into a training set, validation set, and test set consisting of $900$, $100$, and $300$ slide images annotated with $25$ overlapping region categories.

\paragraph{\textbf{SPaSe:}}

The SpaSe~\cite{haurilet2019spase} dataset contains dense, pixel-wise annotations of $25$ classes for $2000$ slide images~\cite{haurilet2019spase}. It is split into training, validation, and test sets containing $1400$, $100$, and $500$ images, respectively.

\subsection{Ablation Study}

In the ablation study, we use the ResNet-101 backbone with an output stride of $16$ and evaluate on the WiSe dataset. First, we examine the best feature vector for adding location encoding in the \textsc{lean}et architecture. Based on Fig.~\ref{ProposedArch} of {\sc lean}et architecture, the location encoding can be added to any of the feature vectors including, $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, $\mathbf{D}$. We empirically find out the best feature for adding location encoding. In addition to it, we also conduct experiments by changing the frequency of the $\sin$ wave encoding in the $\mathbf{L}$ matrix. Finally, we also compare with CoordConv~\cite{liu2018intriguing} approach.

\begin{table}[htb]
\centering
\begin{tabular}{lr}
    \begin{minipage}{.6\linewidth}
\begin{tabular}{@{}p{0.7cm}p{0.7cm}p{0.7cm}p{0.7cm}|c|c@{}}
\toprule
\multicolumn{4}{c|}{Features} &  \multirow{2}{*}{Frequency} & \multirow{2}{*}{mIoU} \\
$\mathbf{A}$ & $\mathbf{B}$ & $\mathbf{C}$ & $\mathbf{D}$ &   &  \\ \hline
   &   &   &  &   1 & 51.07 \\
  \checkmark   &   &   &  &   1 & 51.44 \\
  &  \checkmark  &   &   &   1 & 52.26 \\
  &   &  \checkmark  &   &   1 & 52.12 \\
  &   &   &  \checkmark  &   1 & 50.04 \\ \hline
    &  \checkmark  &   &   &   2 & 51.14 \\
        &  \checkmark  &   &     & 0.5 & 50.32 \\ \hline
\multicolumn{5}{c|}{ $\mathbf{A}$ + CoordConv~\cite{liu2018intriguing}(width+ height)}   & 51.12
\\ \bottomrule
\multicolumn{6}{c}{(a)}
\end{tabular}
    \end{minipage} &

    \begin{minipage}{.3\linewidth}
        \begin{tabular}{cc}
        \toprule
            $\beta$ & mIoU \\ \hline
            0&51.73\\
            1&50.25 \\
            0.5&51.83 \\
            learning & 52.26
            \\ \bottomrule
            \multicolumn{2}{c}{(b)}
        \end{tabular}

\end{minipage}
\end{tabular}
\caption{It shows ablation studies and the impact of hyper-parameters on a validation set of the WiSe dataset. Table (a) shows the result of segmentation by adding location encoding to the features $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, and $\mathbf{D}$. Table (b) shows the effect of manual $\beta$ values and as a learnable parameter.\label{ablationStudy}}
\end{table}

Table~\ref{ablationStudy} shows the ablation studies conducted in the proposed network architecture by changing the various hyper-parameters. The Table~\ref{ablationStudy}(a) shows that the location encoding on the feature vector outperformed the traditional CoordConv~\cite{liu2018intriguing} approach. The experiment also shows that adding location encoding to the intermediate feature $\mathbf{B}$ and $\mathbf{C}$ is comparatively better than adding it to $\mathbf{A}$ and $\mathbf{D}$. The best frequency of $\sin$ wave in the $\mathbf{L}$ is found as 1. While Table~\ref{ablationStudy} (b) shows the learnable $\beta$ value outperforms the manually assigned $\beta$ value.

\subsection{Comparison with State-of-the-Art Techniques}

Comparison results of the proposed network with state-of-the-art techniques are presented in Table~\ref{result1}. From the table, we observe that none of the attention mechanism (e.g., {\sc han}et~\cite{Choi_2020_CVPR}, {\sc dan}et~\cite{fu2019dual}, and {\sc dran}et~\cite{fu2020scene}) is better than Haurilet \emph{et al.}~\cite{haurilet2019spase} on SPaSe dataset. While in case of WiSe dataset, all attention architectures {\sc han}et~\cite{Choi_2020_CVPR}, {\sc dan}et~\cite{fu2019dual}, and {\sc dran}et~\cite{fu2020scene} are better than than Haurilet \emph{et al.}~\cite{haurilet2019wise}. {\sc dran}et~\cite{fu2020scene} obtains the best performance ($35.77$\% mean IoU and $78.64$\% PA) among all existing techniques on SPaSe dataset. While {\sc dan}et~\cite{fu2019dual} obtains the best performance ($44.85$\% mean IoU and $88.80$\% PA) among all existing techniques on WiSe dataset. The proposed network obtains the best results ($46.74$\% mean IoU and $89.80$\% PA) on Wise and ($36.17$\% mean IoU and $79.11$\% PA) on SPaSe dataset as compared to the existing techniques. The Fig.~\ref{qualitativeResult} shows the qualitative performance of the proposed approach with other attention architecture.
\begin{table}[ht!]
\begin{center}
\begin{tabular}{||l||c|c||c|c||} \hline
\textbf{Methods} &\multicolumn{4}{|c||}{\textbf{Datasets}} \\ \cline{2-5}
                 &\multicolumn{2}{|c||}{\textbf{SPaSe}} &\multicolumn{2}{|c||}{\textbf{WiSe}} \\ \cline{2-5}
                 &\textbf{mean IoU} &\textbf{PA}&\textbf{mean IoU} &\textbf{PA} \\ \hline \hline
Haurilet \emph{et al.}~\cite{haurilet2019spase} &35.80  &77.40 &- &- \\
Haurilet \emph{et al.}~\cite{haurilet2019wise}  &-  &-  &37.20 &88.50 \\
HANet~\cite{Choi_2020_CVPR}                    &32.37  &76.54  &39.35 &88.72 \\
DANet~\cite{fu2019dual}                        &33.12  &77.39  &44.85 &88.80 \\
RANet~\cite{fu2020scene}                      &35.77  &78.64  &43.31 &88.77 \\
Ours                                          &36.17   &79.11         &46.74 &89.80 \\ \hline \hline
\end{tabular}
\end{center}
\caption{illustrates the comparison of the proposed method with state-of-the-art techniques on benchmark datasets WiSe and SPaSe datasets. \label{result1}}
\end{table}

\begin{figure}
     \centering
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
\includegraphics[width=\textwidth, trim={0cm
0cm
0cm
0cm
 },clip]{./images/userExp.pdf}
         \caption{Shows the user ranking for various approaches. I and N indicates interactive and non-interactive modes, respectively.}
         \label{userExp}
     \end{subfigure}
     \hfill
     \begin{subfigure}[b]{0.48\textwidth}
         \centering
\includegraphics[width=\linewidth, height = 0.4\linewidth]{images/userInterface.jpg}
         \caption{Illustrates the user interface of the developed mobile application. The mobile screen displays the camera-captured slide image. Various colors highlight the slide's logical regions for better visualization. The {\sc Read All} button plays generated audio sound corresponding to the slide's content.}
         \label{fig_mobile_application}
     \end{subfigure}

\end{figure}

\subsection{Narration System Evaluation}

We establish the effectiveness of the developed slide narration system by comparing the performance with Facebook’s Automatic Alt-Text (\textsc{aat})~\cite{wu2017automatic} and Tesseract \textsc{ocr}~\cite{smith2007overview}. We use $300$ test images of the WiSe dataset for evaluating the system. We select $30$ volunteers who can read and write English. We provide list of $300$ images along with narrating audio files produced by various techniques: the proposed slide narration system, Facebook's Automatic Alt-Text, and Tesseract. Each volunteer can randomly choose $5$ slide images from the given list and evaluate corresponding audio outputs generated by various techniques. The volunteer give a grade (value in the range $[0,\ 10]$) to the generated audio files by various techniques by comparing these audio files with the original slide content. We collect all these grades and plot them as a bar chart (shown in Figure~\ref{userExp}). The user gives the best grade for the audio file corresponding to the ground truth segmentation of the slide. However, the proposed {\sc csns} also performed comparatively better as the ground truth segmentation.
\begin{figure}[!htp]
\includegraphics[width=9cm,height=5cm,trim={
 0.5cm
 1cm
 6.5cm
 0cm
 },clip, page=1]{./images/QualitativeResult.pdf}
\caption{The comparison of the output images of proposed {\sc lean}net architecture and other attention architectures. To visualize the overlapped label region, we use separate image for each label.} \label{qualitativeResult}
\end{figure}
\section{Summary}
This paper presents a classroom slide segmentation network ({\sc lean}et) to segment classroom slide images more accurately. The attention module in the proposed network enriches the learned contextual features by utilizing the location encoding of the logical region (height and width of a region). We develop a Classroom Slide Narration System as an application of classroom slide segmentation for helping {\sc vi} students to read and understand the slide's content. Experiments on benchmark datasets WiSe and SpaSe conclude that the proposed {\sc lean}et achieves the best slide segmentation results as compared to the state-of-the-art methods. We also demonstrate the effectiveness of our developed Classroom Slide Narration System over existing assistive tools in terms of user experiences.

\bibliographystyle{splncs04}

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Multimedia (cs.MM)

Classroom Slide Narration System


The first determination of the actinide Th abundance for a red giant of the Ursa Minor dwarf galaxy

Abstract: The Thorium abundance for the red giant COS82 in the Ursa Minor dwarf spheroidal galaxy is determined based on a high resolution spectrum. This is the first detection of actinides in an extra Galactic object. A detailed abundance pattern is determined for 12 other neutron-capture elements from the atomic number 39 to 68. These elements are significantly over-abundant with respect to other metals like Fe (> 1 dex) and their abundance pattern agrees well with those of the r-process-enhanced, very metal-poor stars known in the Galactic halo, while the metallicity of this object ([Fe/H] ~ -1.5) is much higher than these field stars ([Fe/H] ~ -3.0). The results indicate that the mechanism and the astrophysical site that are responsible for neutron-capture elements in COS82 is similar to that for field r-process-enhanced stars, while the condition of low mass star formation is quite different. An estimate of the age of this object based on the Th abundance ratio is discussed.

\section{Introduction}\label{sec:intro}

The rapid process (r-process) is known to be responsible for about
half of the abundances of elements heavier than Fe in solar-system
material. Since the reaction occurs through very neutron-rich,
unstable nuclei, the understanding of the process is still a
challenging issue in nuclear physics. The astrophysical sites of this
process are still unclear, though numerical simulations have been made
for many possibilities (e.g. Wanajo \& Ishimaru 2006).

An important progress in this field in the past decade is the
discoveries of very metal-poor stars having large over-abundances of
r-process elements. Such objects are expected to record the yields of
nucleosynthetic events in the early Galaxy including the r-process.
Several objects having very large excesses of r-process elements
(e.g. Eu) with respect to other metals ([Eu/Fe]$>1$) are known in the
Galactic halo \citep{sneden03, hill02, christlieb04, barklem05}, and
are called as 'r-II' stars \citep{beers05}. A remarkable result for
these stars is that the abundance patterns of neutron-capture elements
are very similar. Objects having such large excesses of r-process
elements are only found at the metallicity around [Fe/H]$=-3.0$, and
no object having [Eu/Fe]$\gtrsim+1.0$ is known in the range of
[Fe/H]$>-2.5$ in the Galactic halo, though the reason for such
a metallicity distribution is still unknown.

However, the situation is quite different in some dwarf spheroidal
galaxies (dSph), in which high abundance ratios of r-process elements
are found even in stars with relatively high metallicity. The chemical
abundance studies for individual stars in the local group dSph made
large progresses in the past several years, thanks to high resolution
spectrographs mounted at 8--10~m telescopes (e.g. Shetrone et
al. 2001, 2003; Venn et al. 2004). One surprising result derived by
these abundance studies is the existence of stars showing extremely
large enhancements of heavy neutron-capture elements with relatively
high metallicity. The most remarkable star is the red giant COS~82 in
the Ursa Minor dSph. High resolution spectroscopy of this object was
first obtained by Shetrone et al. (2001; in their
paper, this star is referred to as '199'), who reported that this star
is moderately metal-poor ([Fe/H]$\sim -1.5$) but shows large
enhancement of heavy neutron-capture elements ([Eu/Fe]=1.49). Our
previous observing obtained a red spectrum of this
star with higher quality, and has successfully detected some other
heavy elements (e.g. Dy, Er), confirming that the abundance pattern of
elements heavier than Ba almost completely agrees with the r-process
abundance pattern estimated from solar-system material (Sadakane et
al. 2004).  This result is surprising, because no moderately
metal-poor star ([Fe/H]$\gtrsim-2$) in our Galaxy is known to date to
have such large excesses of r-process elements. Tsujimoto \& Shigeyama
(2002) applied their supernova-driven star formation model and
proposed that this phenomenon can be explained by assuming a large
velocity dispersion of the gas from which low mass stars like COS~82
formed.

In this Letter, we report the detection of the Th absorption line at
5989~{\AA} for this object by the re-analysis of our high resolution
spectrum obtained by \citet{sadakane04}. The actinide Th has an
isotope whose half life is 14.05 Gyr, and is produced only by the
r-process. The production of actinides is very sensitive to the
environment of the sites (e.g. entropy, timescale of the explosion)
according to model calculations for the r-process. Therefore, the Th
abundance can be a very strong constraint on the study of the origin
of the neutron-capture elements in this object. While Th abundance has
been determined for more than ten field metal-poor stars and globular
cluster stars, our measurement for COS~82 is the first determination
of an actinide abundance for an extra Galacitic object. We also
discuss the possibility to apply the Th abundance ratio to estimate
the age of this object.

\section{Observation, Analysis and Results}

A high resolution spectrum of COS~82 was obtained by
\citet{sadakane04} with the Subaru Telescope High Dispersion
Spectrograph (HDS, Noguchi et al. 2002). The spectrum covers
4400--7100~{\AA} with a resolving power of 40,000.  The
signal-to-noise ratio per 1.8~km$^{-1}$ pixel is 60 at 5900~{\AA}.
Standard data reduction procedures are carried out with the IRAF
echelle package\footnote{IRAF is distributed by the National Optical
  Astronomy Observatories, which is operated by the Association of
  Universities for Research in Astronomy, Inc. under cooperative
  agreement with the National Science Foundation.} as described by
\citet{aoki05}. Equivalent widths for isolated absorption lines are
measured by fitting gaussian profiles.

A standard abundance analysis was made for the measured equivalent
widths using the model atmospheres of \citet{kurucz93}. The analysis
was made primarily based on the line list of \citet{sadakane04}. We
added some weak Fe {\small I} lines, which enable one to determine the
micro-turbulence more accurately.  The effective temperature ($T_{\rm
eff}$) of 4300~K, determined by \citet{sadakane04}, was adopted in the
analysis. The micro-turbulence ($v_{\rm turb}$) and gravity ($g$) are
determined so that the derived Fe abundance is not dependent on the
strengths of Fe {\small I} lines nor on the ionization stage,
respectively. The results ($\log g=0.6$~dex and $v_{\rm
turb}=1.7$~km~s$^{-1}$) agree with those of \citet{sadakane04} within
the estimated errors. The metallicity  is assumed to be the
Fe abundance ([Fe/H]). The Fe abundance derived by the present
analysis is [Fe/H]$=-1.42$, adopting the solar abundance of
$\log\epsilon$(Fe) = 7.45 \citep{asplund05}. The results of the abundance
analysis are given in table \ref{tab:res}.

The Th abundance is determined from the Th {\small II} 5989~{\AA}
line. The oscillator strengths of Th lines, including the 5989~{\AA}
line, are obtained by \citet{nilsson02}.  This line was investigated by
\citet{yushchenko05} for the field metal-poor star HD~221170. The line
was measured for the same object, using a higher quality spectrum, by
\citet{ivans06}, who reported that the abundance determined from this
line agrees with those from other Th {\small II} lines. We performed
an analysis for HD~221170 using the spectrum obtained with the Subaru
Telescope and the model atmosphere \citep{kurucz93} with the
parameters adopted by \citet{ivans06}, and confirmed that the
derived Th abundance agrees very well with their result. We note that
the partition function of the Th {\small II} was calculated using the
energy levels listed by \citet{bw92}.

Though telluric absorption lines, probably due to water vapor, exist
in this wavelength range, we found that the Th {\small II} 5989~{\AA}
line of COS~82 is not distinctively affected by them.
Figure~\ref{fig:sp} shows the observed spectrum along with the
synthetic ones for three Th abundances. The data for spectral lines
other than the Th {\small II} line were adopted from the list of
\citet{kurucz95}. The wavelength of the observed spectrum is
determined from 137 Fe {\small I} lines in the spectrum. We identified
the absorption feature at 5989.3~{\AA} as a Nd {\small II} line.
Although a Nd {\small II} line at 5989.378~{\AA} with the lower
excitation potential of 6005.270 cm$^{-1}$ is listed by
\citet{kurucz95}, J. E. Lawler (private communication) suggests that
the line should be the transition from 19758.540 to 3066.755
cm$^{-1}$ at $\lambda_{\rm air} =$ 5989.312~{\AA}. Since the transition
probability of this line is unknown, we assume it to be $\log
gf=-2.05$. This $gf$-value and the Nd abundance determined from other
spectral regions well reproduce the strength of the line for COS~82 as
well as for HD~221170.

The Th abundance is determined by $\chi^{2}$-fitting of synthetic
spectra to the observed one from 5988.9 to 5989.2~{\AA}, which is not
severely affected by the 5989.3~{\AA} feature. The derived abundance
is $\log \epsilon$(Th)$=-0.25$. The 2$\sigma$ level of the fitting
error is 0.07~dex. The continuum level is estimated for the spectrum
around the absorption features. The uncertainty of the continuum
placement ($\sim 0.5$\%) results in an possible error of 0.03~dex in
the Th abundance. We estimate the line broadening for relatively clean
lines in the spectrum, and found that the line widths are primarily
determined by the instrumental broadening, and no significant
broadening by macro-turbulence nor rotation is detected. The possible
errors in the Th abundance due to the adopted broadening parameter
($\pm 0.5$~km~s$^{-1}$) is 0.03~dex.

The abundances of other elements are determined using the line list of
\citet{sadakane04} with some modifications. The transition
probabilities of Nd {\small II}, Sm {\small II} and Gd {\small II} are
updated, adding several new lines to the list, using the data of
\citet{hartog03}, \citet{lawler06} and \citet{hartog06}, respectively.
The effects of hyperfine splitting for Ba, La, Eu, and Pr are included
using the line data obtained by \citet{mcwilliam98},
\citet{lawler01a}, \citet{lawler01b}, and \citet{ginibre89},
respectively. We also measure the O abundance from the [O {\small I}]
6300.3~{\AA} line. The results are listed in table~\ref{tab:res}. An
upper limit of the Os abundance is derived from the Os {\small I}
5584~{\AA} line. Estimates of upper limits for other neutron-capture
elements are also attempted, but no meaningful result is derived from
our spectrum. The abundances of Fe and most neutron-capture
elements derived by the present analysis agrees with the results of
\citet{sadakane04} within the errors (see below). Exceptions are Eu
and Gd abundances, for which the differences between the two
measurements are about 0.25~dex. This would be due to the difference
of the spectral features and line data adopted in the analysis. The
abundances of Fe and some neutron-capture elements determined by
\citet{shetrone01} agree well with our results, while their Ce and Sm
abundances are more than 0.3~dex higher than ours. We suspect that
their results are uncertain because of the low S/N ratio in the blue
region of their spectrum where Ce and Sm lines used in their analysis
exist.

The abundance uncertainties for elements for which more than 10 lines
are measured are estimated by $\sigma N^{-1/2}$, where $\sigma$ is the
standard deviation of the abundances from individual lines and $N$ is
the number of lines used. A typical uncertainty of the equivalent
width measurement from our spectrum is 6~m{\AA}.  We performed
abundance analyses for Fe {\small I} lines with equivalent widths from
30 to 200~m{\AA}, changing the equivalent widths by 6~m{\AA}. The
effects of the change of equivalent widths on the derived abundances
are 0.15--0.20~dex, which agree with the $\sigma$ (0.18~dex) for Fe
{\small I} ($\sigma_{\rm Fe I}$). For elements whose abundances are
determined from less than 10 lines, the random error is estimated by
$\sigma_{\rm Fe I}N^{-1/2}$. Errors due to the uncertainty of
atmospheric parameters are estimated by calculating abundances
changing atmospheric parameters by $\Delta(T_{\rm eff})$=150~K,
$\Delta(\log g)$=0.3~dex, $\Delta$([Fe/H])=0.2~dex, and $\Delta(v_{\rm
turb})=0.3$~km~s$^{-1}$. These errors are added, in quadrature, to the
random errors estimated above, and are given in table~\ref{tab:res}
($\sigma_{\rm total}$). We found, however, that the abundances of the
neutron-capture elements determined from the ionized species
systematically scales by changes of the adopted atmospheric
parameters. For the discussion based on the abundance pattern, we
calculate the average of the abundance changes for heavy
neutron-capture elements (La--Er) by changing the atmospheric
parameters, and adopt the deviation from this systematic abundance
change for individual elements as the errors due to the uncertainties
of atmospheric parameters. This results in quite small errors ($\leq
0.05$~dex). These errors and the random errors are added in
quadrature, and are also given in table~\ref{tab:res} as $\sigma_{\rm
n-cap}$.

\begin{figure}
  \begin{center}
    \FigureFile(80mm,150mm){fig1.ps}
  \end{center}
\caption{
Comparisons of synthetic spectra for the region including the Th II
5989~{\AA} line with the observed spectrum. The Th abundances assumed
in the calculations are $\log\epsilon$(Th)$= -0.15, -0.25, -0.35$
(solid lines), and $-\infty$ (dotted line). }\label{fig:sp}
\end{figure}

(see text).

\begin{figure}
  \begin{center}
    \FigureFile(80mm,150mm){fig2.ps}
  \end{center}
\caption{
Upper panel: Scaled abundances of neutron-capture elements for COS~82
(filled circles), CS~22892--052 (open circles), CS~31082--001
(crosses), and the solar-system r-process component (solid line). The
abundances are normalized to the average of the abundances of the 10 elements
from Ba to Er in the logarithmic scale. Lower panel: Abundance
differences in the logarithmic scale between COS~82 and CS~22892--052
(open circles) and that between COS~82 and the solar-system r-process
component (filled circles). Solid lines indicate the averages of the
abundance differences for Ba--Er.}\label{fig:pattern}
\end{figure}

\begin{longtable}{lrrrrrr}
\caption{Elemental Abundances Results}\label{tab:res}
\hline
\hline
species & $\log\epsilon_{\rm sun}$ & $\log\epsilon$& [X/Fe] & $N$ & $\sigma_{\rm totol}$ & $\sigma_{\rm n-cap}$ \\
\endhead
\hline
Fe I  & 7.45 & 6.03 &  ...  & 137 & 0.27 & ... \\
Fe II & 7.45 & 6.02 &  ...  &  13 & 0.18 & ... \\
O I   & 8.70 & 7.45 &  0.17 &   1 & 0.2  & ... \\
Y II  & 2.21 & 1.22 &  0.42 &   6 & 0.23 & 0.14 \\
Zr II & 2.59 & 1.82 &  0.65 &   1 & 0.24 & 0.19 \\
Ba II & 2.17 & 1.36 &  0.60 &   3 & 0.30 & 0.21 \\
La II & 1.13 & 0.52 &  0.81 &   8 & 0.16 & 0.06 \\
Ce II & 1.63 & 0.76 &  0.55 &   4 & 0.20 & 0.14 \\
Pr II & 0.80 & 0.20 &  0.82 &   7 & 0.19 & 0.11 \\
Nd II & 1.45 & 1.00 &  0.97 &  41 & 0.17 & 0.05 \\
Sm II & 0.98 & 0.90 &  1.34 &   2 & 0.21 & 0.13 \\
Eu II & 0.52 & 0.34 &  1.24 &   3 & 0.17 & 0.11 \\
Gd II & 1.09 & 1.10 &  1.43 &   3 & 0.16 & 0.12 \\
Dy II & 1.17 & 1.13 &  1.38 &   3 & 0.17 & 0.10 \\
Er II & 0.97 & 0.73 &  1.18 &   3 & 0.17 & 0.11 \\
Th II & 0.09 &-0.25 &  1.08 &   1 & 0.15 & 0.08 \\
\noalign{\smallskip}
\hline
\end{longtable}

\section{Discussion and concluding remarks}

The upper panel of figure~\ref{fig:pattern} shows the abundance patterns
of neutron-capture elements for COS~82, the solar-system r-process
component \citep{simmerer04}, and the two r-process-enhanced field
stars CS~22892--052 \citep{sneden03} and CS~31082--001
\citep{hill02}. The abundances are normalized to the average of Ba--Er
ones. As reported by \citet{sadakane04}, the overall abundance pattern
of heavy neutron-capture elements (Ba--Er) in COS~82 agrees well with
the r-process abundance pattern. In the lower panel, the abundance
differences between COS~82 and the solar-system r-process component, as
well as that between COS~82 and CS~22892--052, are shown. The Th
abundance of COS~82 is lower than the normalized value of the
solar-system r-process component, while that agrees very well with the
value of CS~22892--052. This point is discussed below in more
detail. The abundance of the light neutron-capture element Y in COS~82
is also lower than the normalized value in the solar-system r-process
component as found in field r-process-enhanced stars.

Although such small deviation exists, an important result derived from
the comparison is that the abundance pattern of neutron-capture
elements, including Th, in COS~82 agrees with the pure r-process
abundance pattern. According to model calculations for the r-process,
the actinide abundances with respect to other neutron-capture elements
are very sensitive to the model parameters (e.g. electron fraction,
entropy; Wanajo et al. 2002). The agreement of the Th abundance ratio
to other neutron-capture elements in COS~82 with that of CS~22892--052
indicates that the mechanism and astrophysical sites that are
responsible for the heavy elements in COS~82 is common with those for
the r-process-enhanced field stars. Recent abundance studies for
r-process-enhanced stars suggest that the r-process occurs under a
quite limited condition (e.g. supernova explosions whose progenitors
have mass in a very narrow range). Our result shows that this is also
applied to stars even in dSph.

The metallicity of COS~82 is more than one order of magnitude higher
than those in the field r-II stars ([Fe/H]$\sim -3$). The chemical
composition of field stars with such metallicity is believed to be
affected by a number of nucleosynthesis events.  However, the
metallicity, as well as the amount of neutron-capture elements, should
be related to how the yields from a nucleosynthesis event are captured
by the clouds from which next generation, low mass stars formed, and
the neutron-capture elements of COS~82 might be provided by a single
event. For instance, \citet{tsujimoto02} proposed that the large
excesses of neutron-capture elements in a few objects in dSph,
including COS~82, are the result of a larger velocity dispersion of
the gas from which low mass stars like COS~82 formed in dSph than in
the Galactic halo.  That is, the yields of one supernova are recorded
in the next generations of stars with smaller mixing with
metal-deficient (or metal-free) gas in dSph.

Finally, we discuss an estimate of the age of COS~82 from the abundance
ratio between Th and other stable elements, which could be an
independent calibration for the age measurements for this galaxy. If
the age of this star is shown to be quite long ($>10$~Gyr), the star
formation of this dSph should be completed with a short time scale.
Th has a half life of 14.05 Gyr, and the age of the object (more
strictly, the duration from the r-process event) is estimated from the
Th abundance ratio to other stable elements, if its initial abundance
ratio is given. Here the average of abundances of the nine elements
from La to Er is used as the reference \footnote{Ba is excluded from
the reference elements, because its abundance error is relatively
large due to the strengths of the lines used in the analysis. The
conclusion in this paragraph is, however, unchanged if this element is
included in the calculation.} (E$_{\rm stable}$).  The Th/E$_{\rm
stable}$ ratio of COS~82 agrees very well with that of CS~22892--052,
while that is 0.3~dex lower than the value in the r-process component in
solar-system material (figure~\ref{fig:pattern}).

Unfortunately, the abundance ratio of Th/E$_{\rm stable}$ produced by
the r-process is not constant. Indeed, the actinides including Th are
significantly enhanced, with respect to other neutron-capture
elements, in the r-II star CS~31082--001, compared to that of the
solar-system r-process component \citep{hill02}. However, studies of
Th abundances for a number of r-process enhanced field stars show that
the abundance ratios distribute in a rather narrow range ($-0.6\leq
$Th/E$_{\rm stable} \leq -0.1$), and CS~22892--052 is one of the
objects that have the lowest ratio. Assuming that the initial
abundance ratio of Th/E$_{\rm stable}$ for COS~82 is within this
range, and that the field halo stars including CS~22892--052 are quite
old (their ages are around 12--13 Gyr), COS~82 is also an old object.
(If a higher value of the initial Th/E$_{\rm stable}$ ratio is
assumed, the derived age becomes longer).

The error ($\sigma_{\rm n-cap}$) in the determination of the Th
abundance ratio is 0.08~dex, while the scatter of the abundances of
the above nine stable elements around those of the solar-system
r-process component, as well as of CS~22892--052, is 0.11~dex. We
estimate the error of the Th/E$_{\rm stable}$ to be 0.13~dex, which
results in the uncertainty of the age estimate to be 6 Gyr. Hence,
though the object is suggested to be as old as field halo stars, the
age estimate using Th chronometer gives only a weak constraint. A
measurement of U/Th ratio, which is a more sensitive chronometer, for
this object will enable one to estimate the age with a much smaller
error ($\lesssim$ 2 Gyr). \\

We are grateful to Dr. J. E. Lawler for providing useful Nd {\small
  II} line data.