Syntax Customized Video Captioning by Imitating Exemplar Sentences

Enhancing the diversity of sentences to describe video contents is an important problem arising in recent video captioning research. In this paper, we explore this problem from a novel perspective of customizing video captions by imitating exemplar sentence syntaxes. Specifically, given a video and any syntax-valid exemplar sentence, we introduce a new task of Syntax Customized Video Captioning (SCVC) aiming to generate one caption which not only semantically describes the video contents but also syntactically imitates the given exemplar sentence. To tackle the SCVC task, we propose a novel video captioning model, where a hierarchical sentence syntax encoder is firstly designed to extract the syntactic structure of the exemplar sentence, then a syntax conditioned caption decoder is devised to generate the syntactically structured caption expressing video semantics. As there is no available syntax customized groundtruth video captions, we tackle such a challenge by proposing a new training strategy, which leverages the traditional pairwise video captioning data and our collected exemplar sentences to accomplish the model learning. Extensive experiments, in terms of semantic, syntactic, fluency, and diversity evaluations, clearly demonstrate our model capability to generate syntax-varied and semantics-coherent video captions that well imitate different exemplar sentences with enriched diversities.


INTRODUCTION
V Ideo captioning [1], [2], [3], which aims to automatically generate natural language sentences to describe video contents, has aroused great interest recent years. Inspired by the success of the sequence-to-sequence model [4] in neural machine translation, the encoder-decoder architecture [5] leveraging CNNs and RNNs to encode video semantics and utilizing RNNs to decode sentences has become a common and effective configuration for most video captioning approaches.
Previous methods [3], [6], [7], [8], [9], [10] are mostly trained with the (conditional) maximum likelihood objective, which encourages the use of the n-grams that appeared in the training samples. Consequently, the generated sentences will bear high resemblance to training sentences in detailed wording [11]. For example, the generated sentences by the captioning models trained on MSRVTT [12] often present similar syntactic structures like "A is doing B at/on/when C". Although such generated captions can describe the video contents, the monotonous and plain sentence forms are of very limited linguistic diversity and expression variability. Compared with those captioning models, people can express things more freely and vividly in daily life. Such an ability, on the one hand, is due to that we have the basic syntactic knowledge about sentence organization. On the other hand, we can acquire abundant corpora during daily reading and talking, and learn how to express things. Therefore, is it Manuscript received XXXX; revised XXXX. possible to learn and leverage sentence syntactic information to guide the video caption generation? Or, more intuitively, can we use the human-edited sentences to customize video captions so as to strengthen the captioning variability in expression?
To answer the above questions, we propose a novel task, namely Syntax Customized Video Captioning (SCVC) in this paper. As shown in Figure 1, given a video and any syntax-valid exemplar sentence, we aim to generate one caption, which should not only express the video semantics but also follow the syntactic structure of the given exemplar sentence. Since exemplar sentences are easy to acquire with no restrictions, our proposed task provides a promising way to strengthen the diversity and expressiveness of video captioning.
In order to solve the SCVC task, our proposed video captioning model consists of three fully coupled components. Firstly, one hierarchical sentence syntax encoder is proposed to capture the syntactic information of the exemplar sentence, in specific a character-level LSTM and a word-level LSTM are stacked to characterize the local lexical features and the global syntactic structures, respectively. Then, one video semantic encoder is introduced to represent the video semantics. Finally, we design a syntax conditioned caption decoder realized in a two-layer LSTM, in which one layer relies on the exemplar syntactic information to modulate the LSTM for customizing the caption syntax, and the subsequent layer relies on the video features to modulate the LSTM for generating the syntactically structured caption describing video semantics. As we do not have the syntax customized groundtruth captions, we propose a new training strategy, which fully leverages the syntactic and semantic information residing in both conventional video captioning data and our collected auxiliary exemplar sentences to train the overall arXiv:2112.01062v1 [cs.CV] 2 Dec 2021 Father tying shoe laces of his son traveling in train.
Woman preparing food ingredients with her audience watching on TV.

Female patient watching tv and remoting control in hand in hospital bed. Woman cook slicing egg and mixing salad in bowl on wooden table.
Water splashes when a coin dropped in a glass. Egg scatters when a knife cuts at a board.

RELATED WORKS
Video captioning has been extensively studied in the past few years. From pioneering template-based methods [2], [13], [14], [15] which defined special grammar rules to compose captions, to recent sequence-to-sequence architectures which leveraged RNNs to encode videos and then decode captions sequentially [3], [6], [7], [8], [9], [10], [16], numerous improvements have been achieved. However, most of these previous approaches are trained to select words with the maximum probability sampled from the learned distribution of the training corpus, resulting in a monotonous set of generated captions bearing high resemblance to the training data. The above works mainly aim at generating precise captions to describe visual contents for keeping fidelity, inspired by the research of text style transfer [17], [18], [19], [20], [21] in the natural language processing community, some recent works for video captioning further put more emphasis on improving the expressiveness or diversity of the predicted video descriptive sentences [11], [22], [23], [24], [25], [26], [27], [28], [29]. Specifically, Gan et al. proposed a StyleNet to control the style in the captioning process so as to produce attractive captions with the desired style like romantic or humorous [25]. You et al. supplied the sentiment label as one additional dimension of the input feature, so as to make the model learn to control the injection of sentiment words in captions for sentiment-conveying [29]. Wang et al. proposed to sample a Part-of-Speech (POS) sequence based on the video representation. By manually altering the sampled POS, the subsequently predicted captions can also be modified [27]. Chen et al. proposed fine-grained control of image caption generation with an abstract scene graph, which makes the level of details such as attributes or relationships can be captured in the caption generation process [22].
Compared with using the style label/sentiment label/POS tag/scene graph to control caption generation, directly leveraging the exemplar sentences to customize the generated captions is more intuitive and natural. Moreover, since the exemplar sentences are easy to obtain and of various syntactic structures, the expressiveness and diversity of video captions can thereby be enriched through imitating those syntax-varied exemplars.

THE PROPOSED MODEL
In this section, we propose one novel model to tackle the SCVC task, with the model overview illustrated in Figure 2. Concretely, the proposed model consists of three components, namely the hierarchical sentence syntax encoder, the video semantic encoder and the syntax conditioned caption decoder. We will detail the model and demonstrate our training strategy in the following.

Hierarchical Sentence Syntax Encoder
To make one video caption imitate the syntactic structure of the given exemplar sentence, how to extract the exemplar sentence syntaxes rather than its semantics is an essential problem. Hence, as shown in the top-right part of Figure 2, we propose a hierarchical sentence syntax encoder as follows.
Specifically, suppose an exemplar sentence S = [w 1 , ..., w n , ..., w N ] with N words, in which the n-th word consists of L characters w n = [w n,1 , ..., w n,l , ..., w n,L ]. A character-level LSTM, namely LSTM c , is first performed on the sequential characters of each word: Here w c n,l is the embedding of the character w n,l in the word w n , and the corresponding hidden vector h c n,l will encode the context in the word to characterize the subword features of w n . We average the L character-level hidden vectors, and obtain the syntactic word feature w n = Avg(h c n,1 , ..., h c n,L ). As words with similar syntactic behaviors, such as part-ofspeech (POS), often have similar subword characteristics like suffix or prefix [30], our character-based word feature w n will capture the local lexical features to support the sentence level syntax encoding. Subsequently, another word-level LSTM, namely LSTM w , is stacked on LSTM c to aggregate the syntactic word features:  sentence S is obtained, which will be used to control the sentence syntax of the predicted video caption. Word Replacement Mechanism based on POS Tags. When conducting experiments, we observe that the sentence syntax encoder often tends to remember the exact words instead of learning the sentence syntactic structure. In order to make the syntax encoder put more emphasis on the sentence syntaxes and lexical word features, we propose a word replacement mechanism based on their POS tags. Specifically, we firstly use the Stanford NLP toolkit [31] to get the POS tag of each word in the sentence. Then, when feeding the sentence in the syntax encoder, we randomly replace the word in the sentence with the word of the same POS tag, and thus discourage the syntax encoder from memorizing the exact word in the exemplar sentence.

Video Semantic Encoder
For encoding video semantic features, we follow the conventional approaches which first use one pretrained CNN to encode each video frame in the input video V , and get a video feature sequence V = [v 1 , ..., v m , ..., v M ]. Then, an LSTM is leveraged to encode the video contexts, with the aggregated hidden vectors denoting the video semantic representation, which will further be utilized to provide semantic meanings of the generated video caption.

Syntax Conditioned Caption Decoder
In this section, we propose to condition the caption generation process on the encoded exemplar sentence syntaxes to achieve the caption syntax customization. The prediction of each word in the caption is on the basis of the global syntax "backbone", and the video semantics are taken as the "flesh" to replenish the caption with semantic meanings. Based on this consideration, we establish a two-layer LSTM to decode the caption, with the first layer modeling sentence syntaxes while the second layer fulfilling sentence semantics.
When decoding the t-th word in the caption, the first layer decoding LSTM, which we denote as LSTM syn , takes the concatenation of the embedding of the previous word e t−1 and the attentively summarized syntactic representation a s t = Att(h syn t−1 , H s ) as input, outputting the hidden vector h syn t to control the sentence syntax at this timestep: (3) In order to incorporate the syntactic information a s t and make it better control the recurrent word decoding procedure, we propose a new Conditional Layer Normalization (CLN) mechanism to modulate LSTM syn as shown in the bottomleft part of Figure 2. Specifically, taking a s t as the conditional guidance signal, the computation flow of Eq. (3) proceeds as: Here CLN performs the modulation on the LSTM gates which leverages a s t to conditionally scale and shift the layer normalized vectors: The scaling and shifting vectors f γ (a s t ) and f β (a s t ) are generated by two independent multi-layer perceptrons (MLPs) with a s t as the input, and the three CLNs in Eq. (4) are independent with each other and do not share weights. By equipping CLNs, the syntactic representation a s t is expected to affect the update procedure of LSTM syn , thereby achieving the caption syntax customization.
The second layer LSTM in our syntax conditioned caption decoder, which we denote as LSTM sem , has the same architecture as LSTM syn . Specifically, LSTM sem takes the concatenation of the syntax hidden vector h syn t and the attentively summarized video semantic representation as inputs, and is conditionally con- t is taken as the condition guidance signal to intensify the video contents in the word decoding procedure, making the predicted caption better preserve the video semantics. Through the above two-layer LSTM modeling, the words in the caption will be sequentially predicted based on the

Training Strategy
As we do not have the syntax customized groundtruth video captions, we propose to leverage the pairwise video captioning data in available datasets as well as our collected auxiliary exemplar sentence corpus to train our proposed model.

Training with Pairwise Video Captioning Data
As illustrated in Figure 3(a), the pairwise video captioning data is utilized for training the proposed model, with the groundtruth caption also being taken as the exemplar sentence. As such, the sentence syntax of the caption is extracted and then coupled with the video semantics to predict the caption. For the training of this syntax customized video captioning procedure, two loss terms are introduced as the learning objective, i.e., the syntactic loss and the semantic loss. Syntactic Loss. In our syntax customized video captioning, the syntactic structure of the generated caption should resemble that of the exemplar sentence. We use the Stanford NLP toolkit [32] to extract the constituency parse tree of the exemplar sentence (here is the input video caption), which is presented as a bracket syntactic token sequence. Then we collect the hidden state vectors of the LSTM syn as H syn =[h syn 1 , ..., h syn t , ..., h syn T ], feed them into a basic decoding model such as [10], and predict the corresponding syntactic token sequence of the input exemplar sentence. Accordingly, the syntactic loss can be defined as: L syn v,c is realized by the typical negative log-likelihood loss. V denotes the input video, and C denotes its accompanying caption in the dataset, which is also used as the exemplar sentence. C syn indicates the extracted syntactic token sequence of the input caption C.
Semantic Loss. Since the generated caption should express the semantic meaning of the video, we collect the hidden vectors H sem = [h sem 1 , ..., h sem t , ..., h sem T ] of the LSTM sem that directly serves for caption decoding, and adopt the negative log-likelihood loss for semantic supervision with C:

Training with An Exemplar Sentence Corpus
Since the syntactic structures of the captions in existing video captioning datasets are simple and monotonous, only relying on them to train our model will limit its ability on expression variability. To this end, we collect a large number of syntaxvaried exemplar sentences to enhance our model's learning ability. Meanwhile, as shown in Figure 3(b), we propose a sentence semantic encoder to replace the video semantic encoder in our model. Doing so will make one single sentence can also be autoencoded with the yielded architecture, and further help the training of the shared hierarchical sentence syntax encoder and syntax conditioned caption decoder. Concretely, the sentence semantic encoder relies on one fully-connected layer to encode the glove embedding [33] of each word, and the obtained word feature sequence will be taken as the sentence semantic representation. By feeding the semantic and syntactic representations of the sentence to the syntax conditioned caption decoder, we follow the same procedure in Sec. 3.3 to reconstruct the input sentence. The corresponding syntactic and semantic losses can also be defined: L syn s,s = −logP (S syn |H syn ; S, S), L sem s,s = −logP (S|H sem ; S, S).
Here S denotes the sentence we feed to the sentence autoencoding procedure, which can be our collected exemplar sentence E or the caption C in the existing video captioning datasets. S syn denotes the parse tree token sequence of S.

Overall Training Objective
The above training objective considers either feeding a (video, caption) pair or one single sentence to our model. However, there is also one case of taking one exemplar sentence E and one video V as the inputs to our model. Although we do not have groundtruth caption to support the L sem loss in this case, the L syn loss can still be involved in training since the parse tree sequence E syn of the exemplar sentence can be pre-extracted. Hence, we can obtain an additional syntactic loss as follows: L syn v,e = −logP (E syn |H syn ; V, E).
Summing up the above terms, the overall training objective of our model is defined as:

Dataset and Exemplar Sentence Collection
We conduct our syntax customized video captioning experiments on the MSRVTT [12] and ActivityNet Captions [34] datasets. We pair each video in these two datasets with 20 different syntax-varied exemplar sentences that are originally collected in the Shutterstock Image Description Corpus [35]. Accordingly, 20 different syntax customized video captions will be generated by our proposed model. The details of the datasets and our exemplar sentence collection are as follows.

Evaluation Metrics
To comprehensively evaluate the predicted syntax customized video captions, we refer to the research in the relating text style transfer task [17], [18] and conduct the objective evaluation from the syntactic, semantic and fluency aspects as follows. Syntactic Evaluation. To evaluate whether our generated captions comply with the syntactic structures of the exemplar sentences, we directly compare their constituency parse tree by computing the syntactic Tree Edit Distance (TED) [36] between them after removing word tokens. Smaller TED value means higher syntactic similarity. Semantic Evaluation. For each sentence, we first remove the stop words and then take the average glove word embeddings of the remaining words as its sentence semantic features. In the semantic feature space, the average cosine similarity (COS) between the syntax customized video caption and all the originally groundtruth video captions in the dataset is used to evaluate their semantic coherence. Sentence Fluency. Besides syntactic and semantic evaluation, another important aspect is the sentence fluency of the generated syntax customized video captions. We use a pre-trained language model to measure the perplexity of generated captions as the fluency score. A state-of-the-art BERT model [37] trained on a large scale lower case English dataset is used for the evaluation.

Implementation Details
To represent videos, we leverage the Inception-Resnet-v2 network [38] pretrained on the ILSVRC-2012-CLS image classification dataset [39] to extract a 1,536 dimensional feature vector for each frame. Videos in the MSRVTT and ActivityNet Captions datasets are represented with evenly spaced 30 and 100 features, respectively. Shorter videos of less than 30 or 100 features are padded with zero vectors. The word embedding size and all the LSTMs' hidden sizes are set as 256. Adam [40] optimizer is used for training. For the word replacement mechanism, the probability to replace the word with their same POS-tagged word is 0.7. The model is implemented with PyTorch [41]. For getting the syntactic token sequence, we first extract the constituency parse tree of the input exemplar sentence with Stanford NLP parser [32]. For example, the parse tree of the sentence " a short clip of news on a white background" is " (ROOT (NP (NP (NP (DT) (JJ) (NN)) (PP (IN) (NP (NN)))) (PP (IN) (NP (DT) (JJ) (NN)))))". We regard the sentence parse as a syntactic token sequence, where each element such as "ROOT", "NN" or "(" is an independent token in the sequence. Such a syntactic token sequence will be used for the syntactic loss as stated in Sec. 3.4.

Compared Methods
Since the proposed syntax customized video captioning is a new task, there is no baseline method for direct comparisons. In this section, we compare our proposed model with the following methods: • The ExemplarOnly method which directly outputs the given exemplar sentence as the video caption while does not consider the video contents. • The conventional sequence-to-sequence (Seq2Seq) video captioning method [10], which only predicts one single caption for one video while does not consider the sentence syntax. • The Template method, which firstly detects visual concepts [42] from each video to generate a bag of videorelated words. Then every content word in the exemplar sentence is simply replaced by one video-related word from the bag that has the same part-of-speech as the removed word. • The Gated Fusion Network (GFN) for video captioning with POS sequence guidance [27]. We replace their intermediately sampled POS sequences from videos with the POS sequences of the given exemplar sentences, and attain the corresponding syntax customized video captions. Table 1 compares our proposed model with the baseline methods in terms of the objective evaluation metrics introduced above. The ExemplarOnly method directly outputs exemplar sentences without considering video contents, making the exemplar and output sentence structures exactly the same and TED scores as 0.0, whereas the semantic coherence to the video is much lower. The Seq2Seq model mainly focuses on describing video contents while neglecting the given exemplar sentence syntaxes, so it gets higher semantic COS scores but inferior TED scores. The Template method simply fills the detected content words from videos in the exemplar sentences while does not consider the global sentence meanings. Doing so makes the generated captions fit the original exemplar syntaxes very well but the captions are not fluent and get high (poor) perplexity scores. Meanwhile, the semantic scores are also lower. Although GFN attempts to leverage POS features to guide video captioning, its controllability of caption syntax is weak and results in high TEDs between exemplar sentences and captions. The main focus of GFN is still to improve the semantic accuracy of video captions and keep their fidelity with the training corpus.

Quantitative Evaluation
Since the Seq2Seq and GFN models often generate simple sentence descriptions with common sentence forms, their perplexity scores are correspondingly lower. The collected exemplar sentences are human-edited, which often take more  various and complex syntaxes and are free in grammar. Thus, the perplexity scores of the ExemplarOnly method are relatively higher. Based on these complicated exemplars, the yielded Template and Ours method will also inevitably present higher perplexity scores. Our proposed model achieves much better TED scores and comparable COS scores than the Seq2Seq and GFN models, which shows that our method can comply with the exemplar sentence syntaxes while describing the video semantics. Also, imitating the complex exemplar sentences does not significantly affect the sentence fluency of our generated captions, with the perplexity scores increasing in a reasonable range compared to the ExemplarOnly method. Figure 4 shows the qualitative results for the syntax customized video captioning task. The Seq2Seq model can generate only one simple caption for one video, which is monotonous and bears high resemblance to the training corpus. The Template method which simply replaces main content words in the exemplar sentences make the generated captions lack of fluency. The GFN model cannot capture sentence syntaxes and produces similar or the same captions for different exemplar sentences. Our generated captions comply with the exemplar sentence syntaxes well, and more concrete video contents like "snow helmet" and "green grass" can also be described in the captions more expressively. Meanwhile, the two different syntax customized video captions further provide us the intuition of diverse video captioning.

Qualitative Evaluation
To better demonstrate the effectiveness of our proposed method on syntax customized video captioning, we further provide more qualitative results of Ours method in Figure 7 and Figure 8 at the end of the paper. It can be observed that on different types and contents of videos, our generated captions can not only present the video semantics accurately, but also well imitate the syntaxes of the given exemplar sentences.

Human Evaluation
Besides the quantitative and qualitative evaluations, we also conduct a human assessment of the generated captions. The Groundtruth caption: Two snowboarders preparing for a decent on top of a snowy mountain. Seq2Seq generated caption: A man is riding a bike on a mountain.

Exemplar Sentence 1: Little rat has been caught inside the mouse trap.
Template syntax customized caption 1: Snow man has parked caught inside the woman person. human evaluators were asked to rate the three aspects of the generated captions -syntax similarity with the exemplar sentence, semantic coherence with the video, and sentence fluency. Each rating aspect is graded in three scales {0.0, 0.5, 1.0}, and the higher, the better. We randomly chose 200 captions generated by each compared method from the test set and invited 10 evaluators (5 males and 5 females) to grade them. The average rating scores are presented in Table 2. These scores are in agreement with the above quantitative evaluation. Although the complex exemplar sentence syntaxes may influence the sentence fluency of our generated captions, they are still realistic and can describe the video semantic contents meanwhile imitating the exemplar sentence syntaxes.

Captioning Diversity Evaluation
Since we collect 20 different exemplar sentences for each video, we will accordingly get 20 different syntax customized captions with the proposed model. To measure the diversity among these generated captions, we follow the evaluation metrics introduced in [43], and report the LSA and Self-CIDEr based diversity scores in Table 3. The collected exemplar sentences are independent of each other, and it is evident that the diversity scores of the ExemplarOnly method are fairly high and can be seen as upper bounds in this diversity evaluation experiment. The conventional Seq2Seq model can only generate one single caption for one video, and thus it cannot achieve diverse video captioning and gets all the scores as 0.0. The GFN method can vary the video captions to some extent, while the captioning diversity remains limited. The Template method and Ours model can generate diverse captions to describe the video contents, and Ours model even gets comparable LSA and Self-CIDER scores to other state-of-the-art methods reported in [43]. Hence, the results verify that our proposed model can indeed strengthen the diversity of video captioning.

The Influence of Exemplar Sentence Length
In our experiments, we find that given different exemplar sentences, there can be different object descriptions coming from the video. Therefore, it is also interesting to study the effect of exemplar sentence length to the number of different object words in the generated captions. Therefore, we first count the different verb and noun words in each generated syntax customized video caption of our model. Then, we plot 6 scatter maps in Fig. 5, where the top three scatters are for MSRVTT dataset and the bottom three are for ActivityNet Captions dataset. The x coordinate of each point in the scatter map means the length of the given exemplar sentence, and the y coordinate means the number of different noun, verb, and noun+verb words (we call them objects in general) in the corresponding syntax customized video caption, respectively. The pearson correlation coefficient between the exemplar sentence length and the object number is also presented in the top-right part of each subfigure. It can be observed that the object number is positively correlated to the exemplar sentence length, no matter for verbs, nouns or their union.
Such results indicate that when we provide longer exemplar sentences, more objects will be incorporated in the generated syntax customized captions, making more representative and diverse video descriptions.

Discussions on the Conventional Metrics
The generated syntax customized video captions one the one hand should semantically describe the video contents, on the other hand should syntactically follow the given exemplar sentences, thus making the caption syntactic structures be greatly changed in the syntax imitation procedure. As such, our generated syntax customized video captions will be greatly different from the original groundtruth captions (in the MSRVTT and ActivityNet Captions datasets) on ngram characteristics and detail wording. Since conventional captioning evaluation metrics, such as BLEU, take the n-gram similarity between the predicted and groundtruth captions as a basic measurement, they are not very appropriate for evaluating our syntax customized video captioning task.
In this section, we still provide the evaluation results in the conventional video captioning metrics in Table 4 for reference. As expected, our proposed model does not achieve high performances in these metrics, while the Seq2Seq and GFN models get good results because they are still focusing on fitting the detail wording patterns in the training corpus. The different evaluation results between the conventional captioning metrics and our human ratings in the main paper also indicate the limitation of the conventional metrics in evaluating syntax-varied captions.

Model Contribution Examination
In this section, we perform three groups of ablation studies on the MSRVTT dataset to examine the contributions of our proposed model, with the results shown in Figure 6. Each point in this figure represents the performance of the model on the validation set during the training process. The xaxis indicates the 10/TED scores, and the y-axis denotes the COS scores. Since lower TED and higher COS means better performance, models at the top right are more desirable.
Loss Components. Comparing the model with only the L v,c =L syn v,c +L sem v,c loss and the model additionally considering the L s,s =L syn s,s +L sem s,s loss in Figure 6(a), we can observe that training with an exemplar sentence corpus as introduced in Sec. 3.4.2 can augment the training data and thereby improve the model performance. By considering both L v,c and L syn v,e terms, the performance curve rises steadily without fierce fluctuation. It shows that even without paired groundtruth syntax customized captions, merely providing supervision on sentence syntaxes can also help the model imitate the syntactic structures of the given exemplar sentences. By combining all the above terms, our proposed model achieves the best performances.
Sentence Syntax Encoding. Ours-WordSyntaxEmbed model in Figure 6(b) drops the character-level LSTM, and only keeps the word-level LSTM to encode the sentence syntactic structure. Ours-NoWordReplace model does not adopt the word replacement mechanism as we stated in Sec. 3.1. The performance superiority of the full model (Ours) over these two ablation models verifies the benefits of the character-level LSTM in encoding subword features and the word replacement mechanism. These designs can help our model capture exemplar sentence syntaxes effectively and filter out unnecessary sentence semantics, which is very crucial for syntax customized video captioning.
Caption Decoding. Ours-Concate model in Figure 6(c) simply concatenates sentence syntactic and video semantic representations and takes them as the inputs to the decoding LSTM without the proposed CLN processing. We can observe that the full model (Ours) achieves higher COS scores than Ours-Concate, which shows that equipping CLN in the decoding LSTM is beneficial to video semantic preservation when performing syntax customization. In Ours-Semantic1Syntax2 model, we feed video semantic representation to the first layer LSTM and sentence syntactic representation to the second. Such a semantic-first syntaxfollow architecture is significantly inferior to our proposed model. The reason mainly dues to that using syntactic information to guide the word prediction in the second layer LSTM will hinder it from choosing appropriate words coherent to the video semantics.

The Influence of Word Replacement Probability
For the word replacement mechanism proposed in Sec. 3.1, we have mentioned that we will randomly replace the word     Table 5.
It can be observed from Table 5 that if the replacement probability is small (e.g., 0.1, 0.3, 0.5), the models get lower COS semantic scores and smaller TED scores. The reason is that in these cases, the models tend to remember the exact words in the exemplar sentences instead of learning their POS tag information and sentence syntaxes, and thus causes the output captions just copy the words from the exemplar sentences. If we increase the replacement probability to higher value (e.g., 0.9), the TED score increases and the COS score decreases a little compared to those of Ours-0.7, while these two settings are comparable to each other in perplexity. Considering the overall better performance of Ours-0.7 in balancing the three metrics, setting replacement probability as 0.7 is suitable for our proposed model.

CONCLUSION
In this paper, we proposed a novel syntax customized video captioning task by imitating different exemplar sentences to strengthen the diversity and expressiveness of video captioning. To solve this task, a hierarchical sentence syntax encoder was proposed to capture both the local subword features and global sentence syntaxes of the exemplar sentence, based on which a two-layer syntax conditioned caption decoder was devised to generate the syntax customized caption expressing video semantic meanings. Comprehensive experiments verify that the generated captions by our model can vividly describe video contents while complying with different exemplar sentence syntaxes, thus indicating our contributions to enrich the video captioning diversity.
Groundtruth caption: A person is drawing a picture of a squidward. Exemplar Sentence: Happy business team with arms crossed at the office. Ours syntax customized caption: Animated cartoon character with glasses isolated at the picture.  (e) Groundtruth caption: She continues speaking to him and has him watch her while she steps on the machine. Exemplar Sentence: Beautiful carrot cake with golden nuts and fresh berries on a rustic board. Ours syntax customized caption: Several people stop with two legs and small shorts on a elliptical bike.
(a) Fig. 8: More qualitative results on the ActivityNet Captions dataset. For each video, we provide one groundtruth caption, one exemplar sentence, and the syntax customized video caption predicted by our proposed model.