Estimation of Reliable Proposal Quality for Temporal Action Detection

Temporal action detection (TAD) aims to locate and recognize the actions in an untrimmed video. Anchor-free methods have made remarkable progress which mainly formulate TAD into two tasks: classification and localization using two separate branches. This paper reveals the temporal misalignment between the two tasks hindering further progress. To address this, we propose a new method that gives insights into moment and region perspectives simultaneously to align the two tasks by acquiring reliable proposal quality. For the moment perspective, Boundary Evaluate Module (BEM) is designed which focuses on local appearance and motion evolvement to estimate boundary quality and adopts a multi-scale manner to deal with varied action durations. For the region perspective, we introduce Region Evaluate Module (REM) which uses a new and efficient sampling method for proposal feature representation containing more contextual information compared with point feature to refine category score and proposal boundary. The proposed Boundary Evaluate Module and Region Evaluate Module (BREM) are generic, and they can be easily integrated with other anchor-free TAD methods to achieve superior performance. In our experiments, BREM is combined with two different frameworks and improves the performance on THUMOS14 by 3.6% and 1.0% respectively, reaching a new state-of-the-art (63.6% average mAP). Meanwhile, a competitive result of 36.2% average mAP is achieved on ActivityNet-1.3 with the consistent improvement of BREM. The codes are released at https://github.com/Junshan233/BREM.


INTRODUCTION
With the conversion of mainstream media information from text and images into videos, the number of videos on the Internet grows rapidly in recent years. Therefore video analysis evolves into a more important task and attracts much attention from both academy and industry. As a vital area in video analysis, temporal action detection (TAD) aims to localize and recognize action instances in untrimmed long videos. TAD plays an important role in a large number of practical applications, such as video caption [11,30] and contentbased video retrieval [3,8].
Recently, a number of methods have been proposed to push forward the state-of-the-art of TAD, which can be mainly divided into three types: anchor-based [16,18,31], bottom-up [14,15,37], and anchor-free [12,26,35] methods. Although anchor-free methods show stronger competitiveness than others with simple architectures and superior results, they still suffer from the temporal misalignment between the classification and localization tasks. Current anchor-free frameworks mainly formulate TAD into two tasks: localization and classification. The localization task is designed to generate action proposals, and the classification task is expected to predict action category probabilities which is naturally used as ranking scores in non-maximum suppression (NMS). However, classification and localization tasks usually adopt different training targets. The feature that activates the classification confidence may lack information beneficial to localization, which inevitably leads to misalignment between classification and localization. To illustrate this phenomenon, we present a case on THUMOS14 [10] in Fig. 1, where a proposal with the highest classification score fails to locate the ground truth action. This suggests that the classification score can't accurately represent localization quality. Under this circumstance, accurate proposals may have lower confidence scores and be suppressed by less accurate ones when NMS is conducted. To further demonstrate the importance of accurate score, we replace predicted classification score of action proposals with the actual proposal quality score, which is tIoU between proposal and corresponding ground-truth. As shown in Tab. 1, mAP is greatly improved, which suggests that accurate proposals may not be retrieved due to inaccurate scores.
Recent attempts adopt an additional branch to predict tIoU between proposal and the corresponding ground truth [12] or focus on the center of an action instance [35]. Although notable improvement is obtained, there is still a huge gap between the performance of previous methods and ideal performance. We notice that previous methods mainly rely on the region view which only considers global features of proposals and ignore local appearance and motion evolvement, which increases the difficulty of recognizing boundary location accurateness, especially for actions with long duration.
In this paper, we propose a new framework that gives insights into moment and region views simultaneously to align two tasks by estimating reliable proposal quality. First, we propose Boundary Evaluate Module (BEM) to acquire boundary qualities of proposals from a moment view. Specifically, BEM focuses on local appearance and motion evolvement for predicting the boundary quality of each temporal location which reflects the distance between the current location to the location of the action boundary. Then, the quality of the generated proposal is calculated by its boundary qualities. However, the duration of realistic actions can vary from a few seconds to minutes and the localization quality of short actions is more sensitive to the boundary error than long actions. To address this, multi-scale boundary quality is adopted in BEM in a divide-and-conquer way which assigns a suitable scale for each proposal depending on its duration. For the region view, we propose a simple but effective module named Region Evaluate Module (REM), which employed the sampled features in proposals as the proposal feature representation and refines proposals. In particular, REM obtains aligned features by sampling at three locations within the action proposals, which contain more contextual information beneficial to estimate reliable proposal quality and accurate boundary. The proposed Boundary Evaluate Module and Region Evaluate Module (BREM) are generic, and they can be integrated with other anchor-free TAD methods to achieve better results. To validate the effectiveness of BREM, we conduct experiments on two popular mainstream datasets THUMOS14 [10] and ActivityNet-1.3 [9]. By combining BREM with a basic anchor-free TAD framework proposed by [33], we achieve an absolute improvement of 3.6% AP@Avg on THUMOS14. When integrating with the stateof-the-art TAD framework ActionFormer [35], we achieve a new state-of-the-art (63.6% AP@Avg) on THUMOS14 and competitive result (36.2% AP@Avg) on ActivityNet-1.3.
Overall, the contributions of our paper are following: 1) Boundary Evaluate Module (BEM) is proposed to predict multi-scale boundary quality and offer proposal quality from a moment perspective. 2) By introducing Region Evaluate Module (REM), the aligned feature of each proposal are extracted to estimate localization quality in a region view and further refine the locations of action proposals.
3) The combination of BEM and REM (BREM) makes full use of moment view and region view for estimating reliable proposal quality and it can be easily integrated with other TAD methods with consistent improvement, where a new state-of-the-art result on THUMOS14 and a competitive result on ActivityNet-1.3 are achieved.

RELATED WORK
Anchor-Based Method. Anchor-based methods rely on predefined multiple anchors with different durations and the predictions refined from anchors are used as the final results. Inheriting spirits of Faster R-CNN [22], R-C3D [31] first extracts features at each temporal location, then generates proposals and applies proposal-wise pooling, after that it predicts category scores and relative offsets for each anchor. In order to accommodate varied action durations and enrich temporal context, TAL-Net [6] adopts dilation convolution and scale-expanded RoI pooling. GTAN [18] learns a set of Gaussian kernels to model the temporal structure and a weighted pooling is used to extract features. PBRNet [16] progressively refines anchor boundary by three cascaded detection modules: coarse pyramidal detection, refined pyramidal detection, and fine-grained detection. These methods require predefined anchors which are inflexible because of the extreme variation of action duration.
Bottom-up Method. Bottom-up methods predict boundary probability for each temporal location, then combines peak start and end to generate proposals. Such as BSN [15], it predicts start, end, and actionness probabilities and generates proposals, then boundarysensitive features are constructed to evaluate the confidence of whether a proposal contains an action within its region. BMN [14] employs an end-to-end framework to generate candidates and confidence scores simultaneously. BU-TAL [37] explores the potential temporal constraints between start, end, and actionness probabilities. Some methods, such as [19,34,38] adopt generated proposals by BSN or BMN as inputs and further refine the boundary and predict more accurate category scores. Our method is inspired by bottom-up frameworks, but we utilize boundary probability to estimate proposal quality instead of generating proposals.
Anchor-Free Method. Benefiting from the successful application of the anchor-free object detection [21,27], anchor-free TAD methods have an increasing interest recently which directly localize action instances without predefined anchors. A2Net [33] explores the combination of anchor-based and anchor-free methods. AFSD [12] is the first purely anchor-free method that extracts salient boundary features using a boundary pooling operator to refine action proposals and a contrastive learning strategy is designed to learn better boundary features. Recent efforts aim to use Transformer for TAD. For example, RTD-Net [26] and TadTR [17] formulate the problem as a set prediction similar to DETR [4]. ActionFormer [35] adopts a minimalist design and replaces convolution networks in the basic anchor-free framework with Transformer networks. Our method belongs to anchor-free methods and is easily combined with anchor-free frameworks to boost the performance.

METHOD
Problem Formulation. An untrimmed video can be depicted as a frame sequence = { } =1 with frames. Action annotations in video consists of action instances Ψ = { , } =1 , where = ( , ) are timestamp of start and end of the -th action instance respectively and is the class label. The goal of temporal action detection is to locate boundaries of actions and recognize categories which cover Ψ as precisely as possible.
Overview. For an untrimmed video denoted as = { } =1 , a convolution backbone (e.g., I3D [5], C3D [28].) is used to extract 1D temporal feature ∈ R / × , where , , denote video frame, feature channel and stride. Then, up-sample is used to for acquiring frame level feature . Multi-scale boundary quality of start and endˆ/ are predicted by (Sec. 3.2). Parallel, several temporal convolutions are used on to generate the hierarchical feature pyramid. For each hierarchical feature, a shared detection head is applied to predict coarse proposals and category confidences. After that, the aligned feature is extracted for each coarse proposal to refine boundaries and scores (Sec. 3.3). The boundary quality of each proposal is interpolated onˆ/ according to the temporal location of boundaries.

Basic Anchor-free Detector
Following recent object detection methods [27] and TAD methods [12,33], we build a basic anchor-free detector as our baseline, which contains a backbone, a feature pyramid network, and heads for classification and localization.
We adopt I3D network [5] as the backbone since it achieves high performance in action recognition and is widely used in previous action detection methods [12,37]. The feature output of backbone is denoted as ∈ R / × . Then, is used to build hierarchical feature pyramid by applying several temporal convolutions. The hierarchical pyramid features are denoted as { ∈ R / × } =1 , where means -th layer of feature pyramid and is the stride for the -th layer.
The heads for classification and localization consist of several convolution layers which are shared among each pyramid feature. For details, for -th pyramid feature, classification head produces category scoreˆ∈ R / ×C , where C is the number of classes. And localization head predicts distance between current temporal location to action boundaries, denoted as Following AFSD [12], the quality branch is also adopted in the baseline model which is expected to suppress low quality proposals.
Based on this baseline model, we further propose two modules named Boundary Evaluate Module (BEM) and Region Evaluate Module (REM) to address the issue of misalignment between classification confidence and localization accuracy. Noteworthily, the proposed BEM and REM are generic and easily combined not only with the above baseline framework but also with other anchor-free methods that have a similar pipeline. The details of BEM and REM would be explained in the rest of this section.

Boundary Evaluate Module
As discussed in Sec. 1, the misalignment between classification confidence and localization accuracy would lead detectors to generate inaccurate detection results. To address this, we propose Boundary Evaluate Module (BEM) to extract features and predict action boundary quality maps from a moment view which is complementary to the region view, thus it can provide more reliable quality scores of proposals.
Single-scale Boundary Quality. Boundary quality maps provide localization quality scores for each temporal location. The quality score is only dependent on the distance from the current location to the location of the action boundary of ground truth.
In practice, we set predefined anchors 1 at each temporal location, denoted as { } =1 , where = [ − /2, + /2] denotes the anchor at -th temporal location and is the predefined anchor size. For a video with action ground truth { , , , } =1 , we define start and end region for -th instance as = [ , − /2, , + /2] and = [ , − /2, , + /2]. The boundary quality maps for start boundary and end boundary , ∈ R are calculated by where tIoU is temporal IoU. The parameter controls the region size, examples for small and large are shown in Fig. 3 denoted as Small scale and Large scale separately. In this way, each score in the quality map indicates the location precision of the start or end boundary. In the inference phase, proposal boundary quality is acquired by interpolation at the corresponding temporal location. Previous works [16,36] formulate the prediction of boundary probability as a binary classification task that can't reflect the relative probability differences between two different locations. However, we define precise boundary quality using tIoU between the predefined anchor and boundary region. Moreover, previous works define positive locations by action length (e.g., locations lie in [ − /10, + /10] are positive samples in [16] and [36], where and are action length and start location of ground-truth). Thus, the model has to acquire the information of the duration of actions. But it is difficult because of the limited reception field, especially for long actions. So, the definition of boundary quality in Eq. 2 is regardless of the duration of actions. Another weakness of previous works is that they define the action boundary using a small region which leads to that only the proposal boundary closing to the ground-truth boundaries being covered. In this work, we can adjust to control the region size. We demonstrate that small region size is harmful to performance in our ablation.
Multi-scale Boundary Quality. Actions with different duration require different sensitivity to the boundary changes. Fig. 3 helps us to illustrate this. If we use Small scale, a short proposal and a long proposal (blue lines) with the same localization error of start boundary acquire the same boundary qualities of 0.41, but the actual tIoU of the long proposal is 0.9. Similarly, if we use Large scale, these two proposals acquire boundary qualities of 0.95, but the actual tIoU of the short proposal is 0.57. Thus, single-scale boundary quality is suboptimal for varied action duration. The scale should dynamically adapt the duration of actions. To address this, we expand the single-scale boundary quality maps into quality maps with multi-scale anchors. Thus, for a proposal, we can choose a suitable anchor depending on its duration (as yellow points show in Fig. 3).
In detail, start and end boundary quality maps are extended to two dimensions corresponding to temporal time steps and anchor scales, denoting as , ∈ R × , where is the number of predefined anchors. We predefine multiple anchors with different size at each temporal location, denoting as { } =1 , where = { ,1 , · · · , , } denoting predefined anchors. The anchor size is defined as representing evenly spaced number from to , where and indicate the maximum and minimum anchor scale. In this paper, is set as 1 that corresponds to the interval time between adjacent input video frames and depends on the distribution of duration of the actions in datasets. We conduct ablation studies about the selection of in Sec. 4.2. Thus the -th anchor at is , = [ − /2, + /2]. As for a ground-truth ( , , , ), its start and end region for -th anchor can be denoted as , = [ , − /2, , + /2] and , = [ , − /2, , + /2]. Then the multi-scale quality maps , ∈ R × are calculated by In the inference phase, the boundary quality of the proposal is obtained by bilinear interpolation according to the boundaries location and the proposal duration (See Sec.3.4). Implementation. To predict multi-scale boundary quality maps, as shown in Fig.2, the backbone feature is first fed into an upsampling layer and several convolution layers to get the framelevel feature ∈ R × with a higher temporal resolution, which is beneficial to predict quality score of the small anchor. Because the anchor scales may have a large range and different scales need different receptive fields, we adopt a parameter-free and efficient method to generate features. In detail, we use linear interpolation in each anchor to obtain the multi-scale anchor feature map, denoted as ∈ R × × × . In particular, for ( , ) ∈ R × , we uniformly sample features in the scope [ − /2, + /2] from which ensures that the receptive field matches the anchor size. This procedure of interpolation can be efficiently achieved by matrix product [14]. After the multi-scale anchor feature map is obtained, we apply max pooling on the sampled features and a 1 × 1 convolution to extract anchor region representation : where ∈ R × × . Finally, two boundary score maps are obtained based on as follows: where (·) and (·) are convolution layers and (·) is sigmoid function.
Training. We denote label maps forˆandˆas , ∈ R × respectively. The label maps is computed by Eq. 4. We take points where / > 0 as positive. L2 loss function is adopted to optimize BEM, which is formulated as follows: where N / is the set of positive points.

Region Evaluate Model
BEM estimates the localization quality of proposals in the moment view that focuses more on local appearance and motion evolvement.
Although it achieves considerable improvement, as illustrated in Tab. 4, we believe that feature of the region view can provide rich context information which is beneficial to the prediction of localization quality. Therefore, we propose Region Evaluate Module (REM), as shown in the right part of Fig. 2, which first predicts coarse action proposals and then extracts features of proposals to predict localization quality scores, action categories, and boundary offsets. Specifically, REM predicts coarse action offset (ˆ,ˆ), action categoriesˆand quality scoreˆfor each temporal location (omitting subscript standing for temporal location for simplicity). For a location with coarse offset prediction which indicates the distance to start and end of the action boundaries, the corresponding proposal can be denoted asˆ= ( −ˆ, +ˆ). Then three features are sampled from pyramid feature at { −ˆ, , +ˆ} via linear interpolation and aggregated by a fully-connected layer. This procedure is illustrated in Fig. 4. Based on the aggregated feature, BEM produces refined boundary offsets (Δˆ, Δˆ), quality scores˜and category scores˜. The final outputs can be obtained by where , , are final action proposal, action category score and location quality score respectively andˆ=ˆ+ˆ.
Training. The loss of REM is formulated as: ℓ =l +l +l +l +l +l , where , are loss weight.l andl are focal loss [13] for category prediction.l andl are loss of quality prediction, which is implemented by binary cross entropy loss. tIoU between proposal and corresponding ground-truth is adopted as target of quality prediction:l where is ground-truth for location .l is generalized IoU loss [23] for location prediction of initial proposal andl is L1 loss for offset prediction of the refining stage: where N indicates the ground-truth action locations, and Δ = 2 · ( −ˆ)/ˆ,ˆis coarse proposal length.

Training and Inference
Training details. Since there are mainly two different strategies for video feature extraction, including online feature extraction [12,31] and offline feature extraction [14,15,19], we adopt different training methods for them. For frameworks using the online feature extractor, BEM and REM are trained jointly with the feature extractor in an end-to-end way. The total train loss function is where is used to balance loss. As for methods with the offline feature extractor, since BEM is independent of other branches, we individually train BEM and other branches, then combine them in the inference phase for better performance.
Inference. The final outputs of REM is calculated by Eq. 8. Thus, the generated proposals can be denoted as {( , , ) } =1 , where = ( , ) and is the number of proposals. In order to obtain boundary quality, we define a function that generates index of appropriate anchor scale in multi-scale boundary quality map according to the action duration, denoted as ( ). We adopt a simple linear mapping: where is a predefined mapping coefficient. For a proposal, controls the anchor size used by it. We explore the influence of in our ablation. Then start and end boundary quality are acquired by bilinear interpolation, where is bilinear interpolation and = − is the length of proposal. After fusing these scores, the final proposals is denoted as {( · · √ , · , , )} =1 .

EXPERIMENTS
Dataset. The experiments are conducted on two popularly used datasets, THUMOS14 [10] and ActivityNet-1.3 [9]. THUMOS14 contains 200 untrimmed videos in the validation set and 212 untrimmed videos in the testing set with 20 categories. Following previous works [14,15,37], we train our models on the validation set and report the results on the testing set. ActivityNet-1.3 contains 19,994 videos of 200 classes with about 850 video hours. The dataset is split into three subsets, about 50% for training, and 25% for validation and testing. Following [14,15,32], the training set is used to train the models, and results are reported on the validate set.
Implementation Details. For THUMOS14 dataset, we sample 10 frames per second (fps) and resize the spatial size to 96 × 96. Same as the previous works [12,14], sliding windows are used to generate video clips. Since nearly 98% action instances are less than 25.6 seconds in the dataset, the windows size is set to 256. The sliding windows have a stride of 30 frames in training and 128 frames in testing. The feature extractor is I3D [5] pre-trained in Kinetics. The mean Average Precision (mAP) is used to evaluate performance. The tIoU thresholds of [0.3 : 0.1 : 0.7] are considered for mAP and average mAP. If not noted specifically, we use Adam as optimizer with the weight decay of 10 −3 . The batch size is set to 8 and the learning rate is 8 × 10 −4 . As for loss weight, , , are set to 5, 1 and 0.5. The anchor scale and mapping coefficient in BEM are {1, 50, 20} and 2. In the testing phase, the outputs of RGB and Flow are averaged. The tIoU threshold of Soft-NMS is set as 0.5.
On ActivityNet-1.3, each video is encoded to 768 frames in temporal length and resized to 96 × 96 spatial resolution. I3D backbone is pre-trained in Kinetics. mAP with tIoU thresholds {0.5, 0.75, 0.95} and average mAP with tIoU thresholds [0.5 : 0.05 : 0.95] are adopted. Optimizer is Adam with weight decay of 10 −4 . Batch size is 1 and learning rate is 10 −5 for feature extractor and 10 −4 for other components. As for loss weight, , , are set to 5, 1 and 1 repestively. The anchor scale and mapping coefficient in BEM are {1, 130, 22} and 2. The tIoU threshold of Soft-NMS is set to 0.85. In order to validate the generalizability of our method, we also evaluate the performance when integrating BREM with methods using the offline feature extractor. ActionFormer [35] is the latest anchor-free TAD method that shows strong performance. Thus we integrate BREM with ActionFormer to validate the effectiveness of BREM. The implementation details are shown in our supplement.
The results on the testing set of THUMOS14 are shown in Tab. 2. Our baseline achieves 53.1% AP@Avg outperforming most of the previous methods. Based on the strong baseline, BREM absolutely improves 3.6% from 53.1% to 56.7% on AP@Avg. It can be seen that the proposed BREM acquires improvement on each tIoU threshold compared with the baseline. Especially on high tIoU thresholds, BREM achieves an improvement of 5.4% on AP@0.7. Similarly, integrating BREM with ActionFormer [35] provides a performance gain of 1.3% on AP@0.5 and yields a new state-of-the-art performance of 63.6% on AP@Avg.  The results on ActivityNet-1.3 validation set are shown in Tab. 3. Integrating BREM with baseline (Base) reaches an average AP of 34.3%, which is 0.7% higher than baseline. And BREM achieves an average AP of 36.2% when combined with ActionFormer (AF) using the pre-training method from TSP [1], which is the best result using the features from [29]. It is worthy to note that BREM brings considerable improvement on middle tIoU thresholds, outperforming ActionFormer by 1.6% on AP@0.75. TCANet [19] is the only model better than ours, but it uses the stronger SlowFast feature [7] and refines proposals generated by a strong proposal generation method [14].

Ablation Study
We conduct ablation experiments on THUMOS14 for the RGB model based on the baseline to validate the effectiveness of our method. The AP at tIoU=0.5, 0.6 and 0.7, and average AP in [0.3 : 0.1 : 0.7] are reported. Each experiment is repeated three times and the average result is presented to obtain more convincing results.
Effectiveness of Model Components. In order to analyze the effectiveness of the proposed BEM and REM, each component is applied in the baseline model gradually. Meanwhile, the result of the combination of BEM and REM is also presented to demonstrate they are complementary to each other. All results are shown in Tab. 4. Obviously, BEM boosts the average AP by 2.2%. The significant improvement brought by BEM confirms that BEM helps to preserve better action proposals based on the more accurate quality score of boundary localization. Meanwhile, REM improves the average AP by 1.2%. This suggests that aligned features are beneficial for refining more accurate boundaries, classification, and quality scores. By combining BEM and REM, the performance is further improved Effectiveness of Boundary Quality. In order to demonstrate the effectiveness of boundary quality, we first analyze its importance by introducing single-scale boundary quality. Then the comparison between single-scale and multi-scale boundary quality is conducted to validate the necessity of introducing more anchor scales. Finally, different settings of boundary anchors are explored. Results are shown in Tab. 5. For single-scale boundary quality with anchor size=4, the AP@Avg drops from 44.2% to 42.6%. We conjecture that the reason is that the estimated boundary quality at the most temporal locations can not reflect the actual location quality because of the small anchor size (see Fig. 3 Small scale). Increasing the anchor size boosts the performance. The best result is reached with anchor size=28, and further increasing the anchor size harms the performance. For multi-scale boundary quality, we gradually increase the largest anchor size ( ). As shown in Tab. 5, increasing improves the performance, and saturation is reached when = 50 because there are few long actions in the dataset thus too large anchors are rarely used. The above results suggest that our single-scale boundary quality can help preserve better predictions in NMS, but a suitable anchor size has to be carefully chosen. Contrary to single-scale boundary quality, multi-scale boundary quality introduces further improvement by dividing actions into different appropriate anchor scales depending on their duration. It can be seen that the anchor size of {1, 50, 20} brings a 1% improvement compared with single-scale boundary quality. Furthermore, it is less sensitive to the choice of anchor size.
Effectiveness of REM. Based on the aligned feature, REM refines the location, category score, and localization quality score of each action proposal. We gradually remove each component to show its effectiveness. The results are shown in Tab. 6. Removing offset, quality, and classification drop the performance by 0.8%, 0.1%, and 0.5% respectively. Refinement of location and category score bring more noticeable improvement to the model than quality score. We preserve quality score refinement in our final model since it can stable the performance and only increases negligible computation. Previous work [12] extracts salient boundary feature by boundary max pooling, while we extract the region feature of the proposal by interpolation which is more efficient and shows competitive performance.
Ablation study on regional feature extraction method in REM. We explore different feature extraction methods in REM, 1) FC: all sampled features in each anchor region are concatenated and a fully connected layer is applied to convert them to the target dimension. 2) Mean: the mean operation is applied to all sampled  Ablation study on mapping coefficient in BEM. The mapping coefficient in BEM controls the corresponding anchor size of the proposal in the inference phase (See Eq. 13). For a proposal, it will use a smaller scale anchor if enlarging . We vary the mapping coefficient ∈ {0.5, 1.0, 2.0, 3.0} in the inference phase and report the results in Tab. 8. The performance is stable if equals to 1.0 or 2.0. Smaller and larger will decrease the performance since the anchor size and the duration of action are not appropriately matched, which also confirms the importance of multi-scale boundary quality.

CONCLUSION
In this paper, we reveal the issue of misalignment between localization accuracy and classification score of current TAD methods. To address this, we propose Boundary Evaluate Module and Region Evaluate Module (BREM), which is generic and plug-and-play. In particular, BEM estimates the more reliable proposal quality score by predicting multi-scale boundary quality in a moment perspective. Meanwhile, REM samples region features in action proposals to further refine the action location and quality score in a region perspective. Extensive experiments are conducted on two challenging datasets. Benefiting from the great complementarity of moment and region perspective, BREM achieves state-of-the-art results on THUMOS14 and competitive results on ActivityNet-1.3.

A EXPERIMENTAL DETAIL
The proposed Boundary Evaluate Module and Region Evaluate Module (BREM) is generic, and it is easily combined with other anchor-free frameworks to achieve better results. Fig. 2 illustrates the overall architecture of anchor free methods combined with BREM, where BEM is integrated as an extra component and REM is adopted to replace the original detection head. In our experiments, BREM is combined with two typical frameworks, a basic anchor-free framework (denoted as Base) and ActionFormer [35]. The implementation details are described in this section.

A.1 The architecture of Boundary Evaluate Module
The implementation of Boundary Evaluate Module (BEM) is shown in Tab 9. The input feature of BEM is frame level feature ∈ R × with the time resolution same as the input of backbone, which preserves more detail information of appearance and motion. In the experiments, the number of sample points is set to 14 for the balance of efficiency and effectiveness.
Finally, the refined action prediction is obtained via Eq. 8 in the main paper.

A.3 Combination with Base
Detailed architecture. We adopt a previous successful feature pyramid proposed by AFSD [12] and its architecture is shown in Fig. 6. I3D [5] is used to extract the semantic feature of videos and the feature of 4 ℎ and 5 ℎ stage (C4 and C5) are used to generate a feature pyramid. Meanwhile, an up-sample layer and a convolutional layer are used to produce the frame-level feature. Finally, pyramid features and frame level feature are fed into REM and BEM, respectively.     , and the channel of all convolutional layers is 512. ⊕ denotes element-wise summation.

A.4 Combination with ActionFormer
Detailed architecture. Contrary to Base, ActionFormer [35] uses off-the-shelf features. Frame level feature is generated by applying a 1 with kernel size=3 and stride=1 on pre-encoded video features. Then is fed into BEM and the original head of ActionFormer is replaced with REM.