Temporal action detection mainly targets activities of high-level semantics and videos with a sparse set of actions (e.g., actions only cover \(30\%\)  of the frames in [1]}). However, in some cases, the goal is to predict action labels at every frame of the video. In such cases, the task is referred to as temporal action segmentation which targets the fine-grained actions and videos with dense occurrence of actions (\(93\%\)  of the frames in [2]}). One can convert between a given segmentation and a set of detected instances in the temporal domain by simply adding or removing temporal background segments [3]}. Temporal action detection similar to object detection belongs to the family of detection problems. Both of these problems aim to localize the instances of interest, i.e., action intervals in temporal domain versus object bounding boxes in spatial domain, Fig. REF  (a and c). When targeting fine-grained actions, temporal action detection (segmentation) is similar to semantic segmentation as both aim to classify every single instance, i.e., frames in temporal domain versus pixels in spatial domain, Fig. REF  (b and d). As a result, many techniques for temporal action detection and segmentation are inspired by the advancements in object detection and semantic segmentation [4]}, [5]}, [6]}.
