RAIST: Learning Risk Aware Traffic Interactions via Spatio-Temporal Graph Convolutional Networks

A key aspect of driving a road vehicle is to interact with the other road users, assess their intentions and make risk-aware tactical decisions. An intuitive approach of enabling an intelligent automated driving system would be to incorporate some aspects of the human driving behavior. To this end, we propose a novel driving framework for egocentric views, which is based on spatio-temporal traffic graphs. The traffic graphs not only model the spatial interactions amongst the road users, but also their individual intentions through temporally associated message passing. We leverage spatio-temporal graph convolutional network (ST-GCN) to train the graph edges. These edges are formulated using parameterized functions of 3D positions and scene-aware appearance features of road agents. Along with tactical behavior prediction, it is crucial to evaluate the risk assessing ability of the proposed framework. We claim that our framework learns risk aware representations by improving on the task of risk object identification, especially in identifying objects with vulnerable interactions like pedestrians and cyclists.


Introduction
Over the years, consistent research efforts towards autonomous driving led to significant developments in perception, planning, and control. A major challenge for autonomous vehicles is doing safe planning and driving decisions in urban traffic. In order to do so, we need to build robust methods for accurately and efficiently understanding traffic like human drivers and learn to analyze complex traffic scenes to make tactical driving decisions like slowing down, changing lanes, or taking turns. In moving traffic, making these decisions recklessly can cause road accidents and fatalities. To avoid such events, learning to identify potential risks in driving scenarios is critical. While visual cues are often the most useful source of perception and holistic scene understanding, utilizing this information to assess the behavior and interaction of the individual road users is non-trivial.
For urban driving scene understanding, we need to anticipate the behavior of dynamic and heterogeneous road agents in the traffic, including the vulnerable (pedestrians and cyclists) and other motorized not-so-vulnerable road agents, both spatially and temporally. Recent works [31,55] have shown the promise of modeling spatial interactions amongst the road users using interaction-graphs. Interaction graphs are designed with the notion that every road user influences every other road user at a particular instant. However, current methods fail to temporally model the behavior of individual road agents. Anticipating the behavior of specific road agents is very crucial for humans to drive safely, and hence, this can be critical to autonomous vehicles for planning the navigation in the impending traffic. For example, temporal intention modeling can help the vehicle make critical distinctions between a scenario with a pedestrian waiting to cross and another with a pedestrian walking parallel to the road. Another example where temporal modeling can be useful is to anticipate the intention of an approaching vehicle in a road intersection, i.e., in what direction is the vehicle going to continue moving?
Besides making the driving decisions, a better understanding of the scene, including road agents' intention modeling, can also be leveraged to assess the risk involved. Deep learning models are often uninterpretable, failing in surprising circumstances. An auxiliary goal of the risk assessment will increase the interpretability towards predicting driver-centric tactical behavior. The risk assessment literature [29] extensively covers the topic of potential risk identification. However, current methods like estimating object importance [21,55] require carefully annotated training labels. Another plausible direction is to identify the risk objects that causally influence the ego-behavior [30]. While proposing a label-efficient causal inference method for identifying risk objects, it lacks interaction modeling and performs poorly, especially when the risk objects are pedestrians.
Main Contributions: We present a novel spatiotemporal learning framework for causal risk assessment Figure 1: An overview of the proposed driving model. The model takes a sequence of video frames as input, applies 3D convolutions (I3D backbone) to extract visual features, employs ROIAlign to extract appearance features. Depth estimation and inpainting modules are used to obtain object-level 3D positions and inpainted frames with the background scenes respectively. A static scene encoder is used to encode the inpainted images to obtain the context which is further combined with the object-level appearance features. The modified features along with the object-level 3D positions are used to construct frame-wise traffic interaction graph learnt via spatio-temporal graph convolutional networks.
from egocentric videos. We show that the proposed framework can generalize to multiple tasks that can benefit from interactive driving scene understanding. We use a combination of detection, tracking, depth estimation, and action recognition to obtain the attributes of road agents, used to formulate the edges of the spatio-temporal graphs. These spatio-temporal graphs are then used to compute a risk metric. The contributions are summarized below: • We use the object attributes to model interaction edges amongst different object nodes at a particular instant. Each object node is also associated with its own representations across multiple time-steps via temporal message passing. We parameterize the interaction edges and temporal message passing with respect to the vulnerability of the participating road agents.
• We use inpainting to remove the dynamic road agents from the input image frames followed by feature extraction, in order to generate a frame-wise static scene prior. This scene prior is combined with each object's appearance features for context awareness, which is eventually used to model the graph edges.
• We aim to validate on the tactical ego-behavior recognition task using HRI Driving Dataset (HDD) [39], evaluating on both the goal-oriented and the cause benchmarks. We also plan to evaluate this proposed approach on the task of risk object identification in a label efficient manner using causal inference [30]. We detail the advantages of using this framework to solve the causal problem in Section 4.

Driving Scene Understanding
There have been significant efforts in driver behavior understanding and tactical behavior recognition [36], [32], [28], [50]. In many classical approaches, hidden Markov models (HMM) were leveraged to recognize driver behaviors. A single node in HMM encodes the states from the ego vehicle, roads, and traffic participants into a state vector. In the proposed framework, we explicitly want to model two of these three states in the form of interactions between the ego and the other traffic agents. The third state of the roads is implicitly modeled using object-level context-aware features obtained from the static scene encoding. In the deep learning era, interesting methods like [39], [47], [31], [35] have been proposed to this end. Traffic trajectory prediction [9,13,14] and tracking [8,[10][11][12], have been widely studied robotics and computer vision, whereas driver behavior modeling has mostly been restricted to traffic psychology and the social sciences [1-4, 7, 15-17, 19, 20, 22, 25, 33, 34, 34, 38, 42, 45, 48].

Graph Convolutional Networks for Driving Scenes
Among other deep learning methods, graph convolutional networks [27] have made significant progress in tasks like action recognition [52], group activity recognition [51], scene graph generation [53], where the need is to learn the underlying semantic structure in the input videos.
For autonomous driving applications, very recent meth-ods like [31], [35], [55] have been proposed that use graph convolutions to learn the spatial semantics of ego-centric driving scenarios. However, no work has focused on the object-level temporal intention modeling using message passing via temporally associated nodes. We intend to use spatio-temporal GCNs [52] to achieve this.

Risk Assessment
It is natural for humans to assess risk before making decisions. This survey [29] presents some existing classical methods for motion prediction and risk assessment in the context of intelligent vehicles. It defines risk as to the probability that the expectation and intention do not match any given traffic scenario. A similar definition of risk as being the object influencing the driver behavior was proposed in [30], and we intend to evaluate our framework's risk awareness using this definition. This task of risk object identification closely touches the boundaries of the task of object importance estimation [21], [55] where the goal is to detect semantically important objects in the ego-centric driving scenes. However, none of these past methods explicitly parameterize for the vulnerability [18] of the traffic participants with respect to the ego like we intend to do.

Methodology
This section describes our risk-aware driving scene understanding algorithm, the feature extraction pipeline, and the spatio-temporal graph-based approach.

Problem Statement
We propose a framework to predict tactical driver behavior and causal reasoning from driver-centric traffic for causal risk assessment from egocentric videos. In this context, we define tactical driver behavior as the high-level driving decision resulting from a cause or the action of one or more participating dynamic road agents. For example, a tactical behavior can be changing lane, and the corresponding cause can be a parked vehicle. In practice, we tackle this as a classification problem with a fixed set of tactical behaviors and the corresponding cause.
Our framework exploits the underlying structure of a traffic scenario, including heterogeneity in traffic agents, by extracting the semantic features of traffic agents and using them to learn pair-wise spatial interactions that evolve temporally, hence learning a more generalizable and interpretable model.

Feature Extraction
Here we elaborate on the multiple feature extraction steps based on the semantics of the static and dynamic traffic scenarios.
Tracking and Location: With a sequence of Γ consecutive RGB frames I 1:Γ , we use Faster R-CNN [41] to de-tect road agents in every frame. In order to identify each unique road agent across time, we then associate the obtained detections using Deep SORT [49]. This gives us N unique road agents, in terms of tracklets, where each tracklet r i stores the presence or absence of a road agent across the Γ time steps. Perspective projection makes it difficult to directly use the 2D pixel positions of objects to model spatial relations amongst them. To overcome this, we perform depth estimation [40] followed by inverse projection to transform every object's 2D pixel position to locate it in a 3D space with respect to the camera frame.
where (u, v) and (x, y, z) represents a point in 2D and 3D coordinate systems respectively. K is the camera intrinsic matrix and δ u,v is the relative depth at (u, v) obtained by depth estimation [40]. In the image plane, we select the center of a bounding box as that object's 2D position.
Appearance: To perform spatio-temporal reasoning on the road agents, we need rich features that are able to encode their inherent characteristics across all Γ time steps from the input I 1:Γ . For this, we use Inception V1 [43] and I3D [5] as the global video descriptors and apply ROIAlign [23] over the 3D video feature map to obtain object-level appearance features from the corresponding regions of interest.
Static Scene Encoding: We believe that the appearance of road agents should also be associated to their static environment WHY?. To this end, we formulate a context extraction module in our framework. With I 1:Γ as the input, this module performs inpainting [54] to generate S 1:Γ image frames free from the dynamic agents of the scene. Following the inpainting, we use ResNet-18 [24] to encode S 1:Γ into frame-wise context embeddings c 1:Γ . We update each object's appearance feature a t for time step t, Vulnerability: Vulnerable road agents are those that have little or no external protection and are at most risk in traffic [18]. In our design, we consider detected objects of classes person, bicycle, car, motorcycle, bus, truck as road agents, categorizing person, bicycle as vulnerable (m = 1) and the rest as non-vulnerable (m = 0). We use this categorization to learn separate set of parameters for vulnerable road agents in the relation modeling.
In summary, we obtain N road agents and their framewise attributes from the input I 1:Γ . For each road agent, we have (a) its tracklet r i , and if it exists in frame I t , we have its (b) category label m t i , (c) spatial position p t i in 3D, and (d) scene-aware appearance feature a t i . To perform spatiotemporal learning on graphs, we use the mentioned three attributes of road users.

Spatio-Temporal Modeling
The extracted features of road agents are used to create spatial interaction graphs with parameterized edges. We use spatio-temporal graph convolutions to learn the desired representations towards traffic scene understanding. We discuss this in detail here.
Graph Definition: In correspondence with I 1:Γ , let G 1:Γ be the sequence of adjacency matrices, each of which can be realized as a graph with weighted edges between all existing nodes at that time step. Such a spatial graph G t can be interpreted as a traffic scene I t , where element G t (i, j) denotes the influence of agent v t i over agent v t j . In any spatial graph G t , each node v t i can be represented as a tuple of its attributes {(a t i , p t i , m t i )|i = 1, 2, . . . , K + 1}, assuming there are K road agents excluding the ego, a non-vulnerable road agent that is spatially located at the origin in the camera frame and whose context-aware appearance feature is based on the global output feature from the video encoder I3D.
Spatial Interaction Modeling: We believe that the interaction between any two road agents should be unique and depend on their respective attributes. To this end, we formulate the pair-wise spatial interactions using appearance and positional relations based on [51].
We represent the interaction value between nodes v t i and v t j with an entry in the adjacency matrix: indicates the vulnerability-dependent positional relation and f a (a t i , a t j ) indicates the contextaware appearance relation. For every node, its edges are normalized, resembling a weighted softmax function. The influence of v t i over v t j is represented as a fraction of the total influence of v t i . The adjacency matrix G t also includes self connections for self-attentional learning.
In order to restrict the interaction edges to objects that are spatially closer in traffic, we use the distance constraint as: where I(.) is the indicator function, d(p t i , p t j ) is the Euclidean 3D distance between objects v t i and v t j , and µ is the distance threshold that constrains the interaction G t (i, j) to be non-zero only if the spatial distance is lower than this upper bound. We use µ = 3 in our framework.
The appearance relation is based on mutual similarity [46] and calculated as the scaled dot product between two embeddings as: where φ(.) and ω(.) are learnable linear transformations over the appearance features and D is the dimension of the embedding space. The parameters are learned to compute a scalar value for the influence of v t i over v t j in the embedding space. We note that this relation makes the overall spatial interaction directional, as φ(.) and ω(.) are different transformations.
For positional relation encoding, we use a vulnerabilitydependent learnable transformation function θ m t i (p t i ) to embed 3D positions. The pair-wise positional relation can be formulated as: where denotes element-wise sum, and γ(.) is the Gaussian mapping function which encodes the embedded input into a high dimensional representation using sine and cosine of different wavelengths.
where v ∈ R d is the input embedding vector and B ∈ R k×d consists of entries sampled from N (0, σ 2 ), and σ is a hyperparameter. Here, γ(.) is used to embed d-dimensional input vector v into 2k-dimensional vector. Such a Fourier feature mapping is effective in overcoming the spectral bias inherent in learning high-frequency functions in low dimensional domains [44] like the positional relation in our case. This mapping is then transformed into a scalar using trainable W p , followed by ReLU activation for non-linearity. Spatio-Temporal Graph Reasoning: With the interaction graphs G 1:Γ formulated, we use spatio-temporal graph convolutional network (ST-GCN) [52] as the graph learning module over them. ST-GCN follows a similar implementation as GCN [27], which is followed by a convolution in the temporal domain. A layer of ST-GCN takes features as the input, leverages the spatial edges to aggregate the node-wise influence, and then uses spatial and temporal convolutional kernels to update the input features. For the purpose of simplification, we express l-th layer of ST-GCN at a particular time-step t, where G t ∈ R N ×N is the graph adjacency matrix, X l t ∈ R N ×C is l-th layer's feature map, ReLU is the activation function, W a t ∈ R C×C and W b t ∈ R C×C are the weight matrices for transformations in spatial and temporal dimensions respectively. In Equation (9), spatially attended interaction features are aggregated over τ consecutive time steps. There is a residual connection in each layer. In practice, this pair of transformations is implemented via 2D convolutional layers on a stack of frames in parallel, and the temporal span τ is used as the size of the temporally convolving kernel.
In the case of traffic scenarios, road agents keep varying across time. For effective temporal message passing, we need their associativity across Γ. In practice, the spatiotemporal adjacency matrix G 1:Γ ∈ R Γ×N ×N , and all the invalid interactions are zeroed based on the information from the tracklets r 1:N . We use the context-aware appearance feature map stacked temporally as the input to the ST-GCN layers and update them after every layer. Following this, the interaction edges are remodeled with the updated features to be used for the next layer of ST-GCN. For learning the bias towards vulnerable road agents, we assign separate temporal convolutional weights to be shared across vulnerable and non-vulnerable agents.

Risk Assessment
In this section, we discuss the task of risk object identification. Our proposed framework is designed to learn the spatio-temporal interactions in traffic, and we explain how it is an appropriate fit for this task.

Problem Statement
We use the driver-centric definition of risk to tackle the task of risk object identification or detecting the traffic agent that has the most significant influence over ego's tactical behavior for a given scenario [30]. Due to the limited annotations of real-world risk object bounding boxes, we choose to experiment on the cause-effect formulation of this task.

Causal Inference Setup
Given a traffic scenario I • if the Stop prediction confidence is high, P r f I where I [1..N ]\i 1:Γ refers to the manipulated input by excluding tracklet r i from the original input I 1..N 1:Γ , and P r f I [1..N ]\i 1:Γ = Go refers to the risk score s i of agent with tracklet r i .
We note that this is a simplistic formulation to detect risk objects in critical Stop scenarios where some road agent might have a particularly high influence causing the ego's behavior to be Stop. The probability maximizing tracklet r o can be obtained from Equation (12) by performing iterative manipulations for each of the N road agents, one at a time. The risk evaluation is weakly supervised and only requires frame-level Stop/Go annotations for training.

Our Approach
The design of our proposed model allows for trivial manipulation of the traffic scenario with respect to each road agent by masking out its attributes before the graph modeling step. The spatio-temporal graph thus generated should closely represent an actual scenario without the particular agent. This avoids the artifacts that are introduced from masking directly in the input video frames [30]. We show improved results compared to the baseline in the next section.
Towards safe risk assessment, we realize a drawback in the above causal inference setup. Let's consider a scenario with multiple pedestrians crossing in front of the ego, and the original tactical behavior is correctly classified as Stop.
In such a case, unless the crossing pedestrians are removed all at once, the driving model's prediction shouldn't change from Stop to Go, i.e., any pedestrian's individual risk score s i needn't be very high. Hence, we can cherry-pick such scenarios from the dataset and identify the corresponding risk groups based on their high interaction values with respect to the ego node in the spatio-temporal graph G 1:Γ . For some of these scenarios, we perform the causal inference step with a single manipulation for the entire risk group and show the risk scores in this visualization.

Dataset
For the tactical behavior prediction task, we evaluate on the HDD dataset [39]. The dataset provides 104-hour driver-centric videos with frame-level annotations of tactical driver behavior and the associated cause. The dataset consists of diverse road scenarios collected within San Francisco Bay Area, including urban, suburban, and highways.
Each frame is labeled by a 4-layer representation to describe the tactical driver behaviors. Among these 4 layers, Goal-oriented action layer (e.g., left turn and right lane change) and Cause layer (e.g., stop for crossing vehicle) consist of the actions with interactions. We leverage the la- Figure 2: Risk-identification using a two-step inference framework. In this scenario, intervening the input by removing the agent changes the driver's behavior (effect) from Stop to Go, indicating the omitted object is the risk object or the cause for Stop. In our framework, this intervention can be achieved by omitting the desired node from the traffic interaction graph. . bels from these two layers to analyze the effectiveness of the proposed interaction modeling framework. For the causal risk object identification task, we use the same dataset, leveraging the layers of Stimulus-driven action with labels of Stop and deviate, and cause denoting the reason for the associated stimulus-driven behavior. In total, the dataset has scenarios labeled with six types of Cause, i.e., Stopping for Congestion, Stopping for Crossing Vehicle, Deviating for Parked Vehicle, Stopping for Pedestrian, Stopping for Sign, and Stopping for Red Light. The first four scenarios are selected to evaluate the proposed risk object identification framework since our proposed framework is based on the interactions between the dynamic road agents only. We intend to utilize the frame-level driver behavior labels (i.e., Go and Stop) to train our driving model for risk assessment. For evaluating the task of risk identification, we will use the object-level annotations [30] provided. Since we want to minimize false-negative risk identifications, we use a recall based metric for our final evaluation.

Implementation Details
For tactical behavior prediction, we use 20-frame clips with resolution 200 × 356 at 3 fps as the input to the framework. We use the I3D network [6] pretrained on the Kinetics action recognition dataset [26] as the backbone for encoding the global video features. The intermediate video feature map from the Mixed 3c layer of I3D is used in the ROIAlign layer to obtain the appearance features. In our implementation, we use the appearance embedding dimension D = 256. For encoding the positional relations using γ(.), we choose d = 5, k = 30 and follow [44] to choose σ = 10. We stack 3 layers of ST-GCN i.e. the pair of equations (8) and (9) with temporal span τ = 3. The last layer of ST-GCN is followed by the classifier that includes average pooling and fully connected layers to obtain the classification logits, which are further converted into class probabilities using the softmax layer. Each layer of a neural network is followed by a layer of batch normalization. We use Adam as the optimizer with default parameters and an initial learning rate of 0.001. We are implementing the framework on PyTorch [37] and performing all experiments on a node with 4 Nvidia Tesla M40 GPUs.

Future Work and Conclusion
We show some preliminary results from our approach, and the work is currently in progress. We are yet to show detailed evaluations of our proposed framework on the task of tactical driver behavior prediction and causal risk identification. In the future, we want to analyze our network and integrate psychology-derived traffic interaction algorithms into our model.