Object Detection and Classification for Autonomous Vehicle

The proposed work presents efficient objection detection and tracking algorithm for autonomous vehicles, which is developed in MATLAB with Image Processing Computer Version Toolbox and Automated Driving Toolbox. The developed algorithm track and detects moving and stationary objects such as other vehicles, pedestrians, and traffic lanes. Accurate and efficient tracking are important to analyze object behavior. For this work various build in detectors from MATLAB tool box were tested and compared. The evaluation of algorithm was carried on for 19 short videos from 8 seconds to 23 seconds, and then it applied to K-dataset and full road experiments by the Automated Driving Lab. The full road tests are between one to five minutes, including CAR to CAR WEST, CAR WEST to CAR, and campus marked roads.


Introduction
Real-time object detection, identification and tracking is a major challenge in developing computer vision algorithms. Accurate identification and recognition of objects in one or two image frames is critical for Advanced Driver Assistance Systems (ADAS) that are integrated in vehicles [1][2][3][4]. ADAS algorithms aid driver to identify and recognize objects which otherwise would be compromised due to lack of attention, anonymous objects, reflection of light etc. Inefficient algorithm design compromises safety and may result in accidents especially at high speed or under heavy traffic conditions. To overcome the said problem ADAS for autonomous vehicles require unique classification and detection of object under different scenarios [5]. Other applications of object detection are advanced robotics, surveillance systems, defense systems, face recognition and space research etc. [4] discusses the typical steps involved in object classification and identification algorithms for computer vision are -1) Classification of objects in image, 2) Localization of objects in image, 3) Object detection in image and 4) Segmentation of image.
In addition to the above steps, ADAS in self-driving vehicles use advance sensor fusion techniques for detecting and extracting characteristics from moving or stationary objects [6]. There are many object detection algorithms have been proposed that are based on fuzzy, statistics, neural networks etc. Many of these algorithms have complex theory and require deeper understanding, implementation and experimentation before incorporating. Real-time images and videos of various objects and structures are bundled as datasets that provide information of the environment which is used for both training and testing. Using this information distance and direction of the object relative to the test vehicle is measured. Detection and reaction are achieved by combining extraction model with response model in the proposed algorithm. Counting is another important aspect of the algorithm, that involves estimating real-time the number of objects in the image. Once the objects are detected, classified, located and counted from the image, the object is tracked from the extracted features by the proposed algorithm. To perform all the afore mentioned tasks in real-time, it requires high performing hardware. In this paper image segmentation is employed to track without any errors using MATLAB automated tool box. The following section of the paper discusses object detection algorithm, road test results and conclusion.

Camera Calibration
Camera calibration estimates camera intrinsic and extrinsic parameters. Intrinsic parameters are including focal length, principal point and image size. Extrinsic parameters are including pitch, yaw and roll, which can also write as a rotation matrix R and translation vector t. Intrinsic parameters are to determine the relation between camera coordinate and pixel coordinate. Extrinsic parameters are to determine the relation between camera image coordinate and real-world coordinate. As figure 1 shows, in the full road tests nineteen figures have been collected with various angles and distances. Angles have to be less than forty-five degree [7]. Camera extrinsic parameters during all experiments are relative to sensor mounting location at the vehicle, which are mounting height, mounting pitch, mounting roll and mounting yaw [8], as figure 2 (a). Mounting height can be directly measured relative to ground, but other angles need to turn manually according to the bird's eye view image. Figure 2 (b) and (c) indicates extrinsic parameters affections on bird's eye view. Road and lanes in bird's eye view should be parallel after successful turned with correct mounting angles. Although Camera Calibrator App can export camera extrinsic parameters, that is not same extrinsic parameters expected. As figure 3 shows, extrinsic parameters from the app are mapping matrix between image coordinate and camera coordinate. However, the expected extrinsic parameters are translation matrix between vehicle coordinate and camera coordinate as figure 2 shows.

Lane Boundaries Detection and Tracking
The algorithm follows a visual perception example by MATLAB document. After a physical defined region of interests, possible lane width and detection sensitivity, a built-in function Segment Lane Marker Ridge to detect lane points at grayscale bird's eye view image, in figure 2 (d) shows [9]. For each parabolic model includes 3 coefficients as a second-degree polynomial equation. In addition, that built-in function can also provide strengths of lanes, boundary type, x maximum position and x minimum position. Weak strength points, such as vehicle shadows and windscreen reflection, need to remove. Lane types as solids, dash etc. could reflect real road conditions. Lane length calculated by subtracting of x maximum and minimum points. After compared with a threshold length defined by a percentage length of maximum lane length in RoI, short length boundaries, such as zebra crossing, need to remove. In the last step, lane vehicle coordinates points would transfer to video frame image coordinate points by vehicle To Image function. Figure 4 shows the step to detect points in bird's eye view and project them to frame.

Vehicle and Pedestrian Detection
Automated driving toolbox and computer vision toolbox provide various pre-trained detectors, including Aggregate Channel Feature (ACF), Gaussian Mixture Model (GMM), Region-based Convolutional Neural Networks(R-CNN), Fast R-CNN and Faster R-CNN [11][12]. The last three detectors are deep learning methods, which need more processing time compared with others in the test. GMM detector is not accurate compared with ACF detectors in nineteen short traffic videos experiments by Automated Driving Lab. Therefore, ACF is the main detector in this work. Although ACF detector has better work compared with other pre-trained detectors, it still has problems [13][14]. It is not always consistent tracking and some frame may miss label. Ground Truth Labeler app provides temporal interpolator algorithm, which can estimate label in the intermediate frame by interpolation. Another main problem is the rectangle label is not consistent with vehicle size, and the size may vary in different frames. Ground Truth Labeler app would also solve the problem, but it may need volume manual work.

Testing and Verification
The tracking algorithm proposed for vehicles and pedestrians is based on Kalman filter which in turn is built on constant velocity model [15]. The pedestrian detection system is a critical part of the system and mandates accuracy and reliability with real-time performance. The pedestrian detection system was tested and verifies taking samples of real traffic scenarios. Although the system provided correct results, it requires tuning and adaptive learning which can consume resources. Therefore, a simulation evaluation tool was used to develop the algorithm to test for real world situations and evaluate the response. Through this algorithm the pedestrian accident data was first analyzed against the other conventional algorithm to find and record its statistical distribution characteristics.

Road Test Results
The algorithm applied to these four tests does not show a great result. It has two main limitations. The first one is when lane width changes or separation, lane points would beyond pre-defined RoI. In complex road conditions, constant RoI will decrease detection accuracies and missing detection. Therefore, the lane detection algorithm has better to detect in constant road condition without lane width changes or division. Another limitation for lane detection algorithm is at the intersection. Since it has a gap but does not reach the threshold, which is not enough strength or length, the detection lanes are not accurate neither able to remove. The detection points could include other direction lanes and all other shadows by vehicles. Theoretically, ground truth labeler app can remove inaccurate points in the frame. However, due to a volume of sampling points, physics work increases by selecting inaccurate points.
Besides hand-operated work by the app, single detection missing frame could update with last tracking points since each frame only takes 0.03 seconds. For a certain period, detection-missing frames, manual work need by ground truth labeler app. To reduce ground truth app processing time, only missing frames import to the app to adding detections. The camera frame rate is 30 frames per second so the frame number calculated as follows

Vehicle and pedestrian detection and tracking results and conclusion
The algorithm applied to these four tests does not show a great result. It has two main limitations. The one of important limitation is the windy and snow shower weather in testing day, as figure 5 shows. Image qualities are not satisfied as the previous test. The detector is only able to detect objects which close to the testing vehicle. Increasing threshold distance and sensitivity is unable to increase detection abilities. In addition, some frame was missing detection due to low image qualities. Therefore, mass manual work needed by ground truth labeler app.

Figure 5 Tracking with test video -Detection in heavy traffic
Another limitation is by ACF detector. ACF detector of vehicles and pedestrians may have inaccurate tracking since it defined as object size. Moreover, the detection rectangle size is not constant, and with small shakings. However, that will not affect object location calculation. ACF may miss detection in heavy traffic condition. Ground truth labeler app would help to manually add detections. Example as figure 5 shows. Overall, considering weather condition, the accuracy of vehicle and pedestrians' detection at a short range is great, as figure 6 shows.  Figure 6. Tracking with test video -Detection in heavy traffic

Tracklet labeler
Tracklet labeler is created based on a timetable is shown in Table 1. For vehicles, pedestrians and stop signs which using rectangle label tracking, have an M x 4 matrix at a specific time. M presents the number of objects at the specific time and four presents image points of tracking rectangle in x, y, width and height. For Lane Marker, it would be 3 element coefficients to present a parabolic lane. Ground truth labeler app would export a group of image points for lanes. Since all these points are based on image coordinate, sensor calibration results are needed to transfer into vehicle coordinate when objects position needed and figure 8 shows output of object detection and line detection.

Conclusion
The proposed algorithm developed in MATLAB detects and tracks objects accurately on test videos for traffic lanes, other vehicles and pedestrians. The algorithm was verified for 19 short videos from Automated Driving Lab, including large curve, ramp and busy traffic scenarios. The tracking of vehicles, pedestrians and lane shows great result compared to ACF detector. Furthermore, the algorithm was applied to Vision Sensor Dataset and Full Road Test with longer than one minute. However, under dim lighting and extreme busy road conditions algorithm showed some limitations in object detection and tracking. This necessitates applying a sensor fusion model which integrates multiple sensors such as vision, radar, and IMU. This model would make the forward collision warning system of the vehicle more effective.