Automating the navigation of unmanned aerial vehicles (UAVs) in diverse scenarios has gained much attention in recent years. However, teaching UAVs to fly in challenging environments remains an unsolved problem, mainly due to the lack of training data. In this paper, we train a deep neural network to predict UAV controls from raw image data for the task of autonomous UAV racing in a photo-realistic simulation. Training is done through imitation learning with data augmentation to allow for the correction of navigation mistakes. Extensive experiments demonstrate that our trained network (when sufficient data augmentation is used) outperforms state-of-the-art methods and flies more consistently than many human pilots. Additionally, we show that our optimized network architecture can run in real-time on embedded hardware, allowing for efficient on-board processing critical for real-world deployment. From a broader perspective, our results underline the importance of extensive data augmentation techniques to improve robustness in end-to-end learning setups.
Moreover, there is another line of work that uses hardware-in-the-loop (HILT) simulation. Examples include JMAVSim @cite_18 @cite_33 which was used to develop and evaluate controllers and RotorS @cite_8 which was used to study visual servoing. The visual quality of most HIL simulators is very basic and far from photo-realistic with the exception of AirSim @cite_13 . While there are multiple established simulators such as Realflight, Flightgear, or XPlane for simulating aerial platforms, they have several limitations. In contrast to Unreal Engine, advanced shading and post-processing settings are not available and the selection of assets and textures is limited. Recent work @cite_40 @cite_19 @cite_48 @cite_10 @cite_9 highlights how modern game engines can be used to generate photo-realistic training datasets and pixel-accurate segmentation masks. The goal of this work is to build an automated UAV flying system (based on imitation learning) that can relatively easily be transitioned from a simulated world to the real one. Therefore, we choose Sim4CV @cite_47 @cite_9 as our simulator, which uses the open source game engine UE4 and provides a full software in-the-loop UAV simulation. The simulator also provides a lot of flexibility in terms of assets, textures, and communication interfaces.
@cite_18 @cite_33 @cite_8 @cite_48 @cite_10 @cite_9 @cite_19 @cite_40 @cite_47 @cite_13
2748378127 2341078590 2518876086 2338554534
We present a photo-realistic training and evaluation simulator (Sim4CV) (http: www.sim4cv.org) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool. In this paper we present a model based on the parallel computational tool of cellular automata CA capable of simulating the process of disembarking in a small airplane seat layout, corresponding to Airbus A320 Boeing 737 layout, in search of ways to make it faster and safer under normal evacuation conditions, as well as emergency scenarios. The proposed model is highly customizable, with the number of exits, the walking speed of passengers, depending on their sex, age and height, and the effects of retrieving and carrying luggage. Additionally, the presence of obstacles in the aisles as well as the emergence of panic being parameters whose values can be varied in order to enlighten the disembarking and emergency evacuation processes are considered in detail. The simulation results were compared to existing aircraft disembarking and evacuation times and indicate the efficacy of the proposed model in investigating and revealing passenger attributes during these processes in all the examined cases. Moreover, we parallelized our code in order to run on a graphics processing unit GPU using the CUDA programming language, speeding up the simulation process. Finally, in order to present a fully dynamical anticipative real-time system helpful for decision-making we implemented the proposed CA model in a field programmable gate array FPGA device, and recreated the results given by the software simulations in a fraction of the time. We then compared and exported the performance results among a sequential software implementation, the implementation running on a GPU, and a hardware implementation, proving the consequent acceleration that results from the parallel CA implementation in specific hardware. In this paper, we propose a new aerial video dataset and benchmark for low altitude UAV target tracking, as well as, a photo-realistic UAV simulator that can be coupled with tracking methods. Our benchmark provides the first evaluation of many state-of-the-art and popular trackers on 123 new and fully annotated HD video sequences captured from a low-altitude aerial perspective. Among the compared trackers, we determine which ones are the most suitable for UAV tracking both in terms of tracking accuracy and run-time. The simulator can be used to evaluate tracking algorithms in real-time scenarios before they are deployed on a UAV “in the field”, as well as, generate synthetic but photo-realistic tracking datasets with automatic ground truth annotations to easily extend existing real-world datasets. Both the benchmark and simulator are made publicly available to the vision community on our website to further research in the area of object tracking from UAVs. (https: ivul.kaust.edu.sa Pages pub-benchmark-simulator-uav.aspx.). The objective of this paper is to develop a vision-based navigation technique for micro aerial vehicles, quadrotor type, to operate in GPS-denied environment. The navigation method has been developed while using appearance-based Visual-Teach-and-Repeat (VT&R) technique. In a teaching phase, a quadrotor is manually navigated along a desired route to collect a set of reference images. In a repeating phase, the quadrotor is able to autonomously follow the desired route using these reference images. Self-localization is developed to determine the current segment of the desired route by a number of Speeded-Up Robust Features (SURF), matched between the current image and the reference images. In this paper, three methods of self-localization are proposed and compared. After performing self-localization, the quadrotor computes appearance-based motion control commands (desired yaw and height) for the next movement in order to keep track of the desired route. This computation is developed on Funnel Lane theory, which was originally proposed in Chen and Birchfield (IEEE Trans. Robot. 25(3), 749---754 (11)) for 2D navigation of a ground vehicle. The paper extends this theory to 3D navigation of a quadrotor.The proposed self-localization methods are tested with several image databases. Finally, an online experiment of proposed VT&R technique is demonstrated using Ar.Drone quadrotor model.