Our work offers a new method for domain translation from semantic label maps
 and Computer Graphic (CG) simulation edge map images to photo-realistic im-
 ages. We train a Generative Adversarial Network (GAN) in a conditional way to
 generate a photo-realistic version of a given CG scene. Existing architectures of
 GANs still lack the photo-realism capabilities needed to train DNNs for computer
 vision tasks, we address this issue by embedding edge maps, and training it in an
 adversarial mode. We also offer an extension to our model that uses our GAN
 architecture to create visually appealing and temporally coherent videos. The topic of image to image translation and more generally video to video translation is of major importance for training autonomous systems. It is beneficial to train an autonomous agent in real environments, but not practical, since enough data cannot be gathered Collins et al. (2018) . However, using simulated scenes for training might lack details since a synthetic image will not be photorealistic and will lack the variability and randomness of real images, causing training to succeed up to a certain point. This gap is also referred to as the reality gap Collins et al. (2018) . By combining a non photo-realistic, simulated model with an available dataset, we can generate diverse scenes containing numerous types of objects, lightning conditions, colorization etc. Chen & Koltun (2017) . In this paper, we depict a new approach to generate images from a semantic label map and a flexible Deep Convolution Neural Network (DCNN) we called Deep Neural Edge Detector (DNED) which embed edge maps. we combine embedded edge maps which act as a skeleton with a semantic map as input to our model (fig 2. 1), The model outputs a photo-realistic version of that scene. Using the skeleton by itself will generate images that lack variability as it restricts the representation to that specific skeleton itself. Instead, we learn to represent skeletons by a neural network and at test time, we sample the closest appropriate skeleton the network has seen at training. Moreover, we have extended this idea to generate photo-realistic videos (i.e. sequence of images) with a novel loss that uses the optical flow algorithm for pixel coherency between consecutive images. Figure 1: in this paper we propose a method for generating photo-realistic images from semantic labels of a simulator scene. This figure provides images related to the Synthia dataset Ros et al. (2016) . Left -semantic map of the scene. Middle -generated image from pix2pixHD Wang et al. (2018b) . Right -Our generated image. The texture and color space in our generated image is more natural giving the image the desired photo-realism. used as a super resolution generator. L1 loss for image generation is known to generate low quality images as the generated images are blurred and lack details Dosovitskiy & Brox (2016) . Instead, Gatys et al. (2016) , Johnson et al. (2016) are using a modified version of the perceptual loss, allowing generation of finer details in an image. Pix2pixHD Wang et al. (2018b) and CRN Chen & Koltun (2017) are using a perceptual loss as well for training their networks, e.g. VGGnet Simonyan & Zisserman (2014) . Moreover, pix2pixHD are using instance maps as well as label maps to enable the generator to separate several objects of the same semantics. This is of high importance when synthesizing images having many instances of the same semantics in a single frame. As for video generation the loss used by Wang et al. (2018a) , Shahar et al. (2011) tend to be computationally expensive while our approach is simpler. We are using two generators of the same architecture, and they are mutually trained using our new optical flow based loss that is fed by dense optical flow estimation. Our evaluation method is FID Heusel et al. (2017) and FVD Unterthiner et al. (2018) as it is a common metric being used for image and video generation schemes. We call this work s-Flow GAN since we embed Spatial information obtained from dense optical flow in a neural network as a prior for image generation and flow maps for video coherency. This optical flow is available since the simulated image is accessible at test time in the case of CG2real scheme. We make Three major contributions: First, our model can generate visually appealing photorealistic images from semantic maps having high definition details. Second, we incorporate a neural network to embed edge maps, thus allowing generation of diverse versions of the same scenes. Third, we offer a new loss function for generating natural looking videos using the above mentioned image generation scheme. please refer to this link for videos and comparison to related work.
Simulation to real images translation and video generation
{"src": "scitldr-AIC_train", "paper_id": "BJxqohNFPB"}