Inferring temporally coherent data features is crucial for a large variety of learning tasks. We propose a network architecture that introduces temporal recurrent connections for the internal  state of the widely used residual blocks. We demonstrate that, with these connections, convolutional neural networks can more robustly learn stable temporal states that persist between evaluations. We demonstrate their potential for inferring high-quality super-resolution images from low resolution images produced with real-time renderers. This data arises in a wide range of applications,  and is particularly challenging as it contains a strongly aliased signal. Hence, the data differs substantially from the smooth inputs encountered  in natural videos, and existing techniques do not succeed at producing acceptable image quality. We additionally propose a series of careful adjustments of typical generative adversarial architectures for video super-resolution to arrive at a first model that can produce detailed, yet temporally coherent images from an aliased stream of inputs from a real-time renderer. Learning expressive and stable representations is a goal that lies at the heart of a vast range of deep learning tasks (Dahl et al., 2011; Radford et al., 2015; . While typical recurrent architectures focus on feedback loops to form persistent latent-spaces (Rumelhart et al., 1988; Chaitanya et al., 2017) , we show that for inference tasks where the result is conditioned on a stream of inputs, these existing architectures unnecessarily complicate the learning task, and fail to reliably stabilize the inference. With our work, we propose a new type of connection for the very widely used building blocks of ResNet architectures (He et al., 2015) that lets the network easily compare internal states in-place. The learned representation can then, e.g., yield a detailed image sequence with natural changes. We demonstrate this with a particularly challenging learning objective: we aim for the synthesis of detailed images from a stream of strongly aliased inputs. Specifically, we show that adversarially trained convolutional neural networks (CNNs) can be leveraged to produce detailed images from unfiltered, low-resolution images generated via point-sampling with a rasterization-based real-time renderer. Real-time graphics are the basis for a wide range of applications: Generating images with a sufficient resolution from low resolution, yet computationally light-weight renderings is a task that is, e.g., important for generating content for the high resolution screens of mobile devices, and is especially interesting for streaming services of games in order to compute the final resolution only on the client. Our work shares its goal with a variety of approaches that have been proposed for generating highquality images for raytracing algorithms (Zhang et al., 2016; Chaitanya et al., 2017) and purely image-based super-resolution algorithms (Sajjadi et al., 2017; . Our architecture differs from previous works as the proposed recurrent connection allows the network to learn a temporally stable latent-space representation that does not negatively impact the residual flow of a ResNet architecture. Also, the temporal connections for deeper layers of the network are important for successful learning, as we will demonstrate below. While the basic concept of depth-recurrent connections could potentially be applied to a variety of sequence-based learning tasks, we focus on demonstrating its potential for pushing forward the limits of real-time rendering. Hence, we additionally outline a series of modifications to existing architectures which are crucial for achieving high quality of the strongly aliased input images from LR RDA modified TecoGAN re-trained DRR Figure 1 : Given a strongly aliased low-resolution input rendering with one sample per pixel, recurrent non-adversarial training ((Chaitanya et al., 2017) with modifications for fair comparisons) produces blurry results, and existing adversarial methods ( , re-trained) introduce strong flickering artifacts. Trained on the same data, due to the proposed DRR connections our network infers more consistent spatio-temporal features (see the supplemental footage for a clear assessment of the temporal differences). typical real-time rendering pipelines. A typical input for our network is shown on the left of Fig. 1 . This application scenario is especially challenging for CNNs, since it requires to work with images that need to be rendered at very high frame rates and, thus, exhibit severe aliasing due to point sampling and typically low resolutions. The aliasing not only distorts the spatial signal, but likewise affects the temporal changes. Therefore, a super-resolution (SR) network can't rely on receiving smoothly changing, filtered inputs that allow for localization of small features. Rather, it has to learn over the course of multiple frames to infer consistent output images (Fig. 1, right ) from spatially and temporally aliased input content. As we will demonstrate in a number of studies below, this task is where our proposed depth-recurrent connections unfold their strength. They enable the network to match the data distribution of the targets, i.e., to synthesize images with a high visual quality in terms of detail as well as their temporal behavior. We show results and comparisons in the paper, and provide many additional evaluations in the supplemental material 1 , where videos more clearly show spatial and temporal differences. As we focus on real-time rendering as our use case scenario, ideally the performance of the inference step needs to surpass the performance of the renderer. For a desired output resolution of 1920×1080, our pre-trained model takes 113ms per frame on average. 2 Although this is not yet fast enough for real-time applications, we expect that techniques such as network compression (Choi et al., 2018; Molchanov et al., 2016) and evaluation of the models with dedicated hardware (NVIDIA Corporation, 2017) will easily yield very significant performance improvements. We have demonstrated how depth-recurrent residual connections can be leveraged to learn stable internal latent-space representations in conditional generator architectures. The DRR connections are particularly promising for iterative models with strongly aliased data, such as low-resolution inputs from a real-time renderer. We have additionally shown how to achieve high quality synthesis in the context of real-time rendering by carefully analyzing and adjusting the network architecture. We anticipate that DRRs could be beneficial for a variety of other tasks such as object tracking (Ning et al., 2017) and physics predictions (Li et al., 2019) . A DATA As source of our data we use the projects "FPS Sample" and "Book of the Dead: Environment" (Unity Technologies, 2019b;a) for the Unity engine, both use the HDRP. We captured a total of 57 120-frame sequences, split 50-5-2 for training, validation and testing. For each frame we have lit color (the final image), unlit diffuse color, view-space surface normals, roughness, screen-space motion and depth for both HR and LR. This data is easy to acquire as it can be inferred from the scene, geometry and materials and is rendered by default in Unity's HDRP. However, the use of unlit color, normals or roughness had no tangible effects during our tests. Most post-processing effect have been turned off, but the HR color is augmented with TAA. HR is rendered and captured at a resolution of 512 × 512, LR at 128 × 128.
A method for persistent latent states in ResBlocks demonstrated for super-resolution of alised image sequences.
{"src": "scitldr-AIC_train", "paper_id": "H1gW93NKvH"}