SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps

Abstract: We present SeamlessGAN, a method capable of automatically generating tileable texture maps from a single input exemplar. In contrast to most existing methods, focused solely on solving the synthesis problem, our work tackles both problems, synthesis and tileability, simultaneously. Our key idea is to realize that tiling a latent space within a generative network trained using adversarial expansion techniques produces outputs with continuity at the seam intersection that can be then be turned into tileable images by cropping the central area. Since not every value of the latent space is valid to produce high-quality outputs, we leverage the discriminator as a perceptual error metric capable of identifying artifact-free textures during a sampling process. Further, in contrast to previous work on deep texture synthesis, our model is designed and optimized to work with multi-layered texture representations, enabling textures composed of multiple maps such as albedo, normals, etc. We extensively test our design choices for the network architecture, loss function and sampling parameters. We show qualitatively and quantitatively that our approach outperforms previous methods and works for textures of different types.

\section{Introduction}\label{sec:introduction}}
\else
\section{Introduction}
\label{sec:introduction}
\fi

\IEEEPARstart{T}{his} demo file is intended to serve as a ``starter file''
for IEEE Computer Society journal papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later.
I wish you the best of success.

\hfill mds

\hfill August 26, 2015

\subsection{Subsection Heading Here}
Subsection text here.

\subsubsection{Subsubsection Heading Here}
Subsubsection text here.

\section{Conclusion}
The conclusion goes here.

\appendices
\section{Proof of the First Zonklar Equation}
Appendix one text goes here.

\section{}
Appendix two text goes here.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi

The authors would like to thank...

\ifCLASSOPTIONcaptionsoff
  \newpage
\fi

\section{Introduction}
This demo file is intended to serve as a ``starter file''
for IEEE Computer Society conference papers produced under \LaTeX\ using
IEEEtran.cls version 1.8b and later.
I wish you the best of success.

\hfill mds

\hfill August 26, 2015

\subsection{Subsection Heading Here}
Subsection text here.

\subsubsection{Subsubsection Heading Here}
Subsubsection text here.

\section{Conclusion}
The conclusion goes here.

\ifCLASSOPTIONcompsoc

  \section*{Acknowledgments}
\else

  \section*{Acknowledgment}
\fi

The authors would like to thank...

\subsection{Inter-map Consistency}
\label{sec:inter_map_consistency}
Whilst our generators $\mathcal{G}$ can generate high-quality tileable pairs of albedo and normal maps, there is no guarantee that those maps are pixel-wise coherent, as no part in the loss function explicitly accounts for this relationship. The $\mathcal{L}_{style}$ loss is computed separately for each map in the texture stack, which may generate non-coherent gradients. Furthermore, our architecture of choice $\mathcal{G}_2$ separates the decoding of each map in different parts of its architecture, which may hinder the generation of spatially-coherent maps.
Nevertheless, we show that this is not a problem in practice. Figure~\ref{fig:consistency_maps} shows crops of our synthesized texture maps. It can be seen that pixel-wise coherency between maps is preserved even for challenging geometric structures. We argue that the role of the discriminator is key to detect inter-layer inconsistencies by yielding lower probabilities for non-coherent maps. Even if separating the decoders may increase the risk of incoherent texture maps, this coherence is forced by the discriminator during training. To test this empirically, for the textures in Figure~\ref{fig:consistency_maps}, we fed the discriminator with a crop of the original stack, a crop with the normals translated 5 px, and translated 100 px, for which we obtain average values of 0.99, 0.35, and 0.32, respectively.
This suggests that, since during training, the discriminator receives the entire texture stack at once, it learns to identify spatial inconsistencies between maps.

\subsection{Loss Function Ablation Study}
\begin{figure}[t!]

	\includegraphics[width=1\linewidth]{figures/results/loss_function.pdf}

	\caption{Ablation study on the impact of the loss function on the quality of the synthesized textures. We evaluate each network using the same input, marked using a blue box. As shown, training $\mathcal{G}$ using the full loss function yields the best results. Each component is weighted by a $\lambda$, as specified in Section~\ref{sec:implementation}.}
	\vspace{-5mm}
	\label{fig:ablation_loss}
\end{figure}
\label{sec:loss_function_ablation}
As described in Section~\ref{sec:synthesis}, the generator $\mathcal{G}$ is trained to minimize a loss function, which is comprised of adversarial, a perceptual and pixel-wise components. Each of these have different impacts on the output synthesis. We hereby study the impact of each of these components. To better isolate the impact of each metric, we perform this study using a limited generator that only outputs albedo maps. As we show on Figure~\ref{sec:loss_function_ablation}, the adversarial loss $\mathcal{L}_{adv}$ provides high-level semantic consistency, the $\mathcal{L}_1$ norm acts as a regularization method which removes artifacts, while the perceptual loss $\mathcal{L}_{style}$ adds additional  details to fully represent the texture. Similar findings are reported in~\cite{Zhou2018}. None of these loss functions yield compelling or artifact-free results on isolation, being the adversarial loss the most important factor of the global function. Further results are included in the Supplementary Material.

\section{Results and Comparisons}

Most tileable texture synthesis methods have not shown results on synthesizing texture stacks. While adding this capability is reasonably easy for some \textit{non parametric} methods~\cite{li2020inverse,Rodriguez-Pardo2019AutomaticPatterns},  others require major changes in their design. In particular, as we have shown in Section~\ref{sec:networkdesign}, expanding the architecture of deep learning methods for generating more than one map requires special attention to balance the map's interdependence with the network's capacity and expected quality.
Therefore, in this section, in order to be able to compare our method with other works, we use a reduced version of our generator that only outputs a single albedo map.

\label{sec:comparison_other_methods}
\begin{figure}[t!]
	\centering
	\includegraphics[width=1\linewidth]{figures/results/networks/comparison_v6.pdf}
	\caption{A comparison of the results of our proposed architectures.  Separating the decoders of the network for each map ($\mathcal{G}_2$) outperforms the joint-decoder baseline ($\mathcal{G}_1$) for both texture maps in the stacks. It generally allows for more varied albedo maps, with a more correct structure and style; as well as more accurate and sharper normal maps. The outputs in this figure are not tiled, for improved visibility of the artifacts created by $\mathcal{G}_1$. In blue, we show an up-sampled crop of the output textures. We refer the reader to the Supplementary Material for more qualitative comparisons.
	}
	\vspace{-4mm}
	\label{fig:comparison}

\end{figure}

\paragraph*{\textbf{Qualitative Analysis}}
\label{sec:qualitative_study}
First, we compare with the Texture Stationarization algorithm proposed in~\cite{moritz2017texture} using examples taken from their own dataset, as their code is not publicly available.
A key difference between both methods is that, while their method aims to maximize the stationarity properties of the textures using external metrics, our method learns them from the texture itself, using a self-supervised approach. This results in our method better preserving the content of the original input.
As shown in Figure~\ref{fig:moritz_smaller}, our method shows compelling results for the kind of textures shown in their paper.
In the \textit{fence} example, our methods better preserves the vertical straight lines. For the \textit{wall} example, both methods shows compelling results, capturing a different repetition pattern.
We include more results using this dataset, in addition to comparisons with the other methods on the Supplementary Material.

As Moritz \etal's dataset contains mainly textures of human-made environments, we gather a different set of images of greater variety in their regularity and content.
Figure~\ref{fig:comparison_all} shows a comprehensive comparison with the other methods. The work by Li \etal~\cite{li2020inverse}, based on \textit{Graph-Cuts}, works reasonably well if the transformation can be done locally at the seams but fails when the required changes are global, as happens in the \textit{basket}.
The \textit{Repeated Pattern Detection} algorithm proposed by Rodriguez-Pardo \etal~\cite{Rodriguez-Pardo2019AutomaticPatterns} is not able to handle many of these challenging cases, which do not represent grid-like textures, as in the \textit{bananas} or \textit{stars}.
The Periodic Spatial GAN (PSGAN) proposed by Bermann \etal~\cite{bergmann2017learning} generates artifacts, not maintaining the content of the texture, as in the \textit{daisies} or the \textit{hieroglyph} examples.
A similar effect is observable in the method of~\cite{niklasson2021self-organising}, that use self-organizing representations through neural cellular automata. The output of the method is seamless but the integrity of the original texture is not preserved in many cases, for example, in the \textit{hieroglyph} or the \textit{stars}.

\begin{figure}[t!]
	\begin{center}
		\includegraphics[width=.98\columnwidth]{figures/results/moritz/moritz_smaller.pdf}
	\end{center}
	\vspace{-3mm}
	\caption{A comparison with the work on Texture Stationarization by Moritz \etal~\cite{moritz2017texture}, using textures from their dataset. The results are tiled $2\times 2$ times to help visualization.
	}
	\vspace{-5mm}
	\label{fig:moritz_smaller}

\end{figure}
Our method favors keeping high-level and bigger semantic structures of the textures, resulting in larger samples with more variety. The modification applied to the texture is the minimum required to make it seamlessly tileable. It provides high-quality outputs for regular textures with uneven illumination and perspective distortion, as the \textit{basket}, textures which are greatly irregular, such as the \textit{bananas}, and stochastic ones, like the \textit{strawberry}.
While non-parametric synthesis methods are typically computationally cheap, obtaining results in less than one minute~\cite{li2020inverse,Rodriguez-Pardo2019AutomaticPatterns,deliot2019procedural}, at the cost of quality and generality, parametric methods are more expensive and slower. Our method is competitive in computational cost when compared to other parametric synthesis methods. Our entire training and sampling process takes less than 45 minutes. In comparison, PSGAN~\cite{bergmann2017learning} needs 4 hours to train, the work by Zhou~\etal~\cite{Zhou2018} requires 6 hours, and the method of Niklasson~\etal~\cite{niklasson2021self-organising} takes 35 minutes to generate textures limited to $200\times200$ pixels.
On the contrary, our method is not limited by the size of the input texture and can synthesize tiles of any size thanks to the fully-convolutional architecture, hardware being the only limiting factor.
We include further and full-resolution results on the Supplementary Material.

\paragraph*{\textbf{Quantitative Comparison}}
Following a similar evaluation scheme as proposed in Section~\ref{sec:networkdesign} for measuring the differences between input and synthesized images, a quantitative evaluation is shown in Table~\ref{tab:quantitative_methods}. We use Moritz's dataset for a fair comparison with every method. All the images we used are included in the Supplementary Material.
The metrics we use for this experiment require that the input and synthesized images have the same resolution. To achieve this, and, in order to account for artifacts in the borders of the generated textures, we tile the synthesized images until they cover the same resolution as their corresponding inputs.

Interestingly, methods based on patches~\cite{rodriguez2019personalised,moritz2017texture,li2020inverse} obtain better SSIM and Si-FID scores than previous deep learning-based methods~\cite{bergmann2017learning,niklasson2021self-organising}.
This difference is not seen in the LPIPS metric. This suggests
that patch-based methods preserve better the structure of the input textures than previous neural parametric models.
 Our method, by combining a variety of loss functions, allows for a better preservation of the style and semantic content of the generated textures. Furthermore, our latent space manipulation algorithm allows for seamless borders between tiles, outperforming both previous non-parametric and parametric methods in perceptual and structural similarity.
The magnitude of the Si-FID metric varies significantly between the values obtained in Table~\ref{tab:quantitative_architectures} and Table~\ref{tab:quantitative_methods}, which indicates that this metric may be overly sensitive to the global statistics of the dataset.

\begin{table}[t!]
	\centering
	\begin{tabular*}{\columnwidth}{l @{\extracolsep{\fill}}ccc}

		&SSIM $\uparrow$&Si-FID $\downarrow$&LPIPS $\downarrow$\\ \cline{2-4}
		Deloit \etal~\cite{deliot2019procedural} &   0.1424 & 1.3471 &0.6207  \\
		Li \etal ~\cite{li2020inverse} &0.2086 & 0.9529     &  0.5818 \\
		Moritz \etal~\cite{moritz2017texture}  & 0.1968   &0.7620  & 0.5171  \\
		Rodriguez \etal~\cite{Rodriguez-Pardo2019AutomaticPatterns}  & 0.2144  & 1.2958&0.5137 \\
		Bergmann \etal~\cite{bergmann2017learning}  &0.1723  & 1.4355 & 0.5624 \\
		Niklasson \etal~\cite{niklasson2021self-organising}  & 0.1753   &1.3171  &0.5328\\
		\textbf{SeamlessGAN} & \textbf{0.2341}  & \textbf{0.6311}  &\textbf{0.4792}  \\
	\end{tabular*}
	\caption{Quantitative comparison between different methods. We show the average results on different perceptual metrics across a variety of textures, shown in the Supplementary Material. As shown, SeamlessGAN consistently outperforms its counterparts at every studied metric. We tile the outputs until they match the spatial resolution of the input examples. Higher is better for SSIM, while lower is better for Si-FID and LPIPS.}
	\label{tab:quantitative_methods}
	\vspace{-4mm}
\end{table}

\begin{figure*}[!p]

	\centering
	\vspace{-30mm}
	\includegraphics[width=1\textwidth]{figures/results/fig_10_comparison_v3.pdf}
	\vspace{-30mm}
	\caption
		Comparison with previous methods. On the left column, we show the input textures. From left to right, the synthesized results of Histogram Blending, by Deloit \etal~\cite{deliot2019procedural}, the GraphCuts algorithm by Li \etal~\cite{li2020inverse}, Repeated Pattern Detection by Rodriguez-Pardo \etal~\cite{Rodriguez-Pardo2019AutomaticPatterns}, PSGAN by Bergmann \etal~\cite{bergmann2017learning}, Self-Organizing Textures by Niklasson \etal~\cite{niklasson2021self-organising}; and ours. Outputs are tiled a similar number of times (at least twice in each dimension) for better visualization. Our method generally captures better the overall structure of the texture, while providing seamless and semantically coherent borders, for enhancing tileability. From top to bottom, we sample $c=4, 12, 5, 17, 13, 1, 19 \text{ and } 43$ input crops before obtaining a tileable texture. More results are included in the Supplementary Material. }

	\label{fig:comparison_all}

\end{figure*}

\paragraph*{\textbf{Limitations and Discussion}} Our method is inherently limited by the capabilities of the adversarial expansion technique to learn the implicit structure of the given texture. That is, if the input does not show enough regularity, the adversarial expansion fails, as shown in Figure~\ref{fig:failurecase}. It is interesting to see that the synthesis of the \textit{dotted} texture fails to reproduce the larger dots, as they are very scarce. However, the remaining structure is very well represented.

Tiling single texture maps, even if they contain seamless borders, may generate perceptible repetitions. This is an intrinsic limitation of any single sample tileable texture synthesis. Approaches such as \textit{Wang Tiles}~\cite{wang2004image} can tackle these limitations, but have additional disadvantages like increased memory and run-time consumption, rendering them less practical for real-time applications. Alternatively, procedural methods~\cite{guehl2020semi} are the most effective way to generate material samples preserving textural properties at multiple scales; however, present several limitations in the range of materials that can be generated to make them fully usable.

\section{Conclusions and Future Work}\label{sec:conclusions}

We have proposed a deep parametric texture synthesis framework capable of synthesizing textures into tileable single-tiles, by combining recent advances on deep texture synthesis, adversarial neural networks and latent spaces manipulation.
Our results show that our method can generate visually pleasing results for images with different levels of regularity and homogeneity. This work is the first method capable of exploiting properties of deep latent spaces within neural networks for generating seamless textures, and opens the opportunity for end-to-end tileable texture synthesis methods without the need for manual input. Comparisons with previous state-of-the-art methods show that our method provides results which better maintain the semantic properties of the textures, while being able to synthesize multiple maps at the same time.

Our method can be improved in several ways. First, the adversarial expansion framework, while powerful, it has some potential pitfalls that hinders its widespread applicability.
The same neural architecture is used for every texture but, as discussed, different choices on the architecture make different assumptions on the nature of the textures. We have proposed a generic architecture that works well for many examples, but recent advances in Neural Architecture Search~\cite{mellor2020neural} may provide better priors on the optimal neural architecture to use for each sample. Furthermore, each texture synthesis network is trained from scratch. This is not only computationally costly, but learning to synthesize one texture may help in the synthesis of other textures, as shown by~\cite{liu2020transposer}. Fine-tuning pre-trained models for generating new textures may provide cheaper syntheses.
Besides, our discriminator, while capable of detecting local artifacts, provides little control for separating such artifacts and global semantic errors. Recent work on image synthesis may provide guidance onto designing better discriminative models~\cite{schonfeld2020u}, training procedures~\cite{yun2019cutmix, sinha2021negative}, image parametrizations~\cite{Wang_2020_CVPR,hudson2021generative,niklasson2021self-organising} or perceptual loss functions~\cite{heitz2020pitfalls}.

Second, we proposed a synthesis solution based on manipulating latent spaces within the generative model, but explicitly training the network to generate tileable textures may provide better results than our approach. Besides, our sampling procedure could be extended for better selection of textures, by comparing histograms of the activations of the discriminator on the selected central area, instead of simply comparing minimum values.

 Finally, our method has the advantage of being fully automatic, however, pre-processing the texture images so they are more easily tileable can help the synthesis process. For example, automatically rotating the textures so their repeating patterns are aligned with the axes was studied by~\cite{Rodriguez-Pardo2019AutomaticPatterns}. Powerful methods for artifact~\cite{Dekel2015NonLocalVar} and distortion~\cite{li2019blind} removal could be applied as a pre-processing operation to the input textures before training the generative models, or as additional components to their loss function for improving tileability or homogeneity.

\begin{figure}[t!]
	\begin{center}
		\includegraphics[width=\columnwidth]{figures/results/failurecase2.pdf}
	\end{center}
	\vspace{-2mm}
	\caption{A limitation of our texture synthesis algorithm: Left: Input texture stacks; right: synthesized stacks. The network fails to replicate the pattern if the occurrence is not frequent enough, as we can see in the base color of these two examples. On the other hand, the synthesized normal map is consistent as its repetitive structure is seen frequently by the network.
		\vspace{-4mm}
	}

	\label{fig:failurecase}

\end{figure}

\section{Experiments}\label{sec:experiments}

In this section, we evaluate our two main contributions: first, in Section~\ref{sec:networkdesign} we study the impact of the design choices of the generator $\mathcal{G}$ to synthesize a multi-layer texture stack, and second, the quality of the tileable texture synthesis through several ablation studies.
We study the inter-map consistency in Section~\ref{sec:inter_map_consistency}, and the impact of the loss function in Section~\ref{sec:loss_function_ablation}.
Finally, we compare our method with  methods on tileable texture synthesis (Section~\ref{sec:comparison_other_methods}).

\subsection{Generator $\mathcal{G}$ Design }
\label{sec:networkdesign}

One of the main challenges when synthesizing texture stacks using a single generator is to preserve the low-level details of each of the texture maps whilst maintaining the local spatial coherence between them; if this coherence is lost, renders that use the synthesized maps will show artifacts or look unrealistic. We propose two different variations of the single-map generative architecture $\mathcal{G}$ presented in~\cite{Zhou2018}, each of which makes different assumptions on how the synthesis should be learned taking into account the particular semantics and purpose of each map. A diagram of each proposed architecture is shown in Figure~\ref{fig:comparison}. For a fair evaluation, we follow the criteria that both networks must have approximately the same number of trainable parameters.

Our baseline, $\mathcal{G}_1$, treats the texture stack as a multiple channel input image, and entangles every texture map in the same layers. It assumes that the maps in the stack share most of the structural information and, as such, there is no need to generate them separately. Thus, the last layer in the decoder outputs every texture map. Our proposed alternative architecture, $\mathcal{G}_2$ finds a shared representation of each texture map, but has a separate decoder for each of them. As such, the residual blocks are shared for all the texture stack, but each decoder can be optimized for the semantics and statistics of each particular map.

To quantitatively evaluate which architecture produces the highest quality output we compare the original texture with the generated one using standard metrics: SSIM, Si-FID, and LPIPS.
The \textit{Structural Similarity Index Measure (SSIM)}~\cite{wang2004image} is a perceptual-aware metric, working on the pixel space, that measures the similarity in structural information, and may be appropriate to evaluate synthesized textures.
The Si-FID~\cite{Shaham_2019_ICCV} is a single image extension of the \textit{Fréchet Inception Score (FID)}~\cite{heusel2017gans}, which measures the difference in deep latent statistics between natural and artificially generated images.
Finally, we use the~\textit{Learned Perceptual Image Patch Similarity (LPIPS)}~\cite{Zhang2018} as a perceptual distance metric in deep image spaces. This metric is widely used for evaluating generative models~\cite{Karras_2020_CVPR,Huang_2018_ECCV,Chan_2019_ICCV,almahairi2018augmented,mardani2020neural}.

The baseline $\mathcal{G}_1$ shows more artifacts than $\mathcal{G}_2$,  most likely due to the fact that the generation parts of the network are not fully separated. A quantitative evaluation is shown in Table~\ref{tab:quantitative_architectures}, showing that $\mathcal{G}_2$ outperforms $\mathcal{G}_1$ from perceptual and statistical standpoints. Those metrics require the input and target images to have the same spatial dimensions. To obtain this, we crop the $50\%$ center area of each generated stack, which doubles the dimensions of the inputs due to the adversarial expansion approach we follow, and compare them with the input textures. In summary, $\mathcal{G}_2$ allows to generate textures that better preserve the properties of the input images without any additional computational cost and without requiring to modify the loss function, or the discriminator design.

\begin{table}[!t]
	\centering
	\begin{tabular*}{\columnwidth}{c @{\extracolsep{\fill}}cccc}
		\centering

		&          &SSIM $\uparrow$ &Si-FID $\downarrow$ &LPIPS $\downarrow$  \\ \cline{2-5}
		\multirow{2}{*}{$\mathcal{G}_1$} & Albedo  & 0.2774      & 0.3462      &  0.4826  \\
		& Normals &  0.2364   &  \textbf{0.3636}     & 0.4870     \\  \cline{1-5}
		\multirow{2}{*}{$\mathcal{G}_2$} & Albedo  & \textbf{0.3123}     &    \textbf{0.3275}   &  \textbf{0.4377}   \\
		& Normals &   \textbf{0.2921}    &  0.4019     &  \textbf{0.4340}    \\ \cline{1-5}
	\end{tabular*}
	\caption{Quantitative comparison between our variations of the generator $\mathcal{G}$. We show the average results on different metrics across different texture stacks, separated by maps. As shown, $\mathcal{G}_2$ outperforms its baseline across metrics and texture maps. $\mathcal{G}_1$ only yields better scores at the Si-FID metric on the normal map.  Higher is better for SSIM, while lower is better for Si-FID and LPIPS.}
	\label{tab:quantitative_architectures}
	\vspace{-4mm}
\end{table}

\section{Self-Supervised Texture Stack Synthesis using Adversarial Expansion}\label{sec:synthesis}

For each input texture $\mathcal{T}$ we train a GAN,
whose generator $\mathcal{G}$ is able to synthesize novel examples $\mathcal{T'}$.
Unlike other GAN frameworks, which take as input a random vector, we use crops $t$ of the original stack as input to guide the generative sampling, such that, $\hat{\mathcal{T'}} = \mathcal{G} (t)$ for $t \in \mathcal{T}$.
The training strategy builds upon the work of Zhou \etal~\cite{Zhou2018}, which uses \textit{adversarial expansion} to train the network as follows:
First, a target crop $t \in \mathcal{T}$ of $2k \times 2k$ pixels is selected from the input stack.
Then, from that target crop $t$, a source random crop $t_s \in t$ is chosen with a resolution of $k\times k$ pixels.
The goal of the generative network will be to synthesize $t$ given $t_s$. This learning approach is fully self-supervised.
The generative model is trained alongside a discriminator $\mathcal{D}$, which learns to predict whether its inputs are the target crops $t \in \mathcal{T}$ or the generated samples $\mathcal{T'} = \mathcal{G}(t_s)$.
Figure~\ref{fig:training_framework} shows an overview of the training strategy.

\subsection{Network Architecture}
SeamlessGAN is comprised by an encoder-decoder convolutional generator $\mathcal{G}$ with residual connections, and a convolutional discriminator $\mathcal{D}$. So as to be able to synthesize textures of multiple different sizes, the networks are designed to be fully-convolutional. We follow the residual architectural design in~\cite{Zhou2018}, with two extensions:
First, building on recent advances on style transfer algorithms, we use \textit{Instance Normalization}~\cite{ulyanov2016instance} before each ReLU non-linearity in the generator and each Leaky-ReLU~\cite{maas2013rectifier} operation in the discriminator. This allows to use normalization for training the networks without the typical artifacts caused by Batch Normalization \cite{ChoiStarGAN:Translation, Deschaintre2018Single-imageNetwork, Nam2018Batch-instanceNetworks, kurach2019large}. Second, in order to allow for the synthesis of multiple texture maps at the same time, we propose a variation in the generator architecture.
In the following, we describe the details of each component, with a particular focus on the elements that are different from \cite{Zhou2018}.
\begin{figure}[t!]
	\centering
	\includegraphics[width=\columnwidth]{figures/diagrams/trainingframework_v2.pdf}
	\vspace{-5mm}
	\caption{An overview of our training framework for learning to synthesize texture stacks through adversarial expansion. At each iteration, from an input stack $\mathcal{T}$, we randomly crop a target crop $t$, from which we select a source crop $t_s$. The goal of the generator $\mathcal{G}$ is to estimate $t$ given $t_s$. We measure the difference between target and estimated crops using a combination of adversarial, pixel-wise and perceptual loss functions.
	}
	\vspace{-7mm}
	\label{fig:training_framework}
\end{figure}
\paragraph*{\textbf{Generator}}

The generative architecture proposed is comprised of three main components:
An \textit{encoder} $\mathcal{E}$, which compresses the information of the input texture $t$ into a latent space, $\mathrm{x}_0\gets\mathcal{E}(t)$,
with half the spatial resolution of the input texture.
Then, a set of \textit{residual blocks}, $\mathcal{R}_i, i \in \{1,l\}$, which learn a compact representation of the input texture $\mathrm{x}_i \gets \mathcal{R}_{i} (\mathrm{x}_{i-1}) + \mathrm{x}_{i-1}$.
Finally, a stack of \textit{decoders} $\mathrm{\textbf{G}}$ that transforms the output of the last residual block $\mathcal{R}_l$ into an output texture $\mathcal{T'} \gets \mathrm{\textbf{G}}(\mathrm{x}_l)$. Residual learning allows for training deeper models with higher levels of visual abstraction and which generate sharper images~\cite{he2016deep, ronneberger2015u,Isola2017ImagetoImageTW}.  Similar residual generators have been used in unsupervised image-to-image translation problems~\cite{Zhu2017UnpairedNetworks}.

Synthesizing a texture stack of multiple maps with a single generative model poses extra challenges over the single map case. Each map represents different properties of the surface, such as geometry or color, resulting in visually and statistically different images. This suggests that independent generative models per map could be needed. However, the texture maps must share pixel-wise coherence, which is not achievable if multiple generative models are used. Inspired by previous work on intrinsic images~\cite{janner2017self, yu2019inverserendernet, Li_2018_CVPR}, we propose the use of a generative model that learns a shared representation of all the texture maps, but has different decoders for each of them. The latent space within the generator will thus encode a high-level representation of the texture, which is then decoded in different ways for each different map in the texture stack.
Specifically, we train a stack of decoders, $\mathrm{\textbf{G}} = \{ G_\mathrm{A}, G_\mathrm{N}\}$, one for each map of the stack (albedo and normals in our case).

\paragraph*{\textbf{Discriminator}}

For the discriminator $\mathcal{D}$, we use a \textit{PatchGAN} architecture \cite{Zhu2017UnpairedNetworks,Isola2017ImagetoImageTW,Ledig2017Photo-realisticNetwork, Zhou2018}, which, instead of providing a single estimation of the probability of the whole image being real, it classifies this probability for small patches of it. This architecture has several advantages for our problem. First, it provides local assessments on the quality of the synthesized textures, which we exploit for obtaining high-quality textures. Second, its architecture design allows to provide some control on what kind of features are learned: by adding more layers to $\mathcal{D}$, the generated textures are typically of a higher semantic quality but local details may be lost, as we show on the Supplementary Material. A comprehensive study on the impact of the depth of $\mathcal{D}$ can be found in~\cite{Zhou2018}.

\paragraph*{\textbf{Loss Function}}
We train the networks following a standard GAN framework~\cite{Goodfellow2014GenerativeNets}. We iterate between training $\mathcal{D}$ with a real sample texture stack $\mathcal{T}$ and a generated sample $\mathcal{T'}$.
The adversarial loss $\mathcal{L}_{adv}$ is extended with three extra loss terms: $\mathcal{L}_1^a$, $\mathcal{L}_1^n$, and $\mathcal{L}_{style}$; corresponding respectively to the pixel-wise distance between the generated and target albedo, normals, and a perceptual loss. The perceptual loss $\mathcal{L}_{style}$ is computed as the sum of the perceptual losses between target and generated albedo maps, and target and generated normal maps. We follow the gram loss described in~\cite{Gatys_2016_CVPR} as our perceptual loss.
We weight the total style loss by weighting different layers in the same way described in \cite{Rodriguez-Pardo2019AutomaticPatterns, Zhou2018,Gatys_2016_CVPR}. Our global loss function thus is defined as:
\begin{equation}
	\mathcal{L} = \lambda_{adv} \mathcal{L}_{adv} + \lambda_{\mathcal{L}_1^a} \mathcal{L}_1^a + \lambda_{\mathcal{L}_1^n} \mathcal{L}_1^n + \lambda_{style} \mathcal{L}_{style}
\end{equation}

\begin{figure}[t!]
	\centering
	\subcaptionbox{$t$}[.242\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/input}}
	\subcaptionbox{$\mathcal{\hat{T}}, l=0$}[.242\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/l1_closeup.pdf}}
	\bigskip
	\subcaptionbox{$\mathcal{\hat{T}}, l=3$}[.242\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/l3_closeup.pdf}}
	\subcaptionbox{$\mathcal{\hat{T}}, l=5$}[.242\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/l5}}
	\vspace{-3mm}
	\caption{Impact of the latent field level $l$ on the quality of the output textures
		 $\mathcal{\hat{T}} = \mathcal{G}(t)$, using the same input
		$t$. Generating the latent field $\mathrm{F}_l$ early layers ($l=0$) generates the best visual results, later layers either create small artifacts ($l=3$) or generate unrealistic textures ($l=5$). A zoom of the central crop is shown as an inset.}\label{fig:latentfield}
	\vspace{-4mm}
\end{figure}

\section{Implementation Details}\label{sec:implementation}
As described in Section~\ref{sec:synthesis}, we follow a standard GAN training framework, by iterating between training the discriminator and the generator.
 We use a batch size of $1$, and an input size of $k=128$.
 All weights are initialized by sampling a gaussian distribution  $\mathcal{N}(0,0.02)$, following standard practice~\cite{Zhu2017UnpairedNetworks}.
 $\mathcal{G}$ has $l=5$ residual blocks with ReLU activations, and $\mathcal{D}$ is comprised of $5$ convolutional layers with Leaky-ReLU non-linearities, and a Sigmoid operation at the end. We refer the reader to the Supplementary Material for an ablation study on the influence of the network size. We use a stride of $2$ for the downsampling operations in $\mathcal{E}$ and transposed convolutions~\cite{long2015fully} for upsampling in the decoders $\mathrm{\textbf{G}}$. We weight each part of the loss function as: $\lambda_{adv} = \lambda_{style}=1$, and $\lambda_{\mathcal{L}_1^a} = \lambda_{\mathcal{L}_1^n} =10$.

 The networks are trained for $50000$ iterations using Adam \cite{Kingma2015Adam:Optimization}, with an initial learning rate of $0.0002$, which is divided by 5, after iterations $30000$ and $40000$. Aside from random cropping, we do not use any other type of data augmentation method.
The models are trained and evaluated using a single NVIDIA GeForce GTX 1080 Ti. Even if training takes around $40$ minutes for each texture stack, once trained, the generator can generate individual new samples in milliseconds, before checking for tileability.
We use PyTorch \cite{Paszke2019PyTorch:Library} as our learning framework.
To accelerate the training process, we leverage mixed precision training and automatic gradient scaling~\cite{micikevicius2017mixed}.
Every operation in the training pipeline is done natively in GPU using Torchvision~\cite{marcel2010torchvision}. These optimizations allow us to train the networks one order of magnitude faster than previous methods~\cite{Zhou2018}.
The input textures $\mathcal{T}$ tested in this paper have, on average, 500 pixels in their larger	dimension, for which our method can generate tiles of at most 1000 pixels. We use Photometric Stereo~\cite{ikeuchi1981determining} for computing the normals of fabric textures, and artist-generated normals for the other samples.
Please refer to the Supplementary Material for further implementation details.

\begin{figure}[t!]
	\centering
	\includegraphics[width=\columnwidth]{rebuttal/consistency.png}
		\vspace{-2mm}
	\caption{Crops of synthesized textures using our method. We observe that pixel-wise coherence between maps is preserved.}
	\vspace{-5mm}
	\label{fig:consistency_maps}
\end{figure}

We tile the latent space at its earliest level ($\mathrm{F}_l, l = 0$), as it provides the best quality results, as shown on Section~\ref{sec:latenttiling}.
For identifying the tileability of textures, we use a search area $S$ that spans a $20\%$ of each spatial dimension of the textures.
For all the results shown, $\gamma = 1$.
We choose $c_{min} = 100$ pixels, and $c_{max}$ equal to the resolution of the whole input texture. This means that for the level of $c_{max}$, only one texture is sampled.
For each $c < c_{max}$, we sample $3$ different random crops.
Typically, for the results shown in the paper, a tileable texture stack is found at high-resolution crop sizes, $c  \geq c_{max} - 10$, thus needing to sample and evaluating less than thirty random crops before finding a satisfactory solution. This entire sampling procedure takes less than five minutes.

\section{Introduction}\label{sec:introduction}}

\IEEEPARstart{R}{ealistic} and high-quality textures are important elements to convey realism in virtual environments. These can be procedurally generated~\cite{tu2020continuous,hu2019novel,guehl2020semi,galerne2012gabor,gilet2014local,guingo2017bi,heitz2018high}, captured~\cite{Guo:2020:MaterialGAN,li2018materials} or synthesized from real images~\cite{efros2001image,kwatra2005texture,Zhou2018, moritz2017texture}. Frequently, textures are used to efficiently reproduce elements with repetitive patterns (for example, facades, surfaces, or materials) by means of spatially concatenating --or \textit{tiling}-- multiple copies of themselves.
Creating tileable textures is a very challenging problem, as it requires a semantic understanding of the repetitive elements, often at multiple scales. For this reason, such a process is frequently done manually by artists in 3D digitization pipelines.

Recent advances in Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) have been applied to texture synthesis problems~\cite{Zhou2018,Fruhstuck2019TileGAN:Texturesb,liu2020transposer,bergmann2017learning,jetchev2016texture, mardani2020neural, hertz2020deep} showing unprecedented levels of realism and quality, however, the output of these methods is not tileable.
Despite recent methods~\cite{ deliot2019procedural,bergmann2017learning,Rodriguez-Pardo2019AutomaticPatterns,moritz2017texture,li2020inverse,niklasson2021self-organising} addressing the problem of tileable texture synthesis, we show that they either assume a particular level of regularity or the generated textures lose a significant amount of visual fidelity with respect to the input exemplars.
Further, most of these methods have only focused on synthesizing single images.
Rendering realistic materials requires more information about their optical properties beyond what represents a single RGB pixel. To this end, it is common to use spatially-varying BRDFs~\cite{deschaintre2019flexible}, which are optical appearance models parameterized by stacks of images, each one representing a different property, such as albedo, normals, or transparency.
As the number of methods that generate texture stacks from physical samples grows~\cite{deschaintre2019flexible,deschaintre2020guided,guo2020materialgan} so does the need to turn them into \textit{tileable texture stacks}.

\begin{figure}[!t]
	\centering
	\vspace{-8mm}
	\includegraphics[width=\columnwidth]{figures/teaser/teaser2.pdf}
	\caption{Na\"ively tiling the original texture causes discontinuities at the seam intersections as shown in the top row. Our method automatically generates a tileable texture stack from an input exemplar which double the size of the input.
		\vspace{-6mm}
	}
	\label{fig:teaser2}
\end{figure}

In this work, we propose a deep generative model, \textit{SeamlessGAN}, capable of synthesizing tileable texture stacks from inputs of arbitrary content.
In contrast to \textit{Wang Tiles}~\cite{cohen2003wang}, by which a single large texture region is created by concatenating multiple different tiles with matching borders, we aim to automatically obtain a seamless single-tile, which allows for reduced memory consumption and enhanced usability in casual scenarios when the user might lack the necessary artistic skills.
Our key idea is to realize that tiling a latent space within a generative network produces outputs with continuity at the seam intersection~\cite{Fruhstuck2019TileGAN:Texturesb}, which can then be turned into tileable images by cropping the central area.
Since not every value of the latent space is valid to produce high-quality outputs, we follow a double strategy: First, we train the generative network using an adversarial expansion technique~\cite{Zhou2018}, which provides a latent encoding of the input texture, which can then be decoded into high-quality outputs that double the spatial extent of the input.
Second, we use the trained discriminator as a local quality metric in a sampling algorithm. This allows us to find the input of the generative process that produces tileable textures similar to the original, as well as multiple candidates per input exemplar.
As opposed to previous work, which focused on maximizing stationarity~\cite{moritz2017texture} and thus might remove important high-level texture features in regular or near-regular textures, our method is focused on maximizing tileability while preserving the original texture as intact as possible, in terms of its stylistic and semantic properties.
To allow for the synthesis of stacks of textures, we propose a neural architecture composed of various decoder networks. Despite not explicitly imposing inter-map consistency, we show that it is implicitly guaranteed by how the generative network is trained. Without losing generality, we show texture stacks synthesis of two maps: an albedo and a normal map.
We demonstrate that our method outperforms state-of-the-art solutions on tileable texture synthesis of single images and show several examples for synthesizing tileable texture stacks. We validate our design choices through several ablations studies and off-the-shelf perceptual quality metrics.

\section{Overview}\label{sec:methodology}

\begin{figure*}[h]
	\centering
	\includegraphics[width=.72\textwidth]{figures/diagrams/DiagramTsv2.pdf}
	\caption[width=1.0\textwidth]{\label{fig:overview}{Overview of \textit{SeamlessGAN}. The input texture stack, represented here by albedo and normal maps $\{A_c, N_c\}$ is fed to an encoder $\mathcal{E}$, which transforms it into a latent space $x_0$. We tile this latent space vertically and horizontally, obtaining a a latent field $\mathcal{F}_0$. $\mathcal{F}_0$ is further processed by several residual blocks $\mathcal{R}_i, i \in \{1, l\}$. The resulting latent variables are transformed by two different decoders,  $\mathcal{G}_a$ and $\mathcal{G}_n$, which will output 4 copies of a seamlessly tileable texture $\{\hat{A},\hat{N}\}$. By cropping the center part of this texture, we obtain a single of those copies, with seamless borders, $\{\hat{A_c},\hat{N_c}\}$. Additionally, this texture can be analyzed by a discriminator $\mathcal{D}$ which provides local estimations of the quality of the synthesis. By analyzing vertical and horizontal search areas $s_v, s_h$, it detects artifacts that may arise when tiling the texture, which gives us an estimation of how \textit{tileable} the texture is. This estimation can be then used by a sampling algorithm for generating high-quality tileable textures. }
	}
\end{figure*}

 In this section, we present the main components of our tileable texture synthesis algorithm \textit{SeamlessGAN}. First, we describe the generative learning framework we use for learning to synthesize textures.
 We then show how to use this framework for generating tileable textures. Finally, we show how our discriminator can be used for detecting artifacts in the synthesized textures. Using this network as a perceptual-aware error metric, we design a sampling algorithm capable of generating artifact-free tileable textures.
\subsection{Learning to synthesize textures}\label{subsec:learningframework}
The first step in our tileable texture synthesis framework is to train a generative adversarial network that learns to generate new examples of a single sample texture. We show how to train such a framework for generating tuples of albedo and normal maps, but the network can be extended to a different number of texture maps.
 Specifically, our generative model $G$ outputs a pair of albedo $\hat{A}$ and surface normals $\hat{N}$. Instead of following the typical GAN framework, which relies on a randomly sampled input vector, we use crops $A_c, N_c \in I$ of the input textures $I$ as input to our generative model.
The generative problem is defined as:
\begin{equation}
\hat{A} , \hat{N} = G(A_c, N_c)
\end{equation}
SeamlessGAN builds upon the work of Zhou et al.~\cite{Zhou2018}. We use \textit{adversarial expansion} to train the network to generate new texture samples.
During training, random crops of $2k \times 2k$ pixels of the input textures are selected, which will form the target $A_t, N_t \in \mathcal{T}$. The crop for both maps in $\mathcal{T}$ is performed over the same pixels of the texture maps, so as to preserve the spatial coherence between them. From that target crop $\mathcal{T}$, a source random crop $S$ is chosen $A_c, N_c \in S$, with a resolution of $k\times k$ pixels. The goal of the generative network will thus be to synthesize $\mathcal{T}$ given $S$. This learning approach is fully self-supervised
The generative model is trained alongside a discriminator $\mathcal{D}$, which learns to predict whether its inputs are the target crops $A_t, N_t \in \mathcal{T}$ or the generated samples $\hat{A} , \hat{N} = G(A_c, N_c)$.

\paragraph{Network architecture}
Our neural network is a generative adversarial network, comprised by an encoder-decoder generator with residual connections, and a convolutional discriminator. So as to be able to synthesize textures of multiple different sizes, the network is designed to be fully-convolutional. We follow the architectural design in~\cite{Zhou2018}, with a few variations. First, building on recent advances on style transfer algorithms, we use \textit{Instance normalization}~\cite{ulyanov2016instance} before each PReLU~\cite{he2015delving} non-linearity operation in both networks. This allows to use normalization for training the networks without the typical artifacts caused by Batch Normalization \cite{ChoiStarGAN:Translation, Deschaintre2018Single-imageNetwork, Ulyanov2016InstanceStylization, Nam2018Batch-instanceNetworks}. Second, in order to allow for the synthesis of multiple texture maps at the same time, we propose a simple variation in the generator architecture.

\paragraph{\textbf{Generator}}
 The baseline architecture proposed in~\cite{Zhou2018} is comprised of three main components: An \textit{encoder} $\mathcal{E}$, which entangles the information of the input texture $\mathcal{T}$ into a deep latent space $x_0\gets\mathcal{E}(\mathcal{T})$,  with half the spatial resolution of the input texture. This latent space is then processed by a set of $l$ \textit{residual blocks} $R_i, i \in \{1,l\}$, which learn a compact representation of the input texture $x_i \gets R_{i} (x_{i-1}) + x_{i-1}$. Finally, a decoder $\mathcal{G}$ transforms the output of the last residual block $R_l$ into an output texture $\mathcal{O} \gets \mathcal{G}(x_l)$. By using residual blocks in the latent space, we are able to train models that only use the levels of visual abstraction that are actually required \cite{he2016deep}, while preserving local details~\cite{ronneberger2015u}

 Synthesizing multiple texture maps with a single generative model poses extra challenges. Each map typically represents different properties of the material, such as micro-geometry or color. Those maps are visually different, meaning that learning to synthesize one of them may not help synthesizing the others. This suggests that different generative models could be needed for each map. However, the texture maps must share pixel-wise coherence, which is not achievable if multiple generative models are used. Following previous work on intrinsic image decomposition~\cite{janner2017self, yu2019inverserendernet, Li_2018_CVPR}, we propose the use of a generative model that learns a shared representation of all the texture maps, but has different decoders for each of them. Such an architecture assumes that the texture properties may be encoded by a dense latent space, but the way in which those latent variables are decoded is different for each texture map.

 As such, our proposed generative network encodes an input texture stack into a deep latent space, and different decoders will transform this latent space into texture maps, which all share pixel-wise correspondence. Specifically, we feed the generative network with a tuple of albedo and normal maps $A_c, N_c$. The encoder $\mathcal{E}$ transforms them into a dense representation $x_0\gets\mathcal{E}(A_c, N_c)$, which is then transformed by the residual blocks, obtaining a shared latent space $x_l$. This is then fed to a decoder for the albedo, which outputs $\hat{A} \gets \mathcal{G}_A(x_l)$ and for the normal map $\hat{N} \gets \mathcal{G}_N(x_l)$.

\paragraph*{\textbf{Discriminator}}

For the discriminator $\mathcal{D}$, we follow previous work and use a \textit{PatchGAN} architecture \cite{ Isola2017ImagetoImageTW,Ledig2017Photo-realisticNetwork, Zhou2018}, which, instead of providing a single estimation of the probability of the whole image being real, it is able to classify this likelihood for small patches of it. This architecture has several advantages for our problem. First, it is able to provide local assessments on the quality of the synthesized textures, which we exploit for obtaining high-quality textures. Furthermore, its fully-convolutional architecture allows to provide some control on what kind of features are learned. By adding more deep layers to $\mathcal{D}$, the generated textures are typically of a higher semantic quality, but lose details that may be beneficial to preserve. A detailed study on the impact of the depth of $\mathcal{D}$ can be found on~\cite{Zhou2018}.

\subsubsection*{Training procedure}
Instead of learning to synthesize tileable textures, we train the models to synthesize the textures without accounting for tileability, then leverage their latent spaces for obtaining seamless textures. We thus train the networks following a standard GAN learning framework~\cite{Goodfellow2014GenerativeNets}. We iterate between training the discriminator with a real sample texture stack $\mathcal{T}$ and a generated sample $\{\hat{A}, \hat{N}\}$. The adversarial loss function $\mathcal{L}_{adv}$ is extended with three extra loss terms: $\mathcal{L}_1^a$, $\mathcal{L}_1^n$ and $\mathcal{L}_{style}$. The $ \mathcal{L}_1$ losses measure the pixel-wise distance between generated and target albedo ($\mathcal{L}_1^a$) and normals ($\mathcal{L}_1^n$). To compute the $\mathcal{L}_{style}$, we use the deep perceptual loss described in ~\cite{Gatys_2016_CVPR}.
We weight the total style loss by weighting different layers in the same way described in \cite{Rodriguez-Pardo2019AutomaticPatterns, Zhou2018,Gatys_2016_CVPR}. Our global loss function is defined as:
\begin{equation}
\mathcal{L} = \lambda_{adv} \mathcal{L}_{adv} + \lambda_{\mathcal{L}_1^a} \mathcal{L}_1^a + \lambda_{\mathcal{L}_1^n} \mathcal{L}_1^n + \lambda_{style} \mathcal{L}_{style}
\end{equation}

Instead of using the network trained for the maximum amount of iterations, we perform the synthesis with the network that minimized the non-adversarial part of the loss function during training: $\lambda_{\mathcal{L}_1^a} \mathcal{L}_1^a + \lambda_{\mathcal{L}_1^n} \mathcal{L}_1^n + \lambda_{style} \mathcal{L}_{style}$. This makes the learning setup more robust to the instability that the discriminator may add to the loss function.

\subsection{Latent space tiling for texture tileability}

Previous work~\cite{Fruhstuck2019TileGAN:Texturesb} showed that by spatially concatenating different latent spaces within a StyleGAN~\cite{karras2019style} generator, it is possible to generate textures that contain the visual information of those tiles while seamlessly transitioning between the generated tiles. Inspired by this idea, we spatially repeat (horizontally and vertically) a single latent space $x_i$ within the generative model, obtaining a latent field $\mathcal{F}_i$. As described in Algorithm~\ref{alg:genseamlesstexture}, we then feed $\mathcal{F}_i$ to the rest of the generator, which outputs a texture stack that contains four copies of the same texture, but which seamlessly transitions between them. By cropping the center portion of this texture stack (with an area of $50\%$ of that of the output texture), we obtain a seamlessly tileable texture stack.
An important parameter to select is the level $j$ at which to split the generative process. As shown in Figure~\ref{fig:latentfield}, generating the latent field $\mathcal{F}$ by tiling earlier levels of the latent space forces the network to transform $\mathcal{F}$ more times, thus resulting in more seamlessly tileable textures. We confirm the results found by~\cite{Fruhstuck2019TileGAN:Texturesb} and found that splitting at earlier levels of the latent space ($j\in \{ 1, 2\}$) yields the best visual results. The output texture $O$ has a resolution of $4\times4$ the resolution of the input $\mathcal{T}$. Thus, after the cropping operation, the texture has a resolution of $2\times2$ that of $\mathcal{T}$.

\begin{figure}[t!]
	\centering
	\subcaptionbox{$A_c$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/input}}
	\subcaptionbox{$\hat{A}, l=1$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/l1}}
	\bigskip
	\subcaptionbox{$\hat{A}, l=3$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/l3}}
	\subcaptionbox{$\hat{A}, l=5$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/latent space/l5}}
	\caption{Impact of the latent field level $l$ on the quality of the output tileable textures, using the same input $A_c$. Generating the latent field $\mathcal{F}_l$ early layers ($l=1$) generates the best visual results, later layers either create small artifacts ($l=3$) or generate unpleasant textures ($l=5$).   }\label{fig:latentfield}
\end{figure}
\begin{algorithm}
	\caption{Algorithm for obtaining a tileable texture stack, given an input texture $\mathcal{T}$, a latent space level $j$ and a generative network, with an encoder $\mathcal{E}$, residual blocks $R_i, i \in \{1,l\}$ and a decoder $\mathcal{G}$}\label{alg:genseamlesstexture}
	\begin{algorithmic}[1]
		\Procedure{GenerateSeamlessTexture}{$\mathcal{T}, j, \mathcal{E}, R, \mathcal{G}$}
		\State $i\gets0$
		\State$x_i \gets \mathcal{E}(\mathcal{T})$ \Comment{Encode into latent space}
		\State $i \gets i + 1$
		\While{$i < j$}
			\State$x_{i} \gets R_i(x_{i-1}) + x_{i-1}$
			\State $i \gets i + 1$
		\EndWhile
		\State $\mathcal{F}_i \gets Tile(x_i)$
		 \Comment{Horizontal and vertical tiling}
		\While{$i < l$}
			\State$\mathcal{F}_{i} \gets R_i(\mathcal{F}_{i-1}) + \mathcal{F}_{i-1}$ \Comment{Rest of $R_i$}
			\State $i \gets i + 1$
		\EndWhile
		\State $ O \gets \mathcal{G} (\mathcal{F}_l)$ \Comment{Decode latent field}
		\State \Return $CenterCrop(O)$
	\EndProcedure
	\end{algorithmic}
\end{algorithm}

\subsection{Discriminator-guided sampling algorithm}

As described in Section~\ref{subsec:learningframework}, the generative model is trained in an \textit{adversarial expansion} fashion. The network receives small crops of the sample texture and learns to output textures of twice the spatial resolution of those crops. When synthesizing new textures, however, the fully-convolutional network design allows us to generate textures of any resolution. Even though the textures generated by Algorithm~\ref{alg:genseamlesstexture} are seamlessly tileable, the latent space tiling may generate unwanted artifacts that hinder the realism of the textures, by generating repeating patterns. In contrast to the method in~\cite{Fruhstuck2019TileGAN:Texturesb}, where the latent spaces are drawn from random vectors, ours are encoded representations of input texture stacks. As such, the selection of the input that the network receives plays a role on the quality of the output textures. Our goal now is to provide an automatic solution to this problem.
So far, we focused on leveraging properties of the generator for synthesizing tileable textures. This model is trained alongside a discriminator, which allows for local estimations of the quality of the generated textures. We argue, following recent work on generative modeling~\cite{schonfeld2020u}, that the discriminator can work as a semantic-aware error metric that can be exploited for detecting artifacts in the generated textures. As discussed in Section~\ref{subsec:learningframework}, the global loss function contains pixel-wise, \textit{style} and adversarial losses. The adversarial loss learns the semantics of the texture, whereas the other components of the loss functions model color distances or repeated pattern activations~\cite{Rodriguez-Pardo2019AutomaticPatterns}.

\begin{algorithm}[]
	\caption{Algorithm for classifying a generated texture  as tileable or not tileable, given an input texture  $O$, a Discriminator $\mathcal{D}$, a search area $s$, a threshold $\tau$  }\label{alg:istileable}
	\begin{algorithmic}[1]
		\Procedure{IsTileable}{$O,\mathcal{D}, $s$, \tau$}
		\State $Tileable \gets False$
		\State $\mathcal{P} \gets \mathcal{D}(O)$ \Comment{Output of discriminator}
		\State $s_v \gets verticalArea(\mathcal{P})$ \Comment{Vertical search area}
		\State $s_h \gets horizontalArea(\mathcal{P})$ \Comment{Horizontal search area}
		\State $s_d \gets (\mathcal{P} \cap \overline{(s_v \cup	s_h)})$ \Comment{Rest of  $\mathcal{P}$ }
		\State $m_v \gets min(s_v)$ \Comment{Worst prediction in $s_v$}
		\State $m_h \gets min(s_h)$ \Comment{Worst prediction in $s_h$}
		\State $m_d \gets min(s_d)$ \Comment{Worst prediction in $s_d$}
		\If{$m_v \leq m_d \times \tau$ and $m_h \leq m_d\times \tau$ }
		\State $Tileable \gets True$
		\EndIf
		\State \Return $Tileable$
		\EndProcedure
	\end{algorithmic}
\end{algorithm}

\begin{figure}[]
	\centering
	\subcaptionbox{$\hat{A_1}$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Successful/albedo}}
	\subcaptionbox{$\mathcal{D}(\hat{A_1})$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Successful/disc}}
	\bigskip
	\subcaptionbox{$\hat{A_2}$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Non-Successful/albedo}}
	\subcaptionbox{$\mathcal{D}(\hat{A_2})$}[.24\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Non-Successful/disc}}
	\caption{Outputs of $\mathcal{D}$ with textures of different qualities. On the left, a generated albedo $\hat{A_1}$ is artifact-free on the search areas, thus the discriminator ($\mathcal{D}(\hat{A_1})$) is fooled to believe $\hat{A_1}$ is real. On its right, a failed generated sample $\hat{A_2}$ cannot fool the discriminator, which finds artifacts $\mathcal{D}(\hat{A_2})$.  }\label{fig:discriminator}
\end{figure}

Empirically, we find that the artifacts caused by the latent space tiling in Algorithm~\ref{alg:genseamlesstexture} appear on vertical and horizontal frames around the center of the textures. This is likely caused by strong discontinuities or gradients on the same areas of the tiled latent spaces, which the rest of the generative network fails to transform into realistic textures. We propose to use the discriminator's predictions on these areas as an measure of how tileable the texture is. A visualization those predictions under artifact-free inputs and textures with artifacts can be seen on Figure~\ref{fig:discriminator}. The difference between the predicted quality of these areas and the rest of the texture will be our loss function that measures how tileable a texture is. Note that this loss function is learned specifically for each texture.
Specifically, as described in Algorithm~\ref{alg:istileable}, we identify whether or not a generated texture $O$ is tileable by looking for artifacts in a vertical and horizontal search area $s$ on the texture, by comparing the predictions $\mathcal{P}$ of $\mathcal{D}$ in $s \in \mathcal{P}$ with the rest of the image $s_d = \mathcal{P} \cap \overline{s}$. As $\mathcal{D}$ outputs $0$ when it believes a patch to be synthetic, we look for artifacts in the predictions by looking for the minimum predicted values on the search areas $s$ and $s_d$. If the minimum value in the vertical and horizontal search areas $s_v, s_h \in s$ are close (measured by $\tau$) to the minimum value in $s_d$, the texture has no artifacts on $s$ and, as such, it is tileable.

\begin{algorithm}[t]
	\caption{Sampling algorithm for generating a set $Samples$ of $n$ textures, crop sizes in the range of $c_{max}$ and $c_{min}$, an input $\mathcal{T}$ and $m$ random crops for each crop size.}\label{alg:samplingAlgorithm}
	\begin{algorithmic}[1]
		\Procedure{SampleTextures}{$n,c_{max}, c_{min}, \mathcal{T}, m$}
		\State $c \gets c_{max}$ \Comment{We start at the largest size}
		\State $Samples \gets \{\}$ \Comment{Empty list by default}
		\While{$length(Samples) < n$ or $c\geq c_{min}$}
		\State $i \gets 0$ \Comment{Sample $m$ crops by crop size}
		\While{$i<m$}
		\State $C \gets RandomCrop(\mathcal{T}, c)$
		\State $O \gets GenerateSeamlessTexture(C)$
		\If{$IsTileable(O)$}
		\State $Samples \gets Samples \cup O$
		\State $ i \gets i + 1 $
		\EndIf
		\EndWhile
		\State $ c \gets c - 1 $ \Comment{Look for smaller samples}
		\EndWhile
		\State \Return $Samples$
		\EndProcedure
	\end{algorithmic}
\end{algorithm}

We propose a discriminator-guided sampling algorithm capable of automatically generating multiple seamlessly tileable textures of different sizes. A main advantage of fully-convolutional generators is that they can generate textures of large sizes, which will organically create less repeating artifacts, as larger textures need to be repeated less times. Consequently, our sampling algorithm will look for tileable textures by randomly cropping the original sample in decreasing sizes. There are two main challenges when finding tileable textures: the size of the input texture has to be adequate so that it models the distribution of the repeating patterns it contains; and the input tile itself must not create strong artifacts when tiling the latent spaces of the generator. Some methods~\cite{lettry2017repeated,Rodriguez-Pardo2019AutomaticPatterns} can estimate the repeating pattern size, but they are limited to regular patterns, which are not common in real-world textures. Consequently, we will sample multiple crop sizes in a decreasing fashion, whilst sampling multiple candidate crops for each crop size. This sampling procedure is described in Algorithm~\ref{alg:samplingAlgorithm}.
This sampling algorithm is designed for square input textures, but it is easily extensible to rectangular shapes

\section{Overview}\label{sec:overview}

Our method takes as input an \textit{untileable} \textit{texture stack} $\mathcal{T}$, which is a layered representation --or SVBRDF~\cite{burley2012physically}-- of a material used by render engines to virtually reproduce it. A typical texture stack is composed of several maps such as albedo, normals, or roughness. For simplicity, and without losing generality, we assume that our texture stacks have two maps: an albedo $\mathrm{A}$, and a normal map $\mathrm{N}$, both are RGB images of the same dimensions.
There are several methods that generate texture stacks for any material using, for example, smartphones~\cite{deschaintre2019flexible, deschaintre2020guided,guo2020materialgan,shi2020match}, however, these stacks are not tileable by default. Having this data as input, we propose a two-step automatic pipeline to generate tileable texture stacks from arbitrary material input exemplars.

In the first step, described in Section~\ref{sec:synthesis}, we use crops $t$ of the input texture $\mathcal{T}$ to train a Generative Adversarial Network (GAN) able to synthesize novel \textit{untileable} texture stacks $\mathcal{T'}$ using adversarial expansion~\cite{Zhou2018}. This training framework has shown to provide state-of-the-art results on single-sample texture synthesis, surpassing previous approaches~\cite{ulyanov2016texture,bergmann2017learning, henzler2021generative}.
Thanks to using a GAN, we learn an implicit representation of the texture parameterised in two neural modules: $\mathcal{G}$ and $\mathcal{D}$.
$\mathcal{G}: t \rightarrow \mathcal{T'}$ is a generator that outputs new untileable textures which double the spatial resolution of the input. $\mathcal{D}$ is a discriminator that, thanks to the adversarial framework used for training, is able to distinguish real from fake textures.

In the second step, described in Section~\ref{sec:sampling}, we produce tileable stacks by means of two key ideas: first, we tile a latent space $\mathrm{x}_0$ of the trained generator $\mathcal{G}$, obtaining a novel texture stack showing continuity at the seams intersection $\mathcal{\hat{T}}$. Second, we implement a sampling process using the trained discriminator $\mathcal{D}$ used as quality metric $\mathcal{Q}$ to find an optimally tileable texture stack $\mathcal{T^*}$. An overview of our sampling step is shown in Figure~\ref{fig:overview}.

In the following, we first describe our GAN architecture, including our proposal to deal with textures stacks. Then, we describe the sampling process to generate tileable ones.

\section{Related work}\label{sec:relatedwork}
We review the texture synthesis methods most  closely related to our work. For a more comprehensive review, please refer to the surveys in~\cite{akl2018survey, raad2018survey}. We will also mention other related work regarding tileable texture synthesis, and deep internal learning.

\subsection{Texture Synthesis}
Traditionally, \textbf{non-parametric texture synthesis} algorithms worked by ensuring that every patch in the output textures approximates a patch in the input texture. Earlier methods included image quilting~\cite{efros2001image,efros1999texture}, GraphCuts~\cite{kwatra2003graphcut}, genetic algorithms~\cite{dong2007optimized}, and optimization~\cite{kwatra2005texture, portilla2000parametric}. More recent approaches use variations of PatchMatch~\cite{barnes2009patchmatch,barnes2015patchtable} as a way of finding correspondences between generated and input images~\cite{kaspar2015self,darabi2012image, zhou2017analysis}.

Despite those methods showing high-quality results for textures of different characteristics, recent work on deep parametric texture synthesis show better generality and scalability, requiring less manual input.
Our approach belongs to the category of \textbf{Parametric texture synthesis}. These methods learn statistics from the example textures, which can then be used for the generation of new images that match those statistics.
While traditional methods used hand-crafted features~\cite{de1997multiresolution, heeger1995pyramid}, recent parametric methods rely on deep neural networks as their parameterization.
Activations within latent spaces in pre-trained CNNs have shown to capture relevant statistics of the style and texture of images~\cite{gatys2016image, gatys2017controlling, johnson2016perceptual, ren2017personalized, rodriguez2019personalised}.
Textures can be synthesized through this approach by gradient-descent optimization~\cite{snelgrove2017high, gatys2015texture} or by training a neural network that learns those features~\cite{dosovitskiy2016generating, niklasson2021self-organising}.
Finding generic patterns that precisely describe the example textures is one of the main challenges in parametric texture synthesis.
Features that describe textured images in a generic way are hard to find and they typically require hand-tuning. Generative Adversarial Networks (GANs), which have shown remarkable capabilities in image generation in multiple domains~\cite{karras2018progressive, karras2019style, karras2020analyzing}, can learn those features from data.
Specifically, in texture synthesis, they have proven successful at generating new samples of textures from a single input image~\cite{Zhou2018} or from a dataset of images~\cite{bergmann2017learning,Fruhstuck2019TileGAN:Texturesb,jetchev2016texture, liu2020transposer, mardani2020neural}.
We build upon the method of Zhou \etal~\cite{Zhou2018} which shows good performance on the synthesis of non-stationary single image textures, and extend it to synthesize texture stacks, as well as generate tileable outputs.

\subsection{Texture Tileability}
Synthesizing tileable textures has received surprisingly little attention in the literature until recent years. Moritz \etal ~\cite{moritz2017texture} propose a non-parametric
approach that is able to synthesize textures from a single example while preserving its \textit{stationarity}, which measures how tileable the texture is. Li \etal~\cite{li2020inverse} propose a GraphCuts-based algorithm. They first find a patch that optimally represents the texture, then use graph cuts to transform its borders to improve its tileability. This method allows for synthesizing multiple maps at the same time. Relatedly, Deliot and Heitz \cite{deliot2019procedural} propose a histogram-preserving blending operation for patch-based synthesis of tileable textures, particularly suited for stochastic textures. The power of deep neural networks for tileable texture synthesis has also been leveraged in the past years. First, Rodriguez-Pardo \etal~\cite{Rodriguez-Pardo2019AutomaticPatterns} exploit latent spaces in a pre-trained neural network to find the size of the repeating pattern in the input texture. Then, they use perceptual losses for finding the optimal crop of the image such that, when tiled, looks the most similar to the original image. Also leveraging deep perceptual losses and using Neural Cellular Automata~\cite{mordvintsev2020growing} as an image parameterisation, Niklasson \etal~\cite{niklasson2021self-organising} generate self-organizing textures that are seamlessly tileable by design, but are limited in resolution and by the quality of the gram matrix perceptual metric used as a loss function~\cite{heitz2020pitfalls}. Bergmann \etal~\cite{bergmann2017learning} achieve tileability in their output textures by training a multi-image GAN in a periodic spatial manifold.
Our proposed method does not follow any of these approaches.
Instead, we build upon a state-of-the-art single-image texture synthesis method, which is able to generate high-quality and high-resolution images, and extend it to generate textures which are tileable. To this end, we propose a sampling algorithm that finds the input of the generative process that maximizes a novel tileability metric. Our goal is to preserve the input original texture as intact as possible while imposing artifact-free continuity at the intersection of the seams when the texture is tiled.  Furthermore, our method has a reduced computational footprint compared to other deep generative texture synthesis methods as we say in the results section.

\begin{figure*}[ht!]
 	\centering
 	\includegraphics[width=\textwidth]{figures/diagrams/DiagramTsv5.pdf}
\vspace{-6mm}
 	\caption[width=1.0\textwidth]{\label{fig:overview}{Overview of \textit{SeamlessGAN}.
 			A crop $t$ of the input texture stack $\mathcal{T}$, is fed to an encoder $\mathcal{E}$, which transforms it into a latent space $x_0$. We tile this latent space vertically and horizontally, obtaining a latent field $\mathrm{F}_0$. $\mathrm{F}_0$ is further processed by several residual blocks $\mathcal{R}_i, i \in \{1, l\}$. The resulting latent variables are transformed by two different decoders, $\mathrm{G}_A$ and $\mathrm{G}_N$, which output 4 copies of a candidate tileable texture $\mathcal{\hat{T}}$. By cropping the center part of this texture, we obtain a single of those copies, with seamless borders, $\mathcal{\hat{T}}_c$. Additionally, this texture can be analyzed by a discriminator $\mathcal{D}$ which provides local estimations of the quality of the synthesis. Using a \text{tileability} evaluation function $\mathcal{Q}$, which, by analyzing two vertical and horizontal search areas $s_v, s_h$, it is able to detect artifacts that may arise when tiling the texture. This gives us an estimation of how \textit{tileable} the texture is. This estimation can then be used by a sampling algorithm for generating high-quality tileable textures. }
 	}
\vspace{-4mm}
\end{figure*}
\subsection{Deep Internal Learning}
Learning patterns from a single image has been studied in recent years, in contexts different to those of texture synthesis. Ulyanov \etal~\cite{Ulyanov_2018_CVPR} show that single images can be represented by randomly initialized CNNs, and show applicability on denoising or image inpainting problems. A similar method is proposed by Shocher \etal~\cite{Shocher_2018_CVPR} for single-image super-resolution. Additionally, single images have shown to be enough for learning  low-level features that generalize to multiple problems~\cite{asano2019critical}. Hypernetworks~\cite{ha2016hypernetworks} in \textit{Implicit Neural Representations}~\cite{sitzmann2020implicit, tancik2020fourfeat} also allow for shift-invariant priors over images, which can then used in generative models~\cite{dupont2021generative}.
Single-image generative models have been explored for domains other than textures. GANs trained on a single image have been used for image retargeting~\cite{Shocher_2019_ICCV}, deep image analogies~\cite{benaim2020structural}, or for learning a single-sample image generative model~\cite{Shaham_2019_ICCV, Hinz_2021_WACV,shocher2019ingan, sushko2021one,vinker2020training}. These methods, while powerful for natural images, are not well-behaved for textures, as shown in~\cite{liu2020transposer} and~\cite{mardani2020neural}. By introducing inductive biases specially designed for textured images, characterized by their repeating patterns, deep texture synthesis methods achieve better performance in textured images than generic single-image synthesis approaches, which need to account for more globally coherent semantic patterns.

\section{Tileable Texture Stack Sampling} \label{sec:sampling}

\subsection{Latent Space Tiling}\label{sec:latenttiling}

After training, the generator $\mathcal{G}$ is able to synthesize novel samples of the texture given a small exemplar of it. Although these novel samples duplicate the size of the input, they are not tileable by default.
Previous work~\cite{Fruhstuck2019TileGAN:Texturesb} showed that by spatially concatenating different latent spaces in a ProGAN~\cite{karras2018progressive} generator, it is possible to generate textures that contain the visual information of those tiles while seamlessly transitioning between the generated tiles. Inspired by this idea, we spatially repeat (horizontally and vertically) the first latent space $\textrm{x}_0$ within the generative model, obtaining a latent field $\mathrm{F}_0$. This field is passed through the residual layers $\mathcal{R}_l$ and the decoders $\mathrm{\textbf{G}}$ to get a texture stack, $\mathcal{\hat{T}}$, that contains four copies of the same texture with seamless transitions between them (see Figure~\ref{fig:overview}).
At first, the latent field $\mathrm{F}_0$ shows strong discontinuities at the borders between the four copies.
Later, as the texture is passed through the network, these artifacts are progressively transformed into seamless borders by the rest of the residual blocks and the decoder of the model.
The intuition behind this idea is that, after training the network with the target texture, each of these latent spaces encode low resolution versions of the input in the spatial domain and semantically-rich information in the deeper layers.

A seamlessly tileable texture stack $\mathcal{\hat{T}}_c$ is obtained by cropping the central region (with an area of $50\%$ of $\mathcal{\hat{T}}$). The predicted texture $\mathcal{\hat{T}}$ has $4\times4$ the resolution of the input crop $t$. Thus, after the cropping operation, $\hat{\mathcal{T}}_c$ has twice the resolution of the input $t$.

A key parameter to select is the level $l$ at which to split the generative process. As shown in Figure~\ref{fig:latentfield}, generating the latent field $\mathrm{F}$ by tiling earlier levels of the latent space forces the network to transform $\mathrm{F}$ more times, thus resulting in more seamlessly tileable textures.
We confirm the results found in~\cite{Fruhstuck2019TileGAN:Texturesb} and noticed that generating this latent field at earlier levels of the latent space ($l\in \{ 0, 1\}$) yields the best visual results, by allowing for smoother and more semantically coherent transitions between tiles. We thus tile the output of the encoder $\mathrm{x}_0 = \mathcal{E}(t)$, before transforming it by the residual blocks $\mathcal{R}_i$.

\subsection{Discriminator-guided Sampling}

Our strategy to tile the latent space guarantees that the generated texture is a continuous function with smooth transitions between the boundaries of the tiles.
However, in contrast to the algorithm in~\cite{Fruhstuck2019TileGAN:Texturesb}, where the latent spaces are drawn from random vectors, ours are encoded representations of input textures.
Thus, the selection of the input that the network receives plays an important role on the quality of the output textures. As shown in Figure~\ref{fig:discriminator}, not all the generated textures are equally valid.
This selection can be posed as an optimization problem: $t^* = \textrm{argmax}_t \mathcal{Q}(\mathcal{G}(t))$, where the goal is to find the crop $t^*$ that maximizes the quality $\mathcal{Q}$ of the generated texture $\mathcal{G}(t) = \hat{\mathcal{T}}_c$.
To solve this optimization problem, one option could be to pose it as an optimization in the generative latent space. However, we found that a simpler solution based on sampling already provides satisfactory results.
The remainder of this section explains the sampling process and the quality metric.

\begin{figure}[tb!]
	\centering
	\subcaptionbox{$\mathcal{\hat{T}}_1$}[.242\linewidth][c]{
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Successful/albedo}

		\vspace{1mm}

		\includegraphics[width=1\linewidth]{figures/results/discriminator/Successful/albedo2}}
	\subcaptionbox{$\mathcal{D}(\mathcal{\hat{T}}_1)$}[.242\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Successful/disc}

		\vspace{1mm}

		\includegraphics[width=1\linewidth]{figures/results/discriminator/Successful/disc2}}
	\bigskip
	\subcaptionbox{$\mathcal{\hat{T}}_2$}[.242\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Non-Successful/albedo}

		\vspace{1mm}

		\includegraphics[width=1\linewidth]{figures/results/discriminator/Non-Successful/albedo2}}
	\subcaptionbox{$\mathcal{D}(\mathcal{\hat{T}}_2)$}[.242\linewidth][c]{%
		\includegraphics[width=1\linewidth]{figures/results/discriminator/Non-Successful/disc}

		\vspace{1mm}

		\includegraphics[width=1\linewidth]{figures/results/discriminator/Non-Successful/disc2}}
	\vspace{-5mm}
	\caption{Outputs of $\mathcal{D}$ with textures of different qualities. (a) The generated albedos $\mathcal{\hat{T}}_1$ are artifact-free on the search areas (b), thus the discriminator $\mathcal{D}(\mathcal{\hat{T}}_1)$ is fooled to believe $\mathcal{\hat{T}}_1$ are real. (c) Failed generated samples $\mathcal{\hat{T}}_2$ cannot fool the discriminator, which finds artifacts $\mathcal{D}(\mathcal{\hat{T}}_2)$ (d). Further examples can be found on the Supplementary Material. }
	\label{fig:discriminator}
	\vspace{-6mm}
\end{figure}

\paragraph*{\textbf{Sampling}}
By using a fully-convolutional GAN, our model can generate textures of any sizes, hardware being the only limiting factor. This is key for tileable texture synthesis as, even if a given texture is seamlessly tileable, larger textures require fewer repetitions to cover the same spatial area, which ultimately results in fewer repeating artifacts, as illustrated in Figure~\ref{fig:multiple_outputs}~(a).
There are two main challenges when finding tileable textures: the input texture needs to contain the distribution of the existing repeating patterns; and the input tile itself must not create strong artifacts when tiling the latent spaces of the generator.

Our goal is thus to find the largest possible tileable texture stack.
To do so, we sample multiple candidate crops for a given crop size $c \in \{c_{min}, c_{max}\}$, where  $c_{max}$ is the resolution of the input $\mathcal{T}$. We sample crop sizes starting at the largest possible size $c = c_{max}$, and stop when we find a suitable candidate according to the tileability metric $\mathcal{Q}$. As shown in
Figure~\ref{fig:multiple_outputs}~(b), this sampling mechanism also allow us to generate multiple tileable candidates for a single exemplar if we choose different parts of  $\mathcal{T}$ as input.

\begin{figure}[t!]
	\centering
	\includegraphics[width=\columnwidth]{figures/results/multiple_outputs_v4.pdf}
	\caption{SeamlessGAN can generate multiple tileable outputs from the same sample $\mathcal{T}$. By cropping different parts of $\mathcal{T}$, we can feed the generator $\mathcal{G}$ with different inputs $t_c$ (shown on the left), generating tileable textures $\hat{\mathcal{T}}_c$ of different sizes. On the example at the top (a), the synthesized images are tiled so they cover the same spatial extent, which shows that, even if the textures are tileable, larger textures generate fewer repeating artifacts. (b) Shows the tile resulting from sampling different parts of the input image. More results are included in the Supplementary Material.
	}
	\vspace{-5mm}
	\label{fig:multiple_outputs}
\end{figure}

\paragraph*{\textbf{Discriminator-guided Quality Function, $\mathcal{Q}$}}
The second component of our sampling strategy is the quality function used to determine whether a stack is tileable or not.
We observed that the artifacts
appear on vertical and horizontal frames around the center of the textures (Figure~\ref{fig:discriminator}). This is likely caused by strong discontinuities or gradients on the same areas of the tiled latent spaces, which the rest of the generative network fails to transform into realistic textures.
Following recent work on generative models~\cite{schonfeld2020u}, we use the discriminator $\mathcal{D}$ as a semantic-aware error metric that can be exploited for detecting local artifacts in the generated textures. This can be done in our case because the global loss function contains pixel-wise, \textit{style}, and adversarial losses. The adversarial loss learns the semantics of the texture, whereas the $\mathcal{L}_1$ and style losses model color distances or repeated patterns ~\cite{Rodriguez-Pardo2019AutomaticPatterns}.  This combination of loss functions allows to balance textural, semantic and perceptual color properties.

We thus design a quality evaluation function $\mathcal{Q}$ that estimates if the generated texture stack $\hat{\mathcal{T}}_c$ is \textit{tileable}, looking for artifacts on a central area, $S \in \mathcal{D}(\hat{\mathcal{T}}_c)$ of the discriminator. This area is composed of two regions, $S = s_v \cup s_h $, where $s_v$ is a vertical area, and $s_h$ is an horizontal one, both centered on the output of the discriminator (Figure~\ref{fig:overview}).
The function $\mathcal{Q}$ leverages the fact that
$\mathcal{D}$ outputs $0$ when it believes a patch to be synthetic. %
However, as the values are sample-dependent, we establish a threshold $\tau \in \mathbb{R}$ using the values of the rest of the image $\overline{S}$ as a reference
$\tau = \gamma \cdot \min(S_r)$, where
$S_r = \mathcal{D}(\hat{\mathcal{T}}_c) \cap \overline{S}$ is the remaining part of the image, and $\gamma \in \mathbb{R}$ is a threshold that allows to control the sensitiveness of $\mathcal{Q}$.
Consequently, $\mathcal{Q}(\hat{\mathcal{T}}_c)$ is $1$ if $\min(s_v)$ and $\min(s_h)$ are greater or equal than $\tau$, considering the texture as \textit{tileable}, and $0$ otherwise. The goal of $\mathcal{Q}$ is to estimate whether or not the central areas, where discontinuities may be present, are as realistic as the rest of the texture.  It works as a classification function which returns a high value if the texture is identified as real by the discriminator in such areas. This allows us to create a sampling strategy that distinguishes between images with local artifacts from seamless textures. By using minimum values instead of the mean estimation of the discriminator, our quality metric focuses on detecting artifacts which may arise during the latent space tiling operation.

Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG); Multimedia (cs.MM)