Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

This paper proposes Markovian Generative Adversarial Networks (MGANs), a method for training generative neural networks for efficient texture synthesis. While deep neural network approaches have recently demonstrated remarkable results in terms of synthesis quality, they still come at considerable computational costs (minutes of run-time for low-res images). Our paper addresses this efficiency issue. Instead of a numerical deconvolution in previous work, we precompute a feed-forward, strided convolutional network that captures the feature statistics of Markovian patches and is able to directly generate outputs of arbitrary dimensions. Such network can directly decode brown noise to realistic texture, or photos to artistic paintings. With adversarial training, we obtain quality comparable to recent neural texture synthesis methods. As no optimization is required any longer at generation time, our run-time performance (0.25M pixel images at 25Hz) surpasses previous neural texture synthesizers by a significant margin (at least 500 times faster). We apply this idea to texture synthesis, style transfer, and video stylization.

Introduction
Image synthesis is a classical problem in computer graphics and vision [6,33].
The key challenges are to capture the structure of complex classes of images in a concise, learnable model, and to find efficient algorithms for learning such models and synthesizing new image data. Most traditional "texture synthesis" methods address the complexity constraints using Markov random field (MRF) models that characterize images by statistics of local patches of pixels.
Recently, generative models based on deep neural networks have shown exciting new perspectives for image synthesis [10,8,11]. Deep architectures capture appearance variations in object classes beyond the abilities of pixel-level approaches. However, there are still strong limitations of how much structure can be learned from limited training data. This currently leaves us with two main classes of "deep" generative models: 1) full-image models that generate whole images [10,3], and 2) Markovian models that also synthesize textures [8,21].
The first class, full-image models, are often designed as specially trained auto-encoders [16,11]. Results are impressive but limited to rather small images arXiv:1604.04382v1 [cs.CV] 15 Apr 2016 (typically around 64×64 pixels) with limited fidelity in details. The second class, the deep Markovian models, capture the statistics of local patches only and assemble them to high-resolution images. Consequently, the fidelity of details is good, but additional guidance is required if non-trivial global structure should be reproduced [6,12,1,8,21]. Our paper addresses this second approach of deep Markovian texture synthesis.
Previous neural methods of this type [8,21] are built upon a deconvolutional framework [37,25]. This naturally provides blending of patches and permits reusing the intricate, emergent multi-level feature representations of large, discriminatively trained neural networks like the VGG network [30], repurposing them for image synthesis. As a side note, we will later observe that this is actually crucial for high-quality result ( Figure 10). Gatys et al. [8] pioneer this approach by modeling patch statistics with a global Gaussian models of the higher-level feature vectors, and Li et al. [21] utilize dictionaries of extended local patches of neural activation, trading-off flexibility for visual realism.
Deep Markovian models are able to produce remarkable visual results, far beyond traditional pixel-level MRF methods. Unfortunately, the run-time costs of the deconvolution approach are still very high, requiring iterative back-propagation in order to estimate a pre-image (pixels) of the feature activations (higher network layer). In the most expensive case of modeling MRFs of higher-level feature patches [21], a high-end GPU needs several minutes to synthesize low-resolution images (such as a 512-by-512 pixels image).
The objective of our paper is therefore to improve the efficiency of deep Markovian texture synthesis. The key idea is to precompute the inversion of the network by fitting a strided 1 convolutional network [31,29] to the inversion process, which operates purely in a feed-forward fashion. Despite being trained on patches of a fixed size, the resulting network can generate continuous images of arbitrary dimension without any additional optimization or blending, yielding a high-quality texture synthesizer of a specific style and high performance 2 .
We train the convolutional network using adversarial training [29], which permits maintaining image quality similar to the original, expensive optimization approach. As result, we obtain significant speed-up: Our GPU implementation computes 512 × 512 images within 40ms (on an nVidia TitanX). The key limitation, of course, is to precompute the feed-forward convolutional network for each texture style. Nonetheless, this is still an attractive trade-off for many potential applications, for example from the area of artistic image or video stylization. We explore some of these applications in our experiments. 1 A strided convolutional network hence replaces pooling layers by subsampled convolution filters that learn pooling during training (for example, two-fold mean pooling is equivalent to blurred convolution kernels sampled at half resolution  [25]. The astonishing complexity of the obtained visual patterns has immediately spurred hope for new generative models: Gatys et al. [8,7] drove deconvolution by global covariance statistics of feature vectors on higher network layers, obtaining unprecedented results in artistic style transfer. The statistical model has some limitations: Enforcing per-feature-vector statistics permits a mixing of feature patterns that never appear in actual images and limit plausibility of the learned texture. This can be partially addressed by replacing point-wise feature statistics by statistics of spatial patches of feature activations [21]. This permits photo-realistic synthesis in some cases, but also reduces invariance because the simplistic dictionary of patches introduces rigidity. On the theory side, Xie et al. [34] have proved that a generative random field model can be derived from used discriminative networks, and show applications to unguided texture synthesis. Full image methods employ specially trained auto-encoders as generative networks [16]. For example, the Generative Adversarial Networks (GANs) use two networks, one as the discriminator and other as the generator, to iteratively improve the model by playing a minimax game [10]. This model is extended to work with a Laplacian pyramid [9], and with a conditional setting [3]. Very recently, Radford et al. [29] propose a set of architectural refinements 3 that stabilized the performance of this model, and show that the generators have vector arithmetic properties. One important strength of adversarial networks is that it offers perceptual metrics [20,4] that allows auto-encoders to be training more efficiently. These models can also be augmented semantic attributes [35], image captions [24], 3D data [5,17], spatial/temporal status [11,13,27] etc.
In very recent, two concurrent work, Ulyanov et al. [32] and Johnson et al. [14] propose fast implementations of Gatys et al's approach. Both of their methods employ precomputed decoders trained with a perceptual texture loss and obtain significant run-time benefits (higher decoder complexity reduces their speed-up a bit). The main conceptual difference in our paper is the use of Li et al.'s [21] feature-patch statistics as opposed to learning Gaussian distributions of individual feature vectors, which provides some benefits in terms of reproducing textures more faithfully. Fig. 1: Motivation: real world data does not always comply with a Gaussian distribution (first), but a complex nonlinear manifold (second). We adversarially learn a mapping to project contextually related patches to that manifold.

Model
Let us first conceptually motive our method. Statistics based methods [8,32] match the distributions of source (input photo or noise signal) and target (texture) with a Gaussian model (Figure 1, first). They do not further improve the result once two distributions match. However, real world data does not always comply with a Gaussian distribution. For example, it can follow a complicated non-linear manifold. Adversarial training [10] can recognize such manifold with its discriminative network ( Figure 1, second), and strengthen its generative power with a projection on the manifold (Figure 1, third). We improve adversarial training with contextually corresponding Markovian patches (Figure 1, fourth). This allows the learning to focus on the mapping between different depictions of the same context, rather than the mixture of context and depictions. Figure 2 visualizes our pipeline, which extends the patch-based synthesis algorithm of Li et al. [21]. We first replace their patch dictionary (including the iterative nearest-neighbor search) with a continuous discriminative network D (green blocks) that learns to distinguish actual feature patches (on VGG 19 layer Relu3 1, purple block) from inappropriately synthesized ones. A second comparison (pipeline below D) with a VGG 19 encoding of the same image on the higher, more abstract layer Relu5 1 can be optionally used for guidance. If we run deconvolution on the VGG networks (from the discriminator and optionally from the guidance content), we obtain deconvolutional image synthesizer, which we call Markovian Deconvolutional Adversarial Networks (MDANs).
MDANs are still very slow; therefore, we aim for an additional generative network G (blue blocks; a strided convolutional network). It takes a VGG 19 layer Relu4 1 encoding of an image and directly decodes it to pixels of the synthesis image. During all of the training we do not change the VGG 19 network (gray blocks), and only optimize D and G. Importantly, both D and G are trained simultaneously to maximize the quality of G; D acts here as adversary to G. We denote the overall architecture by Markovian Generative Adversarial Networks (MGANs).

Markovian Deconvolutional Adversarial Networks (MDANs)
Our MDANs synthesize textures with a deconvolutional process that is driven by adversarial training: a discriminative network D (green blocks in Figure 2) Fig. 2: Our model contains a generative network (blue blocks) and a discriminative network (green blocks). We apply the discriminative training on Markovian neural patches (purple block as the input of the discriminative network.). is trained to distinguish between "neural patches" sampled from the synthesis image and sampled from the example image. We use regular sampling on layer relu3 1 of VGG 19 output (purple block). It outputs a classification score s = ±1 for each neural patch, indicating how "real" the patch is (with s = 1 being real). For each patch sampled from the synthesized image, 1 − s is its texture loss to minimize. The deconvolution process back-propagates this loss to pixels. Like Radford et al. [29] we use batch normalization (BN) and leaky ReLU (LReLU) to improve the training of D.
Formally, we denote the example texture image by x t ∈ R wt×ht , and the synthesized image by x ∈ R w×h . We initialize x with random noise for un-guided synthesis, or an content image x c ∈ R w×h for guided synthesis. The deconvolutio iteratively updates x so the following energy is minimized: Here E t denotes the texture loss, in which Φ(x) is x's feature map output from layer relu3 1 of VGG 19. We sample patches from Φ(x), and compute E t as the Hinge loss with their labels fixed to one: Here s i denotes the classification score of i-th neural patch, and N is the total number of sampled patches in Φ(x). The discriminative network is trained on the fly: Its parameters are randomly initialized, and then updated after each deconvolution, so it becomes increasingly smarter as synthesis results improve. The additional regularizer Υ (x) in Eq. 1 is a smoothness prior for pixels [23]. Using E t and Υ (x) can synthesize random textures ( Figure 3). By minimizing an additional content loss E c , the network can generate an image that is contextually related to a guidance image x c (Figure 4). This content loss is the Mean Squared Error between two feature maps Φ(x) and Φ(x c ). We set the weights with α 1 = 1 and α 2 = 0.0001, and minimize Equation 1 using back-propagation Fig. 3: Un-guided texture synthesis using MDANs. For each case the first image is the example texture, and the other two are the synthesis results. Image credits: [34]'s "Ivy", flickr user erwin brevis's "gell", Katsushika Hokusai's "The Great Wave off Kanagawa", Kandinsky's "Composition VII". with ADAM [15] (learning rate 0.02, momentum 0.5). Notice each neural patch receives its own output gradient through the back-propagation of D. In order to have a coherent transition between adjacent patches, we blend their output gradient like texture optimization [18] did.

Markovian Generative Adversarial Networks (MGANs)
MDANs require many iterations and a separate run for each output image. We now train a variational auto-encoder (VAE) that decodes a feature map directly to pixels. The target examples (textured photos) are obtained from the MDANs. Our generator G (blue blocks in Figure 2) takes the layer relu4 1 of VGG 19 as the input, and decodes a picture through a ordinary convolution followed by a cascade of fractional-strided convolutions (FS Conv). Although being trained with fixed size input, the generator naturally extends to arbitrary size images.
As Dosovitskiy et al. [4] point out, it is crucially important to find a good metric for training an auto-encoder: Using the Euclidean distance between the synthesized image and the target image at the pixel level ( Figure 5, pixel VAE) yields an over-smoothed image. Comparing at the neural encoding level improves results ( Figure 5, neural VAE), and adversarial training improves the reproduction of the intended style further (Figure 5, MGANs).
Our approach is similar to classical Generative Adversarial Networks (GANs) [10], with the key difference of not operating on full images, but neural patches from the same image. Doing so utilizes the contextual correspondence between the patches, and makes learning easier and more effective in contrast to learning the distribution of a object class [10] or a mapping between contextually irrelevant data [32]. In additional we also replace the Sigmoid function and the binary cross entropy criteria from [29] by a max margin criteria (Hinge loss). This avoids the vanishing gradient problem when learning D. This is more problematic in our case than in Radfort et al.'s [29] because of less diversity in our training data. Thus, the Sigmoid function can be easily saturated. Figure 5 (MGANs) shows the results of a network that is trained to produce paintings in the style of Picasso's "Self-portrait 1907". For training, we randomly selected 75 faces photos from the CelebA data set [22], and in additional to it 25 non-celebrity photos from the public domain. We resize all photos so that the maximum dimension is 384 pixels. We augmented the training data by generating 9 copies of each photo with different rotations and scales. We regularly sample subwindows of 128-by-128 croppings from them for batch processing. In total we have 24,506 training examples, each is treated as a training image where neural patches are sampled from its relu3 1 encoding as the input of D. Figure 5 (top row, MGANs) shows the decoding result of our generative network for a training photo. The bottom row shows the network generalizes well to test data. Notice the MDANs image for the test image is never used in the training. Nonetheless, direct decoding with G produces very good approximation of it. The main difference between MDANs and MGANs is: MDANs preserve the content of the input photo better and MGANs produce results that are more stylized. This is because MGANs was trained with many images, hence learned the most frequent features. Another noticeable difference is MDANs  create more natural backgrounds (such as regions with flat color), due to its iterative refinement. Despite such flaws, the MGANs model produces comparable results with a speed that is 25,000 times faster. Figure 6 shows some intermediate results MGANs. It is clear that the decoder gets better with more training. After 100 batches, the network is able to learn the overall color, and where the regions of strong contrast are. After 300 batches the network started to produce textures for brush strokes. After 1000 batches it learns how to paint eyes. Further training is able to remove some of the ghosting artifacts in the results. Notice the model generalizes well to testing data (right).

Experimental Analysis
We conduct empirical experiments with our model: we study parameter influence (layers for classification, patch size) and the complexity of the model (number of layers in the network, number of channels in each layer). While there may not be a universal optimal design for all textures, our study shed some light on how Fig. 8: Different layers and patch sizes for training the discriminative netowrk. Input image credit: "ropenet" from the project link of [19]. the model behaves for different cases. For fair comparison, we scale the example textures in this study to fixed size (128-by-128 pixels), and demand the synthesis output to be 256-by-256 pixels.
Visualizing decoder features: We visualize the learned filters of decoder G in Figure 7. These features are directly decoded from a one-hot input vector. Individual patches are similar to, but not very faithfully matching the example textures (reconfirming the semi-distributed and non-linear nature of the encoding). Nonetheless, visual similarity of such artificial responses seems strong enough for synthesizing new images.
Parameters: Next, we study the influence of changing the input layers for the discriminative network. To do so we run unguided texture synthesis with discriminator D taking layer relu2 1, relu3 1, and relu4 1 of VGG 19 as the input. We use patch sizes of 16, 8 and 4 respectively for the three options, so they have the same receptive field of 32 image pixels (approximately; ignoring padding). The first three results in Fig. 8 shows the results of these three settings. Lower layers (relu2 1 ) produce sharper appearances but at the cost of losing form and structure of the texture. Higher layer (relu4 1 ) preserves coarse structure better (such as regularity) but at the risk of being too rigid for guided scenarios. Layer relu3 1 offers a good balance between quality and flexibility. We then show the influence of patch size: We fix the input layer of D to be relu3 1, and compare patch size of 4 and 16 to with the default setting of 8. The last two results in Fig. 8 shows that such change will also affect the rigidity of the model: smaller patches increase the flexibility and larger patches preserve better structure.
Complexity: We now study the influence of 1) the number of layers in the networks and 2) the number of channels in each layer. We first vary the D by removing the convolutional layer. Doing so reduces the depth of the network and in consequence the synthesis quality (first column, Fig. 9). Bringing this convolutional layer back produces smoother synthesis (second column, Fig. 9). However, in these examples the quality does not obviously improves with more additional layers (third column, Fig. 9).
Testing the D with 4, 64, and 128 channels for the convolutional layer, we observe in general that decreasing the number of channels leads to worse results (fourth column, Fig. 9), but there is no significance difference between 64 channels and 128 channels (second column v.s. fifth column). The complexity requirements also depend on the actual texture. For example, the ivy texture is Fig. 9: Different depths for training the discriminative netowrk. The input textures are "ropenet" from the project link of [19], [34]'s "Ivy", and Pablo Picasso's "self portrait 1907". a rather simple MRF, so the difference between 4 channel and 64 channel are marginal, unlike in the other two cases.
Next, we fix the discriminative network and vary the complexity of the generative network. We notice some quality loss when removing the first convolutional layer from the decoder, or reducing the number of channels for all layers, and only very limited improvement from a more complex design. However the difference is not very significant. This is likely because the networks are all driven by the same discriminative network, and the reluctance of further improvement indicates there are some non-trivial information from the deconvolutional process that can not be recovered by a feed forward process. In particular, the fractionally-strided convolutions does not model the nonlinear behaviour of the max-pooling layer, hence often produces alias patterns. These become visible in homogeneous, texture-less area. To avoid artifacts but encourage texture variability, we can optionally add Perlin noise [28] to the input image.
Initialization Usually, networks are initialized with random values. However we found D has certain generalization ability. Thus, for transferring the same texture to different images with MDANs, a previously trained network can serve as initialization. Figure 10 shows initialization with pre-trained discriminative network (that has already transferred 50 face images) produces good result with only 50 iterations. In comparison, random initialization does not produce comparable quality even after the first 500 iterations. It is useful to initialize G with an auto-encoder that directly decodes the input feature to the original input photo. Doing so essentially approximates the process of inverting VGG 19, and let the whole adversarial network to be trained more stably.
The role of VGG: We also validate the importance of the pre-trained VGG 19 network. As the last two pictures in Figure 10 show, training a discriminative network from scratch (from pixel to class label [29]) yields significantly worse results. This has also been observed by Ulyanov et al. [32]. Our explanation is that much of the statistical power of VGG 19 stems from building shared feature cascades for a diverse set of images, thereby approaching human visual perception more closely than a network trained with a limited example set.

Results
This section shows examples of our MGANs synthesis results. We train each model with 100 randomly selected images from ImageNet, and a single example texture. We first produce 100 transferred images using the MDANs model, then regularly sample 128-by-128 image croppings as training data for MGANs. In total we have around 16k samples for each model. The training take as about 12 min per epoch. Each epoch min-batches through all samples in random order. We train each texture for upto five epochs. Figure 11 compares our synthesis results with previous methods. First, our method has a very different character in comparison to the methods that use global statistics [32,8]: It transfers texture more coherently, such as the hair of Lena was consistently mapped to dark textures. In contrast, the Gaussian model [32,8] failed to keep such consistency, and have difficulty in transferring complicated image content. For example the eyes in [32]'s result and the entire face in [8]'s result are not textured. Since these features do not fit a Gaussian distribution, they are difficult to be constrained by a Gram matrix. The other local patch based approach [21] produces the most coherent synthesis, due to the use of non-parametric sampling. However, their method requires patch matching so is significantly slower (generate this 384-by-384 picture in 110 seconds). Our method and Ulyanov et al. [32] run at the same level of speed; both bring significantly improvement of speed over Gatys et al. [8] (500 times faster) and Li et al. [21] (5000 times faster).  [32], Gatys et al. [8] and input images are from [32].  [32]. Results of [32] and input images are from [32]. Figure 12 further discuss the difference between the Gaussian based method [32] and our method 4 . In general [32] produces more faithful color distributions in respect to the style image. It also texture background better (see the starry night example) due to the learning of mapping from noise to Gaussian distribution. On the other hand, our method produces more coherent texture transfer and does not suffer the incapability of Gaussian model for more complex scenarios, such as the facade in both examples. In comparison [32] produces either too much or too little textures in such complex regions. Figure 13 shows that unguided texture synthesis is possible by using the trained model to decode noise input. In this case, Perlin noise 5 images are 4 Since Ulyanov et al. [32] and Johnson et al. [14] are very similar approaches, in this paper we only compare to one of them [32]. The main differences of [14] are: 1) using a residual architecture instead of concatenating the outputs from different layers; 2) no additional noise in the decoding process. 5 We need to use "brown" noise with spectrum decaying to the higher frequencies because flat "white" noise creates an almost flat response in the encoding of the VGG network. Somer lower-frequency structure is required to trigger the feature detectors in the discriminative network. forwarded through VGG 19 to generate feature maps for the decoder. To our surprise, the model that was trained with random ImageNet images is able to decode such features maps to plausible textures. This again shows the generalization ability of our model. Last, Figure 13 shows our video decoding result. As a feed-forward process our method is not only faster but also relatively more temporally coherent than the deconvolutional methods. Fig. 15: Left: speed comparison between our method and Ulyanov et al. [32]. Right: speed comparison (in log space) between our method and Gatys et al. [8], Li et al. [21], and Ulyanov et al. [32]. The feed-forward methods (ours and [32]) are significantly faster than Gatys et al. [8] (500 times speed up) and Li et al. [21] (5000 times speed up).
Last but not the least, we provide details for the time/memory usage of our method. The time measurement is based on a standard benchmark framework [2] ( Figure 15): Our speed is at the same level as the concurrent work by Ulyanov et al. [32], who also use a feed-forward approach, perform significantly faster than previous deconvolution based approaches [8,21]. More precisely, both our method and Ulyanov et al. [32] are able to decode 512-by-512 images at 25Hz (Figure 15, left), while [32] leads the race by a very small margin. The time cost of both methods scale linearly with the number of pixels in the image. For example, our method cost 10 ms for a 256-by-256 image, 40 ms for a 512-by-512 image, and 160 ms for a 1024-by-1024 image. Both methods show a very significant improvement in speed over previous deconvolutional methods such as Gatys et al. [8] and Li et al. [21] (Figure 15 right): about 500 times faster than Gatys et al. [8], and 5000 times faster than Li et al. [21]. In the meantime our method is also faster than most traditional pixel based texture synthesizers (which rely on expensive nearest-neighbor searching). A possible exceptions would be a GPU implementation of "Patch Match" [1], which could run at comparable speed. However, it provides the quality benefits (better blending, invariance) of a deepneural-network method (as established in previous work [8,21]).
Memory-wise, our generative model takes 70 Mb memory for its parameters(including the VGG network till layer Relu4 1). At runtime, the required memory to decode a image linearly depends on the image's size: for a 256-by-256 picture it takes about 600 Mb, and for a 512-by-512 picture it requires about 2.5 Gb memory. Notice memory usage can be reduced by subdividing the input photo into blocks and run the decoding in a scanline fashion. However, we do not further explore the optimization of memory usage in this paper.

Limitation
Our current method works less well with non-texture data. For example, it failed to transfer facial features between two difference face photos. This is because facial features can not be treated as textures, and need semantic understanding (such as expression, pose, gender etc.). A possible solution is to couple our model with the learning of object class [29] so the local statistics is better conditioned. For synthesizing photo-realistic textures, Li et al. [21] often produces better results due to its non-parametric sampling that prohibits data distortion. However, the rigidity of their model restricts its application domain. Our method works better with deformable textures, and runs significantly faster.
Our model has a very different character compared to Gaussian based models [8,32]. By capturing a global feature distribution, these other methods are able to better preserve the global "look and feels" of the example texture. In contrast, our model may deviate from the example texture in, for example, the global color distribution. However, such deviation may not always be bad when the content image is expected to play a more important role.
Since our model learns the mapping between different depictions of the same content, it requires features highly invariant features. For this reason we use the pre-trained VGG 19 network. This makes our method weaker in dealing with highly stationary backgrounds (sky, out of focus region etc.) due to their weak activation from VGG 19. We observed that in general statistics based methods [32,8] generate better textures for areas that has weak content, and our method works better for areas that consist of recognizable features. We believe it is valuable future work to combine the strength of both methods.
Finally, we discuss the noticeable difference between the results of MDANs and MGANs. The output of MGANs is often more consistent with the example texture, this shows MGANs' strength of learning from big data. MGANs has weakness in flat regions due to the lack of iterative optimization. More sophisticated architectures such as the recurrent neural networks can bring in state information that may improve the result.

Conclusion
The key insight of this paper is that adversarial generative networks can be applied in a Markovian setting to learn the mapping between different depictions of the same content. We develop a fully generative model that is trained from a single texture example and randomly selected images from ImageNet. Once trained, our model can decode brown noise to realistic texture, or photos into artworks. We show our model has certain advantages over the statistics based methods [32,8] in preserving coherent texture for complex image content. Once trained (which takes about an hour per example), synthesis is extremely fast and offers very attractive invariance for style transfer.
Our method is only one step in the direction of learning generative models for images. An important avenue for future work would be to study the broader framework in a big-data scenario to learn not only Markovian models but also include coarse-scale structure models. This additional invariance to image layout could, as a side effect, open up ways to also use more training data for the Markovian model, thus permitting more complex decoders with stronger generalization capability over larger classes. The ultimate goal would be a directly decoding, generative image model of large classes of real-world images.


extfieldsofstudy:
Engineering
Medicine
Computer Science

text:
A Reliable TTP-Based Infrastructure with Low Sensor Resource Consumption for the Smart Home Multi-Platform

With the ICT technology making great progress in the smart home environment, the ubiquitous environment is rapidly emerging all over the world, but problems are also increasing proportionally to the rapid growth of the smart home market such as multiplatform heterogeneity and new security threats. In addition, the smart home sensors have so low computing resources that they cannot process complicated computation tasks, which is required to create a proper security environment. A service provider also faces overhead in processing data from a rapidly increasing number of sensors. This paper aimed to propose a scheme to build infrastructure in which communication entities can securely authenticate and design security channel with physically unclonable PUFs and the TTP that smart home communication entities can rely on. In addition, we analyze and evaluate the proposed scheme for security and performance and prove that it can build secure channels with low resources. Finally, we expect that the proposed scheme can be helpful for secure communication with low resources in future smart home multiplatforms.

Introduction
Thanks to advancing ICT technology, we live in ubiquitous environment where people can get IT services anytime and anywhere. IoT is one of the representative technologies for the ubiquitous environment. By technology, many things are organically connected to the web to provide various services to the users [1][2][3]. It is applied to a wide range of areas including smart agriculture, smart homes, and connected cars [4,5]. Among the application areas, the smart home market has been growing exponentially all over the world as IoT smart devices develop and the services and R&D areas using smart devices expand [6]. Strategy Analytics reported that the world smart home market has grown by 10% annually and the market size will reach 115 billion US dollars by 2019 [7]. Harbor Research also forecasted that the number of IoT terminals installed around the world will be 8 billion units by 2020 and 47% of them will be installed in smart homes. In addition, Gartner expects that the number of smart home-related devices will reach 6.96 billion units by 2020. The world market demand for a new growth engine encourages many companies to participate in the smart home market. According to ADL, a strategic consulting company, the participants in the smart home market can be divided to: (1) electrical power distribution/building automation players; (2) smart building control specialists; (3) building application players; (4) household appliance players; (5) SW, IT, communication equipment players; and (6) service providers [8]. Because smart home services are based on fusion technologies and provided on multiple platforms, various heterogeneities exist in the smart home Sensors 2016, 16, 1036 2 of 15 service environment. Furthermore, various kinds of security threats are posed against user's privacy and service reliability as the smart home market grows. In February 2015, HP warned in its research report that most of the smart home IoT devices are vulnerable to security threats regarding password, encryption and authentication procedures, and can be easily exposed to cyber-crime because user has to provide personal information to use smart home IT devices [9]. According to a Symantec report on the security of smart home devices, they have authentication weakness [10]. Moreover, various sensors used for the smart home have low computing resources. They cannot process the complicated computation tasks required to perform mutual authentication and build secure channels. Even a service provider that has enough computing resources can face overheads in processing data from an exponentially increasing number of smart home sensors [11][12][13][14]. To provide secure and flexible services for the multiplatform smart home environment, the infrastructure is required to connect with multiplatforms and to support low resources. This paper proposes an infrastructure based on Trusted Third Party (TTP) by which mutual authentication and communication can be secured in a multiplatform smart home environment and a scheme to design authentication and secure channels with minimum resources, using PUFs to solve the abovementioned problems.
This study is composed as follows: Section 2 will describe the smart home infrastructure, security smart home requirements, and previous research on smart homes. Section 3 will describe the proposed smart home infrastructure and a protocol for mutual authentication and secure channels in details. Section 4 provides an analysis of the computing resources and storage resources of the proposed scheme, while they will be discussed in Section 5.

Related Works
In this section, we discuss the infrastructure of and security requirements for smart homes, and review previous related works. the smart home service environment. Furthermore, various kinds of security threats are posed against user's privacy and service reliability as the smart home market grows. In February 2015, HP warned in its research report that most of the smart home IoT devices are vulnerable to security threats regarding password, encryption and authentication procedures, and can be easily exposed to cybercrime because user has to provide personal information to use smart home IT devices [9]. According to a Symantec report on the security of smart home devices, they have authentication weakness [10]. Moreover, various sensors used for the smart home have low computing resources. They cannot process the complicated computation tasks required to perform mutual authentication and build secure channels. Even a service provider that has enough computing resources can face overheads in processing data from an exponentially increasing number of smart home sensors [11][12][13][14]. To provide secure and flexible services for the multiplatform smart home environment, the infrastructure is required to connect with multiplatforms and to support low resources. This paper proposes an infrastructure based on Trusted Third Party (TTP) by which mutual authentication and communication can be secured in a multiplatform smart home environment and a scheme to design authentication and secure channels with minimum resources, using PUFs to solve the abovementioned problems. This study is composed as follows: Section 2 will describe the smart home infrastructure, security smart home requirements, and previous research on smart homes. Section 3 will describe the proposed smart home infrastructure and a protocol for mutual authentication and secure channels in details. Section 4 provides an analysis of the computing resources and storage resources of the proposed scheme, while they will be discussed in Section 5.

Related Works
In this section, we discuss the infrastructure of and security requirements for smart homes, and review previous related works.

Smart Home Infrastructure
In general, the smart home infrastructure consists of sensors and a gateway, and service providers, as seen in Figure 1. In the smart home, those sensors provide a single service directly to home occupants, using information they collect in real time, or multiple services through exchanging information with other sensors [15]. When the sensors transmit collected information to a service provider, they have to go through a gateway to communicate with the outside because they have low

Smart Home Infrastructure
In general, the smart home infrastructure consists of sensors and a gateway, and service providers, as seen in Figure 1. In the smart home, those sensors provide a single service directly to home occupants, using information they collect in real time, or multiple services through exchanging information with other sensors [15]. When the sensors transmit collected information to a service provider, they have to go through a gateway to communicate with the outside because they have low resources for use. The gateway functions in two ways: it receives the data the home sensors collect in the smart home and sends the date to service providers or transmits the data from the service providers to the sensors. In addition, the gateway directly sends a control command to the sensors on the basis of the received information and depending on the type of service. Service provider is a server registered for service provision before a sensor is deployed in the smart home. It receives information sensed by a sensor through the gateway and gives a control command or service. Since the smart home infrastructure is mainly built with sensors, gateway and service providers, various and numerous various industries are involved [6], creating a multiplatform environment.

Security Requirements for Smart Home
A variety of security threats exist in smart home environment, just like in existing ICT environments. Therefore, it is necessary to ensure the security and privacy of smart home services. In designing a smart home environment, particular consideration needs to be paid for the multiplatform heterogeneity and limited resource requirements for sensors of low capacity.
Multiplatform: In the smart home, various industries are involved such as server, control and communication required inside of the smart home, as well as smart sensor manufacturing. Therefore, the sensors and the service providers exchange information on heterogeneous smart home multiplatforms [16][17][18][19]. Allseen Alliance, OIC, Thread Group, HomeKit, IIC have currently suggested global standards to improve the multiplatform-based smart home environment, but none of them stands out as a market leader yet. Moreover, since there is no preemptive standard in the market although each standard provides the users and developers with their own security tools and managers, there are lots of challenges to connect with other platforms in security aspects. In such a smart home environment, heterogeneous sensors and/or service providers have to organically communicate with each other in a multiplatform environment so as to provide uniformed or common services to home occupants.
Security: As IoT technology has grown geometrically, so have security threats grown proportionally. Particularly, the smart home is a complex environment combined with various services. Therefore, it can easily be vulnerable to various security threats and malicious attacks can cause serious damage or even threaten the lives of the occupants in the home environment [20]. Access control by mutual authentication must be in place to protect a user from malicious attacks and ensure secure communication [21]. Furthermore, a security channel has to be established, besides mutual authentication, to protect communications from threats such as relay attacks, replay attacks, leaked keys, and forward security. As a way to build a secure sensor environment, Tien-Dung et al. [22] and Mohammad et al. [23] proposed ELK and LHK, respectively, but they are not suitable for the smart home environment. In addition, Adrian et al.'s CoGKTK [24] and Wong et al.'s sGIM [25], which are based on the group multicast transmission technique, also turned out to be unsecure to various attacks.
Privacy: In general, the low capacity smart home sensors exchange information with the service providers through the gateway to provide services. Especially while they are communicating, the personal information of home occupants can leak out due to malicious eavesdropping from outside or inside entities, or user's carelessness, which can eventually result in the breach of privacy to a great extent [26]. Therefore, it is necessary to build a secure channel for each interrelation of the components in the smart home-between the smart home sensors and the gateway; between the gateway and the service providers; between the smart home sensors and the service providers-to protect the occupants' privacy from internal/external security threats and minimize unintended information exposure.
Low Resources: A variety of sensors are used in the smart home. Some of them have abundant computing resources. Such sensors are used in refrigerators, washing machines, and TVs, while others have limited power or computing resources [27,28]. The latter are mainly used for electric lights, thermometers, alarm clocks, and so on. The low resource sensors use small batteries and low-capacity chips to balance performance and cost [29]. As a result, smart home sensors can neither perform complicated computations nor store much data due to their small memory capacity. Particularly, this resource limitation of a sensor is a great hurdle to creating secure channels to protect a user from malicious attacks or breaches of privacy. This is why a light-weighted method, by which a secure channel can be built only with simple computation and low memory, is required for a smart home sensor.

Previous Research on Smart Home
In this section, previous studies related to smart home service and security are reviewed. Jinsung et al. [30] proposed a ZigBee-based intelligent self-adjusting sensor (ZiSAS) as a way to improve the operational efficiency of wireless sensor networks (WSNs), which aims to balance the cost and performance of a sensor in the smart home environment. With consideration of the cost and performance constraints of a smart home sensor, the ZigBee (IEEE 802.15.4 standard) ZiSAS provides four functions as follows to provide efficient intelligent service: first, it provides a flexible middleware architecture for the smart home environment. The flexible middleware architecture consists of three layers in which a sensor can provide services dynamically in the smart home environment. Second, it provides situation-based sensor self-adjusting function to relieve the WSN's hardware/software limitations. Third, it provides event-based sensor control in the smart home environment. Event-based sensor control prevents unnecessary information gathering from all the sensors. Finally, it provides occupants with context-aware services based on sensors' continuous collection of environmental information and analysis of current situations. In addition, Jinsung et al. also showed prototype smart home services using the proposed system. However, it has security and privacy protection problems because not only general data but also the movement of residents detected in ZiSAS are not encrypted to be exchanged. In addition, it has the problem of not supporting multiplatform use without considering the multiplicity among the heterogeneous sensors.
Alessandro et al. [31] proposed a wireless architecture in which the presence, movement, and behaviors of occupants in a smart home are estimated; the power of the smart home is managed; and elderly people are monitored. The proposed wireless architecture is a flexible wireless architecture aiming to satisfy both user acceptance and system performance demands based on the collection of large-scale data sets and training regarding smart home occupants' location and behavior. In the flexible wireless architecture, heterogeneous wireless devices also abstract information physically collected from other devices through the abstraction layers of software stacks in the multiplatform environment, so that service developers can get access to physical data without worrying about hardware compatibility and provide services at the upper layer. In addition, they proposed an integrated and low-cost wireless architecture to guarantee two important key points for future smart homes: user acceptance and low system complexity. However, the proposed model is not encrypted in spite of the sensitive information about the movement of the residents measured in the smart home, therefore, it has the possible issue of violating user privacy and it does not use secure channels for the communication among the sensors.
The smart home has several issues: smart home sensors of low power and low computing resources; the heterogeneity of devices that have to communicate in a multiplatform environment; and, as mentioned above, possible breaches of privacy by malicious attacks or unauthorized access from users' reckless management. To provide secure remote access to home automation networks 'anywhere and anytime', Khusvinder et al. [32] proposed a remote access framework based on a Remote Home Server (RHS) provided by a trusted third party. The proposed framework is a hybrid scheme of direct and third party-based remote communication approaches. It overcomes the weaknesses of existing direct access and third party-based access schemes by a trusted third party having a none-home owner search the IP address of the automated home and allowing a remote user that gets an IP address from the RHS have a direct access to the home. In addition, the system enhances the advance security by distributing the master key shared between the home automation system and RHS as well as one between the home automation system and mobile clients, randomly creating session keys every session, and thus building a secure channel. In addition, the researchers proposed a ZigBee-based home gateway through which the home automation system is connected to the external Internet at low cost and with standalone devices.
As smart home appliances such as lights, smoke-alarms, power switches, baby monitors, and so on have increased in number exponentially, Vijay et al. [33] provided network-level protection, as a solution to maintain privacy and provide security from snooping or intrusion into the family's activities, that system can monitor network activities and detect suspicious behaviors. The proposed scheme applies dynamic security rules with software defined networking (SDN) technology that can block and/or quarantine devices dynamically, based on the home context such as time-of-day or occupancy of a house. Furthermore, they suggested Security Management Provider (SMP) as an external entity that develops, customize, and delivers user extra safeguards for smart home devices at the network level. It is a three-party architecture that consists of SMP role, ISP/home-router-vendor role, and consumer role providing security-as-a-service. Figure 2 shows the proposed infrastructure for multiplatform smart home environments. In the smart home, there are various smart home sensor devices, which are registered with different service providers (SPs) due to their different manufacturers or services and work on the multiplatform. Therefore, they cause the issue of heterogeneity. That is why the gateway exists to solve the heterogeneity problem and open a single communication channel to the outside. Outside the smart home, several SPs exist by sensor platform, form clouds, and communicate with the smart home sensors through the gateway. Finally, TTP exists to solve the heterogeneity between various sensors and SPs and secure the reliability of the communication entities. TTP is in a reliable relation with sensors and all the sensors have physically unclonable PUFs which can process challenge-response and key generation with low resources [34][35][36][37]. The proposed multiplatform smart home infrastructure processes, prior to building a secure channel, mutual authentication between the sensors and the gateway, between the gateway and SP(s), and between SP(s) and the sensors, which are involved in communication by entity. As smart home appliances such as lights, smoke-alarms, power switches, baby monitors, and so on have increased in number exponentially, Vijay et al. [33] provided network-level protection, as a solution to maintain privacy and provide security from snooping or intrusion into the family's activities, that system can monitor network activities and detect suspicious behaviors. The proposed scheme applies dynamic security rules with software defined networking (SDN) technology that can block and/or quarantine devices dynamically, based on the home context such as time-of-day or occupancy of a house. Furthermore, they suggested Security Management Provider (SMP) as an external entity that develops, customize, and delivers user extra safeguards for smart home devices at the network level. It is a three-party architecture that consists of SMP role, ISP/home-router-vendor role, and consumer role providing security-as-a-service. Figure 2 shows the proposed infrastructure for multiplatform smart home environments. In the smart home, there are various smart home sensor devices, which are registered with different service providers (SPs) due to their different manufacturers or services and work on the multiplatform. Therefore, they cause the issue of heterogeneity. That is why the gateway exists to solve the heterogeneity problem and open a single communication channel to the outside. Outside the smart home, several SPs exist by sensor platform, form clouds, and communicate with the smart home sensors through the gateway. Finally, TTP exists to solve the heterogeneity between various sensors and SPs and secure the reliability of the communication entities. TTP is in a reliable relation with sensors and all the sensors have physically unclonable PUFs which can process challenge-response and key generation with low resources [34][35][36][37]. The proposed multiplatform smart home infrastructure processes, prior to building a secure channel, mutual authentication between the sensors and the gateway, between the gateway and SP(s), and between SP(s) and the sensors, which are involved in communication by entity.

Proposed Protocols
Using TTP in the proposed multiplatform smart home environment, the efficient mutual authentication process consists of two stages: the provisioning stage and the authentication stage. In the provisioning stage, each service provider, smart home sensor devices and a gateway are registered on TTP. In the authentication stage, mutual authentication is carried out between smart home sensor devices and the gateway, between the gateway and service providers, and between smart home sensor devices and service providers and the session key with which a security channel can be established is allocated to smart home sensor devices and service providers. It is assumed that TTP, sensors, and TTP are in and gateway and TTP and SP are reliable. The parameters for the proposed protocol are listed in Table 1. Sensors and the gateway are registered on TTP and SP at the provisioning state as shown in Figure 3 prior to being deployed at the field. First, in the (a) sensor provisioning stage (Figure 1), a sensor sends to TTP, through a secure channel, the message asking it (TTP) to register its own ID and ID of TTP and SP that it wants to register. When the TTP is asked to register IDs from a sensor, it randomly selects Challenge vector C[c 1 , c 2 , . . . , c m ] and sends it to the sensor to construct the PUF DB of the relevant sensor. The sensor takes the Challenge vector C from TTP as input value to compute the Response vector R[r 1 , r 2 , . . . , r m ] through PUF and responds to TTP. Then TTP maps Challenge vector C and Response vector R, constructs the PUF DB for concerned sensor, transmits to the SP (that the sensor requested to register) its own ID, Challenge vector C[c 1 , c 2 , . . . , c m ] and ID of that sensor. SP maps the Challenge vector C[c 1 , c 2 , . . . , c m ] and ID of that sensor, constructs PUF Challenge DB, and then sends its own ID and confirmation message to that sensor through TTP, using a secure channel.
As seen in Figure 3b, the gateway sends to TTP along with a requesting message its own ID and IDs of TTP and SP that it wants to register at gateway provisioning stage. Then, TTP selects two large decimal numbers p and q, computes open key n = p * q, and send them back to the gateway so that the gateway and SP can perform zero-knowledge authentication [38]. The gateway selects secret key  1 , v 2 , . . . , v k ] = si 2 mod n (0 < I < k + 1), and sends them back to TTP [39]. TTP that receives the values from the sensor transmits to SP open key vector V[v 1 , v 2 , . . . , v k ] and gateway ID. Then SP that receives the values from TTP sends its own ID and a confirmation message to the sensor through TTP and gateway. As seen in Figure 3b, the gateway sends to TTP along with a requesting message its own ID and IDs of TTP and SP that it wants to register at gateway provisioning stage. Then, TTP selects two large decimal numbers p and q, computes open key n = p * q, and send them back to the gateway so that the gateway and SP can perform zero-knowledge authentication [38]. The gateway selects secret key vector S[s1, s2, … , sk] from 1 < s < n − 1 for zero-knowledge authentication, computes open key vector V[v1, v2, … , vk] = si 2 mod n (0 < I < k + 1), and sends them back to TTP [39]. TTP that receives the values from the sensor transmits to SP open key vector V[v1, v2, … , vk] and gateway ID. Then SP that receives the values from TTP sends its own ID and a confirmation message to the sensor through TTP and gateway.  Figure 4 shows the authentication phase in which each of the sensors, the gateway, TTP and SP perform mutual authentication on one another and construct a secure channel between the sensor and SP. To construct a secure channel with the SP registered at the provisioning stage, the sensor transmits the IDs of sensor, gateway, TTP and SP to the gateway along with a requesting message. Then, the gateway that receives a request from the sensor computes a Witness value x and x = r 2 mod n together, which are necessary to perform zero-knowledge proofs with SP and then sends to SP its own ID, IDs of TTP and SP, a requesting message, along with x. SP transmits the Challenge vector PC[pc1, pc2, … , pcj] to the gateway for Zero-knowledge Proofs and the gateway transmits to SP computed Response value y = (rs1 pc1 s2 pc2 ,,, sj pcj ) mod n that was computed using the private key of  Figure 4 shows the authentication phase in which each of the sensors, the gateway, TTP and SP perform mutual authentication on one another and construct a secure channel between the sensor and SP. To construct a secure channel with the SP registered at the provisioning stage, the sensor transmits the IDs of sensor, gateway, TTP and SP to the gateway along with a requesting message. Then, the gateway that receives a request from the sensor computes a Witness value x and x = r 2 mod n together, which are necessary to perform zero-knowledge proofs with SP and then sends to SP its own ID, IDs of TTP and SP, a requesting message, along with x. SP transmits the Challenge vector PC[pc 1 , pc 2 , . . . , pc j ] to the gateway for Zero-knowledge Proofs and the gateway transmits to SP computed Response value y = (rs 1 pc1 s 2 pc2 ", s j pcj ) mod n that was computed using the private key of S[s 1 , s 2 , . . . , s k ]. Then SP computes x' = (y 2 v 1 pc1 v 2 pc2 , v j pcj ) mod n by using received y value and public key and compares the original x value with the computed x' value [38][39][40]. If two values are identical, SP grants authentication to the gateway. S[s1, s2, … , sk]. Then SP computes x' = (y 2 v1 pc1 v2 pc2 , vj pcj ) mod n by using received y value and public key and compares the original x value with the computed x' value [38][39][40]. If two values are identical, SP grants authentication to the gateway. After SP authenticates the gateway, it selects C value from PUF Challenge DB matching a relevant sensor to perform PUF Challenge-Response with the ID of the sensor, and then transmits the C value to TTP and H(C') value (value of hashed C) to the gateway. The TTP that receives the C value from SP computes PUF(C) through the PUF DB to produce R, which is the PUF Response value, After SP authenticates the gateway, it selects C value from PUF Challenge DB matching a relevant sensor to perform PUF Challenge-Response with the ID of the sensor, and then transmits the C value to TTP and H(C') value (value of hashed C) to the gateway. The TTP that receives the C value from SP computes PUF(C) through the PUF DB to produce R, which is the PUF Response value, hashes R with the current time stamp value T T, and transmits H(R|T T ) to the gateway along with C and T T to the gateway. The gateway that receives the values from SP hashes C value and checks if the hashed C value is identical with H(C') value from SP. If the values are identical, the gateway authenticates SP.

Authentication Phase
After the gateway authenticates SP, the former transmits C, H(R|T T ) and T T from TTP to the sensor. The sensor that receives the values from the gateway produces R' = PUF(C) out of PUF embedded in the magnetic device and computes H(R'|T T ) to confirm whether it is identical with the received H(R'|T T ). If the values are same, the sensor grants authentication to the gateway [34][35][36].
The sensor that authenticates the gateway hashes the current time stamp value T D and R', selects random number N D, transmits H(R'|T D ), T D , and N D to TTP via the gateway. Then, it creates a session key SK = f(R') for SP through the SK generator () [37]. Then, TTP that receives the values from the sensor through the gateway computes H(R|T D ) with transmitted T D and confirms whether the two H(R|T D )s are identical. If they are identical, TTP sends to the gateway the sensor ID and a confirmation message to inform that the sensor is an authentic user and Confirm (message), ID S , R, and N D to SP and removes Challenge C and Response R pair from PUF DB. In addition, the gateway and SP that receive the values from TTP authenticate the sensor.
To get authenticated by the sensor and build a secure channel with it, the SP that receives the values from TTP removes the relevant Challenge C from PUF Challenge DB [36]. The SP generates session key SK = f(R) by using R value from TTP, encodes its own ID and the sensor ID with N D into SK, and transmits to the sensor E SK '(ID P |ID S |N D ), IDS, IDP through the gateway. The sensor that receives the values from SP via the gateway computes the D SK (E SK (ID P |ID S |N D )) using the session key it created itself. The sensor uses its own creation of the session key SK value to get the D SK (E SK (ID P |ID S |N D )). If the decrypted values turn out to be ID P |ID S |N D , the sensor authenticates SP and constructs a security channel using the shared SK. Table 2 shows whether the proposed as well as existing schemes support multiplatform, is designed to consider the security considering the low resources of the device with small computing and battery power, and whether it protects the user's privacy. Also, Table 3 shows how much the proposed as well as existing schemes are secure from the various security threats. The proposed protocol provides mutual authentication between sensor and gateway, gateway and SP and sensor and SP. In the proposed protocol, TTP acts as distributed KS that guides the sensors and SPs secure from relay attack, replay attack, eavesdropping, leaked key, and forward security threats, helps them authenticate each other, and share session keys. Besides, when compared with conventional system, the proposed system is more stable and ensures privacy, supporting the low cost of a smart home sensor and the multiplatform environment. Table 2. Comparative security analysis between smart home protocols.

Jinsung et al. [30]
Alessandro et al. [31] Khusvinder et al. [32] Vijay et al. [33] Proposed Scheme O: Support; ∆: not fully support; X: Not support. Multiplatform: In this smart home infrastructure, which ensure users' privacy, smart home sensors with PUF chips, gateway, and service providers have reliable TTP and organically communicate with each other through it. This infrastructure designed TTP to own PUF DB, so multiplatform service providers or sensors can build a secure channel through mutual authentication, without prior information. It designs the gateway to authenticate service providers and sensors with information it receives from TTP and to build a secure channel. As a result, the smart home infrastructure proposed in this paper solves the issue of heterogeneity and prior information exchange in a multiplatform environment.
Privacy: Privacy exposure in smart home environment is not an issue of a single individual, but a grave threat to his/her whole family. To tackle problems such as malicious eavesdropping or exposure attributable to user's recklessness, the proposed smart home infrastructure and protocol proposes building multi-security channels, not simply station-to-station security channels, between smart home sensors and the gateway, between the gateway and the service providers, and between service providers and home sensors, to prevent the leakage of privacy in advance.
Relay attacks: A malicious attacker may try to perform a relay attack between a sensor and the gateway or between the gateway and SP, but if the malicious attacker sends an authentication request to the gateway while disguising him/herself as a sensor, the attacker may get the values transmitted from SP, but they are not valid and useful as public values. PUF Response R values necessary for producing a session key are transmitted in a hashed state and can be produced from physically unclonable PUF. Therefore, it can't build a secure channel with a SP. Even when the malicious attacker disguises him/herself as an SP and then asks a sensor for authentication through the gateway, it cannot create a session key between a sensor and the SP because the attacker has to get PUF Response R values, with which a session key can be produced. Therefore, the session key between a sensor and SP cannot be produced.
Replay attacks: A malicious attacker may intercept and re-use for malicious authentication the messages between a sensor and TTP and/or between SPs by conducting replay attacks. In Zero-knowledge proofs which is conducted between the gateway and SP, however, it is impossible for a malicious attacker to perform a replay attack and get access to the message because the gateway computes Witness value x using a random value for authentication and furthermore PUF Challenge-Response pairs exchanged between sensors and SPs are removed every time they are used. In addition, the sensor does not use PUF DB. Although the values used are not removed from the DB, C values cannot be re-used because the Challenge C value is transmitted in a hashed state with a time stamp TT value from the TTP. In addition, when a sensor sends an authentication value to the SP, it sends hashed values with time stamp TT and random value secret ND. Therefore, re-use attacks are impossible.
Eavesdropping: A random user may eavesdrop the message exchanges at the mutual authentication stage. Especially, as communication is conducted wirelessly between sensors and the gateway, it can be a big security threat if eavesdropped message contains any confidential information. The clear text exchange in the proposed protocol is composed of public values for zero-knowledge proofs, ID for each authentication entity, non-reused Challenge C, random value, and time stamps, but the Response R value, which is sensitive data required for authentication and creation of session keys, is all hashed before being exchanged.
Leaked keys: When the sensor and SP exchange keys for the construction of a secure channel, and their keys are exposed, there can be a lot of problems from the possible exposure of the sensitive information exchanged during the communication. In our proposed protocol, the key is secure because the key required to construct the secure channel between sensor and SP is created through the PUF Response value, that the PUF of the sensor which can get the Response R is physically unclonable, and the server stores the Challenge C value only on PUF DB and receives the R mapped on the Challenge C from TTP only. In addition, when the sensor performs the authentication process with SP through TTP, it has no risk of confidential value exposure even if the TTP server or SP is hacked since it does not expose confidential value to the others by using zero knowledge proofs.
Mutual authentication: As the platforms of sensors and SP are uniform in a multiplatform smart home environment, various kinds of issues related to heterogeneity can take place. In the proposed protocol, a gateway is positioned between sensors and SPs as a uniform communication channel and it secures reliability among multiplatform devices by mutual authentication via TTP. The system is designed to provide the mutual authentication between sensor and gateway, gateway and SP, and SP and sensor so that each section of the communication process can be authenticated.
Forward security: A malicious attacker may try to take away current session keys between the sensors and SPs and restore past messages. In case the sensors and SPs repeat using a (same) session key at a regular interval, or use parameters that affect past and present in creating a session key, the attacker can restore past session keys, but the proposed protocol does not allow the re-use of PUF Response R values already used in producing sessions, but remove them from the DB every time it is used. Therefore, this system ensures forward security. Table 4 shows storage the resources calculated under the proposed protocol where there are x sensors, one gateway, Y SPs and one TTP at the provisioning phase and authentication phase. Here, it is assumed that x sensors (x < X) are registered on each SP. The sensor with lowest computing power among the sensors, gateway, TTP and SP does not perform complicated computations (in data volume) and often (in the number of operation), except for m-number of PUF computations in the initial registration process. Even for a gateway with medium computing power, the most complicated computation it performs is limited to Y SP and zero-knowledge proofs. Because SP and TTP have enough computing resources, so they perform more complicated computations than a sensor and a gateway do. However, the most complicated computation for a SP is to carry out zero knowledge proofs with the gateway. To TTP, also, (m + 1) * X PUF is the most complicated computation. Computing load is divided and allocated to gateway, SP, and TTP in the order by computing resources, and it was designed for them to perform mutual authentication and security channel building with minimum computation.

Computing Resource Analysis
For example, if there are 100 sensors in the smart home, the calculated amount of each sensor is constant as shown in Table 4 regardless of the number of sensors since it authenticates gateway and SP once. The calculated amount at the gateway is proportional to the number of sensors, so it can be O(100 i ) and SP as well according to zero knowledge proofs. It means that SP performs encryption process so as to do more calculations than gateway and TTP has the calculated amount of O(100 i ) in proportion of the number of sensors. Hence, the sensor has the lowest calculated amount followed by TTP, Gateway, and SP. Especially, TTP lowers the burden to perform low amount of calculation while it authenticates with the most sensors.  Table 5 shows the storage resources calculated under the proposed protocol where there are x sensors, one gateway, Y SPs and one TTP at the provisioning phase and authentication phase. The sensor of lowest storage capacity has its own ID and the IDs of gateway, TTP and SP to which they belong, and stores five additional parameters for the authentication and secure channels. The gateway owns its own ID, X sensor IDs and Y server IDs as well as two parameters for authentication. SP has its own ID and one respective ID for gateway and TTP, and x sensors' IDs. In addition, it possesses mx Challenge C values and x time stamp parameters to perform the authentication with sensors and build secure channels. TTP has its own ID and IDs for x sensors, Y SPs and a gateway, and mX challenge-response pairs to authenticate x sensors. By capacity, storage resources were allocated to sensor, gateway, SP, and TTP in this order and minimum storage capacity was allocated to a sensor lacking storage resources the most.

Storage Resource Analysis
If 100 sensors should save 128 bytes passwords in each entry to authenticate to TTP, SP, or gateway, SP, gateway, and TTP would require the storage space with 128 bytes * 100 units = 12,800 bytes by each. However, it does not have to store the passwords in TTP or SP by using zero knowledge proofs, so it can save storage space.

Conclusions
This paper deals with increasing heterogeneity in multiplatform scenarios and security threats as the smart home market is exponentially growing. It proposes a scheme by which mutual authentication and secure channels can be efficiently constructed with minimum resources. The proposed TTP-based infrastructure can not only provide reliability to multiplatform entities participating in remote communication requiring mutual authentication, but also help sensors of low computing capacity and SPs that have to process data from an ever-increasing number of sensors while minimizing resource consumption in carrying out authentication with the sensors. Using PUFs, it also minimizes the consumption of computing resources by enabling smart home sensors to process authentication and producing session keys without additional hardware. In this paper, it was confirmed through security assessment that the proposed system is secure from common threats. Also, the proposed scheme was analyzed and tested for its computing and storage resources needed by communication entity. At present, the smart home technology remains at the level of "connected home". However, if it is openly applied to markets such as mobile app stores, multiplatform environments for smart home will become more diversified than now, shifting away from one-way service provision as it is now. It is hoped that the scheme proposed in the paper will help build secure communication with fewer resources in future multiplatform smart home environment.


s2fieldsofstudy:
Computer Science
Mathematics

text:
Spatial Context based Angular Information Preserving Projection for Hyperspectral Image Classification

Dimensionality reduction is a crucial preprocessing for hyperspectral data analysis - finding an appropriate subspace is often required for subsequent image classification. In recent work, we proposed supervised angular information based dimensionality reduction methods to find effective subspaces. Since unlabeled data are often more readily available compared to labeled data, we propose an unsupervised projection that finds a lower dimensional subspace where local angular information is preserved. To exploit spatial information from the hyperspectral images, we further extend our unsupervised projection to incorporate spatial contextual information around each pixel in the image. Additionally, we also propose a sparse representation based classifier which is optimized to exploit spatial information during classification - we hence assert that our proposed projection is particularly suitable for classifiers where local similarity and spatial context are both important. Experimental results with two real-world hyperspectral datasets demonstrate that our proposed methods provide a robust classification performance.

I. INTRODUCTION
Hyperspectral imaging techniques have been widely used for a variety of applications such as mineral mapping, surveillance, and environmental monitoring. With its rich spectral information, hyperspectral data provide unique spectral patterns of the materials present in the scene. This capability makes hyperspectral data especially suitable for ground cover classification problems. Although hyperspectral data forms a high dimensional feature space, the information is often effectively represented in a much lower dimensional subspacehence, finding an appropriate subspace is crucial for effective hyperspectral data classification. . Several linear and non-linear dimensionality reduction methods have been proposed in literature to address this issue [1]- [12]. The objective function of such dimensionality reduction methods are typically utilize Euclidean distance information. Different from such Euclidean-based dimensionality reduction methods, angle or correlation-based dimensionality reduction methods have been proposed in literature to exploit specific benefits of angular information -something that is particularly relevant for hyperspectral data. In [13], correlation discriminant analysis (CDA) is proposed to transform the data so that the ratio of between-class and within-class sample correlation is maximized. In [14], the authors propose several supervised spectral angle based dimensionality reduction methods for robust hyperspectral data classification.
In typical remote sensing data classification applications, the process of collecting labeled training data is often very time consuming and expensive. Unlabeled data on the other hand is easily available and hence unsupervised dimensionality reduction methods that effectively learn the most appropriate subspace can hence be readily utilized. With that in mind, we propose an unsupervised counterpart of our recently proposed supervised subspace algorithm, the angular discriminant Analysis (ADA). This unsupervised counterpart is referred to as local similarity preserving projection (LSPP) in this paper. Unlike ADA which requires labeled training samples, LSPP does not require labeled training samples to learn the projection matrix. Additionally, it is well-known that utilizing spatial information in hyperspectral data can dramatically improve the classification accuracies because any such method accounts for the spatial variability of spectral content in local spatial neighborhoods. This follows from the observation that spatially neighboring pixels are highly likely to belong to the same class and have similar spectral characteristics. To incorporate such spatial information of hyperspectral data into our unsupervised projection, we develop a spatial information driven variant of LSPP (SLSPP) which effectively uses the spatial contextual information around each pixel in hyperspectral images to learn the optimal projection.
Recently, sparse representation-based classification (SRC) has been proposed for face recognition [15] and achieved a great amount of attention in various other applications such as digit recognition [16], speech recognition [17], gesture recognition [18], vehicle classification [19] as well as remote sensing data classification [20]- [22]. The central idea of SRC is to represent a test sample as a sparse linear combination of atoms in a dictionary which is formed by all of the available training samples. The class label of the test sample is assigned to a class whose reconstruction error is the lowest. In [21], the authors propose a join sparsity model to incorporate the contextual information of test samples to improve the classification performance of SRC. However, the contextual information of training samples have not been used. Besides the proposed dimensionality reduction methods, we also propose a sparse representation based classifier which takes into account the spatial information for both training and test samples in this work. Experimental results through two real-world hyperspectral datasets demonstrate that our proposed methods outperform other existing approaches.
The rest of the paper is organized as follows. In Sec. II-A, we briefly introduce the recently developed supervised dimen-sionality reduction methods (ADA and LADA), and Sec. II-B and Sec. II-C describe the proposed LSPP and SLSPP dimensionality reduction methods respectively. Sec. III describes the proposed classification method. In Sec. IV, the two practical hyperspectral benchmarking datasets employed to validate the proposed method are described, and the experimental results illustrating the benefits of the proposed work is described in Sec. V.

II. SPECTRAL ANGLE-BASED DIMENSIONALITY REDUCTION
A. ADA and LADA Recently, supervised angular discriminant analysis (ADA) was proposed in [14]. ADA finds a subspace where the angular separation of between-class samples is maximized while the angular separation of within-class samples is simultaneously minimized. It is mainly used to improve the classification performance of NN with cosine angle distance and SRC with OMP as the recovery method.
Let us define {x i ∈ R d , y i ∈ {1, 2, . . . , c}, i = 1, 2, . . . , n} to be the d-dimensional i-th training sample with an associated class label y i , where c is the number of classes and n is the total number of training samples. n = c l=1 n l where n l denotes the number of training samples from class l. Let T ∈ R d×r be the projection matrix, where r denotes the reduced dimensionality. The projection T ADA in ADA can obtained by solving The problem in (1) can be converted to a generalized eigenvalue problem. The authors in [14] also extend ADA into its localized variant named local angular discriminant analysis (LADA) by incorporating a locality preserving property through an affinity matrix. LADA was shown to handle more complex, potentially multimodal data distributions. The projection matrix T LADA of LADA can be obtained by solving The within-class and between-class outer product matrices of LADA are defined as where the weight matrices are defined based on heat kernel A ij as

B. LSPP
In this paper, we seek to make two related contributions within the context of angular discriminant analysis -(1) developing an unsupervised approach to spectral angle based subspace learning, where local spectral angles are preserved following this unsupervised project, and (2) developing a projection that incorporates spatial information when learning such an optimal projection. We first form a unsupervised version of ADA which we refer to as local similarity preserving projection (LSPP). It seeks a lower dimensional space where the correlation or angular relationship between samples that are neighbors in the feature space are preserved. We can also think of it as an angular equivalent of the commonly employed locality preserving projection (LPP) [1]. Let x i be the i-th training sample and P be the d × r projection matrix, where r is the reduced dimensionality. The objective function of LSPP can be simplified as The heat kernel W ij ∈ [0, 1] between x i and x j is defined as where σ is the parameter in the heat kernel. We impose a constraint (P t XDX t P = 1 where D ii = j W ij ) to avoid biases caused by different samples. The larger the value D ii which is corresponding to i-th training sample, the more important i-th training sample is.
The final objective function is defined as The problem in 11 can be solved as a generalized eigenvalue problem as XWX t P = λXDX t P The projection matrix P are the eigenvectors corresponding to the r largest eigenvalues.

C. SLSPP
It is well understood from many recent works [21], [23]- [25], that by aking into account the spatial neighborhood information, hyperspectral image classification accuracy can be significantly increased. This is based on the observation that spatially neighboring samples in hyperspectral data often consist of similar materials,and hence they are spectrally correlated. In order to utilize spatially neighboring samples in a lower-dimensional subspace, the spatial neighborhood relationship of hyperspectral data should be preserved. To address this problem, we propose a spatial variant of LSPP (SLSPP) in this work to further improve the classification accuracies. Let {z k , k ∈ Ω i } be the spatial neighborhood samples around a training sample x i , then the objective function of SLSPP can be reduced to where M = i k∈Ωi W ik z k x t i . The final objective function is defined as argmax P tr P t M P s. t. P t P = I Similar to LSPP, the projection matrix P are the eigenvectors corresponding to the r largest eigenvalues. Note that both LSPP and SLSPP are unsupervised projections in the sense that they do not require labels when learning the projections. We also note that a projection such as SLSPP is particularly beneficial when the backend classifier utilizes the spatial structure in the spatially neighboring pixels for each test pixel during classification. Towards that end, we next propose a modified sparse representation based classifier that utilizes sparse representations of the entire spatial neighborhood of a test pixel simultaneously when making a decision.

III. SPARSE REPRESENTATION-BASED CLASSIFICATION VIA SBOMP A. SBOMP
The simultaneous orthogonal matching pursuit (SOMP) [26] and block orthogonal matching pursuit (BOMP) [27] are all variants of orthogonal matching pursuit (OMP) that explore the block structure of test samples and training samples respectively. In this work, we effectively combine these two recovery methods and propose SBOMP to explore the block structure of both training and test samples simultaneously. SBOMP is illustrated in Algorithm 1. Let x t be a test sample. Assume A i contains spatial neighborhood samples around x i (inclusive of x i ), S contains the spatial neighborhood samples of x t (inclusive of x t ) and K is the sparsity level. SBOMP estimates the coefficientĈ based on the K mostly correlated spatial training samples in A.
, test data S and row sparsity level K. Update the support set Λ m = Λ m−1 ∪ λ by solving

B. Classification via SBOMP
The classification method employed after SLSPP is SBOMP-based Classification (SBOMP-C), as described in Algorithm 2. Since SLSPP can preserve the spatial neighboring samples for both training and test samples in a lowerdimensional subspace, by using SBOMP-C, we can exploit the block structure relationship effectively in the spatial domain between training and test samples. The block diagram of proposed methods are illustrated in Fig. 1.
, test data S and row sparsity level K.   IV. EXPERIMENTAL HYPERSPECTRAL DATA We validate the proposed approaches with two hyperspectral imagery datasets -a standard benchmarking dataset from the University of Pavia (an urban ground cover classification problem), and a hyperspectral imagery data collected by our lab to study wetland species composition.

A. University of Pavia Data
The first experimental hyperspectral dataset employed was collected using the Reflective Optics System Imaging Spectrometer (ROSIS) sensor [28]. This image, covering the University of Pavia, Italy, has 103 spectral bands with a spatial coverage of 610 × 340 pixels, and 9 classes of interests are considered in this dataset. A three-band true color image and its ground-truth are shown in Figure 2. This dataset represents a standardized benchmark as it is a well understood and widely studied dataset that is used to quantify performance of machine learning algorithms with hyperspectral data.

B. Wetland Data
The second hyperspectral data used in this work was acquired by us in Galveston, Texas in October, 2014 which includes two different wetland scenes captured at ground-level (side-looking views) over wetlands in Galveston. This is an unprecedented dataset in that it represents "street view" type hyperspectral images that have very different characteristics and are very useful for on-ground sensing. The two image cubes are referred to as area 1, and area 2, representing different regions of the wetlands that were imaged -In addition to common wetland classes, area 2 has Black Mangrove (Avicennia germinans) trees in the scene, a species which is of particular interest in ecological studies of wetlands, in addition to Spartina. The images were annotated with species information by an expert in Marine biology, who identified the various species in these hyperspectral scenes. This data was acquired using a Headwall Photonics hyperspectral imager which provides measurements in 325 spectral bands with a spatial size of 1004 × 5130. The hyperspectral data uniformly spanned the visible and near-infrared spectrum from 400nm − 1000nm. The objects of interests are primarily vegetation species common in such wetlands. Six different classes were identified in area-1 including soil, symphyotrichum, schoenoplectus, spartina patens, borrichia and rayjacksonia. The second area includes Avicennia germinans, batis, schoenoplectus, spartina alterniflora, soil, water and bridge. Since soil and schoenoplectus are included in both areas, the total number of classes in the combined library are eleven. This dataset represents a "real-world" application of our method for hyperspectral imaging in the field on arbitrary problems of interest.

V. EXPERIMENTAL SETTINGS AND RESULTS
The efficacy of the proposed LSPP, SLSPP and SBOMP are evaluated as a function of training samples per class using the two practical hyperspectral datasets mentioned previously. SLSPP-SBOMP-C indicates the data is projected based on SLSPP, and SBOMP-C is employed as the backend classifier. A nearest-neighbor (NN) classifier with cosine angle distance is used after LSPP and LADA projection, since they do not take spatial information into account when deriving the projection matrix. Similar to SLSPP-SOMP-C, LSPP-NN and LADA-NN. Each experiment is repeated 10 times using a repeated random subsampling validation technique, and the average accuracy is reported. The number of test samples per class is fixed to 100 for every random subsampling.
Wetland data, area -1 Wetland data, area -2 Fig. 3. True color images of the wetland dataset inset with ground truth. Fig. 4 shows the classification accuracies as a function of training samples for the University of Pavia hyperspectral data. The parameter for each algorithm is determined via searching through a wide range of the parameter space and the accuracies reported in this plot is based on the optimal parameter values. As can be seen from this figure, our proposed SLSPP followed by SBOMP-C gives the highest classification accuracies consistently over a wide range of the training sample size. LSPP-NN also gives better classification result compared with LADA-NN.
We perform a similar analysis using the wetland hyperspectral data. Fig. 5 and Fig. 6 plot the classification accuracies with respect to the training sample size per class for wetland area -1 and wetland area -2 respectively. It can be seen from the two plots that the proposed methods generally outperform other baseline methods for the vegetation type of hyperspectral data which demonstrate the diverse applicability of the proposed methods.
The class specific accuracies for different datasets are shown in Table I, Table II and Table III respectively. In this experiment, the training sample size per class is fixed to 10 and the test sample size is 100 per class. Each experiment is repeated 10 times and the average accuracy is reported. As can be seen from the class-specific tables for the wetland data, the spatial information plays a crucial role, especially for the species with complex textures and shapes such as Spartina-Patens, Sedge and Symphyotrichum.
Next, we analyze the effect of the window size (that defines the spatial neighborhood) for the SLSPP method. Fig. 7 depicts the classification accuracies as a function of different window size using the University of Pavia dataset. From this figure, we can see that the optimal window size is 5.
In this work, we also visualize the data distributions after projection for the proposed angle-based SLSPP and the Euclidean-based LPP methods. Figure 8 (a) shows the subset image for University of Pavia and Figure 8 (b) plots all the training samples used in this experiment on an 2 normalized sphere. Figure 8 (c) and (d) show the same samples after an SLSPP and LPP projection on an 2 normalized sphere respectively. U 1 , U 2 and U 3 are the three projections found by SLSPP and LPP corresponding to the largest eigenvalues. As can be seen from this figure, SLSPP is much more effective at    preserving the inter-sample relationships in terms of spectral angle in the lower-dimensional subspace compared to LPP.

VI. CONCLUSION
In this work, we have presented an unsupervised variant (LSPP) of the recently developed supervised dimensionality reduction method -ADA, as well as its spatial variant, SLSPP, that utilize spatial information around the samples when learning the projections. By incorporating spatial information in the dimensionality reduction projection, we are able to learn much more effective subspaces that not only preserves angular information among the training pixels in the feature space, but also their spatial neighbors. We also propose a related sparse representation classifier (SBOMP-C) that is optimized for feature spaces generated by SLSPP. Through experimental results based on two practical realworld hyperspectral datasets, we note that the proposed methods significantly outperform the baseline methods.