AE-OT: A New Generative Model Based on Extended Semi-Discrete Optimal Transport
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Na, Lei
Current generative models like generative adversarial networks (GANs) and variational autoencoders (VAEs) have attracted huge attention due to its capability to generate visual realistic images. However, most of the existing models suffer from the mode collapse or mode mixture problems. In this work, we give a theoretic explanation of the both problems by Figalli’s regularity theory of optimal transportation maps. Basically, the generator compute the transportation maps between the white noise distributions and the data distributions, which are in general discontinuous. However, deep neural networks (DNNs) can only represent continuous maps. This intrinsic conflict induces mode collapse and mode mixture. In order to tackle the both problems, we explicitly separate the manifold embedding and the optimal transportation; the first part is carried out using an autoencoder (AE) to map the images onto the latent space; the second part is accomplished using a GPU-based convex optimization to find the discontinuous transportation maps. Composing the extended optimal transport (OT) map and the decoder, we can finally generate new images from the white noise. This AE-OT model avoids representing discontinuous maps by DNNs, therefore effectively prevents mode collapse and mode mixture.
 Good afternoon everyone, I am Na-Li from Dalian University of Technology. Thank you for watching this video. Thank Professor Chang Bo-Cheng for the invitation. Today I will introduce a new generative model, AEOT, which is based on optimal transport. This is the references for this talk. The work is mainly collaborated with Professor Xing Tong-Yao from Harvard University, Professor Xian Hong-Gu from Stony Brook University, and some other colleges. This is the talk outline. In machine learning, generative models have been becoming more important and popular recently. A generative model creates new data instances that resemble your training data. It converts a white noise from latent space into a dead image. A huge breakthrough for image generating comes from the scheme of autoencoder, which is one kind of non-adversarial model. Here is the structure of AE, one encoder and one decoder. The image in ambient space is encoded to the latent space and then decoded to ambient space again. Training AE is to minimize the lost function, which is the integration of the distance between the original image and the reconstructed image. But encoded latent space may not fill the whole latent space, which always results in low-quality generated images. To solve this problem, AE was proposed, where the decoder approximates real data distributions from a Gaussian distribution in a variation approach. GA is the representative of an adversary model. It includes one generator and one discriminator. The generator is like a folder trying to produce some counterfeit material, which actually transforms random noises into meaningful outputs. The discriminator is like the place trying to detect the forged items, which calculates the distance between the generated distribution and the real distribution. The two neural networks are trained alternately. The generated instances become a negative training example for the discriminator, and the discriminator penalizes the generator for producing implausible results. The training continues until the generated fake data cannot be distinguished from real data by the discriminator. Besides these typical models, there are many other various works such as WAE, AE, WG and so on. In deep learning, the Manifold Distribution Hypothesis, which was pointed out in 2000, is well accepted, which assumes the distribution of a specific class of natural data is concentrated on a low-dimensional manifold embedded in a high-dimensional data space. Let's see, Yen Le Kun's Amnist dataset. Each image is offset 28 by 28. We treat each image as a point in a 784-dimensional space. All the digital images constitute a point cloud in the 784-dimensional space. The point cloud can be flattened into the 2D plan by Hinton's Tisney algorithm. Each image on the left is mapped to a point on the 2D domain. There are totally 10 digits, and here we can see 10 clusters. Each cluster represents 1 digit. So this example shows that even though the point cloud is embedded in a very high-dimensional space, but the real dimension of this manifold is only 2. Now let's put it into mathematical languages. We have a very high-dimensional image space, Rn, like a 784-dimensional space. We also call it ambient space. Every point in this space represents an image. Then we consider a special natural data, such as hydrating digital data. We lay from a curved subspace, which is a manifold. Then we map the manifold to the parameter space, so the dimension of the parameter space equals to the dimension of the manifold. This mapping is called encoding, and inverse mapping is called decoding. One of the major tasks of deep learning is to learn this encoding and decoding map. But encoding and decoding maps are not unique, which will result in different latent calls, so we need to transport from one to the other. Here we show one example by autoencoder. We consider this 2D surface embedded in the 3D space. We randomly sample from the surface to build a data set, and then we train an autoencoder. First we encode it to 2D domain, and then reconstruct it to 3D ambient space. Here in the middle is the latent calls, and in the right is the reconstructed manifold. Since the encoding and decoding are not unique, we can have other latent calls. Here we show two different encoding results. Both of them are the encoded data points in the 2D domain, but with different distributions. Different distributions matter a lot. Next we will show the right latent space preserved original distribution, but the left one does not. Let's consider the left one firstly. If we uniformly sample this latent space, and then put them back by decoding map to the original manifold, it's highly non-uniform. The height part is sparse, and the body part is very dense. This means this kind of encoding map destroyed the original distribution. But in deep learning, we try to preserve the original data distribution from the manifold to the latent space. Now let's look at the second latent space. If we uniformly sample this latent space, and then put them back by the corresponding decoding map to the original manifold, it's still uniform. So we say this encoding decoding preserves the original data distribution. From here we know that for a generative model, at first learn the manifold structure and encode it to a latent space. Then it tries to learn the distribution and find a way to represent the distribution. These are the two major tasks for the generative model. We use this framework to explain the autoencoder. The autoencoder only learns the encoder map fz and the decoding map gsi, but does not learn the data distribution map t. So AE always provides high-quality reconstructed images, but low-quality generated images. However, we include the distribution transformation t, which uses a Gaussian distribution to fit the latent code distribution. The generative step is the composition of the decoding map gsi and the distribution transformation map t. The generator of GAN is also the composition of the decoding map gsi and the distribution transformation map t, but both VAE and GAN mix these two maps together. In the training of the generative models, we usually encounter some difficulties, which is the so-called mod-claps. The difficulties include, first, sometimes the training procedure does not converge and is very sensitive to hyperparameters. Second, if the dataset includes several modes, the generated distribution sometimes misses some modes. And sometimes there are many unrealistic samples in the generated images. Let's see these two simple examples. These are two synthetic datasets. Each yellow cluster in the first column is a Gaussian distribution of 512 points. We use them as the target distribution, which includes many modes. Then we train a GAN and a PECGAN respectively. We can see that the generated distribution of GAN misses some models. For the PECGAN, although it covers all the modes, it also generates unrealistic samples. That means the generative models come to the dialogue of mod-claps. So in this work, we propose a new generative model to overcome these shortcomings. The new model is based on the Manifold Distribution Hypothesis. Decompose the generative task into two parts, Manifold Linear and Distribution Transformation. The first part is realized by AE and the second part is carried out based on semi-discrete optimal transport. Optimal transport problem has a long history. It was first proposed by Moj in 1781. It studies how to transport a certain amount of mass from one distribution to another with the minimum cost. Suppose we are given two subdomains, X and Y. Mu and nu are two density functions defined on X and Y respectively, with the total measures of X, Y are equal. We are trying to find a mapping from X to Y. We can consider X as the producer and Y as consumer. The density function mu can be treated as a producing rate and nu can be treated as a consuming rate. Suppose the cost of moving a unit mass from point X to Y is CXY, then the total transportation cost is defined as this equation. During the transportation, we need the measure to be balanced. That is, the produced mass is equal to the consumed mass, which means we try to find a mapping T pushed forward mu to nu. This is the so-called measure-preserving mapping. Then this mapping must satisfy the following Jacobi equation. More reformulated the following problem, how to find a measure-preserving transportation map that minimizes the transportation cost. Such kind of map is called the optimal transport map. The cost of the map gives us the vast distance between two probability functions. There are many different cost functions. If we choose the cost function to be at a distance, then Brignier proved that the optimal transport map is given by a convex function u. The gradient map is the desired optimal transport map. So the problem is how to find the convex function u we can call it Brignier potential. By measure-preserving condition, we can get this motion-pay equation for the Brignier potential. Therefore, the question seeking the optimal transport map is boiled down to solving this motion-pay equation. But this motion-pay equation is very nonlinear and difficult to solve. Here we only consider the semi-discrete optimal transport map. In this case, the source measure mu is absolutely continuous, defined on a convex domain omega. The target domain is a discrete set, consists of k points p1, p2, and pk. The target measure is a drag measure defined on each point, with the equal total mass as the source measure. Under a semi-discrete transportation map, T, a cell decomposition is induced, such that every x in each cell wi is mapped to a target point pi. The map T is measure-preserving, which means the volume of each cell wi equal to the measure of the target point pi. In 2013, Professor Gu Luo-sun An-yao gave a vibrational proof of this theorem. This theorem provides us a constructive algorithm to the semi-discrete optimal transport map. Here is the pipeline of the computational algorithm. Next, let's discuss the reason of mod-claps. We have known that the distribution transform step can be realized by optimal transport map, which is equivalent to solving motion-pay equation. The field modelist Figuali analyzed the regularity of the solution to motion-pay equation. Intuitively, it says that suppose we have a uniform distribution on the unit disk, and here is the planned domain, which is non-convex and also multi-connected. Here is also a uniform distribution on this domain. Then we calculate the optimal transport map. We map each cell with the same color. Then we can find that the mapping has lots of singularity points and singularity lines. The images of the singularities are not unique and lies in the boundary. From this figure, we can see the optimal transport map is not continuous. Actually, deep learning neural network can only represent continuous mappings, but the transportation maps are discontinuous on singular sites. Namely, the target mappings are outside the functional space of dnn. This conflict induces mod-claps. In this figure, we show the mod-claps caused by the discontinuity of transportation map. Top row shows real data distributions. The bottom row gives the noise distribution. Here, red dot lines are the singularity set. Red cross are mapped to be unrealistic samples by dnn. If the support of the target distribution is convex, then dnn is able to approximate the transportation map, head F as well. But when the support of the target distribution is concave, then dnn represented by F2 cannot approximate the transportation map, head F2, as well, and generates some unreal samples. These are unreal samples. If the target distribution is multi-mode, here, head F3 gives the transport map. When we use continuous approximate, either we will get mod-claps F31. Here, it only captures one mode, missed two other modes. And the other possibility is that it will generate some unreal samples. Here, we use human-physical photo to train a manifold, which is non-convex. After encoding, we map it to the latent space. In latent space, we map the white noise to the latent code. We draw a straight line here, so this straight line will be mapped to a curve here, then it's further mapped to the red curve on the manifold. A curve on the manifold gives us a sequence of morphing from one face to another face. We can see from here, morphing from one face to another face. But if there is a singular curve here, then when we draw a line across this singular curve, we will get some point. This line was mapped here and then here. Then we will get some point in the middle. Something in the middle means this sample is outside the human-physical manifold, which means that the possibility to see this sample is zero. Let's see. Here is the curve across the singularity set. On this curve, we start from boy with black eyes, and later we end with a girl with blue eyes. But in the middle, we see some very strange images. The faces here are the faces with one eye to be black and one eye to be blue. We cannot see this kind of person does not exist, but the possibility to see this kind of face is zero. Go back here. This side means the faces with the black eyes, this side means the faces with blue eyes. Then the point in the middle appears that one eye in blue and one eye in zero. So by using optimal transport theory, we can detect the boundary of the human-physical manifold. Then we propose our new generative model to eliminate mod-claps. As we have discussed, there are two tasks of generative model. One is Manifold Structure Learning, which learn encoder and decoder. The other one is to transform from one distribution to another distribution. This part is actually the most important part. We separate this part and use our geometric model to compute optimal transport map. By doing this, we can eliminate mod-claps and also improve efficiency. Let's show some example results. We have shown this example before. Here GaM model misses some modes, and the Paq GaM model generates some unreal samples. But our AeOt exactly covers all the modes and now we generate the fake samples. Here is the other example. We also compare our AeOt model with some other state-of-the-art methods. This is the comparison result on Amnist data set. This frame is reconstructed by Ae. This one is generated by Lusis. This one is generated by Horses and Malik. And this one is generated by our AeOt model. This is the comparison result for Amnist Fission data set. This was generated by AeOt model. And here is the result on the CIFATN data set. And this is on the slab A data set. If we look into the generated images by other methods, we can see here are some unreal faces. And here by C.R. GaM, C.R. GaM generates totally same images from different latent noises. Here we show the face-morphing result by our AeOt model. When we do this interpolation, we avoid all the singularities. In this table, we compare the FID scores and it shows that under first three data sets, AeOt model achieves the best result. At last, let me conclude. We see from the work that the intrinsic pattern of natural data can be represented by manual distribution hypothesis. So therefore, the deep learning system has two major tasks. One is manifold learning dimension deduction from the image space to the latent space. The other one is the probability distribution transformation from the wet noise to the latent code distribution. We can see optimal transport methods can be used to accomplish the second task. And the regularity theory of Monshun Pei equation explains why mod collapse appears. The AeOt framework we constructed can avoid mod collapse and this model also makes half of the black box transparent. Ok, that's all from my talk. Thank you.
 Good afternoon everyone, I am Na-Li from Dalian University of Technology. Thank you for watching this video.  Thank Professor Chang Bo-Cheng for the invitation. Today I will introduce a new generative model, AEOT, which is based on optimal transport.  This is the references for this talk. The work is mainly collaborated with Professor Xing Tong-Yao from Harvard University, Professor Xian Hong-Gu from Stony Brook University, and some other colleges.  This is the talk outline.  In machine learning, generative models have been becoming more important and popular recently. A generative model creates new data instances that resemble your training data. It converts a white noise from latent space into a dead image.  A huge breakthrough for image generating comes from the scheme of autoencoder, which is one kind of non-adversarial model. Here is the structure of AE, one encoder and one decoder.  The image in ambient space is encoded to the latent space and then decoded to ambient space again. Training AE is to minimize the lost function, which is the integration of the distance between the original image and the reconstructed image.  But encoded latent space may not fill the whole latent space, which always results in low-quality generated images.  To solve this problem, AE was proposed, where the decoder approximates real data distributions from a Gaussian distribution in a variation approach.  GA is the representative of an adversary model. It includes one generator and one discriminator. The generator is like a folder trying to produce some counterfeit material, which actually transforms random noises into meaningful outputs.  The discriminator is like the place trying to detect the forged items, which calculates the distance between the generated distribution and the real distribution.  The two neural networks are trained alternately. The generated instances become a negative training example for the discriminator, and the discriminator penalizes the generator for producing implausible results.  The training continues until the generated fake data cannot be distinguished from real data by the discriminator.  Besides these typical models, there are many other various works such as WAE, AE, WG and so on.  In deep learning, the Manifold Distribution Hypothesis, which was pointed out in 2000, is well accepted, which assumes the distribution of a specific class of natural data is concentrated on a low-dimensional manifold embedded in a high-dimensional data space.  Let's see, Yen Le Kun's Amnist dataset. Each image is offset 28 by 28. We treat each image as a point in a 784-dimensional space.  All the digital images constitute a point cloud in the 784-dimensional space. The point cloud can be flattened into the 2D plan by Hinton's Tisney algorithm.  Each image on the left is mapped to a point on the 2D domain. There are totally 10 digits, and here we can see 10 clusters. Each cluster represents 1 digit.  So this example shows that even though the point cloud is embedded in a very high-dimensional space, but the real dimension of this manifold is only 2.  Now let's put it into mathematical languages. We have a very high-dimensional image space, Rn, like a 784-dimensional space. We also call it ambient space.  Every point in this space represents an image. Then we consider a special natural data, such as hydrating digital data. We lay from a curved subspace, which is a manifold.  Then we map the manifold to the parameter space, so the dimension of the parameter space equals to the dimension of the manifold. This mapping is called encoding, and inverse mapping is called decoding.  One of the major tasks of deep learning is to learn this encoding and decoding map. But encoding and decoding maps are not unique, which will result in different latent calls, so we need to transport from one to the other.  Here we show one example by autoencoder. We consider this 2D surface embedded in the 3D space. We randomly sample from the surface to build a data set, and then we train an autoencoder.  First we encode it to 2D domain, and then reconstruct it to 3D ambient space. Here in the middle is the latent calls, and in the right is the reconstructed manifold.  Since the encoding and decoding are not unique, we can have other latent calls. Here we show two different encoding results. Both of them are the encoded data points in the 2D domain, but with different distributions.  Different distributions matter a lot. Next we will show the right latent space preserved original distribution, but the left one does not.  Let's consider the left one firstly. If we uniformly sample this latent space, and then put them back by decoding map to the original manifold, it's highly non-uniform.  The height part is sparse, and the body part is very dense. This means this kind of encoding map destroyed the original distribution.  But in deep learning, we try to preserve the original data distribution from the manifold to the latent space.  Now let's look at the second latent space. If we uniformly sample this latent space, and then put them back by the corresponding decoding map to the original manifold, it's still uniform.  So we say this encoding decoding preserves the original data distribution.  From here we know that for a generative model, at first learn the manifold structure and encode it to a latent space.  Then it tries to learn the distribution and find a way to represent the distribution. These are the two major tasks for the generative model.  We use this framework to explain the autoencoder. The autoencoder only learns the encoder map fz and the decoding map gsi, but does not learn the data distribution map t.  So AE always provides high-quality reconstructed images, but low-quality generated images.  However, we include the distribution transformation t, which uses a Gaussian distribution to fit the latent code distribution.  The generative step is the composition of the decoding map gsi and the distribution transformation map t.  The generator of GAN is also the composition of the decoding map gsi and the distribution transformation map t, but both VAE and GAN mix these two maps together.  In the training of the generative models, we usually encounter some difficulties, which is the so-called mod-claps.  The difficulties include, first, sometimes the training procedure does not converge and is very sensitive to hyperparameters.  Second, if the dataset includes several modes, the generated distribution sometimes misses some modes.  And sometimes there are many unrealistic samples in the generated images.  Let's see these two simple examples. These are two synthetic datasets.  Each yellow cluster in the first column is a Gaussian distribution of 512 points. We use them as the target distribution, which includes many modes.  Then we train a GAN and a PECGAN respectively.  We can see that the generated distribution of GAN misses some models.  For the PECGAN, although it covers all the modes, it also generates unrealistic samples.  That means the generative models come to the dialogue of mod-claps.  So in this work, we propose a new generative model to overcome these shortcomings.  The new model is based on the Manifold Distribution Hypothesis. Decompose the generative task into two parts,  Manifold Linear and Distribution Transformation.  The first part is realized by AE and the second part is carried out based on semi-discrete optimal transport.  Optimal transport problem has a long history. It was first proposed by Moj in 1781.  It studies how to transport a certain amount of mass from one distribution to another with the minimum cost.  Suppose we are given two subdomains, X and Y.  Mu and nu are two density functions defined on X and Y respectively, with the total measures of X, Y are equal.  We are trying to find a mapping from X to Y. We can consider X as the producer and Y as consumer.  The density function mu can be treated as a producing rate and nu can be treated as a consuming rate.  Suppose the cost of moving a unit mass from point X to Y is CXY, then the total transportation cost is defined as this equation.  During the transportation, we need the measure to be balanced.  That is, the produced mass is equal to the consumed mass, which means we try to find a mapping T pushed forward mu to nu.  This is the so-called measure-preserving mapping.  Then this mapping must satisfy the following Jacobi equation.  More reformulated the following problem, how to find a measure-preserving transportation map that minimizes the transportation cost.  Such kind of map is called the optimal transport map.  The cost of the map gives us the vast distance between two probability functions.  There are many different cost functions.  If we choose the cost function to be at a distance, then Brignier proved that the optimal transport map is given by a convex function u.  The gradient map is the desired optimal transport map.  So the problem is how to find the convex function u we can call it Brignier potential.  By measure-preserving condition, we can get this motion-pay equation for the Brignier potential.  Therefore, the question seeking the optimal transport map is boiled down to solving this motion-pay equation.  But this motion-pay equation is very nonlinear and difficult to solve.  Here we only consider the semi-discrete optimal transport map.  In this case, the source measure mu is absolutely continuous, defined on a convex domain omega.  The target domain is a discrete set, consists of k points p1, p2, and pk.  The target measure is a drag measure defined on each point, with the equal total mass as the source measure.  Under a semi-discrete transportation map, T, a cell decomposition is induced, such that every x in each cell wi is mapped to a target point pi.  The map T is measure-preserving, which means the volume of each cell wi equal to the measure of the target point pi.  In 2013, Professor Gu Luo-sun An-yao gave a vibrational proof of this theorem.  This theorem provides us a constructive algorithm to the semi-discrete optimal transport map.  Here is the pipeline of the computational algorithm.  Next, let's discuss the reason of mod-claps.  We have known that the distribution transform step can be realized by optimal transport map, which is equivalent to solving motion-pay equation.  The field modelist Figuali analyzed the regularity of the solution to motion-pay equation.  Intuitively, it says that suppose we have a uniform distribution on the unit disk, and here is the planned domain, which is non-convex and also multi-connected.  Here is also a uniform distribution on this domain.  Then we calculate the optimal transport map.  We map each cell with the same color.  Then we can find that the mapping has lots of singularity points and singularity lines.  The images of the singularities are not unique and lies in the boundary.  From this figure, we can see the optimal transport map is not continuous.  Actually, deep learning neural network can only represent continuous mappings, but the transportation maps are discontinuous on singular sites.  Namely, the target mappings are outside the functional space of dnn.  This conflict induces mod-claps.  In this figure, we show the mod-claps caused by the discontinuity of transportation map.  Top row shows real data distributions.  The bottom row gives the noise distribution.  Here, red dot lines are the singularity set.  Red cross are mapped to be unrealistic samples by dnn.  If the support of the target distribution is convex, then dnn is able to approximate the transportation map, head F as well.  But when the support of the target distribution is concave, then dnn represented by F2 cannot approximate the transportation map, head F2, as well, and generates some unreal samples.  These are unreal samples.  If the target distribution is multi-mode,  here, head F3 gives the transport map.  When we use continuous approximate, either we will get mod-claps F31.  Here, it only captures one mode, missed two other modes.  And the other possibility is that it will generate some unreal samples.  Here, we use human-physical photo to train a manifold, which is non-convex.  After encoding, we map it to the latent space.  In latent space, we map the white noise to the latent code.  We draw a straight line here, so this straight line will be mapped to a curve here, then it's further mapped to the red curve on the manifold.  A curve on the manifold gives us a sequence of morphing from one face to another face.  We can see from here, morphing from one face to another face.  But if there is a singular curve here, then when we draw a line across this singular curve,  we will get some point.  This line was mapped here and then here.  Then we will get some point in the middle.  Something in the middle means this sample is outside the human-physical manifold,  which means that the possibility to see this sample is zero.  Let's see.  Here is the curve across the singularity set.  On this curve, we start from boy with black eyes, and later we end with a girl with blue eyes.  But in the middle, we see some very strange images.  The faces here are the faces with one eye to be black and one eye to be blue.  We cannot see this kind of person does not exist, but the possibility to see this kind of face is zero.  Go back here. This side means the faces with the black eyes, this side means the faces with blue eyes.  Then the point in the middle appears that one eye in blue and one eye in zero.  So by using optimal transport theory, we can detect the boundary of the human-physical manifold.  Then we propose our new generative model to eliminate mod-claps.  As we have discussed, there are two tasks of generative model.  One is Manifold Structure Learning, which learn encoder and decoder.  The other one is to transform from one distribution to another distribution.  This part is actually the most important part.  We separate this part and use our geometric model to compute optimal transport map.  By doing this, we can eliminate mod-claps and also improve efficiency.  Let's show some example results.  We have shown this example before.  Here GaM model misses some modes, and the Paq GaM model generates some unreal samples.  But our AeOt exactly covers all the modes and now we generate the fake samples.  Here is the other example.  We also compare our AeOt model with some other state-of-the-art methods.  This is the comparison result on Amnist data set.  This frame is reconstructed by Ae. This one is generated by Lusis.  This one is generated by Horses and Malik.  And this one is generated by our AeOt model.  This is the comparison result for Amnist Fission data set.  This was generated by AeOt model.  And here is the result on the CIFATN data set.  And this is on the slab A data set.  If we look into the generated images by other methods, we can see here are some unreal faces.  And here by C.R. GaM, C.R. GaM generates totally same images from different latent noises.  Here we show the face-morphing result by our AeOt model.  When we do this interpolation, we avoid all the singularities.  In this table, we compare the FID scores and it shows that under first three data sets, AeOt model achieves the best result.  At last, let me conclude.  We see from the work that the intrinsic pattern of natural data can be represented by manual distribution hypothesis.  So therefore, the deep learning system has two major tasks.  One is manifold learning dimension deduction from the image space to the latent space.  The other one is the probability distribution transformation from the wet noise to the latent code distribution.  We can see optimal transport methods can be used to accomplish the second task.  And the regularity theory of Monshun Pei equation explains why mod collapse appears.  The AeOt framework we constructed can avoid mod collapse and this model also makes half of the black box transparent.  Ok, that's all from my talk. Thank you.
AE-OT_0_911.png AE-OT_912_4715.png AE-OT_4716_5351.png AE-OT_5352_6731.png AE-OT_6732_9335.png AE-OT_9336_9911.png AE-OT_9912_11315.png AE-OT_11316_13451.png AE-OT_13452_13703.png AE-OT_13704_14435.png AE-OT_14436_16547.png AE-OT_16548_16619.png AE-OT_16620_17051.png AE-OT_17052_18887.png AE-OT_18888_19499.png AE-OT_19500_20795.png AE-OT_20796_22307.png AE-OT_22308_22703.png AE-OT_22704_22931.png AE-OT_22932_25319.png AE-OT_25320_26927.png AE-OT_26928_27779.png AE-OT_27780_28799.png AE-OT_28800_28931.png AE-OT_28932_29699.png AE-OT_29700_30767.png AE-OT_30768_31427.png AE-OT_31428_31619.png AE-OT_31620_33227.png AE-OT_33228_34439.png AE-OT_34440_34979.png AE-OT_34980_35291.png AE-OT_35292_37223.png AE-OT_37224_37345.png