A common framework for learning synthesis with an EBM is to use persistent initialization for MCMC samples with a certain rate of rejuvenation (e.g. 5% chance) from a noise distribution [1]}. This approach can cause instability because shortrun samples used to update the model include a mix of higher-energy burn-in samples and lower-energy realistic images, which can destabilize training by increasing the variance of the gradient of the negative samples in (REF ).
On the other hand, removing rejuvenation can decrease the quality of the learned images because persistent images often become stuck in local modes and develop defects that linger for many updates. Persistent banks also scale poorly to large datasets such as ImageNet because the bank cannot efficiently represent the diversity of the data. Initialization from a cooperative generator network is an appealing alternative because it could enable efficient in-distribution generation of highly diverse appearances. Using a generator for rejuvenation allows samples to begin much closer to the correct energy spectrum, thereby avoiding the instability of noise rejuvenation. However, we observe a major limitation of cooperative learning that affects the generator ability to produce diverse initial states. In particular, a generator trained with the cooperative learning objective has difficulty breaking the symmetry of its activations. This happens because shortrun EBM samples are unable to provide novel diversity if the generator initialization already lacks diversity. This is illustrated in Figure REF . Learning quickly becomes unstable without auxiliary techniques such as batch normalization to break generator symmetry. The lack of diversity of shortrun EBM samples limits the results of cooperative learning even when training succeeds.
<FIGURE>