Architecture.
Following the latest works [1]}, [2]}, we use I3D [3]} as the backbone.
However, different from CoMix which jointly trains the backbone, we simply use the pretrained I3D model on Kinetics [4]}, provided by [3]}, to extract RGB features. For the first three benchmarks, RGB features are used as the input of TranSVAE. For Sprites, we use the original image as the input, for the purpose of visualizing the reconstruction and disentanglement results. We use the shared encoder and decoder structures across the source and target domains.
For RGB feature inputs, the encoder and decoder are fully-connected layers. For original image inputs, the encoder and decoder are the convolution and deconvolution layers (from DCGAN [6]}), respectively. For the TRN model, we directly use the one provided by [7]}.
Other details on this aspect are placed in Appendix.
