Unless specifically mentioned, we set the batch size to 16, the learning rate to 0.0001, weight decay of all learnable weights to 1e-5, dropout \(p\) =0.5 for all the experiments. We optimize the network using an SGD optimizer for 30 epochs. Following the common practice of fine-tuning, we freeze the transferred weights in the initial training epochs. Except for the VGG backbone, we initialize our network using Kaiming uniform initialization proposed in [1]}. We optimize the summed cross-entropy loss on \(\mathbf {n}_T\) , \(\mathbf {n}_S\) , \(\mathbf {n}_{\underline{O}}\) , \(\mathbf {n}_O\) , \(\mathbf {n}_{\underline{\underline{O}}}\) , and \(\mathbf {n}_P\)  for the perception and memory experience. For \(\mathbf {n}_S\)  and \(\mathbf {n}_O\) , we only activate entity instances, with the exception of extending to class/attribute labels for generalized statements (See REF ). For \(\mathbf {n}_{\underline{O}}\)  and \(\mathbf {n}_{\underline{\underline{O}}}\) , we apply cross-entropy loss on each subset. For instance perception, we use a learning rate of 0.001. During the training of memory experience (table REF , table REF ), we optimize our model directly on triples without any visual input. The training converges after 20 epochs with an Adam optimizer [2]}. We train two semantic models to predict relationships between classes and relationships between entities, respectively. For RESCAL, we use the implementation of PyKeen [3]} and set the rank of entity embeddings and predicate embeddings to 1000, which gives a comparable number of learnable parameters as our model. For the social network, we train the model for attribute prediction and relationship prediction on the social network dataset. Concretely, we minimize the cross-entropy loss for \(\mathbf {n}_S\) , \(\mathbf {n}_{\underline{O}}\) , \(\mathbf {n}_O\) , \(\mathbf {n}_{\underline{\underline{O}}}\) , and \(\mathbf {n}_P\) . For SSL, we use a batch size of 128, a learning rate of 1e-5, and total training epochs of 10. Except for the embeddings for time and entities, other weights are frozen. All experiments are conducted on an Nvidia GTX 1089 Ti GPU with a 4 core CPU of 16G memory.
