Most of the promising works developed for material property prediction tasks are using graph neural networks (GNN). GNNs consider non-Euclidean topology to construct a graph representation that can be learned and modified according to the task [1]}, [2]}, [3]}. GNNs developed for material property prediction include CGCNN[4]} , OGCNN[5]} , SchNet[6]} , MegNet[7]}, and other models [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}, [15]}, [16]}, [17]}, [18]}. Developments have also been made in tasks such as material structure generation and prediction [19]}, [20]}, [21]}, [22]}, [23]}, [24]} as well as identifying new materials with specific properties[25]}. Despite progress being made in developing self-supervised ML architectures in the molecular ML, there is a noticeable lack of research works implementing such techniques for the periodic crystalline systems property prediction. In this work, we introduce Crystal Twins (CT): an SSL framework for crystalline material property prediction with GNNs (Figure REF ). In pre-training, the CT framework does not make use of any labeled data to learn crystalline representations, instead, it trains ML models in a self-supervised manner.
In the CT framework, we use the CGCNN[4]} as the encoder to learn expressive representations of crystalline systems (Figure REF a). The GNN encoder (i.e., CGCNN) generates representations of two augmented instances from the same crystal and the objective of CT pre-training is to make the cross-correlation matrix of the two embeddings as close as possible to the identity matrix (Figure REF a). To create augmented instances, we introduce the combination of three different augmentation techniques: random perturbations, atom masking, and edge masking (Figure REF b). The representations learned by the encoder are later used for downstream material property prediction tasks in the fine-tuning stage (Figure REF c). In the pre-training stage, the CT model learns representations without any labeled data. Using the pre-trained weights to initialize the GNN encoder for fine-tuning, CT demonstrates superior prediction performances on 7 challenging datasets. We also compare the performance of the CT model with other competitive supervised learning baselines.
