Knowledge in High-Dimensional Vector.
Pre-trained static and contextual word representations can provide valuable prior knowledge for constructing relational representations [1]}, [2]}. [3]} showed that different neural architectures (e.g., LSTM, CNN, and Transformers) can hierarchically structure linguistic information that varies with network depth.
Recently, many studies [4]}, [5]}, [6]} have shown that such hierarchy also exists in pretraining models like BERT. These results suggest that high-dimensional embeddings, independent of model architecture, learn much about the structure of language. Directly clustering on these high-dimensional embeddings should hardly produce ideal clusters in our desired way, which motivates us to extend current unsupervised clustering-based RE methods to learn the representations tailored for clustering relational data.
