[1]} have used attention model for multi-instance relation extraction. They applied attention over a bag of independent sentences containing two entities to extract the relation between them. First, CNN-based models are used to encode the sentences in a bag. Then a bi-linear attention layer is used to determine the importance of each sentence in the bag. This attention helps to mitigate the problem of noisy samples obtained by distant supervision to some extent. The idea is that clean sentences get higher attention scores over the noisy ones. The sentence vectors in the bag are merged in a weighted average fashion based on their attention scores. The weighted average vector of the sentences is passed to a feed-forward neural network with softmax to determine the relation. This bag-level attention is used only for positive relations and not used for None relation. The reason is that the representations of the bags that express no relations are always diverse and it is difficult to calculate suitable weights for them.
