It has been argued that current machine learning models do not have commonsense, and therefore must be hard-coded with prior knowledge (Marcus, 2018). Here we show surprising evidence that language models can already learn to capture certain common sense knowledge. Our key observation is that a language model can compute the probability of any statement, and this probability can be used to evaluate the truthfulness of that statement.   On the Winograd Schema Challenge (Levesque et al., 2011), language models are 11% higher in accuracy than previous state-of-the-art supervised methods. Language models can also be fine-tuned for the task of Mining Commonsense Knowledge on ConceptNet to achieve an F1 score of 0.912 and 0.824, outperforming previous best results (Jastrzebskiet al., 2018).   Further analysis demonstrates that language models can discover unique features of Winograd Schema contexts that decide the correct answers without explicit supervision. It has been argued that current machine learning models do not have common sense BID4 BID15 . For example, even best machine learning models perform poorly on commonsense reasoning tasks such as Winograd Schema Challenge BID11 BID14 . This argument is often combined with another important criticism of supervised learning that it only works well on problems that have a lot of labeled data. The Winograd Schema Challenge (WSC) is an opposite of such problems because its labeled set size is only on the order of a few hundreds examples, with no official training data. Based on this argument, it is suggested that machine learning models must be integrated with prior knowledge BID15 BID10 .As an example, consider the following question from the WSC dataset:"The trophy doesn't fit in the suitcase because it is too big." What is "it"? Answer 0: the trophy. Answer 1: the suitcase.The main point of this dataset is that no machine learning model today can do a good job at answering this type of questions.In this paper, we present surprising evidence that language models do capture certain common sense knowledge and this knowledge can be easily extracted. Key to our method is the use of language models (LMs), trained on a large amount of unlabeled data, to score multiple choice questions posed by the challenge and similar datasets. In the above example, we will first substitute the pronoun ("it") with the candidates ("the trophy" and "the suitcase"), and then use an LM to compute the probability of the two resulting sentences ("The trophy doesn't fit in the suitcase because the trophy is too big." and "The trophy doesn't fit in the suitcase because the suitcase is too big."). The substitution that results in a more probable sentence will be the chosen answer. Using this simple method, we are able to achieve 63.7% accuracy, 11% above that of the previous state-of-the-art result 1 .To demonstrate a practical impact of this work, we show that the trained LMs can be used to enrich human-annotated knowledge bases, which are known to be low in coverage and expensive to expand. For example, "Suitcase is a type of container", a relevant knowledge to the above Winograd Schema example, does not present in the ConceptNet knowledge base BID13 . The goal of this task is to add such new facts to the knowledge base at a cheaper cost than human annotation, in our case using LM scoring. We followed the Commonsense Knowledge Mining task formulation from BID0 BID12 BID8 , which posed the task as a classification problem of unseen facts and non-facts. Without an additional classification layer, LMs are fine-tuned to give different scores to facts and non-facts tuples from ConceptNet. Results obtained by this method outperform all previous results, despite the small training data size (100K instances). On the full test set, LMs can identify commonsense facts with 0.912 F1 score, which is 0.02 better than supervised trained networks BID8 . We introduced a simple method to apply pretrained language models to tasks that require commonsense knowledge. Key to our method is the insight that large LMs trained on massive text corpora can capture certain aspect of human knowledge, and therefore can be used to score textual statements. On the Winograd Schema Challenge, LMs are able to achieve 11 points of accuracy above the best previously reported result. On mining novel commonsense facts from ConceptNet knowledge base, LM scoring also outperforms previous methods on two different test criteria. We analyse the trained language models and observe that key features of the context that identify the correct answer are discovered and used in their predictions.Traditional approaches to capturing common sense usually involve expensive human annotation to build knowledge bases. This work demonstrates that commonsense knowledge can alternatively be learned and stored in the form of distributed representations. At the moment, we consider language modeling for learning from texts as this supplies virtually unlimited data. It remains an open question for unsupervised learning to capture commonsense from other modalities such as images or videos.
We present evidence that LMs do capture common sense with state-of-the-art results on both Winograd Schema Challenge and Commonsense Knowledge Mining.
{"src": "scitldr-AIC_train", "paper_id": "rkgfWh0qKX"}