Punctuation is a strong indicator of syntactic structure, and parsers trained on text with punctuation often rely heavily on this signal. Punctuation is a diversion, however, since human language processing does not rely on punctuation to the same extent, and in informal texts, we therefore often leave out punctuation. We also use punctuation ungrammatically for emphatic or creative purposes, or simply by mistake. We show that (a) dependency parsers are sensitive to both absence of punctuation and to alternative uses; (b) neural parsers tend to be more sensitive than vintage parsers; (c) training neural parsers without punctuation outperforms all out-of-the-box parsers across all scenarios where punctuation departs from standard punctuation. Our main experiments are on synthetically corrupted data to study the effect of punctuation in isolation and avoid potential confounds, but we also show effects on out-of-domain data.
show how machine reading models can easily be broken with distractor sentences at test time and propose an alternative evaluation scheme, and show how susceptible character-based machine translation models are to noise. Both papers are similar to ours in evaluating the performance of state-of-the-art models under corruptions of the data. There was recently a workshop dedicated to evaluation of NLP models under human adversarial example selection @cite_0 . Historically, NLP models were rarely evaluated on synthetic or otherwise adversarial data, but we believe this is a fruitful research direction. This is largely a philosophical question, and we believe a philosophical argument is in order. John Dewey @cite_2 , the American philosopher, distinguishes three modes of thinking: (i) common reasoning, which identifies pattern in available, historical data, (ii) empirical thinking, which collects new data to vary the experimental conditions, and (iii) experimental thinking, which actively modifies the conditions in controlled experiments to isolate the relevant variables. We believe recent work on breaking NLP models is an attempt to introduce experimental thinking into NLP, which has otherwise been limited -- or handicapped in Dewey's words -- by what data happens to be available.
@cite_0 @cite_2
2768037715 2090299671
This paper presents a summary of the first Workshop on Building Linguistically Generalizable Natural Language Processing Systems, and the associated Build It Break It, The Language Edition shared task. The goal of this workshop was to bring together researchers in NLP and linguistics with a shared task aimed at testing the generalizability of NLP systems beyond the distributions of their training data. We describe the motivation, setup, and participation of the shared task, provide discussion of some highlighted results, and discuss lessons learned. John Dewey, one of America's greatest popular philosophers and educators, emphasises the importance of critical thinking and the vital role education should play. In this progressive work, written more than 80 years ago, Dewey, recognising that we are born with the ability to think, argues that the educator's fundamental role is to train us to think well. At a time when America is lamenting the lack of solid training in the sciences at the elementary and secondary levels, Dewey's enthusiastic correlation between the scientific mind and the natural attitude of childhood marked by curiosity, imagination, and the love of experimental inquiry offers a refreshing and optimistic perspective. He demonstrates how an appreciation of this correlation and a recognition of its value in educational practice can promote individual happiness and reduce social waste.