Distance-based ( windowless ) word assocation measures have only very recently appeared in the NLP literature and their performance compared to existing windowed or frequency-based measures is largely unknown . We conduct a largescale empirical comparison of a variety of distance-based and frequency-based measures for the reproduction of syntagmatic human assocation norms . Overall , our results show an improvement in the predictive power of windowless over windowed measures . This provides support to some of the previously published theoretical advantages and makes windowless approaches a promising avenue to explore further . This study also serves as a first comparison of windowed methods across numerous human association datasets . During this comparison we also introduce some novel variations of window-based measures which perform as well as or better in the human association norm task than established measures . Automatic discovery of semantically associated words has attracted a large amount of attention in the last decades and a host of computational association measures have been proposed to deal with this task ( see Section 2 ) . These measures traditionally rely on the co-ocurrence frequency of two words in a corpus to estimate a relatedness score . There has been a recent emergence of distancebased language modelling techiques in NLP ( Savicki and Hlavacova , 2002 ; Terra and Clarke , 2004 ) in which the number of tokens separating words is the essential quantity . While some of this work has considered distance-based alternatives to conventional association measures ( Hardcastle , 2005 ; Washtell , 2009 ) , there has been no principled empirical evaluation of these measures as predictors of human association . We remedy this by conducting a thorough comparison of a wide variety of frequency-based and distance-based measures as predictors of human association scores as elicited in several different free word association tasks . In this work we focus on first-order association measures as predictors of syntagmatic associations . This is in contrast to second and higher-order measures which are better predictors of paradigmatic associations , or word similarity . The distinction between syntagmatic and paradigmatic relationship types is neither exact nor mutually exclusive , and many paradigmatic relationships can be observed syntagmatically in the text . Roughly in keeping with ( Rapp , 2002 ) , we hereby regard paradigmatic assocations as those based largely on word similarity ( i.e. including those typically classed as synonyms , antonyms , hypernyms , hyponyms etc ) , whereas syntagmatic associations are all those words which strongly invoke one another yet which can not readily be said to be similar . Typically these will have an identifiable semantic or grammatical relationship ( meronym / holonym : stem -flower , verb / object : eat -food etc ) , or may have harder-to-classify topical or idiomatic relationships ( family -Christmas , rock -roll ) . We will show in Section 3.2 that syntagmatic relations by themselves constitute a substantial 25 - 40 % of the strongest human responses to cue words . Although the automatic detection of these assocations in text has received less attention than that of paradigmatic associations , they are nonetheless important in applications such as the resolution of bridging anaphora ( Vieira and Poesio , 2000 ) . 1 Furthermore , first-order associations are often the basis of higher-order vector wordspace models used for predicting paradigmatic relationships : i.e. through the observation of words which share similar sets of syntagmatic associations . Therefore improvements made at the level we are concerned with may reasonably be expected to carry through to applications which hinge on the identification of paradigmatic relationships . After a discussion of previous work in Section 2 , we formulate the exact association measures and parameter settings which we compare in Section 3 , where we also introduce the corpora and human association sets used . Then , by using evaluations similar to those described in ( Baroni et al . , 2008 ) and by Rapp ( 2002 ) , we show that the best distance-based measures correlate better overall with human association scores than do the best window based configurations ( see Section 4 ) , and that they also serve as better predictors of the strongest human associations ( see Section 5 ) . This paper presented the first empirical comparison of window-based and the relatively recently introduced windowless association measures , using their ability to reproduce human association scores as a testbed . We show that the best windowless measures are always at least as good as the best window-based measures , both when it comes to overall correlation with human association scores and predicting the strongest human response . In addition , for several human association sets , they perform significantly better . Although not all parameter settings and corpus sizes could be explored , we conclude that it is worthwhile investigating windowless association measures further . As a side-benefit , we have also introduced new variants of existing frequency-based association measures and shown them to perform as well as or better than their existing counterparts . Although these measures were semi-principled in their construction , a deeper understanding of why they work so well is needed . This may in turn lead to the construction of superior windowless measures . In our own future work , we are especially interested in using higher-order windowless association measures for retrieving paradigmatic relations as well as exploring their use in various NLP applications .
The performance of new windowless word association measures which take the number of tokens separating words into account remains unknown.
They conduct large-scale empirical comparisons of window-based and windowless association measures for the reproduction of syntagmatic human association norms.
The best windowless measures perform on part with best window-based measures on correlation with human association scores.