We establish performance baselines on FEWS with both knowledge-based approaches and a recent neural biencoder model for WSD [1]}. We find that the biencoder, despite being the strongest baseline on FEWS, still underperforms human annotators, particularly on zero-shot senses where the biencoder trails by more than 10%. We also present transfer learning experiments and find adding FEWS as additional training data improves performance on all but the most frequent senses (MFS) in the WSD Evaluation Framework [2]}; this suggests that future improvements on FEWS could generalize other WSD benchmarks. FEWS is available at https://nlp.cs.washington.edu/fews.
