Term importance and salience have a long history in information retrieval: from
term frequency (tf) and inverse document frequency (idf), to recent BERT-based importance measures such as DeepCT [1]}, SPARTA [2]} and Splade [3]}, [4]}. These works mostly focus on sparse lexical retrieval and learn term weights for sparse bag-of-words representations. Term importance in multi-vector dense retrieval is less explored. Our work is probably most related to a recent work from [5]}, which prunes ColBERT by predicting salience scores from a word's embedding with a ReLU gate and L1-norm regularization.
