In NLP, do you distinguish tokens that you don't observe in a training sample and still expect that they may occur in a test sample, between those you know what they are, andthose you don't what they are, and how many of themIf yes, how do you treat them differently to estimate the probabilities in N-grams? Thanks!
In NLP, tokens not seen in training sample, but you know or don't know what they are
natural language processing