This is the corpusabcdefabcdxabcdbbcefefaefwhat I want to extract is abcd, ef, then I try to count all n-grams:abcdef 1abcd 3abc 3...My question is how to distinguish abcd with abcdef and abc, since abcdef just appear once, and abc is the sub-string of 'abcd'. Of course, I can filter with this two conditions, but just want to know any existing algorithm target on solving this problem.
how to extract high frequency phrase
data mining;dataset;statistics;text mining