@ Override public List < String > tokenize ( String text ) { List < String > tokens = new ArrayList <> ( Arrays . asList ( text . split ( "[\\p{Z}\\p{C}]+" ) ) ) ; return tokens ; }
Separates the tokens of a string by splitting it on white space .