public static String [ ] tokenize ( String text ) { final int len = text . length ( ) ; if ( len == 0 ) { return ArrayUtils . EMPTY_STRING_ARRAY ; } final List < String > list = new ArrayList <> ( ) ; int i = 0 ; while ( i < len ) { char ch = text . charAt ( i ) ; int ch32 = text . codePointAt ( i ) ; if ( isSymbolChar ( ch32 ) ) { list . add ( new String ( Character . toChars ( ch32 ) ) ) ; i += Character . isHighSurrogate ( ch ) ? 2 : 1 ; continue ; } if ( isWordChar ( ch ) ) { int wordStart = i ; while ( i < len && isWordChar ( text . codePointAt ( i ) ) ) { i ++ ; } list . add ( text . substring ( wordStart , i ) ) ; continue ; } i ++ ; } return list . toArray ( new String [ list . size ( ) ] ) ; }
Tokenizes a string by splitting on non - word characters . This should be equivalent to splitting on \ W + in Python . The meaning of \ W is different in Android Java 7 Java 8 hence the non - use of regular expressions .