public void setMaxTokenLength ( int maxTokenLength ) { if ( maxTokenLength < 1 ) throw new IllegalArgumentException ( "Max token length must be positive, not " + maxTokenLength ) ; if ( maxTokenLength <= minTokenLength ) throw new IllegalArgumentException ( "Max token length must be larger than the min token length" ) ; this . maxTokenLength = maxTokenLength ; }
Sets the maximum allowed length for any token . Any token discovered exceeding the length will not be accepted and skipped over . The default is unbounded .