private String segmenterNonBreaker ( String line ) { // these are fine because they do not affect offsets line = line . trim ( ) ; line = RuleBasedTokenizer . doubleSpaces . matcher ( line ) . replaceAll ( " " ) ; final StringBuilder sb = new StringBuilder ( ) ; String segmentedText = "" ; int i ; final String [ ] words = line . split ( " " ) ; // iterate over the words for ( i = 0 ; i < words . length - 1 ; i ++ ) { final Matcher nonSegmentedWordMatcher = nonSegmentedWords . matcher ( words [ i ] ) ; // candidate word to be segmented found: if ( nonSegmentedWordMatcher . find ( ) ) { final String curWord = nonSegmentedWordMatcher . replaceAll ( "$1" ) ; final String finalPunct = nonSegmentedWordMatcher . replaceAll ( "$2" ) ; if ( ! curWord . isEmpty ( ) && curWord . matches ( "(" + this . NON_BREAKER + ")" ) && finalPunct . isEmpty ( ) ) { // if current word is not empty and is a no breaker and there is not // final punctuation } else if ( acronym . matcher ( words [ i ] ) . find ( ) ) { // if acronym } else if ( nextCandidateWord . matcher ( words [ i + 1 ] ) . find ( ) ) { // if next word contains initial punctuation and then uppercase or // digit do: if ( ! ( ! curWord . isEmpty ( ) && curWord . matches ( NON_BREAKER_DIGITS ) && finalPunct . isEmpty ( ) && startDigit . matcher ( words [ i + 1 ] ) . find ( ) ) ) { // segment unless current word is a non breaker digit and next word // is not final punctuation or does not start with a number words [ i ] = words [ i ] + "\n" ; } } } sb . append ( words [ i ] ) . append ( " " ) ; segmentedText = sb . toString ( ) ; } // add last index of words array removed for easy look ahead segmentedText = segmentedText + words [ i ] ; return segmentedText ; }
This function implements exceptions for periods as sentence breakers . It decides when a period induces a new sentence or not .