@ Override public Pair < List < String > , AtomicLong > call ( List < String > lstOfWords ) throws Exception { List < String > stops = stopWords . getValue ( ) ; Counter < String > counter = new Counter <> ( ) ; for ( String w : lstOfWords ) { if ( w . isEmpty ( ) ) continue ; if ( ! stops . isEmpty ( ) ) { if ( stops . contains ( w ) ) { counter . incrementCount ( "STOP" , 1.0f ) ; } else { counter . incrementCount ( w , 1.0f ) ; } } else { counter . incrementCount ( w , 1.0f ) ; } } wordFreqAcc . add ( counter ) ; AtomicLong lstOfWordsSize = new AtomicLong ( lstOfWords . size ( ) ) ; return new Pair <> ( lstOfWords , lstOfWordsSize ) ; }
Function to add to word freq counter and total count of words