public static String ptb2Text ( String ptbText ) { StringBuilder sb = new StringBuilder ( ptbText . length ( ) ) ; // probably an overestimate PTB2TextLexer lexer = new PTB2TextLexer ( new StringReader ( ptbText ) ) ; try { for ( String token ; ( token = lexer . next ( ) ) != null ; ) { sb . append ( token ) ; } } catch ( IOException e ) { e . printStackTrace ( ) ; } return sb . toString ( ) ; }
Returns a presentable version of the given PTB - tokenized text . PTB tokenization splits up punctuation and does various other things that makes simply joining the tokens with spaces look bad . So join the tokens with space and run it through this method to produce nice looking text . It s not perfect but it works pretty well .