func Tokenize(r io.Reader) *TokenStream {
	t := newTokenizer(r)
	return NewTokenStream(t.next)
}
Tokenize tokenizes a reader into a stream of tokens. Iterate through the stream by calling Scan() or Next().

Its uses several specs from Unicode Text Segmentation https://unicode.org/reports/tr29/. It's not a full implementation, but a decent approximation for many mainstream cases.

Tokenize returns all tokens (including white space), so text can be reconstructed with fidelity.
package jargon
import (
	"bufio"
	"io"
	"strings"
	"github.com/clipperhouse/uax29/words"
)