<commit_before># text/scanner包函数列表

- [Constants](Constants.md)
- [func TokenString(tok rune) string](TokenString.md)
- [type Position](Position.md)
    - [func (pos *Position) IsValid() bool](IsValid.md)
    - [func (pos Position) String() string](String.md)
- type Scanner
	Scanner用于从io.Reader中读取Unicode字符或词法单元
    - [func (s *Scanner) Init(src io.Reader) *Scanner](Init.md)
    - [func (s *Scanner) Next() rune](Next.md)
    - [func (s *Scanner) Peek() rune](Peek.md)
    - [func (s *Scanner) Pos() (pos Position)](Pos.md)
    - [func (s *Scanner) Scan() rune](Scan.md)
    - [func (s *Scanner) TokenText() string](TokenText.md)
<commit_msg>markdown syntax<commit_after># text/scanner包函数列表

- [Constants](Constants.md)
- [func TokenString(tok rune) string](TokenString.md)
- [type Position](Position.md)
    - [func (pos *Position) IsValid() bool](IsValid.md)
    - [func (pos Position) String() string](String.md)
- type Scanner

	Scanner用于从io.Reader中读取Unicode字符或词法单元

    - [func (s *Scanner) Init(src io.Reader) *Scanner](Init.md)
    - [func (s *Scanner) Next() rune](Next.md)
    - [func (s *Scanner) Peek() rune](Peek.md)
    - [func (s *Scanner) Pos() (pos Position)](Pos.md)
    - [func (s *Scanner) Scan() rune](Scan.md)
    - [func (s *Scanner) TokenText() string](TokenText.md)

{'fraction_non_alphanumeric': 0.1826625386996904, 'fraction_numerical': 0.0, 'mean_word_length': 4.05078125, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5653688', 'n_tokens_mistral': 509, 'n_tokens_neox': 515, 'n_words': 109}