<commit_before>This file contains a range of simple counts related to PTB data.

### Traces per sentence in wsj 2 - 21:

Traces | Count
------ | -----
1      | 12801
2      | 6198
3      | 2312
4      | 739
5      | 221
6      | 63
7      | 23
8      | 6
10     | 1
11     | 1
17     | 1


In the Training set:
  Unique words - 44389
  Pairs of word positions - 33,1247,89 (iw sum of length * (length - 1) / 2)
  Unique arc types - around 4k?
  Unique POS + Spine combos - 4822
  Unique POS + Spine combos without traces or function tags - 1653
  Average Spine options per tag, 36.1
  Average Spine options per word, 0.46
  Total number of arcs, including traces - 986847
  Total number of structural arcs - 950028

  Number of words with the specified number of POS + Spine options:
    26426 1
    7534 2
    3620 3
    2297 4
    1433 5
     936 6
     643 7
     393 8
     293 9
     190 10
     148 11
     105 12
      73 13
      63 14
      46 15
      32 16
      41 17
      15 18
      11 19
      14 20
      14 21
       9 22
       8 23
       8 24
       5 25
       5 26
       1 28
       2 29
       5 30
       5 31
       3 32
       1 33
       2 34
       1 35
       2 43
       1 45
       1 46
       1 53
       1 71
       1 153
  Number of POS with the specified number of Spine options:
     8 1
     2 2
     1 3
     2 4
     2 5
     1 6
     1 9
     2 10
     1 11
     2 12
     1 13
     1 15
     1 17
     1 18
     2 20
     1 27
     1 28
     1 29
     1 35
     1 37
     1 43
     2 45
     2 47
     1 49
     1 52
     1 64
     1 69
     1 72
     1 73
     1 104
     1 112
     1 121
     1 124
     1 178
     1 198<commit_msg>Reformatting<commit_after>This file contains a range of simple counts related to PTB data.
Note, some of these stats were calculated on an earlier version of the parse representation, but they should still be close.

### Training set info

Property                                                  | Value 
--------------------------------------------------------- | -----
Unique words                                              | 44389
Pairs of word positions                                   | 33,1247,89
Unique POS + Spine combinations                           | 4822
Unique POS + Spine combos without traces or function tags | 1653
Average Spine options per tag                             | 36.1
Average Spine options per word                            | 0.46
Total number of arcs, including traces                    | 986,847
Total number of structural arcs                           | 950,028

### Traces per sentence in wsj 2 - 21

Traces | Sentences
------ | ---------
0      | 17467
1      | 12801
2      | 6198
3      | 2312
4      | 739
5      | 221
6      | 63
7      | 23
8      | 6
10     | 1
11     | 1
17     | 1


### Unique (POS + Spine)s seen per word

(POS + Spine) Options | Words
------- | -----
1 | 26426
2 | 7534
3 | 3620
4 | 2297
5 | 1433
6 | 936
7 | 643
8 | 393
9 | 293
10 | 190
11 | 148
12 | 105
13 | 73
14 | 63
15 | 46
16 | 32
17 | 41
18 | 15
19 | 11
20 | 14
21 | 14
22 | 9
23 | 8
24 | 8
25 | 5
26 | 5
28 | 1
29 | 2
30 | 5
31 | 5
32 | 3
33 | 1
34 | 2
35 | 1
43 | 2
45 | 1
46 | 1
53 | 1
71 | 1
153 | 1


### Unique spine options seen per POS

Spine Options | POS
------ | ---
198 | 1
178 | 1
124 | 1
121 | 1
112 | 1
104 | 1
73 | 1
72 | 1
69 | 1
64 | 1
52 | 1
49 | 1
47 | 2
45 | 2
43 | 1
37 | 1
35 | 1
29 | 1
28 | 1
27 | 1
20 | 2
18 | 1
17 | 1
15 | 1
13 | 1
12 | 2
11 | 1
10 | 2
9 | 1
6 | 1
5 | 2
4 | 2
3 | 1
2 | 2
1 | 8
{'fraction_non_alphanumeric': 0.08413597733711048, 'fraction_numerical': 0.19405099150141644, 'mean_word_length': 1.1271084337349397, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6395120', 'n_tokens_mistral': 1680, 'n_tokens_neox': 1180, 'n_words': 565}