<commit_before># English confusion sets, mostly homophones
# Original source: After the Deadline's data/rules/homophonedb.txt
# URL: http://openatd.wordpress.com/ or https://openatd.svn.wordpress.org/atd-server/data/rules/homophonedb.txt
# License: GNU GENERAL PUBLIC LICENSE Version 2
#
# Line format:
# <word1>|<description1>; <word2>|<description2>; <factor>   # optional comment
#   <word1> and <word2> are words that can easily be confused
#   <description> will be used in the error message to explain the word (optional)
#   <factor> is the factor of how much more the other word must be more
#            probable so the text is considered potentially incorrect.
#            Use a higher value for better precision but lower recall.
#   Precision (p) and recall (r) values in the comments come from ConfusionRuleEvaluator
#     and are based on Wikipedia, Tatoeba, and Enron data, usually with 1000 random 
#     sentences (if that many sentence are available for that pair). The number after recall
#     is the number of sentences used for evaluation.
# Order is relevant for ambiguous cases like 'know' ('no' or 'now') where the match
# is used whose pair comes first in this file.

accept|to agree, to endure; except|with the exception of; 100000                  # p=0.997, r=0.792, 1900, 3grams, 2015-08-12
advice|noun; advise|verb; 100000                                                  # p=0.998, r=0.792, 1647, 3grams, 2015-08-12
affect|experience of emotion; effect|outcome, result; 10000000                    # p=0.982, r=0.595, 1805, 3grams, 2015-08-12
allowed|permitted; aloud|using the voice; 10000                                   # p=0.999, r=0.895, 1056, 3grams, 2015-08-12
allusion|passing reference; illusion|an erroneous mental representation; 1000     # p=0.994, r=0.796, 211, 3grams, 2015-08-12
and; end; 1000000000000                                                           # p=0.998, r=0.530, 1925, 3grams, 2015-08-12
ate|past of 'to eat'; eight|the number '8'; 100                                   # p=0.996, r=0.938, 1658, 3grams, 2015-08-12
Australia|the continent; Austria|country in Europe; 10000                         # p=0.995, r=0.279, 1977, 3grams, 2015-08-12
bean; been; 1000                                                                  # p=0.995, r=0.923, 1146, 3grams, 2015-08-12
be; bee|insects; 1000000                                                          # p=0.998, r=0.835, 1162, 3grams, 2015-08-12
been; bin|container; 100000                                                       # p=0.998, r=0.916, 1452, 3grams, 2015-08-12
begin|start; being|living thing; 100000000                                        # p=1.000, r=0.558, 1932, 3grams, 2015-08-12
be; we; 10000000                                                                  # p=1.000, r=0.869, 1889, 3grams, 2015-08-12
bit; but; 10000000                                                                # p=0.996, r=0.799, 1950, 3grams, 2015-08-12
breathe|verb; breath|noun; 100000                                                 # p=1.000, r=0.768, 453, 3grams, 2015-08-12
but; butt; 100000                                                                 # p=0.993, r=0.836, 1145, 3grams, 2015-08-12
buy|purchase; bye|goodbye; 1000                                                   # p=1.000, r=0.832, 1041, 3grams, 2015-08-12
desert|arid land; dessert|last course of a meal; 1000000                          # p=0.996, r=0.491, 1036, 3grams, 2015-08-12
except|with the exception of; expect|regard something as likely; 10000000         # p=0.998, r=0.667, 1909, 3grams, 2015-08-12
extend|verb; extent|noun; 100                                                     # p=0.998, r=0.975, 1780, 3grams, 2015-08-12
fill; full; 1000000                                                               # p=0.997, r=0.770, 1798, 3grams, 2015-08-12
first; fist; 1000000000                                                           # p=0.994, r=0.800, 1076, 3grams, 2015-08-12
for|a preposition as in 'I have something for you'; four|the number '4'; 100000000# p=0.994, r=0.507, 1890, 3grams, 2015-08-12
had|past of 'have'; hat|head covering; 100000000                                  # p=0.998, r=0.647, 1757, 3grams, 2015-08-12
half|noun/adjective; halve|verb; 1000000                                          # p=0.997, r=0.961, 991, 3grams, 2015-08-12
hear; heart; 1000                                                                 # p=0.996, r=0.954, 1971, 3grams, 2015-08-12
in; inn|a hostel; 1000000                                                         # p=0.999, r=0.829, 1145, 3grams, 2015-08-12
kind; king; 1000000                                                               # p=0.997, r=0.533, 1963, 3grams, 2015-08-12
known|past participle of 'know'; know|to be aware of; 1000                        # p=0.999, r=0.965, 1946, 3grams, 2015-08-12
know|to be aware of; no|negation; 100000                                          # p=0.999, r=0.887, 1903, 3grams, 2015-08-12
know|to be aware of; now|in this moment; 1000                                     # p=0.999, r=0.877, 1918, 3grams, 2015-08-12
lies; lines; 1000000                                                              # p=0.999, r=0.556, 1953, 3grams, 2015-08-12
loose|not tight; lose|opposite of win; 100000                                     # p=0.996, r=0.796, 1421, 3grams, 2015-08-12
mice|plural of 'mouse'; nice|pleasant; 1000                                       # p=0.997, r=0.862, 1215, 3grams, 2015-08-12
minuet|piece of music and dance; minute|60 seconds; 1000000000                    # p=0.995, r=0.568, 995, 3grams, 2015-08-12
minuets|pieces of music and dance; minutes|60 seconds; 1000000000                 # p=0.993, r=0.881, 1007, 3grams, 2015-08-12
near|close to; neat|clean or organized; 100000                                    # p=1.000, r=0.784, 1074, 3grams, 2015-08-12
new; news; 1000000                                                                # p=0.997, r=0.542, 1814, 3grams, 2015-08-12
nit; not; 100000000                                                               # p=0.998, r=0.945, 983, 3grams, 2015-08-12
not; note; 100000000                                                              # p=0.997, r=0.525, 1865, 3grams, 2015-08-12
now|at the present moment; no|negation; 1000000                                   # p=0.996, r=0.559, 1919, 3grams, 2015-08-12
our; out; 10000                                                                   # p=0.996, r=0.896, 1901, 3grams, 2015-08-12
passed|paste tense of 'to pass'; past|past times; 10000                           # p=0.999, r=0.858, 1938, 3grams, 2015-08-12
peace|absence of war; piece|a portion of a natural object; 1000000                # p=0.998, r=0.634, 1966, 3grams, 2015-08-12
peak|extremum, summit, top; peek|a secret look; 100000                            # p=0.999, r=0.797, 999, 3grams, 2015-08-12
pray|say a prayer; prey|animal hunted for food; 10000                             # p=0.994, r=0.794, 664, 3grams, 2015-08-12
proof|often noun, also 'proofread'; prove|verb: show, testify, demonstrate; 100000# p=0.997, r=0.850, 1583, 3grams, 2015-08-12
quiet|free of noise; quite|completely, to a degree; 100000                        # p=0.999, r=0.808, 1601, 3grams, 2015-08-12
quite|completely, to a degree; quit|stop, give up; 100000                         # p=0.998, r=0.837, 1519, 3grams, 2015-08-12
right|opposite of 'left'; rite|established ceremony or practice; 100000000        # p=0.995, r=0.698, 1175, 3grams, 2015-08-12
same|identical; sane|mentally healthy; 10000                                      # p=1.000, r=0.962, 1028, 3grams, 2015-08-12
scene|place where some action occurs; seen|past participle of 'see'; 1000         # p=1.000, r=0.920, 1979, 3grams, 2015-08-12
sea|ocean; see|to look at; 10000                                                  # p=0.996, r=0.869, 1947, 3grams, 2015-08-12
sire|noun: title, verb: make by reproduction; sure|certain; 100000                # p=0.997, r=0.936, 987, 3grams, 2015-08-12
than; that; 100000000                                                             # p=0.999, r=0.496, 1902, 3grams, 2015-08-12
than|used in comparisons; then|next in order; 100000                              # p=0.999, r=0.789, 1949, 3grams, 2015-08-12
their|as in 'It’s not their fault.'; there|as in 'Is there an answer?'; 1000      # p=0.994, r=0.966, 1939, 3grams, 2015-08-12
the; to; 10000000                                                                 # p=0.995, r=0.639, 1902, 3grams, 2015-08-12
thing|noun; think|usually a verb; 1000                                            # p=0.997, r=0.953, 1930, 3grams, 2015-08-12
threw|past of throw; through|preposition, e.g. 'go through the door'; 100000000   # p=0.996, r=0.613, 1375, 3grams, 2015-08-12
vary|change, alter; very|to a great extent; 100000                                # p=0.998, r=0.873, 1972, 3grams, 2015-08-12
waist|body part between the ribs and hips; waste|worthless material; 1000         # p=0.996, r=0.860, 903, 3grams, 2015-08-12<commit_msg>[en] better precision for pair the/to, as these words are so common<commit_after># English confusion sets, mostly homophones
# Original source: After the Deadline's data/rules/homophonedb.txt
# URL: http://openatd.wordpress.com/ or https://openatd.svn.wordpress.org/atd-server/data/rules/homophonedb.txt
# License: GNU GENERAL PUBLIC LICENSE Version 2
#
# Line format:
# <word1>|<description1>; <word2>|<description2>; <factor>   # optional comment
#   <word1> and <word2> are words that can easily be confused
#   <description> will be used in the error message to explain the word (optional)
#   <factor> is the factor of how much more the other word must be more
#            probable so the text is considered potentially incorrect.
#            Use a higher value for better precision but lower recall.
#   Precision (p) and recall (r) values in the comments come from ConfusionRuleEvaluator
#     and are based on Wikipedia, Tatoeba, and Enron data, usually with 1000 random 
#     sentences (if that many sentence are available for that pair). The number after recall
#     is the number of sentences used for evaluation.
# Order is relevant for ambiguous cases like 'know' ('no' or 'now') where the match
# is used whose pair comes first in this file.

accept|to agree, to endure; except|with the exception of; 100000                  # p=0.997, r=0.792, 1900, 3grams, 2015-08-12
advice|noun; advise|verb; 100000                                                  # p=0.998, r=0.792, 1647, 3grams, 2015-08-12
affect|experience of emotion; effect|outcome, result; 10000000                    # p=0.982, r=0.595, 1805, 3grams, 2015-08-12
allowed|permitted; aloud|using the voice; 10000                                   # p=0.999, r=0.895, 1056, 3grams, 2015-08-12
allusion|passing reference; illusion|an erroneous mental representation; 1000     # p=0.994, r=0.796, 211, 3grams, 2015-08-12
and; end; 1000000000000                                                           # p=0.998, r=0.530, 1925, 3grams, 2015-08-12
ate|past of 'to eat'; eight|the number '8'; 100                                   # p=0.996, r=0.938, 1658, 3grams, 2015-08-12
Australia|the continent; Austria|country in Europe; 10000                         # p=0.995, r=0.279, 1977, 3grams, 2015-08-12
bean; been; 1000                                                                  # p=0.995, r=0.923, 1146, 3grams, 2015-08-12
be; bee|insects; 1000000                                                          # p=0.998, r=0.835, 1162, 3grams, 2015-08-12
been; bin|container; 100000                                                       # p=0.998, r=0.916, 1452, 3grams, 2015-08-12
begin|start; being|living thing; 100000000                                        # p=1.000, r=0.558, 1932, 3grams, 2015-08-12
be; we; 10000000                                                                  # p=1.000, r=0.869, 1889, 3grams, 2015-08-12
bit; but; 10000000                                                                # p=0.996, r=0.799, 1950, 3grams, 2015-08-12
breathe|verb; breath|noun; 100000                                                 # p=1.000, r=0.768, 453, 3grams, 2015-08-12
but; butt; 100000                                                                 # p=0.993, r=0.836, 1145, 3grams, 2015-08-12
buy|purchase; bye|goodbye; 1000                                                   # p=1.000, r=0.832, 1041, 3grams, 2015-08-12
desert|arid land; dessert|last course of a meal; 1000000                          # p=0.996, r=0.491, 1036, 3grams, 2015-08-12
except|with the exception of; expect|regard something as likely; 10000000         # p=0.998, r=0.667, 1909, 3grams, 2015-08-12
extend|verb; extent|noun; 100                                                     # p=0.998, r=0.975, 1780, 3grams, 2015-08-12
fill; full; 1000000                                                               # p=0.997, r=0.770, 1798, 3grams, 2015-08-12
first; fist; 1000000000                                                           # p=0.994, r=0.800, 1076, 3grams, 2015-08-12
for|a preposition as in 'I have something for you'; four|the number '4'; 100000000# p=0.994, r=0.507, 1890, 3grams, 2015-08-12
had|past of 'have'; hat|head covering; 100000000                                  # p=0.998, r=0.647, 1757, 3grams, 2015-08-12
half|noun/adjective; halve|verb; 1000000                                          # p=0.997, r=0.961, 991, 3grams, 2015-08-12
hear; heart; 1000                                                                 # p=0.996, r=0.954, 1971, 3grams, 2015-08-12
in; inn|a hostel; 1000000                                                         # p=0.999, r=0.829, 1145, 3grams, 2015-08-12
kind; king; 1000000                                                               # p=0.997, r=0.533, 1963, 3grams, 2015-08-12
known|past participle of 'know'; know|to be aware of; 1000                        # p=0.999, r=0.965, 1946, 3grams, 2015-08-12
know|to be aware of; no|negation; 100000                                          # p=0.999, r=0.887, 1903, 3grams, 2015-08-12
know|to be aware of; now|in this moment; 1000                                     # p=0.999, r=0.877, 1918, 3grams, 2015-08-12
lies; lines; 1000000                                                              # p=0.999, r=0.556, 1953, 3grams, 2015-08-12
loose|not tight; lose|opposite of win; 100000                                     # p=0.996, r=0.796, 1421, 3grams, 2015-08-12
mice|plural of 'mouse'; nice|pleasant; 1000                                       # p=0.997, r=0.862, 1215, 3grams, 2015-08-12
minuet|piece of music and dance; minute|60 seconds; 1000000000                    # p=0.995, r=0.568, 995, 3grams, 2015-08-12
minuets|pieces of music and dance; minutes|60 seconds; 1000000000                 # p=0.993, r=0.881, 1007, 3grams, 2015-08-12
near|close to; neat|clean or organized; 100000                                    # p=1.000, r=0.784, 1074, 3grams, 2015-08-12
new; news; 1000000                                                                # p=0.997, r=0.542, 1814, 3grams, 2015-08-12
nit; not; 100000000                                                               # p=0.998, r=0.945, 983, 3grams, 2015-08-12
not; note; 100000000                                                              # p=0.997, r=0.525, 1865, 3grams, 2015-08-12
now|at the present moment; no|negation; 1000000                                   # p=0.996, r=0.559, 1919, 3grams, 2015-08-12
our; out; 10000                                                                   # p=0.996, r=0.896, 1901, 3grams, 2015-08-12
passed|paste tense of 'to pass'; past|past times; 10000                           # p=0.999, r=0.858, 1938, 3grams, 2015-08-12
peace|absence of war; piece|a portion of a natural object; 1000000                # p=0.998, r=0.634, 1966, 3grams, 2015-08-12
peak|extremum, summit, top; peek|a secret look; 100000                            # p=0.999, r=0.797, 999, 3grams, 2015-08-12
pray|say a prayer; prey|animal hunted for food; 10000                             # p=0.994, r=0.794, 664, 3grams, 2015-08-12
proof|often noun, also 'proofread'; prove|verb: show, testify, demonstrate; 100000# p=0.997, r=0.850, 1583, 3grams, 2015-08-12
quiet|free of noise; quite|completely, to a degree; 100000                        # p=0.999, r=0.808, 1601, 3grams, 2015-08-12
quite|completely, to a degree; quit|stop, give up; 100000                         # p=0.998, r=0.837, 1519, 3grams, 2015-08-12
right|opposite of 'left'; rite|established ceremony or practice; 100000000        # p=0.995, r=0.698, 1175, 3grams, 2015-08-12
same|identical; sane|mentally healthy; 10000                                      # p=1.000, r=0.962, 1028, 3grams, 2015-08-12
scene|place where some action occurs; seen|past participle of 'see'; 1000         # p=1.000, r=0.920, 1979, 3grams, 2015-08-12
sea|ocean; see|to look at; 10000                                                  # p=0.996, r=0.869, 1947, 3grams, 2015-08-12
sire|noun: title, verb: make by reproduction; sure|certain; 100000                # p=0.997, r=0.936, 987, 3grams, 2015-08-12
than; that; 100000000                                                             # p=0.999, r=0.496, 1902, 3grams, 2015-08-12
than|used in comparisons; then|next in order; 100000                              # p=0.999, r=0.789, 1949, 3grams, 2015-08-12
their|as in 'It’s not their fault.'; there|as in 'Is there an answer?'; 1000      # p=0.994, r=0.966, 1939, 3grams, 2015-08-12
the; to; 100000000                                                                # p=0.997, r=0.590, 2000, 3grams, 2015-08-12
thing|noun; think|usually a verb; 1000                                            # p=0.997, r=0.953, 1930, 3grams, 2015-08-12
threw|past of throw; through|preposition, e.g. 'go through the door'; 100000000   # p=0.996, r=0.613, 1375, 3grams, 2015-08-12
vary|change, alter; very|to a great extent; 100000                                # p=0.998, r=0.873, 1972, 3grams, 2015-08-12
waist|body part between the ribs and hips; waste|worthless material; 1000         # p=0.996, r=0.860, 903, 3grams, 2015-08-12
{'fraction_non_alphanumeric': 0.11453066608045724, 'fraction_numerical': 0.18800835348428227, 'mean_word_length': 1.5789399092970522, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 34, 'word_list_counts': {'cursed_substrings.json': 140, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5941338', 'n_tokens_mistral': 8176, 'n_tokens_neox': 5375, 'n_words': 1662}