<jupyter_start><jupyter_text>Similar sounding wordsThis is a list of similar sounding words that I have collected from various sources on the web and added to as I find new pairs.Unlike most homophone, homograph, and homonym resources this list is not targeting ESL or educational use. Instead it is designed for finding common errors in speech recognition texts. Specifically I use it with [Caster](https://caster.readthedocs.io/en/latest/) for voice programming.I currently have five different sources. I've downloaded their contents as text files, or in one case HTML and parsed appropriately. I have also linked to the original location of these files both inside the files and in the headings between Jupyter cells below.Unfortunately I wasn't thinking about reproducibility when I started this project, so most of the text files have had a bit of light preprocessing in a text editor. Given that I don't expect these source lists to change in the future, I don't think it will be a problem.<jupyter_code>from bs4 import BeautifulSoup # pip install beautifulsoup4
from disjoint_set import DisjointSet # pip install disjoint-set
import re
from pprint import pformat<jupyter_output><empty_output><jupyter_text>[7esl.html](https://7esl.com/homophones/)<jupyter_code>contents = open("7esl.html", encoding="utf8").read()
parser = BeautifulSoup(contents, 'html.parser')
similar_7esl = []
for element in parser.find_all("p"):
    candidate = element.find("strong")
    if candidate:
        partitions = candidate.text.lower().split(" —– ")
        if len(partitions) > 1:
            words = []
            for p in partitions:
                words.extend(s.strip().replace('’', "''") for s in p.split("/"))
            
            similar_7esl.append(words)
similar_7esl<jupyter_output><empty_output><jupyter_text>[ku.txt](https://web.ku.edu/~edit/wordsall.html)<jupyter_code>contents = open("ku.txt").read().lower().splitlines()[1:]
similar_ku = [s.split(';') for s in contents]
similar_ku<jupyter_output><empty_output><jupyter_text>[singularis.txt](http://www.singularis.ltd.uk/bifroest/misc/homophones-list.html)<jupyter_code>contents = open("singularis.txt").read().lower().splitlines()[1:]
similar_singularis = [s.split(', ') for s in contents]
similar_singularis<jupyter_output><empty_output><jupyter_text>[teachingtreasures.txt](https://www.teachingtreasures.com.au/teaching-tools/Basic-worksheets/worksheets-english/upper/homophones-list.htm)<jupyter_code>contents = open("teachingtreasures.txt").read().lower().splitlines()[1:]
similar_teachingtreasures = [s.split(' ') for s in contents  if s]
similar_teachingtreasures<jupyter_output><empty_output><jupyter_text>[thoughtco](https://www.thoughtco.com/homonyms-homophones-and-homographs-a-b-1692660)<jupyter_code>contents = open("thoughtco.txt").read().lower().splitlines()[1:]
similar_thoughtco = [s.split(' ') for s in contents  if s]
similar_thoughtco<jupyter_output><empty_output><jupyter_text>My personal list of words not found aboveThese were identified through trial and error (actually, just error) during dictation. Pull Requests welcome. These words tend to be commonly confused in dragon, but are not generally recognized as homophones.<jupyter_code>contents = open("numbers.txt").read().lower().splitlines()[1:]
similar_numbers = [s.lower().split(',') for s in contents  if s]
similar_numbers
contents = open("dusty.txt").read().lower().splitlines()[1:]
similar_dusty = [s.lower().split(',') for s in contents  if s]
similar_dusty<jupyter_output><empty_output><jupyter_text>Join it all togetherWe want a list of all possible sets of words. This list of lists will surely contain duplicates (in fact, mostly duplicates).I have done a visual sanity check in all the outputs above, but I'll do another below.<jupyter_code>similar_words = []
similar_words.extend(similar_7esl)
similar_words.extend(similar_ku)
similar_words.extend(similar_singularis)
similar_words.extend(similar_teachingtreasures)
similar_words.extend(similar_thoughtco)
similar_words.extend(similar_numbers)
similar_words.extend(similar_dusty)
regex = re.compile("^[a-z'-]+$")
for similar in similar_words:
    if len(set(similar)) < 2:
        print(similar)
    for word in similar:
        if not regex.match(word):
            print(word)<jupyter_output>39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
one hundred<jupyter_text>DedupRemoving duplicates is not trivial, since the different sets of words may include multiple variations (for example, one set has *your* and *you're* and another includes *yore*). It would be easy enough to just do a double loop, but disjoint sets are my favourite datastructure, and I've never actually had an opportunity to use them in production code before. Read up on the union-find algorithm if you're unfamiliar with it, it's pretty cool.<jupyter_code>word_set = DisjointSet()
for word_list in similar_words:
    for word in word_list[1:]:
        word_set.union(word_list[0], word)
    
wordsets = sorted(sorted(s) for s in word_set.itersets())
wordsets
len(wordsets)<jupyter_output><empty_output><jupyter_text>RedupeThe final output is a dictionary of words mapping to all the words similar to that word, not including that word.<jupyter_code>index = {}
for similar in wordsets:
    for word in similar:
        local = similar.copy()
        local.remove(word)
        index[word] = local
        
index
len(index)
with open("../similar_sounding_words.py", "w") as file:
    file.write("index = " + pformat(index))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10465937169078715, 'fraction_numerical': 0.026826685492410872, 'mean_word_length': 5.060962566844919, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '16855', 'license': 'MIT', 'n_tokens_mistral': 1922, 'n_tokens_neox': 1737, 'n_words': 561, 'path': 'data/parsers.ipynb', 'repo_name': 'dusty-phillips/similar-sounding-words', 'size': '5666'}