<jupyter_start><jupyter_text>Exploring the names database I might like to try some sort of frequency-weighting for namegen, but instead of including each name the same number of times that it appears in the corpus, I will try to find some sort of transformation to make it a little nicer. I should note that I am not using any sort of "best-practices" or theory-driven decision making, because the goal isn't to make these look like real names or anything, but rather to "sort of generate some stuff."<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import gaussian_kde<jupyter_output><empty_output><jupyter_text>Import names database:<jupyter_code>d = pd.read_csv("NationalNames.csv")<jupyter_output><empty_output><jupyter_text>Get a sense of the disribution of occurrences per name:<jupyter_code>def print_summ(arr):
    print("Range: {} to {}".format(str(round(min(arr), 2)), str(round(max(arr), 2))))
    print("Mean: {}".format(str(round(np.mean(arr), 2))))
    print("Std. dev.: {}".format(str(round(np.std(arr), 2))))
    print("Median: {}".format(str(round(np.median(arr), 2))))

print_summ(d.Count)<jupyter_output>Range: 5 to 99680
Mean: 184.69
Std. dev.: 1566.71
Median: 12.0<jupyter_text>How about some natural logs?<jupyter_code>lcount = np.log(d.Count)
print_summ(lcount)
density = gaussian_kde(np.log(d.Count))
# this will be rough
dens_xs = np.linspace(0, np.log(max(d.Count)), 200)
plt.plot(dens_xs, density(dens_xs))
plt.show()
xs = np.arange(1, len(lcount)+1)
plt.plot(xs, np.sort(lcount))
plt.show()<jupyter_output><empty_output><jupyter_text>Just to get a sense of the proportions:<jupyter_code>len(list(filter(lambda val: val < 11, lcount)))/len(lcount)
len(list(filter(lambda val: val < 4, lcount)))/len(lcount)
sum(lcount.astype(int))/len(lcount)<jupyter_output><empty_output><jupyter_text>So it's not going to be too huge, and I can add 1 and then truncate.<jupyter_code>transcount = (lcount+1).astype(int)
sum(transcount)<jupyter_output><empty_output><jupyter_text>Then make the list that way:<jupyter_code>transd = {'name': d.Name, 'count': transcount}
td = pd.DataFrame(transd)
newlist = []
for row in td.itertuples():
    # (Index=_, count=_, name=_)
    newlist.extend([row[2]]*row[1])
#     print(row)
#     if len(newlist) > 50:
#         break
print(len(newlist), newlist[:10])<jupyter_output>6182649 ['Mary', 'Mary', 'Mary', 'Mary', 'Mary', 'Mary', 'Mary', 'Mary', 'Mary', 'Anna']<jupyter_text>Looks good. Saving...<jupyter_code>with open("ext_names.txt", "w") as f:
    f.write("\n".join(newlist))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1514919663351186, 'fraction_numerical': 0.01874521805661821, 'mean_word_length': 6.243767313019391, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '446915', 'license': 'MIT', 'n_tokens_mistral': 1019, 'n_tokens_neox': 952, 'n_words': 242, 'path': 'explore_names.ipynb', 'repo_name': 'cranndarach/namegen', 'size': '2614'}