<jupyter_start><jupyter_text>Historical change Even if the core meaning of community has not changed over the 20th century, it might have evolved new connotations or come to be used in new contexts. Here I examine the temporal stability of "community" using historical word embeddings.Both NLP researchers and social scientists have used word embeddings to study semantic change over time. To do this, distinct embeddings models are trained on different time periods, and then embeddings for particular words can be compared. (Comparing vectors *across* models is a tricky problem, see below.) For instance, the word "gay" shifted from meaning "happy" to meaning "homosexual" in the mid-late 20th century. Multiple historical embeddings papers (Kulkarni et al 2015, Hamilton et al 2016) use this example to demonstrate that embeddings can encode and uncover this kind of semantic change. Mendelsohn et al (2020) extend this work further by noting a more recent shift (since the 1980s) in the valence or connotation of the word "gay", from more negative to more positive. I'll use this conventional example as a comparison and reference in my explorations of "community" below.To measure and visualize semantic change over time, researchers adapt the same metrics of cosine similarity and distance used for other comparisons. First, the simplest approach is to calculate the similarity between a pair of words *within* each time period, and then to show how that similarity measure increases or decreases over time. This can be generalized to compare, for instance, binary axes across time (Kozlowski et al 2019). Second, a single word can be compared with itself *across* time periods; for instance, the distance from a word's vector in the first time period can be calculated for each subsequent time period. Finally, distances can be projected into two-dimensions with dimensionality-reduction techniques like t-SNE, to visualize how a focal word moves in relation to other neighboring words.The most common source for historical word coocurrence data is the Google Ngram corpus (for instance, Kulkarni et al 2015, Hamilton et al 2016, Kozlowski et al 2019). An N-gram is a series of words of length N. Embeddings trained on Google Ngrams typically use 5-grams only. Because the composition of the Google Ngram corpus shifts over the 20th century and especially in the 2000s, it generally isn't considered comparable after the 1990s (Pechenick et al 2015, note that they recommend the fiction corpus over the all-English corpus). This makes it difficult to bridge the gap with contemporary embeddings trained on Wikipedia and Twitter.Alternative data sources include the Corpus of Historical American English (COHA, used in Hamilton et al 2016 and Stoltz and Taylor 2020), and articles published in the New York Times (Mendelsohn et al 2020; Arseniev-Koehler and Foster 2020 also use a 30-year NYT corpus, but do not train separate models on different years).   Notes on prior methods & code HistWords (Hamilton et al 2016)https://nlp.stanford.edu/projects/histwords/https://github.com/williamleif/histwordsI download and use these pretrained historical embeddings from Histwords. These HistWords embeddings are trained on Google N-Grams with skip-gram negative sampling / word2vec. I primarily use the all-English embeddings, but I've also briefly investigated the fiction embeddings. They use 10-year time periods from 1800 through 1990, inclusive. Hamilton's research code is written in Python 2, and he recommends using gensim instead. In a previous iteration of this notebook, I was able to get some of his code, for loading the historical embeddings and calculating pairwise cosine similarities over time, to run in Python 3 with a few modifications. (I don't recommend ad-hoc modifying code like that!)The [viz code](https://github.com/williamleif/histwords/tree/master/viz), including an interactive explorer, can be run in a separate Python 2 conda environment. As the readme notes for macOS, I had to use pythonw in place of python. To install a python 2.7 environment to run the original histwords code:```conda create -n histwords-env python=2.7conda activate histwords-envpip install -r requirements.txtconda install python.app```Then to generate plots in viz/output/: ```pythonw scripts/closest_over_time_with_anns.py community```The static plot code is hard-coded to use the all-English embeddings. This viz code produces more detailed plots than the simplified diagrams in the actual paper. I found the code challenging to read, and I'm not certain that the t-SNE plots it produces exactly match the algorithm outlined in Appendix B of the paper. The key difference (I think!) is that the algorithm in the paper describes using one set of nearest neighbors for all years, and holds all words but the focal word fixed at the latest value when recalculating the position of the focal word. The code actually calculates a new t-SNE on all the nearest neighbors for a given year, which means that other words can move as well. If other words' embeddings change little, results should be fairly similar.Hamilton et al's measure of **polysemy** is also of interest to me, potentially for future investigation, because my intuition is that "community" is a relatively polysemous concept. They operationalize polysemy as "contextual diversity", constructing a cooccurence network on the top 10,000 words, excluding stopwords, using PPMI. They then use a local clustering coefficient on that network, citing Watts and Strogatz. For their regression analyses, they use normalized cosine distances between time t and time t+1. This doesn't feature in their plots of individual words. Kulkarni et al 2015https://github.com/viveksck/langchangetrackhttps://github.com/viveksck/changepointThis paper precedes Hamilton et al, but I haven't read through the code or attempted to run it. According to the paper, they align matrices by piecewise linear regression and then create a time series of distances from the original position of the word at time zero. The y-axes of their figures are those distances. The distance metric they use seems to be cosine distance as well. Kozlowski et al 2019https://github.com/KnowledgeLab/GeometryofCulture/They plot pairwise cosine similarity over time between different binary dimensions of social class and the dimension of affluence. They repeat this plot for six other dimensions. They do this for the Google Ngram corpus in the main paper, and repeat it for the JSTOR Sociology corpus in the appendix.They also plot the correlation of the projections of vocabulary words onto each of these seven dimensions over time. This projection [is equivalent to cosine similarity](https://github.com/KnowledgeLab/GeometryofCulture/blob/master/code/build_cultural_dimensions.RL68) between the words and the dimensions. I need to read the code more closely, but this may also be akin to [the `>>` operator in whatlies](https://github.com/RasaHQ/whatlies/blob/master/whatlies/embedding.pyL132).The repository includes example scripts, rather than complete replication code. Stoltz and Taylor 2020Stoltz and Taylor use pretrained HistWords embeddings. Specifically, they use the genre-balanced COHA embeddings. They plot changes in pairwise cosine similarity. They also plot cosine similarity of particular words to two binary axes simultaneously over time. This paper cites several other papers which use embeddings for comparison, some of which I'm not personally familiar with. The paper is forthcoming in Poetics. The authors generally publish their code, but this paper's code is not yet shared on either author's GitHub. Mendelsoh et al 2020Their code is not available, but their measures of connotations and valence seem promising.  Load packages and embeddings<jupyter_code># load packages
import pickle

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import altair as alt

from collections import OrderedDict
from gensim.models import KeyedVectors
from gensim.matutils import unitvec
from whatlies import EmbeddingSet, Embedding
from whatlies.transformers import Normalizer, Tsne
from sklearn.manifold import TSNE
# set up figures so the resolution is sharper
# borrowed from https://blakeaw.github.io/2020-05-25-improve-matplotlib-notebook-inline-res/
sns.set(rc={"figure.dpi":100, 'savefig.dpi':300})
sns.set_context('notebook')
sns.set_style("darkgrid")
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('retina')

# borrowed from https://github.com/altair-viz/altair/issues/1021
def my_theme(*args, **kwargs):
    return {'width': 500, 'height': 400}
alt.themes.register('my-chart', my_theme)
alt.themes.enable('my-chart')
# load histwords as gensim KeyedVectors
def load_historical_embeddings(year, 
                               path="historical-embeddings/eng-all_sgns/"):
    with open(path + str(year) + "-vocab.pkl", "rb") as f:
        vocab = pickle.load(f)
    
    vectors = np.load(path + str(year) + "-w.npy", mmap_mode="c")
    
    kv = KeyedVectors(vector_size=vectors.shape[1])
    kv.add(entities=vocab, weights=vectors)

    return kv
# alt: load histwords as whatlies EmbeddingSet
def load_historical_embeddingset(year, 
                                 path="historical-embeddings/eng-all_sgns/"):
    with open(path + str(year) + "-vocab.pkl", "rb") as f:
        vocab = pickle.load(f)
    
    vectors = np.load(path + str(year) + "-w.npy", mmap_mode="c")

    return EmbeddingSet.from_names_X(vocab, vectors)<jupyter_output><empty_output><jupyter_text>The gensim syntax for calculating similarity is a bit less awkward than whatlies, and produces the same results, so I'll use gensim here.For reference, cosine similarity is the dot product of the normed vectors. Links to code:  - https://github.com/williamleif/histwords/blob/master/representations/embedding.pyL76- https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/keyedvectors.pyL1143<jupyter_code>emb_hist = OrderedDict()
for year in range(1900, 2000, 10):
    emb_hist[year] = load_historical_embeddings(year)
emb_fic = OrderedDict()
fic_path = 'historical-embeddings/eng-fiction-all_sgns/'
for year in range(1900, 2000, 10):
    emb_fic[year] = load_historical_embeddings(year, path=fic_path)<jupyter_output><empty_output><jupyter_text>Words most similar to community For the all-English embeddings in the 1990s, these are the words most similar to "community". (In gensim, this excludes the word "community" itself, unlike when using whatlies.)The ECSC is the European Coal and Steel Community, a predecessor to the EU.<jupyter_code>emb_hist[1990].most_similar("community", topn=10)
hist_com10 = []
for year in emb_hist:
    w, s = zip(*emb_hist[year].most_similar("community", topn=20))
    hist_com10.append(pd.Series(data=w, name=year))
pd.DataFrame(hist_com10)
fic_com10 = []
for year in emb_fic:
    w, s = zip(*emb_fic[year].most_similar("community", topn=20))
    fic_com10.append(pd.Series(data=w, name=year))
pd.DataFrame(fic_com10)<jupyter_output>/Users/cgilroy/miniconda3/envs/text-env/lib/python3.8/site-packages/gensim/models/keyedvectors.py:2389: RuntimeWarning: invalid value encountered in true_divide
  return (m / dist).astype(REAL)<jupyter_text>For the embeddings based on English fiction, I find it intriguing that LGBT-related words begin to appear on the list in the 1990s. I've expanded the list of nearest neighbors to the top 20 so this is more apparent -- not only "homophobia" and "transgender", but also "lesbian" and "gay".Note that, aside from "gay" (which exists with a non-LGBT meaning) most of these words aren't common enough to appear in the vocabulary at all before 1970 or so. HistWords handles this by setting the vectors for those words to all 0s.<jupyter_code>emb_hist[1960].get_vector("lesbian")[0:10]
emb_fic[1960]["lesbian"][0:10]<jupyter_output><empty_output><jupyter_text>Pairwise cosine similarity For instance, largely as a consequence of the shift in the meaning of the word "gay", the cosine similarity between "community" and "gay" increases over the course of the 20th century. The words become more similar.<jupyter_code>emb_hist[1900].similarity("community", "gay")
emb_hist[1990].similarity("community", "gay")
def get_similarity_over_time(emb_hist, w1, w2):
    df = pd.DataFrame(emb_hist.keys(), columns=["year"])
    
    df["similarity"] = [emb_hist[year].similarity(w1, w2) for year in emb_hist]
    
    df["word1"] = w1
    df["word2"] = w2
    
    return df
df_gay = get_similarity_over_time(emb_hist, "community", "gay")
df_gay
g = sns.relplot(data=df_gay, x='year', y='similarity', 
                kind="line")
g.ax.set(ylim=(-.2, .55))
g.ax.set_title('Pairwise cosine similarity, "community" & "gay"')
g.fig.set_figwidth(8)
plt.show()
df_neighborhood = get_similarity_over_time(emb_hist, "community", "neighborhood")
g = sns.relplot(data=df_neighborhood, x='year', y='similarity', 
                kind="line")
g.ax.set(ylim=(-.2, .55))
g.ax.set_title('Pairwise cosine similarity, "community" & "neighborhood"')
g.fig.set_figwidth(8)
plt.show()
df_nation = get_similarity_over_time(emb_hist, "community", "nation")
g = sns.relplot(data=df_nation, x='year', y='similarity', 
                kind="line")
g.ax.set_title('Pairwise cosine similarity, "community" & "nation"')
g.ax.set(ylim=(-.2, .55))
g.fig.set_figwidth(8)
plt.show()
df_society = get_similarity_over_time(emb_hist, "community", "society")
g = sns.relplot(data=df_society, x='year', y='similarity', 
                kind="line")
g.ax.set(ylim=(-.2, .55))
g.ax.set_title('Pairwise cosine similarity, "community" & "society"')
g.fig.set_figwidth(8)
plt.show()<jupyter_output><empty_output><jupyter_text>These three plots compare community with related words, and show little change over time. Broadly, it's possible that "community" has become less similar to "nation" and more similar to "neighborhood". There's no trend in the similarity between "community"s and "society".Spearman correlation (Hamilton et al 2016) or change point detection (Kulkarni et al 2015) could be used to assess the statistical significance of change. These similarity calculations can be repeated on different sets of historical embeddings:<jupyter_code>df_gay_fic = get_similarity_over_time(emb_fic, "community", "gay")
df_gay_fic<jupyter_output><empty_output><jupyter_text>Self-similarity<jupyter_code># cosine distance from t-0
# like Kulkarni et al
for year in emb_hist:
    # unitvec is in gensim.matutils
    cos_sim = np.dot(unitvec(emb_hist[1900]["community"]), 
                     unitvec(emb_hist[year]["community"]))
    cos_dist = 1 - cos_sim
    print(str(year) + ": " + str(cos_dist))
df_com0 = pd.DataFrame(emb_hist.keys(), columns = ["year"])
cos_dist = []
for year in emb_hist:
    # unitvec is in gensim.matutils
    cos_sim = np.dot(unitvec(emb_hist[1900]["community"]), 
                     unitvec(emb_hist[year]["community"]))
    cos_dist.append(1 - cos_sim)
df_com0["distance"] = cos_dist

g = sns.relplot(data=df_com0, x='year', y='distance', 
                kind="line")
g.ax.set_title('Cosine distance from t0, "community"')
g.fig.set_figwidth(8)
plt.show()
df_gay0 = pd.DataFrame(emb_hist.keys(), columns = ["year"])
cos_dist = []
for year in emb_hist:
    # unitvec is in gensim.matutils
    cos_sim = np.dot(unitvec(emb_hist[1900]["gay"]), 
                     unitvec(emb_hist[year]["gay"]))
    cos_dist.append(1 - cos_sim)
df_gay0["distance"] = cos_dist
df_gay0
g = sns.relplot(data=df_gay0, x='year', y='distance', 
                kind="line")
g.ax.set_title('Cosine distance from t0, "gay"')
g.fig.set_figwidth(8)
plt.show()
df_tape0 = pd.DataFrame(emb_hist.keys(), columns = ["year"])
cos_dist = []
for year in emb_hist:
    # unitvec is in gensim.matutils
    cos_sim = np.dot(unitvec(emb_hist[1900]["tape"]), 
                     unitvec(emb_hist[year]["tape"]))
    cos_dist.append(1 - cos_sim)
df_tape0["distance"] = cos_dist
g = sns.relplot(data=df_tape0, x='year', y='distance', 
                kind="line")
g.ax.set_title('Cosine distance from t0, "tape"')
g.fig.set_figwidth(8)
plt.show()<jupyter_output><empty_output><jupyter_text>In their Figure 5, Kulkarni et al use "tape" as another example of semantic change. They don't include t0, and they use 5-year increments instead of 10-year. Still, this doesn't seem to resemble their figure very closely, especially in terms of the large initial gap between the 1900s and 1910s.  Word migration with t-SNEAs outlined above, I ran the histwords viz code in a separate environment and produced two-dimensional figures illustrating the movement of focal words over time. In addition to showing the magnitude of change, but conveys some information about what that change means in terms of a word's nearest neighbors.The word "community" changes relatively little:  ![](img/community_annotated.png)By comparison, the word "gay" migrates dramatically:![](img/gay_annotated.png) TODO: Finish writing a simple approximation of those visualizations, using either gensim or whatlies. The beginning of this code, for the 1990s only, is below.<jupyter_code>def fit_tsne(values):
    # from histwords/viz/common.py
    mat = np.array(values)
    model = TSNE(n_components=2, random_state=0, learning_rate=150, init='pca')
    fitted = model.fit_transform(mat)
    return fitted
word_list = ["community"]
w, _ = zip(*emb_hist[1990].most_similar("community", topn=99))
word_list += list(w)
tsne_1990 = fit_tsne(emb_hist[1990][word_list])
df_1990 = pd.DataFrame({'word': word_list, 
                        'x': tsne_1990[:, 0], 
                        'y': tsne_1990[:, 1]
                       })
sns.set_style("white")
g = sns.relplot(x='x', y='y', data=df_1990, alpha=0)
g.set_axis_labels('', '')
g.despine(left=True, bottom=True)
g.set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])
g.fig.set_figwidth(8)
g.fig.set_figheight(8)
for _, row in df_1990.iterrows():
    if row.word != 'community':
        g.ax.text(x=row.x, y=row.y, s=row.word, 
                  horizontalalignment='center',
                  verticalalignment='center',
                  alpha=.5)
    else:
        g.ax.text(x=row.x, y=row.y, s=row.word, 
                  c='blue',
                  horizontalalignment='center',
                  verticalalignment='center',
                  alpha=1, 
                  size="xx-large"
                 )
plt.show(g)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09036828727489049, 'fraction_numerical': 0.019955654101995565, 'mean_word_length': 4.576598311218335, 'pattern_counts': {'":': 2, '<': 32, '<?xml version=': 0, '>': 34, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '363618', 'license': 'MIT', 'n_tokens_mistral': 5897, 'n_tokens_neox': 5285, 'n_words': 2046, 'path': 'historical-change.ipynb', 'repo_name': 'ccgilroy/community-discourse', 'size': '18491'}