<jupyter_start><jupyter_text>Training the model<jupyter_code>%%time

#I train a model on the average + std (53+30=83) deviation bilingual context of each word.
# 300 is the standard w2v size
madmod0 = Word2Vec(aligned,window=83,size=300)
# This should take around 1 or 2 minutes
madmod0.wv.most_similar("germany") # if it worked, we should see a bilingual country cluster
# with Alemania in pole position
#So, summing up. 
#1. I have to shuffle the data to obtain a working translingual. If I do not shuffle, the spaces keep apart.
# This can be probably improved in a new version.
#2. If I simply zip the data ("the","la","house","casa"), results are better but, it
# seems to me, not as good as when i shuffle.
#3. Function words and prepositions do not return meaningul clusters
#4. Regarding content words, I get frequent translations and semantically related elements
# in both languages.  
madmod0.wv.most_similar("war")
#5. If a word is (almost) always translated with another word, they should be very close
# in the space. 
madmod0.wv.most_similar("voz")
#5. If a word belongs to a semantically related cluster where most words have a 
# *consistent translation* in the corpus, such word has many bilingual, near neighbours.
madmod0.wv.most_similar("france")
#6. If instead a word does not belong to a group of closely related, consistently 
# translated concepts, but does have a consistent translation in the corpus,
# it tends to have a very close neighbourg followed by a "void" (the second nearest is not that near) 
madmod0.wv.most_similar("solidarity")
#7. If a word is *not* consistently translated, it should display: 
#1 - various translations, but none very close:
madmod0.wv.most_similar("fear")
#... or: 2 - no good translation at all, if the translation cannot be closed in one 
# single word for example. The word also appears isolated (closest neighbours are far away)
madmod0.wv.most_similar("somehow")
# this could be a special way of seeing (1) imbalances in translation
madmod0.wv.most_similar("gentes") 
#gentes does not seem to have a systematic translation in this specific corpus
# (2) strong divergence in polysemy or in function 
madmod0.wv.most_similar("but"),madmod0.wv.most_similar("the") 
#(3) contextual similarities beyond a word's translation...
madmod0.wv.most_similar("palestinian") 
#e.g. palestinian is close to israeli too
#and (4) DISsimilarities between sources and translations. 
madmod0.wv.most_similar("population")#
#words that do not show consistent translations in the space *tend* to cluster monolingually
# and to have more entropic clusters
madmod0.wv.most_similar("quiero"),madmod0.wv.most_similar("struggle")
#words displaying a consistent translation in the space have less entropic distributions,
# depending on how systematic the translation might be (see the City-London translation)
print(madmod0.wv.most_similar("interesting"),"\n")
print(madmod0.wv.most_similar("city"),"\n")
print(madmod0.wv.most_similar("sucesor"),"\n")
print(madmod0.wv.most_similar("bosques"),"\n")

#The original being english, we can look into the "english untranslatables".
# From a random site, a small list of palabras ingles sin traducion. 
sin_traducion = """struggle pump lock realise tip type stand spam insight weekend""".split()
for w in sin_traducion:
    print(w)
    print(madmod0.wv.most_similar(w,topn=3))
    print("\n")
# none of them has a close neighbour in the other language, which is the behaviour we expect
# for elements without a systematic translation
#We can even perform Mikolov's man:woman = king:queen kind of operations! 

#woman:man = x:hombre
#man:woman = x:mujer
#sad:happy = x:feliz

#woman-man+hombre, man-woman+mujer, and so forth
print(madmod0.wv.most_similar(positive=["woman","hombre"],negative=["man"]))
# woman is to man as hombre is to ?
print(madmod0.wv.most_similar(positive=["man","mujer"],negative=["woman"]))
madmod0.wv.most_similar(positive=["sad","feliz"],negative=["happy"])
# sad is to happy as feliz is to ?
#Finally, we can somtimes sum words and find the translation of their sum!
# For example, the closest word to we+can is podemos, the closest to i+hope is espero etc.

madmod0.predict_output_word(["we","can"],topn=3),madmod0.predict_output_word(["we","know"],topn=3),madmod0.predict_output_word(["i","hope"],topn=3)
#This is more or less it. 
# Now I added a long Appendix about variations on this idea
#APPENDIX
### Anoher possibility is to mark every word's context with the id of the sentence it is occurring in.
alignedO = []
for i in range(len(palabras)):
    fra = [p for p in palabras[i]]
    sen = [w for w in words[i]]#
    
    multi = fra+sen
    fatica =  []#
    for w in multi:
        fatica.append(w)
        fatica.append(str(i))
    #random.shuffle(fatica)
    alignedO.append(fatica)

# so a sentence looks like this:
alignedO[20]
#I train the model on the smallest context: words occurring in the same set of sentences should thus have the EXACT kind of context (the same set of ids)
madmod1= Word2Vec(alignedO[:],window=1,size=300)
#This system works very poorly for any content word
w = "germany"
print(madmod1.wv.most_similar(w, topn=3))
print(madmod0.wv.most_similar(w, topn=3))
# but it is working for function words and prepositions!!
madmod1.wv.most_similar("y"), madmod0.wv.most_similar("y")
madmod1.wv.most_similar("to")
#APPENDIX B: SAME IN GERMAN
deu = open(dire+"Translation/B7/epuds-parallel/words/epuds.en-de.de").read()
eng = open(dire+"Translation/B7/epuds-parallel/words/epuds.en-de.en").read()
wrt =  [nltk.wordpunct_tokenize(se) for se in deu.lower().split("\n")]
words = [nltk.wordpunct_tokenize(sen) for sen in eng.lower().split("\n")]
print(len(wrt), len(words))
wrt[0]
for i in range(5):
    sen = words[i]
    print(" ".join(sen))
    print(" ".join(wrt[i]))
    print("\n")
aligned = []
for i in range(len(wrt)):
    se = wrt[i]
    sen = words[i]
    multi = se+sen
    
    random.shuffle(multi) 
    aligned.append(multi)
    
aligned[0]
np.mean([len(s) for s in aligned]), np.std([len(s) for s in aligned])
#I train a model on the average + std deviation context. So its context is bilingual.
madmod_deu = Word2Vec(aligned,window=84,size=300)
madmod_deu.wv.most_similar("germany", topn=3)
madmod_deu.wv.most_similar("kraft")
madmod_deu.wv.most_similar("happy")
madmod_deu.wv.most_similar("mord")
#no near neighs
madmod_deu.wv.most_similar("wozu")
#no near english neighs
madmod_deu.wv.most_similar("darf")
#no near german neighs
madmod_deu.wv.most_similar("wenn")
madmod_deu.wv.most_similar("fragen", topn=20)
w = "freedom"
madmod_deu.wv.most_similar(w), madmod0.wv.most_similar(w)
madmod_deu.wv.most_similar(["freedom","press"]) #concepts at the intersection. 
madmod_deu.wv.most_similar(["religionsfreiheit"])
madmod_deu.wv.most_similar("erfreut")
madmod_deu.wv.most_similar("unemployment".split())
#INTERPRETING<jupyter_output><empty_output><jupyter_text>Interpreting was a bit complicated to pre-process due to the format of the data. I created a txt file with aligned english-german interpreting transcripts and here I will just use that as my source corpus<jupyter_code>import codecs
f = codecs.open("English-German_aligned_interpreting.txt","r","utf8")#.read()
# here I will use it as it is, ofc you can remove </s> and similar signs before starting
mydata = f.read()
doubles = mydata.split("\n#\n")
len(doubles)
print(doubles[100])
print(doubles[100].split())
import random
tokd_alig = []
for d in doubles:
    magic = nltk.wordpunct_tokenize(d.lower())
    random.shuffle(magic)
    tokd_alig.append(magic)

len(tokd_alig), np.mean([len(e) for e in tokd_alig]), np.std([len(e) for e in tokd_alig])
%%time

interp_model = Word2Vec(tokd_alig,window=77,size=300, min_count=1, negative=10, iter=30)
interp_model.wv.most_similar("spain")
len(tokd_alig)
tokn = 0
for tokd in tokd_alig:
    tokn+=sum([len(w) for w in tokd])
print(tokn)
# Finally we can make a down-sampled model trained on a translation corpus as small as an interpreting corpus
random.shuffle(aligned)
downsampled_model = Word2Vec(aligned[:len(tokd_alig)],window=80,size=300, min_count=1, negative=10, iter=30)
wort,n = "germany",5 #friends,
madmod_deu.wv.most_similar(wort, topn=n), downsampled_model.wv.most_similar(wort,topn=n), interp_model.wv.most_similar(wort,topn=n)
#some basic analysis
len(downsampled_model.wv.vocab), len(interp_model.wv.vocab)
#from here on it is possible to compare the spaces' inner distances, avg closest neighbour and so on 
smaller_voc = [k for k in downsampled_model.wv.vocab if k in interp_model.wv.vocab]
print(len(smaller_voc))
smaller_voc = smaller_voc[:]
wow = [downsampled_model.wv.distances(w) for w in smaller_voc]
wow2 = [interp_model.wv.distances(w) for w in smaller_voc]
#hm2 = erste_model.wv.distances(wort)
wow3 = [madmod_deu.wv.distances(w) for w in smaller_voc]
1-np.mean(wow), np.std(wow), 1-np.mean(wow2), np.std(wow2), 1-np.mean(wow3)
1-np.mean(wow), np.std(wow), 1-np.mean(wow2), np.std(wow2)#, 1-np.mean(wow3)
#nearest neighbour avg distance, second nearest neighbour average distance.
nearests_downsampled = [downsampled_model.wv.most_similar(w, topn=10) for w in smaller_voc[:]]
nearests_erste = [erste_model.wv.most_similar(w, topn=10) for w in smaller_voc[:]]
#nearests
first_down = [e[0][1] for e in nearests_downsampled[:]]
first_ers = [e[0][1] for e in nearests_erste[:]]
np.mean(first_down), np.std(first_down), np.mean(first_ers), np.std(first_ers)
second_down = [e[-1][1] for e in nearests_downsampled]
second_ers = [e[-1][1] for e in nearests_erste]
np.mean(second_down), np.mean(second_ers)
second_down = [sum([w[1] for w in e]) for e in nearests_downsampled]
second_ers = [sum([w[1] for w in e]) for e in nearests_erste]
np.mean(second_down), np.mean(second_ers)
wort = "<PASSWORD>" #friends,
downsampled_model.wv.most_similar(wort), erste_model.wv.most_similar(wort)
wort = "<PASSWORD>ell" #friends,doctors,doctor,politician,gefahr,gespielt,understand, vs should,could,must,will,very,gratulieren,verträge
downsampled_model.wv.most_similar(wort), interp_model.wv.most_similar(wort)
madmod_deu.wv.most_similar("sitte")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12424750814171519, 'fraction_numerical': 0.01500049343728412, 'mean_word_length': 5.580519480519481, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '405539', 'license': 'MIT', 'n_tokens_mistral': 3767, 'n_tokens_neox': 3590, 'n_words': 1046, 'path': 'Multilingual_Translation_Embeddings.ipynb', 'repo_name': 'yuri-bizzoni/Multilingual-Translationese-Embeddings', 'size': '10124'}