<jupyter_start><jupyter_text>Preliminaries<jupyter_code># This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import os
# install sent2vec
!pip install git+https://github.com/epfml/sent2vec
# install annoy
!pip install annoy<jupyter_output>Collecting git+https://github.com/epfml/sent2vec
  Cloning https://github.com/epfml/sent2vec to /tmp/pip-req-build-_75a1j6t
  Running command git clone -q https://github.com/epfml/sent2vec /tmp/pip-req-build-_75a1j6t
Building wheels for collected packages: sent2vec
  Building wheel for sent2vec (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / done
[?25h  Created wheel for sent2vec: filename=sent2vec-0.0.0-cp36-cp36m-linux_x86_64.whl size=1137474 sha256=12703f152ae23c528b3cee90794587b2ca576b30b34b54dd0638a5b3dcb3864f
  Stored in directory: /tmp/pip-ephem-wheel-cache-gbc30ioy/wheels/7b/db/9d/db816db406f182ce88bf1b90e6d41313bf16c7ed72a9626ac7
Successfully built sent2vec
Installing collected packages: sent2vec
Successfully installed sent2vec-0.0.0
Requirement already satisfied: annoy in /opt/conda/lib/python3.6/site-packages (1.16.3)<jupyter_text>Write requirements to file, anytime you run it, in case you have to go back and recover dependencies.Latest known such requirements are hosted for each notebook in the companion github repo, and can be pulled down and installed here if needed. Companion github repo is located at https://github.com/azunre/transfer-learning-for-nlp<jupyter_code>!pip freeze > kaggle_image_requirements.txt<jupyter_output><empty_output><jupyter_text>Check what is in the input folder<jupyter_code># Input data files are available in the "../input/" directory.
# Any results you write to the current directory are saved as output.

!ls ../input<jupyter_output>jw300entw  sent2vec<jupyter_text>Load The Raw Data<jupyter_code>import re
import time

start = time.time()
english_sentences = []
with open("../input/jw300entw/jw300.en-tw.en") as f:
    for line in f:
        english_sentences.append(re.sub(r'[\W\d]', " ",line.lower())) # clean and normalize
end = time.time()
print("Loading the english sentences took %d seconds"%(end-start))
print("A sample of the english sentences is:")
print(english_sentences[:10])
print("The length of the list is:")
print(len(english_sentences))
twi_sentences = []
with open("../input/jw300entw/jw300.en-tw.tw") as f:
    for line in f:
        twi_sentences.append(re.sub(r'[\W\d]', " ", line.lower())) # clean and normalize
print("A sample of the twi sentences is:")
print(twi_sentences[:10])
print("The length of the list is:")
print(len(twi_sentences))<jupyter_output>A sample of the twi sentences is:
['  oo   yehowa   boa me babea kumaa yi ma onni nokware     ', 'wɔwoo me too abusua a wonim adwinne di mu wɔ alsace   france   wɔ      mu   ', 'ná papa taa pa twere n agua mu kenkan asase ho nsɛm anaa ewim nneɛma ho nhoma bi anwummere anwummere   ', 'ná me kraman da papa nan so   na na ɔka nsɛntitiriw a epue wɔ n akenkan no mu kyerɛ maame bere a ɔnwene abusua no nneɛma no   ', 'ná m ani gye anwummere a ɛtete saa no ho kɛse   ', 'ná yɛpɛ nyamesom kɛse   ', 'ná yɛyɛ katolekfo amapa   na na nkurɔfo a wohu yɛn sɛ yɛrekɔ asɔre kwasida anɔpa no ka sɛ     abɔ nɔnkron   ', 'arnold abusua no rekɔ asɔre     ', 'ná mekɔ asɔre anɔpa biara ansa na makɔ sukuu   ', 'nanso esiane ɔsɔfo no suban bɔne nti   maame amma me nkutoo ankɔ asɔre bio   ']
The length of the list is:
606197<jupyter_text>Vectorize Subset of English Sentences with sent2vec<jupyter_code>MAXNSENT = 10000 # how many sentences to take from top of data for now (small experiment)
import time
import sent2vec

model = sent2vec.Sent2vecModel()
start=time.time()
model.load_model('../input/sent2vec/wiki_unigrams.bin')
end = time.time()
print("Loading the sent2vec embedding took %d seconds"%(end-start))
def assemble_embedding_vectors(data):
    out = None
    for item in data:
        vec = model.embed_sentence(item)
        if vec is not None:
            if out is not None:
                out = np.concatenate((out,vec),axis=0)
            else:
                out = vec                                            
        else:
            pass
        
    return out
start=time.time()
EmbeddingVectors = assemble_embedding_vectors(english_sentences[:MAXNSENT])
end = time.time()
print("Computing all embeddings took %d seconds"%(end-start))
print(EmbeddingVectors)
print("The shape of embedding matrix:")
print(EmbeddingVectors.shape)
# Save embeddings for later use
np.save("english_sent2vec_vectors_jw.npy",EmbeddingVectors)<jupyter_output><empty_output><jupyter_text>Build and Test Index w/ Annoy for fast Neareast-Neighbor RetrievalFirst build the annoy index for the available English sent2vec vectors<jupyter_code>from annoy import AnnoyIndex

start = time.time()
dimension = EmbeddingVectors.shape[1] # Length of item vector that will be indexed
english_NN_index = AnnoyIndex(dimension, 'angular')  
for i in range(EmbeddingVectors.shape[0]): # go through every embedding vector
    english_NN_index.add_item(i, EmbeddingVectors[i]) # add to index

english_NN_index.build(10) # 10 trees
english_NN_index.save('en_sent2vec_NN_index.ann') # save index
end = time.time()
print("Building the NN index took %d seconds"%(end-start))<jupyter_output>Building the NN index took 1 seconds<jupyter_text>Test the built index<jupyter_code>test_english_NN_index = AnnoyIndex(dimension, 'angular')
test_english_NN_index.load('en_sent2vec_NN_index.ann') # super fast, will just mmap the file
translation_idx = 15 # choose index of sentence to focus on in english_sentences/twi_sentences

annoy_out = test_english_NN_index.get_nns_by_item(translation_idx, 5) # will 5 nearest neighbors to the very first sentence
print(annoy_out)
print("- The sentence we are finding nearest neighbors of:\n")
print(english_sentences[annoy_out[0]])
print("\n\n- The 4 nearest neighbors found:\n")
for i in range(1,5):
    print(str(i) + ". "+ english_sentences[annoy_out[i]])
print("- In other words, if we were translating the english sentence:\n")
print(english_sentences[annoy_out[0]])
print("  where the known correct translation is:")
print(twi_sentences[annoy_out[0]])
print("\n\n- The 4 top translation suggested by our sparse retrieval system above are:\n")
for i in range(1,5):
    print(str(i) + ". "+ twi_sentences[annoy_out[i]])<jupyter_output>- In other words, if we were translating the english sentence:

but mother was so enthusiastic about the truth that she decided to do some bible reading with me   
  where the known correct translation is:
nanso na maame ani gye nokware no ho araa ma ɔne me boom kenkan bible no   


- The 4 top translation suggested by our sparse retrieval system above are:

1. bere a wɔne maame bɔɔ nkɔmmɔ pii akyi no   ɔno nso begye dii sɛ nea na wasua afi bible mu no ne nokware no   
2. ɔkaa maria stossier   yɛn fipamfo hans nuabea kumaa a na wagyina bible mu nokware akyi   ho asɛm kyerɛɛ me   
3. nanso   yɛte nka sɛ yɛn a yɛaka a yenyaa hokwan hyiaa wɔn na yehuu wɔn dɔ ne wɔn mmɔdenbɔ wɔ bible mu nokware ho no na yɛanya nkuranhyɛ kɛse     
4. me nana barima tiee saa ɔkasa no bi   na ɔno nso begye dii sɛ nokware no na wate no
{'fraction_non_alphanumeric': 0.09166332128997726, 'fraction_numerical': 0.026495383380168606, 'mean_word_length': 4.315789473684211, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '621353', 'license': 'MIT', 'n_tokens_mistral': 2797, 'n_tokens_neox': 2709, 'n_words': 831, 'path': 'notebooks/ghananlp-kasa-retrieval-v0-1.ipynb', 'repo_name': 'hkboateng/kasa', 'size': '7473'}