<jupyter_start><jupyter_text>Microbiome Embedding with word2vecIn order to deal with the sparseness of the microbiome data we can attempt to use word2vec to create a dense embedding that represents each OTU. By representing each OTU as a dense vector it will be easier to use for predictive modeling later on. Data is being read from a local pkl file in this demonstration. Proper implementation will pull the data out of S3.<jupyter_code>import pandas as pd
import gensim
from random import shuffle


# pull table from local file
samples = pd.read_pickle(r'C:\Users\bwesterber\Downloads\biom_table.pkl')

# replace nans with zeros
samples = samples.fillna(0)<jupyter_output><empty_output><jupyter_text>Since we are framing this as an NLP problem lets think of each microbiome sample as a sentence and each OTU as a word. Here we construct the sentences that will be put into the word2vec model.<jupyter_code>sentences = []
for sample in range(len(samples) - 1):
    try:
        sentence = list(samples[sample][samples[sample] > 0].index)
        sentences.append([str(x) for x in sentence])
    except KeyError:
        pass<jupyter_output><empty_output><jupyter_text>Here we can augment the dataset use for generating the embeddings by shuffling each sample. In CBOW mode word order shouldnt matter for the embedding, but having a larger training set might improve how word2vec learns the conditional probability of each word. The usefullness of this is still unclear and may not be required.<jupyter_code># shuffle each sample around and append it to the training data
augmentation_constant = 2
generated_sentences = []
for sentence in sentences:
    for augmentation in range(augmentation_constant):
        shuffle(sentence)
        generated_sentences.append(sentence)<jupyter_output><empty_output><jupyter_text>Prepared data is now fed into the gensim word2vec model for training in CBOW mode.<jupyter_code>model = gensim.models.Word2Vec(generated_sentences, size = 100, min_count = 2, window = 100, workers = 4, sg = 0)
model.train(generated_sentences, total_examples = len(sentences), epochs = 5)<jupyter_output><empty_output><jupyter_text>We can now evaluate the model to see which OTUs are most similar to each other in the embedding space. The most_similar method returns the cosine similairty between the input OTU and its nearest neighbors.<jupyter_code>model.wv.most_similar('84239', topn = 10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.060655737704918035, 'fraction_numerical': 0.011475409836065573, 'mean_word_length': 5.242966751918159, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '229902', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 732, 'n_tokens_neox': 674, 'n_words': 287, 'path': 'old/AGP/microbiome_embedding_demo.ipynb', 'repo_name': 'mas-dse-ringhilt/DSE-American-Gut-Project', 'size': '2440'}