Assessment of Mutation Susceptibility in DNA Sequences with Word Vectors

With the advent of natural language processing (NLP) techniques empowered with deep learning approaches, more detailed relationships between words have been unraveled. Word2Vec is quite robust in discovering contextual and semantic relationships. Genome being a long text, is subject to similar studies to unravel yet to be discovered relationships between DNA k-mers. Dna2vec applies Word2Vec approach to whole genome so that DNA k-mers are represented as vectors. The cosine similarity queries on DNA vectors reveal unusual relationships between DNA k-mers. In this study, we examined DNA sequence based prediction of mutation susceptibility. Initially,we generated word vectors for human and mouse genome via dna2vec. On the other hand, we retrieved coordinates of common and all mutations from dbSNP. For each coordinate, we extracted 8 nucleotide k-mers intersecting mutations and results are aggregated. such a way that number of mutations for each 8-mer has been tabulated. These results are incorporated with dna2vec cosine similarity data. Our results showed that for a given k-mer, k-mers with highest cosine similarity coincide with highest mutation count k-mer. In other words, the neighbor with the highest cosine similarity for a k-mer was also seen to be the neighbor overlapping the mutation count. As a result of our studies, human and mouse, dna2vec vs. mutation overlap is 80% and 70%, respectively. In conclusion, dna2vec and other word embedding approaches can be used to reveal mutation or variation characteristics of genomes without sequencing or experimental data, solely using the genome sequence itself. This might pave the way for understanding the underlying mechanism or dynamics of mutations in genomes.


Introduction
Underlying problem of language modeling is that dimensionality. Most known and used technique to solve this problem is word2vec, word embedding method ).
This method includes two models; Continuous Bag of Words (CBOW) and Skip Gram. These models converts word in the corpus into vectors. Purpose of these models is to determine contextually similar words. In line with this objective, they utilize "context window". All words within n units from a target word in context window belong to context of that target word. This architecture based on context window can predict either target or the context. A shallow neural network is used for training of words in these architectures. Output of this neural network gives the semantic or syntactic relation among words. Word vectors can be added or subtracted in order to reveal or extend relationships between words, as shown below: vec(king) − vec(man) + vec(woman) ≈ vec(queen)  Moreover, word2vec can capture word relations such as past-tense, ownership, language spoken in the country or the capital of country information (Levy and Goldberg, 2014). For many machine learning algorithms, an input of fixed-length vector is required. Word2vec generates fixed-length embedding for words which can be later used as input to subsequent machine learning applications. Thus, word2vec has been used in studies in different areas such as document representations for classification of documents and sentiment analysis (Le and Mikolov, 2014;Kusner et al. 2015;Chen, 2017), network detection to predict the link between the edges and nodes (Perozzi et al. 2014;Chen and Lawrence Zitnick, 2015), image caption generation (Pedersoli et al. 2017;Kiros et al. 2014), identification of public user tendency such as political preference or occupational class (Preoţiuc-Pietro et al. 2015;Yang et al. 2018;Preoţiuc-Pietro et al. 2017), limited contextual information transformation (Dos Santos and Gatti, 2014;De Boom et al. 2015), image annotation to provide computer vision and pattern recognition (Uricchio et al. 2017), using biomedical literature to predict drug interactions and capturing medical semantic similarities (Wang et al. 2018;Zhao et al. 2016), emotion detection (Abdul-Mageed and Ungar, 2017;Eisner et al. 2016), improving word similarity detection (Schwartz et al. 2015;Faruqui et al. 2016) and polysemy detection (Arora et al. 2018;Jauhar et al. 2015) and removing gender bias in embeddings (Bolukbasi et al. 2016). The applications of word2vec can extend to domains far from computer science such as urban planning (Yao et al. 2017). Word2vec has been calculated for Turkic languages (Akın and Akın, 2007) as well.
The human genome consisting of 3 billion letters can be considered as a long string and thus subject to word embedding studies. This long string can be divided into constant or variable-length k-mers to produce words. In order to reveal semantic and syntactic relationships between the k-mers in human genome, the dna2vec study was carried out by using word2vec method (Ng, 2017).
Dna2Vec method, using variable-length k-mers training in two-layer neural network, demonstrated correlation between Needleman-Wunsch alignment score and dna2vec cosine similarity of k-mers. Needleman-Wunsch algorithm (Needleman et al. 1970) measures sequence similarity between to k-mers by global alignment approach. In addition, summing dna2vec embedding vectors for two k-mers is equivalent to concatenated k-mer. For example; In this study, we investigated relation between mutation dynamic and vector embeddings. Mutation is change of a letter in the genome, usually causing adverse effects in gene products resulting in diseases. There are no algorithms for predicting mutations and only technique to detect mutations is experimentally sequencing the DNA. In order find a technique to predict mutations, millions of known mutations and their affected words were compared with dna2vec vector embeddings. Our results show there is correlation with vector embedding and mutation count per k-mer suggesting vector embeddings have potential to predict mutations in genome.

Data Retrieval and dna2vec Script
Human genome hg38 and mouse genome mm10 downloaded from NCBI, common SNPs for human and mouse are downloaded from dbSNP (dbSNP link). Human dna2vec model was retrieved from dna2vec repository (GitHub link). Python scripts from dna2vec study are used to generate human and mouse dna2vec models (Ng, 2017). Human genome results were compared with results from mouse genome in order to show that our findings hold true for any organism. Also, mouse genome has comprehensive mutation data available.

Counting mutation per k-mer
We collected sliding window 8-mer around each mutation for nearly 35 million mutations in human and 73 million mutations in mouse. Then we counted occurrence of all k-mers and mutation pairs. Each pair contain coordinate and type of mutation in the k-mer.

Integration mutation and dna2vec data
For all 65,536 8-mers we generated 24 neighbors with hamming one distance for each coordinate of the kmer. For a particular k-mer and its neighbor, we extracted mutation count and dna2vec cosine similarity.

Calculating correlation between mutations and cosine similarity
Using the data generated in previous step for each mutation in every coordinate for all the words we compared number of mutations and cosine similarity and prepared the confusion matrix.

dna2vec Properties
As previously shown addition and subtraction vector embeddings correspond to concatenation and subtraction events in k-mers. Concatenation of a k-mer to another k-mer from right or from left generates two strings with comparable embeddings. (Figure 1A). Similarly, subtracting and then concatenating is applicable to embeddings ( Figure 1B).

Mutation counts
We applied sliding window technique to the genome and we counted each mutation in every coordinate for all 8mers. Figure 2 illustrates technique and genome-wide results for k-mers of interest. Mutation counts for both human and mouse genomes were calculated.

Mutation counts vs. vector similarity
In order to quantify relationship between vector embeddings and mutation counts we used ranking of mutation counts and cosine similarity per coordinate. So, for a k-mer, all possible mutations at a particular coordinate were ranked according to mutation counts (rank 1 is the highest count). Similarly, for a particular k-mer cosine similarity of all possible words generated by changing one letter (i.e. mutation) has been calculated and ranked (highest similarity is ranked 1). Only after, we calculated correlation between mutation occurrence and cosine similarity. We found that 80% of human mutations and 70% of mouse mutations have overlapping top rank with cosine similarity. Table 1 lists k-mers carrying mutations with highest mutation count and highest cosine similarity. Table 1: Mutation counts and cosine similarity values. For the k-mer TGAGCACT, each possible neighbor with hamming distance one has been listed and for each neighbor, mutation count and cosine similarity has been calculated. The numbers in parenthesis indicate ranking of mutation within given coordinate. Figure 3 illustrates overlap of mutation count and cosine similarity rankings per coordinate for a particular 8-mer. For the first coordinate of the k-mer TGAGCACT, T→ C mutation has the highest ranking compared to T→ G and T→ A mutations. On the other hand, word generated by T→A change has the highest cosine similarity ranking when compared to words generated by T→ G and T→ C changes. When all coordinates are examined, at 6 out of 8 coordinates, top ranks for mutation and cosine similarity overlaps. Figure 3: Relation between the mutation count and cosine similarity. Letters above and below indicate ranking of mutation counts and cosine similarity, respectively, in decreasing order.

Confusion Matrix
We prepared the confusion matrix for both human and mouse ( Table 2, Table 3) in order to assess predictive power of cosine similarity to predict experimental mutations. Ranking order per coordinate has been considered as features to be compared. The confusion matrix show that accuracy values are 0.7974 and 0.8322 for mouse and human, respectively.

Conclusions
dna2vec has been previously shown to exhibit concatenation and similarity properties for k-mer vectors. Our study showed that dna2vec can also reveal mutation susceptibility of k-mers. Our findings suggest that by using dna2vec, mutation susceptibility of k-mers can be predicted by processing the genome sequence alone even if no experimental result is available. This indicates that mutation susceptibility information is inherent and embedded within genome sequence and dna2vec can unlock it. With our approach, genomes of numerous organisms can be analyzed for mutation susceptibility, paving the way to comparison mutation dynamics or mechanisms among different organism without experimental mutation data. Moreover, dna2vec has potential to unveil the traces of yet undiscovered biological mechanisms effecting kmers in genome. Finally, dna2vec opens up the genome to various applications of artificial intelligence by converting DNA sequences into fixed length, transformed embeddings.