Goal: Various methods have been developed to analyze the association between organisms and their genomic sequences. Among them, sequence alignment is the most frequently used method for comparative analysis of biological genomes. We intend to propose a novel pairwise sequence alignment method using deep reinforcement learning to break out the old pairwise alignment algorithms. Methods: We defined the environment and agent to enable reinforcement learning in the sequence alignment system. This novel method, named DQNalign, can immediately determine the next direction by observing the subsequences within the moving window. Results: DQNalign shows superiority in the dissimilar sequence pairs that have low identity values. And theoretically, we confirm that DQNalign has a low dimension for the sequence length in view of the complexity. Conclusions: This research shows the application method of deep reinforcement learning to the sequence alignment system and how deep reinforcement learning can improve the conventional sequence alignment method. I.Recent advancements in sequencing technology have enabled the analysis of organisms with long sequences The sequence alignment method has reduced the complexity through the development of a multiple sequence alignment (MSA) method that aligns multiple sequences simultaneously. This method has been improved through various approaches, such as detecting common subsequences, ordering the progressive alignment, or increasing the speed using multithreads based on GPUs In particular, several pairwise alignment algorithms, such as banded alignment, the BLAST, and the MUMmer have been proposed to improve the speed of pairwise alignment To overcome the problems of conventional alignment methods, we proposed a novel alignment method using deep reinforcement learning agent. Reinforcement learning is a way to teach an agent that could choose the best actions by observing an environment in a given system. The conventional tabular based reinforcement learning has a difficulty in expiring and learning the massive and complex systems. To improve this, the deep reinforcement learning method was proposed, and it overcame the limitations by approximately learning the complex systems The key contribution of this paper is as follows:--Apply the deep reinforcement learning algorithm to the sequence alignment system--Investigate the effect of each parameter on the performance of sequence alignment--Verify alignment performance superiority in short fairly dissimilar sequence pairs--Combine the proposed DQNalign with the conventional sequence alignment algorithm--Prove how the proposed DQNalign can obtain the optimal alignment's performanceII.https://github.com/syjqkrtk/DQNalign. Here, the proposed DQNalign algorithm can include one deep neural network structure among the two types that are used to predict the processing direction of alignment efficiently: Dueling Double Deep Q-network (DDDQN), faster Dueling Double Deep Q-network (faster DDDQN). The detailed network architecture will be explained in this section.Through this paper, we will introduce the pairwise heuristic sequence alignment algorithm based on deep reinforcement learning. We call this proposed method as DQNalign. The entire procedure of DQNalign is described as the flow chart in A.To apply deep reinforcement learning to sequence alignment, we try to develop a novel sequence alignment method using reinforcement learning. Instead of observing the entire sequence at once, we propose a novel heuristic sequence alignment method that repeats the small alignment while moving the window of sub-sequence pairs. Through DQNalign, it is possible to solve memory and time complexity problems.The proposed heuristic sequence alignment method can be expressed as shown in B.In order to apply sequence alignment based on deep reinforcement learning, we have defined and used two network architectures: 1) Dueling Double Deep Q-network (DDDQN) and 2) Faster Dueling Double Deep Q-network: Separable convolutional layer based acceleration (faster DDDQN). We considered two types: DDDQN for high accuracy and faster DDDQN for 2x speed, and we used multiple cases of these networks by changing windows size. Detailed description of each network architecture is given in Supplementary materials S1, and their structures are given in Fig. S1 and Fig. S2.C.Most machine learning requires a massive number of input data. In addition, using only a limited set of data in particular sequences can cause bias in deep reinforcement learning. Therefore, we created new training environments according to the model of evolution.In the training procedure, we generated the sequence pairs according to following rules. First, we generated a completely random sequence. Then, we made the other sequence by mutating the random sequence. For convenience, we used the JC69 model to create the SNP mutations D.E.coli and mammalian cases, and the MUMmer method was used only in the E.coli case. Then, we will briefly explain how these conventional methods are adopted to the proposed DQNalign method.In the DQNalign method, the starting point can critically affect the result because the alignment procedure is performed only in one direction. Therefore, we improved stability by applying the preprocessing methods of conventional alignment methods. We applied the longest common substring method to HEV, 1)We extracted the longest common substring using SequenceMatcher function in difflib of Python. Using this function, the most similar portion in the sequence pair was found. Then, this longest common substring was used as a starting point of the DQNalign method.2)https://sourceforge.net/projects/mummer/. MUMmer can obtain the entire global alignment by itself, but dissimilar parts of the sequence pair could be skipped. To improve the coverage of the MUMmer, we apply our DQNalign method to process these ignored parts. For these ignored parts, the DQNalign algorithm aligns until it meets the aligned elements.For the purpose of using the MUMmer software as the preprocessing of our DQNalign method, we used the official version of the MUMmer software in III.To show the feasibility and performance of the overall proposed DQNalign algorithms, we designed the following three simulations: 1) Numerical analysis on the step error probability, 2) Performance difference among DQNalign methods according to parameter changes, and 3) Performance comparison with the conventional alignment methods. Based on these simulations, we will show how the DQNalign method can improve the heuristic sequence alignment performance and how we can adapt the DQNalign method into the conventional alignment methods.A.Before applying the deep reinforcement learning, it is necessary to prove clearly whether the heuristic sequence alignment method in Here, 1)In order to approximate the step error probability in a given environment, we have defined the following three major assumptions and one constraint. Then, we can infer a rough relationship between the window size and the step error probability. First, we assume that the NW algorithm can precisely match the mutation information of the actual sequences. Second, it is assumed that several alignment paths are well-aligned when they have the same highest score in the current window. Third, the alignment at each step in the local best path selection method can be calculated independently. Conversely, for the constraint, we consider the alignment scoring parameters to prevent the indel preferences. We can derive equations on the error probability in case of similar sequences that have positive alignment scores.Here, we calculate the step error probability of the local best path selection method by considering the SNP and the indel cases seperately. The case of the SNP occurrence is depicted in Here, the We confirm that the error probability, Here, 2)To verify the results of this analysis, we generated in-silico sequences according to the model of evolution. Next, we compared the results of alignment in the total length with that in a fixed window size. Under the second assumption, all of the paths with the same score were treated as correct answers. The detailed simulation scenario is described in Table S2. In Fig. S3, we confirmed that the simulation result exhibits a similar tendency compared to the analysis results based on Eq.B.We designed the following two simulations to analyze the performance according to various parameters in the proposed scheme: 1) performance convergence of the deep reinforcement learning-based alignment in the training procedure and 2) performance analysis according to various parameters. Through these simulations, we showed the processes adapting deep reinforcement learning to alignment and considering the optimal parameters of the DQNalign method.1)The performance convergence of DQNalign method in training procedure is shown in In case of similar sequence pair, we could see convergence within a short time regardless of the learning rates in the training procedure. This phenomenon occurs because the environment has rare indel cases in the training procedure. Therefore, the AI learns the wrong behavior that obtains high scores by judging only the forward direction regardless of the environment.However, it is necessary to learn various environments while evading the biased results to align the actual sequence pairs. 2)To observe the effect of the parameters, we used the simulation environments in Table S3. Here, we compared the exact match result of DQNalign method with that of conventional alignment methods. Also, we considered the Needleman–Wunsch algorithm's result to know the ideal outcome of each simulation case. Then, the result of sequence alignment in the generated sequences for the simulation cases in Table S3 is shown in As shown in Moreover, DQNalign method is not limited to a specific neural network structure, which means that it can learn the alignment process using various types of network structures. As can be seen in this figure, we can confirm that DDDQN has slightly better performance than the faster DDDQN. However, in the complexity results in C.E.coli genome sequences 3. three mammalian genome sequences. In this section, we compared the exact match performance and time complexity between various sequence alignment methods: Clustal Omega, MUMmer, and the DQNalign method. The used detailed parameters are described in Table S6 and Table S7. Then, we implemented the pairwise alignment software based on the Clustal Omega from https:// github. com/etetoolkit/ext_apps/blob/master/src/clustal-omega-1.2.1. The implemented Python code is given in our Github link.To confirm the difference between the DQNalign method and the conventional sequence alignment algorithm, three sequence sets were used: 1. HEV genome sequence set 2. two 1)The results of the HEV sequence set in Table S4 are shown in However, in 2)Escherichia coli O157 and Escherichia coli K-12. These alignment results are contained in Supplementary material S3. The brief alignment results are shown in To observe the possibility of the alignment in long sequences, we did simulations on E.coli was not easy in case of using our DQNalign method. There are gene-scale genomic variations in the E.coli samples; therefore, it was hard to connect the whole sequences using only a few hundred-sized windows. Thus, we decided to additionally use the MUMmer method. Here, the overall improved genome alignment result could be obtained by combining MUMmer with our DQNalign method. We used the DQNalign method to align unaligned gaps of the alignment results obtained using the MUMmer methods. Using the DQNalign with MUMmer, we could properly align the entire E.coli sequences. And we can confirm that the DQNalign with MUMmer method achieves the best exact match performance than other conventional methods .As shown in In view of computational complexity, DQNalign method could align the entire sequence within a few hours. Also, DQNalign with MUMmer improves the exact matches in shorter time by providing an accurate preprocessing process (MUMmer) than DQNalign method. Moreover, we can confirmed that the proposed DQNalign method is much faster than the pairwise alignment algorithm of the Clustal Omega. However, we confirm that the MUMmer has a vastly high speed compared to other methods because the MUMmer method focuses on speed optimization in alignment between close sequence pairs.3)Homo sapiens, Mus musculus, and Rattus norvegicus. Since the proposed DQNalign method is one of global alignment methods, we simulated the performance of proposed and conventional methods in view of aligning the orthologous genes of these three organisms. We align the sequence pairs for the coding sequences of the genes of BRCA1, ELK1, and CCDC91, which are related to cancer incidence Finally, we evaluated the performance of proposed DQNalign method for three most famous mammalian genome sequences: As shown in IV.In this paper, we proposed deep reinforcement learning based alignment method to reduce the complexity of pairwise alignment. Using only simple preprocessing based on the longest common substring (LCS), the DQNalign can align the entire sequences by repeating small alignments. We have confirmed that the DQNalign can achieve high accuracy and low complexity by adjusting the window size adaptively through the analysis.We confirmed the advantages and disadvantages of the DQNalign method by analyzing the generated sequences and real HEV sequences. We observed that the DQNalign method has more accurate alignment performance than the conventional methods in dissimilar sequence pairs. We confirmed that the MUMmer method has a much faster speed than our DQNalign method. It is desirable to choose the DQNalign or the conventional alignment method in the consideration of real-time processing and accuracy.The proposed DQNalign method showed the possibility of aligning two long sequences by combining conventional alignment algorithms. It was revealed that the sequences with millions of bases, such as E. coli, could be aligned by combining the DQNalign methods with the Clustal or the MUMmer. Through simulation results, we confirmed that the DQNalign method could be combined efficiently with conventional MUMmer and Clustal methods in view of improving accuracy.Homo sapiens, Mus musculus, and Rattus norvegicus. DQNalign showed higher performance than conventional alignment methods in aligning genetically disease-intuitive genes such as BRCA1, ELK1, and CCDC91. Through these results, we could confirm that the proposed method is superior to the conventional global alignment in view of the actual clinics determining the presence or absence of disease through genetic testing or finding a biomarker.Besides, we tried to find the proposed method's clinical significance by aligning the coding sequence of three type orthologous genes of Mammalian genome: In the future, we will attempt to apply the DQNalign to local alignment and multiple sequence alignment using improved learning methods and proper training strategies for each case. The DQNalign method used deep reinforcement learning based selection rather than human-based features; therefore, the alignment result was not stable for some sequence pairs. Hence, we will improve the stability of the DQNalign method using the alignment results of the several agents that are differently converged by deep reinforcement learning method. Besides, there were several challenges in securing performance on actual sequences, which result from the gap between the modeled sequences and the real sequences used in the learning. Therefore, we are going to solve these issues using a learning method that directly reflects the actual sequences.Three


As sequencing depth of chromatin studies continually grows deeper for sensitive profiling of regulatory elements or chromatin spatial structures, aligning and preprocessing of these sequencing data have become the bottleneck for analysis. Here we present Chromap, an ultrafast method for aligning and preprocessing high throughput chromatin profiles. Chromap is comparable to BWA-MEM and Bowtie2 in alignment accuracy and is over 10 times faster than traditional workflows on bulk ChIP-seq/Hi-C profiles and than 10x Genomics’ CellRanger v2.0.0 pipeline on single-cell ATAC-seq profiles. As studies continue sequencing with deeper coverage, computational processing of these profiles has become increasingly resource consuming. Here the authors designed an efficient computational method called Chromap to align and preprocess high throughput sequencing data from chromatin profiling techniques, including ChIP-seq, Hi-C, or scATAC-seq, with a major decrease in runtime. Single-cell ATAC-seq (scATAC-seq) further enables the profiling of cis-regulatory elements in individual cells8. Standard analysis workflows, such as those used by the ENCODE project9, start with read mapping by the popular short-read aligner BWA-MEM10 or Bowtie211, along with alignment sorting and deduplication by SAMtools12 and Picard13. These steps are the common bottlenecks, which may take hours or days to complete, compared to the downstream analysis steps such as peak calling by MACS214, which usually takes minutes. One reason for such inefficiency is that the comprehensive base-level alignment results for the purpose of variant calling are unnecessary for most chromatin biology studies. Furthermore, alignment filtering, deduplication, and other preprocessing steps are handled by different methods sequentially in a standard workflow, and each step requires parsing from compressed files. Such repeated I/O significantly increases the running time.Chromatin profiling techniques, such as ChIP-seq15, an efficient read aligner based on the minimizer sketch16. It was initially designed for long reads of high error rate and then extended for short accurate reads. Although a few times faster than FM-index-based short-read aligners such as BWA-MEM and Bowtie2, minimap2 more frequently misses short alignments that lack sufficient minimizer seeds. This becomes a severe issue in mapping scATAC-seq data when a large portion of the read sequence is used for barcoding and indexing, and the remaining genomic sequence in a read can be as short as 50 bp. Moreover, minimap2 has to slowly scrutinize the alignment to resolve the high sequencing error rate inherent in the long reads even when in the short-read mode, which could be unnecessary for the highly accurate Illumina short-read sequencing data.Our group previously developed minimap2In this study, we present an efficient read alignment and preprocessing method, named Chromap, based on the minimizer sketch Fig. . Chromap17 (no-splicing mode) and Accel-Align18 on three simulated whole-genome sequencing data sets with various read lengths , and chromatin loops. The chromatin compartments (measured by the first eigenvector) and TADs (measured by the insulation score) called from the two aligners gave highly similar results, achieving Pearson correlation coefficients of 0.995 and 0.998 respectively  where x denotes the minimizer start position on the reference and y denotes the minimizer start position on the read. Then the candidate can be estimated by this anchor as x-y. Co-linear anchors  are a set of anchors that appear in ascending order in both the read and reference, which can be found by a dynamic programming algorithm15 in quadratic time with respect to the number of anchors. While this algorithm can robustly identify chains for noisy long reads (>1000 bp with 5 ~ 10% error rate), we present a more efficient algorithm that can generate candidates for short reads with a low error rate. We generate candidates using all the anchors and then sort the candidates. During a linear scan on the sorted candidates, we merge the same candidates or candidates that have smaller than error threshold difference generated from multiple anchors. The error threshold is a user-defined parameter that constrains the edit distance between read and the genomic region. By allowing an error threshold in candidate merging, Chromap accommodates the insertions and deletions when generating the final candidates for a read. During the merging, Chromap records the multiplicity for each candidate, which is also the number of supporting anchors, and filters the candidates with fewer support than the user-defined threshold. For paired-end reads, chains were first generated for each end and then filtered by the fragment length constraint.We define candidates for a read to be possible mapping start locations on the reference genome, which are estimated by exact minimizer hits  between the read and the reference. Formally, an anchor is a pair (M minimizers sequences mi and the M-1 offsets between adjacent minimizers mi. Chromap uses the function h(m) = (m1 + mM) mod N to quickly map the vector to the h(m)-th entry in the hash table of size N = 2,000,003. The advantage of this mapping function is that the identical reads from both strands can access the same cached information. Furthermore, reads that are nearby in the genome have a greater likelihood of generating the same minimizer vector, and they can also share the same cache information.Chromap stores the raw candidates in a cache for frequent reads to avoid repeated candidate generation for reads from peak regions. The cache is a hash table, where the key is a vector of minimizers and the value is the vector of candidates generated from the set of minimizers. The minimizers vector stores the 23, Chromap maintains a small count array of size N′ = 103 in each cache entry to identify the most frequent minimizer vector from hundreds of different vectors mapped to the same cache entry, namely cache collidings. Chromap uses the function f(m) = (m1 ⊕ mM) mod N' to map the vector to the f(m)-th entry in the count table by computing the XOR of the minimizer codings, which has the same advantage of ignoring the read strand. Chromap then updates the cache table if and only if the count for the minimizer vector is more than 20% of the total count in the count array and is the dominant minimizer vector  among the vectors mapped to the count array entry f(m). As a result, Chromap not only stores in cache the candidates from frequent minimizer vectors, but also avoids unnecessary cache updates from the background noises.Inspired by the count-min sketchChromap supplements the candidates with read-pair information to recover the lost candidates due to the minimizer occurrence limit. For each read end, Chromap will pick its mate’s candidate supported by the most number of anchors and use this mate’s candidate as the estimation for the read coordinate. As a result, for each minimizer in the read end, instead of extracting all the occurrences on the reference, Chromap applies a binary search in the index entry to only select the occurrences within the range estimated read coordinate determined by the fragment size distribution. Chromap then executes the same candidate generation algorithm to supplement the candidates with the minimizer occurrences from the binary searches.25 to pick the optimal candidate coordinate with minimum edit distance to the reference genome. To further accelerate the verification step, we parallelized the algorithm using SIMD instructions on the CPU to align the read with multiple candidates on the reference simultaneously. We also modified the algorithm to efficiently trace back the alignment so that accurate start and end mapping positions can be obtained.Since each read can have multiple candidate mapping positions, we implemented a banded Myers’ bit-parallel algorithmWhen the edit distance exceeds the threshold during the candidate verification step, we check if the length of the mapped read is greater than a certain length threshold. If the length of the mapping passes the length filter, the mapping is kept with an estimated mapping score as the mapped read length minus the edit distance. Note that for some of the Hi-C reads, there can be a small region (<20 bp), which cannot be mapped at the beginning of its 5′ end. To resolve this issue, when the mapping length is too short, the first 20 bp of the read is excluded and a second round of mapping of the remaining region is performed. If a mapping generated in this way passes the length filter, the mapping is then extended backward from its beginning to its maximum exact match. For paired-end data in split-alignment mode, Chromap ignores the constraints from the read-pair, such as the fragment length or strandness.When the data set is small, all the mappings can be kept in the memory and sorted to remove duplicates. For large data sets or limited memory, we provide a low memory mode. It saves mappings in chunks temporarily on the disk and uses an external sort to merge them into the final mapping output in a low memory footprint. For scATAC-seq data, duplicates can be removed at either bulk level or cell level (default) based on the users’ choice.Using the barcode whitelist provided by 10x Genomics, we correct barcodes that are not on the whitelist. Prior to the correction, the barcodes are converted to their bit representations and the abundance of each barcode is computed efficiently using a hash table. For barcodes outside the whitelist, all whitelisted barcodes within one Hamming distance from the barcode to correct are extracted by a set of efficient bit operations. Using the quality score of the mismatched base and the abundance of these whitelisted barcodes as a priori, we compute the posterior probability of correcting the observed barcode to the whitelisted barcodes. We make the correction if the highest probability of the observed barcode being a real barcode is > = 90%. The correction step is performed as part of the read mapping process which is in parallel of loading the next batch of reads.26 with an average sequencing error rate 0.1% and read lengths 50 bp, 100 bp, and 150 bp. The bulk CTCF ChIP-seq data on the human VCaP cell line were downloaded from ENCODE to test the tools on bulk sequencing data. The 10k PBMC scATAC-seq data set is publicly available from 10x Genomics and used to evaluate the performance of the tools on single-cell data. To investigate the impact of alternating BWA-MEM with Chromap on chromatin conformation analysis, we combined the two Hi-C data replicates from a previous study7.In this work, we evaluated Chromap on various data sets including simulated whole-genome sequencing data, bulk ChIP-seq data, 10x Genomics scATAC-seq data, and Hi-C data Table . One milIn this work, we compared Chromap with five state-of-the-art short-read aligners minimap2(v2.17), STAR (v2.7.9a), Accel-Align (GitHub commit code 7217a9f), BWA-MEM (v0.7.17), and Bowtie2 (v2.4.2). STAR is designed to align RNA-seq data which contains spliced alignments across introns, so we used the options “--alignIntronMax 1 --alignEndsType EndToEnd” to forbid spliced alignment. When testing on simulated data, we converted all the alignments into PAF format and used the paftools to calculate the accuracy of alignments. Using the bulk ChIP-seq data, we compared the consensus of alignments and peaks among the aligners after filtering the alignments with MAPQs less than 30 based on ENCODE protocol (7 for Accel-Align). Accel-Align computes MAPQs in a different way, so we compared the distribution of MAPQs from all the aligners in the simulated data and found MAPQ 7 in Accel-Align was highly similar to MAPQ 30 in other aligners. All the methods were tested in a multiprocessing environment with 8 threads. Accel-Align was tested with option “-x” for the fast alignment-free mode.20, on a large Hi-C data set on human cell line K562. We filtered the alignments with MAPQ 0, which follows the default parameter settings in pairtools. Due to complexity introduced by the ligation junction in a Hi-C experiment, direct comparison of alignment coordinates would underestimate the consistency between the methods. Therefore, we compared the contact maps derived from the alignments at various resolutions. We compared the overall distribution of chromatin contacts at 25 kb resolution by using the stratum-adjusted correlation coefficients (SCC); the chromatin compartments measured by the first eigenvector of the normalized contact matrices at 100 kb; the TAD boundary strength measured by the insulation score at 25 kb resolution; and the identified chromatin loops at 10 kb.We compared Chromap and the standard 4D Nucleome Hi-C processing pipeline, which is based on BWA-MEM and pairtoolshttps://pypi.org/project/hicreppy/, v0.0.6)27 between Chromap and BWA-MEM on the same replicate (Chromap R2 vs. BWA-MEM R2) or between two replicates (BWA-MEM R1 vs. BWA-MEM R2). Because the original two replicates have disparate sequencing depths , we first down-sampled R1 to make it match the sequencing depth of R2. The resulting SCC between Chromap R2 and BWA-MEM R2 was 0.998, which was significantly higher than SCC between BWA-MEM R1 and BWA-MEM R2 (0.945). HiCRep was run at 25 kb resolution, and the smoothing factor and the maximum genomic distance were set to 5 and 2 Mb, respectively. For the following chromatin conformation analysis, we merged the alignment results from the two replicates.To confirm that the difference between Chromap alignments and BWA-MEM alignments was smaller than the difference between biological replicates, we computed SCCs by using a Python implementation of HiCRep . For compartments, the eigenvalue decomposition was performed on the 100 kb intra-chromosomal contact maps, and the first eigenvector (PC1) was used to capture the “plaid” contact pattern. The original PC1 was oriented according to a K562 DNase-Seq track (ENCODE accession code: ENCFF338LXW) so that positive values correspond to active genomic regions and negative values correspond to inactive regions. The Pearson correlation of PC1 was 0.995 between Chromap and BWA . Among the 9455 and 9950 loops identified from Chromap and BWA-MEM respectively, we found 8,385 of them were supported by both methods  and adjusted rand index (ARI) calculated by the Python package scikit-learn. We first computed the baseline NMI and ARI between CellRanger v1.2.0 and CellRanger v2.0.0. Chromap vs CellRanger v2.0.0 achieved a higher consistency score than the baseline score, suggesting the results from Chromap were highly consistent with CellRanger and were more consistent than CellRanger version changes Table . To confFurther information on research design is available in the Supplementary InformationReporting Summary


This paper summarizes the OpenEDS 2020 Challenge dataset, the proposed baselines, and results obtained by the top three winners of each competition: (1) Gaze prediction Challenge, with the goal of predicting the gaze vector 1 to 5 frames into the future based on a sequence of previous eye images, and (2) Sparse Temporal Semantic Segmentation Challenge, with the goal of using temporal information to propagate semantic eye labels to contiguous eye image frames. Both competitions were based on the OpenEDS2020 dataset, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display with two synchronized eye-facing cameras. The dataset, which we make publicly available for the research community, consists of 87 subjects performing several gaze-elicited tasks, and is divided into 2 subsets, one for each competition task. The proposed baselines, based on deep learning approaches, obtained an average angular error of 5.37 degrees for gaze prediction, and a mean intersection over union score (mIoU) of 84.1% for semantic segmentation. The winning solutions were able to outperform the baselines, obtaining up to 3.17 degrees for the former task and 95.2% mIoU for the latter. Eye tracking has emerged as a powerful tool for several applications, including health assessment and intervention ,2, and hOne of the core technologies that enables high-quality immersive VR/AR experiences while keeping a low computational cost of rendering the environment is called Foveated Rendering (FR) . FR rendvideo oculography, can be broadly classified into geometric and appearance-based systems [Solutions for image/video-based gaze estimation, also known as  systems . Geometr systems , and fin systems , it is a systems ,19 with  systems ,21, to d systems .In real-world applications, the input to any gaze estimation system is temporal sequences of eye images in the form of a video. However, most popular approaches do not exploit temporal information and instead estimate gaze direction for each frame separately. Intuitively, there exists useful temporal information in videos that can be leveraged to improve current gaze estimation approaches, for example, by incorporating spatio-temporal features captured over a sequence of eye images . SimilarMotivated from the aforementioned gap in the literature, we organized the OpenEDS 2020 Challenge, the second iteration of the OpenEDS-Eye Tracking for VR and AR Challenges, to foster research on spatio-temporal methods for gaze estimation, gaze prediction, and semantic segmentation. In particular, the challenge was divided into 2 competition tracks: (1) Gaze Prediction, which called for designing solutions to predict future gaze directions by leveraging the spatio-temporal information encoded in a sequence of recorded eye images; and (2) Sparse Temporal Semantic Segmentation, which called for designing solutions to propagate semantic segmentation labels for key eye regions, by leveraging a few hand-labeled frames in a given video sequence of eye images. Both tracks were based on a novel dataset, OpenEDS2020, a large-scale anonymized dataset of sequences of high-resolution eye images (640 × 400 pixels) sampled at 100 Hz with semantic segmentation and gaze direction annotations, which features up to 87 subjects with a wide range of appearance variability. The dataset was collected using a VR head-mounted display (VR-HMD) equipped with two synchronized eye-facing IR cameras. Example images of the dataset are shown in In this paper, we describe the rationale behind both challenges, the methodological baselines used, and solutions proposed by the challenge winners of both competition tracks. In addition, we describe in detail the collection and curation process of the OpenEDS2020 dataset, which we publicly release to the research community. We believe this dataset can be instrumental in advancing the current state of the art in gaze estimation, gaze prediction, and semantic segmentation research, and benchmarking of existing or future algorithms.The remainder of this paper is organized as follows. https://research.fb.com/programs/openeds-2020-challenge/, accessed on 25 June 2021) was hosted as part of the OpenEyes: Eye gaze in AR, VR and in the Wild  workshop, organized at the European Conference on Computer Vision in 2020. The challenge was divided into 2 competition tracks: (1) Gaze Prediction Challenge, and (2) Sparse Segmentation Challenge. Both tracks were hosted on the Eval.ai  platform and were active from 1 April through 31 July 2020. To participate in either of the tracks, participants had to request access to the dataset and register in the submission platform. Their goal was to devise solutions for either or both challenge tracks using a defined training/validation set, apply their methods to a test set with hidden labels, and upload their predictions to the submission platform to obtain the performance score of the given solution. Such platform included a public leaderboard where participants could compare the performance of their solution against other teams and the baselines proposed by the organizers. The competition tracks and their evaluation protocol are described below.The OpenEDS 2020 Challenge , or a combination of both. Considering real-time requirements of FR and its goal of reducing power consumption, the prediction of future gaze points based on a short subsequence of the already-estimated gaze locations is considered the most fruitful path of exploration. If predicting future gaze locations with high accuracy is feasible, it would allow an implementation of an FR method that would match closely with the human visual acuity function. As a result, it could encode only a tiny part of the image at a high-quality resolution, providing the highest level of energy savings.Instead of predicting gaze locations, for which the accommodation and vergence of the two eyes would need to be considered, the challenge focused on monocular 3D gaze direction prediction as a first step of a gaze location prediction approach. In particular, the challenge called for the following: (a) Predicting future gaze directions based on the previously estimated gaze direction vectors of the previous eye images; (b) Predicting future gaze directions by leveraging the spatio-temporal information encoded in the sequence of previous eye images.t*10 ms after the last provided sample of the sequence s, and The task consisted in designing a model to predict the 3D gaze direction vector up to 50 ms into the future, given a sequence of previous eye images. For a dataset recorded at 100 Hz, 50 ms is equivalent to 5 frames. Participants of the challenge were scored on a test set, which contained the sequence of previous eye images with hidden ground-truth vectors, using the performance metric of Prediction Error (PE), defined as:The 3D gaze direction vector is defined as the 3D unit  vector in the direction of the visual axis, represented in the 3D Cartesian coordinate system. For training, participants were provided with both eye images and ground-truth vectors, so that they could train appearance-based gaze estimation models using the given dataset. No subject calibration details were provided.Many eye tracking solutions require accurate estimation of eye features from 2D eye images. Typically, this is done via semantic segmentation of key eye regions. To generalize models for per-pixel segmentation of unseen eye images from a diverse population under different eye states  and different makeup conditions, the model training stage requires large-scale, hand-annotated training data that can be costly and time-consuming. However, it is easy to obtain a data acquisition setup that captures medium-to-short duration (5–25 s) video sequences of eye images and manually label a handful of images (5%) for each sequence.Then, the goal of the challenge was to solve label propagation with a limited number of samples per sequence. Solving this problem allows one to have a large set of annotations without spending large amounts of resources on human annotation. We posit that the small fraction of hand-annotated labels can be accurately propagated to the rest of the images in the sequence by leveraging temporal information along with geometric and photometric consistencies arising from the eye images of the same person. Such approaches called for innovative algorithms to leverage the aforementioned cues. Some promising directions were foreseen to be: (a) Temporal co-segmentation using deep learning; (b) Temporal few-shot learning framework; (c) Learning and respecting the natural representation, including the geometry, of human eyes for temporal label propagation; (d) Leveraging synthetic data generation if and where appropriate.Participants of the challenge were given sequences of eye images with a small fraction of them annotated with semantic segmentation masks. The task was to design a model to perform semantic segmentation on a subset of the images that did not have an annotated mask, which we refer to as test subset. The performance metric used to evaluate the proposed solutions was the mean Intersection over Union  over all the key regions of all the images conforming the test subset.We believe that an eye tracking dataset designed for potential spatio-temporal methods should contain a sufficiently representative gaze angle distribution and appearance variability to train gaze estimation or semantic segmentation models, while ensuring variability in terms of eye movements, directions, and velocities to train prediction models.Due to the complexity of annotating accurate gaze direction and eye movements, there is a limited number of publicly available datasets of eye-image sequences with high appearance diversity providing multiple annotations , the recordings from which the eye-image sequences are obtained are unique to each subset. Apart from the data used in the challenge, the new public version of the dataset is enriched with additional per-image/per-sequence annotations. The data collection procedure, curation process of each data subset, and annotations provided are described below.The dataset was captured from 90 voluntary subjects of ages between 20 and 70 years old with appearance variability in terms of ethnicity, gender, and age, and some of them wearing glasses, contact lenses, and/or makeup. There was no assessment by an eye health professional or cut-off by visual performance. These subjects were all recruited internally, and provided written informed consent for using their eye images for research and commercial purposes before taking part in the data collection stage. Subjects were asked to wear a VR-HMD, mounted with two synchronized eye-facing IR cameras at a frame rate of 100 Hz, and were recorded while gazing at specific dot patterns displayed on a blank screen with different target movements. Each recording, lasting approximately 5 min, consisted of a set of patterns: (a) Ring-shaped patterns at ±20 degrees eliciting smooth pursuit movements, at different depths; (b) Random point changes to induce saccades, where the targets were moved in a ±20 degree cone from 50 cm in depth to 600 cm. The background illumination was constant throughout each recording. The dataset was anonymized to remove any personally identifiable information on the subjects for further processing. Original recordings were later divided into one recording per subject and pattern used.This subset of data are aimed at fostering research on spatio-temporal methods for gaze estimation and prediction for tasks involving different eye movements, such as saccades, fixations, and smooth pursuit. Examples of different eye movements included in the dataset are illustrated in Two types of patterns were selected for this data subset, due to the differences in eye movement dynamics associated with them: (a) Ring-shaped smooth pursuit-elicited pattern, intertwining 1s fixations at a fixed depth with 1s-long smooth pursuit movements as smooth transitions between fixation points, with a total of 17 fixations and 16 transitions per recording; (b) Saccade-elicited pattern, with 1s-long randomly located target fixations at different depths and instantaneous (0.1 s) target transitions, with up to 20 fixations per recording. All subjects were recorded following both pattern types. The former pattern was the same for all subjects, while the latter was always different due to its random nature. However, note that although the ring-shaped pattern was the same, the elicited gaze trajectories were different across subjects due the subject-dependent nature of eye movement dynamics .s non-overlapping sequences of f contiguous frames each per recording, with a maximum of 100 frames (1 s) per sequence. This way, each sequence can contain either fixations only, smooth pursuit movement only, a combination of fixation and smooth pursuit, or a combination of fixation and saccadic movement, depending on the pattern segment they fall within. Each eye was processed individually, and right eye (from the camera point of view) and respective ground-truth vectors were flipped horizontally to seamlessly augment the data. These processing steps also allowed to break the symmetry of the ring-shaped pattern, making it difficult to recover the original pattern shape. For the challenge, the type of pattern used for each sequence was not provided, to prevent challenge participants from using that prior information for their proposed gaze prediction methodologies.Frames with invalid ground-truth gaze vector, due to either subject distractions, blinks or incorrect detections see , were maWe devised three different subject-independent partitions of our dataset, with 32 subjects for training, 8 for validation and 40 for test. To do so, we performed stratified sampling in terms of gender, ethnicity, age, and glasses, to ensure a representative sample for testing. The training subset consists of 10 100-frame sequences per eye, type of pattern, and subject, with a total of 4000 images per subject and 128,000 images in total. Assuming that we can use up to 50 frames to initialize a gaze prediction model to predict up to 5 frames into the future, using a sliding window of stride 1 allows us to obtain up to 46 subsequences in a 100-frame sequence, which sums up to 58,880 final sequences in the training set . For validation and test subsets, however, we chose to provide 55-frame sequences to have one set of initialization frames and predictions per sequence, which facilitates evaluation and analysis. To compensate for the difference in number of effective sequences with respect to the training set, we selected approximately 80 55-frame sequences per type of pattern and subject, with 70,400 images for validation and 352,000 for test. For subjects for which there was not enough valid data to obtain such 80 sequences per pattern, the maximum number of valid sequences was selected, and the remaining sequences were obtained from other subjects until we obtained the desired number of sequences, 1280 and 6400, respectively. Please note that the goal of training and validation subsets is both gaze estimation and prediction, while the goal of the test subset is gaze prediction. Therefore, even though the number of images is substantially bigger for the test set, the number of sequences is what we focus on.The distribution of ground-truth gaze angles  for each data split is depicted in Each eye image was provided along with its respective ground-truth 3D gaze vector, corresponding to the visual axis, in the headset coordinate system . Gaze vectors and cornea centers of each eye were obtained using a hybrid model, which combines a deep eye-segmentation model see  with a cBesides the eye-image sequences and associated gaze vectors provided for the challenge, we are including additional metadata, namely: type of pattern used (saccade-based or smooth pursuit-based), 3D target coordinates, target status , 3D point of gaze, 3D cornea center coordinates, internal (anonymized) subject ID, and sequence ID.This subset of data is aimed at exploiting the temporal information available in the form of short temporal sequences of eye images to improve the semantic segmentation accuracy achieved by treating each frame separately. Although there could be multiple approaches to exploit temporal information for improving semantic segmentation, we resort to a simple and practically useful problem of accurate label propagation from a few labeled images to all the frames in a sequence. On the one hand, this problem can serve as a testbed for future improvements in the latest deep-learning-based spatio-temporal models for real-time inference in videos, few-shot learning, geometry-constrained semantic segmentation, and/or co-segmentation. On the other hand, this problem also helps in generating high-quality pseudo-ground-truth data generation for training semantic segmentation networks.label-ratio, R, as:R, which resulted in ∼150 frames (∼30 s of recording) for a total of 29,476 frames with 2605 semantic segmentation annotation masks. We start the curation process with the initial data, i.e., a total of ∼600,000 images in the form of 594 temporal sequences and 11,476 hand-annotated semantic segmentation masks chosen randomly for annotation. All sequences belong to the ring-shaped pattern category, with target depths ranging from 50 cm to 600 cm. Let us define the Human annotations are provided in the form of pixel-level segmentation masks with the following labels: (1) eye region, (2) iris, and (3) pupil. Annotation quality was evaluated by estimating the mIoU score of annotations, ensuring that no labels produced by at least two annotators have less than 80% mIoU. The right eye was flipped horizontally to align lacrimal caruncles across left and right eye annotations, so the annotators have consistency. The lacrimal caruncles were labeled as part of the eye region. In half occlusions, the annotations only include visible parts of the pupil and iris, and eyelashes are labeled as part of the underlying pupil or iris region. Finally, during blinks a thin sliver of the eye region is labeled. Examples of human annotations are provided in Besides the eye images and associated segmentation masks provided for the challenge, we are also releasing the internal subject ID associated with each sequence, so that sequences corresponding to the same subject can be linked.In this section, we describe and evaluate a set of baseline models for each data subset to demonstrate the usefulness and validity of the data for the suggested tasks.We propose a simple baseline model for the Gaze Prediction subset, in which spatio-temporal information is not jointly leveraged. Instead, we disentangle gaze estimation from prediction, estimating first the line of gaze from each eye image individually, and then performing gaze prediction based on the previously estimated gaze vectors.First, we convert the ground-truth 3D gaze vectors to 2D spherical coordinates, representing yaw  and pitch  angles. For gaze estimation, we train a deep-CNN based on a modified ResNet of 13 convolutional layers, as in . The CNNOur gaze prediction approach relies on linear regression. In particular, we use a window of 50 estimated gaze angles to compute the regression parameters using 2 independent regression models, i.e., one per gaze axis in 2D. The estimated parameters are used to predict the next 5 frames into the future. We apply our trained gaze estimation network on the test subset to estimate gaze for the first 50 frames of each sequence, compute linear regression parameters on them and predict the next 5 frames. To set a baseline for spatio-temporal eye-segmentation algorithms, we chose to train a deep-CNN on 1605 images whose corresponding hand-annotated semantic segmentation masks are provided. The network follows an encoder-decoder architecture loosely based on SegNet . ModificWe trained the network for 150 epochs with ADAM optimizer, with initial learning rate of 0.004 and batch size of 128. We used random rotation and intensity scaling for augmentations. Model training and inference was performed on downsampled images of size 128 × 128 pixels. The total number of parameters of the model was 40K and the resulting model size was only 300 KB.The trained network is tested on the hidden set of test samples without exploiting any temporal information, achieving a mIoU score of 0.841 . What follows below is a brief description of the methods explored by the winning teams and the performance numbers produced.The Gaze Prediction challenge received a total of 13 valid entries, while the Sparse Semantic Segmentation challenge received 22 valid entries. From them, the code implementations of the top-3 winning teams of each competition track were evaluated and the final winners announced at the workshop  to predict the future frames. Both team 8 module  to work function  applied backbone  to trainbackbone .random_b chose a rather simple strategy: check if there is a fast change in gradient of the gaze vectors in the last 7 frames of the sequence and if so, add the computed gradient linearly to the predicted frames, otherwise use moving average gaze vector derived from the last 3 frames. The proposed algorithm was applied independently to each axis of the gaze vector. Team Eymazing and team Dmitrykonolov, on the other hand, chose to train a standard Long-Short-Term Memory (LSTM) recurrent neural network [Eymazing, again, used the cosine loss function for their LSTM network, computed only on the predicted frames, while team Dmitrykonolov relied on their pre-designed custom loss function.To train the prediction network, team  network  to predirandom_b, seemed to work best, producing their winning model. This team, however, noted after the competition was over that they were able to produce even better performance numbers using a LSTM recurrent network for training. This conforms with the findings by team Eymazing as well as team Dmitrykonolov.The following general observations can be drawn from the experience of these three winning teams. First, training estimation and prediction networks separately. Second, overfitting was a significant challenge, especially when using complex backbone neural architectures for training the estimation network. As such, Resnet18 with suitable data augmentation methods or the choice of a suitable loss function worked best. Finally, a rather simple prediction algorithm, as proposed by team In BTS Digital, used a UNet segmentation network [tetelias, on the other hand used a DABNet backbone [JHU, trained an ensemble of UNet models followed by heavy dose of pseudo-labeling on all the available unlabeled images.The Sparse Semantic Segmentation challenge called for propagation of semantic labels in a video sequence of eye images. However, none of the winning teams produced a solution to leverage the temporal information in the video sequence of images. Instead, a common theme running across the solution developed by all 3 winning teams was the use of a rather simple encoder-decoder neural network architecture, extensive data augmentations, pseudo-labeling on unlabeled images and model ensemble. Specifically, the first-place winner, team  network  to train network  and dice network . Two rou network  with 90 backbone  pre-traibackbone  for traibackbone  and FMixbackbone  followedIn summary, contrary to our original motivation for seeking novel ideas to propagate knowledge from sequences of eye images to augment semantic segmentation for video frames with missing labels, pseudo-labeling with ensemble averaging proved to be the winning recipe to produce winning solution to the proposed problem. Although the winning numbers are impressive, we still believe that there is room for improvement on this task, should novel methods to leverage sequence information be devised.In this paper, we have reviewed the OpenEDS 2020 Challenge motivation, baselines proposed, winning solutions and findings. Moreover, we have described the OpenEDS 2020 Dataset, a novel dataset of eye-image sequences consisting of up to 87 subjects of varied appearance performing different gaze-elicited tasks. The dataset consists of two subsets of data, one for each competition track of the challenge, i.e., one devoted to gaze prediction and another devoted to eye semantic segmentation, with the goal of fostering the exploitation of spatio-temporal information to improve the current state of the art on both tasks. Obtained results with proposed baseline methods and winning solutions, all of them based on deep learning techniques, demonstrate the usefulness of the data, and serve as a benchmark for future approaches coming from computer vision, machine learning and eye tracking fields.https://research.fb.com/programs/facebook-openeds-2021-challenge/, accessed on 25 June 2021). Based on the observation that all winning teams for the OpenEDS 2020 Gaze Prediction challenge trained the estimation network first and then followed up with the training of the prediction network, the 2021 Gaze Prediction challenge is solely focused on the framing of the prediction problem. The time series of gaze is derived from individuals freely exploring two virtual environments participating in varying activities such as reading, drawing, shooting and object manipulation [In particular, the learnings from the outcome of the Gaze Prediction challenge helped us refine the formulation for the OpenEDS 2021 Gaze Prediction challenge (OpenEDS 2021 Eye Tracking Challenge pulation . The cha


Next-generation sequencing (NGS) represents a significant advancement in clinical genetics. However, its use creates several technical, data interpretation and management challenges. It is essential to follow a consistent data analysis pipeline to achieve the highest possible accuracy and avoid false variant calls. Herein, we aimed to compare the performance of twenty-eight combinations of NGS data analysis pipeline compartments, including short-read mapping , variant calling  and interval padding  methods, along with a commercially available pipeline . Fourteen germline DNA samples from breast cancer patients were sequenced using a targeted NGS panel approach and subjected to data analysis.TP53 gene. We also recommend the BWA-MEM algorithm for sequence alignment, whereas variant calling should be performed using a combination of variant calling algorithms; GATK-HaplotypeCaller and SAMtools for the accurate detection of insertions/deletions and GATK-UnifiedGenotyper for the efficient detection of single nucleotide variant calls.We highlight that interval padding is required for the accurate detection of intronic variants including spliceogenic pathogenic variants (PVs). In addition, using nearly default parameters, the BWA Enrichment algorithm, failed to detect these spliceogenic PVs and a missense PV in the These findings have important implications towards the identification of clinically actionable variants through panel testing in a clinical laboratory setting, when dedicated bioinformatics personnel might not always be available. The results also reveal the necessity of improving the existing tools and/or at the same time developing new pipelines to generate more reliable and more consistent data.The online version contains supplementary material available at 10.1186/s12859-021-04144-1. Massively parallel sequencing, also known as next-generation sequencing (NGS), represents a significant advancement in clinical genetics and has revolutionized the field of molecular genetics, as it enables the investigation of several genes and samples simultaneously . To thisAmong many short-read mapping algorithms, Burrows–Wheeler Aligner (BWA)-Maximal Exact Match (MEM) , Stampy Through the years, divergent variant calling algorithms have been developed which function by distinguishing true variants from alignment errors . The Genhttps://gatk.broadinstitute.org/) suite recently suggested additional interval padding . Although interval padding is clearly stated as an optional parameter in the documentation of various variant calling algorithms and indeed interval padding is being used , BRCA2  and PALB2  genes, respectively. These frameshift PVs were detected by all pipeline combinations except those based on GATK-UG calling, irrespective of interval padding. In addition, one patient carried a missense TP53  PV. Although all pipeline combinations achieved the detection of the PV, this call was filtered out, subsequent to the variant allele frequency (VAF) threshold (≥ 30%). Only BWA-MEM/GATK-UG with null padding (VAF = 31.405%), BWA-MEM/GATK-UG with 50 bp padding (VAF = 30.579%) ] and one in the BRIP1  genes. All pipeline combinations detected both variants. The third patient carried an intronic VUS in the ATM gene [c.2838 + 10G > A]. This variant was detected only upon inclusion of interval padding. Likewise, the BWA Enrichment application failed to detect this intronic VUS.The top tier performing pipelines were selected based on the corresponding MCC value and compared for their concordance for SNV calls. As shown in Fig. ,p value = 0.011) and BWA Enrichment . However, in regards of intronic indels, GATK-HC exhibited an increased FPR  compared to GATK-UG, SAMtools and BWA Enrichment  irrespective of interval padding (FPR = 59.16%) Fig. . In detaIn this study we carried out a comprehensive comparison of the performance of short-read sequence alignment  and variant calling algorithms , in combination with interval padding length , for the analysis of targeted NGS data. Using targeted short-read data of 14 samples from a single NGS panel study of BC patients, we evaluated different pipelines based on several criteria, including mapping efficiency, depth of coverage, variant calling performance, detection of actionable variants and false positive rates. These results provide valuable information about the performance of the selected tools towards the molecular diagnosis of BC susceptibility, as well as insights for the selection of the most accurate variant calling pipeline, towards targeted-panel and exome sequencing data analysis.Data pre-processing and variant discovery were performed according to EuroGentest and European Society of Human Genetics guidelines for the evaluation and validation of NGS applications, for the diagnosis of genetic disorders . AlthougAt first, we investigated mapping efficiencies for the BWA-MEM, Bowtie2 and Stampy aligners. Although, BWA-MEM and Bowtie2 demonstrated highly comparable mapping efficiencies, implementation of BWA-MEM mapping possessed the highest mapping efficiency. Our results agree with studies reporting that BWA-MEM possessed a lower number of misaligned reads compared to Bowtie2 , 9, 28. p value < 0.05) with Bowtie2 exhibiting reduced performance compared to BWA-MEM and Stampy. The top performing tier pipelines based on our comparisons are BWA-MEM/SAMtools with 100 bp padding, followed by BWA-MEM/SAMtools and Stampy/SAMtools pipelines with 50 bp padding, Stampy/SAMtools with 100 bp padding and BWA-MEM/GATK-UG with 50 bp padding. Likewise, Stampy/SAMtools with 50 bp padding followed by BWA-MEM/GATK-UG with zero and 50 bp padding, BWA-MEM/SAMtools with 100 bp padding and BWA Enrichment application, were the top tier exonic SNV calling pipeline combinations. Our results, partly agree with previous data [Herein, we present that the alignment method affects the total variant and exonic SNV calling performance . Nevertheless, there are still reports supporting that indel calling efficiencies are better for the GATK-UG than GATK-HC [The precise detection of indels and intronic variants is more challenging since there are limited guidelines. It is interesting that in our hands, irrespective of interval padding and alignment algorithm, all GATK-UG based pipelines failed to detect indels—including truncating PVs—, compared to GATK-HC, SAMtools and BWA Enrichment. These results match the current knowledge that GATK-HC and SAMtools have a superior ability of calling indels, compared to GATK-UG , 30, 32. GATK-HC , 30, 31  GATK-HC , 10, 34.PTEN, PMS2 and NF1 genes which are known to bear pseudogenes that potentially affect the downstream analysis [While the sensitivity of each pipeline needs to remain at high levels, there is a great need to reduce the number of false positive variant calls. Bowtie2 in combination with GATK-UG calling, demonstrated overall, the highest false positive rate, irrespective of interval padding and variant type. This is due to the fact that it exhibited the highest false positive SNV calling rate. In addition to this, there is evidence that GATK-HC produces a large number of novel indels , 31. Arganalysis , 36. We https://gatk.broadinstitute.org/). So as of GATK version 3.3, they recommend using GATK-HC in all cases, with no exceptions [The Genome Analysis Toolkit has been widely accepted and is regarded as the “Gold Standard”, especially for germline Illumina sequencing data . It is cceptions .BRCA1, BRCA2 and PALB2 genes, which were detected by all pipeline combinations except those based on GATK-UG calling, irrespective of interval padding. In addition, one patient carried a missense TP53 PV. Although all pipeline combinations achieved to detect the missense PV, this call was filtered out, subsequent to the VAF threshold except for three GATK-UG based pipeline combinations that managed to detect it with adequate VAF. However, we cannot ignore that this observation may be attributable to a possibility that this missense PV could be a true mosaic event with a low VAF (< 30%) and not an argument over which variant calling algorithms perform better [As discussed, seven out of fourteen samples carried PVs in established BC susceptibility genes. Among these, two splice site variants in three samples, were only detected upon inclusion of interval padding. Likewise, using the BWA Enrichment algorithm we failed to detect these PVs. Three additional samples carried frameshift PVs in the m better . Hence, Even though all twenty-eight pipeline combinations converge on a relatively large proportion of variants detected, there still exists a significant degree of variability, with near-default parameters. This discordance is a consequence of different alignment and variant calling methods, as well as the use of different alignment and variant calling parameters. It is important to note that the performance of the above-mentioned tools is by no means constant since they are continuing to improve over time, whereas algorithms which are only commercially available (such as NovoAlign ) were noWe finally support the necessity of improving existing tools or developing new algorithms to achieve more reliable and more consistent calling results. Although our findings should be validated using a larger dataset, as well as explored further using different NGS panels, the outcome of our study has important implications for the diagnosis of BC susceptibility through panel testing in diagnostic molecular genetic testing laboratories, where the high quality of the offered clinical genetic tests is of paramount importance.TP53 gene. We additionally suggest that GATK-HC and SAMtools should be used in combination for the accurate detection of indels, since GATK-HC demonstrates high recall rates, while SAMtools demonstrates high precision rates. Moreover, GATK-UG is suggested for the efficient detection of SNV calls. Finally, as precision medicine advances rapidly and NGS technologies are being widely integrated as a routine diagnostic tool, we highlight the necessity of accurate variant calling and bioinformatics expertise.We recommend the inclusion of interval padding and alignment with BWA-MEM for the accurate detection of intronic variants including spliceogenic PVs associated with the disease of interest. We also demonstrate that using nearly default parameters, the BWA Enrichment® failed to detect all the spliceogenic PVs and a missense PV in the Results for the validation experiments described in this study were obtained from a targeted NGS-panel study of BC patients. Each study participant signed an informed consent form and agreed to undergo genetic testing. All study participants were carriers of known PVs or VUSs, which were identified previously by Sanger sequencing.Library preparation was performed on genomic DNA samples using a panel of 94 cancer susceptibility genes (Illumina TruSight Cancer Sequencing panel—#FC-121-0202). The panel contains oligos targeting and enriching more than 1700 exons including coding regions and noncoding exon-flanking regions (~ 50 bp) spanning 94 cancer susceptibility genes  SortSam tool. Duplicates were detected and removed using the Picard MarkDuplicates tool and read groups were added using the Picard AddOrReplaceReadGroups tool. Local realignment around indels was performed using the GATK (v3.6-0) (https://gatk.broadinstitute.org/hc/en-us) IndelRealigner tool and bases were recalibrated according to the best practice guidelines . Depth of coverage was calculated using the GATK DepthOfCoverage tool. An interval file with the coordinates of the genomic regions targeted by the panel was downloaded from Illumina’s repository (https://support.illumina.com/downloads/nextera-flex-for-enrichment-enrichment-manifest-files.html) and used for variant calling. The interval file was used as such , or extended with 50 bp or 100 bp padding. Variant calling was performed using the GATK-UG, GATK-HC and SAMtools (v1.9) (http://samtools.github.io/bcftools/bcftools.html) mpileup and call tools. Alignment and variant calling were also performed using the BWA Enrichment (v2.1.2) application of Illumina, Inc. , which includes BWA mapping and GATK variant calling. Adapter trimming was selected as an advanced option. Since 50 bp or 100 bp interval padding was not an option, 150 bp interval padding was included in the enrichment analysis.Prior to mapping, adapter and low-quality trimming was performed on the FASTQ files, using the Cutadapt tool (v1.9) . AccordiBRCA1,BRCA2,PALB2,RAD51D,ATM,CHEK2,PTEN,TP53) and other cancer predisposition genes . Downstream analyses included variant filtration based on position (coding sequence ± 10 bp flanking regions), DP ≥ 30X and VAF ≥ 30%. Possible mosaic events (VAF < 30%) were excluded from the analysis. Twenty-eight variant calling pipeline combinations were compared, including combinations of sequence alignment, variant calling algorithms and interval padding lengths, along with Illumina’s BWA Enrichment application  and intronic indels (± 1- ± 10). We verified all variants passing quality control filters by Sanger Sequencing. In the sequel, we defined true positive (TP), false positive (FP), true negative (TN) and false negative (FN) variants. True Positives are variant sites confirmed by Sanger Sequencing. True Negatives are sites correctly called as reference . False positives are reference sites miscalled as variants (not confirmed by Sanger Sequencing) and false negatives are variant sites, miscalled as reference.d) of each point from the diagonal—“Random Guess” line. To further assess the pipeline’s performance, confusion matrices were analysed using the MCC, p, r and F1 metrics [http://bioinformatics.psb.ugent.be/webtools/Venn/).Pipelines were ranked in the receiver operating characteristic (ROC) space. False positive and true positive rates (FPR and TPR) were calculated for the entity of variant calls and plotted on the ROC space. In the ROC space, each point corresponds to an instance of a confusion matrix . The pipelines were ranked based on the perpendicular distance  statistical computing language was used for the statistical analyses presented in this manuscript. The Shapiro–Wilk’s and Levene’s tests were used to test normality and equality of variances for variables calculated for three or more groups. The one-way ANOVA parametric test was used to compare the means of homogeneous, normally distributed and independent numerical variables. The non-parametric Kruskal–Wallis test was used to compare numerical variables of three or more groups when one-way ANOVA assumptions  were not met. Post-hoc multiple comparisons were performed using the Bonferroni method [p value of less than 0.05 was considered to be statistically significant.The R (v3.3.2)  target genes in alphabetical order.Additional file 2: Table S2. EuroGentest and the European Society of Human Genetics, guidelines for the evaluation and validation of NGS applications.Additional file 3: Figure S1. Sequencing and Mapping Evaluation. a. Number of reads and clusters generated per run. b Number of unmapped reads per sample and alignment method.Additional file 4: Table S3. Number of reads per sample and mapping tool.Additional file 5: Table S4. Minimum, mean and maximum depth of coverage, per gene and alignment algorithm. Depth of coverage results upon NextSeq 500 High-Output kit 2x75 cycles and 2x150 cycles sequencing.Additional file 6: Figure S2. Hierarchical clustering of the tools. The d, MCC, p, r and F1 values were used to perform hierarchical clustering analysis based on the Lance–Williams agglomerative hierarchical clustering algorithm, which at each stage recomputes dissimilarities between clusters.Additional file 7: Figure S3. Box plot of Matthew Correlation coefficient (MCC) comparisons per alignment, variant calling or padding method. Each dot represents one observation and horizontal bold lines denote median MCC values. Boxes extend from the 25th to the 75th percentile of each group’s distribution of values. Vertical extending lines (whiskers) denote the upper and lower adjacent values. Statistical analyses were performed using the non-parametric Kruskal-Wallis or one-way ANOVA tests.Additional file 8: Figure S4. Box plot comparisons of False Positive Rates (FPR). Only statistically significant differences are shown. Each dot represents one observation and horizontal bold lines denote median FPR values. Boxes extend from the 25th to the 75th percentile of each group’s distribution of values. Vertical extending lines (whiskers) denote the upper and lower adjacent values. Statistical analysis was performed using the non-parametric Kruskal-–Wallis test.Additional file 9. Detailed commands and parameters used for data pre-processing, sequence alignment, post-alignment processing and variant discovery.


Improvements in sequencing technology continue to drive sequencing cost towards $100 per genome. However, mapping sequenced data to a reference genome remains a computationally-intensive task due to the dependence on edit distance for dealing with INDELs and mismatches introduced by sequencing. All modern aligners use seed–filter–extend methodology and rely on filtration heuristics to reduce the overhead of edit distance computation. However, filtering has inherent performance–accuracy trade-offs that limits its effectiveness.Motivated by algorithmic advances in randomized low-distortion embedding, we introduce SEE, a new methodology for developing sequence mappers and aligners. While SFE focuses on eliminating sub-optimal candidates, SEE focuses instead on identifying optimal candidates. To do so, SEE transforms the read and reference strings from edit distance regime to the Hamming regime by embedding them using a randomized algorithm, and uses Hamming distance over the embedded set to identify optimal candidates. To show that SEE performs well in practice, we present Accel-Align an SEE-based short-read sequence mapper and aligner that is 3–12As sequencing technologies continue to increase read length while improving throughput and accuracy, we believe that randomized embeddings open up new avenues for optimization that cannot be achieved by using edit distance. Thus, the techniques presented in this paper have a much broader scope as they can be used for other applications like graph alignment, multiple sequence alignment, and sequence assembly.The online version contains supplementary material available at 10.1186/s12859-021-04162-z. G as the reference genome, and a set of substrings R as the sequencing reads, the task of read alignment is to find the origin location of each read rR in G. However, due to sequencing errors, and differences between the reference genome and the sequenced organism, a read might not exactly align at any candidate location in the reference genome. Thus, an aligner has to perform approximate string matching to tolerate errors.Over the last decade, DNA sequencing technology has achieved dramatic improvements in both cost and throughput. With the $100-per-genome sequencing goal emerging as a realistic target in the near future, the amount of genomic data generated by sequencing is only poised to grow faster. The first, and often one of the most time consuming steps, in analyzing genomic datasets is sequence alignment—the process of determining the location in the reference genome of each sequencing read. Sequence alignment can be boiled down to a string matching problem. Given a string ungapped or gapped depending on whether they use Hamming or Levenshtein distance for computing mismatch between reads and the reference  with at least 0.99 probability, where N into strings of length 3N such that Hamming distance of embedded strings is at most square of the edit distance between original strings. Recent studies have demonstrated that this algorithm, which we henceforth refer to as 3N-embedding (3NE), works well in practice for performing edit similarity joins for even relatively large edit distances [The first algorithm was proposed by  who showistances .N consisting of four possible characters . The output is an embedding string of length 3N consisting of the four characters and possibly multiple repeats of a pad character (P). In each iteration, the algorithm appends a character from the input string, or the pad if it runs out of the input string, to the output string. Then, it uses a random binary bit string to decide if the input index should be advanced. The net effect of this algorithm is that some input characters appear uniquely in the output string, while others are randomly repeated multiple times. Using the theory of simple random walks, [Accel-Align uses 3NE for embedding both reference strings and the read itself. Listing 1 shows the pseudo-code for the embedding algorithm. The input string is a DNA sequence of length m walks,  establis2N-Embedding (2NE). Listing 2 shows the pseudocode for 2NE which is conceptually similar to 3NE with the exception that each character in the input string is copied to the output string at most two times. Thus, 2NE implements a mapping f: N to 2N.An initial implementation of 3NE in Accel-Align showed us early on that despite the simplicity of the algorithm, it was computationally intensive to embed billions of candidate locations across millions of reads. We describe several optimizations later in Section 2.4 that reduced the overhead of embedding, but one of the first optimizations we designed was a variant of the embedding proposed by , which wund that  had alsoembedding distance. While doing so, we dynamically keep track of the top two candidate locations with least embedding distance and forward them to the third phase for further extension, scoring, and mapping quality computation.We use one of the two embedding algorithms described above to embed all reference strings and the read. Then, we compute the Hamming distance between each embedded reference and the embedded read. We refer to this distance as the p, then the expected normalized position is P is the best candidate position, l is the INDEL length threshold we set to 5% of the read length . Finally, we take the starting position with the least embedding distance compared to the first seed of read as the final position.Before extension, Accel-Align will rectify the position of the best candidate. As mentioned in seeding, the positions are adjusted using the offset of the k-mer in the read to get normalized candidate positions. However, this approach can shift the candidate position by a few nucleotides when there are INDELs. For example, consider a read with an insert in the first seed. If the second seed produces a candidate position Accel-Align can be configured to run in alignment-free mapping mode where only the identified candidate location is reported, or full-alignment mode where base-by-base extension is performed and the CIGAR string is reported. For the mapping mode, we simply pick the best candidate location, which is the one whose reference strings has the least embedding distance, and report its normalized location. For the full-alignment mode, we perform a global extension at the best candidate location using lib-ksw  with matIn addition to the CIGAR, popular aligners also report a mapping quality (MAPQ) that represents the degree of confidence in the alignment for each read. The basic idea behind MAPQ is that if multiple candidates align to the read with comparable alignment score, the aligner cannot be confident in its final choice. However, if alignment scores between the chosen candidate and others are very different, the aligner can be reasonably confident in its choice. Thus, MAPQ computation typically requires aligners to extend at least two candidate locations so that alignment scores can be compared. SEE, in contrast, makes it possible to avoid such extensions as we can exploit embedding distance once again to compute MAPQ. If two candidates are spaced apart with respect to edit distance from a read, then the embedded reference strings will also be spaced apart with respect to Hamming distance from the embedded read. Thus, Accel-Align uses Bowtie2’s MAPQ estimation procedure  adjustedSect.2N-embedding, we implemented three other optimizations, namely, pipelining, early-stop, and prioritizing, that reduced the overhead of embedding without any change in functionality or accuracy.A naive SEE implementation would perform seeding, embedding, and extension as described so far in sequence. However, during initial experimentation we found that while embedding reduced the overhead of extension, the computational task of embedding and Hamming distance computation added non-negligible overhead. Thus, in addition to the 2NE algorithm described in Instead of embedding all candidates and then computing the Hamming distance, our first optimization is to pipeline these steps. We do this by modifying the embedding step so that the read is embedded first. Then each candidate location is embedded one by one, and the embedding algorithm simply updates the embedding distance in each iteration by comparing the output character of the candidate generated in that iteration with the corresponding character in the embedded read. This pipelining of embedding and distance computation provides three major benefits. First, as embedded strings are no longer generated in their entirety, it reduces memory consumption and associated overheads of allocating and freeing memory for storing embedded candidates. Second, it reduces the overhead caused by a needless second loop over the embedded candidates to calculate the Hamming distance. Third, as the output character generated by the algorithm is used immediately for distance computation, it improves processor cache utilization.Using pipelining to produce the embedding distance for each candidate enables us to apply the second optimization based on the observation that only the top-two candidates with the least embedding distance are selected for further extension. Thus, if we have already encountered a candidate with a very low embedding distance, there is no point in continuing the embedding process for another candidate whose distance has already exceeded the previously observed minimum. Thus, we parameterize the embedding algorithm with a threshold such that the algorithm stops embedding a candidate as soon its embedding distance exceeds the threshold. Instead of storing the embedded distance of all candidates, we dynamically track the lowest and second-lowest distances, and simply use the latter as the threshold parameter.Our third optimization is a policy that drives pipelining and early-stop mechanisms. It is based on the intuition that if candidates with low embedding distance are prioritized before others, the overall cost of embedding will be low. This is due to the fact that the threshold will be set to a relatively low value during the early stages of embedding. As a result, early-stop will be applied to most candidates. However, the embedding distance of candidates is not known to us in advance. Therefore, we use candidate counting, a technique used by SFE aligners for count filtering , to prioIn this section, we present an evaluation of Accel-Align using both simulated and real data to compare its performance and accuracy with respect to four state-of-the-art short-read aligners, namely, BWA-MEM v0.7.17) , Bowtie2 , BowtieAll experiments were run on a server equipped with a quad-core Intel(R) Core(TM) i5-7500 CPU clocked at 3.40GHz, 32GB RAM, and a 256GB SATA SSD. In an offline phase not reported here, we used each aligner to pre-index the reference genome. Then, in each alignment experiment, we ran the aligner five times and gathered execution statistics. As all aligners read the index from secondary storage, the first run is typically “cold” as data is not in memory. Table For benchmarking, we used hg37 (hs37-1kg) as the reference genome. We used Mason2  to generTable Table Figure Comparing Accel-Align with the Accel-Align-AS configuration, we see that Accel-Align-AS assigns a few more alignments (0.7%) to a higher MAPQ. As the only difference between these two configurations is the use of alignment score in Accel-Align-AS versus the embedding distance in the default version for computing MAPQ, we see that using embedding distance for MAPQ makes the aligner marginally more conservative in estimating MAPQ. Bowtie2’s MAPQ estimation procedure assigns MAPQ by comparing the best alignment score with respect to a threshold minimum score, and the second best score. We know that embedding distance computed for each candidate location is always larger than original edit distance. Thus, when embedding distance is used in MAPQ estimation, alignments that are further away from minimum with respect to alignment score end up moving closer to the minimum. This intuitively explains why MAPQ assignment is more conservative. However, it is important to note that the mapping location and difference in the overall error rate is the same for both Accel-Align and Accel-Align-AS. Thus, for the rest of this evaluation, we use the default version of Accel-Align.N into a new string of length 2N. To test the sensitivity of performance with respect to read length, we used Mason2 to generate three more paired-end datasets with 10M reads, where each dataset was configured with a read length of either 125bp, 150bp, or 175bp. Table The computational cost of the embedding step is proportional to the read length, as each read and at least one candidate are converted from length Both Accel-Align and Minimap2 can be configured to run in alignment-free mapping mode where they report the position without the CIGAR string. The mapping mode completely eliminates the overhead of edit-distance computation. Although such mapping is useful in several applications that do not require base-by-base alignment, we use it in this context to isolate the benefit of embedding. To compare Accel-Align with Minimap2 in mapping mode, we used the two aligners to perform alignment-free mapping of the four paired-end Mason2 datasets. Figure ths Fig.  with alino-embed mode where embedding is not used, and all candidate locations identified by seeding are directly forwarded for extension, (ii) the default mode using 2NE, and (iii) using 3NE instead of 2NE. Table To further isolate and understand the benefit of embedding in Accel-Align, we modified Accel-Align to run in three modes. (i) a Comparing 2NE and 3NE cases, we can see that 2NE provides a 23% improvement in performance with no discernible difference in accuracy. Comparing 2NE and the no embedding cases, we see that embedding provides a 8NO-KSW and NO-WFA modes, we do not use embedding and pass all candidates for extension with either lib-ksw or lib-wfa. In 2NE-KSW and 2NE-WFA modes, we use 2NE for embedding to select top two candidate locations, and pass them for extension with lib-ksw or lib-wfa. Table Recall that Accel-Align uses lib-ksw for extension. However, embedding is orthogonal to the type of extension technique used. To understand the effectiveness of embedding in the presence of alternate, more efficient extension methods, we incorporated the recently published Wavefront Alignment Algorithm (lib-wfa)  as an alThere are two important observations to be made from Table To evaluate the accuracy of Accel-Align on real data, we used the human whole-exome sequencing dataset NA12878 (accession No.: SRR098401). We built a pipeline similar to prior work  to detecWe compare the aligners with respect to several metrics as shown in Table To better understand the difference between detected variants, we show a Venn diagram of variants detected by various aligners in Fig. To understand the impact of k-mer size on overall accuracy, we modified Accel-Align to use 25-mers instead of 32-mers. Table In order to isolate and understand the behavior of Accel-Align in the presence of SNP and INDEL variants, we show the F-score for SNP and INDELs separately in Table In this work, we presented Accel-Align-a fast sequence mapper and aligner that uses the SEE method for quickly identifying optimal candidates based on randomized embedding. Using an extensive evaluation, we showed that Accel-Align specifically, and SEE more generally, can provide an order of magnitude improvement over some state-of-the-art aligners on commodity CPUs. We are pursing several avenues of future work. First, we are working on extending Accel-Align to support additional functionalities like soft clipping and local alignment. Currently we do not support this as neither our current embedding approach, nor lib-ksw, support local extension. In order to support soft clipping, Accel-Align cannot embed the entire read, as a position with a high embedding distance can still be the original mapping position chosen by local extension after soft clipping. Instead, we are investigating the use sub-sequences of a read between exactly-matched seeds as the embedding target.Another natural extension of this work that we are pursuing is investigating the effectiveness of SEE for long-read alignment. Research on string-similarity join has demonstrated that these randomized embeddings excel when applied to long strings with even up to 20% mismatch . This suAs sequencing technologies continue to increase read length while improving throughput and accuracy, we believe that randomized embeddings open up new avenues for optimization that cannot be achieved by using edit distance. Accel-Align clearly demonstrates the potential of using embedding for short-read sequence alignment. We believe that the benefit of SEE methodology and low-distortion embedding are not limited to short-read alignment; any computational biology problem that is limited by the overhead of edit distance can benefit from embedding. As embedding transforms strings from edit to Hamming regime, computational tools like Locality Sensitive Hashing can also be applied on the resulting strings. Thus, the techniques presented in this paper have a much broader scope as they can be used for other applications like spliced RNA-seq and bi-sulfite alignment, multiple sequence alignment, and even sequence assembly.Additional file 1. The link to the used datasets, command line to generate simulated reads, and link to repository of Accel-Align.