Being able to identify the genetic variants responsible for specific bacterial phenotypes has been the goal of bacterial genetics since its inception and is fundamental to our current level of understanding of bacteria. This identification has been based primarily on painstaking experimentation, but the availability of large data sets of whole genomes with associated phenotype metadata promises to revolutionize this approach, not least for important clinical phenotypes that are not amenable to laboratory analysis. These models of phenotype-genotype association can in the future be used for rapid prediction of clinically important phenotypes such as antibiotic resistance and virulence by rapid-turnaround or point-of-care tests. However, despite much effort being put into adapting genome-wide association study (GWAS) approaches to cope with bacterium-specific problems, such as strong population structure and horizontal gene exchange, current approaches are not yet optimal. We describe a method that advances methodology for both association and generation of portable prediction models. Discovery of genetic variants underlying bacterial phenotypes and the prediction of phenotypes such as antibiotic resistance are fundamental tasks in bacterial genomics. Genome-wide association study (GWAS) methods have been applied to study these relations, but the plastic nature of bacterial genomes and the clonal structure of bacterial populations creates challenges. We introduce an alignment-free method which finds sets of loci associated with bacterial phenotypes, quantifies the total effect of genetics on the phenotype, and allows accurate phenotype prediction, all within a single computationally scalable joint modeling framework. Genetic variants covering the entire pangenome are compactly represented by extended DNA sequence words known as unitigs, and model fitting is achieved using elastic net penalization, an extension of standard multiple regression. Using an extensive set of state-of-the-art bacterial population genomic data sets, we demonstrate that our approach performs accurate phenotype prediction, comparable to popular machine learning methods, while retaining both interpretability and computational efficiency. Compared to those of previous approaches, which test each genotype-phenotype association separately for each variant and apply a significance threshold, the variants selected by our joint modeling approach overlap substantially. Such–P values in a GWAS.Large cohorts of bacterial sequences have also been a tempting target for machine learning and “deep learning” methods such as convolutional neural networks, which are able to relate arbitrary high-dimensional inputs to measured outputs with high accuracy and without the need for specialized model descriptions for each new problem . Rather However, some issues crucial to understanding bacterial populations remain unaddressed. First, bacterial populations tend to exhibit a strong population structure, meaning samples cannot be treated as independent. In the context of prediction, this can result in the selection of features unrelated to the phenotype but common to the background of associated strains (lineage effects). While not necessarily a problem in the training data set, if new data are drawn from different strains, this can lead to much poorer prediction than expected. Examples of this effect are ubiquitous in artificial intelligence, for example, automatically designed clocks which only work in the lab they were built , 23. In Here, we set out to develop a method which combines the desirable attributes of both of these classes of approaches when analyzing the genetics underlying bacterial traits. We wished to retain the simplicity and interpretability of traditional GWAS approaches and combine this with the flexibility and accuracy of machine learning methods which can be fitted to the entire data set at once. These models were previously applied to human GWAS data sets for inference –27 and p–This pangenome-wide approach reflects the polygenic nature of complex traits better than older fixed effect methods which must select only some population structure covariates to include . Unlike 30–Using large genomic data sets from four different species and sixteen varied phenotypes, we find that an elastic net model  selects https://pyseer.readthedocs.io/en/master/predict.html).Our approach models the entire pangenome of the population to include the large proportion of variation which resides in the accessory genome. It explicitly addresses issues of population structure and consistent performance between trained and new (test) data sets. The method is broadly usable, not requiring programming knowledge or manual adjustments for new data sets, and allows for the sharing of models between researchers. We have implemented the elastic net model in the pyseer microbial package as a new “prediction” module and consistent pangenome variant calling in two further packages. An extensive tutorial for all of these methods is available online  of the phenotype.The elastic net uses a linear prediction model as with a standard linear regression run between a phenotype and all genetic variants and tries to find slopes (effect sizes) for all the variants which best predict the phenotype. A shrinkage term (λ) is used to prevent overfitting, adding a cost to each fitted slope proportional to its value. This has the effect of making many of the genetic variants have a slope of zero, and so they can be removed from the model entirely. To predict the phenotype in new samples, these fitted slopes form a simple linear model that can be applied to a new set of input variants. Furthermore, this also allows the calculation of We use an alignment-free representation variation as input. Alignment-free approaches have proven particularly popular in bacterial populations, removing the need for selection of a particular reference and simultaneously modeling both gene content and sequence variation , 12, 35.P value from GWAS and must be considered along with minor allele frequency (MAF) and other confounders. To include a population structure explicitly in the model, we first divide the population into strains (or lineages). Each sample’s contribution to the fitting is then downweighted by the prevalence of the strain in the elastic net so that repeated observations of the same genotype count for less, known as “sequence reweighting.” We also use this to select the value shrinkage term λ. We designed a “leave-one-strain-out” (LOSO) cross-validation rather than randomly leaving samples out. This aims to avoid correctly predicting the strain (which is frequently correlated with phenotype) rather than the phenotype itself, since such an approach is less robust when common strains dominate the data or when the fitted population model is not representative of a population in which predictions will be made. When applied together, we refer to this as a “weighted” model, as opposed to “unweighted” models which use neither of these adjustments.As the elastic net includes predictors that are correlated due to population structure in the same model fitting, it will typically downweight these, to some extent incorporating this evolutionary history into its model. However, as we note below, effect size alone does not have the same interpretation as a Whole-genome models can be used to construct a linear model to predict phenotypes in new data. In this section, we evaluate these predictions compared to those of other models and variant calling methods using a variety of data sets and phenotypes.Mycobacterium tuberculosis resistance is equally well predicted by the elastic net.(i)  We first evaluated the predictive performance of our models, with and without population structure control, compared to that of a more complex deep learning model. We used an M. tuberculosis data set with antibiotic resistance to four first-line antibiotics . As M. tuberculosis has no accessory genome and minimal core gene variation  Prediction of pneumococcal resistance using different variant types. We also investigated the advantages of the use of unitigs over other variant calling methods. Using the same Streptococcus pneumoniae antimicrobial resistance in children (SPARC) data set described above for β-lactam and erythromycin resistance, we compared computational resources and prediction accuracy using SNPs, k-mers, and unitigs . For common variants, M reaches an asymptote for a given population: the main requirement is therefore based on N. For all methods, the CPU time was modest, but memory usage may pose a problem. SNPs are tractable on a laptop, but unitig analysis likely requires a computing cluster for the model fitting (using a fitted model on test data requires negligible resources). k-mers require an enormous amount of memory, which would not scale to larger data sets. Though the unitig analysis was easy to schedule on our cluster, future improvements to reduce memory use could include accessing the variants as they are needed from a disk or fitting the elastic net in chunks, with resampling  Reduced intercohort accuracy is ameliorated with consistent genetic calls and population structure control. Random splits of single data sets in test and training data, while convenient for analysis, may mask inter-data set differences such as class imbalance (different resistance rates), unobserved lineages, and technical errors   . To testcalling) . The res10.1128/mBio.01344-20.1FIG S1S. pneumoniae data sets. Minimum unitig size is by definition the de Bruijn graph k-mer size, which was chosen to be 31 for all datasets. Download FIG S1, PDF file, 0.4 MB.Histograms of the unitig length distributions in Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the R2, 0.439), this gave very similar results to those using mean allele frequency [AF] imputation for missing unitigs .Between-cohort predictive accuracy was considerably lower than within-cohort accuracy but still outperformed an intercept-only model in all cases. The use of unitigs proved successful: repeating the SPARC-Maela comparisons with SNPs led to extremely poor predictions for every sample, as the selected SNPs were called as missing in the other cohort, leading to a mean value prediction for every sample . To fix this issue with SNPs would likely require a labor-intensive mapping and joint recalling of variation, whereas with sequence elements, the simple search implemented in unitig-caller can be used. To deal with missing unitig data, which may truly be missing or miscalled, we assumed it was all truly missing. On the same SPARC-Maela comparison  of the receiver operating characteristic (ROC) is misleadingly high (0.9185/0.9728 for the weighted/unweighted GPS model) and would encourage the reporting of error rates as more intuitive summaries of accuracy for bacterial traits such as resistance.R2 was 72% higher with sequence reweighting versus 82% higher without sequence reweighting), and more of the selected variants were in the causal loci.We found that sequence reweighting generally reduced prediction accuracy for this phenotype, although it is the LOSO strategy in particular which gave slightly more representative accuracy estimates for out-of-cohort prediction  Virulence phenotypes can be predicted with sequence reweighting, preventing overestimation of accuracy. Most work on prediction of bacterial phenotypes has focused on antibiotic resistance, but many more complex phenotypes relating to bacterial virulence are now available. For these phenotypes, which are under weaker or no selection, instead of a few strong effects, multiple smaller effects are expected in the genome  was 0.89 —these may be overestimates, especially as the revised estimates introduced here used more information from the genome.We applied our method to predict the duration of asymptomatic carriage in a subset of the Maela cohort, which can easily be visualized in the manner of a linear regression . We shownly 0.27 , right. Streptococcus pneumoniae isolated from Dutch adults, we fitted a model which selected 9,701 unitigs. This model was able to predict meningitis versus carriage genomes  and gave a similar h2 estimate to that originally reported (0.65 versus 0.70). Comparing tissue infection with carriage of Streptococcus pyogenes gave a model with 5,817 unitigs, which had a higher error rate than the S. pneumoniae model , but the phenotype also had a correspondingly lower h2 estimate of 0.343. Detailed performance of these models is given in We also tested virulence prediction in two streptococcal species, which would be a useful application for routine pathogen surveillance but has not been as thoroughly explored as resistance prediction. Using 10.1128/mBio.01344-20.8TABLE S2S. pneumoniae. Download Table S2, PDF file, 0.1 MB.Prediction accuracy of the elastic net in other binary datasets, all of which are Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the S. pneumoniae genomes)   using prgenomes) . We chosR2 > 0.9).chosen uniformly at random across the genome, after linkage disequilibrium (LD) pruning .chosen uniformly at random from 1 to 3 prespecified genes , true negatives (TN), false positives (FP), and false negatives (FN), we calculated the power, the proportion of true causal variants in the output: TP/(TP + FN). This allowed us to analyze the overlap of selected variables which gave good prediction with those found in a GWAS, which are individually associated with a phenotype of interest and may therefore provide mechanistic insight into the trait being analyzed. We also calculated the proportion of false positives in the output, the number of variants selected in the output which are not true causal variants divided by the total number of variants being tested: FP/(TP + FP + FN + TN). This is a measure of the “noise” in the selected predictor set.We chose the first setting to emulate a polygenic trait, with many variants of roughly equal effects associated across the genome. LD pruning was only used to select causal variants, and all variants were used as input to the model. The second setting more closely resembles antibiotic resistance, where multiple alleles in either one or a small number of genes contribute to the effect, with multiple occurrences independent of genetic background. We ran the elastic net (α = 0.01) and lasso regression (α = 1) as well as both GWAS models (fixed effects and linear mixed model) previously implemented in pyseer. Variants were output by the genome-wide model if they had a nonzero coefficient and by the GWAS models if their First, our simulations were able to show that using the correlation filtering step (reducing input size by 25%) reduced power on average by 4% and 8% in the worst case see , where m10.1128/mBio.01344-20.9TABLE S3pbpX. (c) Varied sample sizes, variants from both regions. (d) Varied heritability, variants from both regions. Download Table S3, PDF file, 0.1 MB.True causal variants retained after filtering on sample correlation. In parenthesis is the total number of the retained variants. (a) Varied sample sizes, uncorrelated true variants are chosen from across the genome. (b) Varied sample sizes, variants chosen from Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the Over all of our simulations, we found that either the elastic net or a fixed-effect GWAS had the highest power depending on the setting, and both always had higher power than the linear mixed model . This is10.1128/mBio.01344-20.2FIG S2pbpX gene, sample size was varied. (b) Different numbers of true variants were chosen from across the genome, sample size was varied. (c) Fifty true variants were chosen from different set ups: LD-pruned variants across the genome, and the genes pbpX, pbp1a and penA. Sample size was varied. (d) Fifty true variants were chosen from different set ups: LD-pruned variants across the genome, pbpX, pbp1a, and penA. Sample size was varied. (e) Varied heritability of a continuous phenotype. Fifty true variants were chosen from one or two genes. Sample size was fixed at 3,000. (f) Varied heritability of a binary phenotype. Fifty true variants were chosen from one, two, or three genes. Sample size was fixed at 3,000. Download FIG S2, PDF file, 1.0 MB.Power and false-positive rates of GWAS and elastic net regression methods. (a) Different numbers of true variants were chosen from the Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the P value from a GWAS is important. It is possible to do this with least angle regression . As the number of causal variants is generally not known These results show that the variants selected by the elastic net are causal at similar rates to those with GWAS methods. The elastic net’s selected variants can be used as an effective trade-off between the regimes of the two commonly used GWAS models, having higher power than the linear mixed model and a lower false-positive rate than fixed-effect models. As many bacterial GWAS results must be followed up with lab work, these results suggest a dual approach of variable selection with the elastic net followed by ranking results with the linear mixed model may be useful when the first set of variables selected is large. This is possible in a single step in pyseer.(i) Variant selection for pneumococcal antimicrobial resistances. Next, we tested whether our method selects causal variants for some phenotypes where these are known and compared our results to those from GWAS approaches. First, we analyzed a well-studied phenotype and data set: sensitivity/resistance to β-lactams in the SPARC cohort of 603 S. pneumoniae genomes. Resistance is mainly conferred by allelic variation of three genes , which are easily detected by most GWAS and machine learning methods with SNP calls as input  with this strongly selected phenotype; only a few noncausal genes are included, usually with only a few SNPs and much lower ranking than the causal genes. The elastic net selects many more noncausal SNPs across the MAF spectrum, though combining with P values and number of hits allows these to be effectively filtered. It should be noted that effect size does not appear to be an effective filtering criterion without taking into account minor allele frequency, which may have implications for other machine learning methods where a P value cannot easily be integrated. Both methods calculate comparable heritability estimates for this trait, for the LMM, h2 = 0.89, and for the elastic net, h2 = 0.81.S. pneumoniae genomes). We previously attempted GWAS on this data set using a fixed-effect GWAS model in the original description of our SEER software (folA-dyr and folP) due to being administered jointly with sulfamethoxazole as co-trimoxazole  which were not easily found in our previous attempt. Again, the erythromycin results contained many peaks in their Manhattan plots , mel , and mef . While this was clearly more successful than our previous analysis, when considering the noise when mapping to a single reference, these causal mechanisms would not stand out  Using sequence reweighting. (b) Without sequence reweighting. Download FIG S3, PDF file, 1.0 MB.Manhattan plots for trimethoprim resistance in Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the 10.1128/mBio.01344-20.4FIG S4S. pneumoniae. Plotted variants are unitigs selected by the elastic net using sequence reweighting; P values are from the LMM. (a) Using sequence reweighting. (b) Without sequence reweighting. Download FIG S4, PDF file, 1.5 MB.Manhattan plots for erythromycin resistance in Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the 10.1128/mBio.01344-20.5FIG S5x axis is the average effect size (beta) of unitigs covering the locus, y axis is the minimum P value of any unitig in the locus; size relates to the total number of unitigs mapped to the gene, color is the average MAF of the mapped unitigs. (a) Using sequence reweighting. (b) Without sequence reweighting. Download FIG S5, PDF file, 0.7 MB.Summary of genes with overlapping selected unitigs in the weighted GWAS. Each point is a gene, Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the (ii) Heritability and mapping of gonococcal resistance. We also applied our method to a combined cohort of 1,595 Neisseria gonorrhoeae genomes where resistance to five different antibiotics has been measured. These data were previously used to do GWAS using an LMM, with selected loci then entering a reduced dimension epistasis analysis  had 4,612 unitigs selected, with the top hits mapping to the four 23S rRNA sequences in the genome. The original analysis only identified a SNP in one of these repeated rRNA sequences, likely due to the impossibility of mapping variation in these repeats at a single base level: this is an advantage of being able to report multiple mappings of sequences at the final stage. Cefixime (CFX) identified the penA region, as in the original analysis, and also suggested an association in the promoter of opaD. Ciprofloxacin (CIPRO) had hits throughout the genome, as in the original analysis and similar to the analysis of erythromycin described above—combining the LMM with the elastic net may reduce candidate regions in these cases. Penicillin (PEN) had a hit in the porB region, as in the original analysis, along with hits in lgtE, mexB (and efflux pump), and a prophage. Tetracycline (TET) similarly had a replicated hit in the porB region, along with the cysN promoter, an alternative pilE allele, and rsmE . These Manhattan plots can be seen in blaTEM and tetM, causal for PEN and TET, respectively, to which we can also map unitigs rather than needing to recall variation with respect to this reference panel. This confirms further hits to these genes. Our method therefore broadly replicated the results from the LMM and added new candidate hits due to testing unitigs rather than just SNPs, as expected.The mapping of resistance genes for these antibiotics using our approach was similar to this GWAS. The original analysis looked at ∼8,700 SNPs with a MAF of >0.5%; we used 5.3 × 1010.1128/mBio.01344-20.6FIG S6N. gonorrhoeae. Plotted variants are unitigs selected by the elastic net without sequence reweighting; P values are from the LMM. (a) Azithromycin. (b) Cefixime. (c) Ciprofloxacin. (d) Penicillin. (e) Tetracycline. Download FIG S6, PDF file, 2.6 MB.Manhattan plots for antimicrobial resistances in Copyright © 2020 Lees et al.2020Lees et al.Creative Commons Attribution 4.0 International license.This content is distributed under the terms of the h2 using our elastic net and unitig method and compared these to those calculated with previous methods . Separately, using 3,000 samples, h2 was varied at 0.1, 0.3, 0.6, and 0.9. We did not use fewer than 500 samples, as we expect a significant decrease in discovery power , l is the link function , λ > 0 is the magnitude of the penalty, and 0 < α < 1 is the amount of mixing between L1 and L2 penalties. Minimizing this function reduces the squared distance between predicted and observed values when λ = 0; as λ increases, predictors are shrunk toward zero to trade-off prediction accuracy with overfitting.We use a high-dimensional regression model which includes all genetic variants and covariates of interest function , 65, mixGiven the strong linkage disequilibrium present in bacterial populations , 66, manR2 value of the model over the test data. The user can change the number of folds. The amount of mixing (α) between L1 penalties, which lead to sparse predictive models with mostly zero valued slopes, and L2 penalties, which lead to models with shrunken but nonzero predictors, could also be chosen by cross-validation to maximize prediction accuracy. However, we propose users choose a value of around 0.01 throughout . A sparse model is preferred for prediction for speed and easier consistency between populations, but this leads to a loss of power in the context of GWAS, as potentially causally related predictors may be removed from the model  is generally insufficient to capture all of the variation within the samples. In particular, accessory gene content has been shown to be associated with core variation , be assoS. pneumoniae data sets, we observed a 13- to 18-fold reduction in the number of unitigs compared to that of k-mers.Early bacterial GWAS methods used k-mers, sequence words of fixed or variable length, to assay variation throughout the population independent of gene annotation or variant calling method , 12, 35.y and x (defined as phenotype and variant as above) such thatWe used two techniques to circumvent this issue while still including as much pangenomic variation as possible. Following the idea of screening methods in ultrahigh-dimensional data , 73, we https://github.com/johnlees/unitig-counter), also including tools to extend unitigs by traversing neighboring nodes in the graph, and calculate distances between unitigs based on the graph using Dijkstra’s algorithm.We also followed the method used in DBGWAS , which awi, or by changing the folds used in cross-validation. We discuss each of these in turn.Population structure causes correlations between the genetic variants (unitigs) that make up Fixed-effect models typically use a principal components-type analysis to include the main axes of variation in the population as covariates. In a new data set, projection of variation onto these existing axes could be used but would require large overlap between variant calls in each data set to be accurate. Random-effect models use a kinship matrix to include the sample covariance matrix in each association. For a new data set, this would require calculation of covariance against the original data set, which reduces model portability.We therefore opted to use a definition of population structure which does not introduce extra predictors. This makes application of the model more straightforward in new data sets. The use of a definition of clusters which naturally extends to new populations, which may have very different strain frequencies and/or large numbers of novel clusters, further increases robustness in the face of between-data set variation. Any method which produces discrete cluster membership definitions independent of cluster frequency is suitable for this purpose, such as sequence type, clonal complex, or percentage identity cutoff. We opted to use the “strain” definition provided by the PopPUNK software throughout our analysis due to its speed and biological basis  However,i(x), which is 1 if x is in cluster i or 0 otherwise. The model is fit to all the data except the first cluster, and then its accuracy is tested on the cluster not used in the fitting. This is repeated for every cluster to find the value of λ which maximizes fit accuracy over all these fits. This may be referred to as leave-one-strain-out (LOSO) cross-validation or leave-one-cluster-out (LOCO) cross-validation. This is more realistic than random selection of folds, as random samples would maintain relative strain frequencies between training and test data, whereas new populations usually vary greatly in their genetic background .Prediction of unobserved phenotypes ix = 0) or imputed  in C++ using the SeqAn library , led to reports in the top decile for almost all data sets and methods and were harder to intuitively compare between different models and data sets. We therefore report the false-negative rate and false-positive rate along with the totals selected.For continuous phenotypes, we report R2 value, as defined above.We implemented the association model in version 1.3.2 of the pyseer bacterial GWAS package, which is written in python . This taVariant matrices are potentially very large, and so to optimize speed and memory use, we read these into a sparse matrix structure. Variants with allele frequency of >50% have their genotypes flipped to increase sparsity—these sites are flipped back during prediction. This sparse structure can be saved to disk to avoid repeated parsing of variant input files. Only haploid variant calls (0/1), allele frequency, and sample order are saved in this file. After extracting the nonzero coefficients from the fitted elastic net, the input variant file is reread with minimal parsing to output information about the selected variants. A SHA256 hash of the input file is calculated to ensure consistency with the original input file.The fitted models are saved as an associative array, with variant names as keys  and allele frequencies and fitted slopes as values. New variant call files are read with minimal parsing to extract just those sites which appear in the model, and at the end-of-file, the appropriate imputation procedure is applied to model terms which were not found. This allows both rapid prediction in large new data sets and an easy and portable way to share predictive models.P values (adjusted by any of pyseer’s other models), and allele frequencies are included in the output. Where the true phenotype is known, prediction accuracy is reported using R2 and a confusion matrix if the phenotype is binary. If clusters were provided, accuracy statistics for within each cluster are also included in the output.For fitted models, slopes, Streptococcus pneumoniae ATCC 700669 [Neisseria gonorrhoeae FA1090 [We used the tools within pyseer to interpret unitigs. Specifically, we used bwa-mem with the shortest possible seed to map unitig sequences to a single reference genome . Bonferhttps://pyseer.readthedocs.io/en/master/predict.html).The new code in pyseer includes automated tests and unit tests we wrote using test data distributed with the package. Documentation and a tutorial are available online , documented on readthedocs (http://pyseer.readthedocs.io/), and available for install on bioconda (https://anaconda.org/bioconda/pyseer). The unitig-counter package is available at the GitHub repository  and available for install on bioconda (https://anaconda.org/bioconda/unitig-counter). The unitig-caller package is available at the GitHub repository  and available for install on bioconda .The pyseer package is available as source code at the GitHub repository (


The implementation is focused on uses common in experimental psychology. An arbitrary number of within-subject factors, each with an arbitrary number of levels, can be used. A non-parametric, randomization- and permutation-based formulation of repeated measures ANOVA was defined and implemented. Methods for testing interactions with categorical and continuous between-subject variables are implemented. Post-hoc tests for exploring interactions are automated. Simulations indicate correct control of false positive rate for all types of test. The software provides output with statistics including   Specifications Tablehttps://github.com/thomasgladwin/teg_RMA. The method has been used, e.g., in psychological studies on anticipatory attentional bias and trial-to-trial history effects on attentional bias variability The primary overall aim of the method is to take a matrix of observations, with within-subject factors nested within each other over columns and independent observations on different rows, and a specification of levels of factors, and provide a test of all main effects and interactions. The test can be parametric, via linear regression and Greenhouse-Geisser correction, or non-parametric, via permutation tests. The implementation is in Matlab teg_RMA_create_ANOVA_dummy. A vector levels is specified that contains, for every factor, the number of levels in that factor. All combinations of levels are generated in a combinations-matrix, with one combination per row. For instance, if the first factor has two levels and the second factor has three levels, then the combinations-matrix will be . The effect coding matrices for the main effects are subsequently created based on the combinations-matrix. For a 2 × 3 design, these will be [−1 −1 −1 1 1 1]’ for the 2-level factor and ’ for the 3-level factor. Note that this representation assumes a nesting of the second factor within the first factor, which must be followed by the columns of the data matrix. The process for generating the effect coding matrices is: for each level n after the first, a contrast vector is added to the matrix in which the first level is coded as −1, level n is coded as 1, and all other levels are coded as 0. These effect coding matrices provide a basis for all linear contrasts involving the levels of a factor.The first step of the method is to translate an arbitrary number of factors, each with an arbitrary number of levels, into sets of {−1, 0, 1} effect coding matrices to be used for testing. This is done in the function tuple”-way interactions, from 2-way to N-way, where N is the number of factors. For each tuple, a combinations-matrix is again created, now representing all possible combinations of tuple factors. For instance, if there are 5 factors, there are 10 possible 3-way interactions . For each specific interaction, all combinations of the contrast vectors representing the levels of the involved factors are stored in a combinations-matrix. For instance, for an interaction between a 2-level and a 3-level factor, the combinations-matrix would beSubsequently, similar effect coding matrices are created for all “The 1′s in the first column represent the single [−1 −1 −1 1 1 1]’ contrast vector of factor 1. The 1 and 2 in the second column represent the two contrast vectors of factor 2. ’. An effect coding matrix for the interaction is created that contains, for each row of the combinations-matrix, the elementwise-multiplied products of the indicated contrast vectors. In the 2 × 3 example, the 2-way interaction would thus be represented as the effect coding matrixteg_RMA_ANOVA. For each test, the first step is to create a reduced data matrix which only reflects variability due to the combinations of levels relevant to the effect. For instance, when testing the main effect of the 2-level factor in a 2 × 3 design, the raw data matrix with six columns of independently varying values has its data “reduced” to two sets of three columns within which the values per row are the same, and represent the centered difference scores between mean of the last three columns and the mean of the first three columns. This reduction is done by performing a regression separately for each centered  row of the raw data. The vector of scores per row serve as observations, with the effect coding matrix being used as the set of predictors, and the predicted vector replaces the raw data. Thus, variance over cells unrelated to the current test is removed. For instance, for a 2-level factor, each row will have columns consisting of plus and minus some value. This procedure is illustrated in the following example, for one line from a raw data matrix containing the data of a 3 × 2 within-subject design, for which the main effect of the second factor is to be tested:Subsequently, the effect coding matrices are used, together with the data matrix, to test each main effect and interaction as follows. The core function for this is N participants, the first N values of the vectorized effect coding matrix will be −1, and the second N values will be +1.Subsequently, the reduced data matrix is vectorized, each column placed under each other. The vectors of the effect coding matrix are repeated for every participant, so that the −1 / 0 / 1 elements are matched correctly to the vectorized reduced data matrix. For instance, for a 2-level factor and p-value of the tested effect via the F-test of the ratio of the mean square model (the sum of squares of the regression prediction divided by the model degrees of freedom) and mean square error (the sum of the squares of the regression error divided by the error degrees of freedom). The model degrees of freedom are the number of predictor columns, and the error degrees of freedom are (N – 1) * the number of predictor columns, where N is the number of participants. Partial eta squared is calculated as the model sum of squares divided by the sum of the model and error sum of squares Finally, all columns are centered, and the data vector is regressed onto the effect coding vectors . If there is only one degree of freedom , the regression test is performed as normal. This results in the L, of the covariance matrix of the reduced data matrix is calculated using Matlab's eig function. Epilson is the square of the sum of the eigenvalues divided by the product of the number of non-zero eigenvalues and the sum of the squared eigenvalues:If the effect coding matrix consists of multiple columns , Greenhouse-Geisser correction is performed using previously described calculations F-test. We note that there are alternative ways to correct for sphericity such as Huynh-Feldt correction The model degrees of freedom and error degrees of freedom were multiplied by epsilon before performing the N-way interactions, follow-up tests are performed in two ways, both aimed at providing a descriptive understanding of what caused a statistically significant interaction. First, the interaction is systematically explored by “unpacking” the interaction: all lower-order, (N-1)-way interactions are performed for each level of the final factor of the interaction, recursively until a single factor is being tested for a specific combination of levels of the other factors involved in the interaction. Second, pairwise t-tests between all conditions defined by all combinations of levels of the factors involved in the higher-order interaction are performed. Because these sets of tests are “protected” by a significant interaction effect and are considered here to have a descriptive purpose not served by losing statistical power, they are not corrected for multiple testing.For higher-order, In the above formulation of the analysis, all tests are performed per effect, independently from all other tests. Accordingly, addition of a between-subject variable as specified below does not affect the results of within-subject tests. Note that this approach to tests involving between-subject effects is dissimilar to, e.g., SPSS, and will give slightly different results. The current effect-wise approach was considered preferable for certain experimental goals: first, to remove fluctuations in results that depend on the choice of included variables , and second, because this approach is conceptually clearly close to common research questions about whether the size of a specific contrast is associated with individual differences rather than the traditional focus on noise-reduction of ANCOVA.k groups to (N – k) * the number of within-subject predictor columns.For between-group effects, the between-group factors are transformed into effect coding matrices in the same way as the within-group factors. Interactions with within-subject effects are tested by multiplying the elements of the vectorized effect coding matrix for within-subject effects with the centered contrast scores representing the contrast vectors of the between-group effect coding matrix. Note that centering is essential to avoid group differences from inducing false interaction effects in the presence of a within-subject effect. All resultant pairs of multiplied vectors are used for the final regression, in the same way as for within-subject effects, adjusting the degrees of freedom error given cv, in a line-by-line way, all participants’ effect coding values were multiplied with their respective centered cv value. The same test as for the within-subject effect was then performed to determine whether the within-subject effect depends on the cv. Correlations between the cv and both pairwise contrasts and individual conditions are reported to explore the cause of a significant result involving an interaction term.To test interactions with a continuous between-subject variable, A non-parametric version of the repeated measures ANOVA was implemented as follows. The method is followed as above, up to the final regression. Then, instead of calculating statistics as above, for a user-specified number of iterations  the direction of within-subject effects is randomly, with 50% chance, reversed per participant, thereby preserving the dependencies between columns. This generates samples from a null hypothesis in which the positive versus negative direction of the effect per condition is exchangeable, providing a test of whether the consistency over participants of effects in the observed data is unlikely under this null hypothesis. For interactions with continuous variables, instead of this random reversal of effects, the individual scores in the cv are randomly permuted, removing any systematic relationship between the continuous variable and the within-subject effects. For each iteration the regression is performed and the F-values are stored. This provides a null-hypothesis distribution of F-values. The p-value is defined as the proportion of F-values in the null distribution equal to or greater than the F-value of the unpermuted, observed data.For further information on such randomization- and permutation-based testing procedures, please see, e.g., Hooton The current paper describes an implementation of an effect coding approach to repeated measures ANOVA, with independent effect-wise tests, automated drilling-down into interactions, and a non-parametric permutation test for when data do not have an approximately normal distribution. The paper is hoped to be of use to researchers interested either in automated analyses or in open-source code that exposes the details of calculations for understanding or adjustments.p-value, although an effect size is also produced . Further, the primary interest in the relationship between within-subject effects and between-subject variables was in terms of separate effects; adding between-subject variables should not influence other tests. This led to a “line-by-line” analysis, in which each test was done via a regression analysis specifically generated for that test. This also allowed for a formulation of the method that can be broken down and understood precisely by reasonably statistically informed researchers or students, in terms of effect coding and regression. It also allows variations and improvements to be made where desired or necessary, by breaking in to the method at a point at which all the information necessary for the test is provided, thus possibly efficiently re-using the logistics of organizing data and creating effect coding matrices. It is also envisioned that the software might have educational purposes for students or researchers interested in the “behind the scenes” machinery and computations of within-subject analyses.Within-subject effects and interactions involving them are common in, e.g., experimental psychology. It is often the case that researchers wish to work from the perspective of null hypothesis significance testing (NHST). Despite its limitations, NHST plays a valid role in experimental research Simulations were performed to check whether the implementation correctly recovered simulated effects in the data and controlled for false positives. Initially, ad-hoc simulations were run on a range of designs, with multiple within- and between-subject factors and continuous variables. Within-subject and continuous scores were drawn from a normal distribution (Matlab's randn function). Random between-group assignment was done using the floor function on multiplications of values drawn from the uniform distribution (using the rand function). Dependency between within-subject conditions was created by creating a matrix of random noise, in which for a random selection of half the columns, the column was replaced by the weighted sum of (1 – dep) times itself and dep times the mean of the other columns, the scaling factor dep being the dependency factor. Both the parametric and permutation tests consistently controlled the false positive rate and recovered simulated effects as expected. More simulations were run to check whether false positive rates were controlled for the parametric tests, for a design with 3 × 4 level within-subject factors, 3 × 3 level between-subject factors, and a single continuous between-subject variable. Dependency factors were varied from 0 to 1 in steps of 0.1. For each dependency factor, 25,000 iterations were performed, and the p-values of all tests were stored. The mean p-values over all iterations were used to indicate whether false positive rate was controlled. Over all dependencies and all effects, the worst-case p-value was 0.054, indicating that the false positive rate was controlled for all types of effect. Visual inspection did not suggest any pattern over the p-values related to the degree of dependency.The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.


Triticum aestivum L.) lines was evaluated to compare single- and multi-trait models for predicting grain yield (GY), harvest index (HI), spike fertility (SF), and thousand grain weight (TGW). The panel was phenotyped in two locations and two years in Florida under drought and moderately drought stress conditions, while the genotyping was performed using 27,957 genotyping-by-sequencing (GBS) single nucleotide polymorphism (SNP) makers. Five predictive models including Multi-environment Genomic Best Linear Unbiased Predictor (MGBLUP), Bayesian Multi-trait Multi-environment (BMTME), Bayesian Multi-output Regressor Stacking (BMORS), Single-trait Multi-environment Deep Learning (SMDL), and Multi-trait Multi-environment Deep Learning (MMDL) were compared. Across environments, the multi-trait statistical model (BMTME) was superior to the multi-trait DL model for prediction accuracy in most scenarios, but the DL models were comparable to the statistical models for response to selection. The multi-trait model also showed 5 to 22% more genetic gain compared to the single-trait model across environment reflected by the response to selection. Overall, these results suggest that multi-trait genomic prediction can be an efficient strategy for economically important yield component related traits in soft wheat.The performance of genomic prediction (GP) on genetically correlated traits can be improved through an interdependence multi-trait model under a multi-environment context. In this study, a panel of 237 soft facultative wheat ( From 2007 to 2050, farmers will need to increase the production of cereals by 60% to feed over 9.5 billion people in the world . MeanwhiCompared to traditional marker-assisted selection, GP does not require prior knowledge about a few, large-effect quantitative trait loci, since all genotypic markers are curated in training prediction models . Essentip) is much larger than the number of observations (n) due to the improved availability of genotypic data compared to phenotypic. As such, penalized GP models such as ridge-regression best linear unbiased prediction (rrBLUP), least absolute shrinkage and selection operator (LASSO), and elastic net are employed to control the trade-offs between lack of fit and model complexity [A key component in GP is the choice and optimization of models that are used to estimate the marker effect. Statistical models with different capacities are required to handle the ever-growing magnitude of phenotypic and genotypic data. The number of predictor variables  and wheat data, which showed higher prediction accuracies for NNs when the genotypic by environmental (G × E) effect was ignored, while lower prediction accuracies were observed when G×E effect was involved. In addition to model selection, the genetic structure of the trait, marker density, sample size, and composition of training population (TP) and validation population (VP) are also important factors in GP accuracy [A few multivariate and/or multi-environment predictive models have been proposed for binary, ordinal, count, and continuous traits. Several studies applied a Bayesian-based model for multi-trait analysis and observed improved accuracies compared to single-trait analysis ,19,21. Baccuracy ,39,40,41Yield component traits such as harvest index (HI), spike fertility (SF: ratio of grain number per spike to chaff weight per spike), and thousand grain weight (TGW) play important roles in the determination of grain yield (GY) in wheat. Strong genetic correlations were observed among GY, HI, SF, and TGW under different environmental conditions ,44,45,46The experiment was conducted over two growing seasons from 2016 to 2018 at Citra and Quincy, Florida . Citra i−1. The GAWN panel was planted in an incomplete block augmented design with repeated checks  in each block with 237 unreplicated new entries [The genetic material used for the study consisted of 240 (237 + 3 checks) facultative soft wheat genotypes selected from the Gulf Atlantic Wheat Nursery (GAWN). The genotypes were developed by public wheat breeding programs  targeting for the south and southeastern regions of the USA. The panel is referred to as GAWN panel. The genotypes in the panel generally require a short duration of cold treatment to satisfy the vernalization for flower induction. The panel was evaluated at two locations in Florida: Citra and Quincy during 2016–2018 (two years). To induce terminal drought stress at Citra, irrigation was stopped 2 weeks before anthesis (GS60) until maturity. Contrary to that, 1–2 supplemental irrigations were applied when needed in Quincy. Citra is considered to be a drought-stressed environment, and Quincy is moderately drought stressed. All trials were planted in six-row plots (3 m length × 1.5 m width) using a seeding rate of 100 kg h entries . Three r−1. To measure SF, ten random spikes were sampled from the field at physiological maturity, dried for 72 h at 60 °C, and threshed by a single head thresher to determine chaff weight (the non-grain part of a spike), which was calculated as the difference between total spike dry weight and spike grain weight. Spike fertility (SF) was calculated as a ratio of grains m−2 to spike chaff weight m−2 [−2 was obtained by using seeds from SF sample (grains per spike) multiplied to the number of spikes m−2. To get spike number m−2, we harvested tillers at maturity from 0.5 m2 middle two rows and counted and later converted to m2. The HI was calculated as the ratio of grain weight m−2 to total dry matter m−2. TGW was measured by weighing 1000 grains counted through a seed counter . Phenotypic data for days to heading (DTH), grain yield (GY), harvest index (HI), spike fertility (SF), and thousand grain weight (TGW) were collected. Days to heading (GS 59) were collected using the Zadoks scale . GY was ight m−2 . Grain mijkY is the observed value; µ was the general mean; j = 1 to 223); i = 1 to 4, corresponding to Citra 2017, Citra 2018, Quincy 2017, and Quincy 2018); k = 1 to 12; ith environment; jth genotype by ith environment interaction effect; and H2) from each environment was calculated using the following formula, H2 = (σ2G)/(σ2G + σ2e), where σ2G and σ2e were variances due to genotype and error, respectively. Genotype and block were considered as random effects. In order to estimate variance values, we used the following model below:The best linear unbiased estimates (BLUEs) and standard errors were calculated for DTH, GY, HI, SF, and TGW using the following equation assuming genotype as a fixed effect and environment and block as random effects:ibutions . To evalPearson correlation analyses among four phenotypic traits were also calculated. Pstl and Msel restriction enzymes [α = 0.001). This procedure typically lowers heterozygous calls due to sequencing errors, genome duplications, and homologous sequences on different genomes [Fresh green seedling leaf tissues of each line were used to get genomic DNA through the LGC Genomics Oktopure robotic extraction platform along with Sbeadex magnetic microplate reagent kits. Genotyping by sequencing (GBS) was performed using Illumina HiSeq 2500 after double digestion of genomic DNA with  enzymes . SNP cal enzymes ,63. The  enzymes . Pre and enzymes . The SNP genomes ,65,66. IThree statistical models including the Multi-environment Genomic Best Linear Unbiased Predictor (MGBLUP), Bayesian Multi-trait Multi-environment (BMTME) model, and Bayesian Multi-output Regressor Stacking (BMORS), and two deep learning (DL) models including Single-trait Multi-environment Deep Learning (SMDL) and Multi-trait Multi-environment Deep Learning (MMDL) were compared for predicting GY, HI, SF, and TGW. Yij is the best linear unbiased estimate (BLUE) of predicted trait for jth genotype in ith environment; i = 1 to 4, corresponding to Citra 2017, Citra 2018, Quincy 2017, and Quincy 2018); j = 1 to 223); the genetic main effect is assumed as a joint distribution of genotype effect with a multivariate normal distribution G represents the genomic relationship matrix; the G matrices were calculated as X is a matrix of the centered and standardized SNP marker matrix and p is the number of SNP markers; tjh genotype by ith environment interaction effect; the term jth line in the ith environment distributed as According to Montesinos-López et al. ,67, a brH is a Σ is a Ω is a Y is the vector of multivariate responses of jth genotype and ith environment, X is a vector of ′G denotes the genomic relationship matrix; the G matrices were calculated as a and b; a; and b.For the BMTME model, a matrix-variate normal distribution is assumed denoted as The BMORS model is a two-stage predictive model originally proposed by Spyromitros-Xioufis et al. ,69. In taccuracy ,71. ThisSingle-trait Multi-environment Deep Learning (SMDL) and Multi-trait Multi-environment Deep Learning (MMDL) models delineated by Montesinos-López et al.  were alsDAPC) [All five predictive models were evaluated using a five-fold cross-validation (CV) approach for their prediction accuracies. Under this CV, the dataset was partitioned into five subgroups of equal size; four of the five subgroups  were used to fit each prediction model, while the remaining subgroup  was used to assess the correlation between the observed and predicted trait values. This process was repeated five times, with each subgroup being used as the prediction set once. A stratification method was employed to evaluate the influence of population structure on prediction accuracies for all three models. Briefly, the population was split into 10 clusters based on the discriminant analysis of principal components (DAPC)  clusteriR = 2SH [2H is the heritability for grain yield and S is the selection differential (in unit of kg ha−1). To be specific, all 237 lines were ordered according to their GEBV calculated from each model in each environment. Then, the top 10% lines were chosen as the selected population . The selection differential was calculated as the difference of grain yield between the means of selected lines and the whole population: S = S – P, where μS is the mean yield of 10% selected lines based on GEBV and μP is the mean yield of the population. The response to selection for all three models at each environment were computed with and without correction for DTH. The mean of RTS was calculated for each environment and each model across five folds. The standard error of RTS was calculated based on The standard error of prediction accuracy for each environment and each model was calculated based on  R = H2S , where HDAPC analysis was performed using an “adegenet” package [Phenotypic data analysis, including BLUPs and heritability calculation, and correlation analyses, were performed using R (R Development Core Team 2018). Basic models (1–2) were fit with the “lme4” package . Genetic package . The res package . Cross-vAll data generated or analyzed during this study are available in the supplemental files, including phenotypic data in “multi-trait GS phenotypic data.csv” and genotypic data in “multi-trait GS genotypic data.txt”.−1, 42.7%, and 40.9 g, respectively) in 2017, compared to other three environments. For Citra, a hotter and drier environment compared to Quincy had the lowest BLUEs of GY, HI, and SF  in 2018. Citra 2018 had the lowest value for TGW (34.1 g) and the highest value for SF (98.3 g). In general, Citra 2017 and Citra 2018 showed higher broad-sense heritability values than Quincy 2017 and Quincy 2018 for all four traits. For GY, Citra 2018 had the highest heritability (0.80), while Quincy 2018 had the lowest value (0.24). For HI, Citra 2017 had the highest heritability (0.78), while Quincy 2018 had the lowest value (0.26). Quincy 2017 showed the lowest heritability for SF (0.22), and Citra 2018 had the highest value (0.68). Quincy 2018 showed the lowest heritability for TGW (0.44), and Citra 2018 had the highest value (0.87). In general, a higher coefficient of variation was shown in Citra 2017 and Quincy 2017 compared to that in Citra 2018 and Quincy 2018 for all four traits.A description of GY, HI, SF, and TGW phenotypic traits is presented in Genetic correlations among four traits and four environments are presented in DAPC algorithm, and the panel was clustered into 10 groups (Population structure was determined by using the 0 groups . Each sup > 0.05), the low heritability of GY in this environment contributed to the negative predication accuracies in general. Overall, statistical models including MGBLUP and BMTME showed higher prediction accuracies than DL models (SMDL and MMDL). The BMOR model showed the highest prediction accuracies in the majority of the cases except for SF in Citra 2017. However, the differences between statistical models and DL models were minimal in some environments and traits. For example, DL models were comparable to statistical models for HI across environments. For SF, two DL models showed higher prediction accuracies compared to two statistical models in Quincy 2017 and Quincy 2018. When comparing results from four environments, Citra 2017 showed high prediction accuracies for all four traits. For GY, Quincy 2018 had lower prediction accuracies compared to other environments. Citra 2018 had lower prediction accuracies for SF and TGW compared to other environments. For HI, the prediction accuracies varied between models and environments. For DL models, optimal epoch and neuron combinations were identified based on the results of response surface research (Supplement When populations were stratified (noted as “stratified”), prediction accuracies ranged between −0.22 and 0.62 for GY, 0.03 and 0.55 for HI, 0.16 and 0.83 for SF, and 0.21 and 0.85 for TGW among four environments and three models . A similFor MGBLUP and BMOR models, the averaged prediction accuracies across environments were higher for TGW, which was followed by SF, HI, and GY in order . For theWe also applied the BMOR model to predict whole environments using the remaining environments as training datasets . Predict−1 for GY, 0.09 to 4.94% for HI, 0.45 to 3.90 grains g−1 of chaff weight for SF, and 0.99 to 2.02 g for TGW among four environments and three models (Response to selection (RTS) was compared in the same fashion as prediction accuracy for each model × environment combination. When populations were not stratified, RTS ranged from −0.05 to 0.5 ton hae models . In gene−1 for GY, 0.13 and 5.53% for HI, 0.26 and 4.46 for SF grains g−1 of chaff weight, and 1.01 g and 2.70 g for TGW among four environments and five models   . The hig−1) . For SF,−1) . For TGW−1) . In gene−1 for GY, 0.45 to 5.22% for HI, 6.52 to 13.33 grains/g of chaff weight for SF, and 3.03 to 4.36 g for TGW  for prediction accuracy and response to selection of all four traits when comparing to the single-trait and multi-environment model (MGBLUP), single-trait and multi-environment deep learning model, and multi-trait and multi-environment deep learning model. This result is in concordance to previous studies that reported that multi-trait and multi-environment GP models could be implemented to increase the prediction accuracy and RTS for low-heritability traits correlated with higher-heritability traits ,83,84,85The use of a stratified cross-validation scheme with all five models did not increase the prediction accuracy compared with using an un-stratified cross-validation scheme in the present study. One possible reason is that the alleles of representative quantitative trait loci (QTL) associated with target traits are commonly shared between training and validation populations in both stratified and un-stratified schemes. Ward et al.  also fouAlthough the BMOR model showed higher prediction accuracy and comparable RTS to that of the BMTME model, it does not estimate the covariances between traits and environments because it implements univariate analysis at both stages . HoweverThe study demonstrates that the multi-trait model has in general higher predictive accuracy than the single-trait model under a multiple-environmental analysis and has the capacity to predict the performance of genotypes for different test environments. It is useful for plant breeding scenarios where several economically important traits are inter-correlated. The findings of the present study could be potentially applied in plant breeding to achieve more cycles of selection by unit of time for multiple traits, to assess accurately genotype performance due to the low number of testing environments or due to a lack of replication, and to predict the performance of genotypes for stressed environments with low heritability. The analysis also showed that statistical models were superior to DL models for the studied traits, but DL models were comparable to statistical models in many cases. In conclusion, our study showed that for our population and traits of interest, multi-trait and multi-environment models can be exploited to achieve generally higher increases in prediction accuracy and RTS in several focal traits.


As expected, all methods performed better with larger sample sizes and effect sizes. The performance of clustering and dimensionality reduction approaches to correct for population structure were considerably variable according to the choice of parameters. Notably, the multi-locus elastic net (lasso) approach was consistently amongst the highest-performing methods, and had the highest power in detecting causal variants with both low and high effect sizes. Most methods reached the level of good performance  for identifying causal mutations of strong effect size [log odds ratio (OR) ≥2] with a sample size of 2000 genomes. However, only elastic nets reached the level of reasonable performance  for detecting markers with weaker effects (log OR ~1) in smaller samples. Elastic nets also showed superior precision and recall in controlling for genome-wide linkage, relative to single-locus models. However, all methods performed relatively poorly on highly clonal (low-recombining) genomes, suggesting room for improvement in method development. These findings show the potential for multi-locus models to improve bacterial GWAS performance. BacGWASim code and simulated data are publicly available to enable further comparisons and benchmarking of new methods.Genome-wide association studies (GWASs) have the potential to reveal the genetics of microbial phenotypes such as antibiotic resistance and virulence. Capitalizing on the growing wealth of bacterial sequence data, microbial GWAS methods aim to identify causal genetic variants while ignoring spurious associations. Bacteria reproduce clonally, leading to strong population structure and genome-wide linkage, making it challenging to separate true ‘hits’ (i.e. mutations that cause a phenotype) from non-causal linked mutations. GWAS methods attempt to correct for population structure in different ways, but their performance has not yet been systematically and comprehensively evaluated under a range of evolutionary scenarios. Here, we developed a bacterial GWAS simulator (BacGWASim) to generate bacterial genomes with varying rates of mutation, recombination and other evolutionary parameters, along with a subset of causal mutations underlying a phenotype of interest. We assessed the performance  of three widely used single-locus GWAS approaches (cluster-based, dimensionality-reduction and linear mixed models, implemented in  Genomes used for measuring linkage disequilibrium (LD).Mycobacterium: 3295 samples susceptible to pyrazinamide, downloaded from ftp://ftp.patricbrc.org/AMR_genome_sets/Mycobacterium/pyrazinamide/Susceptible.Escherichia: 1582 samples susceptible and resistant to gentamicin, downloaded from ftp://ftp.patricbrc.org/AMR_genome_sets/Escherichia/gentamicin.Streptococcus: 2169 samples resistant to trimethoprim, downloaded from ftp://ftp.patricbrc.org/AMR_genome_sets/Streptococcus/trimethoprim/sulfamethoxazole/Resistant.2. Simulation datasets.https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_400/9956420.Simulation dataset for sample size 400: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_700/9956426.Simulation dataset for sample size 700: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_1000/9956429.Simulation dataset for sample size 1000: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_2000/9956441.Simulation dataset for sample size 2000: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_lowLD_Sample_size_3000/9956444.Simulation dataset for sample size 3000 and low LD: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Medium_LD_dataset/9956456.Simulation dataset for moderate LD simulations: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_High_LD_dataset/9956477.Simulation dataset for high LD simulations: Microbial populations contain measurable phenotypic differences with important clinical and environmental consequences, such as antibiotic resistance, virulence, host preference and transmissibility. A major challenge is to discover the genes and mutations in bacterial genomes that control these phenotypes. Bacterial genome-wide association studies (GWASs) are families of methods to statistically associate phenotypes with genotypes, such as point mutations and other variants across the genome. However, compared to sexual organisms such as humans, bacteria reproduce clonally meaning that causal mutations tend to be strongly linked to other mutations on the same chromosome. This genome-wide linkage makes it challenging to statistically separate causal mutations from non-causal false-positive associations. Several GWAS methods are currently available, but it is not clear which is the most powerful and accurate for bacteria. To systematically evaluate these methods, we developed BacGWASim, a computational pipeline to simulate the evolution of bacterial genomes and phenotypes. Using simulated genomes, we found that GWAS methods varied widely in their performance. In general, causal mutations of strong effect (e.g. those under strong selection for antibiotic resistance) could be easily identified with relatively small samples sizes of around 1000 genomes, but more complex phenotypes controlled by mutations of weaker effect required 3000 genomes or more. We found that the elastic net (lasso) approach, a method that has only recently been applied to the GWAS problem, was particularly good at identifying causal mutations in highly clonal populations, with strong linkage between mutations – but there is still room for improvement. The BacGWASim computer code is publicly available to enable further comparisons and benchmarking of new methods.Recent progress in sequencing technologies and consequently the rapid expansion of bacterial genomic data repositories have provided enormous opportunities to identify the genomic elements underlying clinically, environmentally and industrially important bacterial phenotypes and their evolutionary responses to changing environmental circumstances. Such discoveries could immensely improve our knowledge of the molecular mechanisms of important microbial phenotypes such as antibiotic resistance and virulence; thus, contributing to the development of new drugs, vaccines and antibiotics.Neisseria meningitidis into a lethal pathogen causing cerebrospinal meningitis using multilocus sequence typing  and power [true positives/[true positives+false negatives)] to identify true causal variants in the simulated bacterial populations. As an additional performance metric, we used the F1 score, defined as the harmonic mean of precision and power, which has been used elsewhere to evaluate the performance of GWAS tools . These mBacGWASim was developed with the goal of simulating a range of evolutionary parameters that can potentially confound microbial GWASs. In this first release, we primarily focused on simulating population structure and genome-wide LD, and evaluating GWAS methods to identify SNPs underlying a phenotype. BacGWASim starts with a bacterial whole genome and its annotations, and simulates a population across a pre-defined or simulated phylogenetic tree a. A binaalf) [A neutral model of speciation-extinction developed by Genhard  implemenalf)  was usedalf was used for the simulation of protein-encoding genes and dawg v.2 for the intergenic regions [alf phylogenetic tree simulator in order to remove confounding effects of differences in genetic diversity on GWAS. The resulting sequences of coding and intergenic regions at the tips of phylogenetic tree are then combined, while accounting for gene loss and transfer. Synthetic sequencing reads with Illumina-specific sequencing errors were generated for each simulated sample using the art next-generation sequencing read simulator [et al. [bwa [For simulation of realistic genomic content, BacGWASim accepts any bacterial whole genome and the corresponding feature annotations as starting input. The elements of the genome are then extracted and classified in two groups of protein-encoding genes and intergenic regions (defined as sequence not annotated as a coding sequence). These two categories are often under distinct evolutionary constraints and their evolution is best captured by different evolutionary parameters.  regions . These iimulator . The emp [et al.  were useal. [bwa  with defal. [bwa .gcta [After genome simulation, a binary or continuous phenotype was then simulated for each member of the population based on an additive genetic model (equation 1) implemented in gcta :jy is the phenotype liability of individual ijj, x is number of reference alleles for the ith causal variant of the jth individual, ip is the frequency of the ith causal variant, iu is the allelic effect of the ith causal variant, and je is the residual effect generated from a normal distribution with mean of 0 and variance of var (2h - 1), where 2h is the heritability of the phenotype defined by the user. Cases were sampled from the individuals with phenotype liabilities (y) exceeding the threshold of a normal distribution truncating the proportion of K , and controls were sampled from the remaining individuals.where A user-defined number of causal variants, minor allele frequency cut-off and range of effect sizes in units of OR are used to randomly select a set of ‘true’ causal variants from the pool of simulated markers. The phenotype labels are then simulated according to the presence/absence of the causal variants in each genome, user-defined values of heritability and prevalence of the desired phenotype. Across all the simulations in this paper, the ratio of case to control was set to 1 (simu-cc=population size/2−population-size/2), heritability was set to 1 (simu-hsq=1) and phenotype prevalence  was set to 0.5 (simu-k=0.5).S. pneumoniae estimated by Chewapreecha et al. [alf accepts the recombination rate (ρ/θ) via the lgtRate parameter. The maximum recombination tract length was set to the length of each gene considered. To simulate homologous recombination, the orthRep parameter was set to 1 in all scenarios. (iii) Effect size distribution: 18 causal markers with ORs (effect sizes) of 2, 3, 4, 7, 10, 11, 15 and 20  with minor allele frequency >0.1 were randomly chosen for phenotype simulation. The LD values between the selected markers were measured using bcftools [2 >0.6 were discarded. This filtering step to remove strongly linked causal markers was included to ensure that these markers (including those of different effect sizes) were identifiable in the simulated datasets. We note that causal markers could still be linked to non-causal alleles, posing a challenge for GWAS methods to correctly identify the causal variant.Bacterial genomes were simulated within ranges of three features of interest, keeping other parameters constant. (i) Sample size: bacterial populations with a range of 400 to 3000 sampled genomes were simulated to evaluate the effect of sample size on GWAS. (ii) Recombination rate: recombination rates in highly recombining a et al.  card dataThis was employed using linkage agglomerative clustering based on pairwise identity-by-state (IBS) distances for population structure correction, and using Bonferroni corrections for multiple tests with the typically accepted genome-wide false-positive rate (FPR) of 0.05 .This was implemented in pyseer v 1.2.0 with multidimensional scaling of pairwise k-mer based genetic distances to correct for population structure, and using a Bonferroni correction  for the number of unique SNP patterns . In practice, however, the total number of patterns in our simulations were similar to the total number of SNPs (i.e. ~100 000). We also removed markers tagged with the errors ‘bad-chisq’, ‘pre-filtering-failed’, ‘lrt-filtering-failed’, ‘perfectly-separable-data’, ‘firth-fail’ and ‘matrix-inversionerror’ after the analysis . In our seer) after the analysis [This was used to correct for population structure, using unique patterns to estimate significance threshold, and by removing the tagged markers mentioned above , recall=true positives/, and F1=2×/. GWAS analysis can be considered as a classification task on a highly asymmetric dataset: in the case of our simulations, there were 99 984 true negative and only 16 true positives. Positive hits  were defined below a P value threshold of 0.05 after Bonferroni correction for 100 000 tests. Therefore, we used precision, recall and their harmonic mean (the F1 score) as our main performance metric, because they can handle this uneven class distribution. We also calculated the FPR .The performance of each GWAS method was assessed based on the mean values of precision, recall and F1 scores, and the corresponding S. pneumoniae . A range of genome-wide LD was then simulated to approximate the range of LD observed in bacterial species with high, moderate and low recombination rates. While not an exact match, these simulations approximate the LD landscapes of S. pneumoniae, Escherichia coli and M. tuberculosis, respectively , likely because the phenotypes investigated were under strong positive selection . Howevertic nets ; howevertic nets . FaST-LMtic nets , achievetic nets . With retic nets . Elastictic nets .A general pattern observed across all methods is that after a rapid surge in recall rate by increasing the sample size to 1000, the power improvement slows down and reaches a plateau around 3000 samples . BreakinWe next investigated how GWAS power varies across the four methods to correct for population structure: single-locus models including cluster-based approaches, dimensionality reduction and LMMs, and the multi-locus model using elastic nets. In single-locus models, the association between each of the markers (SNPs or k-mers) and the desired phenotype is investigated separately. Therefore, the covariance between the markers due to population structure needs to be included explicitly. Multi-locus models, however, include all the genetic variants at once, and by this means, covariance between the variants are implicitly included in the analysis. Our findings show that the multi-locus elastic net model has superior power relative to single-locus models in controlling for population structure and genome-wide LD, especially in small sample sizes  and whenplink across a range of thresholds. The threshold was defined as the maximum number of individuals allowed to group in one cluster. By relaxing the threshold, a larger number of samples with lower genetic similarity are included in each cluster; therefore, population structure will not be fully captured. In contrast, setting a very strict threshold will make the association test conservative by over-correcting for population structure.One of the classic methods for controlling the confounding effect of population structure is to identify clusters of related individuals within the overall population and then test for association conditional on these subpopulations. Subpopulations can be inferred using a variety of methods  and thenUsing simulated datasets of 3000 genomes with low LD , we variIn dimensionality reduction, a relatedness matrix of samples is projected onto a lower number of components, and then a certain number of these components are chosen as fixed-effect factors in the linear regression (or logistic regression in the case of binary phenotypes) to account for population structure. The two most popular and nearly identical methods for dimensionality reduction in GWASs are the principal component ANOVA-standardized relationship matrix based on SNP data  and the seer method implemented in pyseer on a simulated high-recombining dataset of 3000 samples  and has been shown to control type I error without loss of power compared to GWAS performed without population stratification correction . gemma [gemma  and FaSTgemma  are two gemma , dbgwas  [dbgwas  or pysee [dbgwas .gemma and FaST-LMM implemented in pyseer on the same simulated high-recombining dataset of 3000 samples .Higher genome-wide LD is expected to reduce the precision of association testing, because hitchhiking non-causal mutations are identified as false positives. To assess the influence of LD on GWAS performance, we tested each method across a range of simulated datasets with low, moderate or high genome-wide LD . In genes method . At highs method . All mets method . Still, s method . However high LD . Q-Q ploseer, BugWAS, dbgwas and others. Although a full exploration of core and pangenome associations is beyond the scope of the current work, we expect our major findings to hold when causal variants in the pangenome are included. Our evolutionary model is also ‘neutral’ in that it does not consider positive selection on causal variants (e.g. antibiotic-resistance mutations that have a selective advantage). Future versions of BacGWASim could explicitly model an increased replication rate  of bacteria encoding causal variants, perhaps more realistically capturing the evolution of certain phenotypes.We developed a platform to simulate bacterial genomes and phenotypes based on the emergence and evolution of causal variants along a phylogenetic tree. The simulator is tunable for relevant evolutionary parameters. Although it was designed for benchmarking GWAS tools, it could also be used for other applications in bacterial population genomics and epidemiology. Here, we focused on how GWAS method performance is affected by sample size, recombination (LD) and causal variant effect sizes. We only considered point mutations in the core genome as causal variants, but this could be extended to causal variants in the pangenome (i.e. gene presence/absence), which can be identified with k-mer approaches  such as . E. coli or M. tuberculosis) [S. pneumoniae. These results highlight the importance of assessing the LD landscape of the target organisms before deciding on a sample size and GWAS design. It also suggests significant room for improvement in GWAS method development, particularly for highly clonal bacteria for which we are currently only able to detect lineage-specific associations [Our results confirmed that currently popular GWAS methods perform poorly when applied to bacteria with relatively clonal population structures yielding moderate or high LD (e.gulosis) . In suchulosis) . More enciations . Althougciations  and treeciations  due to tciations , they wogemma. The clustering approach implemented in plink also performed well, but varied significantly depending on the clustering threshold, which can be challenging to optimize. Elastic nets implemented in pyseer also provide the possibility to perform a GWAS on k-mers or unitigs, whereas this is not as easily implemented in gemma or plink. Although elastic nets had the highest precision of the methods tested, there is significant room for improvement, as mentioned above. However, particularly for highly clonal populations, there may be a limit to what can be learned from GWAS approaches, and some combination of experimental and observational studies may be required.Of the GWAS methods evaluated here, the elastic net multi-locus model  generally performed best, followed by the mixed model approach implemented in Our results also help explain the success of early bacterial GWASs with low sample sizes. We found that, in a high-recombining population, a sample size ~1000 is sufficient to reliably detect causal variants of strong effect (log OR>=2) with high power (>0.80). Such strong effect sizes may be common for antibiotic-resistance mutations, and other variants under strong positive selection. However, samples sizes >3000 will likely be needed to detect variants of lower effect (log OR ~1), which may be more common for more ‘complex’ phenotypes with lower heritability. Of course, the sample size required to achieve a desired power will vary depending on the recombination rate and population structure of the organism of interest. Thus, BacGWASim provides a tool for study-specific power calculations. Although other simulators of bacterial evolution are available, these tend not to model phenotype evolution, which is essential for benchmarking GWAS methods, nor do they provide as much control over genetic variation at the level of nucleotides, genes and genomes . We prevThis work is somewhat limited in initial scope, as we focus on a subset of evolutionary parameters which we deem most relevant to GWAS benchmarking. Future studies could further explore variation in the mutation rate, causal allele frequencies, high LD among causal variants, case–controls ratios, environmental factors or selective pressures that differ between sub-populations. In an age of a rapidly growing array of options for performing GWASs, we hope that our results are instructive in quantifying general trends, and that our simulation platform can continue to be used to benchmark novel methods as they appear.Mycobacterium: 3295 samples susceptible to pyrazinamide, downloaded from ftp://ftp.patricbrc.org/AMR_genome_sets/Mycobacterium/pyrazinamide/Susceptible (2019).1. Genomes used for measuring LD in Escherichia: 1582 samples susceptible and resistant to gentamicin, downloaded from ftp://ftp.patricbrc.org/AMR_genome_sets/Escherichia/gentamicin (2019).2. Genomes used for measuring LD in Streptococcus: 2169 samples resistant to trimethoprim, downloaded from ftp://ftp.patricbrc.org/AMR_genome_sets/Streptococcus/trimethoprim/sulfamethoxazole/Resistant (2019).3. Genomes used for measuring LD in https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_400/995642 (2019).4. Saber, M. M., Shapiro, B. J., Simulation dataset for sample size 400 available at: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_700/995642 (2019).5. Saber, M. M., Shapiro, B. J., Simulation dataset for sample size 700 available at: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_1000/9956429 (2019).6. Saber, M. M., Shapiro, B. J., Simulation dataset for sample size 1000 available at: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Sample_size_2000/9956441 (2019).7. Saber, M. M., Shapiro, B. J.,Simulation dataset for sample size 2000 available at: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_lowLD_Sample_size_3000/9956444 (2019).8. Saber, M. M., Shapiro, B. J., Simulation dataset for sample size 3000 and low LD available at: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_Medium_LD_dataset/9956456 (2019).9. Saber, M. M., Shapiro, B. J., Simulation dataset for moderate LD simulations available at: https://figshare.com/articles/bacterial_GWAS_benchmark_simulations_High_LD_dataset/995647 (2019).10. Saber, M. M., Shapiro, B. J., Simulation dataset for high LD simulations available at: Click here for additional data file.