Timp2 and Abcg8 deficient mice. Our results suggest that the integration of functional genomics data into networks, which itself is informative of protein function and interactions, can successfully be utilized as a complementary approach to quantitative genetics to predict disease risks. All supplementary material is available at http://cbfg.jax.org/phenotype.An ultimate goal of genetic research is to understand the connection between genotype and phenotype in order to improve the diagnosis and treatment of diseases. The quantitative genetics field has developed a suite of statistical methods to associate genetic loci with diseases and phenotypes, including quantitative trait loci (QTL) linkage mapping and genome-wide association studies (GWAS). However, each of these approaches have technical and biological shortcomings. For example, the amount of heritable variation explained by GWAS is often surprisingly small and the resolution of many QTL linkage mapping studies is poor. The predictive power and interpretation of QTL and GWAS results are consequently limited. In this study, we propose a complementary approach to quantitative genetics by interrogating the vast amount of high-throughput genomic data in model organisms to functionally associate genes with phenotypes and diseases. Our algorithm combines the genome-wide functional relationship network for the laboratory mouse and a state-of-the-art machine learning method. We demonstrate the superior accuracy of this algorithm through predicting genes associated with each of 1157 diverse phenotype ontology terms. Comparison between our prediction results and a meta-analysis of quantitative genetic studies reveals both overlapping candidates and distinct, accurate predictions uniquely identified by our approach. Focusing on bone mineral density (BMD), a phenotype related to osteoporotic fracture, we experimentally validated two of our novel predictions (not observed in any previous GWAS/QTL studies) and found significant bone density defects for both  Many recent efforts to understand the genetic origins of complex diseases utilize statistical approaches to analyze phenotypic traits measured in genetically well-characterized populations. While these quantitative genetics methods are powerful, their success is limited by sampling biases and other confounding factors, and the biological interpretation of results can be challenging since these methods are not based on any functional information for candidate loci. On the other hand, the functional genomics field has greatly expanded in past years, both in terms of experimental approaches and analytical algorithms. However, functional approaches have been applied to understanding phenotypes in only the most basic ways. In this study, we demonstrate that functional genomics can complement traditional quantitative genetics by analytically extracting protein function information from large collections of high throughput data, which can then be used to predict genotype-phenotype associations. We applied our prediction methodology to the laboratory mouse, and we experimentally confirmed a role in osteoporosis for two of our predictions that were not candidates from any previous quantitative genetics study. The ability of our approach to produce accurate and unique predictions implies that functional genomics can complement quantitative genetics and can help address previous limitations in identifying disease genes. Understanding the genetic bases of human disease has been an overarching goal of biology since the foundation of genetics as a scientific discipline. Efforts in quantitative genetics have utilized new laboratory technology to quickly genotype and phenotype large populations in order to determine which sequence features are most related to specific phenotypes. There are currently two major quantitative genetics approaches used to identify these genotype-phenotype associations Despite promising results, each of these approaches for quantitative genetics have common and unique unresolved issues that limits their utility. Both QTL and GWAS approaches can suffer from sampling biases. Population structure and proper selection of representative case and control groups are challenges for many GWAS, while linkage disequilibrium and limited genetic diversity are challenges for many QTL studies e.g. expression, interactions, etc.) to identify candidate genes that may be involved in a phenotypic outcome.Thus, there is a strong need for complementary approaches to quantitative genetic techniques that are independent from the biases inherent in linkage mapping QTLs and GWAS. Many of the shortcomings of quantitative genetics could be attenuated by considering the functional roles of proteins and by evaluating existing large-scale experimental evidence. As such, we propose a complementary, alternative approach to discovering gene-phenotype associations that applies machine learning techniques to functional genomic measurements of the activities of genes and proteins . Our results concretely demonstrate that the combination of quantitative genetics and functional genomics approaches can more comprehensively associate genes with phenotypes or diseases, which may aid in identifying risk factors and potential drug targets.We propose that functional genomics approaches can complement the potential shortcomings of quantitative genetics results in two ways: first, by identifying candidates that may have been missed due to biases in sampling or low allele frequency; and second, by prioritizing candidates in loci containing many genes due to limited mapping resolution. Here, we adopt a state-of-the-art machine learning algorithm (support vector machine) to analyze the functional network of the laboratory mouse to identify genes involved in phenotypes and diseases. We show that our approach significantly out-performs previous naïve functional genomic methods used to predict phenotypes. Further, we demonstrate that our results are complimentary to quantitative genetics methods since a statistically significant number of our predictions fall within QTLs or GWAS loci, but several of our most confident predictions fall outside of these regions as well. We have experimentally validated a phenotypic role for two genes predicted to be involved in bone mineral density (BMD), a risk factor for osteoporotic fracture, which were not identified by any previous quantitative genetics study  are available online at http://cbfg.jax.org/phenotype.In this study, we develop a new algorithm that accurately predicts the phenotypic effects of genetic perturbations based on functional genomic data, and we demonstrate that this approach complements the results of quantitative genetics studies. In the following sections, we first demonstrate that combining a genome-wide functional relationship network with a state-of-the-art machine learning algorithm can produce improved predictions of genotype-phenotype relationships compared to previous naïve algorithms. Second, we observe that both existing prior knowledge and the biological nature of phenotypes affect the accuracy of our approach. Third, we quantitatively show that our results compensate for some of the shortcomings of prior GWAS and QTL studies by providing complementary predictions based on large-scale functional genomic data. Finally, we experimentally demonstrate the power of this alternative approach by e.g. gene pairs known to interact) and negative examples (e.g. gene pairs not known to interact). Given these learned conditional probabilities for each data source and prior probabilities for gene relationships, Bayesian inference is used to generate a network defined by pair-wise posterior probabilities of functional relationships between all genes. In order to use this network to predict genotype-phenotype associations, we must exclude phenotypic data to avoid circularity. Therefore we gathered diverse genomic data for inputs, including protein-protein physical interactions Integrated functional relationship networks have the advantage of summarizing multiple complex datasets into a concise, visually interpretable graph representation where genes are nodes and connections between them represent the probability that two genes work together Given this integrated functional relationship network, we applied two methodologies to generate hypotheses about genotype-phenotype associations. Each method is supervised, and thus requires a starting set of genes known to be associated with each phenotype or disease of interest. Here, we used the mammalian phenotype (MP) ontology i.e. the inferred probabilities that genes are functionally related) as features for support vector machine (SVM) classification i.e. positive examples) are used for training. We then apply the trained SVM to classify all genes for each phenotype , which is most important for subsequent biological validation where only a handful of candidates can be reasonably examined. At 1% recall (roughly 200 predictions), our SVM approach achieved a median of 75% precision, compared to 43% for the summed weight method; and at 10% recall (roughly 2000 predictions) our SVM approach outperforms summed weight by 40% to 15%. The comparisons of precisions at multiple levels of recall confirm the overall improved quality of our algorithm over naïve methods  ontology  methods .i.e. the number of genes annotated to each MP term) Prediction algorithms often show drastic differences in baseline performance related to the number of training examples  or which are measured based on a subjective “severity score”.As discussed above, there is a significant effect on prediction accuracy based on the number of known gene annotations to each phenotype. In addition to this size effect, there appears to be a strong correlation between prediction accuracy and the ability of a phenotype to be accurately and reproducibly measured. For example, among the phenotypes most accurately predicted by our approach are “decreased IgE level” (MP:0002492) and “decreased circulating free fatty acid level” (MP:0002702). Each of these is a concrete phenotype that is measurable using an unbiased metric such as “concentration” or “width”. In contrast, among the most poorly predicted phenotypes are “head bobbing” (MP:0001410), “disheveled coat” (MP:0001511), and “lethargy” (MP:0005202), which are more qualitative in nature , which confirms our observation that more quantitative phenotypes tend to perform better. This phenomenon is not surprising in that functional genomic evidence is more likely to be informative for well-defined phenotypes. However, this effect is also promising in that better defined phenotypes are more likely to reflect specific molecular-level changes that may be more relevant from a drug target or clinical diagnosis perspective.While it is difficult to concretely assess the notion of “phenotypic specificity,” we generally observe in our overall results that more concrete phenotypes tend to perform more accurately. In order to quantify this effect, we conducted a small, blinded survey of 19 laboratory biologists. Survey respondents ranked phenotypes on a scale from 1  to 5 . Based on these results, we found a small, but significant, difference between the top and bottom 20 phenotypes ranked by overall precision . Similarly, genomic regions within 5cM of a QTL peak contain 16% of mouse genes, but contain 71 of our top 100 predictions .We selected bone mineral density (BMD) as an example to evaluate our approach since this is an extensively studied heritable trait in both human populations and mammalian model organisms While this overlap is significant, a large number of our most confident predictions (∼30% of the top 100) do not fall within 5cM of a QTL peak and are thus not likely candidates from previous studies. We consider these to be candidates likely missed by prior quantitative genetics studies due to sampling, population biases, epistasis, or other circumstances. In fact, two of our most confident predictions for genes associated with BMD are not candidates from previous quantitative genetics studies, but both have been experimentally verified as described below.Timp2 and Abcg8. These genes are our two most confident predictions for involvement in BMD that are not quantitative genetics candidates and that have existing, live knockout strains available for testing  11, outside of all previous QTLs identified on Chr 11. The Abcg8 gene lies on the distal end of Chr 17. One previous QTL study found a BMD QTL on Chr 17 that contains over 100 genes, including Abcg8Abcg8 was not considered as a possible candidate gene in this study because there is no known polymorphism within 20 kb of Abcg8 in the two strains crossed for linkage mapping (NZB/B1NJ and RF/J). None of the 20 BMD loci identified in human GWAS lie near either Timp2 or Abcg8  database . It is also linked to the metalloproteinases Mmp8 and Mmp14, the collagen Col1a1, and the glycoprotein Sparc, all of which have been associated with bone defects in the literature Abcg8 are the collagens Col1a2 and Col1a1, which are involved in osteoporosis (OMIM #166710). Additional interactors of Abcg8 include the bone related genes Sparc and the proteoglycan Bgn. All of these connections are supported by multiple evidence sources, including expression, physical interactions and phylogenetic profiles  and “abnormal bone structure” (MP:0003795) . Second,database  and S2. Timp2 or Abcg8 knockout mouse strains. However, there is limited evidence in the literature that these genes may play a role in bone biology. Polymorphisms in the human Timp2 gene were found to be weakly associated with increased risk of non-vertebral osteoporotic fracture in a small study of post-menopausal women Abcg8 is involved in cholesterol absorption and serum cholesterol levels Despite the connections apparent in our functional network, to our knowledge no bone related phenotypes have been reported for either Timp2 and Abcg8 knockout mice at 16 weeks of age. Animals homozygous for deletion were compared to heterozygous littermate controls. As shown in Timp2−/− male mice  and a significant increase of roughly 6.5% was observed in −/−Abcg8 male mice . Most individual mouse QTLs account for a 3–6% change in vBMD http://www.informatics.jax.org/external/ko/) to assess the likelihood of identifying genes with a significant affect on BMD. While these strains were not randomly selected, we can use these results to gain an estimate of how often single gene deletions affect bone density. Of 206 strains tested, only 20 exhibited a significant alteration in BMD, indicating a roughly 10% background rate. Thus, our confirmation of 2 out of 2 predictions is well above the expected result by chance.We examined femoral volumetric BMD (vBMD) in male Timp2−/− mice  and an increase in cortical thickness in the Abcg8−/− mice  as shown in −/−Timp2 and the increase density seen in −/−Abcg8, along with the noted differences in bone morphology, indicate that these genes are likely related to osteoporotic fractures. Neither of these genes is a candidate from any previous quantitative genetics study of BMD, which indicates that our approach produces results that are complementary to GWAS and QTL studies.Furthermore, we observed morphological defects in the bones of both strains, including an increase in periosteal circumference in the We have developed a novel SVM-based classification method to predict genotype-phenotype associations based on a probabilistic functional relationship network integrated from diverse data sources. Through bootstrapping and cross validation we confirmed its superior performance compared to previous approaches. Using osteoporosis related phenotypes as an example, we have computationally demonstrated and experimentally validated how integration of functional genomics data can facilitate disease gene identification in a complementary manner to quantitative genetics approaches.This study demonstrates the potential for integrative functional relationship networks to be applied in new ways, especially when successfully combined with sophisticated machine learning techniques. Functional networks have been intensively studied during recent years, resulting in multiple networks available for several model organisms We have also shown that our integration of functional genomics data is able to identify potential disease genes not yet identified by any quantitative genetics screens. The caveats of quantitative genetics, including sampling biases, rare allele effects, epistasis, and potentially limited explanatory power have been recognized for years http://cbfg.jax.org/phenotype.To avoid circularity in the phenotype prediction process, we created a functional relationship network that excludes phenotypic data. We pre-computed this network following the Bayesian method described in We utilized the annotations to the mammalian phenotype (MP) ontology pn, corresponds to all genes known to be associated with the phenotype or its descendents in the MP hierarchy. All other genes were considered as negative examples; the total number of negative examples is denoted as nn.We obtained the phenotype annotations for mouse from MGI f, is calculated for each gene, x, as the sum of all links between the gene and all positive examples from the gold standard:i, x)K, iy equals to 1 or −1 depending on whether gene i is annotated to the phenotype term or not, p is any gene annotated to the term in study, and n is any of the other genes.For each phenotype, we constructed a specific feature space consisting of the network connection weights to all positive examples of the phenotype. Therefore the number of features varies across different phenotypes and is equal to 2)O for a 25-round bootstrap validation to generate the prediction results for each phenotype term. Another intuitive algorithm is to directly input all original data into a linear SVM Alternately, we could use the connection weight to all examples as the input vectors, regardless of whether they are positive or negative examples. However, this construction is excessively time-consuming for SVM calculations. For example, in a Lagrangian SVM, the running time is approximately In order to limit over-fitting, and because each phenotype is only associated with a limited set of genes, we consider bootstrap cross-validation to be an ideal method for estimating error rates X is the raw output value for a gene, y represents positive examples, n represents negative examples, yσ and nσ are the standard deviations of the raw outputs for positive and negative examples, and yμ and nμ are the mean of the raw outputs for positive and negative examples. Based on these distributions we estimate the probability of a gene being associated with a phenotype as p(y|X), given its observed value X. Outputs with a value lower than the average of negative examples are assigned as zero. Transforming to probability does not affect the ordering of results, and consequently has no effect on our performance evaluation metrics. We provide the complete list of predictions in terms of probability on our supporting website http://cbfg.jax.org/phenotype.The outputs from the SVMs represent the distances from the examples to the separating hyperplane To assess the performance of the phenotype predictions, we obtained precision-recall curves and summary statistics for each phenotype. We computed the precision at various recall rates as previously described Recall is defined as the percentage of genes annotated to a given phenotype that were classified as having that phenotype:There is a trade off between precision and recall in that the most confident predictions are more likely to be accurate, whereas in order to achieve high levels of recall, we must accept a lower level of precision. As such, precision values are measured at many levels of recall to produce a curve. In order to produce single number summary statistics from these curves, we use the area under the precision-recall curve (AUPRC) as well as the precision at fixed levels of recall, including 1%, 10%, 20% and 50%.Timp2tm1Pds/J (Stock Number 0008120) and B6.129-Abcg8tm1Elk/J (Stock Number 008763) mice were originally purchased from the resource colonies of The Jackson Laboratory  and colonies were maintained by pair mating heterozygous mice. After weaning, mice were maintained in groups of 3–5 in polycarbonate boxes (130 cm2) on bedding of sterilized white pine shavings under conditions of 12 hours light; 12 hours darkness. All mice used in this study had free access to water and diet for the duration of the study.All studies and procedures were approved by the Institutional ACUC of The Jackson Laboratory. The B6.129S4-ex vivo by peripheral quantitative computed tomography (pQCT). Specifically, mice were killed at 16 weeks of age and femurs were isolated and fixed in 95% ethanol for 14 days. Femurs were measured for density using an SA Plus pQCT densitometer  as previously described 3. Periosteal circumference and cortical thickness measures were made at the exact midshaft of the femur.Total femoral volumetric BMD (vBMD) and femoral geometry was assessed Figure S1Well defined, high level MP terms were obtained from MGI, which represent a wild sampling of phenotypes. Precisions at different levels of recall were calculated for both the summed weight method (left) and the network-based SVM method (right), where the latter shows significant improvement.(1.35 MB TIF)Click here for additional data file.Table S1Supporting evidence for top connectors to the candidate genes.(0.06 MB DOC)Click here for additional data file.Table S2Interaction weights (posteriors) of local networks surrounding Timp2 and Abcg8.(0.07 MB DOC)Click here for additional data file.Table S3Top 100 genes predicted for association with ‘abnormal bone mineralization’.(0.17 MB DOC)Click here for additional data file.Text S1Integration of diverse data for constructing a functional relationship network.(0.07 MB DOC)Click here for additional data file.Text S2Training SVMs on raw data as a baseline for performance evaluation.(0.06 MB DOC)Click here for additional data file.


When a test for diagnosis of infectious diseases is introduced in a resource-limited setting, monitoring quality is a major concern. An optimized design of experiment and statistical models are required for this assessment.Interferon-gamma release assay to detect tuberculosis (TB) infection from whole blood was tested in Hanoi, Viet Nam. Balanced incomplete block design (BIBD) was planned and fixed-effect models with heterogeneous error variance were used for analysis. In the first trial, the whole blood from 12 donors was incubated with nil, TB-specific antigens or mitogen. In 72 measurements, two laboratory members exchanged their roles in harvesting plasma and testing for interferon-gamma release using enzyme linked immunosorbent assay (ELISA) technique. After intervention including checkup of all steps and standard operation procedures, the second trial was implemented in a similar manner.P < 0.0001) and error variance was no longer affected by laboratory personnel in charge, indicating that a marked improvement could be objectively observed.The lack of precision in the first trial was clearly demonstrated. Large within-individual error was significantly affected by both harvester and ELISA operator, indicating that both of the steps had problems. After the intervention, overall within-individual error was significantly reduced (BIBD and analysis of fixed-effect models with heterogeneous variance are suitable and useful for objective and individualized assessment of proficiency in a multistep diagnostic test for infectious diseases in a resource-constrained laboratory. The action plan based on our findings would be worth considering when monitoring for internal quality control is difficult on site. Assuring quality is essential for clinical laboratories in the field of infectious diseases. Beneficiaries are not only patients obtaining a diagnosis on site but also future patients receiving benefits of clinical research supported by qualified laboratories. Quality assurance in modern laboratories is realized by total quality management including external quality assurance (EQA) and internal quality control (IQC) -3.In most resource-constrained countries, however, regulations on quality assurance have not been laid down by the authorities and accuracy and precision of clinical measurements have not been monitored systematically . Under sCurrently, immunoassay is commonly used to make a serological diagnosis of infectious diseases involving human immunodeficiency virus, a variety of hepatitis virus and other sexually transmitted or blood-borne pathogens ,7, whichRecently, a two-step immunoassay to detect tuberculosis (TB) infection has also been developed and used extensively . It consMycobacterium tuberculosis-specific antigens (TB-Antigen) for detecting TB infection. In the ELISA-based IGRA , one milliliter of the whole blood was collected into the Nil tube for negative control, Mitogen for positive control, and TB-Antigen separately. The blood in the tubes was mixed and placed in the incubator for 18 hours at 37°C . Approximately 200 μl of plasma were harvested from each tube after centrifugation .IGRA is a general method to measure interferon-gamma induced by Interferon-gamma concentrations in the plasma were measured by ELISA, using microtiter plate washer and reader  with the analysis software provided by the manufacturer . In this study, interferon-gamma concentrations obtained from this calculation were directly used for further analysis.Two trials were carried out in Hanoi TB and Lung Disease Hospital, Viet Nam. Between the first and second trial, statistical analysis was made and an intervention was planned to ensure counterchecking and correct questionable manipulations. Each trial consisted of two runs. In each run, three milliliters of blood were collected from volunteers after informed consent had been obtained. Study protocols using IGRA were approved by ethical committees of the Ministry of Health, Viet Nam and International Medical Center of Japan respectively.Two laboratory members, A and B, performed either plasma harvest or ELISA operation or both: Harvest included labeling and placing plasma storage tubes properly and transferring plasma from centrifuged blood collection tubes to these tubes by pipetting. ELISA was a process including preparing reagents and transferring plasma samples into the microtiter plate. ELISA ended with calculation of interferon-gamma concentration. Because their roles were changed occasionally due to the limited manpower of the laboratory, their performance in both Harvest and ELISA was the subject to be analyzed.A single specimen obtained from routine blood collection was not sufficient to assess two staff members' performance. Because additional blood sampling was not easily accepted in many countries including Viet Nam, BIBD was attempted to obtain analytical information from small volume of plasma samples in this study: Of four possible combinations of harvester and ELISA operator, two combinations were cyclically chosen, using the limited amount of specimen. Allocation of observed combinations by BIBD in this study was described in Table To identify outliers, Mahalanobis distance D was calculated, which took the distance from the mean and correlation into account . When D To assess effects of factors of interest and error variance, we used a fixed-effect model:of which,ijk: Interferon-gamma concentration in the plasmayμ: Grand mean of all measurementsi: Harvest with i levels: i = 1, 2 (= A and B)αj: ELISA with j levels: j = 1, 2 (= A and B)βk: Blood specimen with k levels: k = 1, 2,..., 12γijk: Within-individual error; following normal distribution with mean = 0 and variance = σ2: N εIn this clinical setting, effects of interaction terms were not considered in the above model, because harvesting plasma and performing ELISA are independent steps and it is unlikely that the exchanging of staff roles in itself could increase the chances of error.To determine whether individuals of Harvest or ELISA affect within-individual error, we assessed a fixed-effect model with heterogeneous variance of error in the following way:ij2).where error follows the normal distribution N. Differences in error variance of two trials and error variance affected by a given factor were considered to be significant when Out of 72 measurements obtained from the first trial, seven outliers were identified: One was in Nil condition, three in TB-Antigen and three in Mitogen .To assess effects of individuals for Harvest and ELISA and character of errors involved in the first trial, we first performed three-way ANOVA using a fixed-effect model, in which three factors, Harvest, ELISA and individual blood specimens may have possible effects on the interferon-gamma concentration respectively. This model decomposes the total variance into between-individual error (or bias) and within-individual error (or imprecision). Herein, "between-individual error" indicates deviation in interferon-gamma values caused by the difference between Harvesters or ELISA operators, and "within-individual error" represents fluctuation of interferon-gamma values measured by a single Harvester or ELISA operator.P = 0.017). In the other two conditions, the effects of ELISA and Harvest were also considerably large but did not reach significant levels, as compared with the corresponding mean square errors. These findings indicate that their performance is unstable. Problems specific to ELISA and Harvest should be considered, although not statistically significant in all conditions.As shown in Table We then analyzed which factor affected within-individual error. Because two laboratory members were involved in each step of this experiment, we assumed that within-individual error, i.e. error variance, could be different depending on the personnel in each step. Thus, we chose a fixed-effect model with heterogeneous variance of error affected by Harvest and ELISA Table .P = 0.0040), when error variance caused by ELISA operator was not considered. Difference of error variance caused by ELISA operators A and B was also significant (P = 0.024), when error variance caused by Harvester was not taken into account. These findings imply that under the model, the error variance was affected significantly by different Harvesters or ELISA operators, respectively.In Nil condition, difference in error variance was statistically significant between Harvesters A and B  there was a considerable number of outliers. Within-individual error was large and between-individual error was also comparably large, and b) within-individual error was affected by both Harvesters and ELISA operators at least when Nil was measured.Based on these results, an intervention was introduced: 1) reviewing all procedures of Harvest and ELISA, 2) reconsidering and strengthening standard operation procedures, 3) checking working condition of machines, and 4) developing a checklist for countercheck. First, we attempted to find out which procedure of harvesting and ELISA operation would be unstable and all questionable manipulations were listed up. Essential laboratory skills, such as mixing the solution by pipetting, were reviewed. Secondly, standard operation procedures were rechecked and corrected seeing that the laboratory personnel were handling three blood collection tubes and three other plasma storage tubes from each blood donor at a time, they should take every care to identify the tubes during Harvest and ELISA and to confirm the right position of corresponding tubes. Thorough instruction for handling ELISA plates and tubes with manipulation of the pipette was given to avoid carry-over error or contamination. After intensive discussions, more attention was paid to basic laboratory practice and reduction of preventable mistakes. Thirdly, performance of the ELISA plate washer and reader and the quality of distilled water were also checked. Technical requirements from the manufacturer, such as temperature for reagent reservation, time of incubation, were strictly followed. Finally, a checklist for the countercheck of each step was developed for practical use.To assess the overall improvement after intervention, CV was compared between the two trials. Because variation due to Harvest and ELISA was of interest, CV adjusted by the effect of specimens was calculated and used. The CV had decreased remarkably in each condition of the second trial, as compared with that of the first trial, indicating the overall improvement of test performance after intervention Table .In the second trial, only one outlier was seen in Nil condition ; the number of outliers was lower than that of the first trial.We then proceeded to analyze the change of parameters that had possibly contributed to overall improvement of test performance. As shown in Table In contrast to the first trial, there were no significant differences of error variance affected by Harvesters or ELISA operators Table . This fiP < 0.0001).We further evaluated the decrease in overall within-individual error statistically. For this purpose, we used a fixed-effect model with heterogeneous variance between the two trials. Under the assumption that influence of Harvest and ELISA was not changed between the two trials, estimated overall error variances of the two trials were compared. As shown in Table We have demonstrated that a study design BIBD and statistical analysis using fixed-effect models with heterogeneous variance of error are useful for objective and quantitative assessment of laboratory testing for the first time. A series of experiments in our study clearly showed that proficiency of the personnel was improved by an appropriate intervention between the first and second trials of a two-step ELISA-based immunoassay for tuberculosis newly introduced to a resource-constrained laboratory.Design of clinical experiments including block designs can be used to estimate effect of factors and their possible interaction . In blocPrevious studies in clinical fields other than laboratory medicine showed the advantage of BIBD over the sample size -14. In tIn the first trial before intervention, we found that within-individual error was large and between-individual error tended to be so. However, a number of outliers also affected both within- and between-individual errors. The cause of outliers was probably due to mixing up of specimen tubes or contamination of samples resulting from unfamiliar handling of multiple samples, although this was not easily determined ,16. UsinAfter timely intervention including checkup of all steps and standard operation procedures, marked improvement was observed in all parameters including CV, a general parameter for precision of measurements . In caseWe propose as a consequence the following action plan to improve diagnostic capacity in resource-constrained settings. This could be generalized not only for complicated immunoassay for infectious disease but also for other kinds of clinical tests:• Set the target CV derived from simple one-way ANOVA model of specimen . This value should be defined before the commencement of study.• Design experiment to evaluate between- and within-individual error.• Conduct experiment.• Analyze data with ANOVA model with and without heterogeneous error variance.• If CV exceeds the target, review the operating procedures.• Conduct experiment a second time.• Consequently analyze data to ascertain any improvement. • Return to step 5 until CV becomes less than the target.In-house quality control for effective transfer of skills is a topic of interest in our proposal and this should be carried out easily, at a low cost, whilst assuring objective and quantitative assessment in a clinical laboratory where resources such as reagents, manpower and feasibility of sample collection are limited. Our plan meets the above requirement. Measurements could be sent via the internet and analyzed in a statistical way by a joint-research facility inside or outside the country and an immediate feedback should be sent in an appropriate manner. Such continuous efforts to share information are important to maintain quality levels over a long distance .In this age of evidence-based medicine and development of new diagnostic technologies, quality of laboratory tests is essential. There is an urgent need for validation and standardization of the new assays before they are adopted into clinical diagnostics . Until sOur study has some limitations. It was obvious that outliers influenced statistical analysis in the first trial and exact causes of error in each condition were not clearly specified by the present analysis itself . ThroughIn a setting where a modern quality control system has not been entirely established, a laboratory test could be assessed quantitatively and such objective assessment is helpful for quality improvement of the test, if an appropriate design of experiment and statistical method are chosen. The design of experiment BIBD and analytical models for ANOVA were useful for objective assessment of individual skills in each stage of a multi-step immunoassay for tuberculosis in a laboratory with limited resources. A proposed plan to assess the level of proficiency might be useful for skill improvement of clinical testing especially for infectious diseases when monitoring is difficult to assure the sustainability of the technology transferred.The authors declare that they have no competing interests.NTLH participated in supervising the on-site implementation of the study, drafting the paper or substantially revising it. NI was responsible for making conception, design and overall supervision of the study, analysis and interpretation of data, drafting the paper or substantially revising it. NK participated in making conception and design of the study, analysis and interpretation of data, drafting the paper or substantially revising it. LTH carried out the immunoassays. DBT participated in on-site implementation. VTXT carried out the immunoassays. IM participated in technical transfer and supervision. NH was responsible for technical transfer and supervision. KH was responsible for technical transfer and supervision. SS participated in conception and design of the study. LTL participated in conception, design and supervision of the study. All authors read and approved the manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2334/9/66/prepub


Hypothesis testing is an important activity of empirical research and evidence-based medicine. A well worked up hypothesis is half the answer to the research question. For this, both knowledge of the subject derived from extensive review of the literature and working knowledge of basic statistical concepts are desirable. The present paper discusses the methods of working up a good hypothesis and statistical concepts of hypothesis testing. Many scientists, even those who do not usually read books on philosophy, are acquainted with the basic principles of his views on science. The popularity of Popper’s philosophy is due partly to the fact that it has been well explained in simple terms by, among others, the Nobel Prize winner Peter Medawar . Popper makes the very important point that empirical scientists (those who stress on observations only as the starting point of research) put the cart in front of the horse when they claim that science proceeds from observation to theory, since there is no such thing as a pure observation which does not depend on theory. Popper states, “… the belief that we can start with pure observation alone, without anything in the nature of a theory, is absurd: As may be illustrated by the story of the man who dedicated his life to natural science, wrote down everything he could observe, and bequeathed his ‘priceless’ collection of observations to the Royal Society to be used as inductive  evidence.Karl Popper is probably the most influential philosopher of science in the 20The first step in the scientific process is not observation but the generation of a hypothesis which may then be tested critically by observations and experiments. Popper also makes the important claim that the goal of the scientist’s efforts is not the verification but the falsification of the initial hypothesis. It is logically impossible to verify the truth of a general law by repeated observations, but, at least in principle, it is possible to falsify such a law by a single observation. Repeated observations of white swans did not prove that all swans are white, but the observation of a single black swan sufficed to falsify that general statement .et al., 2001).A good hypothesis must be based on a good research question. It should be simple, specific and stated in advance . This is a long-winded sentence, but it explicitly states the nature of predictor and outcome variables, how they will be measured and the research hypothesis. Often these details may be included in the study proposal and may not be stated in the research hypothesis. However, they should be clear in the mind of the investigator while conceptualizing the study.The hypothesis must be stated in writing during the proposal state. This will help to keep the research effort focused on the primary objective and create a stronger basis for interpreting the study’s results as compared to a hypothesis that emerges as a result of inspecting the data. The habit of post hoc hypothesis testing (common among researchers) is nothing but using third-degree methods on the data (data dredging), to yield at least something significant. This leads to overrating the occasional chance associations in the study.For the purpose of testing statistical significance, hypotheses are classified by the way they describe the expected difference between the study groups.The null hypothesis states that there is no association between the predictor and outcome variables in the population . The null hypothesis is the formal basis for testing statistical significance. By starting with the proposition that there is no association, statistical tests can estimate the probability that an observed association could be due to chance.The proposition that there is an association — that patients with attempted suicides will report different tranquilizer habits from those of the controls — is called the alternative hypothesis. The alternative hypothesis cannot be tested directly; it is accepted by exclusion if the test of statistical significance rejects the null hypothesis.A one-tailed (or one-sided) hypothesis specifies the direction of the association between the predictor and outcome variables. The prediction that patients of attempted suicides will have a higher rate of use of tranquilizers than control patients is a one-tailed hypothesis. A two-tailed hypothesis states only that an association exists; it does not specify the direction. The prediction that patients with attempted suicides will have a different rate of tranquilizer use — either higher or lower than control patients — is a two-tailed hypothesis.  A one-tailed hypothesis has the statistical advantage of permitting a smaller sample size as compared to that permissible by a two-tailed hypothesis. Unfortunately, one-tailed hypotheses are not always appropriate; in fact, some investigators believe that they should never be used. However, they are appropriate when only one direction for the association is important or biologically meaningful. An example is the one-sided hypothesis that a drug has a greater frequency of side effects than a placebo; the possibility that the drug has fewer side effects than the placebo is not worth testing. Whatever strategy is used, it should be stated in advance; otherwise, it would lack statistical rigor. Data dredging after it has been collected and post hoc deciding to change over to one-tailed hypothesis testing to reduce the sample size and P value are indicative of lack of scientific integrity.et al., 2001). In some ways, the investigator’s problem is similar to that faced by a judge judging a defendant [A hypothesis  is either true or false in the real world. Because the investigator cannot study all people who are at risk, he must test the hypothesis in a sample of that target population. No matter how many data a researcher collects, he can never absolutely prove (or disprove) his hypothesis. There will always be a need to draw inferences about phenomena in the population from events observed in the sample  occurs if an investigator rejects a null hypothesis that is actually true in the population; a type II error  occurs if the investigator fails to reject a null hypothesis that is actually false in the population. Although type I and type II errors can never be avoided entirely, the investigator can reduce their likelihood by increasing the sample size .False-positive and false-negative results can also occur because of bias .  Such errors are troublesome, since they may be difficult to detect and cannot usually be quantified.The likelihood that a study will be able to detect an association between a predictor variable and an outcome variable depends, of course, on the actual magnitude of that association in the target population. If it is large (such as 90% increase in the incidence of psychosis in people who are on Tamiflu), it will be easy to detect in the sample. Conversely, if the size of the association is small (such as 2% increase in psychosis), it will be difficult to detect in the sample. Unfortunately, the investigator often does not know the actual magnitude of the association — one of the purposes of the study is to estimate it. Instead, the investigator must choose the size of the association that he would like to be able to detect in the sample. This quantity is known as the effect size. Selecting an appropriate effect size is the most difficult aspect of sample size planning. Sometimes, the investigator can use data from other studies or pilot tests to make an informed guess about a reasonable effect size. When there are no data with which to estimate it, he can choose the smallest effect size that would be clinically meaningful, for example, a 10% increase in the incidence of psychosis. Of course, from the public health point of view, even a 1% increase in psychosis incidence would be important. Thus the choice of the effect size is always somewhat arbitrary, and considerations of feasibility are often paramount. When the number of available subjects is limited, the investigator may have to work backward to determine whether the effect size that his study will be able to detect with that number of subjects is reasonable.After a study is completed, the investigator uses statistical tests to try to reject the null hypothesis in favor of its alternative (much in the same way that a prosecuting attorney tries to convince a judge to reject innocence in favor of guilt). Depending on whether the null hypothesis is true or false in the target population, and assuming that the study is free of bias, 4 situations are possible, as shown in The investigator establishes the maximum chance of making type I and type II errors in advance of the study. The probability of committing a type I error  is called α  the other name for this is the level of statistical significance.If a study of Tamiflu and psychosis is designed with α = 0.05, for example, then the investigator has set 5% as the maximum chance of incorrectly rejecting the null hypothesis (and erroneously inferring that use of Tamiflu and psychosis incidence are associated in the population). This is the level of reasonable doubt that the investigator is willing to accept when he uses statistical tests to analyze the data after the study is completed.The probability of making a type II error  is called β (beta). The quantity (1 - β) is called power, the probability of observing an effect in the sample (if one), of a specified effect size or greater exists in the population.If β is set at 0.10, then the investigator has decided that he is willing to accept a 10% chance of missing an association of a given effect size between Tamiflu and psychosis. This represents a power of 0.90, i.e., a 90% chance of finding an association of that size. For example, suppose that there really would be a 30% increase in psychosis incidence if the entire population took Tamiflu. Then 90 times out of 100, the investigator would observe an effect of that size or larger in his study. This does not mean, however, that the investigator will be absolutely unable to detect a smaller effect; just that he will have less than 90% likelihood of doing so.Ideally alpha and beta errors would be set at zero, eliminating the possibility of false-positive and false-negative results. In practice they are made as small as possible. Reducing them, however, usually requires increasing the sample size. Sample size planning aims at choosing a sufficient number of subjects to keep alpha and beta at acceptably low levels without making the study unnecessarily expensive or difficult.et alpha at 0.05 and beta at 0.20 (a power of 0.80). These are somewhat arbitrary values, and others are sometimes used; the conventional range for alpha is between 0.01 and 0.10; and for beta, between 0.05 and 0.20. In general the investigator should choose a low value of alpha when the research question makes it particularly important to avoid a type I  error, and he should choose a low value of beta when it is especially important to avoid a type II error.Many studies sP value, the probability of obtaining the study results by chance if the null hypothesis is true. The null hypothesis is rejected in favor of the alternative hypothesis if the P value is less than alpha, the predetermined level of statistical significance . “Nonsignificant” results — those with P value greater than alpha — do not imply that there is no association in the population; they only mean that the association observed in the sample is small compared with what could have occurred by chance alone. For example, an investigator might find that men with family history of mental illness were twice as likely to develop schizophrenia as those with no family history, but with a P value of 0.09. This means that even if family history and schizophrenia were not associated in the population, there was a 9% chance of finding such an association due to random error in the sample. If the investigator had set the significance level at 0.05, he would have to conclude that the association in the sample was “not statistically significant.” It might be tempting for the investigator to change his mind about the level of statistical significance ex post facto and report the results “showed statistical significance at P < 10”. A better choice would be to report that the “results, although suggestive of an association, did not achieve statistical significance (P = .09)”. This solution acknowledges that statistical significance is not an “all or none” situation.The null hypothesis acts like a punching bag: It is assumed to be true in order to shadowbox it into false with a statistical test. When the data are analyzed, such tests determine the Hypothesis testing is the sheet anchor of empirical research and in the rapidly emerging practice of evidence-based medicine. However, empirical research and, ipso facto, hypothesis testing have their limits. The empirical approach to research cannot eliminate uncertainty completely. At the best, it can quantify uncertainty. This uncertainty can be of 2 types: Type I error  and type II error . The acceptable magnitudes of type I and type II errors are set in advance and are important for sample size calculations. Another important point to remember is that we cannot ‘prove’ or ‘disprove’ anything by hypothesis testing and statistical tests. We can only knock down or reject the null hypothesis and by default accept the alternative hypothesis. If we fail to reject the null hypothesis, we accept it by default.


Prunus persica (L.) Batsch] is one of the most economically important fruit crops that, due to its genetic and biological characteristics , has become a model plant in genomic studies of fruit trees. Our aim was an in-depth study of the extent, distribution and structure of peach genetic variation in North American and European commercial varieties as well as old Spanish varieties and several founders used in the early USA peach breeding programmes. For this we genotyped 224 peach cultivars using 50 SSRs evenly distributed along the 8 linkage groups of the Prunus reference map.Peach [Genetic distance analysis based on SSRs divided the peach cultivars in three main groups based mainly on their fruit characteristics: melting flesh peaches, melting flesh nectarines and non-melting varieties. Whereas non-melting flesh peaches had a higher number of alleles than melting peaches and nectarines, they were more homozygous. With some exceptions , the founder US cultivars clustered together with the commercial melting peaches, indicating that their germplasm is well represented in modern cultivars. Population structure analysis showed a similar subdivision of the sample into subpopulations. Linkage disequilibrium (LD) analysis in three unstructured, or barely structured, subpopulations revealed a high level of LD conservation in peach extending up to 13-15 cM.Using a much larger set of SSRs, our results confirm previous observations on peach variability and population structure and provide additional tools for breeding and breeders' rights enforcement. SSR data are also used for the estimation of marker mutation rates and allow pedigree inferences, particularly with founder genotypes of the currently grown cultivars, which are useful to understand the evolution of peach as a crop. Results on LD conservation can be explained by the self-pollinating nature of peach cultivated germplasm and by a bottleneck that occurred at the beginning of modern breeding practices. High LD suggests that the development of whole-genome scanning approaches is suitable for genetic studies of agronomically important traits in peach. P. domestica and P. salicina), apricot (P. armeniaca) and cherry (P. avium and P. cerasus). It originated in China where it was domesticated 4-5,000 years ago [Peach is the most important of the stone fruit crops, that also include plum (ears ago . Its culPrunus species [Microsatellite or simple-sequence repeat (SSR) markers, have been very useful for studying the extent and distribution of genetic variability in wild and cultivated plants including various  species -4. Our r species  indicatePrunus species  that have an effective gametophytic self-incompatibility system, peach is self-fertile. The outbreeding rate of peach has been estimated to be ~15% [In this paper we re-examine this collection of genotypes, with the addition of several American founder accessions, with a set of 50 SSRs that cover the peach genome . With th be ~15% , implyin be ~15% . Self-fe be ~15% , and the be ~15% -12. UndeThe 50 SSRs studied amplified 318 alleles, an average of 6.36 alleles per locus (A) and 2.08 the average effective number of alleles (Ae) (Table F) values were positive, with a mean of 0.26 for all loci. The average of power of discrimination (PD) was 0.60, with CPPCT022 being the marker with the highest ability to discriminate between two random cultivars (PD = 0.92). The probability of confusion calculated using the PD values, was extremely low: C = 3.32 × 10-24.We observed an average SSR heterozygosity (Ho) of 0.34, while the expected heterozygosity (He) was 0.46. Consequently, the fixation index (The number of different genotypes amplified by each marker ranged from 3 (BPPCT039 and CPPCT013) to 27 BPPCT015) with an average of 12.1. The whole set of 50 SSRs used was able to identify 209 different genotypes among the 224 cultivars studied Figure . Ten gro with an The percentage of heterozygous loci at each cultivar ranged from 0 to 79.4%, with an average of 33.4%. Three cultivars had all the SSR loci tested in homozygosis: the non-melting cultivars 'Cofrentes-6' and 'Auberge Blanc', and the nectarine 'Independence', whereas 'Elberta' had the highest rate of heterozygosis. These values are shown in Additional file A tree constructed from the SSR data divided the cultivars into clusters characterized by correspondence with fruit characteristics: melting peach/nectarine/non-melting peach Figure . These mi = 5.54) than melting peaches (Ai = 2.65) and nectarines (Ai = 2.46). In contrast these varieties were more homozygous (Ho = 0.26 compared to 0.39 in melting peaches and 0.31 in nectarines). The proportion of unique alleles was similar in melting peaches and nectarines (10.8% and 11.6% respectively) and higher in non-melting peaches (15.7%).We calculated the variability in the 3 main groups, separated by fruit morphology: melting peaches, melting nectarines and non-melting peaches. Given that these groups have different sample sizes, we compared the number of alleles per cultivar (Ai) as shown in Table K) from two to eight. With K = 2 we detected a subdivision between melting and non-melting cultivars. Moving to K = 3, the cluster of melting cultivars split into two subpopulations, one principally peaches and the other, nectarines. With increasing K, three populations remained almost invariable , N with 50 nectarines (dark green) and NMP with 21 non-melting peaches (light green). An accession was considered to belong to a population if its membership coefficient was ≥ 80% in 2, i.e. the association between each of the alleles at the first locus and at the second one. A total number of 4,306, 5,814 and 3,301 pairs of alleles were compared in the MP, N and NMP subpopulations, respectively, with 515 in MP, 649 in N and 444 in NMP being placed in the same linkage group (intra-chromosome comparisons).It is known that population structure increases linkage disequilibrium (LD) in the genome. For this reason we calculated the LD for each of the three unstructured subpopulations described. After removing low frequency alleles (considering MAF ≥ 0.05), we calculated interallelic r2 values plotted against distance. In all three populations, the LD clearly decays with distance. LD in melting peaches rapidly declines as pairs of SSRs become more distant, whereas in both nectarines and non-melting peaches, this is slower. The curves that describe the variation of r2 with distance show that it reaches significant values at 13.3 cM in MP (r2 ≥ 0.15), 15.9 cM in N (r2 ≥ 0.07), and 15.2 cM in NMP (r2 ≥ 0.18). These distances decrease considerably at a significance threshold of α ≤ 0.01 .In Figure The percentage of allele comparisons with significant LD between pairs of alleles of SSRs mapped in different linkage groups (unlinked comparisons) was 13.9% in the melting peach population, 13.4% in nectarines and 18% in non-melting peaches. When considering intra-chromosome comparisons, these figures rose to 25%, 20% and 31%, respectively, with the ratios decreasing with distance in all three cases. The variation with genetic distance of the percentage of pairs of alleles in LD for markers mapped in the same linkage group in each of the 3 subpopulations is shown in Figure In all three populations we observed a region, from 1 to 11-14 cM, where the LD decays considerably with distance, then slowly reaching a constant level.Prunus genome. Most of the varieties (208) used here coincide with the 212 previously genotyped in [Prunus genome was previously proposed [We genotyped 224 peach and nectarine cultivars with 50 SSR markers covering the whole proposed . Here weMost of the modern North American and European cultivars have originated from those produced in the US breeding programs of the early twentieth century, using a reduced number of parents as founders ,9,15. EiNone of the founders grouped together with the nectarine varieties, supporting an earlier hypothesis  that difIn agreement with previous results , the den-3 per allele, much lower than our previous estimate of 1.1 × 10-2 [Of the 224 possible genotypes, 209 (93%) were recovered and 199 (89%) of these cultivars had a unique genotype. An average of 25 SSRs differed between two cultivars , much higher than the background LD found in each of the three subpopulations .2 values and variation of the percentage of loci in LD with distance showed that, in all three populations, LD decayed at 13-15 cM. After this point, between 32-36% of the loci were in significant LD (α ≤ 0.05). When increasing the threshold to α ≤ 0.01, LD decayed at 2-6 cM depending on the population studied. This is similar to that observed in crops such as sugarcane with LD extending to 10 cM [2 ≥ 0.1 and 5-6 cM at the higher threshold of r2 ≥ 0.2 [Prunus reference map is 499 [In all cases, LD conservation was high and decreased with distance. Raw rto 10 cM , cotton r2 ≥ 0.2 , and grar2 ≥ 0.2 , would rp is 499 ,22, withAlthough SSRs are currently the markers used for genetic analysis in peach, a high number of SNP markers are being developed, providing additional opportunities for high-resolution LD mapping. These studies require knowledge of LD extension and intensity. Tests with simulated data to predict the extent of LD with SNPs based on LD estimates using SSRs , indicatThe number of individuals analyzed in each subpopulation was small (21 to 50 cultivars) and diversity analysis shows that they were closely related. It has been reported that populations with a broad genetic basis have lower levels of LD than narrow-based populations, so the use of more diverse peach material that exploits the recombination events occurred in its history is likely to result in a population with reduced LD extension for association mapping purposes ,26. ChinThe study of a large set of SSR markers in a collection of peach cultivars from Europe and North America supported previous data indicating a relatively low level of genetic variation, and a distribution of the molecular variability that places together entries with some of the major commercial fruit characters, such as peaches, nectarines and non-melting flesh peaches. Our results indicate a strong subpopulation structure and a high level of linkage disequilibrium conservation, which may have been caused in part by the fact that most cultivars examined originated from the small set of founders used by the early U.S. breeding programs. These data provide a standard where other molecular variability studies based on sets of cultivars from different origins can be compared, and a foundation for the development of tools for genome-wide surveys of variability based on SNP markers allowing for association genetics studies in this important horticultural crop.Of the set of 224 peach cultivars, most (208) coincide with the 212 previously studied . Eight oPrunus reference map 'Texas'  × 'Earlygold' (peach) .AB) between pairs of alleles A and B at two loci, in either the same or different linkage group, were calculated according to Weir's method [2AB, as in [2 = n*r2AB, where n is the number of individuals in the sample (see [21df = 3.841) and α ≤ 0.01 (χ21df = 6.635). Therefore, we did not correct the LD estimate for sample size since it was already accounted for by the significance value. For the curve that represents the observed r2 values, we used the average values for each of 4 subsets of equal number of pair comparisons covering adjacent intervals along distance.Given that the phases between alleles at two heterozygous loci are unknown, we calculated composite linkage disequilibrium (LD) coefficients, a measure reported to have good statistical properties and suggested for routine testing of LD : after rs method  with GDAs method . These cs method , r2AB, aB, as in . For thiple (see ), and siStructure v.2 [K subgroups of individuals with distinctive allele frequencies. Individuals can be members of multiple subpopulations with a different coefficient, with the sum of all being equal to 1. To check for population stratification in our sample we ran the program under the admixture model assumption with correlated alleles. The run used 106 interactions after a burn-in of 105 for a value of K ranging from 2 to 8.Population structure was studied with the ture v.2  softwareMJA and PA conceived and designed this study and wrote the manuscript. MJA obtained molecular marker data and performed the statistical analysis. EKA obtained molecular marker data and participated in the variability analysis. WH contributed in the molecular data uptake and data analysis. All authors read and approved the final manuscript.Description of the 224 peach cultivars used. (a) Most pedigree data were obtained from Okie's Handbook [Handbook ; (b) - UClick here for filePrunus reference map 'Texas' × 'Earlygold'Map position of the SSRs used in this study in the .Click here for file


In all cases, the “winner's curse” effect was corrected analytically. Confidence intervals were also derived. Simulations were performed to compare and verify the performance of different estimators (which incorporates various means of winner's curse correction) and the coverage of the proposed analytic confidence intervals. Our methodology only requires summary statistics and is able to handle both binary and continuous traits. Finally we applied the methods to a few real disease examples  and estimated that hundreds to nearly a thousand variants underlie these traits.Recently genome-wide association studies (GWAS) have identified numerous susceptibility variants for complex diseases. In this study we proposed several approaches to estimate the total number of variants underlying these diseases. We assume that the variance explained by genetic markers (Vg) follow an exponential distribution, which is justified by previous studies on theories of adaptation. Our aim is to fit the observed distribution of Vg from GWAS to its theoretical distribution. The number of variants is obtained by the heritability divided by the estimated mean of the exponential distribution. In practice, due to limited sample sizes, there is insufficient power to detect variants with small effects. Therefore the power was taken into account in fitting. Besides considering the most significant variants, we also tried to relax the significance threshold, allowing more markers to be fitted. The effects of false positive variants were removed by considering the local false discovery rates. In addition, we developed an alternative approach by directly fitting the  We developed a methodology to tackle the problem by fitting distributions to the GWAS results. We assumed that the effect sizes of all susceptibility variants in the genome, as measured by the variance explained (Vg), follow an exponential distribution. For binary traits, the variance explained is computed based on the liability threshold model. The model proposes a latent continuous liability, which is assumed to follow a normal distribution with mean 0 and variance 1. The variance in liability explained can be directly interpreted as the locus-specific heritability. The method is described in details in another paper z-statistics.Our aim is to fit the observed distribution of Vg from GWAS to its theoretical distribution. In practice, the sample size of a study is limited and there is inadequate power to detect variants with small effect sizes. As a result, variants with larger effects are over-represented and the Vg of the discovered variants would not follow a standard exponential curve. To correct for this bias, we also considered the statistical power corresponding to every given Vg. In addition, we developed an alternative estimator of the number of variants by directly fitting a suitable distribution to the observed et al.The assumption of an exponential distribution of effect sizes is theoretically justified. Orr et al.et al., we did not construct specific combinations of allele frequencies and ORs, instead we fit appropriate distributions to actual GWAS data to estimate the number of variants. The array of methodologies presented in this paper has not been previously described.The problem of the total number of variants in the genome is rarely addressed. Recently a study by Pawitan not exactly the same as in Park et al. We are considering the entire range of effect sizes while in Park et al the range of effect sizes is limited to those observed for known susceptibility variants. In other words, Park et al. aimed at estimating the total number of variants within the range of effect sizes that have been observed in GWAS conducted to date. On the other hand, we aimed at providing a framework to evaluate the total number of variants underlying a disease, regardless of the effect sizes of variants. The number of variants estimated from Park et al. will therefore be smaller than the number from our approach. For example, for Crohn's disease, Park et al. estimated the number of loci to be ∼150 within the range of effect sizes of known loci while our estimate is around a thousand (as detailed in later sections). Park et al At the time of submission, we also noticed an interesting paper by Park et al. λ is the only parameter that characterize the distribution. The mean is simply given by 1/λ. Therefore the number of susceptibility variants can be derived given λ:We assume that the Vg of all susceptibility variants in the genome follow an exponential distribution. The probability density function (pdf) of an exponential distribution can be expressed asp-values below a certain threshold α. The probability density of the Vg of these observed significant variants is directly proportional to the standard exponential density multiplied by the power:pwr is a function that returns the corresponding power for a given Vg, assuming allele frequency, prevalence, type I error  and sample size are all fixed. Examples of the power-adjusted pdf for different sample sizes are shown in 1 and estimated the corresponding power. Here we take advantage of the fact that the power corresponding to a given Vg is grossly similar regardless of the risk allele frequency  can be found in the appendix.The above pdf can be used for maximum likelihood estimation of λ that gives the closest theoretical mean to the observed sample mean of Vg from GWAS. In other words, we solve the following equation,λ  procedure Typically in GWAS we only consider the variants that withstand a Bonferroni correction  to be significant. However, for fitting the density curve and estimation of 0 given the test statistic.1 given the test statistic, or Pr(H1|Z = z), is given by (1-lfdr). Local fdr is calculated for the z-value of every SNP  in a GWAS. The Vg for each SNP is weighted by 1-lfdr corresponding to that particular SNP. The weighted mean of Vg can be expressed byThe local fdr is the posterior probability of Hz = 0 is often close to or equal to 1, due to limited sample size. The actual number of truly associated variants is usually larger than that predicted by local fdr when z is around 0. Inclusion of all variants will lead to an underestimation of mean Vg, as the weights for SNPs with z-values around 0 will be inappropriately low. Therefore we set an fdr threshold and only considered z-values corresponding to a fdr lower than a particular value. Here we assume that when we restrict our consideration to z-values below the fdr threshold, we can accurately assess the non-null density of z (denoted f1(z) by Efron), and hence “recover” the Vg correponding to these non-null z-values. The statistical power (required for the construction of the power x density curve) is calculated according to the z-value cutoff. Local fdr was calculated by the locfdr program by Efron Although theoretically we may include all SNPs in distribution fitting, it should be noted that the local fdr around z-statistics. Since we assume the Vg of associated variants follow an exponential distribution with parameter λ, we can derive λ. We may then fit the most significant z-values to λ. This is similar to our earlier proposal of fitting observed Vg of significant markers to the power x density curve, but this time the z-values instead of Vg are used for distribution fitting.Another approach is to consider z-value as a sum of the true z-value and a random “noise” component which is standard normal:δ. For simplicity, we first restrict all δ to be positive and denote them by +δ. Let g be a function that converts Vg to δ, ieδ are all positive, a single Vg can only give a single value of δ, in other words, there is a one-to-one correspondence of Vg and δ)Note that the observed z-values do not equal the true z-values. Denoting the true z-value by δ, then+δ to Vg and λ.The density of δ is given byδ is symmetrical i.e. f(δ) = f(−δ), so the above densities are twice of the overall density which includes all values of δ. The overall density of δ is given byNow we consider all values of δ, positive and negative. The distribution of δ and Y:f1(z) is determined by parameter λ only as we integrate over δ.Now we turn to the evaluation of α, the z-value threshold critz is given by N is the total number of markers in the study. The observed significant z-values hence follows a truncated critz and - critz. The pdf of the truncated Since the sample size is limited, we can only detect variants with effect sizes larger than a certain extent. Assuming a Bonferroni correction is used and family-wise error rate is set at z-values isλ can be obtained by numerical maximization of the above likelihood function.The log likelihood for a set of significant 1) for each included marker. Pr(H1) may be estimated by 1 minus the local fdr for each marker. The weighted likelihood isAgain one may wish to relax the significance threshold to allow more variants for distribution fitting. To remove the effects of the false positive variants, we may employ a weighted likelihood approach with weights equal to Pr for the chosen SNPs, resulting in bias in effect size estimates. The selection bias is most prominent when the study has weak power and when the significance threshold is extreme. This is particularly pertinent to genome-wide association studies, in which the association signals are usually weak and very stringent significance thresholds are often imposed to guard against multiple testing. As overestimation of effect size will lead to overestimation of mean Vg, or underestimation of λ, we investigated various statistical methods based on conditional likelihood to correct for the winner's curse.Winner's curse refers to the possible overestimation of effect size of et al.The details of the winner's curse correction approaches applied here were described by Ghosh z-values exceeding a certain threshold c, where c is chosen such that the family-wise error rate (FWER) or false discovery rate are controlled below a certain level. The selection of SNPs passing the significance cutoff distorts the original normal distribution. The conditional likelihood after selection can be expressed by In our case, we are only interested in SNPs with z-values, Zhong and Prentice β and used an equivalent form of the above conditional likelihood in their paper. Here for clarity, we focused on the formulation using the z-statistics. A number of corrected estimates of the effect size have been proposed based on the above result. Below we briefly describe five estimators considered in our simulations.Instead of working with the μ to the observed mean.The first one is the MLE estimatorμ.The second one is the mean of the normalized conditional likelihood,The third estimator is the mean of the first two estimatorsz with median at the observed z-value. The estimator can be expressed aszobs is the observed z-statistic for the SNP under study. Note that Zhong and Prentice β, but since z and β are directly related through the fixed SE (z = β/SE), it does not make any difference if one works with the sampling distribution of z.The fourth estimator produces a sampling distribution of The fifth estimator is a weighted average of the uncorrected regression coefficient estimate  and the standard error (SE) can be estimated from z-values. It should be noted that the SE is in fact a function of the risk allele frequency. For the same level of relative risk and the same z-statistic, a larger SE will result if risk allele frequency is lower. Ignoring the risk allele frequency will result in less accurate estimates of SE. Hence it is recommended that one considers the actual risk allele frequencies if they are available. In view of this limitation, the results that are based on the last correction method (i.e the “MSEmedian” estimators) should be regarded as rough estimates only.All the above correction methods work for binary as well as continuous outcomes. The first four methods just require z-statistics and investigated the performance of the various estimators in recovering the underlying parameter that generates the statistics. The z-statistics were composed of two groups: the first group corresponded to H0 with distribution N ; the second group corresponded to H1 (i.e. truly associated variants) with distribution N, where δ was determined by the parameter of the distribution of Vg as described below.We simulated z-value (denoted δ) can be obtained. Since the same Vg gives rise to similar power and z-values, we simply fix the allele frequency at 0.5.We simulated the Vg of associated variants according to the exponential distribution,z-values, 99.5% of which are null. The non-null z-values were derived from their corresponding exponentially distributed Vg. We assumed a prevalence of 0.001 and a local fdr cut-off of 0.3 throughout. We investigated the performance of a number estimators under three different sample sizes  and four different λ . The root mean squared error from 300 simulations was computed. The root mean squared error is given byB is the number of simulations, i th simulation.In each replication we simulated 100,000 λ were evaluated. without any winner's curse corrections. Lastly, we fit the truncated convolution density z-values in each group.In total fourteen estimators of z and calculation of power given a level of Vg. In fact fewer assumptions are required for continuous traits. For example, no extra information on the prevalence of disease is required.Although our previous discussions focus on variance explained of binary outcomes, the method can easily be extended to continuous traits. The only difference lies in the conversion of Vg to z-statistic and the same Vg would always give rise to the same power. In summary, we have the following relations between z and Vg :n is the sample size.A given level of Vg corresponds to a specific z, where g is the function to convert Vg to z. In the case of a continuous outcome, one can easily obtain a closed form expressionRecall that we require the derivative  methods .λ found in each bootstrap run is recorded and the confidence interval of λ can be obtained from the empirical distribution of λ in bootstrap runs. The non-parametric bootstrap approach takes into account of all possible sources of uncertainties, such as uncertainties in the estimate of the odds ratios and allele frequencies and variation brought about by winner's curse correction methods. This approach is relatively straightforward and can be applied to any of the proposed estimation methods. However, the raw data on the genotypes and phenotypes of all subjects are required.Non-parametric bootstrap can be applied to all proposed estimation methods. for construction of CIs. To perform non-parametric bootstrap, one resamples with replacement the subjects from the case-control dataset and re-calculates the test statistic for each SNP in each run. Then repeat the entire procedure of distribution fitting as described above. The z-values, based on the estimated value of λ. Then we could re-estimate λ in each simulated sample and obtain a distribution for λ. For example, we considered fitting the pdf of the alternative z-values, or z-values based on the pdf z-values is to first derive the cdf by integrating z-values can be generated from the inverse cdf.Parametric bootstrap is another way to obtain the CI when raw data is not available. The basic idea is to generate many simulated samples (each with the same of number of observations) from the known distribution of Vg or As the bootstrap methods are more computationally intensive than analytic methods, we will focus on the latter in simulations and applications to a few real datasets. Bootstrap methods however may have better performance particularly when the number of observations is small. If access to the raw data is available, non-parametric bootstrap is also applicable and has the merit of being free from distributional assumptions.A common analytic approach for calculating CI is to assume normality of the parameter and obtain the SE by inversion of the Fisher information matrix. Alternatively, one may obtain CI by inverting hypothesis tests, such as the maximum likelihood ratio test (MLRT).λ by fitting to the distribution of observed Vg with correction for winner's curse. Note that we assume the actual Vg is exponentially distributed and hence fit the power x exponential density curve to the data. What we obtain from association studies however is not the actual Vg but Vg with random sampling variation. Put it another way, the Vg is derived from the allele frequencies and odds ratios. Both of them have sampling variations  and would not be identical when we repeat the study. Nevertheless our approach is to fit the estimated “true” Vg to the exponential density multiplied by power. The uncertainties of the allele frequencies and ORs have not been dealt with in the model.These analytic methods for calculation of CI however cannot directly be applied when we estimate sampling distribution of Vg, then the analytic approaches for obtaining CI are applicable. But since the form of this distribution is difficult to obtain, we cannot obtain CIs by analytic means directly. Correction for winner's curse introduces further variations. The effect to the variance and CI of the final λ estimate is not straightforward. In addition, different ways of winner's curse correction have different effects on the variances of λ, as shown in our simulations.Hypothetically, if one is able to fit the observed data to the actual z-value.Analytic means of calculating CI can be applied when we fit the significant variants to the convolution density x represent the observed data, λ is the parameter to estimate and Two standard analytic methods in deriving the confidence interval were used. The first one is based on inverting the observed Fisher information evaluated at the MLE. It may be expressed as the negative of the 2nd derivative of the log-likelihood function,not rejected, or more simply, the lambdas that are not significantly different from the MLE. The confidence interval with coverage 1−The second approach is derived from the maximum likelihood ratio test (MLRT). It is based on the duality of hypothesis testing and interval estimation. To construct a confidence interval with coverage 1−z-values are generated as previously described. Ninety-five percent confidence intervals of the parameter λ were computed using two analytic methods (inversion of Fisher information matrix and MLRT), for markers passing Bonferroni correction and for those passing the local fdr threshold of 0.3. We evaluated coverage probabilities and the width of the CIs.It is less clear whether these analytic approaches would work if we worked with weighted maximum likelihood. To assess the coverage probability of the proposed analytic CIs, we performed a brief simulation study on a sample size of 3500 cases and 3500 controls and lambdas of 1000, 2000 and 3000. One thousand simulations were carried out. In each simulation run, random The above distribution fitting methods were applied to a few real disease examples. As estimation of the local fdr requires all summary statistics to be available, we focused on studies that had full summary statistics released before. We included three large-scale studies, one on lipid levels p-value threshold (<5×10−5) and the 2nd-stage threshold (<0.0008 or 0.05/63 by Bonferroni correction) given the effect size. λ was estimated by the same distribution fitting procedure as detailed previously.For the Crohn's disease dataset not exactly be the true model in real scenarios (please see the discussion section for further details). Particularly, the effect size distribution of complex traits may be more skewed towards zero than an exponential distribution, as suggested in Park et al. z-values . The mean of a gamma distribution is simply the product of the shape and scale parameters.Although we have focused on the exponential distribution in this study, we noted that this is only theoretical and may p-value threshold and then removing the false positive effects by local fdr weighting, the root mean squared error can be reduced. The mean, bias and SD of different estimators were shown in λ is large) and the sample size was not large. For example when Ncase = Nctrl = 3500 and λ is 4000, the SD of the fdr-adjusted estimators was only about one-third of the SD of the Bonferroni- adjusted counterparts. The bias of the fdr-adjusted estimators were also in general smaller than that of the Bonferroni- adjusted ones, but for the last two  rather than the raw data, although the availability of raw data allows non-parametric bootstrap to evaluate confidence intervals. If only the z or p-values of the most significant markers are available, the models can still be fit, but one should be aware that distribution fitting on a small number of markers is often unstable.One advantage of our methodology is that all proposed estimators require only the summary statistics . A rare variant with a large effect size may have similar Vg as a common variant with smaller effect size. For example, under a prevalence of 0.01, a variant with risk allele frequency (RAF) 0.5 and allelic RR 1.1 has the same Vg (0.0639%) as another variant with RAF 0.0005 and RR 6.348 with Vg . The exponential distribution of Vg is compatible with numerous rare variants with large RR or numerous common variants with small RR, or a combination of both. Since current GWAS panels only focus on common variants, we are estimating the parameter of the exponential distribution based on the set of common SNPs, and hoping the result can be extrapolated to rare variants.quantify the number of susceptibility variants, albeit approximately, in a statistical framework. We also noted that GWAS have been relatively successful for the traits studied here: over 20 loci have been identified for type 2 diabetes and around 30 loci have been identified for Crohn's disease We have also applied the methodology to a number of real disease traits. The results suggested that roughly hundreds of susceptibility variants underlie these traits. This is not particularly striking, as many researchers might have expected that a large number of variants are implicated from the polygenic model for complex diseases. However, we have shown how to To accurately infer the number of total susceptibility variants in the genome from GWAS data is a very difficult problem, and the methodology presented here should only be interpreted as a rough estimate based on the exponential distribution assumption. We stress that the work presented here is not a perfect solution to the problem. Instead, we made our best attempt to estimate the number of variants, and more importantly, we hoped to provide a useful and rigorous statistical framework to attack the problem.λ underestimated. In addition the strong correlation may distort the assumed exponential distribution of effect sizes. One way to alleviate this problem is to prune the SNPs before analyses so that they are roughly independent. On the other hand, pruning reduces the number of markers available for distribution fitting and will increase the variability of the estimates, especially when the effect sizes are small and there are few markers considered significant even in the entire set of genotyped SNPs. There is no perfect solution to this sort of bias-variance tradeoff. However, as the sample size becomes larger, more SNPs will pass the significance threshold and there will be adequate SNPs for stable parameter estimates even if the SNPs are pruned prior to analysis.One difficulty in dealing with GWAS data is that the SNPs are correlated, while most distribution fitting methods in statistics apply to independent observations. Often clusters of SNPs may show high significance but may just contain one or few true independent signals. As a result of the “redundant” significant signals, the mean Vg may be overestimated and Also, the significant associations in a GWAS may not be causal variants themselves, but maybe merely tagging them. We have to assume that the variance explained by significant results in a GWAS is close to that of the causal variants if they were typed.Current GWAS technologies mainly aim at capturing common variants. As a result, the best thing we can do is to infer the total number of underlying based on the effect sizes of common variations. The contribution of rare variants and structural variations to complex diseases is largely unexplored in the literature. In addition, gene-gene or gene-environmental interactions may also play a role in the etiology of complex diseases, but our current understanding regarding interactions is still very limited. To make predictions on the total number of susceptibility variants in the genome, inevitably we have made the assumption that the other variants that are not captured share a similar distribution of Vg as common variants. Of course this assumption may not work very well in practice. Nonetheless, we think this limitation may become less of a concern in the future when studies accumulate and technologies improve. While now we can only fit the Vg of common variants, with more rare and structural variants discovered in the future, their effect sizes may also be fit using the proposed framework. Interactions may be regarded as multi-locus genotypes and their Vg can be readily calculated and distribution fitting can be done as well. The framework we presented does not pose any restrictions on the actual nature of the susceptibility variants. With accumulation of more studies we will hopefully be able to obtain more reliable estimates considering a greater variety of genetic variations.We assumed an exponential distribution of Vg in this study, which is justified by theories in adaptation not be exactly true in practice. For example, Park et al.Despite the advantages, we must point out that the exponential assumption is a theoretical prediction after all, and may entire range of effect sizes rather than the observed range, which is a bit trickier. The unobserved effect sizes are likely to be smaller than the observed ones and cannot be modeled easily. Note that non-parametric methods are usually highly flexible. They can fit the observed data-points well (and may perform well within the range of effect sizes observed). However, flexible models also increase the risk of overfitting. They may be less reliable at predicting future data-points as they may magnify the “noise” or random fluctuations in the original data. The danger of overfitting is exaggerated when we have to extrapolate beyond the observed spectrum of effect sizes. The risk of overfitting is also heightened when the number of loci (i.e. data-points) available for fitting is small in the first place, as explained earlier.As first mentioned in the introduction, Park et al. As a result, we have focused on the more restrictive parametric models in the current study, and in particular the exponential model. This is not because the exponential model is perfect, but it is the only parametric model with theoretical support and represents the best “educated guess” of effect size distribution in our view. One may change the exponential assumption to other models such as gamma distributions, as was done in this study. With greater sample sizes for association studies and more loci available for fitting, one may be able to reliably fit statistical distributions with more free parameters, for example the gamma distribution (which is very flexible) with free shape and rate parameters. The reliance on the exponential distribution can then be relaxed. The framework presented in this paper can potentially be extended to deal with other types of distributions.It should be noted that our methodology requires an external estimate of the total heritability. As there are many different statistical approaches and designs that are available for heritability studies, one may need to be aware of the limitations of different heritability estimates. For example, twin and adoption studies can separate shared environmental from shared genetic factors in families, while family-based studies (e.g. those based on siblings and parent-offsprings only) cannot make a distinction between the two types of shared effects.λ) We have proposed an fdr-based approach to relax the significance threshold so that more markers are available for fitting. Simulations showed that this approach had superior performance when compared to the conventional way of including only the markers that passed the Bonferroni threshold. In practice, confounding factors such as population stratification may be present to increase the false positive rate. Many methods such as principal component analysis are available for correcting population stratification not considered and as admitted by the author, it is assumed that all SNPs yet to be found have smaller effects than the weakest one discovered to date. This will lead to an overestimation of the number of variants.Goldstein et al.p values up to 0.5) from a schizophrenia GWAS demonstrate predictive power collectively. It will be much harder to estimate the number of variants making up this combined effect as the Vg of each variant is probably tiny. For instance, it may be difficult to distinguish between 100 variants having an average Vg of 0.01% versus 10 variants having an average Vg of 0.1%.Estimation of the number of variants underlying a trait is a very challenging task. We believe that for most complex diseases, the Vg of susceptibility variants are likely to be rather small. It may be possible to investigate the aggregate effect of these variants, for example Purcell In conclusion, we have developed a novel statistical approach to estimate the number of susceptibility variants in the genome. The performance of different proposed estimators were tested by simulations and applied to some real data examples. Despite the limitations, we believe this study is a useful step towards the understanding of the genetic architecture of complex diseases.Text S1Supplementary methods: Equivalence of the maximum likelihood and method of moments estimates of lambda and the approach to continuous traits.(0.10 MB PDF)Click here for additional data file.Table S1Power estimates for different risk allele frequencies under the same Vg.(0.15 MB DOC)Click here for additional data file.Table S2Mean and bias of different estimators from simulations.(0.11 MB DOC)Click here for additional data file.Table S3SD of different estimators from simulations.(0.07 MB DOC)Click here for additional data file.Table S4Estimates and confidence intervals of lambda for a few complex traits, with inclusion threshold based on Bonferroni correction.(0.04 MB DOC)Click here for additional data file.