Christopher J Mattocks
email: chris.mattocks@salisbury.nhs.uk

Michael A Morris

Gert Matthijs

Elfriede Swinnen

Anniek Corveleyn

Els Dequeker

Clemens R Müller

Victoria Pratt

Andrew Wallace

Mike Zoccoli

Jana Camajova

Petra Křenková

Patricia Norambuena

Alexandra Stambergova

Milan Macek

Patrick M Bossuyt

Els Voorhoeve

Bert Bakker

Sarah Berwouts

Tom Janssens

Ivo Salden

Trudi Mcdevitt

David Barton

Jean Amos-Wilson

A standardized framework for the validation and verification of clinical molecular genetic tests

Keywords: Validation, verification, molecular genetics, statistics, diagnostic

The validation and verification of laboratory methods and procedures prior to their use in clinical testing is an essential component of providing a safe and useful service to clinicians and patients. This paper outlines the principles of validation and verification in the context of clinical human molecular genetic testing. We describe implementation processes, types of tests and their key validation components, and suggest some relevant statistical approaches that can be used by individual laboratories to ensure that tests are performed to defined standards.

Introduction

The process of implementing a molecular genetic test for diagnostic use is complex and involves many levels of assessment and validation. The key components of the process, as detailed by the ACCE framework are analytical validation, clinical validation, clinical utility and consideration of the ethical, legal and social implications of the test [START_REF] Haddow | ACCE: A Model Process for Evaluating Data on Emerging Genetic Tests[END_REF] . After making a decision to set up a diagnostic test, the technology to be used must be chosen and built into a suitable laboratory process. The development stage involves assessment of both the diagnostic and technical utility of the process to ensure that the measurements obtained are relevant to the diagnostic question [s] and that the analyte[s] can be unambiguously identified (i.e. that there are no confounding factors). The final stage of the laboratory process is to determine whether the performance of the test, in terms of accuracy, meets the required diagnostic standards. Whether this is achieved by performing analytical validation or verification depends on the existence of a suitable performance specification that details the expected accuracy of the test under given conditions. The results of the analytical validation or verification determine if, and how, the test will be implemented and set the requirements for performance monitoring (ongoing validation) of the test. A simplified process diagram illustrating these concepts is given in Figure 1.

The validation or verification of methods, as defined in Table 1, are formal requirements for the accreditation of laboratories according to the two major international standards applicable to genetic testing laboratories, ISO 15189 [START_REF]International Organization for Standardization: Medical laboratories -Particular requirements for quality and competence[END_REF] and ISO 17025 [START_REF]International Organization for Standardization: General requirements for the competence of testing and calibration laboratories[END_REF] . Although the general requirements are clearly stated (Table 1), the standards provide very little guidance about the detailed requirements or procedures.

To provide more detailed and specific guidance, Eurogentest [START_REF] Eurogentest | EU Contract[END_REF] setup a working group comprising clinical and laboratory scientists and experts on quality assurance and statistics from both Europe and the USA. The aims were to develop a framework for validation that could be widely implemented in laboratories to improve the overall quality of genetic testing services, while respecting the need for flexibility imposed, for example, by regional requirements and regulations, as well as practical constraints such as test volume and resources. In a recently generated parallel initiative, Jennings et al [START_REF] Jennings | Recommended principles and practices for validating clinical molecular pathology tests[END_REF] have provided a thorough discussion of FDA regulation together with a good review of validation procedures.

However specific interpretation of the standards and practical guidance for molecular genetic tests is still lacking. In this paper we propose a generic scheme for the validation and verification of molecular genetic tests for diagnostic use.

Scope

This paper is specifically focussed on processes involved in analytical validation and verification of tests in human molecular genetics so as to provide working detail the first component of the ACCE framework [START_REF] Haddow | ACCE: A Model Process for Evaluating Data on Emerging Genetic Tests[END_REF] . These processes seek to confirm that a particular laboratory process or test delivers reliability consistent with the intended diagnostic use.

Analytical validation/verification relates only to laboratory processes, and makes no assessment of the how the decision to set up a test is made, the clinical validation, clinical utility or the ethical, legal and social implications of the test [START_REF] Haddow | ACCE: A Model Process for Evaluating Data on Emerging Genetic Tests[END_REF] . In particular, the clinical relevance of the test and the suitability of the chosen measurements with respect to diagnosing a particular genetic disorder are left to professional judgement.

There is much debate about the exact boundary between development and validation, and good cases can be made for different divisions. For the purpose of simplicity we have defined a definitive boundary placing all concepts that relate to test utility in development (i.e. out of scope) and parameters relating to test accuracy in validation (i.e. within scope).

These limitations of the scope should not be taken as assigning different importance to the various processes; making clinically-useful and appropriate measurements is clearly critical to setting up a valid diagnostic test. For this reason we have included a brief section outlining the processes involved and important factors that should be considered at the development stage.

Although we are concerned with appropriate use of statistics and sample sizes, this paper is not intended to be a treatise on the subject, but a practical guide for diagnostic molecular geneticists to aid them in designing, performing and reporting suitable validation or verification for the tests they wish to implement. References have been provided where more complex statistical concepts are involved, but it is recommended that the advice of a statistician is sought in case of doubt. Above all, we seek to promote a pragmatic approach; whilst validation and verification must be carefully considered, test implementation must also be achievable and not overburdening.

Whilst there is much literature addressing validation on a more general level [START_REF] Prence | A practical guide for the validation of genetic tests[END_REF][START_REF]QSOP23 -Commercial and inhouse diagnostic tests: evaluations and validation[END_REF][START_REF] Eurachem | The fitness for purpose of analytical methods a laboratory guide to method validation and Related Topics[END_REF] , we propose a first attempt to identify and organize the components required for validation/verification in the context of molecular genetic diagnostics and have consequently included some simplification of statistical principles and their interpretation. It is intended that this paper be a starting point for the ongoing development of validation/verification guidelines that will necessarily become more sophisticated as knowledge and experience in the area increases and scenarios that are not adequately covered by the present paper are identified. Although these recommendations are aimed primarily at molecular genetic testing, we believe the principles and concepts are also applicable in the context of cytogenetics.

To help guide the validation process and provide a format for recording validations, a standardized validation pro forma (template) has been provided in the Supplemental data. An initial draft of this form was developed from an amalgamation of standard forms used in a range of small and large laboratories that undertake genetic testing. This prototype underwent a field trial to assess its utility and ease of use and appropriate amendments made. We recognise that a single format is unlikely to suit all laboratories so we recommend this form should be used as a starting point for the development of a suitable format for local needs. This paper can be used as detailed explanatory notes for the validation form.

The implementation process

Development

The purpose of development is to establish a testing procedure and broadly show that it is fit for the intended purpose, in terms of what is being tested and of the desired laboratory procedure. This involves defining the analyte[s] to be tested and designing an appropriate methodology, including any assay-specific reagents (e.g. primers), controls and a testing work-flow. The development process should be used to gain the necessary experience with the test, with the aim of identifying any critical parameters that may affect performance and any necessary control measures and limitations that need to be considered. Examples of critical parameters may include primer design, location of known polymorphisms, G+C content of the region of interest, fragment length, type of mutations to be detected and location of mutations within fragments. Suitable control measures might include the use of positive, negative and no-template controls, running replicates of the test and a quality scoring system. It therefore follows that the amount of development required will depend on how novel the testing procedure is, both on a general level (i.e. in the literature) and within the laboratory setting up the test. For example, development of a sequencing test for a new gene in a laboratory with extensive experience in sequencing may simply be a case of primer design, whereas setting up an entirely new methodology would require much more extensive work.

Assessment of utility

Before a test can be validated it is necessary to establish (a) that the particular measurements are diagnostically useful and (b) that the correct analyte[s] and only the correct analyte[s] are being measured. This could involve, for example, ensuring that primers do not overlay known polymorphisms in the primer binding site and that they are specific to the target of interest. It should be noted that use of a CE-marked kit does not preclude assessment of utility; care should still be taken that the test is measuring suitable analyte[s] for the intended purpose since in vitro diagnostic device (IVDD) compliance relates only to technical performance and not clinical or diagnostic validity. Three other critical concepts that should be considered at this stage:

• Selectivity

How good is the method at distinguishing the signal of the target from that of other components? For example, a PCR product run on a denaturing polyacrylamide gel to detect the presence of the CFTR Phe508del (F508del) mutation associated with cystic fibrosis will also detect the rarer Ile507del (I507del) mutation, without distinguishing between them. For most genetic tests, selectivity issues are best avoided by careful design (e.g. BLAST [START_REF] Zhang | A greedy algorithm for aligning DNA sequences[END_REF] primers to avoid non-specific amplification) or by applying adapted controls and/or limitations.

• Interference

Are there any substances whose presence in the test can affect the detection of the target sequence? If so will this cause the reaction to fail or is there a possibility of an incorrect result? For most genetic tests this is likely to relate to substances that cause a reaction to fail (e.g. heparin or ethanol in a DNA sample as a result of the stabilisation or extraction procedure). While failures may not generate false results, there can be issues relating to the utility and timeliness of tests if failure rates are too high. In situations where interference could cause incorrect results, great care needs to be taken to avoid interfering substances, for example by running a pre-test quality check on samples or including more controls.

Because of their complex nature, multiplex assays are particularly susceptible to interference which could give rise to incorrect results. Validation and verification of this type of assay can be particularly demanding and is beyond the scope of this paper.

The Clinical Laboratory Standards Institute (CLSI) has published a guideline which deals comprehensively with this specialist topic [START_REF] Wilson | Validation and verification of qualitative multiplex nucleic acid assays[END_REF] .

• Carry-over [cross-contamination]

This relates to residual products from previous or concurrent analyses that may be introduced into an assay (e.g. via a contaminated pipette). Stringent procedural precautions should be employed as a matter of routine to minimize the risk of such cross-contamination. In particular, physical separation of pre-and post-PCR areas for both reagents and laboratory equipment is critical. Other controls / precautions may include the use of no-template controls and uracil-N-glycosylase treatment [START_REF] Longo | Use of uracil DNA glycosylase to control carryover contamination in the polymerase chain reaction[END_REF][START_REF] Pruvost | Minimizing DNA contamination by using UNGcoupled quantitative real-time PCR on degraded DNA samples: application to ancient DNA studies[END_REF][START_REF] Hartley | Dealing with contamination: enzymatic control of carryover contamination in PCR[END_REF] .

Performance specification

Once a suitable test procedure has been established and it is judged that there is sufficient local knowledge of critical parameters, it is necessary to demonstrate that: (a) the test performs to a suitable level of accuracy for the intended purpose: that is, it produces results that can satisfactorily answer the clinical question allowing for uncertainty of measurement; and that (b) this level of accuracy is routinely maintained.

The level of testing required is dependent on the availability of a suitable performance specification. This should define all the test conditions necessary to achieve a particular level of accuracy together with measurable parameters that can be used to show that this is the case -specifically: (a) An estimate of the test accuracy including measurement uncertainty (e.g. confidence limits).

(b) Control measures required to ensure routine maintenance of accuracy.

(c) Limitations on critical parameters that will ensure the desired level of accuracy.

For validation of a specific test, limitations may include factors like input DNA concentration or details on how DNA extraction needs to be performed. When a technology is being validated (as opposed to a specific test), there may also be limitations related to physical parameters such as PCR fragment length or G+C content. It should be stressed that a performance specification will only apply within particular limits of certain critical parameters, so care should be taken to ensure the new test falls within these limits. For example, the performance specification for a hypothetical method for mutation scanning (>95% sensitivity for mutations in fragments <300bp long and 25-60% G+C content) would not be applicable to a new test involving a 400bp fragment or fragments with 70% G+C content.

Validation

Full validation is required when there is no suitable performance specification available, for example with novel tests or technologies. This process involves assessing the performance of the test in comparison to a 'Gold Standard' or reference test that is capable of assigning the sample status without error (i.e. a test that gives 'true' results). In simple terms validation can be seen as a process to determine if we are "performing the correct test". In the field of medical genetics, with the almost complete absence of reference tests or certified reference materials, the reference should be the most reliable diagnostic method available. It is worth noting that the gold standard does not have to comprise results from a single methodology; different techniques could be used for different samples and in some cases the true result may represent a combination of results from a portfolio of different tests. To avoid introducing bias, the method under validation must not, of course, be included in this portfolio.

Validation data can be used to assess the accuracy of either the technology (e.g. sequencing for mutation detection) or the specific test (e.g. sequencing for mutation detection in the BRCA1 gene). Generally speaking, the generic validation of a novel technology should be performed on a larger scale, ideally in multiple laboratories (inter-laboratory validation) and include a much more comprehensive investigation of the critical parameters relevant to the specific technology, to provide the highest chance of detecting sources of variation and interference.

Verification

If a suitable performance specification is available, it is only necessary to establish that the new test is meeting this specification within the laboratory; this process is called verification.

In simple terms verification can be seem as a process to determine that "the test is being performed correctly". Verification should usually be appropriate for CE-marked IVDD-compliant kits, but care should be taken to ensure that the performance specification is sufficient for the intended use of the kit, particularly with kits that are self-certified. Most diagnostic genetic tests are classified by the IVD directive as 'low risk' and can be self-certified by the manufacturer without assessment by a third party. Such tests can be identified by the absence of a number following the CE mark (Article 9: IVDD Directive 98/79/EC) 14,[START_REF]Conformity Assessment Procedures under the In Vitro Diagnostic Medical Devices Directive 98[END_REF] . If, at any stage, the test procedure associated with the performance specification is modified (e.g. if reaction volumes of a CE-marked kit are reduced), verification is not appropriate and validation is required [START_REF] Camajova | Variability in the use of CE-marked assays for in vitro diagnostics of CFTR gene mutations in European genetic testing laboratories[END_REF] .

Other applications of verification may include a new test being implemented using a technology that is already well established in a laboratory (e.g. a sequencing assay for a new gene), or a test for which a suitable performance specification is available from another laboratory where the test has already been validated. In all cases, it is essential that laboratories obtain as much information as possible concerning the validation that has been performed.

Reporting validation and verification

The plan, experimental approach, results and conclusions of the validation or verification should all be recorded in a validation file along with any other relevant details (see section 6).

In addition the validation plan and outcome should be formally reviewed and approved.

When reporting validations or verifications in peer-reviewed publications, it is strongly recommended that the STARD initiative (Standards for Reporting of Diagnostic Accuracy) [START_REF] Bossuyt | Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy[END_REF] is followed as far as possible.

Performance monitoring (Ongoing validation)

Once a test validation has been accepted (i.e. the utility and accuracy has been judged to be fit for the intended diagnostic purpose) it is ready for diagnostic implementation. However, this is not the end of performance evaluation. The performance specification derived from the validation should be used to assess the 'validity' of each test run and this information should be added to the validation file at appropriate intervals. In many cases, the accumulation of data over time is an important additional component of the initial validation which can be used to continually improve the assessment of test accuracy and quality. The ongoing validation should include results of internal quality control, external quality assessment, and non-conformities related to the test or technique as appropriate.

Types of test

The core aim of validation is to demonstrate that the accuracy of a test meets the diagnostic requirements. Essentially all tests are based on a quantitative signal, even if this measurement is not directly used for the analysis. Whilst measuring the proportion of a particular mitochondrial variant in a heteroplasmic sample is, for example, clearly quantitative, the presence of a band on a gel is commonly considered as a qualitative outcome. However, the visual appearance of the band is ultimately dependent on the number of DNA molecules that are present, even though a direct measurement of this quantity is rarely determined. These differences in the nature of a test affect how estimates of accuracy can be calculated and expressed.

For the purposes of this paper, we are concerned with two types of accuracy. Determining how close the fundamental quantitative measurement is to the true value is generally termed "analytical accuracy". However, it is often necessary to make an inference about the sample or the patient based on the quantitative result. For example, if the presence of a band on a gel signifies the presence of a particular mutation, test results are categorized as either "positive" or "negative" for that mutation, based on the visible presence of the band. Such results are inferred from the quantitative result, but are not in themselves quantitative. Determination of how often such a test gives the correct result is termed "diagnostic accuracy". The term diagnostic accuracy is generally used to describe how good a test is at correctly determining a patient's disease status. However, genotype does not necessarily equate directly to disease status (phenotype) for various reasons including incomplete penetrance / modifying factors or simply that the patient is pre-symptomatic. The purpose of these guidelines is to enable laboratories to establish how good their tests are at correctly determining genotype: clinical interpretation of the genotype is not considered in this context. Therefore, for the purposes of this paper, the term diagnostic accuracy will be taken to relate exclusively to the ability of a test to correctly assign genotype irrespective of any clinical implication.

We distinguish three broad test types (quantitative, categorical and qualitative) that can be sub-divided into five groups according to the method for interpreting the raw quantitative value to give a meaningful result.

The following sections discuss each of these test types in more detail and provide guidance on appropriate measurement parameters in each case. A summary of the characteristics of the different test types and examples is given in table 2, together with recommendations for appropriate measurement parameters and timing of validation.

Type A Quantitative tests

For a quantitative test, the result is a number that represents the amount of a particular analyte in a sample. This can be either a relative quantity, for example determining level of heteroplasmy for a particular mitochondrial allele, or an absolute quantity, for example measurement of gene expression. In either case the result of a quantitative test can be described as continuous since it can be any number [between two limits], including decimal numbers.

trueness and precision [START_REF] Hauck | Making sense of trueness, precision, accuracy and uncertainty[END_REF][START_REF] Menditto | Understanding the meaning of accuracy, trueness and precision[END_REF] . Trueness expresses how close the test result is to the reference value. Typically, multiple measurements are made for each point and the test result is taken to be the mean of the replicate results (excluding outliers if necessary). Since quantitative assays measure a continuous variable, mean results are often represented by a regression of the data (a regression line is a linear average). Any deviation of this regression from the reference (i.e. the line where reference result equals test result) indicates a systematic error, which is expressed as a bias (i.e. a number indicating the size and direction of the deviation from the true result).

There are two general forms of bias. With constant bias test results deviate from the reference value by the same amount regardless of that value. With proportional bias the deviation is proportional to the reference value. Both forms of bias can exist simultaneously (figure 2).

Whilst measurement of bias is useful (figure 3), it is only one component of the measurement uncertainty and gives no indication of how dispersed the replicate results are (i.e. the degree to which separate measurements differ). This dispersal is called precision and provides an indication of how well a single test result is representative of a number of repeats. Precision is commonly expressed as the standard deviation of the replicate results, but it is often more informative to describe a confidence interval (CI) around the mean result. For example, a result for a test investigating mutation load in a tumour sample might be described as 7% (95% CI: 5-10%).

Precision is subdivided according to how replicate analyses are handled and evaluated. Here there is some variability in the use of terminology, however, for practical purposes, we recommend the following scheme based on ISO 3534-1 [START_REF]Statistics-Vocabulary and Symbols-Part 1: General Statistical Terms and Terms Used in Probability[END_REF] , and the International Vocabulary of Metrology [START_REF]International Vocabulary of Metrology -Basic and General Concepts and Associated Terms VIM, 3rd edition[END_REF] :

Repeatability refers to the closeness of agreement between results of tests performed on the same test items, by the same analyst, on the same instrument, under the same conditions in the same location, and repeated over a short period of time.

Repeatability therefore represents "within-run precision".

Intermediate Precision refers to closeness of agreement between results of tests

performed on the same test items in a single laboratory but over an extended period of time, taking account of normal variation in laboratory conditions such as different operators, different equipment and different days. Intermediate precision therefore represents "within-laboratory, between-run precision" and is therefore a useful measure for inclusion in on-going validation Reproducibility refers to closeness of agreement between results of tests carried out on the same test items, taking into account the broadest range of variables encountered in real laboratory conditions, including different laboratories.

Reproducibility therefore represents "inter-laboratory precision" 22

In practical terms internal laboratory validation will only be concerned with repeatability and intermediate precision and in many cases both can be investigated in a single series of well designed experiments. Reduced precision indicates the presence of random error. The relationship between the components of analytical accuracy, types of error and the metrics used to describe them is illustrated in figure 3.

Any validation should also consider robustness which, in the context of a quantitative test, could be considered a measure of precision. However, robustness expresses how well a test maintains precision when faced by specific, designed "challenge", in the form of changes in pre-analytic and analytic variables. Therefore, reduced precision does not represent random error. Typical variables in the laboratory include sample type (e.g. EDTA blood, LiHep blood), sample handling (e.g., transit time or conditions), sample quality, DNA concentration, instrument make and model, reagent lots, and environmental conditions (e.g., humidity, temperature). Appropriate variables should be considered and tested for each specific test.

The principle of purposefully challenging tests is also applicable to both categorical and qualitative tests and should be considered in these validations as well. Robustness can be considered as a useful prediction of expected intermediate precision.

Since trueness and precision represent two different forms of error, they need to be treated in different ways. In practice, systematic error or bias can often be resolved by using a correction factor; constant bias requires an additive correction factor whilst proportional bias requires a multiplicative correction factor. For example, results from a test that has +5% bias can be multiplied by 100/105. Random error, in contrast, cannot be removed but its effects can generally be reduced to acceptable levels by performing an appropriate number of replicate tests.

For the purposes of this paper a basic understanding of the concepts described above is the main objective. However, it is worth outlining some of the complexities that can arise in estimating the analytical accuracy of quantitative tests. In molecular genetics, quantitative measurements are most often relative i.e. two measurements are taken and the result is expressed as a proportion (e.g. the percentage of heteroplasmy of a mitochondrial mutation).

In such cases it is preferable to perform both measurements in a single assay to minimize the effects of proportional bias, since the assay conditions are likely to affect both measurements in a similar way.

If the measurements must be taken in separate assays, each measurement is effectively an absolute measurement and must be quantified in comparison to a set of calibration standards run with each test batch. In this scenario it is important to assess the variation in each test/standard pair since even minor variation can dramatically affect the overall analytical accuracy. This is most effectively achieved by monitoring the efficiencies of the two reactions over time [START_REF] Sivaganesan | A Bayesian method for calculating real-time quantitative PCR calibration curves using absolute plasmid DNA standards[END_REF] .

For quantitative tests, particularly those requiring absolute quantification, it is most effective to estimate analytical accuracy on an ongoing basis by running a set of calibration standards (standard curve) with each batch or run. In this case it is important that linearity is evaluated [START_REF] Tholen | Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach[END_REF] and that the lower and upper standards are respectively below and above the expected range of the results since precision cannot be assessed on extrapolated results.

Where possible, calibration standards should be traceable to absolute numbers or to recognized international units.

Other factors that may need to be evaluated include the limit of detection defined as the lowest quantity of analyte that can be reliably detected above background noise levels and the limits of quantification which define the extremities at which the measurement response to changes in the analyte remain linear.

A detailed description of the determination of these limits is given in CLSI document EP17-A [START_REF] Tholen | Protocols for Determination of Limits of Detection and Limits of Quantitation; Approved Guideline[END_REF] . In situations where test results are likely to fall close to these extremities or there are It should be noted that limit of detection is sometimes referred to as 'sensitivity' i.e. how sensitive a methodology is to detecting low levels on a particular analyte in a large background. Usage of the term sensitivity in this context should be avoided as it may be confused with sensitivity described in section 4.3 (i.e. the proportion of positive results correctly identified by a test).

It can be seen that the analysis of all but the simplest quantitative assays can be complex and it is recommended that statistical advice is sought to determine what factors need to be measured and how this is best achieved.

Categorical tests

Categorical tests (sometimes referred to as semi-quantitative [START_REF]COFRAC Guide de validation de méthodes en biologie médicale[END_REF] ) are used in situations where quantitative raw data, which could have any value including decimals, are grouped into categories to give meaningful results. For example, fluorescent capillary analysis might be used to determine the size of PCR products (in base pairs) by analysing the position of the peaks relative to an internal size standard. The quantitative results from this analysis will include numbers with decimal fractions but the length of the product must be a whole number of base pairs; a fragment cannot be 154.3 bp long. Therefore cut-offs must be used to assign the quantitative results to meaningful categories. The parameters used to describe the estimates of analytical accuracy for a quantitative test (figure 3) can be used to describe the performance of the categorical test in much the same way. However, there is an added level of complexity here since the primary (quantitative) result is manipulated (i.e. placed into a category). The categorized results for these tests retain a quantitative nature (although this is distinct from the quantitative primary data) and, in practice, trueness and precision can be determined at the category level as well as at the level of the primary result. We divide categorical tests into two subgroups, depending on the number and type of categories and the degree of importance placed on knowing how accurate a result is (figure 4).

Type B Categorical tests

This group includes tests where there are [essentially] unlimited categories, such as the sizing example cited above. In this case, each cut-off forms the upper boundary of one category and lower boundary of the next so that all results can be categorized (except for those that have failed). Generally, less stringent levels of accuracy are acceptable with this type of test. In this case, estimation of precision can be performed before implementation (e.g. ±1bp ), whilst trueness is dealt with by running a standard curve with each experiment (i.e. a size standard).

Type C Categorical tests

When the number of pre-defined categories is limited, for example with allele copy number determination, accuracy tends to be critical and a more definitive approach is often required.

The most informative way to express accuracy for this type of test is the probability that a particular [quantitative] result falls into a particular category. Here the cut-offs are defined at particular level[s] of probability, typically 95% CI, which means that each category has its own unique upper and lower boundaries with regions between where results would be classified as un-reportable.

Results can be assigned to the appropriate categories by a process of competitive hypotheses testing. For example, a test to determine constitutional allele copy number has three expected results: normal [2n], deleted [n] and duplicated [3n]. The odds ratios p(2n):p(n) and p(2n):p(3n) can be used to assign results (figure 5). It should be noted that mosaic variants may give rise to intermediate values; detection of mosaics should be considered under quantitative tests. A good example of this methodology is described in the MLPA spreadsheet analysis instructions freely available from NGRL (Manchester) [START_REF] Wallace | MLPA analysis spreadsheets -User Guide[END_REF] . In this case the validation of accuracy is predominantly done on an ongoing basis by running replicate control assays during the actual test run to determine the extent of the random error observed within that particular run.

Qualitative tests

This is the extreme form of a categorical test, in which there are only two result categories, positive and negative. This binary categorization can be either based on a cut-off applied to a quantitative result, for example peak height or a mathematical measure representing peak shape, or on direct qualitative observation by the analyst, for example the presence or absence of a peak (in the latter case, as discussed in section 4, the underlying data will generally be quantitative in nature, even though no formal quantification is performed). In terms of accuracy, categorization can be either correct or incorrect with respect to the "true" (reference) result. A simple contingency table can be used to describe the four possible outcomes (table 3).

The diagnostic accuracy of a qualitative test can be characterized by two components, both of which can be calculated from the figures in the contingency table ;       (i) Sensitivity = the proportion of positive results correctly identified by the test =TP/(TP+FN);

(ii) Specificity = the proportion of negative results correctly identified by the test =TN/(TN+FP).

Additionally the overall accuracy can be characterized by the total number of true results as a proportion of the total results [(TP+TN)/(TP+TN+FP+FN)], although in practice this parameter is rarely used. For comparison with quantitative tests (figure 3) the relationship between the components of accuracy is depicted in figure 6.

There is an inverse relationship between sensitivity and specificity (figure 7). As more stringent cut-offs are used to reduce the number of false positives (i.e. increase specificity), so the likelihood of false negatives increases. Therefore the desirable characteristics of a test must be considered in the context of the required outcome and the diagnostic consequences.

For example, laboratory procedures for mutation-scanning tests often involve a primary screen to determine which fragments carry mutations, followed by a second confirmatory test by sequencing to characterize the mutations present. In the primary screen, sensitivity is much more critical than specificity, to avoid missing mutations that are present; the only consequence of poor specificity is to increase the work load for the confirmatory sequencing.

Obviously there is a limit to the lack of specificity that can be tolerated, even if only on the grounds of cost and efficiency.

In situations where sensitivity and specificity are both critical it is desirable to use two cutoffs, to minimize both false-positive and false-negative rates. In this case, results falling between the two cut-offs can either be classified as test failures or be passed for further analysis.

Type D Qualitative tests

Where the test is designed to measure only two states of the analyte (e.g. a specific SNP genotyping assay), the quantitative results for each state can be expected to be normally distributed (figure 8a). In this case, results can be assigned to appropriate categories by competitive hypothesis testing, as described for type C categorical tests (see section 4.2.2).

Again this model can be used as an ongoing validation method, minimizing the need for implementation validation. Test accuracy can also be described in terms of sensitivity and specificity given particular cut-offs. This method would require much more stringent validation prior to implementation (see section 5.3.2).

Type E Qualitative [binary] tests

In many cases, particularly for mutation scanning methods, it is necessary to use qualitative description to distinguish between a single normal state (negative result) and any number of mutated states (positive result). Whilst quantitative results for the normal state would be expected to be normally distributed, positive results would not, since they combine many (known or potential) different mutations, each with its own distribution (figure 8b). Although it is still theoretically possible to use basic hypothesis testing to assign a probability that a result is not normal, competitive hypotheses cannot be used since it is not possible to know the mean quantitative result for all possible mutations (unless they have all been tested). In this scenario, assessment of accuracy is therefore best performed in a pre-implementation validation using a suitable number of positive (i.e. known mutant) and negative (known normal) samples (see section 5.3.2)

Sequencing

Direct sequencing (currently, fluorescent dideoxy-terminator sequencing by capillary electrophoresis) is the method of choice for a wide range of clinical genetic tests and is widely considered to be the 'gold standard' (reference) method for identifying and characterizing DNA variations. As such it is often not possible to develop a suitable reference for comparative validation of new sequencing-based tests. In this situation it is recommended that validation be treated as a verification that sequencing is performing to the required standard, in the context of the new test. Factors to be considered should include confirmation that the new test specifically targets the region of interest (i.e. BLAST primers, and check sequence), that both alleles are reliably amplified (i.e. ensure no SNPs are located in primer binding sites) and that the sequencing data generated is consistently of suitable quality across the whole region of interest (e.g. monitoring PHRED scores across the region of interest). It is important to note that, since sequencing methodologies can vary, for example by clean-up method, thermal cycling regime, or whether single or bi-directional sequencing is used for analysis, the validation scheme should be carefully tailored to the application. This is of particular importance when a new sequencing test is being 'imported' from another laboratory since most laboratories will have their own particular sequencing methodology and this is unlikely to be identical to the local method.

As with other tests, it is important to participate regularly in an external quality assurance (EQA) scheme where possible. In the case of sequencing, this may be dealt with at the technology level in addition to disease-specific schemes, for example the MSCAN and SEQ DNA schemes run by the European Molecular Genetics Quality Network (EMQN) [START_REF]The European Molecular Gentics Quality[END_REF] .

Considerations for experimental design 5.1 Extrapolation of results (validation constraints)

The results of a validation can be applied beyond its immediate coverage; however some rationale needs to be applied to such extrapolation. Let us consider the validation of a mutation-scanning technology that tested 100 different mutations in a particular gene (5,000 bp) resulting in a sensitivity of "≥ 97% (95% CI)" (see section 6.1.2 for calculating and reporting sensitivities). What does this actually mean in practice? Only a very small number of the possible mutations in the region of interest were actually covered; there are 15,000 possible single-base substitutions in 5,000 bp and virtually limitless insertion/deletion mutations. If only substitutions were tested in the validation, the estimated sensitivity could only reasonably be considered to apply to these types of mutations.

However, assuming all different types of mutation were broadly covered by the validation (e.g. all possible nucleotide substitutions, different lengths of insertion and deletion etc.), it would be reasonable to say that sensitivity of mutation scanning in this gene using this method had been shown to be ≥ 97% (95% CI).

It is often appropriate to examine particular categories separately, based on specific knowledge of a test system. For example, it is known that certain single base insertions or deletions in homopolymer stretches can be refractory to detection by high resolution melting.

To gain a realistic understanding of how relevant this might be, particular attention might be paid to this group of variations by including a disproportionate number in the validation. The specific gene or disease should also be considered: if only amino acid substitutions are expected, a reduced sensitivity to single-base insertions would be irrelevant.

Broadly speaking, the limits of extrapolation can be defined by coverage of the parameters considered critical to the successful outcome of the test. That is to say if mutation type is considered a critical factor in achieving a correct result with the given test, then as many different types of mutations need to be included in the validation as possible. Equally, if the G+C content of the template is considered a critical factor, the validation is only applicable to fragments within the G+C content range covered by the validation. This means that a validation of a technology could be applicable to a new gene even if the validation was carried out exclusively on another gene or genes, provided the test carried out on the new gene falls within the critical parameters of the validation (obviously in this case it is critical to ensure that the correct fragments are being amplified.) In this case the gene itself is not a critical factor.

Potentially critical factors should be identified and evaluated at the development stage, based on prior experience and expertise with the technology being validated. However, with primary validation of new technology, attempts should be made to identify the key parameters by performing an evaluation covering as many different potential factors as possible (full or partial factorial). It is also recommended that inter-laboratory reproducibility is evaluated (see section 4.1).

Sample selection

The limits of extrapolation of the validation results is ultimately defined by the choice of samples, which itself is generally limited by the availability of positive controls. For this reason, it is essential that the sample profile is clearly detailed in the validation report, together with an analysis of how this relates to the factors considered critical to the performance of the test.

Positive (mutant) samples should be chosen to represent as broad a range of results as possible, bearing in mind the desire or requirement for extrapolation of the results. This will depend on the purpose of the test under validation. For example, for validation of a method for mutation scanning, samples containing as many different mutations as possible should be included in the validation. In this context it is not normally important that the mutations / variations chosen for the validation are actually pathogenic, as this is not normally relevant to whether or not they will be detectable. It should be noted that including multiple examples of the same mutation in the same amplicon will not increase the power of the study to determine sensitivity, as each repeat cannot be considered different with respect to sensitivity. It is also valuable to include examples where potentially confounding variations exist (i.e. is it possible to detect a mutation in a fragment containing a certain common polymorphism?).

In general it is desirable to include samples containing mutations that represent the range of possible variation in parameters that are important to the technique under test. For example, key parameters for a technique that relies on heteroduplexing or melting would include G+C content of the fragment, position of the mutation in the fragment, and the actual nucleotide change.

In some cases, particularly when validating a new technology, local limitations of sample availability may necessitate an inter-laboratory collaboration to collect a suitable number of samples to attain the required power for diagnostic validation.

Sample size [numbers]

The number of samples used in a validation determines its statistical power, which is a measure of how much confidence can be placed on the results of the validation. Therefore validation sample size is ultimately one of the most important factors in determining the analytical utility of the test. Unfortunately definitive guidelines defining specific sample sizes cannot realistically be given as the requirement is so dependent on a wide range of factors including the nature and performance of the test, what the critical parameters are, how the test will be used in practice, and the confidence level required for clinical utility. A large number of tools for determining sample size given certain input criteria (e.g. confidence interval) are freely available on the internet (e.g. http://www.statpages.org/#Power accessed May 2010).

The Clinical and Laboratory Standards Institute provide a number of evaluation protocols (prefixed EP) making reference to sample size requirements for a variety of situations [START_REF] Tholen | National Committee on Clinical Laboratory Standards: Evaluation of Precision Performance of Quantitative Measurement Methods[END_REF][START_REF] Krouwer | National Committee on Clinical Laboratory Standards. Method Comparison and Bias Estimation Using Patient Samples; Approved Guideline-Second Edition[END_REF][START_REF] Krouwer | National Committee on Clinical Laboratory Standards Preliminary Evaluation of Quantitative Clinical Laboratory Methods[END_REF][START_REF] Garrett | National Committee on Clinical Laboratory Standards. User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline-Second Edition[END_REF][START_REF] Carey | National Committee on Clinical Laboratory Standards. User Verificaton of Performance for Precision and Trueness; Approved Guideline-Second Edition[END_REF] .

Whilst these tools will give useful estimates of the numbers of samples required, the limiting factor is often the availability of suitable control samples [START_REF] Maddalena | Technical standards and guidelines: molecular genetic testing for ultra-rare disorders[END_REF] , even in the case of verification which requires less stringent analysis and therefore fewer samples. In this case it is critical to understand the statistical relevance of using the given sample size and how the confidence level achievable with this sample size impacts on the usage of the test. It is recommended that statistical advice is sought and this is carefully reviewed in the context of the clinical utility. As Jennings et al 2009 5 state: "Although supporting evidence is essential to scientific understanding, it must be recognised that statistically significant data may not be available for every aspect of every validation study. There is simply not enough patient material, financial support, or scientist/technologist time to support such rigorous data collection. Therefore, medical judgement in the context of in-laboratory data and the larger health care system is essential to deciding when a test is ready to be introduced for patient care."

Whatever the availability of samples or the outcome of the validation, it is important to accurately record all details in the validation file including confidence levels and the basis of any decisions made.

Qualitative tests

Estimating power

In the case of qualitative tests, there is a useful rule of thumb that can be used to estimate the power of a study given a particular number of samples. This can be illustrated by the following two qualitative validations of a methodology for mutation scanning: a) Validation using 30 different mutations. b) Validation using 300 different mutations.

If all mutations were correctly identified in both validations, the measured sensitivity would be 100% in both cases. However we are likely to be much more confident in the results of validation (b) because a wider range of different mutations has been tested. This difference relates to the confidence that certain mutations, which cannot be detected by the technique, have not been excluded from the validation by the random selection of samples. This confidence increases as more different mutations are tested. This problem is referred to in statistics as sampling error. For a qualitative test, the goal is to determine a sample size that will provide sufficient power to determine sensitivity and specificity to the desired level of confidence for the particular application.

Precise calculations can be complex, but for practical purposes the 'rule of 3' provides a sufficiently accurate estimate of power according to sample size [START_REF] Jones | An introduction to power and sample size estimation[END_REF][START_REF] Hanley | If nothing goes wrong, is everything all right? Interpreting zero numerators[END_REF][START_REF] Rümke | Uncertainty as to the acceptance or rejection of the presence of an effect in relation to the number of observations in an experiment[END_REF] . This states that, at 95% confidence, the probability of an event that is not seen in validation of sample size n is 3/n. An illustration of the use of the rule of 3 using the examples above is given in table 4.

With molecular genetic tests, technologies are often highly sensitive and the target of validation is often a sensitivity approaching 100%; a test that does not achieve a measured sensitivity of 100% is often not considered suitable for diagnostic purposes. Whilst it is likely a false negative would be found given a big enough sample size, this expectation does mean that sample numbers calculated using the rule of 3 generally yield the required results.

In practical terms the 'rule of 3' will give very accurate estimates for studies where n>60; below this the estimates become overcautious, which is not a bad thing in diagnostics. This rule is valid for any proportion so it can be used for either sensitivity or specificity.

Study design

As we have seen, the power of validation data is related to sample size. The number of positive samples (mutant) will prescribe the power to estimate sensitivity, and the number of negative samples (normal) that of specificity. For most applications it is sufficient to include equal numbers of mutant and wild type samples; this will give equal power to estimate both sensitivity and specificity. This has a practical implication: for the validation of a mutation scan over 50 amplicons using 100 mutant samples, it is not useful to screen all samples for all amplicons (i.e. total of 50 x 100 = 5000 tests). This equates to 100 analyses of mutant samples (power to estimate sensitivity = 97% [1 -3/100 by 'rule of 3']) but 4900 analyses of normal samples (power to estimate specificity = 99.94% [1 -3/4900]). There is clearly a disproportionate power to estimate specificity, which in this case is likely to be the less important measure. It would be sufficient to perform 100 analyses of normal samples (total analyses 200), although it would be sensible to evenly distribute these analyses amongst the amplicons. In situations where sensitivity or specificity is considered to be particularly important, it may be appropriate to weight the numbers of mutant and normal samples appropriately.

It is critical that validation be performed without any knowledge of the actual status of each sample (i.e. blinded analysis), especially in the case of categorical and qualitative tests. To eliminate systematic errors or bias, consideration should also be given to sample order, which should be randomized as much as is practically possible. It may also be beneficial to introduce redundancy into the experiment (by for example duplication) to ensure coverage of all the required results. Whilst this is not critical to the validation results per se, it can save time repeating failed analyses. In addition these data can be used in the determination of precision (repeatability and/or reproducibility).

Reporting the results

Comprehensive and clear documentation of validation is extremely important, both during the preimplementation phase and during ongoing validation. When reporting the results of a validation experiment, it is important to include the derived estimates of diagnostic accuracy including confidence intervals and all details that may affect the interpretation of these In all cases where estimates of accuracy are reported, some measure of the confidence that is applicable to the estimate should also be given. The confidence applied to quantitative measures is essentially the precision (with the exception of measures of probability, which are measures of confidence in themselves). This can most usefully be expressed as a confidence interval around the mean of the replicate results. The following is a simple guide to calculating confidence intervals. In cases, where the measured sensitivity and/or specificity is 100% and the sample size is ≥60, the 'rule of 3' (as described in section 5.4.1) reference is sufficiently accurate to determine the confidence interval. Only the lower confidence limit need be stated, as the upper limit is 100%.

It is important to note that using the "rule of 3" in this context is only valid if all tested mutations are detected. In situations where the measured diagnostic accuracy are less than 100%, more complex statistics are required to calculate the confidence interval. It is recommended that the exact method based on the binomial distribution is used since confidence intervals near 100% need to be skewed (i.e. the interval above and below the measured result are not equal) to avoid upper confidence limits ">100%". A detailed description of the method, together with instructions on performing the calculations in Microsoft Excel, is available on the NIST engineering statistics handbook website [START_REF]Validation procedures: Clinical laboratory standards institute (CLSI) publish a range of protocols and standards that may be useful for diagnostic genetic applications[END_REF] . In all cases where measured diagnostic accuracy is less than 100%, it is recommended to consult a competent statistician.

Conclusion

This paper has outlined the basic principles for including validation and verification in an implementation process for molecular genetic testing. We have described the different types of tests and the key components for validation, and suggested some relevant statistical approaches. The standardized validation pro forma provided in the Supplemental data can be used to guide and record validations and verifications, for the purposes of quality management and accreditation. Any suggestions for additions or alterations should be addressed to the corresponding author. Verification: "Confirmation, through the provision of objective evidence, that specified requirements have been fulfilled" (doing test correctly) Validation: "Confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled" (doing correct test) Principle requirements of ISO 17025:2005 3 5.4.2 "Laboratory-developed methods or methods adopted by the laboratory may also be used if they are appropriate for the intended use and if they are validated". 5.4.5.2 "The laboratory shall validate non-standard methods, laboratorydesigned/developed methods, standard methods used outside their intended scope, and amplifications and modifications of standard methods to confirm that the methods are fit for the intended use. The validation shall be as extensive as is necessary to meet the needs of the given application or field of application. The laboratory shall record the results obtained, the procedure used for the validation, and a statement as to whether the method is fit for the intended use". 5.4.5.3 "NOTE 1 Validation includes specification of the requirements, determination of the characteristics of the methods, a check that the requirements can be fulfilled by using the method, and a statement on the validity. NOTE 3 Validation is always a balance between costs, risks and technical possibilities. There are many cases in which the range and uncertainty of the values (e.g. accuracy, detection limit, selectivity, linearity, repeatability, reproducibility, robustness and cross-sensitivity) can only be given in a simplified way due to lack of information".

Principle requirements of ISO 15189:2007 2 5.5.1 "[...] If in-house procedures are used, they shall be appropriately validated for their intended use and fully documented". 5.5.2 "The methods and procedures selected for use shall be evaluated and found to give satisfactory results before being used for medical examinations. A review of procedures by the laboratory director or designated person shall be undertaken initially and at defined intervals". 5.6.2 "The laboratory shall determine the uncertainty of results, where relevant and possible". As the cut-off is moved to the left false negative rate is reduced but false positive rate increases. used to estimate the probability that a result is not negative, but not that it is not positive.

  significant clinically relevant boundaries within the linear range (e.g. the intermediate expansion/mutation boundary in Huntington's disease), it is useful for both implementation and ongoing validation to use controls on or close to the boundary[s].

1 .

 1 Calculate the mean of the replicates (M) 2. Calculate the standard deviation of the replicates (s) 3. Calculate the standard error s M = s/√N (where N = number of replicates) 4. Calculate degrees of freedom df = N-1 5. Find t for these df using a Student's t table

Figure 1 :

 1 Figure 1: The process of implementing a molecular genetic test for diagnostic use. The

Figure 2 :

 2 Figure 2: Types of bias. In each case the broken line represents the perfect result where all

Figure 3 :

 3 Figure 3: Performance characteristics, error types and measurement metrics used for quantitative tests (adapted from Menditto et al) 19 .

Figure 4a :

 4a Figure 4a: A type B categorical test to size PCR fragments. Each category (indicated by

Figure 4b :

 4b Figure 4b: A type C categorical test for allele quantification. Each category (shaded) has

Figure 5 :

 5 Figure 5: Multiplex ligation dependant probe amplification to detect exon copy number

Figure 6 :

 6 Figure 6: The relationship between performance characteristics, error and measurement uncertainty used for qualitative tests (adapted from Menditto et al) 19 .

Figure 7a :

 7a Figure 7a: The relationship between sensitivity and specificity. The figure shows

Figure 7b :

 7b Figure 7b: Cut-off location. Positioning the cut-off to the right encompasses more of the

Figure 7c :

 7c Figure 7c: Use of two cut-offs. It is possible to minimize both false positive and false

Figure 8a :

 8a Figure 8a: Truly binary test. Only two states of the analyte that can be measured; one wild

Figure 8b :

 8b Figure 8b: Arbitrarily binary test. There are many different possible states of the analyte;

Table 1 .

 1 Validation and verification: definitions and summarized requirements of the major international standards for accreditation of genetic testing laboratories.

	Definitions
	(from ISO 9000:2005)
	Also see the VIM 20

Table 2 :

 2 Types of test NB. In addition to the parameters detailed below appropriate robustness testing should be carried out for all types of test. Metric used for implementation validation a. Sensitivity = True Positive / (True Positive + False Negative) Metric used for implementation or ongoing validation b. Specificity = True Negative / (True Negative + False Positive) Metric used for ongoing validation c. Accuracy = True Result / (True Result + False Result) ++ Recommended parameter d. Precision should be measured in terms of repeatability and intermediate precision (as well as reproducibility for interlaboratory validations) + Applicable parameter (less used) e. The term 'probability' is used to describe situations where a probability that the result is correct can be assignedprimarily in on-going validation (e.g. competitive hypothesis testing)

		Description	Examples	Sensitivity a Specificity b	Accuracy c	Trueness	Precision d	Limits of detection	Probability e
	A	Quantitative tests. The result can have any value between two limits (including decimals).	Determination of methylation load (%); characterization of a mosaic mutation; heteroplasmy of mitochondrial variants.	v			++	++	++
	B	Categorical tests where the quantitative signal is placed into an ordinal series to give the final result.	Sizing a PCR product; determination of triplet repeat size (FRAXA, Huntington disease, etc.)			+	++	++	++	+
	C	Categorical tests where the quantitative signal is placed into one of a limited series of final result. predefined categories to give the	Determination of copy number using dosage in CMT and HNPP; PCR or MLPA.: exon deletion / duplication in BRCA1; PMP22 gene			+	To establish correction factors and/or cut-offs		++
		Qualitative tests where the true
	D	quantitative signal can have one of many possible values, but the required result can only have one	Mutation scanning for unknown mutations e.g. by sequencing or high resolution melt.	++	++	+	To establish correction factors and/or cut-offs	++
		of two possible values.
	E	Qualitative [binary] tests where the true quantitative signal can only have one of two possible values	Genotyping for a specific mutation e.g. CFTR Phe508del in cystic fibrosis or HFE Cys282Tyr in hemochromatosis .	++	++	+	To establish correction factors and/or cut-offs	++	+
		Legend	Notes

Table 3 : Possible outcomes for a qualitative validation experiment

 3

		Reference result
		+	-
	Test result	+ True positive (TP) -False Negative (FN) True Negative (TN) False Positive (FP)

Table 4 : The effect of sample size on statistical power to determine sensitivity.

 4

	Validation	n	Experimental sensitivity	3/n (probability of a FN)	Maximum sensitivity
	a)	30	100%	0.1 (or 10%)	≥90% (95% CI)
	b)	300	100%	0.01 (or 1%)	≥99% (95% CI)
	FN = False negative

Table 5 : Confidence intervals for experiments with apparently equivalent sensitivities.

 5

	Experiment	Experimental result	Experimental sensitivity	Confidence interval (95% confidence)	Confidence range
	A	1 FN in 150	99.3%	96.9 to 99.9%	3.09%
	B	2 FN in 300	99.3%	97.9 to 99.9%	1.96%
	C	20 FN in 3000	99.3%	99.0 to 99.6%	0.53%
	FN = False negative

Acknowledgements

Thanks to Patrick M Bossuyt (Dept. Clinical Epidemiology -Biostatistics & Bioinformatics -Amsterdam -The Netherlands), Ian Mann (Swiss Accreditation Service) and Paul Strike (Salisbury Foundation Trust, UK) for their help and advice, and Milan Macek Jr (Charles University, Prague) for hosting the meetings. This work was funded by Eurogentest and the UK Department of Health.

6. Lower confidence limit = M -(t x s M ) 7. Upper confidence limit = M + (t x s M ).

For a comprehensive discussion on expression of uncertainty in relation to quantitative tests refer to the European co-operation for Accreditation document EA-4/14 [START_REF]EA guidelines on the expression of uncertainty in quantitative testing[END_REF] .

Qualitative tests

When reporting estimates of accuracy for a qualitative test, the measured sensitivity and specificity are not useful figures on their own as they only relate to the specific samples tested in the validation (e.g. the proportion of gold-standard positives correctly identified). To apply the estimates to a wider population and to allow the validation results to be realistically compared to others, a confidence interval must be given. This is a function of the measured results and the sample size. Table 5 gives an example with the results of three experiments, of different sample sizes but for which the measured sensitivities were identical. It is clear that the larger sample size of experiment C gives a much smaller confidence interval. Such ambiguities are very common in the reporting of diagnostic accuracy [START_REF] Harper | Reporting of precision of estimates for diagnostic accuracy: a review[END_REF] . At best, they can preclude any realistic comparison of different validation experiments; at worst, they can provide misleading diagnostic information with potentially serious consequences for patient care.

To improve this situation, estimates of accuracy should always be based on valid calculations and be given with appropriate confidence intervals, for example the lower and upper limits between which there is 95% confidence that the sensitivity/specificity for the wider population falls.

Conflict of interest.

The authors declare no conflict of interest.


Ellen Reisinger

David Meintrup

Dominik Oliver

Bernd Fakler

Gene expression associated with the onset of hearing detected by differential display in rat organ of Corti

Keywords: Deafness genes, hair cells, quantitative real-time PCR, deafness loci, hair cell differentiation

The exquisite performance of the mammalian hearing organ results from a finely orchestrated array of cell types whose highly specialized functions are determined by their gene expression profile. In rodents, this profile is established mainly during the first two weeks of postnatal maturation. Here, we used the differential display technique on the rat organ of Corti to uncover transcripts upregulated in expression between postnatal stages P0 and P14. A total of 176 different genes were identified whose amount of mRNA increased during early postnatal development. The transcripts code for proteins serving a broad spectrum of cellular functions including intracellular signaling, control of growth/differentiation, regulation of protein synthesis/degradation/modification, metabolism and synaptic function. In addition, the set of upregulated transcripts contained several proteins of yet unknown function, hypothetical proteins, and so far unknown mRNA sequences. Thus, this study unravels the broad and specific transcription program that operates the maturation of the mammalian hearing organ.

Further, as 49 of the genes found here map to at least one unspecified deafness locus, our study provides candidate genes for these and novel deafness loci.

Introduction

The mammalian hearing organ, the cochlea, perceives and processes sound with extremely high sensitivity and frequency selectivity [START_REF] Dallos | The active cochlea[END_REF] . This remarkable performance essentially results from the concerted action of a variety of highly specialized cell types that are organized in a well-ordered microarchitecture and serve quite a number of distinct functions, including ion transport and homeostasis, micromechanics and active amplification, sensory transduction, and electrical signal propagation [START_REF] Dallos | The active cochlea[END_REF][START_REF] Hudspeth | How hearing happens[END_REF] .

On the molecular level, this specialization, and hence perception and processing of sound, relies on the expression of a multitude of proteins, some of them specifically found in the inner ear, e.g. tectorin, cochlin and prestin [START_REF] Weber | Thyroid hormone is a critical determinant for the regulation of the cochlear motor protein prestin[END_REF][START_REF] Zheng | Genomic characterization and expression of mouse prestin, the motor protein of outer hair cells[END_REF][START_REF] Li | Expression of full-length Cochlin p63s is inner ear specific[END_REF][START_REF] Killick | Antibodies to the sulphated, high molecular mass mouse tectorin stain hair bundles and the olfactory mucus layer[END_REF] . This notion is supported by the considerable number of genes shown to be causative for inherited, non-syndromic forms of hearing impairment or deafness, both in humans and mouse models [START_REF] Petit | Molecular genetics of hearing loss[END_REF][START_REF] Steel | A genetic approach to understanding auditory function[END_REF] . Furthermore, a series of proteins found both in the inner ear and in other tissues are known to be causative for inherited diseases which are associated with deafness. For example, Usher syndrome proteins like SANS are required for proper function of auditory hair bundles and photoreceptors in the retina [START_REF] Weil | Usher syndrome type I G (USH1G) is caused by mutations in the gene encoding SANS, a protein that associates with the USH1C protein, harmonin[END_REF] . For humans, the central database has so far collected a total of more than 140 different gene loci associated with non-syndromic deafness [START_REF] Van Camp | Hereditary Hearing Loss Homepage[END_REF][START_REF] Dror | Hearing Loss: Mechanisms Revealed by Genetics and Cell Biology[END_REF] . However, until now only around 50 have been assigned to proteins with defined function, and the majority still awaits identification and examination of the respective molecular pathomechanism [START_REF] Van Camp | Hereditary Hearing Loss Homepage[END_REF][START_REF] Dror | Hearing Loss: Mechanisms Revealed by Genetics and Cell Biology[END_REF][START_REF] Friedman | Human nonsyndromic sensorineural deafness[END_REF] .

In rodents, pups are born deaf and the inner ear undergoes extensive remodeling during postnatal development [START_REF] Pujol | Development of Sensory and Neural Structures in the Mammalian Cochlea[END_REF] culminating in the onset of hearing around postnatal day 12 (P12) in rats [START_REF] Rübsamen | The development of cochlear function[END_REF] . Accordingly, proteins essential for perception and processing of sound are expected to be 'switched on' or to be upregulated on the transcriptional level during this period. Known examples that confirm to this developmental pattern and cause inherited hearing disorders in humans when mutated include, e.g., prestin (SLC26A5) [START_REF] Toth | A new mutation in the human pres gene and its effect on prestin function[END_REF][START_REF] Liu | Prestin, a cochlear motor protein, is defective in nonsyndromic hearing loss[END_REF][START_REF] Oliver | Expression density and functional characteristics of the outer hair cell motor protein are regulated during postnatal development in rat[END_REF] , KCNQ4 [START_REF] Beisel | Longitudinal gradients of KCNQ4 expression in spiral ganglion and cochlear hair cells correlate with progressive hearing loss in DFNA2[END_REF] and Cav1.3 [START_REF] Petit | Molecular genetics of hearing loss[END_REF][START_REF] Steel | A genetic approach to understanding auditory function[END_REF][START_REF] Brandt | CaV1.3 channels are essential for development and presynaptic activity of cochlear inner hair cells[END_REF][START_REF] Hafidi | Developmental expression of Ca(v)1.3 (alpha1d) calcium channels in the mouse inner ear[END_REF] . Thus, postnatal maturation renders the rat organ of Corti an ideal model system for investigating genes required for hearing by analysis of transcripts being differentially expressed prior to and after the onset of hearing. The Differential Display on cDNA (cDNA-DD) of microdissected organs of Corti exhibited several benefits for our experimental design: first, it requires only small amounts of tissue without the need of mRNA-or cDNA pre-amplification. Indeed, the high sensitivity of the cDNA-DD technique might increase the chance to detect expression differences for low abundance transcripts that are underrepresented in DNA array outcomes [START_REF] Hildebrand | Gene expression profiling analysis of the inner ear[END_REF] . Second, the cDNA-DD allows detecting not only genes with database entries, but also new mRNA transcripts or splice variants can be identified. Third, several previous studies proved the reliability and robustness of the cDNA-DD [START_REF] Stankovic | Real-time quantitative RT-PCR for low-abundance transcripts in the inner ear: analysis of neurotrophic factor expression[END_REF][START_REF] Liang | Differential display of eukaryotic messenger RNA by means of the polymerase chain reaction[END_REF][START_REF] Diachenko | Combining the technique of RNA fingerprinting and differential display to obtain differentially expressed mRNA[END_REF] .

In this study, we used the cDNA-DD to identify genes that are upregulated in the rat organ of Corti between the very immature postnatal stage P0 and P14, which is two days after the onset of hearing.

Materials and Methods

Differential Display

Tissue preparation

Organs of Corti from six rats at day of birth and apical parts of the organs of Corti of seven rats at postnatal stage P14 were dissected under 5 to 10-fold magnification and immediately frozen in liquid nitrogen prior to mRNA isolation. Dissected tissue consists of the organ of Corti and minor parts of stria vascularis and spiral ganglion. mRNA isolation and cDNA synthesis mRNA was prepared with DYNABeads (Dynal, Oslo, Norway) according to the manufacturers instructions. mRNA concentrations in P0 and P14 samples were determined by UV absorption. cDNA was derived by reverse transcription of approximately 150 ng mRNA (each) with Superscript II (Invitrogen, Karlsruhe, Germany) in supplemented buffer and 10 µM DTT, 1 mM dNTPs, 1 pmol oligo-dT primer, 0.3 µl RNasin (Promega, Madison, WI) for 2 h at 42°C.

Differential Display

The differential display on reverse transcribed mRNA of P0 and P14 organs of Corti was performed as described [START_REF] Meintrup | A statistical model providing comprehensive predictions for the mRNA differential display[END_REF] . Briefly, arbitrary P-primers of the 'Delta Differential Display' kit (Clontech, Palo Alto, CA) were combined with each other or with two-base-anchored oligo dT-primers.

The differential display PCR was set up with two dilutions of reverse transcribed P0 and P14 mRNA (dilution A, using 2.4 ng/µl mRNA; and B, 0.6 ng/µl) and for negative controls with mRNA (without reverse transcriptase) or H 2 O as templates.

Bands displaying from P14 cDNA with no corresponding band at P0 or bands with higher intensity in both dilutions of P14 template cDNA compared to both P0 lanes were considered as 'differential'. The underlying gel pieces were cut out, DNA was eluted in 40 µl 1x TE at 100°C for 5 min and reamplified by a PCR using the original primer combination and 20-25 cycles of a standard PCR protocol.

Identification of cDNA bands

Reamplified DNA fragments were analyzed by direct sequencing with fluorescent dye terminator cycle sequencing (Applied Biosystems, Weiterstadt, Germany). cDNAs for which no sequence information was obtained with this procedure were subcloned into pBlusecript SK -(Stratagene) and three to five clones of each subcloned band were sequenced. In Supplementary Tables 2 and3, only those genes were included where at least two inserts from independent clones were comprised of identical or overlapping sequences.

Quantitative Real-Time PCR

RNA preparation and reverse transcription for qPCR analysis

Quantification of transcripts from inner ear was performed as described [START_REF] Reisinger | Cod106, a novel synaptic protein expressed in sensory hair cells of the inner ear and in CNS neurons[END_REF] . mRNA of the respective developmental stage was prepared from the apical turns of two times six or seven organs of Corti. Total RNA was isolated with Trizol reagent (Invitrogen, Karlsruhe, Germany) as recommended by the manufacturer. RNA was transcribed into cDNA in 1x first strand buffer with 10 µM DTT, 0.5 mM dNTPs, 0.3 x hexanucleotide Mix (Roche, Mannheim, Germany), 50 pmol oligo-dT-primer, 20 U RNasin (Promega) and 1 µl Superscript II (Invitrogen) for 10 min at 25°C, 10 min 37°C and 70 min at 42°C. These cDNA samples were ethanol precipitated prior to qPCR.

Design of intron-overspanning primers

Primer pairs were designed by means of Primer Express 2.0 software (Applied Biosystems).

Primers were either placed on neighboring exons, when introns spanned more than 1kb, or one of the primers annealed over the intron splice site, so that 5-8 nucleotides on the 3' site bound to the downstream exon. Amplification efficiency was evaluated in a separate qPCR experiment followed by melting curve analysis. Primer sequences are given in Supplementary Table 1.

Real-Time qPCR

Quantitative PCR was performed with SYBr Green (2x Master Mix, Applied Biosystems) in an ABI PRISM 7000 or 7500 sequence detection system (Applied Biosystems). For the quantification of 18S rRNA samples were diluted 1:100 or 1:1000. All qPCR experiments were performed in triplicate; standard deviations (SD) of C t values were in average 0.12 cycles, which reflects an SD in cDNA copy numbers of 5.3%. The SDs were smaller for high abundance genes, e.g. 18S rRNA quantification resulted in an SD of 0.02 cycles or 1.5% SD for the number of mRNA molecules, but higher for low abundance genes, e.g. Scamp3 (C t values between 35 and 38 which is close to the detection limit).

To calculate mean C t values, non-logarithmic numbers (2^(42-C t )) were calculated, those numbers were averaged and back-calculated to get the (logarithmic) C t value. Transcript abundance in each sample was determined using the comparative C T method, as described [START_REF] Reisinger | Cod106, a novel synaptic protein expressed in sensory hair cells of the inner ear and in CNS neurons[END_REF][START_REF] Livak | Analysis of Relative Gene Expression Data Using Real-Time Quantitative PCR and the 2DeltaDeltaCT Method[END_REF] .

Results

Profiling upregulated transcripts by means of cDNA-DD

In this study, we used a cDNA-DD with reverse transcribed mRNA of P0 and P14 organs of Corti to identify genes upregulated during postnatal development of the rat hearing organ (Fig. 1a). 28 combinations of 'arbitrary' and 'oligo-DT' primers were used for PCR-amplification of cDNA fragments from both transcriptomes. Separation of the radioactively labeled PCR products on high-resolution acrylamide gels showed distinct patterns between the two postnatal stages (Fig. 1b) and allowed for identification of cDNA fragments with higher abundance in the P14 amplifications. Altogether, 302 cDNA fragments were found upregulated in P14 organs of Corti, of which 289 were successfully recovered from the gel and reamplified. Of these, 166 allowed for direct sequencing, while 115 cDNA-fragments were identified only after subcloning. 14 of the sublconed fragments contained more than one DNA sequence (according to the criteria described in Materials and Methods). In total, sequences of 297 cDNA fragments were assigned to differential bands.

Identification and categorization of transcripts

All sequences were screened for redundancy by comparison against each other (using FASTA [START_REF] Pearson | Improved tools for biological sequence comparison[END_REF] ) and against the public 'non-redundant' or 'EST' databases on www.ncbi.nlm.nih.gov using BLAST [START_REF] Altschul | Gapped BLAST and PSI-BLAST: a new generation of protein database search programs[END_REF] . In total, 176 individual genes were detected by this cDNA-DD, of which 114 were identified by cDNA fragments found in one cDNA-DD band each, while 62 genes were covered by DNA fragments extracted from two to eight bands (Table 1).

From the 176 genes, 116 fragments could be attributed to genes with well-established function (Fig. 2a, Supplementary Table 2). Another 19 sequences originated from genes encoding proteins with unknown function, while 13 cDNA fragments were assigned to predicted or hypothetical proteins. Further, 20 sequences matched to expressed sequence tags (ESTs) or uncharacterized mRNA sequences (Fig 2a), with two significantly exceeding the database entries which were therefore submitted for publication (dbEST 50172343, 50172344). Eight sequences were deposited in public databases (accession numbers 50172335 to 50172342) as they did not match any database entry for transcribed sequences but corresponded to genomic DNA entries.

The proteins with known function were grouped with respect to their role in cell physiology (Fig. 2b). As summarized in Supplementary Table 2, a substantial fraction of the developmentally upregulated genes are involved in (I) intracellular signaling pathways (including kinases/phosphatases and interacting molecules, GTP-binding proteins, receptors and enzymes modifying second messengers) as well as in (II) extracellular signaling dedicated mainly to regulation of growth and differentiation and in (III) transmembrane signaling and transport represented by subunits of ion channels and solute carriers. Further functional categories defined by the identified transcripts are (IV) synaptic/vesicular proteins, proteins involved in (V) regulation of protein biosynthesis and degradation (including transcriptional elements, chaperones and components of the protein degradation machineries), and (VI) proteins involved in maintenance of the cellular (sub-)structure and morphology. In addition, quite a number of the identified gene products exhibit enzymatic activities (VII) and participate in metabolism of various subcellular compartments (see Supplementary Table 2).

Together, the diversity of gene products detected by our cDNA-DD approach demonstrated that maturation of the hearing organ requires a broad transcriptional program covering a variety of cell physiological functions.

Association of the differential genes with hearing disorders

Some proteins found here to be upregulated during inner ear maturation had been assigned to hearing disorders. Three of the genes, namely Sbf2, Pmp22 and myelin protein zero lead to variants in Charcot-Marie-Tooth Syndromes which are associated with hearing impairments [START_REF] Chapon | Axonal phenotype of Charcot-Marie-Tooth disease associated with a mutation in the myelin protein zero gene[END_REF][START_REF] Kovach | A Unique Point Mutation in the PMP22 Gene Is Associated with Charcot-Marie[END_REF] .

In a mouse model, deficiency in neurotrophic tyrosine kinase receptor type 2 (Ntrk2) was shown to be emybronic lethal but also led to a defect in hair cell innervation [START_REF] Fritzsch | The combined effects of trkB and trkC mutations on the innervation of the inner ear[END_REF][START_REF] Schimmang | Developing inner ear sensory neurons require TrkB and TrkC receptors for innervation of their peripheral targets[END_REF] . In addition, for a series of genes identified here, a close protein family member had been shown to cause deafness in humans. For example, protocadherin 15, sharing 42% sequence similarity with protocadherin α (Supplementary Table 2, section II), is now well established to cause Usher 1F syndrome [START_REF] Ahmed | Mutations of the protocadherin gene PCDH15 cause Usher syndrome type 1F[END_REF][START_REF] Alagramam | Mutations in the novel protocadherin PCDH15 cause Usher syndrome type 1F[END_REF] as well as DFNB23 36 .

We next compared the genetic locus of the human homologous genes with deafness loci that had not been associated with a specific gene. If the position of the human gene lay within the finite linked deafness loci defined by linkage analysis, we considered this gene to be a potential candidate for a deafness gene. We found 49 genes that matched these criteria and were at the same time not assigned to a genetic human disorder in OMIM database (Supplementary Table 3).

For example, two genes with yet unknown function, namely reticulon 3 and transmembrane protein 16A, both match to the deafness locus DFNB63, whereas in the genomic region comprising the deafness loci DNFNA24, DFNA27 and DFNB55, insulin-like growth factor binding protein 7 (Igfbp7) is encoded.

Upregulation of transcripts in the postnatal period

The transcriptional upregulation implied in the cDNA-DD approach was verified in experiments that quantified the postnatal expression by quantitative PCR (qPCR). For this validation experiment we selected genes matching to several deafness loci, different genes encoded within the same deafness locus, and genes which might play an important role for organ of Corti function. The qPCR experiments revealed a substantial up-regluation between P0 and P14 for 21 out of the 27 tested genes (Fig. 3a). Upregulation was more than five-fold for seven genes, including Igfbp7, Na-K-ATPase β3, and Sepp1, whose human genes lie within several deafness loci. With this approach, we could not find a significant up-regulation for Rab9, protocadherin α, transmembrane protein 16A, prefoldin, Prkab2 and pleiotrophin.

For five genes we tested the course of up-regulation during five postnatal developmental stages of the organ of Corti (Fig 3b). SLC20A1 was found to exhibit a linear increase in mRNA level between P0 and P21, while SNAP91 and Mpp6 display transient upregulation with peak levels around P10. mRNA expression of Chp (Ca 2+ bp-P22) and Gpm6b elevates between postnatal days 1 and 9 and remains about constant thereafter (Fig. 3b). Thus, the up-regulation of the individual genes seems to happen at different time points during postnatal maturation of the inner ear.

Estimation of Completeness of Results

Identifying a total amount of 176 distinct proteins poses the questions on how many genes may actually be upregulated in the maturating organ of Corti in total. We estimated this by statistical analysis using a custom developed algorithm [START_REF] Meintrup | A statistical model providing comprehensive predictions for the mRNA differential display[END_REF] , which is based on the correlation that redundancy of randomly amplified genes decreases when the total number of templates increases.

We estimated that 16 % (90% confidence interval (CI) of [0.145, 0.178]) of the genes expressed in the organ of Corti are upregulated during postnatal maturation. Further, 441 differentially expressed genes are estimated to be theoretically detectable by the cDNA-DD using an infinite number of primer combinations (90% CI: [397, 485]). Accordingly, the cDNA-DD with a subset of 28 primer combinations that identified 176 distinct genes uncovered approximately 40% of the accessible differentially expressed genes. Finally, the total number of genes expressed in the organ of Corti is estimated to 2734 (90% CI [2485, 3037]). We consider this estimation as a lower limit for the actual protein encoding genes in the rat organ of Corti because of limitations of the cDNA-DD technique (discussed in ref [START_REF] Meintrup | A statistical model providing comprehensive predictions for the mRNA differential display[END_REF] ).

Discussion

In the present study, we identified a set of transcripts that are upregulated during postnatal maturation of the organ of Corti. The massive change in gene expression probably underlies the extensive physiological and morphological changes that lead to maturation steps essential for the onset of hearing.

Reliability of the Method

To uncover differentially expressed transcripts in the mammalian inner ear, we chose the cDNA-DD method mainly because of its high sensitivity. Indeed, we successfully detected genes with low copy numbers, such as Itgbl1 or Scamp3 (according to qPCR) and a series of 11 transcription factors. Further, this approach is known to produce a low fraction of false positives, which was reassessed here by qPCR. While 21 out of 27 genes were confirmed to be substantially upregulated (Fig. 3), transcript levels of six genes were unchanged in these experiments. This could be due to differences between organ of Corti microdissections for the cDNA-DD and for qPCR. For example, if a gene is upregulated in hair cells but down-regulated in spiral ganglion neurons, different amounts of spiral ganglion tissue attached to the organ of Corti in the different preparations could lead to such discrepancies.

Applying the cDNA-DD approach, we uncovered differences in mRNA expression, assuming that transcript abundance correlates with protein amounts for most of the genes. This presumption was found to be supported by earlier findings on the developmental upregulation of gene products involved in cochlear function. For example, immunostaining for KCNJ16 (Supplementary Table 2, section III) showed a rapid increase of protein levels in the rat cochlea from P10 to P16, correlating with observed changes in endocochlear potential 37 . Another wellstudied example is the polyamine metabolism (Supplementary Table 2, section If): Highest ornithine decarboxylase activity was observed at postnatal days 9 and 10, and it declined to about 1-2% in adulthood 38,39 . Moreover, deficiency in spermine synthase, an enzyme acting in this pathway, was shown to cause deafness in mice 40 . These examples strongly support the idea that differential transcripts found in our study might indeed reflect a postnatal upregulation of the respective proteins in the rodent inner ear.

In summary, the cDNA-DD approach turned out to reliably uncover upregulated transcripts in the inner ear. However, some proteins like KCNQ4, prestin or TMC1, have not been detected by this approach, despite our expectation to do so. For KCNQ4, whose mRNA has a GC content of 62%, we assume that it would not be amplified under our experimental conditions, but only upon addition of DMSO. Other proteins might have been missed as this cDNA-DD uncovered only 40% of the detectable genes.

Significance for identification of deafness genes and cochlear biology

Normal hearing requires proper expression and function of an extensive set of specialized proteins, a notion reflected in the high number (~ 140) of gene loci correlated with various forms of non-syndromic hearing loss. Currently, 90 of these loci, mostly spanning 100 genes or more [START_REF] Van Camp | Hereditary Hearing Loss Homepage[END_REF][START_REF] Friedman | Human nonsyndromic sensorineural deafness[END_REF]41 , still await identification of the gene underlying this particular form of deafness It is reasonable to suggest that in rodents many of those genes should be upregulated between birth and the onset of hearing. Here, we found 49 upregulated genes that have not yet been described as disease-associated but lie within a human deafness locus with so far unidentified underlying gene. Therefore, they are attractive candidates for geneticists that aim to identify genes defective in the respective forms of deafness. Further, interaction partners of the proteins unraveled herealthough not directly identified in this study -might also be reasonable candidate genes for deafness loci.

Apart from the added value of our study for identification of candidate genes in novel deafness loci, this work gives insight into the molecular machinery that enables the organ of Corti to transduce mechanical into electrical energy. The series of proteins serving most different cellular functions (Supplementary Table 2) might inspire research on different aspects of cochlear biology, such as establishment of the cytoskeleton, inter-and intracellular signaling and regulation of gene activation by transcription factors.

In the present study we show that maturation of the mammalian cochlea is associated with the transcriptional upregulation of a considerable number of proteins implicated in a wide array of cellular functions in the inner ear. Notably, many of the 176 distinct genes identified here match to so far uncharacterized deafness loci. Thus, this study provides a rational set of target candidates for the deafness genes within deafness loci that often span more than 100 genes 41

Table and

Frequency with which genes were identified within this cDNA-DD.

Fig. 1 cDNA

 1 Fig. 1

Fig. 2

 2 Fig.2

Fig. 3

 3 Fig.3

  Figure 1

  .

	36.	Ahmed ZM, Riazuddin S, Ahmad J et al: PCDH15 is expressed in the neurosensory
		epithelium of the eye and ear and mutant alleles are responsible for both USH1F and
		DFNB23. Hum Mol Genet 2003; 12: 3215-3223.
	37.	Hibino H, Higashi-Shingai K, Fujita A, Iwai K, Ishii M, Kurachi Y: Expression of an
		inwardly rectifying K+ channel, Kir5.1, in specific types of fibrocytes in the cochlear
		lateral wall suggests its functional importance in the establishment of endocochlear
		potential. Eur J Neurosci 2004; 19: 76-84.
	38.	Henley CM: Postnatal developmental changes in inner ear ornithine decarboxylase
		(ODC). Hearing Research 1991; 55: 45-49.
	39.	Brock M, Henley CM: Postnatal changes in cochlear polyamine metabolism in the rat.
		Hear Res 1994; 72: 37-43.
	40.	Wang X, Levic S, Gratton MA, Doyle KJ, Yamoah EN, Pegg AE: Spermine Synthase
		Deficiency Leads to Deafness and a Profound Sensitivity to {alpha}-
		Difluoromethylornithine. J Biol Chem 2009; 284: 930-937.
	41.	Alsaber R, Tabone CJ, Kandpal RP: Predicting candidate genes for human deafness

disorders: a bioinformatics approach. BMC Genomics 2006; 7: 180.

Table 1 . Gene frequency

 1

	frequency	genes	bands
	1	114	114
	2	38	76
	3	12	36
	4	2	8
	5	3	15
	6	3	18
	7	2	14
	8	2	16
	9	0	0
	Sum	176	297

Acknowledgements

We thank Susanne Weidemann, Nina Katrin Dankenbrink-Werder and Jost Ludwig for technical assistance and advice, Henrike Berkefeld for help with the preparation of organs of Corti and Tina Pangrsic, Tobias Moser and Nikolaj Klöcker for critically reading the manuscript. The project was supported by the Graduiertenkolleg GRK 843.

Supplementary information is available at European Journal of Human Genetics' website.

Conflict of interest.

The authors declare no conflict of interest.