Comparative analysis of specificity and sensitivity between Cobas 6800 system and SARS-CoV-2 rRT-PCR to detect COVID-19 infection in clinical samples

Fast and reliable testing for the COVID 19 infection is the need of the hour for the development of effective and reliable tools and assays. However, it is difficult to find the performance relativity among all these tests which are poorly understood. In this study, we aimed to evaluate the two different platforms where we determine the difference of sensitivity and specificity between the fully automated analyzer (Roche Diagnostics Cobas 6800 SARS-CoV-2 test) under FDA Emergency Use Authorization (EUA) and the laboratory designed test (SARS-CoV-2 rRT-PCR) based on the protocol developed by ICMR (Indian Council for Medical Research). The study was conducted for individual samples. We performed our study with two different approaches, first with validation method consisting of 188 samples (2 batches) on cobas 6800 instrument (Roche Molecular Systems, Branchburg, NJ) soon after we received US FDA EUA on 1 June 2021, all these samples were tested earlier with laboratory designed tests on 25th and 26th May 2021. Over all agreement between the two tests is of 88% and the coefficient of agreement between the two testing platform Cohen’sκ coefficient was found to be 0.76 (95% CI, 2.5897–13.4103) suggesting the substantial agreement between the two platforms. However, in some of the cases, both tests have shown a little disagreement. An overall discordance rate between two systems was found 11.1%. The difference may be due to the limit of detection, variation in the sequences of the primer design or may be due to other factors depicting the importance of comparing the two platforms used in the testing for SARS-CoV-2. Second approach includes head to head evaluation which comprises 1631 samples showed overall agreement of 99% and kappa value of 0.98. These results showed that cobas is effective and reliable assay for the detection of SARS-CoV-2 infection.


Introduction
In the final months of 2019, a novel beta-coronavirus, defined as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was identified as the cause of atypical pneumonia of unknown etiology in Wuhan city of China (https:// www. who. int/ emerg encies/ disea ses/ novel-coron avirus/ 2019/ techn ical; https:// www. undp. org/ conte nt/). The emerging disease hence has been called as  or Corona virus disease (Gorbalenya et al. 2020;Zhu et al. 2020). Till May 2020, over 5.8 million confirmed cases of COVID-19 and more than 360,000 deaths worldwide were recorded. This pandemic has created the need for rapid and accurate diagnostic modalities (Pujadas et al. 2020). Among the various diagnostic paradigms, molecular diagnostic has become the primary mode of detection. rRT-PCR becomes the reliable and more realistic way of detection of SARS-CoV-2 virus. Researcher has reported different genes for targeting the presence of SARS-CoV-2 virus which includes envelope (E), and confirmatory genes including nucleocapsid (N), RNA-dependent RNA polymerase (RdRp) and open reading frame (ORF) 1a (You et al. 2021). But time to time, the efficacy of rRT-PCR for COVID-19 diagnosis has been questioned. At the time of pandemic, US Food and Drug Administration (FDA) emergency-use authorization (EUA), have designed a tool for analyzing a sample by high throughput technique to endure with the unrivaled demand of testing for SARS-CoV-2. The Cobas 6800 System (Roche Molecular Systems, Branchbug, NJ, USA) is highly automated fully integrated analyzers tool (Pfefferle et al. 2020). On 12 March 2020, The US FDA approved Cobas 6800 System for the testing of SARS -CoV-2. In addition to this many reports has also assessed the performance of Cobas 6800 and approved it as highly reliable, sensitive and accurate method for SARS -C oV-2 RNA assays. In this study, we share our experience with two different testing platforms, the Cobas 6800 SARS-CoV-2 test (Roche Molecular Systems, Branchburg, NJ, USA) and a laboratory design test (LDT) real-time rRT-PCR using the Indian council of Medical research (ICMR) 2019-nCoV primers and probes.

Methodology
In our institute, on 25th and 26th may 2021, a total of 188 respiratory samples (nasopharyngeal) were obtained from asymptomatic, symptomatic patients or contact persons in viral transport medium and tested on laboratory design rRT-PCR test. The protocol recommended by the ICMR Delhi was as follows: viral RNA was extracted from 200 μL of the sample using the automated purification system (GENETIX purifier). Quantification rRT-PCR (qRT-PCR) was performed using one step multiplex real-time rRT-PCR kit with the previously published primer/probe for detecting two target gene (N gene and ORF 1ab) in human sample, with an analytical accuracy of < 5 RNA copies/reaction it makes COVID-19 one Step rRT-PCR Kit a highly sensitive and specific kit for the detection of SARS COV 2 RNA (Corman et al. 2020). The easy protocol, minimum assay time and high specificity and sensitive make it an ideal choice for your laboratory for the detection of COVID 19 suspected patients.
The samples were also analyzed on Cobas 6800 (Roche Molecular Systems), approved by US Food and Drug Administration (FDA) emergency-use authorization (EUA for the testing of SARS-CoV-2 on 5th June 2021. All of the samples were inactivated by the heating the sample at 56 °C for 10 min. As precautionary measure, samples were processed inside BSC II bio safety cabinet before being delivered on to the Cobas 6800 instrument. The Cobas SARS-CoV-2 reagent kit was used. The limit of detection was calculated as 4.4 copies per reaction. All the protocols were performed as per the manufacturer's instructions (Pujadas et al. 2020).
Subsequently Head to Head comparison also done on 1631 samples collected on 14th and 15th June 2021 and processed on 15th June 2021. 600 µL of sample from viral transport medium vial were transferred into barcoded secondary tubes, in batches of 94 samples plus one negative control and one positive and run on cobas instrument following the manufacturer's instructions. The results can be seen directly on the system screen or can be printed as a report. Results can be analyzed as positive if either both ORF1 (target 1) and E (target 2) genes are or the ORF1 gene detected. If only E gene (target 2) is detected, the result should be reported as SARS-CoV-2 presumptive positive.
For presumptive positives cases, a quantitative algorithm was designed based on the previous study (Yang et al. 2020) by considering 5 parameters in the present study which includes (1) cases with consistent symptoms of COVID-19 with fever, cough, sore throat and upper respiratory symptoms; (2) cases with suspected symptoms for COVID-19 such as hypoxia, loss of test and insomnia; (3) cases with contact history; (4) additional PCR tests and (5) antibody test for COVID-19. For clarity in the results, scores (0-5) were assigned to each parameters as shown in Table 1. The assignment of scores was done on the basis of the severity and strength of the parameters which strongly support the COVID-19 infection. The result is considered as "confirmed positive", "true positive" or "false positive" when the total score is more than 5, 3-5 or < 3, respectively.

Statistics
Statistical analysis were performed using GraphPad Prism version 8.3.0 for Windows, Graph PadSoftware (SanDiego, CA, USA, www. graph pad. com). Cycle thresholds (Ct) were analyzed using oneway ANOVA or using Bland-Altman analysis by plotting the difference between two measurements on the Y axis, and the average of the two measurements on the X axis. The degree of agreement was quantified by the kappa value.

Agreement between the Cobas SARS-CoV-2 and the rRT-PCR platform on validation method
The results of comparative evaluation of the cobas and laboratory based tests on validation method consisting of 188 samples are summarized in

Agreement between the Cobas SARS-CoV-2 and the rRT-PCR platform on head to head approach
The comparison between the two tests on 1631 samples is summarized in Table 3. Around 65 samples results were found inconclusive which was send to NCDC Delhi for the confirmation and details of which is described further in the study. The diagnostic approaches showed overall agreement of 99% (1503/1566), positive agreement of 100% (1500/1500), negative agreement 95% (63/66) and showed kappa value of 0.9 on 1566 samples with valid result of both tests. Three discordant results were also obtained. Samples were found cobas positive and found negative with laboratory designed test, the possible reason could be the very low SARS-CoV-2 viral load (CT values for ORF1 (target 1) and E (target 2) genes of 33 and 36.5, respectively. Keeping in view of cobas manufacturer's instructions, samples can be considered as SARS-CoV-2 positive. For further investigation these samples were followed up and found these were tested SARS-CoV-2 positive by laboratory designed test 18 days earlier. Out of which, 2 were recovered from the infection but one was on the path of recovery.

Evaluation of inconclusive COVID-19 PCR results using quantitative algorithm
A total of 65 inconclusive cases were found in Cobas 6800, which were further sent to NCDC (Delhi) for the confirmation. In the 65 inconclusive NCDC assay for Cobas 6800, T2 was frequently detected than T1. The mean Ct values for T1 (32.5) and for T2 (33.5) and hence the cut off value was decided as 33. In case of inconclusive cases (n = 65) for Cobas 6800, 46 cases were found confirmed positive by NCDC assay with quantitative algorithm score ranging between 5 and 9. The entire "confirmed positive" cases either had found positive in additional PCR tests result or had antibodies related to COVID 19 infection. In case of deemed "false positive" cases (19), most of them (10/19) had nonspecific symptoms for COVID-19 and had exposure to confirmed cases.

Comparison the detection of ORF gene between Cobas SARS-CoV-2 and the rRT-PCR platform on various clinical specimens
In Government medical college and hospital Jammu, a total of 88 samples were detected with the Cobas 6800 (17 with a Ct < 25, 58 with a Ct between 25 and 34 and 13 with a Ct > 35). All the samples with a Ct < 35 (n = 75) were considered positive. Among the 13 samples with a Cobas 6800 Ct > 35, all were negative, where as in case of rRT-PCR, a total of 41 samples were detected (9 with a Ct < 25, 17 with a Ct between 25 and 34 and 15 with a Ct > 35). All the samples with a Ct < 35 (n = 26) were considered positive. Among the 15 samples with an rRT-PCR Ct > 35, all were negative. We found R 2 = 0.586. Deming regression revealed  a strong correlation with a slope of 0.724631 and an intercept of 10.84171 (Fig. 1a). Bland-Altman analysis showed that the mean difference (Cobas 6800 ORF ct minus rRT-PCR ORF) ct was − 3.88075, and with SD of + 5.681198 (Fig. 1b).

Comparison of the detection of gene E and ORF for Cobas 6800
Amplification of gene E (115/129, 89%) was more frequently positive compared to amplification of gene ORF (91/129, 70%). A R 2 of 0.94 and the Deming regression line revealed a good correlation with a slope of 0.89 and an intercept of + 1.99 (Fig. 2a). Bland-Altman analysis showed that the mean difference (Cobas 6800 ORF ct minus E gene) ct was1.8, and with SD of 32.4 (Fig. 2b).

Discussion
In view of unrivaled health and economic crisis due to the COVID-19 pandemic impact, demand for medical devices, sampling devices and diagnostic instruments for timely diagnosis has increased worldwide. To increase the testing capacity, with minimum loss in specificity and sensitivity, US FDA designed fully automated sample to result solution tool for the detection of SARS-CoV-2 infection. EUA has provided a fast and reliable approach for testing large numbers of patients in a reasonable time frame over the conventional molecular approach including several manual steps, RNA extraction, Master mixing, rRT-PCR and compilation of results after analyzing and interpretation. Cobas 6800 is an automated instrument allowing handling and testing of large number of samples reducing 50% intensive labor and hands-on time.

Fig. 2 Correlation curve (a) and Bland-Altman plot (b) for the Ct values of E and ORF gene of Cobas 6800
In this study, the sensitivity and specificity between the Cobas SARS-CoV-2 test (Roche Molecular Diagnostics) and rRT-PCR protocol was studied and compared. Cobas showed excellent overall agreement with the comparator rRT-PCR protocol on validation method with 20 discordant results (Table 2). On further investigation, it has been reported that eight patients samples were negative for SARS-CoV-2 virus by the rRT-PCR because the RdRp and N gene were undetected where as these samples were examined by the Cobas SARS-CoV-2 test and both tested positive for SARS-CoV-2 as the ORF1 and E gene targets were detected. All of these were in close contact with the COVID-19 patients. Other seven patients who diagnosed with COVID-19, after one month of treatment and quarantine, the throat swab showed a negative result by the rRT-PCR protocol because only the N gene was detected. However, these samples were positive results by the Cobas SARS-CoV-2. Rest five discordant cases are off the recent cases of COVID-19 with mild symptoms, showed a negative result with N gene detected by rRT-PCR, whereas Cobas SARS-CoV-2 test reported them as positive with both E and ORF 1 gene detected.
Overall percent agreement between the two platforms was found to be substantial which is indicated by Cohen's κ coefficient of 0.76 (95% CI, 2.5897-13.4103). Study also suggested a little discordance rate of 11.1% between the two studied platforms. This is in accordance with the other similar studies (Pujadas et al. 2020;You et al. 2021) which suggested the limit of detection (LOD) is lower for E gene as compared to N gene. Present study demonstrated about 4% (Cobas 6800) of the inconclusive results. It is found that majority of the inconclusive results are due to false positives cases with the high frequencies of asymptomatic cases, low prevalence or positive rate (Moriarty et al. 2020) and multiple PCR targets in an assay. It is important to detect these false-positive results in the interest of infection prevention and better care for patients (Schizas et al. 2020;Katz et al. 2020). Our analysis also showed that lowering the Ct cutoff from 37 (laboratory designed test for SARS-CoV-2 rRT-PCR) to 33 in the Cobas 6800 assay resulted in decrease number of false positive and enhance the clarity of interpretation of the results. In addition to this, it also increase the specificity with only a slight loss of sensitivity. This is also interested to note that there is need to update laboratory design protocol to reduce false amplification due to the viscosity of suspension, not proper vortexing of the PCR Master Mix and uneven suspension of the fluorescent dyes. These modifications in the protocol, in addition to the lowered Ct cutoff, may also help in the reduction of falsepositive results (Poljak et al. 2020;Vogels et al. 2020).
Due to the encouraging results obtained on the validation method, we immediate started head-to-head analysis of cobas on 1631 samples against the laboratory designed test. Overall agreement of 99% (1503/1566), positive agreement of 100% (1500/1500), negative agreement 95% (63/66) and kappa value of 0.9 on 1566 samples with valid result of both tests detected on head-to-head analysis. Three discordant results were also obtained. These were found cobas positive and found negative with laboratory designed test, the possible reason could be the very low SARS-CoV-2 viral load (CT values for ORF1 (target 1) and E (target 2) genes of 33 and 36.5, respectively, suggesting slightly higher analytical sensitivity of cobas over the laboratory designed test approach.
Two divergent assessments one the laboratory based rRT-PCR assay used with various extraction experiments and fully automated Cobas 6800 platform with Cobas 6800 kit, with more exhaustive functions, yielded a Ct value inversely correlated to the proportion of virus. In this study, Ct value below 35 in Cobas 6800 qualitative results are highly consonant and shows good correlation with the laboratory based rRT-PCR assay. In contrast to this, Ct value above 35, the laboratory based rRT-PCR failed to detect about one third of the SARS-CoV-2 genes while Cobas 6800 detected at least one of both targets. However, this observation is impacted by the extraction method (Jacot et al. 2020;Opota et al. 2020).
As compared to ORF, E gene is the most frequent detected target with the Ct value more than 35 in Cobas 6800 test. This suggests relatively higher sensitivity of the Cobas 6800 system for the E gene than the ORF which results in the declination of the positivity rate. This is the probably most related reason for the laboratory based rRT-PCR assay leads into most of the negative results in such cases. This restricts us to determine and assessed the limit of detection between the two methods studied. However, the limit of detection is much better in Cobas 6800 than the laboratory based on rRT-PCR test based on ICMR, Delhi protocol. This is in accordance with the other similar study (Pujadas et al. 2020;Opota et al. 2020). This variation in performance of detection between the two operating systems may be explained by the efficacy of primer/probe of target genes, sample volume and the initial amount of specimen.
Other probable reason for this difference includes the genomic sequences for SARS-CoV-2 designed by the primer and probe based on the sequences for accurately detected SARSCoV-2 virus by qRT-PCR by the various research groups across the world. USA,CDC protocol primarily targets N gene, whereas Taiwan CDC protocol target RdRp, E and N gene and Cobas 6800 automated system target ORF1 and E gene. These findings are in according with other reports which suggested that the N gene have a higher limitation of detection than the E gene (Pujadas et al. 2020;You et al. 2021). The laboratory-based test based on the modified USA CDC protocol has a lower input volume for the initial specimen (200 µL) compared with commercially available viral (universal) transport mediums have a volume of 3.0 mL but to enhance the sensitivity of the SARS-CoV-2 detection methods, 1.0 mL viral transport medium is used to achieve a concentrated specimen (You et al. 2021).

Conclusion
In conclusion, the results of both approaches, validation method consisting of 188 samples and head-to-head comparison on 1631 samples against the current diagnostic standard showed that cobas is a fast and reliable assay for qualitative detection of SARS-CoV-2 infection. The importance of the present study also suggests the role of multiple molecular diagnostic with high throughput analytical platform for testing of SARS-2 virus to control the spread of COVID-19. The performance of two analytic platform, Cobas 6800 and laboratory design test was studied on various clinical samples of SARS-2 infection. We found substantial agreement between the two analytic methods, although overall outcome of the study shows the limit of detection of Cobas 6800 is lower than the Laboratory design test. The results also suggest, the Cobas 6800 SARS-CoV-2 test has more sensitive which requires a minimum viral load and also shorten the time for diagnosis.
This study also has many limitations which includes limited availability of reagents for high number of clinical samples for testing, limiting understanding of COVID-19 infection, due to the no clinical references for comparison. However, more studies are required to compare analytic platforms, with more clinical data to come to the conclusion of testing decisions. However, the data presented would not allow the test to be validated in the registration process, but, as an added value for knowledge.