There is an even greater onus to track quality control metrics for NGS assays because of the number of steps and elements that must successfully work together to produce consistent results. The Standards and Guidelines for Validating NGS Bioinformatics, published in January by AMP and CAP, highlight the importance of tracking QC metrics over time because “trends in these metrics can indicate an emerging issue with an NGS process that has not yet manifested itself in failed tests.” Trends of abnormal precision in an assay can stem from variability in elements of the assay that can change over time including reagents, operators, and instruments. Clinical labs use reference materials to provide consistent, run-to-run, ground truth quality control data to be used in trend analysis. While the accuracy of an assay - defined as the ability to detect the variants it claims - is expected to remain consistent, the assay’s precision - the closeness of results from independent runs - is determined by the amount of input material and the depth of sequencing. If both parameters remain consistent, then variability in precision can be attributed to sources such as reagents, operators, and instruments. Variability in these elements would manifest in the assay performing as if it had less input material or lower sequencing depth.
While tried and true Levey-Jennings plots are must-haves for any robust NGS QC program, they’re only helpful for analyzing the performance of one variant at a time. They make it very laborious to determine overall assay performance drift and troubleshoot because they only give you a piecemeal view.
To address this difficulty, we’re excited to introduce the SeraCare Confidence Score™, a novel precision metric designed specifically for NGS assays that provides a singular view into your assay’s precision. The Confidence Score offers clinical labs a comprehensive look into how different reagent lots, operators, and instruments impact the precision of their assay, streamlining the troubleshooting process to accelerate time to resolution and maximize assay uptime.
What is the SeraCare Confidence Score?
Simply put, the Confidence Score helps you determine the precision of your assay using all of the known variants in a reference material. Specifically, the Confidence Score is calculated by comparing the aggregated variance of the reference material’s variants from their target values in a run (effective depth) to the variance that can be expected from the DNA input amount and sequencing depth (expected depth).
This focuses on known variants as we are particularly interested in assessing the precision of measuring the count of different variant alleles (the numerator of the variant allele frequency). To approximate the probability of observing a variant allele, the average allele frequency of each variant is calculated over a set of runs that use independent libraries. The difference between this average and the allele frequency in a run measures the deviation for that variant and can be compared with the expected depth to get a relative score of deviation. To get the score for a sample or run, all the scores of the known variants detected in the run are averaged.
Interpreting and Applying the Confidence Score
The score can be added to the recommended set of QC metrics already used in a variety of scenarios from assay development through assay monitoring, including establishing new kit lot equivalency, pinpointing variants that have higher than expected variability (unexpected additional sources of variance), routine assessment of instrument equivalency, and supervising new operator performance. Not engaging in the proper analysis for these scenarios can lead to unexpected failed runs because problems were not detected early enough. Typically, these analyses require a lab to assess performance by looking at a subset of variants by variant type to reduce the amount of labor and time is necessary to analyze the data. Instead, the Confidence Score makes it simpler to identify trends and abnormal performance by aggregating all the variant data.
Drops or trends in the score should prompt investigations into the differences in a run or series of runs and the review of other quality control metrics that will offer more specific insight into where and what may be causing the variability.
For example, in a dataset composed of 33 runs of the Seraseq Solid Tumor Mutation Mix gathered during an interlab study, analysis using the Confidence Score identified two variants (Figure 1) with significantly lower scores that in turn caused the sample and run scores to also be more variable than expected (Figure 2). If the assay was still in development, the relevant amplicons could have been improved. But because the assay was already in use, the variants were removed from this analysis. Removing these variants normalized the scores by sample (Figure 3) and identified a run by a new operator that performed with more variability than runs by the other operators, offering the opportunity to provide additional training to the new operator and align their performance with the standards of the lab. There also was another run identified with more variability than expected that would prompt a troubleshooting investigation using the recommended QC metrics and associated run meta-data to understand the source of variability.
Figure 1 – Confidence Score by variant analysis
Figure 2 – Confidence Score by sample analysis (prior to removal of two target regions)
Figure 3 – Confidence Score by sample analysis (post-removal)
These essential analytical reports are available in the iQ NGS QC Management software and only need the VCFs from a highly multiplexed reference material to get started. Having these reports a few clicks away makes identifying and troubleshooting performance drift a matter of minutes, not hours.
Learn more about SeraCare’s complete QC solution for clinical labs running NGS assays here.
Are you looking for clarity on validation guidance?
Our new eBook offers insights for more effective and less costly assay management and will walk you through key considerations and guidelines that will ensure a smooth and successful validation.
"Nothing is more frustrating than finding a sample that is positive for a relevant variant but cannot be tested multiple times due to sample depletion."