Since the introduction of the GS20 in 2005 by 454 Life Sciences, Next Generation Sequencing (NGS) has found many applications in clinical diagnostics. As a result of this transition from the long-held gold standard, Sanger sequencing, the primary challenge for clinical laboratories has shifted from data acquisition to ensuring these tests are safe and effective for guiding medical management decisions.
Many laboratories struggle to gain a thorough understanding of the analytic performance characteristics of their NGS tests. The difficulty arises from the fact that these assays are comprised of highly complex, fragmented workflows, and have many different intended uses. However, across the various practices currently used for NGS assay development, validation, and performance monitoring, there is a common goal: results must be as accurate, precise, and consistent as possible.
WHAT QUALITY CONTROL CHALLENGES DO NGS LABORATORIES FACE?
Today, there are many different offerings on the market across all components of the NGS workflow. Different tests use different extraction methods, target enrichment chemistries, library preparation techniques, sequencing platforms, and bioinformatic analysis pipelines. Each of these variables can have a profound impact on performance. Furthermore, external factors such as specimen characteristics, sample handling and stability, and the presence of interfering substances can also affect results.
Combined variation from these sources can lead to different types of failures: Random variation may reduce data quality so that results are not interpretable, leading to increased cost and delayed reports; on the other hand, systematic bias can cause incorrect results, the most catastrophic outcome of which is errors in patient treatment. Identifying the root cause of a failure can be a difficult task, yet is essential to determine risk to patients. Depending on the risk, a laboratory must then determine the time and resources needed for corrective actions.
In order to minimize patient risk and reduce costly downtime, highly effective Quality Control strategies are especially important for laboratories that are running NGS.
RELIABLE REFERENCE MATERIALS ARE A FOUNDATION OF AN EFFECTIVE NGS QC PROGRAM
As we previously wrote, according to US Food and Drug Administration guidelines, there are two primary ways to gauge the analytic performance of a diagnostic test: through use of a validated control, or by comparing to an orthogonal technology. In the case of NGS-based somatic mutation testing, there are currently no validated controls available on the market. Therefore, clinical laboratories who wish to produce their own reference materials must expend significant time and resources: ‘Home-brew’ materials must be collected or formulated, a constant supply must be maintained, and orthogonal testing must be performed. These efforts require constant attention from laboratory and Quality personnel, and must be accomplished via validated, SOP-driven processes to minimize the risk of errors such as sample-swaps or contamination.
To avoid the cost, time, and complexities of home-brew manufacturing, clinical NGS laboratories must have reliable access to high-quality reference materials.
KEY CHARACTERISTICS OF NGS REFERENCE MATERIALS
- Materials must challenge NGS assays across a broad range of variant types. Mutations that are analytically difficult for detection via NGS such as large insertion/deletion mutations and structural variants must be present, as these mutations are often the first to be affected by variability.
- NGS-based assays are highly multiplexed, and require reference materials that are also highly multiplexed. Cell lines or residual patient samples that are positive for a single mutation of interest may be cost-effective for highly targeted single-site tests; however, the information to cost ratio is far too low to be sustainable for comprehensive NGS screening. Use of highly multiplexed materials that are positive for a wide variety of mutations significantly increases the amount of information per run.
- It is necessary to evaluate test specificity, not just sensitivity. Successful detection of positive mutations provides assurance of a low False Negative rate. However, it is just as important to estimate False Positive rate, as these errors also lead to incorrect reporting that can harm patients. Residual patient samples do not have well-characterized genomic backgrounds, making it difficult to distinguish true variants from False Positives. Additionally, cell line admixtures are especially poor at evaluating specificity, as the background genomic data are confounded by the presence of many different genotypes.
- A robust, highly sensitive comparator method must be used to generate reference results. Use of a different NGS target enrichment library is not a true orthogonal method since results are generated via the same workflow components. Sanger sequencing may produce successful results for high quality starting materials carrying variants at relatively high allele frequencies, but will be insufficient for degraded samples with fragmented DNA that have low-frequency mutations. Therefore, a method such as digital PCR should be used to generate reference results.
- Reference materials must remain constant over time to allow well-controlled evaluation of change. Clinical tests are constantly changing as new targets are added, workflow components are updated, and new bioinformatics analyses are developed. A constantly changing reference material offers little value for assessing a constantly changing test since potential sources of variation are confounded.
- Unchanging reference results allow generation of an assay-specific baseline. QC data can then be compared with this baseline to determine whether there is a statistically significant difference that could indicate changes in assay performance. Establishing an assay-specific baseline is essential since results for a reference material on one NGS test are likely to differ from results produced by a different test.
- Flexibility and customization are vital to ensure in vitro diagnostic developers and laboratories have access to the right materials for the right tests. Highly targeted companion diagnostics need materials that are very focused to ensure robust detection across a defined set of biomarkers. On the other hand, comprehensive screening tests require a broad range of variant types across many different genes. Customization must also be rapid to ensure timelines are not delayed.
The ability to produce reference materials with these essential qualities is beyond what many laboratories are willing or able to accomplish via ‘home-brew’ practices. This is to be expected for organizations whose primary focus is high-throughput testing rather than manufacturing. External sourcing not only grants materials that are very high quality, but also ensures scientific rigor through independent evaluation. Health care providers, regulatory bodies, and payers should recognize laboratories that use these types of standards rather than opaque internal materials and practices.
For information about how SeraCare can help fulfill your quality control needs for Precision Medicine, please see us at booth 711 at Association for Molecular Pathology 2016 in Charlotte, NC. You can also visit us onliine or contact us here or via the comments below.
Next up: The most effective ways to use QC data to monitor the health of your NGS assay.
