After 17 months of deliberations since its first open meeting February 20, 2015 on NGS IVD assay oversight, the U.S. Food and Drug Administration (FDA) issued DRAFT guidance for Stakeholders and FDA staff. The document “Use of Standards in FDA Regulatory Oversight of Next Generation Sequencing (NGS)-Based In Vitro Diagnostics (IVDs) Used for Diagnosing Germline Diseases” was published online on July 6, 2016. This document (PDF located here) is for analytical validity and not for clinical validation.
The FDA, the U.S. Federal Government’s Precision Medicine Initiative, the IVD industry and the entire medical community all realize how critical genetic profiling is to the diagnosis and treatment of disease and how central it is to precision diagnostics. In fact, today there are over 600 laboratories worldwide performing NGS germline testing. Most importantly, germline DNA sequencing data will always be part of a person’s health record so the data will have lasting impact.
Although somewhat unusual for our industry (but not unexpected considering the importance of these guidelines to manufacturers), Illumina posted a press release within hours stating its support for the FDA highlighting the draft guidelines “…provide appropriately flexible and adaptive regulatory oversight of these tests…”. It is not often that a commercial entity is first to praise a United States Regulatory Body.
The Draft Guidance clearly emphasizes that NGS IVD for Germline Disease is to be an aid in diagnosis and specifically limited to targeted gene panels and whole exome sequencing (WES). The document is quite thorough and encompasses many, if not all, of the critical elements for development and validation: a) specimen collection, processing and storage, b) DNA extraction, c) DNA processing and library preparation, d) generation of sequence reads and base calling, e) sequence alignment / mapping, f) variant calling, g) variant annotation and filtering, h) variant classification / interpretation, and lastly i) generation of the test report. The general outline is very similar to the ACMG NGS guidelines for clinical laboratories published in 2013 (Rehm, Bale et al. 2013).
However, the FDA document is much more specific in the data required to obtain release of a product as an IVD. Considering the on-going public debate on the need for enhanced assay quality, and also the need for greater transparency to ensure accurate patient testing results, I was surprised the FDA used a rather loose term “as needed” when specifying which Controls and Reference Materials are needed to achieve confidence in a genetic test. In fact and from an analytical viewpoint, large areas of medically actionable genes fall within unknown confidence areas by NGS sequencing. Goldfeder et al. reported that 82.1% of exonic bases in ACMG-reportable genes from CLINVAR fell within high-confidence regions leaving 17.9 % falling within unknown or low confidence areas. (Goldfeder, Priest et al. 2016).
The Recommendations state that IVD manufacturers should include controls and reference materials per sample and per run etc. “as needed”, in order to establish the quality of performance. “As needed” implies sequencing metrics like base coverage and Q-scores are enough to confirm assay run performance on that day. In my opinion, not requiring a positive control in an NGS assay is really unexpected. By not opining to require running a positive control leaves the decision to the manufacturer’s discretion. Running a positive control is likely to be minimized or eliminated considering the high costs of all the other reagents, flow cells, hardware and software. It also, discounts the experience many clinical labs already have with NGS. Gargis et. al. recommend running a well characterized control for each and every assay with patient samples to show the assay is working and to use a rotating control schedule to bring genetic variety to the overall QC strategy. (Gargis, Kalman et al. 2012)
Population-level data is available via the Clinical Genome Resource and there are NGS results published that demonstrate the importance of running positive controls. For example, the Laboratory for Molecular Medicine at Harvard-Partners Healthcare has tested over 3,000 probands for cardiomyopathy thus has extensive experience and a data base (Alfares, Kelly et al. 2015) on which genes and which variants are associated with cardiomyopathy and which genes and variants are difficult to assess analytically. Expert design of the run control should be based on patient testing experience and should be required. Furthermore, the Molecular Characterization Lab at the Frederick National Laboratory for Cancer Research of the National Cancer Institute (NCI) (Sims, Harrington et al. 2016) have shown pathogenic variants are missed by different versions of the same pipeline. It seems to me that if the intended use of an IVD is in the aid to diagnosis cardiomyopathy or inherited cancer or cystic fibrosis or any other inherited disease, the FDA should “insist on” and use the appropriate regulatory language to include positive controls in the assay. We have to ensure that the genetic data from that patient on that assay on that day are analytically valid. Starting with “as needed” will lead to cutting QC corners on Day 1 and will continue to fuel to lack of quality and lack of transparency we encounter today.
The FDA is seeking comments on the draft guidelines on or before October 6th 2016, and the link to submit comments is located here. I highly recommend that interested parties including clinical lab personnel, patient groups and physicians take the time to carefully assess the long term impact of a genetic test to an individual’s long term care and profiling and take time to respond to the FDA. Just ask if the best quality control systems are in truly in place for an IVD labeled product. Everyone should remember that the genetic data will reside in the patient’s record forever. I know I will respond by saying “as needed” should be replaced by “is required” for NGS Controls and Reference Materials.
References and Abstracts (also linked to PubMed above):
Alfares, A. A., et al. (2015). "Results of clinical genetic testing of 2,912 probands with hypertrophic cardiomyopathy: expanded panels offer limited additional sensitivity." Genet Med 17(11): 880-888.
Hypertrophic cardiomyopathy (HCM) is caused primarily by pathogenic variants in genes encoding sarcomere proteins. We report genetic testing results for HCM in 2,912 unrelated individuals with nonsyndromic presentations from a broad referral population over 10 years. METHODS: Genetic testing was performed by Sanger sequencing for 10 genes from 2004 to 2007, by HCM CardioChip for 11 genes from 2007 to 2011 and by next-generation sequencing for 18, 46, or 51 genes from 2011 onward. RESULTS: The detection rate is ~32% among unselected probands, with inconclusive results in an additional 15%. Detection rates were not significantly different between adult and pediatric probands but were higher in females compared with males. An expanded gene panel encompassing more than 50 genes identified only a very small number of additional pathogenic variants beyond those identifiable in our original panels, which examined 11 genes. Familial genetic testing in at-risk family members eliminated the need for longitudinal cardiac evaluations in 691 individuals. Based on the projected costs derived from Medicare fee schedules for the recommended clinical evaluations of HCM family members by the American College of Cardiology Foundation/American Heart Association, our data indicate that genetic testing resulted in a minimum cost savings of about $0.7 million. CONCLUSION: Clinical HCM genetic testing provides a definitive molecular diagnosis for many patients and provides cost savings to families. Expanded gene panels have not substantively increased the clinical sensitivity of HCM testing, suggesting major additional causes of HCM still remain to be identified.
Gargis, A. S., et al. (2012). "Assuring the quality of next-generation sequencing in clinical laboratory practice." Nat Biotechnol 30(11): 1033-1036.
Goldfeder, R. L., et al. (2016). "Medical implications of technical accuracy in genome sequencing." Genome Med 8(1): 24.
As whole exome sequencing (WES) and whole genome sequencing (WGS) transition from research tools to clinical diagnostic tests, it is increasingly critical for sequencing methods and analysis pipelines to be technically accurate. The Genome in a Bottle Consortium has recently published a set of benchmark SNV, indel, and homozygous reference genotypes for the pilot whole genome NIST Reference Material based on the NA12878 genome. METHODS: We examine the relationship between human genome complexity and genes/variants reported to be associated with human disease. Specifically, we map regions of medical relevance to benchmark regions of high or low confidence. We use benchmark data to assess the sensitivity and positive predictive value of two representative sequencing pipelines for specific classes of variation. RESULTS: We observe that the accuracy of a variant call depends on the genomic region, variant type, and read depth, and varies by analytical pipeline. We find that most false negative WGS calls result from filtering while most false negative WES variants relate to poor coverage. We find that only 74.6 % of the exonic bases in ClinVar and OMIM genes and 82.1 % of the exonic bases in ACMG-reportable genes are found in high-confidence regions. Only 990 genes in the genome are found entirely within high-confidence regions while 593 of 3,300 ClinVar/OMIM genes have less than 50 % of their total exonic base pairs in high-confidence regions. We find greater than 77 % of the pathogenic or likely pathogenic SNVs currently in ClinVar fall within high-confidence regions. We identify sites that are prone to sequencing errors, including thousands present in publicly available variant databases. Finally, we examine the clinical impact of mandatory reporting of secondary findings, highlighting a false positive variant found in BRCA2. CONCLUSIONS: Together, these data illustrate the importance of appropriate use and continued improvement of technical benchmarks to ensure accurate and judicious interpretation of next-generation DNA sequencing results in the clinical setting.
Rehm, H. L., et al. (2013). "ACMG clinical laboratory standards for next-generation sequencing." Genet Med 15(9): 733-747.
Next-generation sequencing technologies have been and continue to be deployed in clinical laboratories, enabling rapid transformations in genomic medicine. These technologies have reduced the cost of large-scale sequencing by several orders of magnitude, and continuous advances are being made. It is now feasible to analyze an individual's near-complete exome or genome to assist in the diagnosis of a wide array of clinical scenarios. Next-generation sequencing technologies are also facilitating further advances in therapeutic decision making and disease prediction for at-risk patients. However, with rapid advances come additional challenges involving the clinical validation and use of these constantly evolving technologies and platforms in clinical laboratories. To assist clinical laboratories with the validation of next-generation sequencing methods and platforms, the ongoing monitoring of next-generation sequencing testing to ensure quality results, and the interpretation and reporting of variants found using these technologies, the American College of Medical Genetics and Genomics has developed the following professional standards and guidelines.
Sims, D. J., et al. (2016). "Plasmid-Based Materials as Multiplex Quality Controls and Calibrators for Clinical Next-Generation Sequencing Assays." J Mol Diagn 18(3): 336-349.
Although next-generation sequencing technologies have been widely adapted for clinical diagnostic applications, an urgent need exists for multianalyte calibrator materials and controls to evaluate the performance of these assays. Control materials will also play a major role in the assessment, development, and selection of appropriate alignment and variant calling pipelines. We report an approach to provide effective multianalyte controls for next-generation sequencing assays, referred to as the control plasmid spiked-in genome (CPSG). Control plasmids that contain approximately 1000 bases of human genomic sequence with a specific mutation of interest positioned near the middle of the insert and a nearby 6-bp molecular barcode were synthesized, linearized, quantitated, and spiked into genomic DNA derived from formalin-fixed, paraffin-embedded-prepared hapmap cell lines at defined copy number ratios. Serial titration experiments demonstrated the CPSGs performed with similar efficiency of variant detection as formalin-fixed, paraffin-embedded cell line genomic DNA. Repetitive analyses of one lot of CPSGs 90 times during 18 months revealed that the reagents were stable with consistent detection of each of the plasmids at similar variant allele frequencies. CPSGs are designed to work across most next-generation sequencing methods, platforms, and data analysis pipelines. CPSGs are robust controls and can be used to evaluate the performance of different next-generation sequencing diagnostic assays, assess data analysis pipelines, and ensure robust assay performance metrics.
