AMP Reference Material Forum: Themes and Highlights

On November 14, 2017, AMP hosted a forum to discuss genetic testing reference material availability and needs. The forum attracted attendees including EQA providers, developers in industry and government, as well as scientists from clinical laboratories. Topics for discussion included reference material use and needs for assay validation, quality control, and proficiency testing. Throughout the talks, a few themes emerged and were discussed by multiple speakers.

Reference Materials Without “Hard” variants Give Only Part of the Story

A recurrent topic was that assessing numerous variants of each variant type during validation is critical, and “hard” variants must be included in both validation and daily QC. Steve Lincoln presented “An inter-laboratory study of complex variant detection in clinical testing” and showed that “hard” variants are very challenging for clinical detection. He defined “hard” variants as falling within three different categories (Table 1). First, the variant itself may be a difficult type, such as large indels, single exon CNVs, or structural variants. Second, the variant may be in a hard genomic region, such as in a homopolymer, a region that is not uniquely mappable due to pseudogenes, or regions of extreme GC content. Finally, variants may be hard to detect because they are in areas of low sequencing coverage. Although at first glance, “difficult” variants may appear rare, in fact, about 9% of pathogenic variants are difficult variant types, across several clinical areas.

The interlaboratory study consisted of 8 different laboratories, using a total of 10 different workflows, most clinically validated. The labs tested a highly multiplexed biosynthetic reference material enriched for hard variants in hereditary cancer testing. The study revealed that only 10 out of the 22 hard variants in the reference material were detected by all 10 NGS workflows. Split read analysis wasn’t part of several bioinformatics pipelines; hence larger indels that were not initially reported we were often found upon closer inspection of the raw sequencing data. Benign, easy SNPs were almost 100% detected across all labs, so an important conclusion of the study was that use of well characterized reference genomes such as NA12878 as validation or QC material would give labs a false sense of security in their ability to detect pathogenic variants. He further went on to recount Invitae’s experience with a testing issue which was not promptly detected and necessitated retesting thousands of patient samples; retesting costs time and money, which is why QC control with difficult variants is important.

The value of using reference materials with multiple challenging variants was echoed by Francisco De La Vega in his talk about “Reference Materials for Validation of Structural Variants Identification by NGS Cancer Gene Panels”. Structural variants are another difficult variant type and one of the most actionable biomarkers. The prevalence can be up to 50% of actionable variants. Fusions preserve oncogenic kinase domains, and may juxtapose it with a promoter or coiled-coil domain. Dr. De La Vega discussed the development and validation of a capture-based, clinical NGS assay that looks for split reads in order to detect fusion genomic breakpoints. He used highly multiplexed, reference materials that contained genomic structural variants NCOA4-RET and TPR-ALK at different allele frequencies to determine that false negatives occurred at 2.5% VAF, so this helped set the clinical thresholds and limits of detection appropriately.

Biosynthetics Have Unique Qualities as Reference Materials

A second recurrent theme was that although there will always be a need for clinical, patient specimens in validation and proficiency testing, reference materials, particularly biosynthetic NGS reference materials have unique advantages to supplement these studies. Sandi Deans discussed “Reference Materials for EQA: An EQA Providers Perspective from IQN Path”. IQN Path began as an umbrella organization for EQA in Europe and has now become a global initiative with member EQA providers in the United States, China and Australia. As proficiency providers, they recognize that there are distinct advantages of using biosynthetic reference materials in the proficiency challenges. A primary advantage is less heterogeneity. It is very important for all EQA samples to be the same and one of the limitations of clinical tissue specimens is there can be inconsistent expression and inconsistent content from one aliquot to the next. As manufactured products, reference materials can ensure uniformity between samples. A second advantage for EQA providers is that there is more variety in variants: biosynthetic reference materials make rare mutations available. They allow the provider to choose different clinically relevant mutations important for targeted therapy. Finally, biosynthetic reference materials offer standardization and ensure availability and scalability. They meet ISO standards and enable investigation of LOD for variant allele frequency. As an example, she discussed a ctDNA Pilot EQA Study for EGFR and KRAS/NRAS mutations in plasma. The initial pilot study had 32 labs participate, and they are currently in the planning stages for a second pilot study that will be even larger. Use of clinical specimens would not be possible for such large EQA studies, so patient-like, full process biosynthetic reference materials have been essential for producing challenging EQA schemes.

Francisco De La Vega went even further in describing how biosynthetic reference materials can sometimes be advantageous over other types of validation samples. He described validation of a CNV detection assay that works by analysis of read coverage depth with associated read normalization and GC correction. At first, his group tried to use a custom blend of cell lines for their validation studies. They “inevitably found a lot of other mutations” that required them to go back and design dPCR assays to verify that they are not false positives. The use of poorly characterized reference materials added a lot of time and cost to their study. He then used a biosynthetic reference material with engineered full gene increase in copy number. The reference material contained synthetic DNAs that are spiked into GM24385 genomic DNA, which is a homogeneous, highly characterized reference genome and completely resolves the issue of “background” unexpected variants. He stated that another advantage of highly characterized, biosynthetic reference materials is that “the ground truth is known,” and this facilitates assessment of assay sensitivity, specificity, and LOD.

Beyond these themes, a resounding request from almost all of the speakers was for more reference materials of all types…cell line based, patient specimen-derived, and biosynthetic. The speakers advocated for a wider variety of variant types, for a wider number of disease states, and for reference materials in a greater variety of formats and matrices.

To learn more about SeraCare’s growing library of patient-like, highly multiplexed reference materials purpose-built for NGS, click here.