I am pleased to share findings from a newly published peer-reviewed study with foundational circulating tumor DNA (ctDNA) pre-analytical and analytical testing in multiple technologies and assay chemistries. The study, “Multi-laboratory Assessment of a New Reference Material for Quality Assurance of Cell-Free Tumor DNA Measurements,” was just published in The Journal of Molecular Diagnostics (He, Stein et al. 2019). I use the term “foundational” since validation of the technology used to design, develop, and manufacture the ctDNA reference materials that are central to the study helps to establish a critical tool set that is required to standardize testing through comparison analysis, providing a much needed confidence boost to liquid biopsy-based cancer diagnostic or disease monitoring tests.
Before I discuss the study, I’ll highlight two recent publications that demonstrate why we need high-quality validated reference standards manufactured in large quantities, and why ctDNA testing is so promising for cancer patients.
The Promise of Liquid Biopsy
The promise of liquid biopsy cancer diagnostics is to use a minimally invasive procedure to detect metastasis or disease recurrence as early as possible. A hallmark for cancer treatment is that early detection and targeted treatment often leads to longer disease-free periods and better outcomes. ctDNA driver mutations or resistance markers are shed from the tumor into the circulation and can be detected in plasma. Many studies have shown concordance between driver genes found in the primary tumor to those found in blood. Sensitivity has been an issue since measuring just one variant allele mutation, e.g. a single nucleotide variant C>T mutation, for every thousand wild-type reads is a technical challenge when the sample is a blood draw.
Widespread adoption of a technically complex assay like ctDNA is even more complicated since it is offered by dozens of testing labs as an LDT and analytical performance is kept confidential making comparisons from lab to lab or assay to assay very difficult. To address this issue, AstraZeneca took matters into their own hands and initiated a comparison study of their own. This was just published in the Journal of Clinical Oncology in March. The study (Stetson, Ahmed et al. 2019) used patient samples to assess concordance between four labs offering ctDNA testing. Not unexpectedly, discordance was especially evident at the very lowest levels <1% VAF. In the Stetson study, twenty-four matched tumor plasma samples were sent to four commercial labs. Variant-level concordance was high in plasma samples above 1% variant allelic fraction (100% PPV for 3 of the 4 labs) but very poor overall (PPV ranged from 36% to 80%). As will be discussed below, ctDNA assays must approach 0.1% VAF to meet early cancer detection needs of patients.
Here is Table 2 from Stetson et al. showing the sensitivity of four commercial assays:
These results not only made clinicians pause when ordering tests, they also negatively impacted pharmaceutical companies validating genotypic biomarkers or enrolling patients in trials. The discordant results also prompted several projects to improve ctDNA testing via inter-laboratory concordance studies. One well-designed study underway is the Foundation for NIH Biomarkers study. This effort seeks to validate commercially available reference materials as aids to the development, validation, and QC NGS analysis of ctDNA-based molecular assays. The first phase (1a) of the F-NIH study is NGS assay analytical validation and a 1b study with patient samples is designed to assess commutability of the reference materials. The second phase is clinical validation of up to 14 genotypic biomarkers. BloodPAC is also engaged in inter-laboratory concordance studies for the same purpose.
On the positive side of ctDNA testing, recent reports from Memorial Sloan Kettering Cancer Center (MSKCC), Dana Faber Cancer Center, and several other cancer treatment hospitals/laboratories are very encouraging and highlights the potential of ctDNA testing in cancer patient management. Li, et al. show that ultra-deep sequencing (20,000 X) with clonal hematopoiesis filtering, can detect de novo driver mutations at <0.5% VAF (Li, Janku et al. 2019). Here is Figure 1c. showing lung cancer driver mutations and resistance mutations for 7 genes.
Note the overall number of driver mutations, especially those below 0.5% (green arrow) and the number of resistance mutations in red. Actionable resistance mutations including EGFR T790M, EGFR C797S, and ERBB2 amplification were found frequently enough to raise expectations that one of the main purposes for blood testing, early resistance monitoring, can be met. In addition to resistance, the authors state that their method is quite promising since targetable lung cancer driver mutations could be identified in plasma prospectively when tissue failed or could not be biopsied due to patient safety concerns.
Validation of ctDNA Reference Material Manufacturing Technology
Now let’s get back to the aforementioned multi-laboratory assessment. No need to repeat everything in this blog post that is reported in the manuscript, but I do want to provide contextual information to those readers who are interested in liquid biopsy standardization, or who are simply just interested in why we did the work in the first place. Let me highlight three observations from the report:
- An assessment of sample prep or A.K.A. the pre-analytical phase. I mentioned earlier how difficult it is to measure a single base change from the tumor cells from a blood draw. A prerequisite is achieving a high and consistent DNA extraction yield. Reference material stability is also critical considering manufacture and distribution around the globe. The data in Table 2 (He et al.) shows that the reference materials at 0%, 0.125%, 0.5% and 2% VAF manufactured by this novel technology can be used to compare DNA yields from 7 commercial DNA extraction methods with statistical precision. For example, one extraction kit using a manual extraction protocol will yield more DNA than the automated protocol (P< 0.05) for the same technology. One lab also shows that by adding RNA to the DNA elution buffer, the DNA yield almost doubles demonstrating the carrier effect works yet again. Maximizing DNA extraction is the first critical step to achieving maximal variant detection sensitivity. To accomplish this aspect using patient samples would have required about a unit of blood from a cancer patient, thus not a feasible approach.
- The use of droplet digital PCR to establish a reference value or “ground truth” for the material to which other methods can be compared. We show ddPCR is a reliable, low cost, and accurate method to characterize multiple types of variants (40 in total) to very low levels 0.125% VAF (~19 copies / ml). ddPCR was used to establish the initial “ground truth” variant allelic fraction measurements for comparison to low plex single gene assays and to high plex multigene NGS assays. At SeraCare, each variant in the reference material is quantitated using a validated ddPCR assay.
- The results provide a preliminary view of analytical sensitivity to 0.125% VAF for multiple variants SNVs, insertions and deletions. As discussed above, ctDNA assays must approach 0.1 VAF% to be sensitive enough for routine use. One of the most promising approaches to high sensitivity testing is the SimSen-Seq Assay developed by Drs. Tony Godfrey and Anders Stahlberg. Shown in Figure S2 from (He et al.) is the SimSen-Seq NGS Data for Reference Materials for Four Variants: PIK3CA (H1047R), KRAS (G12D), BRAF (V600E), and NRAS (Q61R). The errors bars are 1 standard deviation from n = 3 measurements and clearly show reproducible detection from 2% to 0.13%.
Lastly and not to be overlooked is overall variant detection. Figure 6 compares all NGS results tested by each of the seven laboratories at n=3 and shows detection precision at these levels.
Reference Materials are Necessary for Clinical Success
We are in the earliest days of ctDNA adoptions into clinical practice, thus the ability to compare assay results using a reference material made in bulk by GMP procedures is critical not only to assess the utility of the technology used to make the reference materials, but also to establish ground truth measurements that assay developers and clinical labs can rely on to standardize and make improvements. This technical approach to design, develop, and manufacture a set reference material for IVD manufacturers and clinical labs to identify and correct assay biases before expensive clinical trials are started is fundamental to the advancement of liquid biopsy.
So how did the collaborative effort start? In 2016, SeraCare and The National Institutes of Standards and Technology (NIST) jointly funded a Collaborative Research and Development Agreement (CRADA) to establish the requirements for the most useful reference materials and controls for ctDNA measurements. This is a collaboration of scientists and our Gaithersburg neighbors with an aligned purpose. SeraCare is focused on the design, development, manufacture, and distribution of quality control materials applied to clinical genomics, and NIST’s mission is to promote innovation and measurement science. NIST’s core competencies include the development and use of standards for biological applications including liquid biopsy. NIST directs resources to areas of greatest commercial need, and clinical genomics is a high priority. The NIST lab is very experienced with complex assay quantitation, statistics, qPCR, and ddPCR assay development and validation. Within the CRADA umbrella, we set out to understand what the requirements are for a fit-for-purpose reference material and to assess the variables of a liquid biopsy workflow through an inter-lab study. NIST was highly instrumental in recruiting leading laboratories for interlab testing including Frederick National Laboratory for Cancer Research, Boston University School of Medicine / Sahlgrenska Cancer Center, Johns Hopkins University School of Medicine, University of North Carolina School of Medicine, Asuragen, ArcherDX, and National Cancer Institute.
The design requirements for a ctDNA multiplex reference material are complex and many, and earlier in the blog I stated the results are foundational for future products. By this I mean we now have a very good assessment of how these materials work across a range of assay formats and extraction kits.
Let me highlight some important lessons learned for the performance of the reference materials from the study. First at the molecular level. The DNA must have similar size 160-170 base pairs to natural ctDNA, and it did. The mix should contain clinically relevant variants that are difficult to detect such as EGFR exon 19 deletions c.2236_2250del15 and be able to detect more common, easier to detect SNVs BRAF c.1799T>A. The mutation mix used for the study contained 40 variants (SNVs, insertions, and deletions) and it demonstrates we can make a highly multiplexed material that tested just once in an NGS assay, generates a tremendous amount of high quality data, easily increases productivity, and saves the lab precious sequencing costs by not having to run multiple samples at $400 per sample.
Through many designs of experiments at SeraCare R&D, we’ve optimized design of the biosynthetic variants, spike-in to genomic DNA, fragmentation, sizing, and end-repair to best represent natural patient-like reference standard. The %VAF must be manufactured to be below a clinical cutoff. Based on the (Li, Janku) study discussed above, ~7% patients have driver or resistance mutations between 0.1-0.5% VAF. From a manufacturing viewpoint, this means, using BRAF V600E as an example, that for every 1000 molecules there are 999 “T’s” and just 1 of the variant nucleotide sequence “A”; see COSMIC 476 c.1799T>A. We applied precise and accurate manufacturing approaches and QC measurements afforded by allele-specific ddPCR for each variant in the mix at these lowest target levels 0.125% VAF. These same methods can be applied to ten-fold less.
Not to be overlooked is the macroscale, the physical form of the reference material, and how the DNA is packaged and in what diluent. Although the half-life of natural ccfDNA in normal circulation is very short, ~16 min due to tissue uptake and degradation, it has been shown that naturally circulating DNA is mostly protected from nucleases (Lo, Zhang et al. 1999). At SeraCare, we decided to replicate this feature of ccfDNA by mimicking the apoptotic body form found in circulation by using a highly reproducible, liposome encapsulation method developed and patent pending by SeraCare. This method protects the DNA from nucleases and damaging polymerization, and it improves long term stability at 4oC. Finally, the reference material was formulated in a synthetic SeraCon Matribase (SeraCare). This diluent is formulated to ensure total protein of 5.9-6.6 g/dL (human albumin); endogenous ions, salts, glucose etc. are always at the same physiological specification, and it ensures there is no assay-interfering hDNA carried over from human plasma or enzyme inhibitors that might impact library prep or sequencing. The data also show that the material for this version is not ideally suited for specificity assessment due to background noise in the wild type material. This feature has already been addressed in newer versions of the technology which are more suited for specificity testing. (Stay tuned as this is a future blog post topic).
Conclusions
So, based on the data, where do we stand and where do we go from here?
- Flexible design. We have a flexible reference material manufacturing technology. Each of the steps described above to make reference materials with different mutations and at differing allelic fractions is easily reproduced for different assay formats and cancers.
- Large scale. The materials are made in large scale for broad distribution to clinical labs around the world. The materials can be used for proficiency testing or for in-kit positive controls.
- Stable. The materials are stable in SeraCon Matribase for > 3-year real time at 4oC.
- Analytical sensitivity. Due to the design features, materials can be made to bracket clinically relevant cut-offs and help set assay sensitivity levels to 0.125% VAF or lower.
- Routine QC. Guidelines like New York State Department of Health and the FDA recommend daily run controls and the form and function of these materials are ideal to QC systems and daily QC.
- Data to make improvement for the next-generation product. The results identify the need for an improved method to assess specificity. We have solutions that are being tested and implemented in new product designs. Stay tuned.
Scientific Poster Video
Multi-Center Evaluation of Circulating Tumor DNA Assays
This multicenter study was made possible with the help of CANCER-ID, a consortium of experts from companies and institutions looking to establish standard protocols for and clinical validation of blood-based biomarkers. This trial was established to help standardize complex ctDNA diagnostic assays by testing highly complex reference standards.
Download this poster and watch the video to learn more about this trial which covers the entire workflow from sample preparation through variant reporting.
He, H. J., et al. (2019). "Multi-Laboratory Assessment of a New Reference Material for Quality Assurance of Cell-Free Tumor DNA Measurements." J Mol Diagn.
We conducted a multi-laboratory assessment to determine the suitability of a new commercially-available reference material with 40 cancer variants in a background of wild type DNA at four different variant allele fractions (VAF): 2%, 0.5%, 0.125%, and 0%. The variants include single nucleotides, insertions, deletions, and two structural variations selected both for their clinical importance, and to challenge the performance of next-generation sequencing (NGS) methods. Fragmented DNA was formulated to simulate the size distribution of circulating wild-type and tumor DNA in a synthetic plasma matrix. DNA was extracted from these samples and characterized with different methods and multiple laboratories. The various extraction methods had differences in yield, perhaps due to differences in chemistry. Digital PCR assays were used to measure VAFs to compare results from different NGS methods. Comparable VAFs were observed across the different NGS methods. This multi-laboratory assessment demonstrates that the new reference material is an appropriate tool to determine the analytical parameters of different measurement methods and ensuring their quality assurance.
Li, B. T., et al. (2019). "Ultra-deep next-generation sequencing of plasma cell-free DNA in patients with advanced lung cancers: results from the Actionable Genome Consortium." Ann Oncol.
BACKGROUND: Noninvasive genotyping using plasma cell-free DNA (cfDNA) has the potential to obviate the need for some invasive biopsies in cancer patients while also elucidating disease heterogeneity. We sought to develop an ultra-deep plasma next-generation sequencing (NGS) assay for patients with non-small-cell lung cancers (NSCLC) that could detect targetable oncogenic drivers and resistance mutations in patients where tissue biopsy failed to identify an actionable alteration. PATIENTS AND METHODS: Plasma was prospectively collected from patients with advanced, progressive NSCLC. We carried out ultra-deep NGS using cfDNA extracted from plasma and matched white blood cells using a hybrid capture panel covering 37 lung cancer-related genes sequenced to 50 000x raw target coverage filtering somatic mutations attributable to clonal hematopoiesis. Clinical sensitivity and specificity for plasma detection of known oncogenic drivers were calculated and compared with tissue genotyping results. Orthogonal ddPCR validation was carried out in a subset of cases. RESULTS: In 127 assessable patients, plasma NGS detected driver mutations with variant allele fractions ranging from 0.14% to 52%. Plasma ddPCR for EGFR or KRAS mutations revealed findings nearly identical to those of plasma NGS in 21 of 22 patients, with high concordance of variant allele fraction (r = 0.98). Blinded to tissue genotype, plasma NGS sensitivity for de novo plasma detection of known oncogenic drivers was 75% (68/91). Specificity of plasma NGS in those who were driver-negative by tissue NGS was 100% (19/19). In 17 patients with tumor tissue deemed insufficient for genotyping, plasma NGS identified four KRAS mutations. In 23 EGFR mutant cases with acquired resistance to targeted therapy, plasma NGS detected potential resistance mechanisms, including EGFR T790M and C797S mutations and ERBB2 amplification. CONCLUSIONS: Ultra-deep plasma NGS with clonal hematopoiesis filtering resulted in de novo detection of targetable oncogenic drivers and resistance mechanisms in patients with NSCLC, including when tissue biopsy was inadequate for genotyping.
Lo, Y. M., et al. (1999). "Rapid clearance of fetal DNA from maternal plasma." Am J Hum Genet 64(1): 218-224.
Fetal DNA has been detected in maternal plasma during pregnancy. We investigated the clearance of circulating fetal DNA after delivery, using quantitative PCR analysis of the sex-determining region Y gene as a marker for male fetuses. We analyzed plasma samples from 12 women 1-42 d after delivery of male babies and found that circulating fetal DNA was undetectable by day 1 after delivery. To obtain a higher time-resolution picture of fetal DNA clearance, we performed serial sampling of eight women, which indicated that most women (seven) had undetectable levels of circulating fetal DNA by 2 h postpartum. The mean half-life for circulating fetal DNA was 16.3 min (range 4-30 min). Plasma nucleases were found to account for only part of the clearance of plasma fetal DNA. The rapid turnover of circulating DNA suggests that plasma DNA analysis may be less susceptible to false-positive results, which result from carryover from previous pregnancies, than is the detection of fetal cells in maternal blood; also, rapid turnover may be useful for the monitoring of feto-maternal events with rapid dynamics. These results also may have implications for the study of other types of nonhost DNA in plasma, such as circulating tumor-derived and graft-derived DNA in oncology and transplant patients, respectively.
Stetson, D., et al. (2019). "Orthogonal Comparison of Four Plasma NGS Tests With Tumor Suggests Technical Factors are a Major Source of Assay Discordance." JCO Precis Oncol(3): 1-9.
PURPOSE Discordance between plasma and tumor variant calling has been attributed primarily to tumor heterogeneity, whereas technical variables remain largely unexplored.MATERIALS AND METHODS To measure these variables, we tested four next-generation sequencing (NGS) gene panel assays for mutations in circulating tumor DNA (ctDNA) using replicate sets of 24 plasma samples and compared the results with matched tumor-normal tissue pairs.RESULTSOur orthogonal approach identified false-negative (FN) and false-positive (FP) variants with high confidence and revealed substantial variability among the ctDNA assays, with a range of sensitivity (38% to 89%) and positive predictive value (36% to 80%). Most discordance in our cross-vendor study was observed below 1% variant allele frequency. FP variants displayed mutational biases and tended to be novel variants not found in somatic databases. Of the 56 unique variants called by all four ctDNA assays, 41 (68%) resulted from technical discordance.CONCLUSION These findings suggest that most NGS assay discordance is a result of technical variations and, to a lesser extent, biologic factors such as clonal hematopoiesis of indeterminate potential and tumor heterogeneity.
