Next-generation sequencing (NGS) allows deeper insights than ever before into the human genome and a host of diseases and conditions. So it makes sense that there is a worldwide movement to employ NGS in a growing number of applications. But as the saying goes, with great power comes great responsibility.
Because of the many complexities unique to NGS, careful consideration must go into the design and implementation of robust NGS-based assays. We’ll unpack issues ranging from analytical validity, to statistics, to sample type considerations over a couple of posts. In part one, we’ll talk about key things to examine as you design your assay. Part two will focus on important factors in the development phase.
Robustness is Key
To maximize your assay’s effectiveness, it should be robust enough so that it is very likely to lead to accurate results and very unlikely to miss specifications. With NGS-based assays, standard operating procedures (SOPs) are often dozens – if not hundreds – of pages long. All of those steps should be easy to carry out and should be tolerant of expected variance.
Recently, several FDA-approved companion diagnostics (CDxs) have included “guard banding” studies where different steps were carried out with temperatures, concentrations, etc., shifted from their target values in order to assess the impact on the assay. Such studies are related to what is known as Design of Experiments (DOE), which is very helpful and often involves doing something that scientists were told to avoid (i.e., changing many variables at once). Guard banding studies can be revealing, and in the case of the Oncomine® Dx Target Test, showed that PCR temperatures must be within -1 °C to +0.5 °C of the specified temperature. Is your thermocycler accurate enough across the entire block to meet that tolerance?
It’s All About Validity and Utility
No matter how robust an assay is, it still must prove its worth on three fronts – analytical validity, clinical validity, and clinical utility. Because even the most robust assay is only useful if it can inform patient care.
Analytical Validity
First, a clinical NGS assay must demonstrate analytical validity by establishing that the assay measures what it claims to measure and meets end-user requirements. Analytical validation is never trivial and requires numerous studies that make extensive use of patient samples and reference materials. Because it is hard to be certain that a donor-derived sample does not contain an analyte, limit of blank (LOB) studies often use biosynthetic – or contrived – NGS reference materials that are known not to contain the analyte. Similarly, limit of detection (LOD) studies often use biosynthetic NGS reference materials that contain the analyte at specific levels. Precision studies measure the same sample(s) many times, which is not always possible with remnant samples, and can also benefit from using synthetic reference materials. In regard to precision, which measures the sources of variance, the CLIA “Survey Procedures and Interpretive Guidelines for Laboratories and Laboratory Services” indicates that “There are no specific guidelines for the number of times a material must be tested to establish statistical limits. In general, twenty replicate tests should be considered the minimum for determining a standard deviation.” 20 runs on a HiSeq® can be a significant time commitment.
Regarding the performance for meeting end user requirements, those may be set by the state for LDTs and will be indicated by the FDA through guidance documents for cleared assays. For approved assays, there may not be a minimum performance level and there will probably be extensive dialogue with the FDA. If clinical validity is required, then performance must be sufficient for the result to be clinically valid.
Clinical Validity
The FDA mandates that assays must demonstrate clinical validity in order to be cleared or approved. If a disease and the clinical role of the analyte in that disease are well understood, then establishing clinical validity is straightforward. If they are not well understood, then establishing clinical validity can require extensive clinical trials. Currently, many FDA-approved cancer assays are companion diagnostics that take a quantitative internal measurement of one or more analytes and output a qualitative (e.g., yes/no) result. Clinical validity is then established by showing that the group of patients that obtains a particular result is likely to have a favorable response with a drug (taking into account the risk of using that drug). This does not require understanding precisely how the analyte is tied to the response, and some patients might still benefit from using the drug even if they receive the opposite result from the test.
The sometimes blurry link between clinical validity and analytical validity can make it difficult to design assays that may be used as substitutes for companion diagnostics. For example, the PCR-based companion diagnostics that test for KRAS codon 12/13 mutations have LODs that typically range from 2-6%, but can be much higher (up to 30%) for some variants under certain circumstances. NGS assays that are designed for circulating cell-free DNA (ccfDNA) have LODs that can be well below 1% and can be adapted for other input materials. This then raises the question if the detection by NGS of an EGFR exon 19 deletion at 30% and a KRAS G12R mutation at 0.5% should preclude treatment with an EGFR inhibitor. With the PCR-based companion diagnostics, it would probably not. However, being unable to link an assay result to a clinical decision impacts the clinical validity of the assay.
Clinical Utility
Clinical utility is a requirement for reimbursement from insurance companies and clinical utility generally requires clinical validity because – outside of research – a test should only be performed if the result from that test will affect how that patient will be treated. Cost is also a factor. In the KRAS example, being able to detect KRAS mutations by NGS at 0.5% may not lead to higher reimbursement than for assays that can only detect them reliably at 5% if such low frequencies do not yet have established clinical utility. Finally, the insurance company will also have its own interpretation of clinical utility. For example, Aetna recently started to cover targeted solid organ genomic sequencing panels (5-50 genes) for – and presently only for – non-small cell lung cancer (NSCLC). It should be noted that this requires a diagnosis of NSCLC by other means, such as histology, and potential diagnostic methods like liquid biopsies are still considered investigational. So, for reimbursement, you may want your NGS assay to cover at least five genes.
What to Look For in Your AssayDiagnostic markers are important because they can be used to establish that a particular disease is present. In the case of lung cancer, there is not yet an established diagnostic marker, and looking for mutations with methods such as liquid biopsies in the absence of other signs of cancer can be seen as problematic. For example, every day, your body produces about 100 billion neutrophils that live for about half a day. Every time a cell divides, there is a chance that a new mutation or other genomic alteration appears somewhere in the genome. Every day, some neutrophil (and probably more than one) is likely to have a mutation or other genomic alteration that is associated with cancer and that may be detected in the blood. Thus, the baseline in the absence of cancer is unlikely to be just one copy, and may be higher than several copies. At the recent FDA-AACR Liquid Biopsies in Oncology Drug and Device Development Part II workshop, the term “clonal hematopoiesis” was used by several speakers to refer to an apparent increase in mutations in the cell-free DNA of older people. Similarly, your daily exposure to the sun and its UV radiation can cause mutations to appear in your skin cells; however, the vast majority of these cells are unlikely to be cancerous.
The Promise of Precision Medicine
In addition to the genes that are commonly mutated or otherwise altered in lung cancer, there is always a chance that a driver mutation from a different type of cancer that is uncommon in lung cancer will be present. Presently, there are several clinical trials such as NCI-MATCH that are looking to determine whether there is a benefit to selecting treatments based on genetic testing. Thus, if you believe that potential off-label use of drugs based on genetic testing might be successful, then you will want your lung cancer assay to look for the relevant alterations. Additionally, if you believe that the alterations present in a cancer are critical in its treatment, then diagnosing lung cancer as specifically lung cancer becomes irrelevant.
As you’re designing your assay - taking analytical validity, clinical utility, and clinical validity into account - it’s important to think ahead to the development phase and how you’ll acquire enough research specimens bearing the variants your assay will interrogate. Biosynthetic NGS reference materials can expedite this process for less money since they’re precisely quantitated, highly multiplexed, and contain rare and challenging variants in a sustainable supply.
In part two, we’ll explore key considerations for the assay development phase, including advice on platforms; sample type; and detecting mutations, fusions, and amplifications. In the meantime, here’s a great guide on how to develop a clinical NGS assay without losing your mind (or your shirt).