Genomic Precision

 A SeraCare blog focused on precision medicine and advanced clinical diagnostics

The Trouble with Troubleshooting Clinical NGS

Posted by Peter Duncan on Nov 15, 2018 11:00:00 AM
Figure 1 Frustrated Actor in PocketHose Infomercial

Have you ever seen those late night infomercials? One of the things I love whenever I have watched these is the over-the-top acting to depict whatever frustration the target audience must be feeling if they can’t coil up their garden hose or manage their closet space.

As frustrated as these actors are, I can’t help but imagine what an actor’s depiction of the level of frustration that a clinical genomics lab director might be feeling when things don’t go according to plan while carrying out an NGS assay.

Figure 1 Frustrated Actor in PocketHose Infomercial

As many already know, NGS diagnostic testing is classified as “High Complexity.”1 The workflow to generate an actual answer is incredibly complex and can take several steps across several days to complete. Multiple reagents, operators, instruments, and computer programs all have to work in concert, reproducibly, to generate results that a clinician can rely on in order to determine the best course of treatment. The more complex a system, the more likely it is that things will go awry. It’s entropy, applied in real time.


Figure 2 Typical NGS Workflow

When the inevitable happens, troubleshooting is the name of the game. The following is a simplified version of the course of action that is typically taken:

  1. error observed
  2. hypothesis of root cause
  3. confirm hypothesis
  4. remediate
  5. confirm root case has been addressed
  6. proceed with testing. 

In more expanded terms, this process can take several days to several weeks to complete. In the meantime, results to clinicians are delayed, reagents are spent usually at the cost to the lab, key personnel are allocated to get things back up and running again, and the backlog of testing increases for the lab further delaying test results for patients. In severe cases, lab results may have to be retracted, which may include contacting many patients to explain their results may be erroneous and will require repeat testing.

Exacerbating this concept, some of the issues that a lab faces are acute (improper pipetting, damaged reagents, etc.) while others are chronic and can present themselves over longer windows of time and may be harder to trace. For example, a manufacturer’s reagent will naturally have performance drift over time. This drift can be well within the manufacturer’s specifications. But if a lab happens to validate at different intervals throughout the time span of a few years, they may, in fact, fail their validation depending on when the validation runs are conducted with respect to the performance of reagent lots they have received. This type of event can be incredibly time consuming and expensive to resolve.


Figure 3 Hypothetical representation of longitudinal validation of reagent lots. The reagent performance meets manufacturing spec (red line) but fails validation for “User Y.”

Wouldn’t it be better to proactively monitor assay performance so that proper planning could happen before an assay needs to be taken off line? For example, an assay might not be technically failing, but it is trending downwards in key performance areas. Proactively allocating patient samples to another instrument while conducting a root-cause investigation might be a better approach than waiting for the assay to actually fail and dealing with the aftermath.

This type of scenario is exactly what we had in mind at SeraCare when we developed the Confidence Score.

The Confidence Score is a single precision metric designed specifically with highly multiplexed NGS assays in mind that provides a singular view into an assay’s performance. The Confidence Score offers clinical labs a comprehensive view into how different reagent lots, operators, and instruments impact the precision performance of their assay, streamlining the troubleshooting process to accelerate time to resolution and maximizing assay uptime.

The Confidence Score is calculated by comparing the expected performance from validation runs based on the sample input and sequencing depth with the observed performance in a run. A score of “1” is the desired result and indicates that the level of variance observed in the run is equivalent to the variance around the averages seen in the validation runs. If a lab is seeing the Confidence Score drifting downwards over time, then they may want to proactively investigate what is creating this effect using the tools within iQ NGS QC Management software.

In the below example, we are tracking the Confidence Score over time using data from a study of our Seraseq Solid Tumor Mutation Mix reference material published in the Journal of Applied Laboratory Medicine (Figure 4). We see that the Confidence Score is trending downwards. With just a few mouse clicks, we can isolate the cause of one of the outliers to a specific operator (Figure 5). The lab can now proactively address this issue with additional training instead of reactively isolating the root cause, and therefore avoid excess cost, time, or even the loss of lab credibility.


Figure 4 Longitudinal tracking of Confidence Score with a downward trend demonstrated


Figure 5 Identification of source of outlier Confidence Score – in this case, a new operator was the cause

The Confidence Score can be also be used to monitor reagent performance, instrument performance, bioinformatics versions, chip or flow cell lots, automated liquid handlers, or any other variable that the lab believes may have an impact on assay performance. As a result, labs can longitudinally monitor assay performance to ensure that results of a given assay are consistent.

Building on this concept, SeraCare will be launching a new capability within iQ NGS QC Management that allows multiple labs to benchmark assay performance with one another. For example, if you are a lab that has multiple locations running the same assay, leveraging both the Confidence Score within iQ NGS QC Management and SeraCare’s Seraseq reference materials can ensure that each location is performing consistently, ensuring the best quality results.

Late night infomercial actors can be quite entertaining to watch and ironically, were the inspiration for why I wrote this blog post. However, in all seriousness, the clinical genomics community is waking up every day to solve very important challenges for the benefit of humankind and there is no substitute to having the best tools to tackle these challenges effectively and efficiently.

Whenever I think about how our company helps in this endeavor, I keep coming back to the term “operational innovation.” By that, I mean complementing the amazing technological advancement we are seeing in clinical genomics with new products, technologies, and processes that enable labs to be more successful operationally. One of the components of operational innovation is how SeraCare helps labs stay up and running longer – and a key area where we can help is to make troubleshooting be less troublesome.

poster inset video QC management part2QC Solutions for Clinical NGS, Part 2: Easy Troubleshooting

Welcome to our video series on how to use SeraCare's QC Solutions. This video focuses on how to more quickly troubleshoot your NGS assays so you can get back up and running faster and proactively monitor assay performance to stay up and running longer.


Topics: QC Challenges, QC Management Software, iQ NGS