Chapter 2: Consistency – Reducing Inter- and Intra- Reader Variability

By sdavis | October 1, 2018 | Blog

The Long and Short of It Series

Accurate and repeatable tumor size measurement is vital in assessing the efficacy of oncology treatments. Objective measurements of tumor response to therapy benefit individual patient care decisions, such as whether to continue administering the current medication. For patients enrolled in clinical trials, measurement-based categorization of therapeutic response contributes to pools of data used to judge the effectiveness of a certain treatment.

Clinical trial response criteria such as Response Evaluation Criteria in Solid Tumors version 1.1 (RECIST 1.1) include strict definitions of disease progression and response based on change in tumor diameter. Unfortunately, radiologists, oncologists and other downstream clinicians face significant challenges controlling variability and maintaining data integrity within the parameters of size-based methodologies. This problem exists whether the goal is consistent, quantitative, decision making for routine patients or whether the goal is adherence to rigid clinical trial response criteria. Variability at the point-of-read is the most problematic. 1

Notes Jonathan D. Clemente, M.D., Chief of Radiology, Charlotte Radiology, “A measurement that is off by 1-2 mm in any dimension could potentially change the measured volume of an index lesion, the radiologist’s interpretation of past response to treatment, and the future course of treatment for that patient.”

Research has shown that reader variability, whether caused by differences in measurement from the same (intra-) or multiple (inter-) readers or interpretation bias or fatigue, is a by-product of the calipers currently available in Picture Archiving and Communication Systems (PACS). More specifically, it is a by-product of the need for humans to objectively and exhaustively search for the true longest diameter of a tumor before those calipers can be utilized—a task for which the human brain is not well-suited.

Download the
entire e-book!

If you enjoyed reading this chapter of the Long and Short of It series, you can download the entire e-book by filling in the information below.

Clinical studies have found that inter-reader variability can be as high as 25 percent when using calipers, and intra-reader variance and errors can be as high as 27 percent when they are used by technicians to perform repeated measurements of small tumor masses.3,4

This variability has historically been tolerated due to lack of viable alternatives on the market. However, the resulting inconsistencies in care and implications for clinical trials and treatment effectiveness demand implementation of strategies to significantly reduce or ideally eliminate its occurrence.

3D delineation and diameter measures by four expert radiologists - which is the true diameter?

Far-Reaching Implications

The prevalence of intra- and inter-reader variability has repercussions that are far-reaching and can potentially impact the safety and efficacy of care while driving up costs and slowing clinician productivity. Many radiologists are discouraged by their inability to be precise in their measurements, not to mention the time required to do them (including reviewing prior studies to ensure the same lesion is being re-measured) and skip measuring lesions altogether in favor of providing only qualitative information in reports—much to the chagrin of oncologists and other downstream clinicians. One study found that 94 percent of oncologists want measurements at baseline on all lesions.5 Yet, 93 percent of radiologists admit they omit this step.6

Because all oncology protocols rely on percentage differences to assess treatment response, a lack of quantitative measurements makes it exceedingly difficult, if not impossible, to make accurate and informed treatment decisions. In the case of clinical trials, no measurements mean no trial data and often means that oncologists or other clinicians have to do the quantification, leading to inefficiency and even greater variability.

Further, even when quantitative measures are provided, the high probability of variability causes many downstream physicians, including subsequent radiologists, to re-measure because they do not trust the data that was provided by the initial reader. Complicating matters, some PACS do not have the capability to annotate location information, forcing subsequent readers to become investigators who must track down the original study and identify the appropriate lesions. These duplicated efforts not only add to the variability challenge, but also drive up the cost of care.

It’s no wonder that clinicians blame reader variability and the distrust it sows for productivity losses, increased costs, reader fatigue and burnout. Notably, it is also a frequent source of animosity between clinicians.


Technology-Aided Standardization

There are numerous opportunities to address reader variability regardless of the source. One of the most effective ways is through standardization, such as clearly defined protocols, mandated radiologist and technician training, and creation of standard tumor measurement models that comply with protocol tumor assessment criteria. For example, the well-defined handling of target, non-target and new lesions within RECIST 1.1 provides a basis for standardized, objective assessment of solid tumors.

However, because protocol-oriented standards are rooted in clinical trials, they only benefit the approximately 5 percent of cancer patients participating in those trials7 or receive treatment at clinics participating in trials. That means 95 percent of cancer patients may not benefit from quantitative analysis of tumor metrics when determining treatment response or disease progression—a less-than-ideal approach.

There are clear potential benefits to implementation of standardized, quantitative, decision making for routine patients. However, the dramatic increase in number of patients requiring rigorous measurements would inevitably impact patient throughput. Tools that allow radiologists to perform measurements quickly and correctly are therefore needed.

Which leads us to the application of technology.

Numerous studies have shown that, when technology is leveraged, standardization is optimized. In fact, one study found that computer-assisted measurements reduced inter-reader variability by half compared to manual measurements.8 This is particularly true when technology-aided standardization involves the deployment of semi-automated tools, which eliminate much of the variation at the outset, before it can negatively impact downstream physicians and care/clinical trial outcomes.

Computer-assisted measurements reduced inter-reader variability by half compared to manual measurements

These tools are not designed to replace the radiologists or supplant their expertise and training. Rather, they empower radiologists to identify the lesion and then “take over” the measuring to ensure true and consistent long-short values. A further step is to provide automation in locating the lesions at each follow-up time point that the radiologist has identified as needing to be tracked. They also automate Therapy Response Assessment to provide decision support for interpretation of the protocol rules, which leads to significant improvements in the efficiency, consistency, and quality of the imaging read process.

Extending this outward, measuring lesions in a semi-automated manner that does not slow radiologist productivity reduces variability and accelerates the overall imaging process. This, in turn, means more patients can benefit from the quantitative analysis that may be limited to only those participating in clinical trials.

The most effective technology-aided standardization tools are those that track a lesion over time. This ensures that tumor size changes between reads can be easily produced and included in reports, while minimizing reader variability and eliminating incomplete follow-up studies. They also do away with the need to re-locate the lesion for every read, reducing the time required for re-measurement.


Reader Variability Resolved

Radiologists and multidisciplinary clinicians are starting to look to semi-automated tools to solve the well-known issues associated with reader variability. They create consistency in the way lesions are measured among and between radiologists and between time points. They also enable the use of evidence-based metrics, which empower radiologists to deliver more consistent and standardized reports to support clinical decision-making by downstream physicians.

“For radiologists, it’s not about being able to work faster or harder. It’s about being able to work smarter. It’s about having the valuable tools radiologists need to improve the consistency and quality of their reporting by minimizing inter-reader variability on complex serial oncologic imaging studies, and to practice the best medicine possible,” said Dr. Clemente. “…Consistent and accurate measurement of lesions over time can help drive positive patient outcomes.
That’s pretty clear.”

Did you enjoy this chapter of our The Long and Short of It Series?
Click below, in case you missed a chapter.

Chapter 1


  1. A. McErlean, D. Panicek, E. Zabor, C. Moskowitz, R. Bitar, R. Motzer, H. Hricak, and M. Ginsberg. Intra- and Interobserver Variability in CT Measurements in Oncology. Radiology. 2013 269:2, 451-459.
  2. C. Klifa. Minimizing Variability in Imaging Reads for Clinical trials. Median Technologies blog. May 16. 2017.
  3. J. Delgado San Martin, P. Worthington, J Yates. Non-invasive 3D time-of-flight imaging technique for tumor volume assessment in subcutaneous models. Lab Animals. 2015. 49(2):168-71.
  4. R. Klette, K. Schluns, A. Koschan. 1998. Computer vision, three-dimensional data from images. New York: Springer
  5. T. Jaffe, N. Wickersham, D. Sullivan. Quantitative Imaging in Oncology Patients: Part 2, Oncologists’ Opinions and Expectations at Major U.S. Cancer Centers. American Journal of Roentgenology. 2010 195:1, W19-W30.
  6. Jaffe, TA, Wickersham, NW., Sullivan, DC. Quantitative Imaging in Oncology Patients: Part 1, Radiology Practice Patterns at Major U.S. Cancer Centers. American Journal of Roentgenology. 2010. 195:1, 101-106.
  7. B. Zimmerman. Just 5% of cancer patients participate in clinical trials: 5 things to know. Becker’s Hospital Review. December 11, 2017
  8. C. Kilfa. May 16, 2017.