AI Corner: GPT4 Looks at Carrigan's Review of FDA Economic Flaws: Really Good.

Main blog on ACLA comment here.

Here, AI discusses the Carrigan 26 page economic appendix to the 107 page ACLA comment on the FDA LDT rule (12 04 2023).   It's one of the most interesting uses of AI that I've seen.

##

CHAT GPT4  [Summary of ACLA Comments on FDA's Proposed Rule "Medical Devices; Laboratory Developed Tests"]

GPT4 Reads and Summarizes:

The American Clinical Laboratory Association (ACLA) submitted an extensive critique, conducted by Christopher Carrigan, of the FDA's Preliminary Regulatory Impact Analysis (RIA) for the proposed rule "Medical Devices; Laboratory Developed Tests". This analysis scrutinizes the FDA's methodology and conclusions, highlighting several significant shortcomings in the RIA.

1. Underestimation of Costs:

The FDA's analysis predominantly considers compliance costs while largely ignoring the ancillary and substantial health-related and mortality costs arising from the new regulatory framework on Laboratory Developed Tests (LDTs).

The report argues that a large majority of LDTs would become economically unviable due to the high compliance costs relative to their revenues. Carrigan suggests that almost 90% of current LDTs could be forced out of the market, which would result in significant health costs due to lost tests, far exceeding the proposed rule’s benefits.

2. Overestimation of Benefits:

The FDA’s benefits assessment primarily stems from the supposed reduction in misdiagnoses due to “problematic” LDTs. Carrigan criticizes this approach, noting that the FDA inappropriately generalizes from a single, limited study (Pfeifer et al., 2022) to all LDTs.

A more recent study (Zehir et al., 2023) is cited, which corrects previous errors and suggests that LDTs generally exhibit excellent analytical performance. Applying these findings would significantly reduce the estimated benefits of the proposed rule.

3. Alternatives and Distributional Effects:

The FDA's analysis of alternatives primarily focuses on variations in compliance timelines, which Carrigan finds inadequate. He suggests considering management-based regulation, similar to current CMS oversight under CLIA, and other less stringent alternatives.

The report also stresses the importance of analyzing distributional effects, particularly how the rule might disproportionately impact vulnerable groups, such as individuals with rare diseases, low-income individuals, and racial minorities.

4. Recommendations:

Carrigan recommends a reevaluation of the RIA to address these issues. Quantifying ancillary costs, applying benefit transfer appropriately, evaluating reasonable regulatory alternatives, and considering distributional effects would provide a more accurate and transparent view of the proposed rule’s impacts.

The corrected analysis, according to Carrigan, would likely reveal that the quantified costs substantially exceed the quantified benefits of the Proposed Rule.

In conclusion, Carrigan’s detailed critique suggests that the FDA’s current analysis is significantly flawed, both in its assessment of costs and benefits and in its failure to consider reasonable regulatory alternatives and distributional impacts. A more thorough and accurate analysis is needed to ensure that the proposed rule's benefits justify its costs and that it does not disproportionately affect vulnerable populations.

User

Can you explain in more detail what mistakes were made by FDA in overestimating benefits (which were primarily dollarzed by extended life years)?

GPT

Certainly. The critique of the FDA's overestimation of benefits in the Regulatory Impact Analysis (RIA) for the "Medical Devices; Laboratory Developed Tests" proposed rule, primarily centers on the methods used to estimate the reduction in misdiagnoses and the associated economic value of extended life years. 

Here's a breakdown of the key mistakes identified:

1. Reliance on a Single, Limited Study:

The FDA based its estimation of the proportion of "problematic" LDTs (Laboratory Developed Tests) on a single study by Pfeifer et al. (2022). This study focused on a narrow area (metastatic colorectal cancer testing) and a small sample size of laboratories.

The critique points out that extrapolating from this specific and limited context to the entire landscape of LDTs is inappropriate and leads to an overgeneralization. This is because the diversity and complexity of LDTs across various conditions and diseases are not adequately represented in this single study.

2. Disregarding Updated and Contradictory Evidence:

A more recent study, Zehir et al. (2023), which reanalyzed the same data used in the Pfeifer study, found that the performance of LDTs was significantly better than originally reported. The Zehir study corrected for errors in the initial study and found that LDTs demonstrated “excellent” analytical performance, rather than being predominantly “problematic.”

Ignoring this more recent and corrected analysis led to a significant overestimation of the benefits of the proposed rule, as it falsely assumed a high rate of misdiagnoses due to LDT errors.

3. Incorrect Assumptions about Test Accuracy:

The FDA's analysis assumed that every misdiagnosis due to a “problematic” LDT would be eliminated under the proposed regulatory framework. This assumption is flawed because no diagnostic test, including FDA-approved tests, is perfect.

A more nuanced and realistic approach would compare the incremental improvements in accuracy between LDTs and FDA-approved tests. However, the FDA did not adequately address this, leading to an overestimation of benefits.

4. Inappropriate Benefit Transfer Application:

he RIA applied a benefit transfer approach, taking estimates from one context and applying them to another. However, the principles of benefit transfer require a careful selection of studies and contexts that are sufficiently similar. The FDA’s approach failed to meet these criteria, leading to an inaccurate and inflated estimation of benefits.

5. Failure to Consider Broader Research:

There's existing research indicating that the analytical accuracy of LDTs is not significantly different from that of FDA-approved tests. Studies like Kim et al. (2018) and Moncur et al. (2019) show high accuracy rates for LDTs, which are comparable to FDA-approved diagnostics.

By not integrating these broader findings, the FDA’s RIA skewed the perceived benefit of replacing LDTs with FDA-approved tests.

In summary, the FDA's overestimation of benefits stemmed from a narrow and potentially biased selection of evidence, incorrect assumptions about test accuracy, and an inappropriate application of benefit transfer methods. This led to an inflated view of the potential benefits of the proposed rule in terms of reducing misdiagnoses and extending life years.

User

But Pfeifer wasn't a study of life years or dollar benefits at all.  It was a limited study of test accuracy (and in that, perhaps superceded by Zehir 2023).  How did FDA get from percent errors in lab tests to 33 billion a year (roughly) in lost life benefits?  

GPT

You are correct in noting that the Pfeifer study was focused on test accuracy and not directly on the quantification of life years or dollar benefits. 

The transition from the findings of test accuracy in the Pfeifer study to the FDA's estimation of roughly $33 billion a year in lost life benefits involves a few critical steps and assumptions, which have been scrutinized for their validity. Here's how the FDA likely made this transition:

1. Establishing a Baseline of Misdiagnoses:

The FDA began by establishing a baseline estimate of misdiagnoses attributable to LDTs. They used the Pfeifer study as a basis to determine the proportion of LDTs that might be "problematic," which in their interpretation meant likely to lead to misdiagnoses.

2. Extrapolating to a Larger Scale:

The FDA then extrapolated this proportion to the broader landscape of LDTs. This extrapolation was a critical point of contention, as it generalized findings from a narrow study to a wide range of LDT applications, many of which were not represented in the initial study.

3. Estimating the Number of Fatal Misdiagnoses:

From this generalized proportion of "problematic" LDTs, the FDA estimated the number of misdiagnoses that could potentially result in fatal outcomes. This estimation involved assumptions about the rate of misdiagnoses leading to mortality, which again expanded the scope of the original study.

4. Valuing Life Years Saved:

The FDA then used standard methods to value the economic benefit of preventing these misdiagnoses. This typically involves estimating the value of a statistical life year (VSLY). By assuming that correcting these misdiagnoses would save lives, they assigned a dollar value to the life years saved.

The VSLY is a commonly used metric in health economics to quantify the benefit of life-extending interventions. It reflects the public’s willingness to pay for marginal increases in lifespan.

5. Calculating Total Benefits:

Finally, by multiplying the estimated number of lives saved (or life years extended) by the value of a statistical life year, the FDA arrived at an annual dollar benefit of the proposed rule.

Critique of the Methodology:

The critique of the FDA's approach primarily focuses on the initial steps of this process: the extrapolation of the Pfeifer study's findings to a broad range of LDTs and the subsequent estimations of the rate of fatal misdiagnoses.

The critics argue that both the extrapolation and the assumptions about the rate of fatal outcomes were overly broad and not sufficiently supported by the data from the Pfeifer study. Moreover, more recent studies like Zehir et al. (2023) suggest that the rate of misdiagnoses due to LDT inaccuracies might be significantly lower than what the FDA assumed.

In summary, while the Pfeifer study did not directly address life years or dollar benefits, the FDA used its findings as a starting point to estimate the potential benefits of reducing LDT errors, leading to their substantial annual benefit figure. The validity of this extrapolation and the assumptions used in the process have been key points of debate.

###

###

This has been excellent.  You have been very helpful.  In light of our discussion can you go back and summarize the 26 page economic critique again, for experts, but in a user-friendly Q&A format.

GPT

Absolutely, presenting the summary of the 26-page economic critique in a user-friendly Q&A format for experts is a great approach. Here we go:

Q: What is the primary focus of the ACLA's critique of the FDA's RIA?

A: The American Clinical Laboratory Association's critique, conducted by Christopher Carrigan, primarily focuses on the FDA's methodological shortcomings in estimating the costs and benefits of the proposed rule "Medical Devices; Laboratory Developed Tests." It highlights issues in underestimating costs, overestimating benefits, failing to consider reasonable regulatory alternatives, and overlooking significant distributional effects.

Q: How does the ACLA critique the FDA's estimation of the costs of the proposed rule?

A: The critique argues that the FDA significantly underestimates the costs by focusing mainly on compliance costs while ignoring ancillary and health-related costs. It suggests that the high compliance costs relative to the revenues of Laboratory Developed Tests (LDTs) could force a majority (about 90%) of LDTs out of the market, resulting in substantial health costs due to lost tests.

Q: What are the key points in the critique about the FDA's overestimation of benefits?

A: The critique identifies several errors in the FDA's benefits estimation:

Inappropriate extrapolation from a limited study (Pfeifer et al., 2022) to all LDTs, leading to an overgeneralization about the proportion of "problematic" LDTs.

Disregard for updated evidence (e.g., Zehir et al., 2023) suggesting better LDT performance.

Incorrect assumptions about the elimination of all misdiagnoses due to “problematic” LDTs under the new rule.

Failure to apply benefit transfer appropriately and to consider the broader research indicating comparable accuracy between LDTs and FDA-approved tests.

Q: How does the critique address the FDA's handling of alternatives and distributional effects?

A: The ACLA critique points out that the FDA's analysis of alternatives is insufficient, mainly focusing on variations in compliance timelines. It suggests considering management-based regulation and other less stringent alternatives. Regarding distributional effects, the critique notes that the FDA's analysis is limited and does not adequately address the impact on vulnerable groups such as individuals with rare diseases and low-income individuals.

Q: What are the ACLA's recommendations for revising the RIA?

A: The ACLA recommends:

A reevaluation of the RIA to quantify ancillary costs with available data.

A more accurate application of benefit transfer and inclusion of broader research findings.

A thorough assessment and quantification of reasonable regulatory alternatives.

An in-depth analysis of distributional effects, especially considering impacts on vulnerable populations.

Q: What are the implications of the critique for the proposed FDA rule?

A: The critique suggests that the FDA's current analysis overstates the benefits and understates the costs of the proposed rule. It implies that a more accurate and comprehensive analysis might reveal that the quantified costs substantially exceed the quantified benefits, questioning the overall efficacy and justification of the proposed regulatory changes.