AI Comparison of HHS ONC Proposal, Google & Amazon Comments (June 2023)

Main Blog: https://www.discoveriesinhealthpolicy.com/2023/12/hhs-scandalously-bad-x-2-costbenefit.html

AI backgrounder / deep dive

https://brucedocumentblog.blogspot.com/2023/12/chat-gpt-on-hhs-onc-ai-ehr-final-rule.html

###

Attached are comment letters from Google and Amazon on an HHS rule on EHR's and AI.   Please write a consolidated viewpoint of their comments.   Please summarize the consolidated comments in 12 major points.  Thank you!  You are writing for a policy expert.

ChatGPT

I have reviewed the comment letters from Google and Amazon on the HHS rule concerning Electronic Health Records (EHRs) and Artificial Intelligence (AI). Here's a consolidated viewpoint of their comments summarized into 12 major points:

Support for Enhanced Interoperability: Both companies emphasize the importance of improved interoperability in healthcare, facilitating better data access for patients and clinicians.

Empowering Patients and Clinicians: There is a shared belief that access to more comprehensive health data empowers patients and clinicians to make informed decisions.

Advocating for Transparency in AI/ML Tools: Both stress the need for transparency in the development and use of AI/ML tools, particularly in high-risk applications.

Risk-Based Approach to Regulation: They advocate for a risk-based approach to the regulation of AI/ML tools, focusing on high-risk use cases.

Data Privacy and Protection: The importance of data privacy and the protection of intellectual property and trade secrets in any transparency requirements is highlighted.

Quality Assurance Across Product Life Cycle: Emphasis on maintaining quality across the product life cycle, especially for high-risk AI/ML use cases.

Human-AI Collaboration: The necessity for human oversight in AI/ML systems is recognized, especially in high-risk scenarios, to ensure safety and accuracy.

Promoting Data Diversity: They call for diverse patient data in AI/ML development to address healthcare biases and ensure effective care for all patient populations.

Development of Reference Data Sets: The need for developing and using reference data sets to benchmark AI/ML technologies is emphasized.

Enabling Continuous AI/ML Learning: They highlight the importance of regulatory frameworks that allow for continuous improvement of AI/ML models.

Standardization and International Alignment: The companies advocate for the alignment with international standards for consistency and effectiveness in AI/ML use.

Incorporation of Third-Party Tools and Real-Time Data: Both emphasize the integration of third-party tools and real-time data capabilities in EHR systems to enhance healthcare delivery and decision-making.

These points reflect a consensus on the need for responsible, transparent, and innovative use of technology in healthcare, focusing on patient-centered care and data-driven decision-making.

User

Do they go so far as to suggest that HHS ONC should not finalize the proposed rule at this time?

ChatGPT

Both Amazon and Google, in their comments on the HHS ONC proposed rule regarding Electronic Health Records (EHRs) and Artificial Intelligence (AI), did not explicitly suggest that HHS ONC should not finalize the proposed rule at this time. However, they did express concerns and sought clarifications:

Amazon's Concerns: Amazon pointed out that the proposed rule covers a broad array of AI/ML-based tools, extending beyond its intended scope of predictive technologies. They encourage ONC to refine the definition to focus specifically on AI/ML tools that use data to train algorithms and predictive models for tasks associated with human intelligence, such as computer vision, natural language processing, and speech recognition​​.

Google's Perspective: Google highlighted the need for clarity regarding low-risk predictive DSI (Decision Support Intervention) use cases. They suggest that the rule should clearly state that low-risk use cases, such as non-clinical applications (e.g., EHR integrations for administrative notes and billing), are not covered by the proposed regulations. Google argues that providing such clarifications would help avoid consumer confusion and unnecessary compliance costs​​.

In summary, while neither company explicitly recommends against finalizing the rule, they both stress the need for more precise definitions and clarifications to ensure that the rule effectively targets high-risk use cases and avoids unnecessary regulatory burdens on low-risk applications.

User

What were the 2 or 3 greatest areas of dissimilarity between the independent views of Amazon and Google?

ChatGPT

The two or three greatest areas of dissimilarity between the independent views of Amazon and Google on the HHS ONC proposed rule regarding EHRs and AI are:

Focus on Data Diversity and Bias:

Amazon: Amazon places a significant emphasis on encouraging data diversity. They stress the need for sufficient data on diverse patient populations to reflect real-world use and address underrepresented attributes, thus providing safe and effective care. Amazon also highlights the importance of identifying and addressing existing biases in healthcare delivery that could influence AI/ML model performance. They support the development of reference data sets to benchmark technology performance and address healthcare biases​​.

Google: Google's focus, in contrast, is more on the regulatory scope and clarity, particularly for low-risk predictive DSI (Decision Support Intervention) use cases. They advocate for clarity in the proposed rules to ensure that low-risk use cases, such as EHR integrations for administrative notes and billing, are not covered by the proposed regulations. Google emphasizes avoiding consumer confusion and unnecessary compliance costs due to ambiguous application of the rules​​.

Approach to AI/ML Tools and Services:

Amazon: Amazon agrees with ONC on the importance of transparency in the development and use of AI/ML tools, particularly for higher-risk applications. They stress that AI/ML tools should be fair, appropriate, valid, effective, and safe. Amazon has developed Service Cards to provide customers with information on their AI/ML services, showcasing a commitment to transparency and responsible use of AI/ML in healthcare​​.

Google: Google, while also concerned with transparency and the appropriate use of AI/ML, does not specifically mention initiatives like Service Cards or similar tools for transparency. Their comments are more focused on the need for regulatory clarity and the exclusion of low-risk predictive DSI tools from certain regulatory burdens.

These dissimilarities highlight Amazon's focus on addressing bias and promoting diversity in data used for AI/ML, and their efforts towards transparency in AI/ML tools, compared to Google's emphasis on regulatory clarity and the distinction between high-risk and low-risk AI/ML applications.

Bonus:  Ten Main Points of UHG Letter (13pp)

The UnitedHealth Group (UHG) made several key points in their response to the Proposed Rule from the Office of the National Coordinator for Health Information Technology (ONC). Here's a concise summary:

Support for ONC's Engagement: UHG appreciates ONC's public engagement and transparency efforts during the comment process.

Definitions and Scope: They discuss the definition and scope of electronic health information (EHI) and the interoperability requirement.

Clarifications in Health IT Development: UHG seeks clarifications on the definitions related to self-developed health IT and the conditions under which health care providers are not considered health IT developers.

Exception for Trusted Exchange Framework: UHG supports the new exception for entities in the Trusted Exchange Framework under certain conditions.

FHIR Standards and USCDI Adoption: They express concerns about the transition complexity and timeline for adopting new FHIR standards and migrating from USCDI v1 to v3.

Predictive Decision Support Intervention (DSI): UHG suggests narrowing the definition of predictive DSI to clinical uses and advocates a risk-based approach to classifying DSIs.

Intervention Risk Management (IRM): There's a request for clarity on the proposed IRM structure and its relation to FDA regulations.

Disclosure Requirements: UHG seeks clarification on disclosure requirements for certified health IT developers, particularly concerning third-party predictive DSI developers.

Transparency in DSIs: UHG supports more transparency but has concerns about the practicality and risks associated with the requirement to disclose predictive DSI “source attributes.”

Additional Comments and Requests for Information: UHG provides additional comments on various aspects of the Proposed Rule, including the adoption of health IT standards, data segmentation, and lab data standards.

These points reflect UHG's focus on ensuring clear, practical, and risk-based approaches to health IT development and interoperability, with a keen interest in safeguarding against overly broad definitions and burdensome requirements.

User

What is UHG most upset about?  You can pick two topics and briefly explain why you chose them.

ChatGPT

UnitedHealth Group (UHG) expressed significant concerns in two main areas regarding the Proposed Rule by the Office of the National Coordinator for Health Information Technology (ONC):

Disclosure of Predictive Decision Support Intervention (DSI) Source Attributes:

UHG is particularly concerned about the requirement for health IT modules that enable or interface with predictive DSIs to allow end-users to review predictive DSI source attributes and summary information regarding intervention risk management practices. They find the list of required attributes for disclosure to be overly broad and potentially impractical to implement. UHG is worried that this requirement could lead to the sharing of sensitive or proprietary information with end users. They also express concerns that such detailed disclosures could result in user interfaces that are overly complex and designed primarily to meet regulatory requirements rather than user needs​​.

Documentation and Disclosure Requirements for Intervention Risk Management (IRM) Practices:

UHG has reservations about the nature of the information required to be disclosed concerning the IRM practices of certified health IT developers. They are concerned that depending on the level of detail expected by ONC Authorized Certification Bodies (ONC-ACBs), third-party DSI developers might be obliged to share proprietary technical and governance information. UHG urges ONC to provide clarity on the level of detail required in the documentation that demonstrates the employment of IRM practices and to consider the implications of obligating third-party developers to disclose potentially sensitive information​​.

These concerns indicate UHG's apprehension about the potential overreach of disclosure requirements, which could compromise proprietary information and lead to impractical implementation challenges.

What is Substack? An Interesting, Simple Business Model

 What is substack?   I have the impression it is a blog website that gives easy access to subscription controlled entry if desired.

ChatGPT

Substack is indeed a platform that closely aligns with your description. It's essentially an online service that allows writers and content creators to publish newsletters and host blogs. One of its key features is the ability to offer subscription-based content. This means creators can provide some or all of their content behind a paywall, allowing them to generate income directly from their subscribers.

Substack is popular for several reasons:

Ease of Use: It offers a simple, user-friendly interface for publishing written content. This makes it accessible for a wide range of users, from professional journalists to hobbyist writers.

Monetization: The subscription model allows writers to earn money directly from their audience, rather than relying solely on advertising revenue. This can create a more direct and sustainable relationship between creators and their readers.

Community Building: It provides tools for creators to engage with their subscribers, fostering a sense of community around their content.

Flexibility: Writers can choose how much of their content is free and how much is behind a paywall. This flexibility allows them to balance reaching a wide audience with earning revenue.

Creative Freedom: Substack doesn't exert editorial control, so creators have the freedom to write about what they want, how they want.

Overall, Substack represents a shift in how written content is distributed and monetized online, empowering individual creators by providing them with the tools to build and monetize their audience directly.

Chat GPT on HHS ONC AI (EHR) Final Rule

 

See original article here

https://www.discoveriesinhealthpolicy.com/2023/12/hhs-scandalously-bad-x-2-costbenefit.html

See AI review of Google (etc) letters here

https://brucedocumentblog.blogspot.com/2023/12/ai-comparison-of-hhs-onc-proposal.html

 

You

20231214 Press Etc re ONC AI Final.pdf

PDF

Attached, you have a number of new press articles about a final rule from HHS about electronic health records and artificial intelligence. From reading these articles, please provide ten important takeaways - and focus specifically on rules or implications for applications that involve or impact artificial intelligence in health care.

ChatGPT

ChatGPT

The document is a comprehensive collection of press releases and articles regarding the U.S. Department of Health and Human Services (HHS) finalizing the Health Data Technology and Interoperability: Certification Program Updates Algorithm Transparency and Information Sharing (HTI-1) rule. This rule is focused on advancing health IT interoperability and transparency, particularly in relation to artificial intelligence (AI) in healthcare. The document highlights various aspects of the rule, its implications, and reactions from different stakeholders in the healthcare sector.

I will now summarize the key takeaways related to AI in healthcare from this document.

Here are ten important takeaways from the HHS final rule on electronic health records and AI in healthcare:

1. Algorithm Transparency: The rule introduces first-of-its-kind transparency requirements for AI and predictive algorithms in certified health IT.

2. Scope of Impact: ONC-certified health IT is used by over 96% of hospitals and 78% of office-based physicians in the U.S., significantly impacting healthcare delivery.

3. Assessment and Decision Making: Clinical users will gain access to essential information about AI algorithms, assisting in assessing their fairness, validity, effectiveness, and safety.

4. USCDI Version 3 Adoption: The United States Core Data for Interoperability Version 3 will be the new standard in the Certification Program, enhancing patient data accuracy and promoting equity.

5. Information Blocking Revisions: The rule revises certain definitions and exceptions related to information blocking, supporting better information sharing.

6. Interoperability-Focused Reporting Metrics: New reporting metrics for certified health IT developers will provide insights into the usage of certified health IT in care delivery.

7. Voluntary Commitments from Healthcare Companies: Leading healthcare companies have made commitments to align with the rule, focusing on safe and responsible AI deployment in healthcare.

8. Risk Management Framework: There's an emphasis on comprehensive tracking of AI applications, evaluating potential harms, and mitigating risks.

9. Non-Regulatory AI Tools: The rule impacts certified HIT developers primarily, but healthcare providers using non-regulated AI tools are also advised to stay informed.

10. Future Regulatory Actions: This rule is part of a broader governmental effort to ensure the safe, secure, and responsible use of AI in healthcare, indicating potential future regulations.

These takeaways reflect a significant step towards ensuring transparency, safety, and efficacy in the use of AI technologies in the healthcare sector.

 

You

I use a vender's software for tumor recognition on pathology slides in our pathology department. The software is currently sold as "RUO" research use but we have validated for local clia lab use in a supportive, not diagnostic, manner. Does this rule apply to our applicaiton in our path lab in our hospital, or does it only apply to stuff integrated in an EHR. 

ChatGPT

ChatGPT

The HHS final rule primarily applies to health IT systems that are certified under the ONC Health IT Certification Program, which typically includes electronic health records (EHR) systems. In your case, since you're using a vendor's software designated for research use only (RUO) and validated locally for CLIA lab use in a non-diagnostic, supportive capacity, it may not fall directly under the scope of this rule. The rule is more focused on software integrated into EHR systems and its transparency requirements are geared towards those systems. However, it's important to stay informed about any potential broader implications or future regulations that might affect all types of health IT, including specialized applications like yours in pathology.

 

You

How does the final rule affect the use of Ciinical Decision Support software? I understand that in some circumstances CDS is also regulated by FDA as a medical device, so it is confusing if there are both FDA rules and HHS ONC rules. ?? 

ChatGPT

ChatGPT

The HHS final rule and FDA regulations intersect but have distinct focuses regarding Clinical Decision Support (CDS) software. The FDA regulates CDS as a medical device in specific circumstances, primarily when it provides specific treatment or diagnostic recommendations. The HHS final rule, under the ONC Health IT Certification Program, would apply to CDS that's integrated into certified EHR systems, emphasizing transparency, interoperability, and information blocking aspects.

Therefore, a CDS could potentially be subject to both sets of regulations, depending on its functionality and integration with EHR systems. It's important to understand the specific requirements of each regulatory framework and how they apply to your particular CDS software.

 

You

20231214 HHS ONC FINAL RULE AI EHR (INSPECTION)916pp.pdf

PDF

Can you summarize any remarks in this final rule about interaction between HHS CDS rules and FDA CDS (medical device) rules? 

ChatGPT

ChatGPT

The HHS final rule addresses the interaction between HHS CDS rules and FDA CDS (medical device) rules by focusing on transparency and risk management in Clinical Decision Support (CDS) systems. It introduces requirements for transparency about the design, development, and evaluation of CDS tools, especially those using AI and predictive models. The rule emphasizes the need for health IT developers to disclose information about these tools, including their training data and fairness assessments. This aligns with broader goals of ensuring safety, effectiveness, and non-discriminatory practices in healthcare technologies. However, the rule does not directly modify FDA's regulatory approach but complements it by enhancing transparency and user awareness in the use of CDS systems.

3 / 3

 

You

You have (prior prompt) the final rule circa 900 pdf pages, from HHS about electronic health records and artificial intelligence. From reading this, please provide ten important takeaways - and focus specifically on rules or implications for applications that involve or impact artificial intelligence software and services in health care.

ChatGPT

ChatGPT

I have reviewed the HHS final rule on electronic health records and AI in healthcare, which spans over 900 pages. I will summarize ten key takeaways focusing on the implications for AI software and services in healthcare. This will take a little time to analyze the document in detail, so I appreciate your patience.

After reviewing the extensive HHS final rule document, here are ten important takeaways related to AI in healthcare:

1. Algorithm Transparency: The rule mandates transparency for AI algorithms in certified health IT, requiring disclosure of design, development, and performance aspects.

2. Certification Program Updates: Revisions to certification criteria for health IT, including for decision support systems and predictive models, are introduced.

3. Interoperability Enhancement: The rule emphasizes improving interoperability, particularly in the context of AI applications in health IT.

4. Risk Management Practices: Developers must apply risk management for predictive decision support interventions, focusing on aspects like fairness and safety.

5. Information Blocking Provisions: Revised definitions and exceptions related to information blocking support better data sharing, crucial for AI integration.

6. USCDI Version 3 Adoption: Adoption of this standard aims at improved data accuracy and equity, impacting AI applications' data handling.

7. Decision Support Interventions (DSI): New certification criteria for DSIs are established, focusing on health equity and transparent AI use in decision-making.

8. Predictive DSI Definition: A new definition for Predictive Decision Support Intervention is adopted, with specific certification requirements.

9. Public Transparency: Developers are required to make summary information about their AI risk management practices publicly available.

10. Governance Requirements: Policies and controls for governance are specified, ensuring data integrity and ethical use in AI applications.

These points highlight the rule's focus on enhancing transparency, interoperability, and responsible AI use in health IT.

###

Self Improvement: Improving a Consulting Sales Call with New Client

Potential New Client "X" reached out to me whether I could provide strategic advice on a certain problem area in their business.   These are heavily sterilized notes in which I asked Chat GPT3.5 for advice on what I could have done better during the introductory call.

I am Dr Quinn.   How did I do on this call?   DId I ask good questions?  What could I have done to make this a more effective sales call with this biotech company?

ChatGPT

Dr. Quinn, based on the call transcript, you did quite well in engaging with the biotech company, XXX. Here are some observations on your performance and suggestions to make future calls more effective:

Positives:

Engagement and Active Listening: You actively engaged in the conversation by asking questions and showing genuine interest in XXX work. This helps build rapport and demonstrates your commitment to the discussion.

Expertise: Your background and knowledge in healthcare, consulting, and policy were evident, and you effectively conveyed your ability to provide valuable insights to XXX.

Relevance: You brought up relevant points about the steep rise in XXX DISEASE and the impact of XXX DISEASE INTERVENTIONIS. This demonstrates your understanding of the industry and its challenges.

Offered Assistance: You offered to provide a RESEARCH REPORT  tailored to XXX's product, showing a willingness to collaborate and support their goals.

Areas for Improvement:

More Specific Questions: While you engaged in the conversation, some of your questions could have been more specific. For example, you could have asked about XXXspecific challenges WHEN IT FACES PROBLEM XXX.   WHAT DOES IT SEE AS ITS BIG CHALLENGES TO SURMOUNT IN AREA XXX.  

Clarification: At times, it would have been beneficial to ask for more details or clarification when discussing XXX technology or plans. This would help you gain a deeper understanding of their needs.

Closing Statement: It's good practice to end the call with a clear statement about next steps. You could have confirmed the timeline for your proposal or discussed when you would follow up.  [PS - IT WAS CLEAR I WOULD OFFER A STATEMENT OF WORK AND BUDGET TOMORROW]

Aligning Services: While you offered to provide a XXX REPORT you could have discussed how your services align with XXXs specific needs and challenges, showcasing the value you can bring. [PS - WE DID THIS TO SOME EXTENT]

Summarizing Key Points: To ensure alignment and clarity, consider EXPLICITLY summarizing key takeaways at the end of the call. This can help both parties ensure they're on the same page and understand the next steps.

Overall, you had a productive call with XXX NEW COMPANY, but there's room for improvement in terms of asking more targeted questions, clarifying details, and ensuring a clear path forward. These adjustments can make your future sales calls even more effective

Lewis Black Reviews Digital Pathology Conference

 Main article:

https://www.discoveriesinhealthpolicy.com/2023/12/ai-corner-ai-reports-on-digital.html

AI in Precision Oncology: 

A Lewis Black-Inspired Rant on the Future of Cancer Care

By [Your Name], Channeling My Inner Lewis Black

Ladies and gentlemen, gather 'round as I regale you with tales from the "Artificial Intelligence in Precision Oncology" conference. Now, don't get too excited—it's not a sci-fi convention. No, it's where a bunch of brainiacs talk about AI in cancer treatment while the rest of us try to figure out how to program our DVRs.

First up, we had Douglas Flora, MD, who, I swear, must have mistaken the podium for a pulpit. He preached the gospel of AI in oncology like it's going to save us all. And, okay, maybe it will. But let's not forget, these are the same people who can't make a printer work reliably!

Then there's Ben Freeberg, discussing AI's application in oncology. He talked about reducing costs and improving outcomes. Great! But when was the last time anything in healthcare got cheaper? I'll believe it when I see it, preferably on a receipt.

Eric Stahlberg, PhD, enters the scene talking about digital twins. Digital twins! We're barely handling the human ones, and now we've got virtual doppelgangers running around in the digital world. They're supposedly helping with precision medicine, but let's be honest, I can't even get Siri to understand my coffee order.

Day two, and Stephen Wong, PhD, starts talking about integrating AI models into healthcare. He says it's "complex and nuanced." That's academic speak for "good luck figuring this out without a PhD in computer science."

Sonya Makhni, MD, talks about deploying AI solutions. She says it needs a "novel approach." I'll give you a novel approach: How about making a healthcare system that doesn't confuse both patients and AI?

Yuan Luo, PhD, comes up with deep reinforcement learning for diagnosis. Now, if only we could apply that to reinforce common sense in healthcare billing!

Day three, Jithesh Veetil, PhD, talks about public-private partnerships in digital pathology. That's right, folks. We need more meetings, more bureaucracy! Because what healthcare really lacks is more red tape.

And then, the pièce de résistance: a panel on digital pathology and AI. They're discussing hurdles in implementation. Here's a hurdle for you: try explaining this stuff to your grandma without her thinking you're writing an episode of Star Trek.

In conclusion, this conference showed me one thing: the future of cancer treatment is bright, assuming these geniuses can make AI as reliable as my old VCR. And let's be real, the biggest breakthrough in AI we need right now is one that finds the TV remote.

Alternate MOLDX picture (night)

 

AI Corner: GPT4 Looks at Carrigan's Review of FDA Economic Flaws: Really Good.

Main blog on ACLA comment here.

Here, AI discusses the Carrigan 26 page economic appendix to the 107 page ACLA comment on the FDA LDT rule (12 04 2023).   It's one of the most interesting uses of AI that I've seen.

##

CHAT GPT4  [Summary of ACLA Comments on FDA's Proposed Rule "Medical Devices; Laboratory Developed Tests"]

GPT4 Reads and Summarizes:

The American Clinical Laboratory Association (ACLA) submitted an extensive critique, conducted by Christopher Carrigan, of the FDA's Preliminary Regulatory Impact Analysis (RIA) for the proposed rule "Medical Devices; Laboratory Developed Tests". This analysis scrutinizes the FDA's methodology and conclusions, highlighting several significant shortcomings in the RIA.

1. Underestimation of Costs:

The FDA's analysis predominantly considers compliance costs while largely ignoring the ancillary and substantial health-related and mortality costs arising from the new regulatory framework on Laboratory Developed Tests (LDTs).

The report argues that a large majority of LDTs would become economically unviable due to the high compliance costs relative to their revenues. Carrigan suggests that almost 90% of current LDTs could be forced out of the market, which would result in significant health costs due to lost tests, far exceeding the proposed rule’s benefits.

2. Overestimation of Benefits:

The FDA’s benefits assessment primarily stems from the supposed reduction in misdiagnoses due to “problematic” LDTs. Carrigan criticizes this approach, noting that the FDA inappropriately generalizes from a single, limited study (Pfeifer et al., 2022) to all LDTs.

A more recent study (Zehir et al., 2023) is cited, which corrects previous errors and suggests that LDTs generally exhibit excellent analytical performance. Applying these findings would significantly reduce the estimated benefits of the proposed rule.

3. Alternatives and Distributional Effects:

The FDA's analysis of alternatives primarily focuses on variations in compliance timelines, which Carrigan finds inadequate. He suggests considering management-based regulation, similar to current CMS oversight under CLIA, and other less stringent alternatives.

The report also stresses the importance of analyzing distributional effects, particularly how the rule might disproportionately impact vulnerable groups, such as individuals with rare diseases, low-income individuals, and racial minorities.

4. Recommendations:

Carrigan recommends a reevaluation of the RIA to address these issues. Quantifying ancillary costs, applying benefit transfer appropriately, evaluating reasonable regulatory alternatives, and considering distributional effects would provide a more accurate and transparent view of the proposed rule’s impacts.

The corrected analysis, according to Carrigan, would likely reveal that the quantified costs substantially exceed the quantified benefits of the Proposed Rule.

In conclusion, Carrigan’s detailed critique suggests that the FDA’s current analysis is significantly flawed, both in its assessment of costs and benefits and in its failure to consider reasonable regulatory alternatives and distributional impacts. A more thorough and accurate analysis is needed to ensure that the proposed rule's benefits justify its costs and that it does not disproportionately affect vulnerable populations.

User

Can you explain in more detail what mistakes were made by FDA in overestimating benefits (which were primarily dollarzed by extended life years)?

GPT

Certainly. The critique of the FDA's overestimation of benefits in the Regulatory Impact Analysis (RIA) for the "Medical Devices; Laboratory Developed Tests" proposed rule, primarily centers on the methods used to estimate the reduction in misdiagnoses and the associated economic value of extended life years. 

Here's a breakdown of the key mistakes identified:

1. Reliance on a Single, Limited Study:

The FDA based its estimation of the proportion of "problematic" LDTs (Laboratory Developed Tests) on a single study by Pfeifer et al. (2022). This study focused on a narrow area (metastatic colorectal cancer testing) and a small sample size of laboratories.

The critique points out that extrapolating from this specific and limited context to the entire landscape of LDTs is inappropriate and leads to an overgeneralization. This is because the diversity and complexity of LDTs across various conditions and diseases are not adequately represented in this single study.

2. Disregarding Updated and Contradictory Evidence:

A more recent study, Zehir et al. (2023), which reanalyzed the same data used in the Pfeifer study, found that the performance of LDTs was significantly better than originally reported. The Zehir study corrected for errors in the initial study and found that LDTs demonstrated “excellent” analytical performance, rather than being predominantly “problematic.”

Ignoring this more recent and corrected analysis led to a significant overestimation of the benefits of the proposed rule, as it falsely assumed a high rate of misdiagnoses due to LDT errors.

3. Incorrect Assumptions about Test Accuracy:

The FDA's analysis assumed that every misdiagnosis due to a “problematic” LDT would be eliminated under the proposed regulatory framework. This assumption is flawed because no diagnostic test, including FDA-approved tests, is perfect.

A more nuanced and realistic approach would compare the incremental improvements in accuracy between LDTs and FDA-approved tests. However, the FDA did not adequately address this, leading to an overestimation of benefits.

4. Inappropriate Benefit Transfer Application:

he RIA applied a benefit transfer approach, taking estimates from one context and applying them to another. However, the principles of benefit transfer require a careful selection of studies and contexts that are sufficiently similar. The FDA’s approach failed to meet these criteria, leading to an inaccurate and inflated estimation of benefits.

5. Failure to Consider Broader Research:

There's existing research indicating that the analytical accuracy of LDTs is not significantly different from that of FDA-approved tests. Studies like Kim et al. (2018) and Moncur et al. (2019) show high accuracy rates for LDTs, which are comparable to FDA-approved diagnostics.

By not integrating these broader findings, the FDA’s RIA skewed the perceived benefit of replacing LDTs with FDA-approved tests.

In summary, the FDA's overestimation of benefits stemmed from a narrow and potentially biased selection of evidence, incorrect assumptions about test accuracy, and an inappropriate application of benefit transfer methods. This led to an inflated view of the potential benefits of the proposed rule in terms of reducing misdiagnoses and extending life years.

User

But Pfeifer wasn't a study of life years or dollar benefits at all.  It was a limited study of test accuracy (and in that, perhaps superceded by Zehir 2023).  How did FDA get from percent errors in lab tests to 33 billion a year (roughly) in lost life benefits?  

GPT

You are correct in noting that the Pfeifer study was focused on test accuracy and not directly on the quantification of life years or dollar benefits. 

The transition from the findings of test accuracy in the Pfeifer study to the FDA's estimation of roughly $33 billion a year in lost life benefits involves a few critical steps and assumptions, which have been scrutinized for their validity. Here's how the FDA likely made this transition:

1. Establishing a Baseline of Misdiagnoses:

The FDA began by establishing a baseline estimate of misdiagnoses attributable to LDTs. They used the Pfeifer study as a basis to determine the proportion of LDTs that might be "problematic," which in their interpretation meant likely to lead to misdiagnoses.

2. Extrapolating to a Larger Scale:

The FDA then extrapolated this proportion to the broader landscape of LDTs. This extrapolation was a critical point of contention, as it generalized findings from a narrow study to a wide range of LDT applications, many of which were not represented in the initial study.

3. Estimating the Number of Fatal Misdiagnoses:

From this generalized proportion of "problematic" LDTs, the FDA estimated the number of misdiagnoses that could potentially result in fatal outcomes. This estimation involved assumptions about the rate of misdiagnoses leading to mortality, which again expanded the scope of the original study.

4. Valuing Life Years Saved:

The FDA then used standard methods to value the economic benefit of preventing these misdiagnoses. This typically involves estimating the value of a statistical life year (VSLY). By assuming that correcting these misdiagnoses would save lives, they assigned a dollar value to the life years saved.

The VSLY is a commonly used metric in health economics to quantify the benefit of life-extending interventions. It reflects the public’s willingness to pay for marginal increases in lifespan.

5. Calculating Total Benefits:

Finally, by multiplying the estimated number of lives saved (or life years extended) by the value of a statistical life year, the FDA arrived at an annual dollar benefit of the proposed rule.

Critique of the Methodology:

The critique of the FDA's approach primarily focuses on the initial steps of this process: the extrapolation of the Pfeifer study's findings to a broad range of LDTs and the subsequent estimations of the rate of fatal misdiagnoses.

The critics argue that both the extrapolation and the assumptions about the rate of fatal outcomes were overly broad and not sufficiently supported by the data from the Pfeifer study. Moreover, more recent studies like Zehir et al. (2023) suggest that the rate of misdiagnoses due to LDT inaccuracies might be significantly lower than what the FDA assumed.

In summary, while the Pfeifer study did not directly address life years or dollar benefits, the FDA used its findings as a starting point to estimate the potential benefits of reducing LDT errors, leading to their substantial annual benefit figure. The validity of this extrapolation and the assumptions used in the process have been key points of debate.

###

###

This has been excellent.  You have been very helpful.  In light of our discussion can you go back and summarize the 26 page economic critique again, for experts, but in a user-friendly Q&A format.

GPT

Absolutely, presenting the summary of the 26-page economic critique in a user-friendly Q&A format for experts is a great approach. Here we go:

Q: What is the primary focus of the ACLA's critique of the FDA's RIA?

A: The American Clinical Laboratory Association's critique, conducted by Christopher Carrigan, primarily focuses on the FDA's methodological shortcomings in estimating the costs and benefits of the proposed rule "Medical Devices; Laboratory Developed Tests." It highlights issues in underestimating costs, overestimating benefits, failing to consider reasonable regulatory alternatives, and overlooking significant distributional effects.

Q: How does the ACLA critique the FDA's estimation of the costs of the proposed rule?

A: The critique argues that the FDA significantly underestimates the costs by focusing mainly on compliance costs while ignoring ancillary and health-related costs. It suggests that the high compliance costs relative to the revenues of Laboratory Developed Tests (LDTs) could force a majority (about 90%) of LDTs out of the market, resulting in substantial health costs due to lost tests.

Q: What are the key points in the critique about the FDA's overestimation of benefits?

A: The critique identifies several errors in the FDA's benefits estimation:

Inappropriate extrapolation from a limited study (Pfeifer et al., 2022) to all LDTs, leading to an overgeneralization about the proportion of "problematic" LDTs.

Disregard for updated evidence (e.g., Zehir et al., 2023) suggesting better LDT performance.

Incorrect assumptions about the elimination of all misdiagnoses due to “problematic” LDTs under the new rule.

Failure to apply benefit transfer appropriately and to consider the broader research indicating comparable accuracy between LDTs and FDA-approved tests.

Q: How does the critique address the FDA's handling of alternatives and distributional effects?

A: The ACLA critique points out that the FDA's analysis of alternatives is insufficient, mainly focusing on variations in compliance timelines. It suggests considering management-based regulation and other less stringent alternatives. Regarding distributional effects, the critique notes that the FDA's analysis is limited and does not adequately address the impact on vulnerable groups such as individuals with rare diseases and low-income individuals.

Q: What are the ACLA's recommendations for revising the RIA?

A: The ACLA recommends:

A reevaluation of the RIA to quantify ancillary costs with available data.

A more accurate application of benefit transfer and inclusion of broader research findings.

A thorough assessment and quantification of reasonable regulatory alternatives.

An in-depth analysis of distributional effects, especially considering impacts on vulnerable populations.

Q: What are the implications of the critique for the proposed FDA rule?

A: The critique suggests that the FDA's current analysis overstates the benefits and understates the costs of the proposed rule. It implies that a more accurate and comprehensive analysis might reveal that the quantified costs substantially exceed the quantified benefits, questioning the overall efficacy and justification of the proposed regulatory changes.