Thursday, January 30, 2020

Medicaid Block Grants - Another Example that CMS Can Do "Anything" Under SSA 115A (CMMI)

Back when the ACA was relatively new, I pointed out that the creation of Center for Medicare and Medicaid Innovation, CCMI, sometimes called "Innovation Center" by CMS, had large ramifications.

CMMI is created by Section 1115A of the Social Security Act.  CMMI can run demonstration programs, for the purpose of which it may waive any section of the Social Security Act for CMS.

Think about that - why not a demonstration program that (A) waives some fundamental legal principal, B) applies nationwide, and (C) for 100 years?

CMS announced Medicaid block grants for "healthy adults" on January 30, 2020.  Back in her initial hearings in 2017, the Administrator, Seema Verma, highlighted "healthy adults in Medicaid" as an area of major interest for her.  (As opposed to disabled adults on Medicaid or elderly people on Medicare, or children on Medicaid.)

On block grants, NYT here.
CMS press release here.
CMS letter to Medicaid directors here.
CMS fact sheet here.

##

Prior to the Trump presidency, in 2016, Republician legislators were very much against the power of CMMI under a Democratic president, here.

See the full text of 1115A here


January 30, 2020: CMS Transmittal on Enhanced NTAP ABX Payments

https://www.cms.gov/files/document/se20004.pdf

Increasing Access to Innovative Antibiotics 
for Hospital Inpatients 
Using New Technology Add-On Payments: 
Frequently Asked Questions 


MLN Matters Number: SE20004
Article Release Date: January 21, 2020
Related CR Transmittal Number: N/A
Related Change Request (CR) Number: N/A

Effective Date: N/A        Implementation Date: N/A

This MLN Matters Special Edition Article informs providers of changes made by the Centers for Medicare & Medicaid Services (CMS) to develop an alternative New Technology Add-On Payment (NTAP) to increase access to innovative antibiotics for hospital inpatients.

SE20004 answers Frequently Asked Questions (FAQs) about NTAP. BACKGROUND Antimicrobial Resistance (AMR) represents an urgent clinical and economic crisis for the American health care system. Bacteria resistant to existing antibiotic drugs annually infect more than 2 million Americans, resulting in thousands of deaths.

Seniors are uniquely vulnerable to AMR due to age-related immunosuppression and greater exposure to infection (from catheters or chronic disease). A recent CMS internal analysis indicated that Medicare beneficiaries account for the majority of cases of both new diagnoses of drug-resistant infections and resulting deaths in United States hospitals.

Drug resistance causes Medicare beneficiaries to spend hundreds of thousands of additional days in hospitals each year, costing taxpayers billions in additional health care costs annually.

Note: For more information, see the CMS publication entitled, Securing Access to LifeSaving Antimicrobial Drugs for American Seniors, available at

https://www.cms.gov/blog/securing-access-life-saving-antimicrobial-drugs-american-seniors  .

Thursday, January 23, 2020

Jurisdiction E MAC Under Bid: Due November 4, 2019

Jurisdiction E, one of the largest MACs, especially from the perspective of molecular testing, is up for re-bid.

The federal Request for Proposals (RFP) for Jurisdiction E posted on August 20, 2019, with responses due November 4, 2019.   See website here, solicitation number  75FCMC19R0023.  The anticipated award date for the new contract was posted as August 4, 2020.  (Click through the documents at the bottom, 5 at a time, to see the most interesting document, the 150 page statement of work.  On my PC, I had to rename as a PDF-type file to open it.)  See my cloud copy of the statement of work here.  Pp. 47-50 describe medical director duties; there should be 3 FTE for this MAC.

In 2017/2018, NGS MAC protested the limit on total share of MAC workload that can be held by any one MAC.   It lost.  Here.  CMS caps workload of any one corporate entity to 26% of national MAC workload, and a group of companies shall not hold more than 40% of MAC workload.








Wednesday, January 22, 2020

FDA approves larotrectinib for any NTRK tumor, but NHS says "no"

NTRK Fusion Drug Not Cost Effective for Tumor-Agnostic Use, UK's NICE Suggest

NEW YORK – The UK's National Institute for Health and Care Excellence (NICE) has issued a new draft report recommending against the histology-independent use of larotrectinib (Bayer's Vitrakvi) for advanced adult and pediatric cancer patients treated through the National Health Service (NHS), even for rare cancer types or when no other satisfactory treatment options exist.
At the request of the UK's Department of Health and Social Care, NICE undertook an appraisal consultation on the use of larotrectinib — a drug that targets tumor specific fusions involving the tyrosine receptor kinase enzyme-coding gene NTRK. Under current conditional marketing authorization from the European Medicines Agency, larotrectinib can be used to treat NTRK fusion-positive solid tumors in adults or children with locally advanced, metastatic, or non-surgically resectable cancer cases without appropriate treatment alternatives.....


###


FDA approves larotrectinib for solid tumors with NTRK gene fusions


On November 26, 2018, the Food and Drug Administration granted accelerated approval to larotrectinib (VITRAKVI, Loxo Oncology Inc. and Bayer) for adult and pediatric patients with solid tumors that have a neurotrophic receptor tyrosine kinase (NTRK) gene fusion without a known acquired resistance mutation, that are either metastatic or where surgical resection is likely to result in severe morbidity, and who have no satisfactory alternative treatments or whose cancer has progressed following treatment.
This is the second tissue-agnostic FDA approval for the treatment of cancer.




This indication is approved under accelerated approval and continued approval for this indication may be contingent upon verification and description of clinical benefit in confirmatory trials.FDA granted this application priority review, breakthrough therapy designation and orphan product designation. A description of FDA expedited programs is in the Guidance for Industry: Expedited Programs for Serious Conditions-Drugs and Biologics.

Friday, January 10, 2020

PL 116-94: DELAY OF PAMA (S.105)

Further Consolidated Appropriations Act of 2020 (FCAA).  See PL 116-94 (HR 1865) online herehere.  See: Division N, Title 1, Subtitle A, Section 105.


 SEC. 105. LABORATORY ACCESS FOR BENEFICIARIES.
    (a) Amendments Relating to Reporting Requirements With Respect to 
Clinical Diagnostic Laboratory Tests.--
        (1) Revised reporting period for reporting of private sector 
    payment rates for establishment of medicare payment rates.--Section 
    1834A(a) of the Social Security Act (42 U.S.C. 1395m-1(a)) is 
    amended--
            (A) in paragraph (1)--
                (i) by striking ``Beginning January 1, 2016'' and 
            inserting the following:
            ``(A) General reporting requirements.--Subject to 
        subparagraph (B), beginning January 1, 2016'';
                (ii) in subparagraph (A), as added by subparagraph (A) 
            of this paragraph, by inserting ``(referred to in this 
            subsection as the `reporting period')'' after ``at a time 
            specified by the Secretary''; and
                (iii) by adding at the end the following:
            ``(B) Revised reporting period.--In the case of reporting 
        with respect to clinical diagnostic laboratory tests that are 
        not advanced diagnostic laboratory tests, the Secretary shall 
        revise the reporting period under subparagraph (A) such that--
                ``(i) no reporting is required during the period 
            beginning January 1, 2020, and ending December 31, 2020;
                ``(ii) reporting is required during the period 
            beginning January 1, 2021, and ending March 31, 2021; and
                ``(iii) reporting is required every three years after 
            the period described in clause (ii).''; and
            (B) in paragraph (4)--
                (i) by striking ``In this section'' and inserting the 
            following:
            ``(A) In general.--Subject to subparagraph (B), in this 
        section''; and
                (ii) by adding at the end the following:
            ``(B) Exception.--In the case of the reporting period 
        described in paragraph (1)(B)(ii) with respect to clinical 
        diagnostic laboratory tests that are not advanced diagnostic 
        laboratory tests, the term `data collection period' means the 
        period beginning January 1, 2019, and ending June 30, 2019.''.
        (2) Corrections relating to phase-in of reductions from private 
    payor rate implementation.--Section 1834A(b)(3) of the Social 
    Security Act (42 U.S.C. 1395m-1(b)(3)) is amended--
            (A) in subparagraph (A), by striking ``through 2022'' and 
        inserting ``through 2023''; and
            (B) in subparagraph (B)--
                (i) in clause (i), by striking ``through 2019'' and 
            inserting ``through 2020''; and
                (ii) in clause (ii), by striking ``2020 through 2022'' 
            and inserting ``2021 through 2023''.
    (b) Study and Report by MedPAC.--
        (1) In general.--The Medicare Payment Advisory Commission (in 
    this subsection referred to as the ``Commission'') shall conduct a 
    study to review the methodology the Administrator of the Centers 
    for Medicare & Medicaid Services has implemented for the private 
    payor rate-based clinical laboratory fee schedule under the 
    Medicare program under title XVIII of the Social Security Act (42 
    U.S.C. 1395 et seq.).
        (2) Scope of study.--In carrying out the study described in 
    paragraph (1), the Commission shall consider the following:
            (A) How best to implement the least burdensome data 
        collection process required under section 1834A(a)(1) of such 
        Act (42 U.S.C. 1395m-1(a)(1)) that would--
                (i) result in a representative and statistically valid 
            data sample of private market rates from all laboratory 
            market segments, including hospital outreach laboratories, 
            physician office laboratories, and independent 
            laboratories; and
                (ii) consider the variability of private payor payment 
            rates across market segments.
            (B) Appropriate statistical methods for estimating rates 
        that are representative of the market.
        (3) Report to congress.--Not later than 18 months after the 
    date of the enactment of this Act, the Commission shall submit to 
    the Administrator, the Committee on Finance of the Senate, and the 
    Committees on Ways and Means and Energy and Commerce of the House 
    of Representatives a report that includes--
            (A) conclusions about the methodology described in 
        paragraph (1); and
            (B) any recommendations the Commission deems appropriate.

PL 116-94: Extension of PCORI


https://www.congress.gov/bill/116th-congress/house-bill/1865/text

Division N, Title 1, Subtitle A, Section 104



SEC. 104. EXTENSION OF APPROPRIATIONS TO THE PATIENT-CENTERED 
      OUTCOMES RESEARCH TRUST FUND; EXTENSION OF CERTAIN HEALTH 
      INSURANCE FEES.
    (a) In General.--Section 9511 of the Internal Revenue Code of 1986 
is amended--
        (1) in subsection (b)--
            (A) in paragraph (1)--
                (i) by inserting after subparagraph (E) the following 
            new subparagraph:
            ``(F) For each of fiscal years 2020 through 2029--
                ``(i) an amount equivalent to the net revenues received 
            in the Treasury from the fees imposed under subchapter B of 
            chapter 34 (relating to fees on health insurance and self-
            insured plans) for such fiscal year; and
                ``(ii) the applicable amount (as defined in paragraph 
            (4)) for the fiscal year.''; and
                (ii) by striking ``and (E)(ii)'' in the last sentence 
            and inserting ``(E)(ii), and (F)(ii)''; and
            (B) by adding at the end the following new paragraph:
        ``(4) Applicable amount defined.--In paragraph (1)(F)(ii), the 
    term `applicable amount' means--
            ``(A) for fiscal year 2020, $275,500,000;
            ``(B) for fiscal year 2021, $285,000,000;
            ``(C) for fiscal year 2022, $293,500,000;
            ``(D) for fiscal year 2023, $311,500,000;
            ``(E) for fiscal year 2024, $320,000,000;
            ``(F) for fiscal year 2025, $338,000,000;
            ``(G) for fiscal year 2026, $355,500,000;
            ``(H) for fiscal year 2027, $363,500,000;
            ``(I) for fiscal year 2028, $381,000,000; and
            ``(J) for fiscal year 2029, $399,000,000.'';
        (2) in subsection (d)(2)(A), by striking ``2019'' and inserting 
    ``2029''; and
        (3) in subsection (f), by striking ``December 20, 2019'' and 
    inserting ``September 30, 2029''.
    (b) Health Insurance Policies.--Section 4375(e) of the Internal 
Revenue Code of 1986 is amended by striking ``2019'' and inserting 
``2029''.
    (c) Self-insured Health Plans.--Section 4376(e) of the Internal 
Revenue Code of 1986 is amended by striking ``2019'' and inserting 
``2029''.
    (d) Identification of Research Priorities.--Subsection (d)(1)(A) of 
section 1181 of the Social Security Act (42 U.S.C. 1320e) is amended by 
adding at the end the following: ``Such national priorities shall 
include research with respect to intellectual and developmental 
disabilities and maternal mortality. Such priorities should reflect a 
balance between long-term priorities and short-term priorities, and be 
responsive to changes in medical evidence and in health care 
treatments.''.
    (e) Consideration of Full Range of Outcomes Data.--Subsection 
(d)(2) of such section 1181 is amended by adding at the end the 
following subparagraph:
            ``(F) Consideration of full range of outcomes data.--
        Research shall be designed, as appropriate, to take into 
        account and capture the full range of clinical and patient-
        centered outcomes relevant to, and that meet the needs of, 
        patients, clinicians, purchasers, and policy-makers in making 
        informed health decisions. In addition to the relative health 
        outcomes and clinical effectiveness, clinical and patient-
        centered outcomes shall include the potential burdens and 
        economic impacts of the utilization of medical treatments, 
        items, and services on different stakeholders and decision-
        makers respectively. These potential burdens and economic 
        impacts include medical out-of-pocket costs, including health 
        plan benefit and formulary design, non-medical costs to the 
        patient and family, including caregiving, effects on future 
        costs of care, workplace productivity and absenteeism, and 
        healthcare utilization.''.
    (f) Board Composition.--Subsection (f) of such section 1181 is 
amended--
        (1) in paragraph (1)--
            (A) in subparagraph (C)--
                (i) in the matter preceding clause (i)--

                    (I) by striking ``Seventeen'' and inserting ``At 
                least nineteen, but no more than twenty-one''; and
                    (II) by striking ``, not later than 6 months after 
                the date of enactment of this section,''; and

                (ii) in clause (iii), by striking ``3'' and inserting 
            ``at least 3, but no more than 5''; and
        (2) in paragraph (3)--
            (A) in the first sentence--
                (i) by striking the ``the members'' and inserting 
            ``members''; and
                (ii) by inserting the following before the period at 
            the end: ``to the extent necessary to preserve the evenly 
            staggered terms of the Board.''; and
            (B) by inserting the following after the first sentence: 
        ``Any member appointed to fill a vacancy occurring before the 
        expiration of the term for which the member's predecessor was 
        appointed shall be appointed for the remainder of that term and 
        thereafter may be eligible for reappointment to a full term. A 
        member may serve after the expiration of that member's term 
        until a successor has been appointed.''.
    (g) Methodology Committee Appointments.--Such section 1181 is 
amended--
        (1) in subsection (d)(6)(B), by striking ``Comptroller General 
    of the United States'' and inserting ``Board''; and
        (2) in subsection (h)(4)--
            (A) in subparagraph (A)(ii), by striking ``Comptroller 
        General'' and inserting ``Board''; and
            (B) in the first sentence of subparagraph (B), by striking 
        ``and of the Government Accountability Office''.
    (h) Reports by the Comptroller General of the United States.--
Subsection (g)(2)(A) of such section 1181 is amended--
        (1) by striking clause (iv) and inserting the following:
                ``(iv) Not less frequently than every 5 years, the 
            overall effectiveness of activities conducted under this 
            section and the dissemination, training, and capacity 
            building activities conducted under section 937 of the 
            Public Health Service Act. Such review shall include the 
            following:

                    ``(I) A description of those activities and the 
                financial commitments related to research, training, 
                data capacity building, and dissemination and uptake of 
                research findings.
                    ``(II) The extent to which the Institute and the 
                Agency for Healthcare Research and Quality have 
                collaborated with stakeholders, including provider and 
                payer organizations, to facilitate the dissemination 
                and uptake of research findings.
                    ``(III) An analysis of available data and 
                performance metrics, such as the estimated public 
                availability and dissemination of research findings and 
                uptake and utilization of research findings in clinical 
                guidelines and decision support tools, on the extent to 
                which such research findings are used by health care 
                decision-makers, the effect of the dissemination of 
                such findings on changes in medical practice and 
                reducing practice variation and disparities in health 
                care, and the effect of the research conducted and 
                disseminated on innovation and the health care economy 
                of the United States.''; and

        (2) by adding at the end the following new clause:
                ``(vi) Not less frequently than every 5 years, any 
            barriers that researchers funded by the Institute have 
            encountered in conducting studies or clinical trials, 
            including challenges covering the cost of any medical 
            treatments, services, and items described in subsection 
            (a)(2)(B) for purposes of the research study.''.

Tuesday, January 7, 2020

June 2018: Not Enough People Listened to Pathologists on Theranos !

https://www.healthnewsreview.org/2018/06/pathologists-predicted-the-theranos-debacle-but-their-voices-were-missing-from-most-news-coverage/


Pathologists predicted the Theranos debacle, but their voices were missing from most news coverage

The following guest post is written by Benjamin Mazer, MD, a resident physician in the departments of pathology and laboratory medicine at Yale-New Haven Hospital. His views are his own, and don’t represent those of his employer.
_____
Theranos, the fraudulent laboratory company whose rise and fall is recounted in a recent exposé entitled “Bad Blood,” was a darling of investors and news outlets for a more than a decade. The company suffered a rapid change of fate in late 2015 after Wall Street Journal reporter John Carreyrou revealed the shaky underpinnings of its technology, which launched the company into a gauntlet of financial and legal jeopardy.
For many of us in the pathology community, the writing was on the wall long before Carreyrou’s article was published. Had journalists consulted pathologists as expert sources, the news coverage of Theranos might have been less fawning and more skeptical. Patients might have been spared erroneous tests.
Pathologists are physicians who are experts in laboratory medicine. When patients undergo laboratory testing – on bodily fluids, cells, or biopsied tissues – those tests typically are overseen by pathologists. We ensure that tests are run using accurate and reliable methods and we provide actionable interpretations for complex tests, such as cancer diagnoses or genetic testing.

Unchecked enthusiasm

Elizabeth Holmes, founder of Theranos, promised to disrupt the system, suggesting that traditional pathology methods were stale, outdated, and even inhumane. Health and technology reporters regularly quoted her making such claims. Yet as “Bad Blood” revealed, Holmes hired a dermatologist as her laboratory director after a pathologist questioned the company’s commitment to safety and accuracy, ultimately quitting as director. This makes me wonder how little she trusted pathologists – or perhaps how much she wanted to obscure her company’s deficiencies.
It was a pathologist, however, who provided a tip to Carreyrou, which in turn led to the reporting that brought down the company. Before that, journalists didn’t seek us out. Consider this glowing 2014 article in WIRED magazine. No pathologists were interviewed to provide perspective on Holmes’ astonishing claims. The same year, Fortune magazine published an equally enthusiastic article, in which a hospital CEO and an orthopedic surgeon provided admiring quotes. (I wonder if the same journalists would have called up a cardiologist for an expert opinion about an innovative new cancer treatment.) In a 2015 Inc. magazine profile, a couple of paragraphs are dedicated to questioning the validity of the tests, but they’re lost in a sea of flattery. Criticism is chalked up to unnamed “competitors and some in the medical community.”

Pathologists’ unrest

For years, I’ve discussed Theranos amongst my pathologist colleagues, most of whom were skeptical from the beginning. The promises were, quite simply, too good to be true. If the thousands of laboratory tests being done on standard venous blood samples could be so easily replicated with finger-stick blood, it surely would have been done. Both the source of the tested blood, as well as the volume, are critical elements to laboratory testing. Many tests do now have both standard and finger-stick options, such as blood sugar testing in diabetes. Finger-stick tests require careful validation – by correlating results to standard tests – and often still do not achieve perfect accuracy.
Could pathologists have warned the public sooner? No one can replay history, but consider this detailed examination of Theranos published by a pathologist before Carreyrou published his stories. Robert Boorstein, MD, deftly breaks down Theranos’ business model, which claimed their technology would allow blood collection and testing to occur in retail drugstores. While news reporters parroted such claims, Boorstein found that blood samples were not being tested in-store because the company had not hurdled necessary FDA regulations. He wrote:
As it operates today, it appears that Theranos has moved to a typical hub-and-spoke model with minimal advantages and several disadvantages compared with competing labs.
Given these observations, Theranos does have one factor that works in its favor: This is the general belief by many smart people that Theranos “can’t be making it up.” Obviously, I have no better idea about this as any other outsider. Having said that, it is always useful to remind oneself that “if it sounds too good to be true,” it probably is!
Pathologists would have provided knowledgeable perspective for some of Holmes’ more extravagant visions, such as her promise that a Theranos test that would be able to “see the onset of pancreatic cancer 17 years before a tumor forms.” Diagnosing cancer, however, doesn’t happen by algorithm – it requires a pathologist’s examination and judgment.

Room for improvement

Let me be clear – there are surely many laboratory tests that could be automated or improved. In my view, Theranos could have developed accurate fingerstick testing for some tests (as other companies have in the past). However, the company promised not only easier blood draws, but faster, more accurate, and cheaper testing – and not just for a single test, but for an entire menu of options. All this from an ambitious but inexperienced engineering student.
Notable Silicon Valley successes have perhaps primed us to be open to such an incredible possibility, but it seems to me that journalists and investors threw common sense out the window. Anyone should know that simultaneously better, faster, and cheaper technologies are hard to create. Admittedly, laboratory testing is a topic that spans biology, engineering, medicine, and the law. Too often, even diligent journalists may reach for the most accessible quote, instead of the most reliable one, when they’re faced with a deadline.
It’s easy to blame news outlets, as I have done. There will be no shortage of journalism autopsies detailing the inadequacies that led us to overly rosy takes on Theranos. Physicians also have a responsibility when they act as expert sources for journalists. A recent set of media ethics guidelines from the American Medical Association implores doctors to only provide only information that is “commensurate with their medical expertise.” Laboratory testing is a conundrum, as all physicians utilize laboratory tests – it’s easy to think that because you know how to interpret a test, you also understand how it’s performed. Viewing laboratory testing as a mysterious black box plagues other doctors as much as it does journalists.

A case for pathologists

Theranos will soon be yesterday’s news, but the promise of easier testing still holds immense appeal. Stories about liquid biopsies – blood-based tests for cancer – continue to appear in the news and continue to be misreported. If journalists writing these stories consulted pathologists, they could help readers understand why such tests are a long way off. Liquid biopsies are essentially genetic tests and most cancer diagnoses today are still made by examining cells under a microscope. Because only a minority of cancers are now treated based on genetic mutations, blood-based diagnostic tests will not replace traditional biopsies anytime soon.
Even as I argue that journalists should speak to pathologists as experts in laboratory testing, I must point out that like any physician, pathologists can have conflicts of interests, which might bias their view.
Still, I encourage journalists to reach out to pathologists for topics such as:
  • Cancer diagnosis and screening
  • Laboratory safety, quality, and accuracy
  • Laboratory regulation
  • Genetic testing and hereditary diseases
  • Precision medicine
  • Infectious diseases
How to find a pathologist? Reach out to pathology faculty at academic medical centers or call the College of American Pathologists to help coordinate an interview with an appropriate expert. (Disclosure: I volunteer for this organization, but it had no involvement in this article). We want to lend our expertise and engage in a dialogue to help journalists and consumers better understand how laboratory medicine affects patients.


Google's Blog and Two Articles on Healthcare AI & ML

https://ai.googleblog.com/2019/12/lessons-learned-from-developing-ml-for.html?m=1



##
This article summarizes two publications recently from the Google team, one in Nature Materials and one in JAMA.
##
The Nature Materials article Chen et al. can be read via an Author Share Online Link (purchase to download PDF though).
https://www.nature.com/articles/s41563-019-0345-0.epdf?author_access_token=ZGMRwJitg3pRuX_kutDYXtRgN0jAjWel9jnR3ZoTv0NiZsoPIBujbc403fHYFYLjHom3wQtBLejAM2bArfXxfG4Nv5ex9ozdDOtcUou5Ws9AIQrx_iwUOmisAGRQcQp613cGPa7yADfIhNS-Txy5vA%3D%3D

##
The JAMA article from Nov 12, 2019:  JAMA 322:1806-16, Liu et al.
https://jamanetwork.com/journals/jama/fullarticle/2754798?guestAccessKey=fd274bef-2813-446f-bb10-e5134640922f
##
##


Machine learning (ML) methods are not new in medicine -- traditional techniques, such as decision trees and logistic regression, were commonly used to derive established clinical decision rules (for example, the TIMI Risk Score for estimating patient risk after a coronary event). In recent years, however, there has been a tremendous surge in leveraging ML for a variety of medical applications, such as predicting adverse events from complex medical records, and improving the accuracy of genomic sequencing. In addition to detecting known diseases, ML models can tease out previously unknown signals, such as cardiovascular risk factors and refractive error from retinal fundus photographs.

Beyond developing these models, it’s important to understand how they can be incorporated into medical workflows. Previous research indicates that doctors assisted by ML models can be more accurate than either doctors or models alone in grading diabetic eye disease and diagnosing metastatic breast cancer. Similarly, doctors are able to leverage ML-based tools in an interactive fashion to search for similar medical images, providing further evidence that doctors can work effectively with ML-based assistive tools.

In an effort to improve guidance for research at the intersection of ML and healthcare, we have written a pair of articles, published in Nature Materials and the Journal of the American Medical Association (JAMA). The first is for ML practitioners to better understand how to develop ML solutions for healthcare, and the other is for doctors who desire a better understanding of whether ML could help improve their clinical work.

How to Develop Machine Learning Models for Healthcare

In “How to develop machine learning models for healthcare” (pdf), published in Nature Materials, we discuss the importance of ensuring that the needs specific to the healthcare environment inform the development of ML models for that setting. This should be done throughout the process of developing technologies for healthcare applications, from problem selection, data collection and ML model development to validation and assessment, deployment and monitoring.

The first consideration is how to identify a healthcare problem for which there is both an urgent clinical need and for which predictions based on ML models will provide actionable insight. For example, ML for detecting diabetic eye disease can help alleviate the screening workload in parts of the world where diabetes is prevalent and the number of medical specialists is insufficient. Once the problem has been identified, one must be careful with data curation to ensure that the ground truth labels, or “reference standard”, applied to the data are reliable and accurate. This can be accomplished by validating labels via comparison to expert interpretation of the same data, such as retinal fundus photographs, or through an orthogonal procedure, such as a biopsy to confirm radiologic findings. This is particularly important since a high-quality reference standard is essential both for training useful models and for accurately measuring model performance. Therefore, it is critical that ML practitioners work closely with clinical experts to ensure the rigor of the reference standard used for training and evaluation.

Validation of model performance is also substantially different in healthcare, because the problem of distributional shift can be pronounced. In contrast to typical ML studies where a single random test split is common, the medical field values validation using multiple independent evaluation datasets, each with different patient populations that may exhibit differences in demographics or disease subtypes. Because the specifics depend on the problem, ML practitioners should work closely with clinical experts to design the study, with particular care in ensuring that the model validation and performance metrics are appropriate for the clinical setting.

Integration of the resulting assistive tools also requires thoughtful design to ensure seamless workflow integration, with consideration for measurement of the impact of these tools on diagnostic accuracy and workflow efficiency. Importantly, there is substantial value in prospective study of these tools in real patient care to better understand their real-world impact.

Finally, even after validation and workflow integration, the journey towards deployment is just beginning: regulatory approval and continued monitoring for unexpected error modes or adverse events in real use remains ahead.
Two examples of the translational process of developing, validating, and implementing ML models for healthcare based on our work in detecting diabetic eye disease and metastatic breast cancer.
Empowering Doctors to Better Understand Machine Learning for Healthcare

In “Users’ Guide to the Medical Literature: How to Read Articles that use Machine Learning,” published in JAMA, we summarize key ML concepts to help doctors evaluate ML studies for suitability of inclusion in their workflow. The goal of this article is to demystify ML, to assist doctors who need to use ML systems to understand their basic functionality, when to trust them, and their potential limitations.

The central questions doctors ask when evaluating any study, whether ML or not, remain: Was the reference standard reliable? Was the evaluation unbiased, such as assessing for both false positives and false negatives, and performing a fair comparison with clinicians? Does the evaluation apply to the patient population that I see? How does the ML model help me in taking care of my patients?

In addition to these questions, ML models should also be scrutinized to determine whether the hyperparameters used in their development were tuned on a dataset independent of that used for final model evaluation. This is particularly important, since inappropriate tuning can lead to substantial overestimation of performance, e.g., a sufficiently sophisticated model can be trained to completely memorize the training dataset and generalize poorly to new data. Ensuring that tuning was done appropriately requires being mindful of ambiguities in dataset naming, and in particular, using the terminology with which the audience is most familiar:
The intersection of two fields: ML and healthcare creates ambiguity in the term “validation dataset”. An ML validation set is typically used to refer to the dataset used for hyperparameter tuning, whereas a “clinical” validation set is typically used for final evaluation. To reduce confusion, we have opted to refer to the (ML) validation set as the “tuning” set.
Future outlook
It is an exciting time to work on AI for healthcare. The “bench-to-bedside” path is a long one that requires researchers and experts from multiple disciplines to work together in this translational process. We hope that these two articles will promote mutual understanding of what is important for ML practitioners developing models for healthcare and what is emphasized by doctors evaluating these models, thus driving further collaborations between the fields and towards eventual positive impact on patient care.

Acknowledgements
Key contributors to these projects include Yun Liu, Po-Hsuan Cameron Chen, Jonathan Krause, and Lily Peng. The authors would like to acknowledge Greg Corrado and Avinash Varadarajan for their advice, and the Google Health team for their support.