Using Figures to Make Sense of Things: The Six Question Figure versus an FDA Risk Benefit Figure

In an earlier blog, we used a "figure" to represent the logical relationships between the Six Questions for a clinical utility framework.   The framework along is reproduced below.

Surprisingly, we were able to go on for a ten-page publication discussing some implications of the "Six Questions" and the corresponding figure.  That made it fun - unwrapping or unfolding what seem to be fairly simple questions and diagrams.

Along the way, I used this figure to convey an FDA Risk-Benefit analysis.  The FDA's general question and criterion is, would an expert in the field be reasonably assured that the benefit exceeds the risk for the drug in question?

Let's use this simple figure:

This represent a drug randomized control trial.   In the treatment arm, patients succeed on a benefit measure (such as "overall survival"), and the benefit is 10 plus or minus 3.    However, we also measure adverse events, which are worse in the treatment arm than the placebo arm; the risks (adverse events) are tallied at 5 plus or minus 3.

What you see above is a bar for "the benefit" and a bar for "the risk."  How do we get to a risk benefit analysis to answer the actual question?

Let's refer to the figure below.

The risk benefit analysis is on the right side.  We tally up the benefits (10 plus minus 3).  We SUBTRACT the risks (5 plus minus 3).  We are left, at the far right, with the "net benefit" which is about 5 plus minus 3.   Since the plus-minus is a 95% confidence interval, we can be confident the benefit exceeds the risk.  QED.

Of course, the table and logic are simple, but the FDA advisory board and FDA decision is not. Actual FDA decisions get very complex very fast, even if the inner logic is about as simple as is shown above with a couple bars and a couple arrows.

The benefits are usually well-defined, they are a primary study endpoint such as "survival."  The statistics can be non-trivial, but the main ones are more or less confidence intervals.   Now it gets hard, though.   The risks and benefits are usually incommensurate, unlike the cartoon above.  The risks are gnarly, variable, range wildly in degree, and are very different from one another (vomiting versus death), and have no good statistics since they "just happen" - they aren't a single primary endpoint.   So while the concept of having a reasonable (95%) assurance that the risks are greater than the benefits is easy to show, the real life analysis and decision is far more complicated, even though the inner logical structure is the way it is shown here.

Similarly, the "Six Questions" figure and framework is pretty logically sound, and you can draw it in 60 seconds, but applying it to any real-world situations takes time and effort and content knowledge and elbow grease and discussion. Still, as I concluded in my main blog on the topic, isn't it better to hear:

• "We agree on questions 1,2,3,5, and some specific discussion points for you on questions 4 and 6" -- than to hear what we used to hear, 
• "You don't have enough clinical utility, so do more clinical utility."

"Experimental" View of FDA LDT Categories and Timelines

Note: I am not certain whether all these categories and timelines are what the FDA intends in its guidance document... but here's an alternative presentation of most of the data as I understand it as of August 2014.

EVENT	TIME (FG +)	WHO IS IN	WHO IS NOT IN
1	Time Zero: Final Guidance (FG) Published		·   Forensic Tests (Never) ·   High Complexity HLA Diagnostics for Tissue Typing/Transplant (Never)
2	FG + 6 months	· “Notification” for All LDTs, but: · “Registration and Listing” if Entity has IVDs –OR- if the entity has declined to use the simpler Notification option · This date applies to: (1) LDTs previously on the market or (2) marketed before the 6 month date · LDTs entering market: Notify FDA before clinical use.
3	FG + 12 months	· PMA for High Priority Enforcement:  ü  Same intended use as an IVD companion diagnostic; ü  Same intended use as any PMA Class III IVD; ü  LDTs for safety/efficacy blood products (e.g. HIV tests)	·   No action except notification for: ü  Class I LDTs, ü  Traditional 1976 LDTs (Are you one?  If: Solely ASR components; One hospital/health system; Interpreted by laboratory professional); ü  LDTs for rare diseases, but only within health systems; ü  Unmet Need LDTs (Are you one?  No IVD, so keep monitoring that status; One Hospital/Health system; FDA waives the ASR rule) ·         Also at this date: Safe harbor for tests which are in the several high risk categories AND entered the market before the 1 year deadline AND the approval paperwork is under FDA review ·         NB:  No action but notification for those tests in the lower priority categories in later rows
4	FG + 1-5 years	· Development of priority list, likely to include: ü  Devices that act like companion diagnostics, even if no IVD, ü  Screening tests for serious diseases, ü  Tests for high risk infectious disease, ü  Other “priorities” TBD ü  Likely the fact that FDA views LDTs [with RUO or manufactured components like chips or plates or designed by commercial labs or licensed by academia to companies] as “not true LDTs” will mean they are in earlier review categories, also	·   ?? Suppose you are a “screening test for serious disease” (enforce, left column) but also “unmet need” (don’t enforce, right column)…or other mixed rule categories.
5	FY 6+ years	· Roll out clearance/approval reviews to Class II (510k) LDTs	·   Unclear if and when exemption for “ASRs tests for hospitals for their own patients” stops applying