Using Figures to Make Sense of Things: The Six Question Figure versus an FDA Risk Benefit Figure

In an earlier blog, we used a "figure" to represent the logical relationships between the Six Questions for a clinical utility framework.   The framework along is reproduced below.

Surprisingly, we were able to go on for a ten-page publication discussing some implications of the "Six Questions" and the corresponding figure.  That made it fun - unwrapping or unfolding what seem to be fairly simple questions and diagrams.

Along the way, I used this figure to convey an FDA Risk-Benefit analysis.  The FDA's general question and criterion is, would an expert in the field be reasonably assured that the benefit exceeds the risk for the drug in question?

Let's use this simple figure:

This represent a drug randomized control trial.   In the treatment arm, patients succeed on a benefit measure (such as "overall survival"), and the benefit is 10 plus or minus 3.    However, we also measure adverse events, which are worse in the treatment arm than the placebo arm; the risks (adverse events) are tallied at 5 plus or minus 3.

What you see above is a bar for "the benefit" and a bar for "the risk."  How do we get to a risk benefit analysis to answer the actual question?

Let's refer to the figure below.

The risk benefit analysis is on the right side.  We tally up the benefits (10 plus minus 3).  We SUBTRACT the risks (5 plus minus 3).  We are left, at the far right, with the "net benefit" which is about 5 plus minus 3.   Since the plus-minus is a 95% confidence interval, we can be confident the benefit exceeds the risk.  QED.

Of course, the table and logic are simple, but the FDA advisory board and FDA decision is not. Actual FDA decisions get very complex very fast, even if the inner logic is about as simple as is shown above with a couple bars and a couple arrows.

The benefits are usually well-defined, they are a primary study endpoint such as "survival."  The statistics can be non-trivial, but the main ones are more or less confidence intervals.   Now it gets hard, though.   The risks and benefits are usually incommensurate, unlike the cartoon above.  The risks are gnarly, variable, range wildly in degree, and are very different from one another (vomiting versus death), and have no good statistics since they "just happen" - they aren't a single primary endpoint.   So while the concept of having a reasonable (95%) assurance that the risks are greater than the benefits is easy to show, the real life analysis and decision is far more complicated, even though the inner logical structure is the way it is shown here.

Similarly, the "Six Questions" figure and framework is pretty logically sound, and you can draw it in 60 seconds, but applying it to any real-world situations takes time and effort and content knowledge and elbow grease and discussion. Still, as I concluded in my main blog on the topic, isn't it better to hear:

• "We agree on questions 1,2,3,5, and some specific discussion points for you on questions 4 and 6" -- than to hear what we used to hear, 
• "You don't have enough clinical utility, so do more clinical utility."