Should reform be based on statistical significance?

Having spent roughly half my career teaching research methods and statistical analysis to medical professionals, I believe the difference between statistical significance and real-world importance must be understood by anyone who makes decisions in health care.  Statistical significance is a measure of the probability (p) that random chance explains the outcome of a test to see if an experimental effect—such as a new drug, a change in care delivery, or an alternative mechanism for reimbursement—makes a difference.  Statistical significance increases as the p value declines.

However, statistical significance often has no practical importance for our daily lives or long-range plans.  I’ve taught hundreds of students to resist the temptation to overreact to studies based on statistical significance of the findings.  Presentations at last week’s annual meeting of the American Society of Clinical Oncology illustrated this very important point.  Several researchers suggested that cancer patients who took new drugs lived significantly longer than comparable patients who took a placebo or older medication under controlled conditions.  

The statistically significant difference sounds impressive, all other things being equal, but should we immediately start paying for a new drug if it extends life only three months and costs $50,000?  Of course not!  Today’s push for health reform is based on widespread agreement that our country cannot afford to spend more on medical care, and we could surely find a more productive way to spend an extra $50,000 if we had it.  

Today’s economic realities and political circumstances are forcing us to learn to live within our means.  We cannot adopt some new approach to medical care just because it is supported by statistically significant research.  (For the record, I am a very strong supporter of medical research.  This blog post questions the use of research reports, not the research itself.)

To complicate matters, a recent article in the Journal of the American Medical Association (JAMA; 26 May 2010, p. 2058-64) http://jama.ama-assn.org/cgi/content/abstract/303/20/2058 suggests that reports and interpretations of studies with statistically non-significant findings are frequently inconsistent with the actual results.  In other words, more than a few of today’s “scientific” publications convey impressions not supported by the data. 

We now run not only the risk of overreacting to good research, but to bad research as well.  I hope that you will join me in pressuring our policy-makers to put statistical significance into proper perspective.  Or am I the only one who fears that policy analysts are putting too much faith in data and too little in strategic vision of an efficient and effective health system with limited resources?  Please share your thoughts.

What can countries learn from each other?

Having spent the past week at conferences in Amsterdam and Grenoble with several of my London-based colleagues from Xerox/ACS, I had an excellent opportunity to gauge the current status of three top-rated health systems that are commonly cited as models for the U.S. to emulate.  Ironically, I discovered that policy-makers in the Netherlands, France, and Great Britain are looking to us for solutions just as much as we are looking to them.  Our respective systems for financing and delivering care are facing the same problems, in spite of significantly different organizational characteristics.

The costs of health care have exceeded available economic resources in these three top-rated European countries.  American health policy analysts might look to them as models for universal coverage, but their economic challenges are no different than ours.  These countries are slashing providers’ budgets on a scale comparable to the cuts that Democrats in Congress recently imposed to pass insurance overhaul “reforms.”  So much for political promises that the United States could maintain accustomed levels of service and move toward European-style universal coverage!

The extent of covered benefits is also declining in these health systems that presumably provide more health care for their citizens.  Residents of all three countries told me that many services traditionally included in the universal plans are now only covered under supplemental plans that require the individual to pay an additional premium.  Further, the role of private providers is growing perceptibly in state-directed systems that many American liberals would have us emulate.


Those of you who follow my commentaries on health care will not be surprised that the principle themes of my speeches at last week’s European conferences were efficiency and effectiveness—that is, eliminating waste in health care delivery.  Perhaps I should not be surprised by the strong, positive audience responses to my message.  Indeed, a French study just concluded that more than 30% of all hospital stays there are inutile (useless).  The world’s top-rated health systems are suddenly realizing that they must harness medical expenditures that do not contribute to the health of individual citizens or national populations.    

The United States does not have a top-rated national health system, but I proudly believe our best providers are global leaders in applying performance improvement to health care.  Americans can still learn from European countries that produce healthier populations with lower shares of GDP, but we can teach them a thing or two about lowering costs (efficiency) and improving quality (effectiveness).  Some of our providers and payers have developed world-class solutions that will solve problems in Europe and elsewhere.  I expect more Americans will be asked to speak at international conferences in the future.      

What’s an ACO?

The new reform laws raise more questions than they answer.  One particular question was asked a lot at two national conferences I attended last week: what’s an Accountable Care Organization?  Policy-makers seem to be placing high hopes on ACOs—integrated and aligned systems where all who care for an individual patient are at risk for delivering care of defined quality as inexpensively as possible [my definition]. 

Countless health care leaders will attend conferences over the next year to see if becoming an ACO is the key to their organizations’ survival under “ObamaCare.”  I’ll step out on a limb and argue that becoming an ACO will be critical to the future success of most providers.  I really like the general concept.  However, I fear for three reasons that it will do more harm than good if the bureaucrats define ACO too narrowly when they translate Congressional intent into regulations.

First, the rule-writing process will take years.  Very few health care organizations can afford to wait for the “feds” to tell them how to qualify for financial incentives that may ultimately be directed to ACOs.   Integration is imperative now.  The American economy cannot continue to support the waste that fragmentation generates in our health care delivery system.

Second, an ACO must integrate more than hospitals and physicians.  To deliver the full benefits of alignment, an ACO must encompass key economic units on the supply and demand sides of a medical marketplace—accountably and transparently.  In my view, a successful ACO will not only put doctors and physicians on the same balance sheet, but also bring payers, purchasers, and patients into the mix. 

Third, competition between ACOs is needed to produce the full benefits of multi-stakeholder integration.  I am concerned that regulations will turn the solution into a problem.  A “one size fits all” definition of ACO would stifle the competitive innovation and diversity that caused Congress to favor ACOs in the first place.  For example, Kaiser-Permanente and the Geisinger Clinic are two of the highly integrated delivery systems examined by Congress, but they achieve their impressive successes in different ways. 

I fear bureaucratic rule-making will produce a narrow definition that excludes different paths to acceptably accountable care.  (It’s happened before.  Kaiser-Permanente ultimately chose not to seek designation as a federally qualified HMO under the 1973 Health Maintenance Organization Act that was based on the Kaiser model.)  So what’s an ACO?  It is a new and improved delivery system engineered to produce efficient and effective health care through partnerships of all key players in local markets.  Ideally, ACOs will be as different as the marketplaces they serve.  The less the concept is standardized beyond its generic attributes, the better. 

How would you define an ACO?  Is your health care organization ready to rise to the challenge now, or are you waiting for the Secretary of HHS to define it for you later?  

What are the real “costs” of reform?

Critics of health reform found some fuel for their fire in reports just issued by the Congressional Budget Office and the CMS Actuary.  Both official government agencies increased their previous estimates of federal costs of insurance overhaul—thus renewing questions about reform’s long-run economic impact and political tactics used to get the bills through the House and Senate.  Both reports put the new cost projections well above an economic threshold that swing-vote legislators were unwilling cross only two months ago.

As an experienced forecaster and economist, I doubt both the original and the revised estimates for three reasons. 

• The cost estimates are based on highly speculative assumptions.  The future values of variables in the estimating equations are SWAGs at best.  In particular, I believe they grossly overestimate consumers’ economic capacity to hold up their end of the “bargain.”
• The federal government has a very poor record of estimating the future costs of previous health care legislation.  Actual government expenditures have been well above earlier estimates much more often than they have been below, but they’ve never been right. 
• In their defense, the federal employees responsible for estimating reform’s costs were not given the time or resources to do a good job over the past year.  Indeed, the CBO Director recently issued a very reasonable statement about the unreasonable pressures put on his staff, but it did not get the attention it deserved.

Nevertheless, I believe that the estimated federal cost of the latest reforms is a red herring.  To me, the critical question is whether the reforms do anything to improve the quality of care and to stop the persistent increase in expenditures.  The latest laws fail on both these counts.  Insurance overhaul simply shifts a substantial portion of the rising costs of a dysfunctional delivery system from the federal government to payers and purchasers (including state governments).  A lot of excellent ideas for improving the system were lost last fall when Democrats in Congress began to focus on passing a law rather than improving the delivery system. 

I think we’ve got to get refocused on efficiency and effectiveness in the medical marketplace.  We do not need estimates of reform’s costs for the federal government to know that the status quo is unaffordable for everyone.  We will be even deeper in a financial hole if we spend the coming years fighting over the minutia of regulations when we ought to be re-engineering the entire health care delivery system.  What do you think?  Alternative points of view are welcome and will be published.  That’s what a healthy debate is all about.   

What’s an “affordable” health plan?

As a health futurist, I feel comfortable predicting that one of the hottest political issues for the next few years will be designing the “affordable” health plan that uninsured Americans must purchase in 2014.  The insurance overhaul laws delegate the general task to the Secretary of HHS, but they also dictate specific services that must be included in the mandated packages.  The process of combining regulatory authority with statutory mandates will be one big can of worms.  (The equivalent French expression, translated as a nest of poisonous snakes, provides an even better metaphor.) 

The reform laws themselves set the stage for a bitter battle by decreeing that plans must include many “basic” services to comply with the law—but defining them is one of more than a thousand “the Secretary shall…” delegations of authority in the legislation.  Having observed the promulgation of federal regulations for nearly 40 years, I do not foresee this process going smoothly.  It sets up intense battles between some of the most powerful constituencies in Washington.  Providers, payers, purchasers, pharma, and other suppliers and will fight to protect their interests under the negative-sum budget constraints built into the laws (if Congress continues to honor its current intent, another area of uncertainty). 

The new laws also identify preventive services that must be covered.  Many of these tests are politically popular, but they are also expensive and imprecise.  For example, the “affordable” health plans must pay all costs of breast and prostate cancer tests that are not clearly supported by scientific evidence or economic analysis.  Ironically, the debatable prevention mandates were legislated by the same Congress that made a major commitment to comparative-effectiveness research in the economic recovery law of 2009.  This policy conflict will further complicate the Secretary’s efforts to translate Congressional intent into viable, deficit-reducing regulations between now and 2014.     

In spite of these problems, a basic and affordable health plan must be developed.  It is an essential step for creating an efficient and effective health system.  However, I fear the politics of implementing insurance overhaul will get in the way of doing what needs to be done.  I am optimistic, on the other hand, that partnerships of dedicated stakeholders will develop good, progressive health plans in state and local markets.  I expect that I will be deeply involved in this quest.  Please join me by sharing your concepts of health plan design that reflect the medical science, technologies, and economic realities of the next decade.   In your view, what services should and should not be covered in a basic, affordable health plan?