You are here: Home » News » Health-care Policy » Hospital ratings often don’t measure up – study

Hospital ratings often don’t measure up – study

LocalHealthGuide | December 8, 2008

When choosing a hosptial you can now go online to see how they are ranked for quality and safety at a number of websites.

But how useful are these rankings to the consumer?

Not very, a new study suggests.

The study appears in the current issue of the journal Health Affairs. Michael Rothberg, an assistant professor of medicine at Tufts University School of Medicine, was the paper’s lead author.

In the study, the Rothberg and his colleagues, compared how five rating services rated hospitals in the Boston area.

They found the services frequently disagreed, with one service giving a hospital a high rating, while another giving the same hospital a low rating.

In the study, the researchers imagined a patient living in the Boston area who would be willing to travel up to one hour to get high-quality care.

They limited the choice of hospitals to institutions with at least 250 beds, reasoning a patient wouldn’t travel a hour to get care at smaller community hospital.

Using these criteria, the researchers identified nine hospitals within 30 miles of Boston.

They then looked to see how five well-known online rating services ranked these nine hospitals.

The rating services studied were:

Hospital Compare, a service of the U.S. Department of Health and Human Services.
HealthGrades
Leapfrog Group
U.S. News & World Report
Massachusetts Healthcare Quality and Cost, a state-run service known as Mass QC.

They then looked at how these five services rated the hospitals on four common conditions that do not necessarily need emergency treatment:

Total hip replacement surgery
Coronary artery bypass surgery
Percutaneous coronary intervention (such as opening a narrowed coronary artery with a balloon-tipped catheter)
Pneumonia that was acquired in the community (as opposed to a pneumonia picked up in a hospital or institution).

They also looked at death rates for myocardial infarctions (heart attacks).

“For any given diagnosis, there was little overall agreement among the rating services,” the authors report.

In the case of heart by-pass surgery, for example, two hospitals that were ranked first by at least one of the rating services were also ranked fourth and last by one of the other services.

And two hospitals that had been ranked last for by-pass surgery by at least one service were ranked first or second by one of the other services.

Even within one service’s ratings, a hospital would often rank high for the treatment of one condition but low for the treatment of another.

Even ratings based on mortality rates would not be of much help to a consumer. For example, the hospital ranked first by HealthGrades had the second highest mortality. That same hospital was ranked seventh by the state-run Mass QC rating service. And, the hospital ranked first by Mass QC was ranked seventh by HealthGrades.

“In this study of five leading health care rating services, we found that these services failed to consistently agree on either top- or bottom-performing hospitals in a single metropolitan area,” the reseachers write. “Among the nine institutions studied, hospitals ranked first or second by one system were often ranked seventh and eighth by another.”

Part of the problem, the researchers write, is that the services looked at different things to come up with their rankings. Some focused on processes, for example, whether a certain procedure was done according to guidelines, while others might have focused on outcomes, whether, for example, a procedure was successful.

The rating services also looked at data for different time periods. U.S. News & World Report, for example, based its 2007 ratings on mortality data for fiscal years 2003-2005.

“Four of the services used at least some data that were two to four years old,” the researchers write.

In addition, services were often comparing data on different kinds of patients. Three of the services, for example, limted their data to Medicaid and Medicare patients, who tend to be either older or sicker.

Finally, the researcher found that most rating services did not use statistical analyses that could help clarify their confusing results.

To improve the quality of ratings the researchers urge hospitals to actively embrace the rating process.

Currently, hospitals have been passively accepting the requirements that they report data on such quality measures, they write.

Instead of being passive players, the researchers suggest that hospitals step in and “help design measures that can be efficiently collected and represent the true quality of care offered.”

To learn more:

Read the article “Choosing the Best Hospital: The Limitations of Public Quality Reporting” in the journal Health Affairs (fee or subscription may be required).
Visit the website of the Puget Sound Health Alliance, a local group that is working to improve health care quality, an effort that includes posting quality ratings of local hospitals and medical groups on its Community Checkup website.