Nursing Facilities’ “Quality Measures” Do Not Reflect Actual Quality of Care Provided to Residents

The Center for Medicare Advocacy wanted to determine whether nursing facilities that had one-star in their health survey ratings on Nursing Home Compare were able to boost their overall ratings from one star to two stars through the designation of five stars in the self-reported quality measure domain.  The finding – that many facilities in the five states reviewed were able to boost their overall ratings in this way – undermines the validity of “quality measures” as reflecting the actual quality of care provided to residents by nursing facilities.

When facilities that are challenged with poor performance cite their “quality measures,” beware.  Quality measures do not reflect the quality of care residents receive.

Background

The Centers for Medicare & Medicaid Services’ (CMS’) website Nursing Home Compare provides public information about nursing facilities that participate in Medicare or Medicaid or both.  It provides information about three domains – health surveys, staffing, and quality measures – and an overall rating. 

• The health survey rating is based on unannounced annual and complaint surveys conducted by state survey agencies, under contract with CMS;
• The nurse staffing rating is based on total nurse staffing (registered nurses (RNs), licensed practical nurses and nurse aides) and RN staffing; and
• The quality measures rating is based primarily on self-reported and unaudited resident assessment information provided by facilities.   

CMS assigns star ratings to each of the three domains, and an overall score which starts with the health survey rating and revises the overall rating downward or upward to reflect, respectively, one-star and five-star ratings on the staffing and quality measure domains.  Ratings in each domain and in the overall rating range from one to five stars, with one star reflecting the lowest performance (“much below average”) and five stars, the highest performance (“much above average”).

The three domains are separately scored, each with highly complex scoring rules.  As described in CMS’s Technical Users’ Guide,[1] the rating system is based on the following principles:

• The health survey domain “calculates a health inspection score based on points assigned to deficiencies identified in each active provider’s two most recent recertification health inspections prior to November 28, 2017, as well as on deficiency findings from the most recent two years of complaint inspections prior to November 28, 2017.”[2]  Ratings are “fixed” for each state: the top 10% receive 5 stars; the middle 70% of facilities receive a rating of two, three, or four stars, with an equal number (approximately 23.33%) in each star rating category; and the bottom 20% receive one star.[3]  

  CMS “froze” the survey domain for a year, effective November 27, 2017, to reflect implementation of the new federal survey process.[4]  Deficiencies that are cited in surveys conducted after November 27, 2017 are reported on Nursing Home Compare, but they are not used in calculating the health survey rating.
   

• The staffing domain reports two components – total nurse staffing (RN, licensed practical nurses, and certified nurse aides) and on RN staffing, which both are case-mix adjusted.[5]  Since May 2018, the staffing domain has been based on payroll-based journal information (PBJ), not, as before, on self-reported information.[6] 

  Scoring uses “cut points,” [7] which CMS adjusted so that the ratings looked the same after April 2018, when CMS first began to report PBJ-reported staffing data.  The New York Times reported that 70% of nurse facilities had lower staffing numbers, averaging 12% lower under PBJ than under the prior self-reported system, and that, in the last quarter of 2017, a quarter of all nursing facilities reported at least one day in the three-month period when there was no RN present in the facility.[8] 
   

• Nursing Home Compare reports 27 quality measures (24 assessment-based quality measures that are self-reported by facilities and 3 measures that are both assessment-based and claims data-based),[9] but the quality measure rating is based on only 16 of the 27 measures (9 long-stay self-reported measures for residents in the facility for 100 days or more; 4 short-stay self-reported measures for residents in the facility for fewer than 100 days; and 3 claims-based short-stay measures).[10]

  Each measure is separately scored and assigned points (20-100) based on the national distribution of the measure.[11]  CMS then groups facilities “into quintiles based on the national distribution of the QM,” with the lowest-performing facilities receiving 20 points and the highest-performing facilities, 100 points.[12]  The point scores for each of the 16 measures are added together and ratings assigned according to cut points.[13]
   

• A five-star rating in the quality measure domain boosts a facility’s overall rating by one star, under the scoring rules used by CMS.[14] 

This Analysis

The Center began by identifying nursing facilities listed on Nursing Home Compare on August 7, 2018 with overall ratings of two stars.  Next the Center looked at these two-star facilities to determine how many had five stars in the quality measure domain.  Finally, the Center calculated how many facilities had one star in the health survey domain and five stars in the quality measure domain. 

Facilities with overall ratings of two stars report five stars in quality measures, from 39% of the two-star facilities in Iowa to 80% of the two-star facilities in California. 

Even more disturbing, many facilities with overall ratings of two stars had one star in health surveys but five stars in quality measures, with the quality measure rating boosting the overall rating to two stars.  These boosts occurred in 28 facilities (33%) in Iowa and in 167 facilities (71%) in California.

State	Number of facilities with 2 stars in overall rating	Number of 2-star facilities with 5 stars in quality measure domain	Percentage of 2-star facilities with 5 stars in quality measure domain	Number of 2-star facilities with 1 star in health survey domain and 5 stars in quality measure domain	Percentage of 2-star facilities that boosted their rating from 1 to 2 stars, based on their quality measure rating
California	235	188	80%	167	71%
Connecticut	47	28	60%	26	55%
Iowa	84	33	39%	28	33%
New York	115	65	56%	46	40%
Wisconsin	68	35	51%	33	48%

Conclusion

These findings confirm that the quality measure domain overstates the “quality” of a facility, is completely unreliable, and should not be used in determining facilities’ overall ratings.  As The New York Times reported in 2014, facilities “game” the rating system, particularly in their misuse of the quality measure rating.[15]

 

August 9, 2018 – T. Edelman