Cardiovascular Risk Calculators – Use or Lose?

Clinical practitioners have used calculators such as the Framingham Risk Score to assess the probability that a patient will develop coronary heart disease (CHD) and to guide primary prevention therapy for decades.¹  In 2013, the American College of Cardiology (ACC) and the American Heart Association (AHA) released an updated risk calculator which has been widely criticized and may lead to over-prescribing of cholesterol-lowering agents (namely statins), aspirin, and antihypertensive drugs.^2,3   Over-use of preventative therapies is not without risk: statins increase the risk for type 2 diabetes and aspirin can cause gastrointestinal bleeding.^4,5 While many of these risk calculators have been validated in multiple cohorts, questions still remain.  Do these widely used tools potentially over-expose patients to preventative therapies by over-estimating risk?

A prospective epidemiologic study sought to validate five cardiovascular risk calculators (outlined in Table 1) in the cohort of patients enrolled in the Multi-Ethnic Study of Atherosclerosis (MESA) study.⁶ MESA is a multicenter, prospective, epidemiologic study of cardiovascular disease (CVD) in a cohort believed to be representative of the population of the United States. Of note, MESA was not originally used to validate any of these risk calculators.  The widely-used Framingham-based calculators were developed based on a population that was mainly white, in a small geographical region, with a health profile that is “less modern.”

Participants in MESA did not have CVD at the time of enrollment and new CHD/CVD events were recorded over 10-years of follow-up.  The analysis was limited to men and women between the ages of 50 and 74 years without diabetes at baseline. The authors excluded patients with diabetes because two of the risk calculators, the ATPIII-Framingham Risk Score-CHD and the Reynolds Risk Score for men, are not intended for use in patients with diabetes.  The study authors examined the discrimination (the ability to correctly identify higher and lower-risk subjects) and calibration (the ability to correctly match observed versus predicted event rates) of these 5 risk calculators. They also sought to determine if the routine use of modern preventative therapies (statins, aspirin, antihypertensive therapy) in the MESA cohort led to overestimation of cardiovascular risk.

Table 1: Summary of Risk Score Calculators Analyzed

Out of the 4227 MESA participants included in this analysis, 46% (1961 participants) were male. The mean age of the cohort was 61.5 years. A majority of the participants were non-Hispanic whites (41.9%), with African American (26.3%) and Hispanic (20.2%) participants making up the next largest ethnic groups. Overall, the MESA cohort is more representative of the US population than other cohorts that have been used to validate risk calculators in the past. At baseline, total cholesterol (mean = 197mg/dL, high-density lipoprotein (mean = 52mg/dL), systolic blood pressure (mean = 126mmHg) were measured.  Slightly more than half of the MESA cohort were using some sort of cardiovascular drug at baseline. The therapies most commonly used were antihypertensive treatments (32.7% of men, 36.2% of women), aspirin (27.8% of men, 21.4% of women), and lipid-lowering drugs (about 15% of both men and women).  Median follow-up was 10.2 years, with 89% of the subjects having 9 or more years of follow-up.

The ability of each risk score to appropriately discriminate between higher and lower risk patients was determined using the c-statistic.¹² Values for a c-statistic can range from 0.5-1.  A value = 0.5 indicates that a model is no better than chance at predicting an outcome or event. A c-statistic = 1 indicates that the model is perfect in predicting an outcome. A c-statistic of ≥0.7 suggests that the model has reasonably good predictive value. In this cohort, the risk scores had c-statistics ranging from 0.69-0.71 when used to predict CVD events in men. In women, the risk scores had c-statistics between 0.67-0.72 except the FRS-CHD where the calculated c-statistic was only 0.60.  In individuals estimated to have between 7.5-10% risk of an event using the AHA-ACC-ASCVD Pooled Cohort risk score (e.g. those who would be a candidate for statin therapy) had actual event rates of only 3% for men and 5.1% for women.

The ability to correctly match observed versus predicted event rates was evaluated by reporting the number of expected events and the number of observed events in the MESA cohort (see Table 2). Events were censored at 10-years of follow-up and participants with fewer than 10-years of follow-up had their 10-year risk estimated by an exponential survival function. Hosmer-Lemeshow plots were used to assess whether or not the observed event rates matched predicted event rates and chi-squared statistic was calculated separately for men and women for each score and outcome.¹³  Nearly all of the risk scores overestimated risk in both men (from 9% to 154%) and women (from 8% to 67%). The exception — the RRS underestimated risk in women (-21%). Table 2 summarizes the results between risk scores in men and women and the corresponding c-statistics.  A sensitivity analysis was done to evaluate the cumulative effect of multiple preventative treatments such as aspirin, statin therapy, antihypertensive therapy, and revascularizaton. The authors did not find that preventative therapy was linked to over-estimation of risk.

Table 2: Predicted and Observed Events for Each Risk Score

There are several possible explanations why these calculators over-estimation risk. The authors posit that over time a different set of risk factors influence outcomes.  The risk calculators developed and validated many years ago are therefore are no longer calibrated for modern cohorts. For example, the incidence of cigarette smoking can be accounted for, but the number of cigarettes smoked and the overall composition of cigarettes may be markedly different now. Other unmeasured and unaccounted for risk factors may also influence outcomes such as changes in salt and trans-fat intake.

This analysis has several limitations. The authors acknowledge that MESA participants may be more health-conscious and have better general health practices than the “general population.” As with any epidemiological cohort study, missing data can be a problem. However, the investigators estimated they missed no more than 9% of CVD events.  Patients with diabetes represent a large part of our patient population and because they were excluded from the analysis, we simply do not know how well these calculators perform in this critically important patient population.

Four of the 5 risk calculators significantly overestimate CVD risk. The RRS score appears to be the most accurate, but significantly underestimates risk in women. Overestimating risk leads to unnecessary initiation of agents that can cause side effects and may result in higher health care costs.  Decades of research and expertise are channeled into creating these tools, but the truth is they are not fool-proof. Should we continue to use these risk calculators to estimate CVD risk? Do you favor using the RSS over the other risk calculators?  What about people with diabetes? Will you continue to use the ACC-AHA-ACVSD Pooled Cohort risk calculator knowing that it promotes the overuse of statins? We would love to hear your thoughts!