Clinical practitioners have used calculators such as the Framingham Risk Score to assess the probability that a patient will develop coronary heart disease (CHD) and to guide primary prevention therapy for decades.1 In 2013, the American College of Cardiology (ACC) and the American Heart Association (AHA) released an updated risk calculator which has been widely criticized and may lead to over-prescribing of cholesterol-lowering agents (namely statins), aspirin, and antihypertensive drugs.2,3 Over-use of preventative therapies is not without risk: statins increase the risk for type 2 diabetes and aspirin can cause gastrointestinal bleeding.4,5 While many of these risk calculators have been validated in multiple cohorts, questions still remain. Do these widely used tools potentially over-expose patients to preventative therapies by over-estimating risk?
A prospective epidemiologic study sought to validate five cardiovascular risk calculators (outlined in Table 1) in the cohort of patients enrolled in the Multi-Ethnic Study of Atherosclerosis (MESA) study.6 MESA is a multicenter, prospective, epidemiologic study of cardiovascular disease (CVD) in a cohort believed to be representative of the population of the United States. Of note, MESA was not originally used to validate any of these risk calculators. The widely-used Framingham-based calculators were developed based on a population that was mainly white, in a small geographical region, with a health profile that is “less modern.”
Participants in MESA did not have CVD at the time of enrollment and new CHD/CVD events were recorded over 10-years of follow-up. The analysis was limited to men and women between the ages of 50 and 74 years without diabetes at baseline. The authors excluded patients with diabetes because two of the risk calculators, the ATPIII-Framingham Risk Score-CHD and the Reynolds Risk Score for men, are not intended for use in patients with diabetes. The study authors examined the discrimination (the ability to correctly identify higher and lower-risk subjects) and calibration (the ability to correctly match observed versus predicted event rates) of these 5 risk calculators. They also sought to determine if the routine use of modern preventative therapies (statins, aspirin, antihypertensive therapy) in the MESA cohort led to overestimation of cardiovascular risk.
Table 1: Summary of Risk Score Calculators Analyzed
Calculator |
Year Published |
Target Age Group |
Cardiovascular End Points Examined |
AHA-ACC-ASCVD Pooled Cohort Equations2 |
2014 |
30-74 years |
Myocardial infarction (MI), stroke, death from coronary heart disease (CHD) |
Reynolds Risk Score (RRS)5,6 |
2007-2008 |
Women: 45-80 years Men: 50-80 years |
MI, stroke, coronary revascularization, death from CHD |
Adult Treatment Panel (ATP) III-Framingham Risk Score (FRS)- CHD7 |
2002 |
>20 years |
MI, death from CHD |
FRS-Cardiovascular Disease (CVD)8 |
2011 |
30-74 years |
Angina, MI, stroke, heart failure, peripheral vascular disease (PVD), death from CHD |
FRS-CHD9 |
1998 |
30-74 years |
MI, angina, coronary insufficiency, death from CHD |
Out of the 4227 MESA participants included in this analysis, 46% (1961 participants) were male. The mean age of the cohort was 61.5 years. A majority of the participants were non-Hispanic whites (41.9%), with African American (26.3%) and Hispanic (20.2%) participants making up the next largest ethnic groups. Overall, the MESA cohort is more representative of the US population than other cohorts that have been used to validate risk calculators in the past. At baseline, total cholesterol (mean = 197mg/dL, high-density lipoprotein (mean = 52mg/dL), systolic blood pressure (mean = 126mmHg) were measured. Slightly more than half of the MESA cohort were using some sort of cardiovascular drug at baseline. The therapies most commonly used were antihypertensive treatments (32.7% of men, 36.2% of women), aspirin (27.8% of men, 21.4% of women), and lipid-lowering drugs (about 15% of both men and women). Median follow-up was 10.2 years, with 89% of the subjects having 9 or more years of follow-up.
The ability of each risk score to appropriately discriminate between higher and lower risk patients was determined using the c-statistic.12 Values for a c-statistic can range from 0.5-1. A value = 0.5 indicates that a model is no better than chance at predicting an outcome or event. A c-statistic = 1 indicates that the model is perfect in predicting an outcome. A c-statistic of ≥0.7 suggests that the model has reasonably good predictive value. In this cohort, the risk scores had c-statistics ranging from 0.69-0.71 when used to predict CVD events in men. In women, the risk scores had c-statistics between 0.67-0.72 except the FRS-CHD where the calculated c-statistic was only 0.60. In individuals estimated to have between 7.5-10% risk of an event using the AHA-ACC-ASCVD Pooled Cohort risk score (e.g. those who would be a candidate for statin therapy) had actual event rates of only 3% for men and 5.1% for women.
The ability to correctly match observed versus predicted event rates was evaluated by reporting the number of expected events and the number of observed events in the MESA cohort (see Table 2). Events were censored at 10-years of follow-up and participants with fewer than 10-years of follow-up had their 10-year risk estimated by an exponential survival function. Hosmer-Lemeshow plots were used to assess whether or not the observed event rates matched predicted event rates and chi-squared statistic was calculated separately for men and women for each score and outcome.13 Nearly all of the risk scores overestimated risk in both men (from 9% to 154%) and women (from 8% to 67%). The exception — the RRS underestimated risk in women (-21%). Table 2 summarizes the results between risk scores in men and women and the corresponding c-statistics. A sensitivity analysis was done to evaluate the cumulative effect of multiple preventative treatments such as aspirin, statin therapy, antihypertensive therapy, and revascularizaton. The authors did not find that preventative therapy was linked to over-estimation of risk.
Table 2: Predicted and Observed Events for Each Risk Score
Risk Score |
Predicted Events, % |
Observed Events, % |
Absolute Difference |
Discordance |
c-Statistic |
Total (n=4227) |
|
|
|
|
|
FRS-CHD |
9.41 |
6.22 |
3.18 |
51 |
0.68 |
FRS-CVD |
13.28 |
10.60 |
2.68 |
25 |
0.71 |
ATPIII-FRS-CHD |
6.83 |
3.17 |
3.66 |
115 |
0.71 |
RRS |
7.43 |
7.64 |
-0.21 |
-3 |
0.72 |
AHA-ACC-ASCVD |
9.16 |
5.16 |
4.00 |
78 |
0.71 |
Men (n=1961) |
|
|
|
|
|
FRS-CHD |
12.80 |
8.36 |
4.44 |
53 |
0.69 |
FRS-CVD |
18.29 |
13.31 |
4.98 |
37 |
0.71 |
ATPIII-FRS-CHD |
11.15 |
4.39 |
6.76 |
154 |
0.71 |
RRS |
10.89 |
9.99 |
0.89 |
9 |
0.70 |
AHA-ACC-ASCVD |
11.84 |
6.37 |
5.46 |
86 |
0.71 |
Women (n=2266) |
|
|
|
|
|
FRS-CHD |
6.47 |
4.37 |
2.10 |
48 |
0.60 |
FRS-CVD |
8.94 |
8.25 |
0.69 |
8 |
0.70 |
ATPIII-FRS-CHD |
3.10 |
2.12 |
0.98 |
46 |
0.67 |
RRS |
4.44 |
5.60 |
-1.17 |
-21 |
0.72 |
AHA-ACC-ASCVD |
6.84 |
4.10 |
2.74 |
67 |
0.70 |
There are several possible explanations why these calculators over-estimation risk. The authors posit that over time a different set of risk factors influence outcomes. The risk calculators developed and validated many years ago are therefore are no longer calibrated for modern cohorts. For example, the incidence of cigarette smoking can be accounted for, but the number of cigarettes smoked and the overall composition of cigarettes may be markedly different now. Other unmeasured and unaccounted for risk factors may also influence outcomes such as changes in salt and trans-fat intake.
This analysis has several limitations. The authors acknowledge that MESA participants may be more health-conscious and have better general health practices than the “general population.” As with any epidemiological cohort study, missing data can be a problem. However, the investigators estimated they missed no more than 9% of CVD events. Patients with diabetes represent a large part of our patient population and because they were excluded from the analysis, we simply do not know how well these calculators perform in this critically important patient population.
Four of the 5 risk calculators significantly overestimate CVD risk. The RRS score appears to be the most accurate, but significantly underestimates risk in women. Overestimating risk leads to unnecessary initiation of agents that can cause side effects and may result in higher health care costs. Decades of research and expertise are channeled into creating these tools, but the truth is they are not fool-proof. Should we continue to use these risk calculators to estimate CVD risk? Do you favor using the RSS over the other risk calculators? What about people with diabetes? Will you continue to use the ACC-AHA-ACVSD Pooled Cohort risk calculator knowing that it promotes the overuse of statins? We would love to hear your thoughts!
1. DeFilippis AP, Young R, Carruba CJ, et al. An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort. Ann Intern Med. 2015; 162: 266-275.
2. Goff DC Jr, Lloyd-Jones DM, Bennett G, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014; 129: S49-73.
3. Ridker PM, Cook NR. Satins: new American guidelines for prevention of cardiovascular disease. Lancet. 2013; 382: 1762-5.
4. Sattar N, Preiss D, Murray HM, et al. Satins and risk of incident diabetes: a collaborative meta-analysis of randomized statin trials. Lancet. 2010; 375: 735-742.
5. Huang ES, Strate LL, Ho WH, et al. Long term use of aspirin and the risk of gastrointestinal bleeding. Am J Med. 2011; 124(5): 426-433.
6. Bild DE, Bluemke DA, Burke GL, et al. Multiethnic study of atherosclerosis: objectives and design. Am J Epidemiol. 2002; 156: 871-81.
7. Ridker PM, Buring JE, Rifai N, et al. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA. 2007; 297: 611-19.
8. Ridker PM, Paynter NP, Rifai N, et al. C-reactive protein and parental history improve global cardiovascular risk prediction: the Reynolds Risk Score for men. Circulation. 2008; 118: 2243-51.
9. National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III). Third Report of the National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III) final report. Circulation. 2002; 106: 3143-421.
10. D’Agostino RB Sr, Vasan RS, Pencina MJ, et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008; 117: 743-53.
11. Wilson PW, D’Agostino RB, Levy D, et al. Prediction of coronary heart disease using risk factor categories. Circulation. 1998; 97: 1837-47.
12. Harrell FE Jr, Lee KL, Mark DB, et al. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996; 15: 361-87.
13. Lavalley MP. Statistical primer for cardiovascular research; logistic regression. Circulation. 2008; 117: 2395-2399.
The interesting thing with
The interesting thing with risk calculation is that no patient really has 20% chance of anything. In fact, your risk is actually either 0% or 100%. Clearly, risk calculators have limitations as outlined here, so I don’t feel any risk calculator should decide whether or not to treat. It’s merely a tool that you can use with the patient to make them aware of their risk and the modifiable risk factors they can work on. From there, it’s reasonable to inform the patient of their risk based on the calculator and hopefully they quit smoking, move more, and control their BP. If not, then a statin is likely necessary. Also, the 10-year risk isn’t helpful in patients 18-40 years old, yet this is probably the best time to intervene as atherosclerotic plaque is being laid down during this time. Waiting until someone’s in the 50’s is often too late as most event occur in the 6th and 7th decades of life. Prevention must start earlier, regardless of which calculator you use.
I agree with Dave, risk
I agree with Dave, risk calculators are a rough guidance only. How well they predict relates to the population used to establish them, the parameters selected, and the period in history when they were developed, as therapies not accounted for in their development change over time. In this case, the use of glucose measurement [A1C is a rough measure at best] as well as inflammatory marker use in the Reynolds formulae have been associated with eventual outcomes. The Reynolds score seems to be better at risk prediction in the current MESA population for men, but not women. If you drop people with diabetes at entry, that would certainly not help…but it should affect men and women pretty much the same [I’m generalizing here…]. The Reynolds score for women was developed in a cohort of almost exclusively white women. In the MESA population the percentage of African American women is quite high increasing the likelihood of poorer CV outcomes, hence the underestimation (?)
Overuse of statins
Early in the paper you say:
“In 2013, the American College of Cardiology (ACC) and the American Heart Association (AHA) released an updated risk calculator which has been widely criticized and *may lead to* over-prescribing of cholesterol-lowering agents (namely statins), aspirin, and antihypertensive drugs.
At the end of the paper you say:
Will you continue to use the ACC-AHA-ACVSD Pooled Cohort risk calculator knowing that it *promotes the overuse* of statins?
Early in the paper you treat it is a possibility but at the end treat it as a fact. Do we know this for a fact?
Overuse of statins vs. increased use
Douglas – This analysis of the MESA cohort confirms the findings from at least one earlier study – the ACC-AHA-ASCVD Pooled Cohort risk calculator leads to more patients being judged to be “at risk” and therefore leads to increased use of statins (and other cardioprotective drugs). So, in that sense, we know with certainty that statins will be prescribed more if the ACC-AHA-ASCVD Pooled Cohort risk calculator is used (when compared to other risk calculators). Whether this represents “over use” (implying that they are being prescribed more than is necessary) is certainly debateable. Will more patients be saved (from CVD morbidity and mortality) or harmed (due to ADRs) if statins are used more frequently? That’s certainly is an important question that no analysis (that I’m aware of) has definitively answered to date.
That it is still debatable is
That it is still debatable is my point. It can’t be stated as a fact that the ACC/AHA calc/guidelines lead to overuse.
Will we ever have a
Will we ever have a definitive answer on primary prevention? My guess is likely not. The best that we can do is keep refining our equations to try to target those who will most likely benefit.
Four of the 5 calculators had
Four of the 5 calculators had a C-statistic >/= 0.7 for men, and 3 of 5 in women. Does this not mean that they have reasonably good predictive value?