PRETestConsult.com has developed an ACS risk module that will enable clinicians (and patients) to calculate the pretest probability of ACS (acute coronary syndrome) when a patient presents acutely with chest pain. This critical review represents my personal impression of the scientific validity and clinical usefulness of their ACS risk module.
Background information regarding the value of estimating the pretest probability of ACS
Most US-based patients who develop acute chest pain, which could be due to acute cardiac ischemia, usually go to a hospital ED where they are routinely evaluated by an EP (emergency physician). With respect to the diagnostic possibility of an ACS event, many EPs adopt both a rule-out and rule-in clinical posture based on Bayesian thinking. A ruled-out clinical posture can be loosely defined as the EP's final clinical impression (after history taking and physical examination and EKG testing) that an ACS event is unlikely to be present, and it represents the EP's final clinical belief that it is safe to discharge the chest pain patient from the ED (presuming that there is no other disease causing the chest pain that requires hospital admission). The threshold definition of "unlikely" varies and different EPs adopt different ruled-out threshold values. The rule-out threshold is also called the "test-do not test" threshold and it is the threshold value below which further diagnostic testing is no longer indicated, because the disease is deemed not to be present. If an EP's final estimated probability of ACS is below the "test-do not test" threshold, then the EP assumes that he has reasonably and safely excluded ACS. With respect to the disease entity of ACS, most EPs adopt a rule-out threshold value of <5% and many EPs would be more comfortable with a lower threshold value eg. 1-2% probability of ACS. PRETestConsult.com adopts a rule-out threshold value of 2% as being optimum -- based on a calculation which is available on this website [1]. Although I may disagree with some of the presumptions and figures used in their calculation, I am content to assume that a rule-out ACS threshold of 2% is a reasonable choice. This rule-out threshold issue is actually much more complicated than it may initially appear, because PRETestConsult.com adopts a specific definition of ACS that only includes those ACS patients who will have an AMI or die or have a revascularisation procedure within 45 days, while many "gestalt" EPs will be mentally adopting a rule-out threshold of 1-5% that applies globally to all ACS patients (unstable angina patients + AMI patients) even though they have no scientific method of proving that they can exclude ACS to that degree of certainty (<1-5% predicted likelihood of ACS). I will discuss this complex issue in greater detail later in this review.
If an EP estimates that a chest pain patient has a predicted probability of ACS that is greater than the ruled-out threshold value (above the "test-do not test" threshold value) then he will presumably recommend that the patient be admitted to a chest pain obervation unit for further clinical evaluation. If an EP has a particularly strong clinical impression that an ACS event is definitely present -- that the predicted probability of ACS is above a certain threshold value, called the rule-in threshold value -- then he may also decide to institute specific ACS-treatment in the ED setting before triaging the patient to a cardiac ICU (rather than a chest pain observation unit). How certain should an EP be about a "definite" diagnosis of ACS before instituting targeted anti-ACS treatment in the ED setting? I strong suspect that individual clinicians will define the term "definite" differently. Some clinicians will state that they are only definitely certain that a particular diagnosis is present if they are 95-100% certain that the disease is present, while other clinicians will be content with lower levels of diagnostic certainty eg. >90% certain or >80% certain. Can a "gestalt" EP be >80% certain that an ACS event is present in a chest pain patient using only information derived from the clinical history, clinical examination and ED-presentation EKG? I know of no clinical studies that can prove that EPs can achieve that level of diagnostic certainty. A cardiologist may reach that level of diagnostic certainty after utilising a series of sequential diagnostic tests (eg. serial cardiac marker testing, nuclear imaging and/or echocardiography, coronary angiography), but an EP can only rationally expect to achieve a significantly lower degree of diagnostic certainty in the ED setting. Therefore, when I use the term "rule-in threshold" for an EP in an ED setting, it does not imply that the disease has definitively been ruled-in. Rather it implies that the EP believes (based on the history and examination and EKG findings) that the probability of ACS is high enough to justify admission to a cardiac ICU + the expeditious institution of anti-ACS treatment in the ED setting. I have never seen a precise threshold value for a ruled-in ACS threshold for an EP in an ED setting, and I presume that different EP use different, and arbitrary, ruled-in ACS threshold values. Some EPs may only institute specific anti-ACS treatment in the ED if they are >80% certain that an ACS event is present, while other EPs may choose to use a lower ruled-in threshold value eg. 60% predicted probability of an ACS event. I know of no scientific method of precisely determining what should be the appropriate ruled-in threshold value for deciding that specific anti-ACS treatment should be instituted by an EP in the ED setting, and I will examine this complex issue in much greater detail later in this review.
If the estimated pretest probability of ACS is above the rule-out threshold (eg. 2% probability of ACS) but below an individual clinician's arbitrary rule-in threshold (eg. 60-80% probability of ACS) then the predicted probability of ACS is in an intermediate zone. If the estimated probability of ACS is in the intermediate zone, then the patient should be admitted to hospital for further diagnostic testing in order to finally verify, or finally exclude, the presence of ACS. From the perspective of an individual EP, he will probably not feel justified to institute anti-ACS therapy while the patient is in the ED and he will probably discuss the issue of patient triage (cardiac ICU versus chest pain observation unit) with the cardiologist prior to admitting the patient.
In summary, I suspect that most EPs routinely try to place a suspected ACS patient into one of three clinical decision-making bins, and then manage the patient accordingly.
i) Bin 1 -- final estimated probability of ACS is below the rule-out threshold (eg. 2% probability of ACS) => discharge from the ED.
ii) Bin 2 -- final estimated probability of ACS is in the intermediate zone between the rule-out threshold (eg. 2% probability of ACS) and the rule-in threshold for an EP in an ED setting (eg. 60-80% probability of ACS) => patient admitted to a chest pain observation unit (or a cardiac ICU -- depending on the cardiologist's choice) .
iii) Bin 3 -- final estimated probability of ACS is above the ruled-in threshold for an ED setting (eg. 60-80% probability of ACS) => institute anti-ACS treatment in the ED setting and admit the patient to a cardiac ICU.
In this review, I will attempt to determine whether PRETestConsult.com's ACS risk prediction module (which primarily utilises information derived from the patient's history, examination and presentation EKG) can help an EP appropriately place a chest pain patient into one of those three bins.
How does PRETestConsult.com's ACS module actually work?
The ACS module is based on the attribute matching profiling methodology
of determining the pretest probability of ACS. PRETestConsult.com claims
to have used a sample base of >15,000 suspected ACS patients who underwent
attibute matching profiling. I am including a copy of Jeff Kline's comments
on attribute matching (which I obtained from reference number 2) in this
box.
|
Attribute
matching represents a novel method of explicit pretest probability assessment.
This method that requires a computer, or personal digital assistant and
a large database. Attribute matching basically asks the doctor to create
a profile, based upon several attributes (such as age, gender, race, pulse
rate, prior history, physical examination findings). The computer then
searches through the database and returns only the patients who match all
of the properties of the attributes. In this way, attribute matching shows
the user a group of patients who share some very similar traits as the
patient in front of the doctor. The trick to attribute matching is to use
a database that has a large number of patients and for the outcomes of
the patients to be known and to be correct. The process demands a large
number of patients to ensure a reasonably large match size, which produces
a stable match. In fact, attribute matching is the only method of
pretest probability that can provide any degree of certainty about match
stability, defined as the 95% confidence interval for the proportion, which
is a standard method of estimating the boundaries as to where the true
probability lies. The smaller the confidence interval, the more certain
we can be of the pretest probability estimate. If only one patient in the
database matches the attribute profile of a given patient, then not much
of a conclusion can be made about pretest probability, except that the
patient is probability unusual. If the database returns 100 patients who
match the attribute profile of the unknown patient, then a stronger inference
can be made. Computation of pretest probability depends upon the database
“knowing” the outcomes. Pretest probability can be computed as:
100%* # with the disease of interest / all patients matched from the database Suppose for a moment that in the case of the 100 matched subjects, the database erroneously says that 10 patients had the disease, when the truth was that they did not. Then the pretest probability from the outset would be doomed to be off by at least 11% (that’s 10/90). So obviously, the database must have valid, accurate outcomes, to estimate the pretest probabilityaccurately. Also, the database must be big enough to produce a large enough match size to produce a reliable estimate of the pretest probability in enough patients to make the system useful. Finally, the researchers who are constructing an attribute matching system must use relevant variables as the matching attributes. For example, to determine the pretest probability of coronary artery disease, then the variable of “recent travel” might be too irrelevant to create an accurate profile. Attribute matching represents a new but promising method for accurate pretest probability assessment. A commercially available system known as the ACS PREtest Consult has been developed for the prediction of acute coronary syndrome in the emergency department. The system asks for eight parameters to be input. The database for this particular attribute matching system represents 15,000 ED patients evaluated for possible acute coronary syndrome with at least an EKG. All patients had 30 day follow-up, and many had a provocative study such as a treadmill test. The ACS PREtest Consult was compared to the ACI-TIPI method for pretest probability assessment in an independent validation population consisting of 6,000 ED patients evaluated for chest pain. Eight percent of these patients had acute coronary syndrome within 30 days. The ACS PREtest Consult instrument categorized 24% of allpatients as having a very low pretest probability, defined as <2% . In comparison,ACI-TIPI, categorized only 4% as having a pretest probability <2%. Both systems were equally accurate, meaning that the true prevalence of ACS was 1.7% in the very low group assigned as very low by the ACS PREtest Consult and the true prevalence was 1.6% in the group that ACI-TIPI deemed very low risk. Moreover,only 1.0% of patients who hada pretest probability estimate <2% plus a single negative troponin I measurement developed acute coronary syndrome within 30 days. This is about the level of certainty that can be obtained by putting a patient through a standard chest pain protocol, including 2 troponin measurements and a negative exercise treadmill electrocardiography test. Thus, the ACS PREtest consult may allow more low-risk patients to be safely discharged without subjecting them to the risk of a false positive provocative test (possible prompting unnecessary cardiac catheterization), nor the wasted time, expense and inconvenience of an unnecessary hospital stay. Also, the results of the matching query can be displayed for the patient to see, so that he or she can become involved in the decision-making process, and the results can be stored and time-stamped for later proof of logic behind the decision-making in the event of civil litigation. The downside of attribute matching includes the fact that it is not free, and that it requires a computer to operate and a large database must exist before using the tool clinically. The attribute matching is a relatively new concept, and more validation may be necessary before it is widely accepted. |
To see the ACS module in action, go to http://evidencemd.com/pt/nav.php and click on evaluate, and then click on ACS risk. It is also possible to download a beta version of their ACS risk module for palm-pilot from their official website (see reference number 3).
It is very important to understand that the ACS risk module uses attribute matching profiling to determine the 45 day risk of an ACS event, which is defined as an ACS event that results in an AMI, revascularisation or cardiac death within the next 45 days. The definition does not include all ACS patients, and it specifically does not include ACS patients who do not have an AMI/revascularisation/death within 45 days.
Is PRETestConsult.com's ACS risk module clinically useful from the perspective of ruling-out an ACS event in an ED setting?
Consider how the ACS risk module handles a common clinical scenario -- a 51 year old white male, who presents to the ED with chest pain. If the hypothetical patient is a non-smoker, a non-diabetic and a non-hypertensive, and has no history of CHD, and the chest pain is not reproduced by palpation and the initial EKG in the ED is normal, then the ACS risk module predicts that the pretest probability of ACS for that hypothetical patient is 11.2% (based on 66 ACS outcomes in 550 matched patients). This 11.2% figure is greater than the "test-do not test" threshold of 2% and PRETest.Consult.com recommends that this patient be admitted to hospital because of a >2% risk of a serious ACS event (AMI/revscularisation/death) within 45 days.
What is very interesting from my perspective, is the fact that PRETestConsult.com's ACS risk module cannot generate a predicted probability of ACS of <2% if the patient is a white male who is >51 years of age. The lowest predicted ACS risk that I could generate for a >51 year old white male is 4.2% (by changing the values of the input data and inputting "chest pain reproduced by palpation" while keeping all the other input parameters unchanged). That ACS prediction value is still above the threshold value of 2%, and that chest pain patient should therefore also be admitted to hospital. In other words, PRETest.Consult.com's ACS risk module has no useful rule-out utility in the most important target population of chest pain patients in an ED setting - white male patients >51 years of age. An EP doesn't even have to bother to use PRETEstConsult.com's ACS risk module as a rule-out ACS prediction tool for those patients, because the ACS risk prediction tool will never generate a predicted ACS rate of <2%.
How useful is the ACS risk module for white male patients between 39-50 years of age? If the hypothetical white male patient is a non-smoker, a non-diabetic and a non-hypertensive, and has no history of CHD, and the chest pain is not reproduced by palpation and the initial EKG in the ED is normal, then the ACS risk module predicts that the pretest probability of ACS for that hypothetical patient is 2.2% (based on 11 ACS outcomes in 397 matched patients). That figure is still above the the 2% threshold figure recommended by PRETestConsult.com and that patient should theoretically be admitted to hospital. The only way that I could generate a pretest probability of ACS for a 39-50 year old male patient that is <2% is to choose "chest pain reproduced by palpation" while keeping all other input parameters unchanged. That specific choice would result in a 0.2% predicted pretest probability of ACS if all the other input parameters are left unchanged (based on 0 ACS outcomes in 25 matched patients). It is important to note that only a very small proportion of 39-50 year old white male patients can be classified as having "chest pain reproduced by palpation" (25 versus 397 in their sample population), which means that PRETest.Consult.com's ACS risk module cannot generate a pretest probability risk of <2% for the majority of white male patients in the 39-50 year old age group.
What also particularly bothers me about PRETest.Consult.com's ACS risk module is that it ignores other historical factors that could greatly influence a "gestalt" clinician's clinical decision-making in borderline cases. Consider the following two clinical scenarios.
Clinical scenario number 1: A 48 year old white male patient presents to the ED with chest pain. The pain has been present for 30 minutes. The pain consists of a sharp pain that is localised to a palm-sized area of the left anterolateral chest. The pain waxes-and-wanes and is mainly present in short bursts lasting 20-30 seconds. The pain is occasionally, but inconsistently, aggravated by torso movements, and it is unaffected by breathing or coughing. There is no history of chest trauma. There is no pain radiation to the arms or jaw. No other significant symptoms. No history of smoking or diabetes or hypertension or CHD. Cholesterol level unknown. Normal vital signs. Normal physical examination. No reproducible chest pain or diaphoresis. Normal EKG and chest X-ray.
Clinical scenario number 2: A 48 year old white male patient presents to the ED with chest pain. The pain has been present for 30 minutes. The pain consists of a poorly localised chest tightness across the anterior chest. The pain is constant and is unaffected by breathing, coughing and torso movements. The patient also complains of some aching in both arms and jaw. No other significant symptoms. No history of smoking or diabetes or hypertension or CHD. Cholesterol level unknown. Normal vital signs. Normal physical examination. No reproducible chest pain or diaphoresis. Normal EKG and chest x-ray.
According to PRETestConsult.com's ACS risk module, both patients have a predicted pretest probability of ACS of 2.2%. What should an EP do if the two hypothetical clinical scenario patients refuse to be admitted to hospital when informed by the EP that their predicted pretest probability of ACS is only 0.2% above the recommended threshold admission value of 2%? I personally think that clinical scenario patient number 2 may be making a grave mistake if he decides to leave the hospital because he perceives that he only has a 2.2% likelihood of having an ACS event, and I think that the EP is partly responsible if he does not provide the patient with additional information. What additional information? It is important to remember that PRETestConsult.com's ACS risk module does not predict the pretest probability of ACS that does not result in a MI/revascularisation ot death within 45 days. Many ACS patients, if correctly identified and expeditiously treated, will not necessarily develop a MI or death, or require a revascularisation procedure within 45 days. I personally think that clinical scenario 2 patient could be experiencing an ACS event, and I think that it would be foolish to discharge that patient just because a clinical prediction tool (which is based on 11 ACS outcomes in 397 matched patients) predicts that the patient only has a 2.2% probability of a serious ACS event (death, MI or a need for a revascularisation procedure within the next 45 days). How can an EP rationally and definitively reassure clinical scenario number 2 patient that although he may be having an ACS event, that there is only a 2.2% chance of him experiencing a serious ACS-related outcome within the near-future, and that he therefore doesn't necessarily need to be admitted to hospital for further evaluation? In other words, I think that there will often be a discrepancy between a "gestalt" clinician's predicted probability of ACS estimation (which includes all ACS patients) and PRETestConsult.com's predicted probability of ACS estimation (which is only focused on those ACS patients who are predicted to have an AMI/death/revascularisation within 45 days). I wonder how individual EPs are going to handle this discrepancy if they intuitively believe that a particular chest pain patient (like clinical scenario number 2 patient) should be classified as having an intermediate probability of ACS (higher than the rule-out threshold but lower than the rule-in threshold), but PRETestConsult.com's ACS risk module classifies the patient as having a <2% probability of a serious ACS event within 45 days.
Is PRETestConsult.com's ACS risk module clinically useful from the perspective of ruling-in an ACS event in an ED setting?
With respect to a suspected ACS patient, an EP needs to make a number of appropriate medical decisions. First of all, if his clinical suspicion of ACS exceeds a certain threshold value (eg. 2-5% predicted probability of ACS), then he would presumably recommend that the patient be admitted to hospital for further diagnostic testing so that the diagnosis of ACS can be more definitively established (action number 1). Secondly, if an EP firmly believes that the patient is experiencing an ACS event and that the predicted probability value exceeds a certain threshold value (eg. 60-80% predicted probability of an ACS event), then he may subsequently undertake three actions -- i) expeditiously institute anti-ACS treatment in the ED setting; ii) expeditiously consult a cardiologist; iii) expeditiously arrange admission to a cardiac ICU rather than a chest pain observation unit (action number 2).
Does PRETestConsult.com's ACS risk module help an EP decide whether to undertake action number 2 because the predicted ACS risk value exceeds a certain probability value?
Consider the following two clinical case scenarios:
Clinical case scenario number 1: A 69 year old white male presents to the ED with chest pain. The chest pain has been present for 30 minutes and consists of a poorly localised heaviness across the anterior chest, plus some aching in both arms and jaw. The patient has a known history of CHD and diabetes. He is a non-smoker. Cholesterol level unknown. Vital signs are normal. There is no diaphoresis or chest pain reproducible by palpation. EKG shows ST depression (see EKG sample number 1).
EKG sample number 1:
What is your predicted probability of an ACS event estimation for this patient and does it exceed a threshold value that justifies action number 2 (expeditious institution of anti-ACS treatment, expeditious cardiology consultation, and expeditious admission to a cardiac ICU)?
Clinical case scenario number 2: A 39 year old white male presents to the ED with chest pain. The chest pain has been present for 30 minutes. The pain is vaguely localised to the anterior chest. There is no arm or jaw pain. The patient has no history of CHD or diabetes, but he does have a history of untreated hypertension. Cholesterol level is unknown. He is a non-smoker. Vital signs are normal except for an elevated blood pressure of 200/120. There is no diaphoresis or chest pain reproducible by palpation. EKG shows ST depression and some inverted T waves (see EKG sample number 2).
EKG sample number 2:
What is your predicted probability of an ACS event estimation for this patient and does it exceed a threshold value that justifies action number 2 (expeditious institution of anti-ACS treatment, expeditious cardiology consultation, and expeditious admission to a cardiac ICU)?
What does PRETestConsult.com's ACS risk module predict the ACS risk will be for those two hypothetical patients?
PRETestConsult.com's ACS risk module's predicted pretest probability for clinical scenario patient number 1 is 18.2% (based on 11 ACS outcomes in 61 matched patients).
PRETestConsult.com's ACS risk module's predicted pretest probability for clinical scenario patient number 2 is 66.2% (based on 2 ACS outcomes in 3 matched patients).
Are you surprised by the results? Did you expect that the predicted probability of an ACS event for clinical scenario number 1 patient would be much less than the predicted value for clinical scenario number 2 patient? Would those predicted probability of ACS values help you decide whether to institute action number 2 (expeditious institution of anti-ACS treatment, expeditious cardiology consultation, and expeditious admission to a cardiac ICU)? How do you explain to clinical scenario number 1 patient why his EKG demonstrates ischemic-appearing ST segment depression if the clinical prediction tool predicts that he only has a 18.2% chance of having an ACS event? What is likely causing the ST segment depression in the other 50 patients in PRETestConsult.com's sample population of 61 matched patients, considering that only 11 patients ended up with a diagnosis of ACS?
It is my personal impression that PRETestConsult.com's ACS risk module is seriously flawed from a rule-in ACS in an ED setting perspective, and I also believe that it has no/little utility in helping an EP to decide whether action number 2 (expeditious institution of anti-ACS treatment, expeditious cardiology consultation, and expeditious admission to a cardiac ICU) is recommended for a chest pain patient in an ED setting.
Consider my personal reasons for rejecting the scientific legitimacy and clinical utility of PRETestConsult.com's ACS prediction tool as rule-in ACS prediction tool.
As a "gestalt" physician, I am >60% certain that clinical scenario number 1 patient is having an ACS event, which justifies action number 2 (expeditious institution of anti-ACS treatment, expeditious cardiology consultation, and expeditious admission to a cardiac ICU). I cannot prove that my "gestalt" prediction is accurate because it is merely based on my personal weighting of the evidence. My personal belief is based on the additive effect of the following facts-: patient age 69 years + white male + known CHD + "classical" anginal chest pain presentation with pain radiation to the jaw/arms + very ischemic-appearing EKG with horizontal planar depression of the ST segments in contiguous leads (anterolateral leads). Likewise, although I think that clinical scenario number 2 patient may be having an ACS event and I therefore think that he requires admission to hospital for further evaluation, I am definitely less than 60% certain that the etiology of the chest pain is primarily due to coronary ischemia secondary to ACS (ACS is usually due to acute plaque rupture within a coronary artery and secondary local thrombosis). The chest pain could be due to the effects of LVH outstripping the "effective" coronary blood supply in the presence of severe afterload. In particular, I think that the ST depression in clinical scenario number 2 patient's EKG is due to LVH and I believe that there is no evidence of cardiac ischemia/injury in his EKG (click here for my detailed interpretation of the EKG). In other words, my "gestalt" ACS predictions are exactly opposite to PRETestConsult.com's ACS predictions -- I think that clinical scenario number 1 patient has a >60% likelihood of having an ACS event, while PRETestConsult.com's ACS prediction value is 18%; and I think that clinical scenario number 2 patient has considerably less than a 60% likelihood of having an ACS event, while PRETestConsult.com's ACS prediction value is 66%. In a personal back-and-forth e-mail communication with Jeff Kline, who is the originator of the ACS pretest probability module based on attribute matching profiling, Jeff Kline argued that even if PRETestConsult.com's predictions appear counter-intuitive, they are more accurate and scientific than any "gestalt" clinician's predictions. Do you agree?
I personally reject Jeff Kline's argument for two major reasons. First of all, the attribute matching system's sample sizes are far too small. There were only 3 matched patients for clinical scenario number 2 patient! That is a ridiculously small sample size and I do not think that it can be used as a solid foundation for any EBM-type scientific prediction. Secondly, we are also talking about different ACS endpoint definitions. PRETest.consult.com's ACS endpoint definition only includes those ACS patients who develop a MI, or die or require a revascularisation procedure within 45 days. It does not include the majority of ACS patients -- those ACS patients who stabilise following expeditious anti-ACS therapy and who do not subsequently develop an AMI or die or require an urgent revascularisation procedure. Note that PRETestConsult.com claims that only 18% of their high risk patients (white males >51 years who have known CHD and ST segment depression >0.5mm in their presentation EKG) actually had an ACS outcome event. What was causing the ST depression in the other 72% of high risk patients? I suspect that the majority of those ST segment depression cases are due to ACS events that stabilise following effective in-hospital treatment and that do not subsequently evolve into AMI events, and that only a minority of those cases are due to other etiological causes of ST segment depression eg. LVH, digoxin effect, PE. As far as I know, PRETestConsult.com has not analysed the EKGs of those patients who did not meet their arbitrary definition of an ACS event and we therefore do not know how many of those 72% group patients (50 out of 61 patients) actually had an ACS event that subsequently stabilised with therapy. I personally think that all high probability (>60% predicted probability) ACS patients require hospital admission and expeditious anti-ACS therapy and I do not think that it is useful to only make risk predictions about the future likelihood of a serious ACS event -- because I believe that we cannot predict which particular ACS patients (among the group of all suspected ACS patients) will definitely have an adverse event in the near-future (MI or death or urgent revascularisation). PRETestConsult.com's ACS prediction tool does not attempt to identify all ACS patients, and I therefore conclude that its ACS prediction values have no "real life" value in helping an EP decide whether action number 2 (expeditious institution of anti-ACS treatment, expeditious cardiology consultation, and expeditious admission to a cardiac ICU) is appropriate for a suspected ACS patient in an ED setting.
Is PRETestConsult.com's use of presentation cardiac marker results to generate a posttest probability of ACS scientifically valid and clinically useful?
Re-consider the scenario of clinical case scenario number 1 patient.
Clinical case scenario number 1: A 69 year old white male presents to the ED with chest pain. The chest pain has been present for 30 minutes and consists of a poorly localised heaviness across the anterior chest, plus some aching in both arms and jaw. The patient has a known history of CHD and diabetes. He is a non-smoker. Cholesterol level unknown. Vital signs are normal. There is no diaphoresis or chest pain reproducible by palpation. EKG shows ST depression (see EKG sample number 1).
PRETestConsult.com's ACS risk prediction tool states that this patient has a 18.2% pretest probability of having an ACS event. What is the ACS posttest probability if the presentation serum troponin level is negative (or positive).
If you use PRETestConsult.com's ACS risk module, you will obtain the following ACS posttest probability values for a serum troponin I test result.
Negative presentation serum troponin I result = 10.79% ACS posttest probability.
Positive presentation serum troponin I result = 60.63% ACS posttest probability.
Rough copy of PRETestConsult.com's screen appearance
AGE Gender Male: Race History of coronary artery disease Yes Chest pain, reproduced by palpation No Diaphoresis No ECG Normal EKG ST depression Value Is there T wave inversion more negative than -0.5 mm? No Current Smoker No Hypertension No Diabetes Yes Total Serum Cholesterol unknown.
ACS Pre Test Probability
|
ACS Post Test Probabilities
| Negative | Positive | |
| presentation CKMB (5 ng/mL) | 11.32% | 70.76% |
| presentation troponin I (0.4 ng/mL) | 10.79% | 60.63% |
| presentation troponin T (0.1 ng/mL) | 12.34% | 56.9% |
| presentation myoglobin (70 ng/mL) | 11.66% | 54.75% |
| presentation myoglobin and troponin | 5.21% | 50.3% |
How do you interpret these ACS posttest probability results? How do they influence your medical decision-making? As an EP, are you more likely to institute anti-ACS therapy in the ED for this particular patient if he has a positive presentation serum troponin result compared to a negative presentation serum troponin result?
I cannot fathom how the provision of these posttest ACS probability values helps an EP make better clinical decisions from the perspective of appropriately binning suspected ACS patients, and treating them accordingly. More importantly, I think that the actual posttest probability values are scientifically invalid and therefore clinically meaningless. Consider my argument.
The 18.2% ACS pretest probability prediction figure includes only those ACS patients who develop a serious complication within 45 days (MI or death or revascularisation) and it does not include all suspected ACS patients. Therefore, I think that it makes no sense to apply a subsequent diagnostic test result (which has a specific LR- or LR+ value) to generate a new posttest ACS probability figure if the precise definition of ACS is different for the cardiac marker studies (which starts off with all suspected AMI ACS-subset cases to diagnose only AMI ACS-subset cases) than PRETestConsult.com's arbitrary definition of ACS pretest probability (which seemingly starts off with only death/revascularisation/AMI ACS-subset cases to subsequently diagnose only AMI ACS-subset cases). When I asked Jeff Kline how he obtained the LR-/LR+ figures for the cardiac marker test results, he provided no specific details and he simply referred me to the meta-analysis paper co-authored by Lau [4].
If one examines the meta-analysis paper [4], one will note that it is merely a meta-analysis of presentation and serial cardiac marker studies and it does not provide scientific legitimacy for any particular sensitivity/specificity value. In fact, the authors of the paper suggest that the results of presentation cardiac marker studies cannot be accurately interpreted because they are not time-standardised (the presentation cardiac marker results vary considerably because they were taken at different times following chest pain onset) or threshold-standardised (different test criteria).
Here is the table, from that paper, for the test performance of seven
studies of presentation serum troponin I in the diagnosis of AMI.
It is important to note that there are no studies for the diagnosis of
ACS (which includes unstable angina patients and AMI patients).
|
|
|||||||
|
Study
|
Population Category
|
Prevalence of AMI
(%) |
Subjects Evaluated
(Enrolled) |
Test Criteria
|
Sensitivity
(%) |
Specificity
(%) |
Study Quality
|
|
Apple et al, 1995[39]
|
IV
|
6
|
98 (no data)
|
Troponin I >3.1 ng/mL
|
100
|
91
|
C
|
|
Mair et al, 1996[5]
|
II
|
39
|
101 (no data)
|
Troponin I >0.1 ng/mL
|
23
|
95
|
C
|
|
D'Costa et al, 1997[27]
|
II*
|
20
|
316 (no data)
|
Troponin I >1.0 ng/mL
|
79
|
No data*
|
C
|
|
Hamm et al, 1997[56]
|
III
|
6
|
773 (no data)
|
Troponin I >0.1 ng/mL
|
66
|
89
|
A
|
|
Laurino et al, 1997[9]
|
I*
|
22
|
115 (no data)
|
Troponin I >0.6 ng/mL
|
32
|
No data*
|
C
|
|
Tucker et al, 1997[21]
|
IV
|
15
|
177 (no data)
|
Troponin I >0.6 ng/mL
|
4
|
98
|
B
|
|
Kontos et al, 1999[57]
|
IV*
 |
9
|
620 (721)
|
Troponin I >2.0 ng/mL
|
39
|
No data*
|
C
|
|
For definition of population category and study quality, see legend for Table 1. * Insufficient data. Not included in meta-analysis.
|
|||||||
Note the large variation in sensitivity values for the different studies. There are two major reasons why they vary so greatly -- they have different "test criteria" threshold levels and they were not time standardised.
Which sensitivity/specificity values did PRETestConsult.com choose to use to generate its posttest ACS probability values? That specific information is not available on its website. Also, note that these published sensitivity/specificity values are only applicable to the diagnosis of AMI, and not ACS (which also includes serum troponin-negative unstable angina ACS patients). Therefore, I have to presume that the posttest probability values generated by PRETestConsult.com's ACS risk module must specifically refer to the diagnosis of AMI, and not ACS, although that fact is not definitively deducible from viewing the ACS risk module's screen (which only states "ACS posttest probability").
Consider this complex issue in greater detail.
When a Bayesian clinician uses EBM information from the medical literature and applies it to "real life" patients in clinical practice, he should be using the following simple Bayesian equation as the logical basis for his Bayesian thinking.
Pretest probability of disease "X" x (LR+ of a diagnostic test for disease "X") = Posttest probability of disease "X".
[* I have used the term "probability", which most people can readily understand, rather than "odds" (which is actually the correct method of mathematically solving a Bayesian equation), because I am only attempting to demonstrate a simple principle, and it will not alter the logic of my reasoning if you prefer to substitute the word "odds" whenever I use the word "probability" in the following series of illustrative examples]
It is very critical that the diagnostic test generating the LR+ value specifically apply to the specific disease entity under consideration -- disease "X" -- and not to a variant/subset of that disease. Consider the following illustrative example.
If an EP evaluates a patient with RLQ abdominal pain in the ED and decides that the patient has a 40% pretest probability of appendicitis, then he may decide to order an appendiceal CT to make a more definitive diagnosis.
Then the following simple Bayesian equation applies to his "rule-in" mental thought process.
Pretest probability of appendicitis x (LR+ of a positive CT scan for appendicitis) = Posttest probability of appendicitis.
For that Bayesian equation to be perfectly logical, it is important that the sensitivity/specificity of the CT scan (which generates the LR+ value of the positive CT scan result) be derived from research studies that are seeking the same specific diagnosis (appendicitis) using similar patients.
If the EP specifically wants to know what's the posttest probability of a ruptured appendix (rather than appendicitis), then he cannot logically think in the following manner.
Pretest probability of appendicitis x (LR+ of a positive CT scan for appendicitis) = Posttest probability of a ruptured appendix.
The above Bayesian equation is logically imperfect because there is discordance between the two diseases -- "appendicitis" for the pretest probability, and a "ruptured appendix" (which is only a subset of all appendicitis cases) for the posttest probability. There is also discordance between the diagnostic test's LR value (which applies to all appendicitis cases) and the posttest disease probability (which applies to only a subset of appendicitis cases).
The following Bayesian equation is also logically imperfect.
Pretest probability of a ruptured appendix x (LR+ of a positive CT scan for appendicitis) = Posttest probability of a ruptured appendix.
In this specific instance, the specific disease sub-entities under consideration are identical, but the sensitivity/specificity values for the CT scan (which are used to generate the LR+ value of the positive CT scan) applies to appendicitis (all cases), and not specifically to a ruptured appendix (a subset of all appendicitis cases). If a clinical researcher uses a different clinical endpoint (ruptured appendix rather than appendicitis) in his clinical research appendiceal CT scan study, then his study will generate different sensitivity/specificity values that will result in a different LR+ value for a positive CT scan, which will then be applicable to clinical practice if the clinician is specifically thinking along those lines.
Therefore, if a clinician wants to estimate the posttest probability of a ruptured appendix for a particular patient in clinical practice, then he should think along the following Bayesian lines.
Pretest probability of a ruptured appendix x (LR+ of a positive CT scan for appendiceal rupture) = Posttest probability of appendiceal rupture.
In this case, there is concordance between the three elements in the Bayesian calculation (pretest probability of a specific disease sub-entity, LR of a diagnostic test for that specific disease sub-entity, and posttest probability of that specific disease sub-entity). [* As an aside, the solved Bayesian equation is only scientifically valid if the clinician can find a sufficient number of "good" quality research studies using that specific endpoint to generate a scientifically valid LR+ value]
How does this pattern of Bayesian thinking apply to the use of serum troponin testing to rule-in AMI in chest pain patients?
Consider clinical scenario number 1 patient again (see above).
If a "gestalt" EP thinks that clinical scenario number 1 patient has a 60% pretest probability of ACS, then his Bayesian thinking is logically imperfect if he thinks along the following lines -- after using the sensitivity/specificity values of a presentation serum troponin I test from some/all of those seven studies listed in the table to generate a specific (and hopefully scientifically valid) LR+ value.
60% pretest probability of ACS x (LR+ of a positive serum troponin I test to diagnose AMI) = Posttest probability of ACS.
In this instance, the LR+ values were derived from studies used to diagnose AMI, and not ACS. AMI is a sub-entity of ACS, and AMI and ACS are not precisely the same disease entities. Therefore, there is discordance between the disease entities under consideration (ACS) and the diagnostic test's LR+ value (which is only derived from AMI studies).
Using the same logic, the following example of Bayesian thinking is also logically imperfect -- because there is discordance between the pretest disease entity (ACS) and the posttest disease entity (AMI), and also discordance between the diagnostic test's LR + value (which only applies to AMI) and the pretest disease entity (ACS).
60% pretest probability of ACS x (LR+ of a positive serum troponin I test to diagnose AMI) = Posttest probability of an AMI.
The only perfectly logical line of Bayesian thinking in this context would be as follows.
Pretest probability of an AMI x (LR+ of a positive serum troponin I test to diagnose AMI) = Posttest probability of an AMI.
In other words, the EP needs to change his thinking-pattern when he orders, and interprets, a presentation serum troponin test result, and he should estimate the patient's pretest probability of AMI (and not ACS) prior to using the serum troponin test result to generate a posttest probability of AMI estimation (presuming that he has a specific clinical reason to primarily focus on diagnosing AMI rather than ACS).
Because there are no research studies using serum troponin testing to diagnose/exclude ACS, I think that it is not rational to think along the following lines.
Pretest probability of ACS x (LR+ of a serum troponin test for ACS) = Posttest probability of ACS.
Therefore, if I am correct, how can the Bayesian thinking underlying PRETestConsult.com's use of a Bayesian equation be logical when it seems to be based on the following Bayesian equation?
Pretest probability of an ACS sub-entity (only those ACS cases that will result in an AMI or revascularisation or death within 45 days) x (LR+ of a positive serum troponin test for AMI) = Posttest probability of ACS (???? I don't really know which ACS sub-entity is actually under consideration).
Although I am dismayed by the logically imperfect Bayesian thinking exemplified by the above Bayesian equation, I also think that there are other important reasons to reject PRETestConsult.com's use of serum troponin testing to generate posttest ACS probability values.
A major reason why I think that PRETestConsult.com's ACS posttest probability values are clinically meaningless is because the presentation serum troponin I results (used to generate the LR+ value) are not time-standardised.
Clinical scenario number 1 patient only had chest pain for 30 minutes, and the sensitivity of a serum troponin I test at 30 minutes is so low as to have no/little utility as a rule-out AMI diagnostic test for that patient (sensitivity is probably between 0-20%). I therefore think that it is totally unscientific to apply an arbitrary single set of presentation serum troponin I LR-/LR+ values to a particular patient if that patient has chest pain of significantly shorter duration as compared to the majority of patients in the research study generating the LR values.
Another factor that PRETestConsult.com did not seemingly consider is the degree of elevation of the presentation serum troponon I test. For example, if a single presentation serum troponin I test is just above the threshold value of 0.4ng/ml (eg. 0.6 ng/ml) four hours after chest pain onset in a low probability ACS patient, then that positive serum troponin test result is much more likely to be a false-positive test result (rather than a true-positive test result) compared to a presentation serum troponin I test result of 10ng/ml at four hours.
I think that all EPs, who choose to use a single set of presentation serum troponin results to modify their estimated pretest probability of AMI (not ACS)value in order to generate a revised posttest probability of AMI value, must always consider two critical elements -- the absolute degree of elevation of the serum troponin I and the time that has passed since chest pain onset. I think that a "cookbook" approach using a single set of sensitivity/specificity values for all suspected AMI patients (without taking time since chest pain onset into precise consideration) is totally unscientific, and I think that it will produce clinically meaningless results.
Finally, does it really matter if clinical scenario number 1 patient's presentation serum troponin test is negative? Do you think that an EP should withold anti-ACS therapy in the ED (or mandatory admission to a cardiac ICU) because a presentation serum troponin test is negative -- if the chest pain patient is having ongoing chest pain associated with a frankly ischemic-appearing EKG? As an EP, do you treat ACS patients in the ED (who have ongoing chest pain and overtly ischemic ST segment depression) differently if you predict that they are somewhat more likely (rather than less likely) to develop an AMI-type ACS event rather than a non-AMI-type ACS event?
Conclusion:
Although I have retired from clinical practice, I am still intrigued by the issue of what represents "good quality" clinical decision-making. In fact, I am presently more interested in the methodology of how clinicians logically reason their way through clinical problems than I was when I was actively practicing emergency medicine. The clinical problem of ACS is a very common and very important problem that EPs need to solve. First of all, an EP needs to ensure that he does not miss the diagnosis of ACS because the inadvertent ED-discharge of an ACS patient (who has a substantial risk of developing a MI and/or dying) may result in a personal tragedy for the patient and a potential medical malpractice nightmare for the physician. Secondly, an EP needs to optimise his ED-triage decisions to ensure that high probability ACS patients (who are very likely to be having an acute coronary ischemic event) are triaged to a cardiac ICU rather than a chest pain observation unit. Thirdly, I believe that an EP should institute anti-ACS therapy in the ED setting if his clinical suspicion of ACS exceeds a certain threshold value (a threshold value that is "somewhere" between a 60-100% probability of ACS). I remain very interested in the best method of achieving those three goals, and I have spent endless hours thinking about the best method of optimising those three clinical decisions.
A few years ago, I wrote a lengthy and rambling essay for the soapbox section of my website called "Cardiac risk stratification and clinical reasoning: Can EBM improve clinical judgements!" [5]. In that essay, I discussed the pitfalls and complexities that plague an EPs attempt to diagnose ACS in an ED setting. I attempted to demonstrate how difficult it is for an EP to take EBM evidence (LR+/LR- values of different diagnostic tests) from the medical literature and usefully apply it to "real life" patients in clinical practice. At the end of my essay, I lamented the fact that there was no useful scientific research on pretest probability of ACS estimations, because I felt that a Bayesian clinician would never be able to accurately determine the final postest probability of ACS if he only had objective (scientific) evidence regarding the LR values of ACS diagnostic tests, but didn't know how to objectively (scientifically) determine the pretest probability of ACS. As a "gestalt" clinician I was always cognizant of the fact that my "subjective" pretest probability of ACS estimations were unscientific, and I was always looking for more objective (scientific) methods of determining the pretest probability of ACS. I therefore welcomed the fact that clinical researchers, like Jeff Kline, were researching the problem of how best to accurately predict the "pretest probability of ACS" and I was hoping that they would produce a clinically useful "pretest probability of ACS" product for chest pain patients with suspected ACS.
It must be obvious from my detailed critical review of PRETestConsult.com's ACS risk module, that I do not think that they have produced a scientifically valid and clinically useful product. I have evaluated their product from the perspective of whether their ACS risk module could help an EP accurately achieve three goals:-- i) determine which chest pain patients are at such a low risk of ACS that they can be safely discharged from the ED; ii) determine which suspected ACS patients are at such a high risk of ACS that they should be admitted to a cardiac ICU rather than a chest pain observation unit; and iii) determine which ACS patients have such a high probability of an ACS event that aggressive anti-ACS therapy should be instituted in the ED.
I personally think that PRETestConsultcom's ACS risk module is not clinically useful for ruling-out ACS because it cannot generate pretest probability of ACS predictions that are below the target threshold (<2% predicted risk of ACS) for the major target population of chest pain patients -- white male patients >40 years of age. What is the clinical value of a "rule-out" clinical prediction tool if it cannot generate a ACS probability prediction that is below an admision-to-hospital threshold for the majority of "problematic" chest pain patients? I also still think that an experienced "gestalt" clinician can outperform any ACS clinical prediction tool in borderline cases because an experienced clinician is capable of synergistically combining much more clinical information (eg. type of chest pain, exact location of chest pain, duration and persistence of the chest pain, presence of associated pain radiation to the arms/jaw, presence of a third heart sound or basal rales, actual pattern and specific appearance of the depressed ST segments) to generate a more nuanced clinical prediction than that provided by a clinical prediction tool that only utilises limited information of unrefined quality (eg. all ST segment depression >0.5mm cases, rather than a nuanced EKG interpretation that is based on the specific appearance-pattern-location of the ST segment depression).
Also, although PRETestConsult.com claims to have a sample population of 15,000 chest pain patients in its data base, I have noted that the actual sample number for any particular patient profile is very small. Take, as an example, the situation of a 60 year old black male, who has a known history of CHD. Can you really rely on the scientific validity of PRETestConsult.com's attribute matching profiling methodology if you discover that the ACS risk module's ACS prediction value of 22% is based on only 2 ACS outcomes in 9 matched patients?
The second issue relates to the usefulness of PRETestConsult.com's ACS prediction values as a "rule-in ACS" tool. From an EPs standpoint, he doesn't need to be absolutely certain that a suspected ACS patient is experiencing an ACS event to decide that the patient should be triaged to a cardiac ICU (rather than a chest pain observation unit) and to decide that the patient should receive anti-ACS therapy while he is still in the ED setting. He only has to be "relatively" certain (eg. >60%, or >70%, or >80% certain). Although PRETestConsult.com's ACS risk module generates an ACS prediction value, the actual ACS prediction value only refers to a subset of ACS patients (those who die or have an AMI or have a revascularisation procedure within 45 days of ED presentation). If a "gestalt" EP suspects that a chest pain patient has a 70% probability of having an ACS event (based on a combination of clinical and EKG criteria) and PRETestConsult.com's ACS risk module predicts a 45 day serious-ACS risk of 20%, how should the EP utilise those different prediction values to optimally manage the patient in the ED setting? It may/may not be clinically useful to be able to inform the patient that he has a 20% risk of a serious adverse event within the next 45 days (MI or death or a need for urgent revascularisation). However, I think that it is definitely useful to be able to accurately inform the patient that he has a 70% predicted risk of having an ACS event in the present or near-future, and that the 70% predicted value exceeds an officially recommended threshold value for cardiac ICU admission (+ anti-ACS therapy in the ED while he is awaiting admission to the cardiac ICU). PRETestConsult.com's ACS risk module does not attempt to achieve that latter goal, and I think that an imperfect "gestalt" ACS prediction judgment may prove to be a better "real life" clinical guide even though "gestalt" clinical judgements cannot be scientifically validated.
Regarding PRETestConsult.com's use of presentation cardiac marker results to modify the pretest ACS probability predictions to generate a posttest ACS probability value, I think that it is the weakest part of the ACS risk module (because it is scientifically invalid and of questionable clinical utility) and I think that it should never be used by EPs in clinical practice.
Finally, I am very cognizant of the fact that I have many 'a priori' biases and strong personal opinions. I am consequently very interested in receiving positive or negative feedback from readers, because the feedback may induce me to modify my opinions. I therefore invite interested readers to send me their personal viewpoints. Please note that I am also very willing to post any positive/negative feedback comments in the commentary section below.
Jeff Mann.
Retired physician.
September 2004.
jmannemg@earthlink.net
References:
1. From Clinical case number 2 -- Chest pain - Acute Coronary Syndrome -- from Studymaker: "Use of pretest probability to reduce unneccesary test ordering in the Emergency Department"
http://www.studymaker.com/cme/prview.php?book=PTP1&StepNum=9
http://www.studymaker.com/cme/getrev.php?zform=PTP1&book=PTP1 for registration.
A more detailed explanation of how to calculate the test threshold for ACS (slide show consisting of 21 slides) is available at pretestconsult.com's official website, in the download section.
http://www.pretestconsult.com/site/downloads.php
Click on the link -- View the ""Pretest Probability Primer" by Jeffrey Kline.
2. From section 2: Methods of pretest probability assessment from the Studymaker: "Use of pretest probability to reduce unnecessary test ordering in the Emergency Department".
http://www.studymaker.com/cme/prview.php?book=PTP1&StepNum=4
http://www.studymaker.com/cme/getrev.php?zform=PTP1&book=PTP1 for registration.
3. PRETestConsult.com -- http://www.pretestconsult.com/site/home.php -- see trial software.
4. Balk EM. Ioannidis JP. Salem D. Chew PW. Lau J. Accuracy of biomarkers to diagnose acute cardiac ischemia in the emergency department: a meta-analysis. Annals of Emergency Medicine. 37(5):478-94, May 2001.
5. Mann J. Cardiac risk stratification and clinical reasoning: Can EBM improve clinical judgements!
Available in the soapbox section of my website at http://www.homestead.com/emguidemaps/JeffMannEMguidemaps.html (use a google search engine "Jeff Mann EM guidemaps" if the link doesn't work).
Commentary section:
Insightful comments by readers will be included in this section.