messori and colleagues are to be congratulated on their interesting report on the italian observational study of antithrombin iii ( at iii ) use in intensive care units . audits of this sort are difficult and time consuming to conduct .
observational studies are also difficult to analyse and interpret because of the heterogeneity of real - life patient populations , the lack of standardised treatment regimens , the lack of standardised indications for treatment , and the lack of predefined endpoints for assessing survival ( e.g. 28 day survival or in - hospital mortality ) . for these reasons
we take issue with messori and colleagues over the very strong conclusion they draw on the basis of their survey . in our opinion , this nonrandomised observational study is not able to provide scientific evidence to support the authors ' statements .
as pointed out by pocock and elbourne , observational studies have one crucial deficiency : the design is not an experimental one .
experimental design involves randomisation , the use of entry criteria , and the rigorous use of standard definitions of index medical conditions such as disseminated intravascular coagulation and septic shock .
none of these are to be found in the survey by messori and colleagues . in the absence of rigorous experimental methodology ,
it is not possible to be sure that the findings of an observational study are predictive of the results of a randomised , controlled trial ( rct ) .
the published finding that observational studies usually agree with rcts comes from studies in indications unrelated to sepsis and septic shock .
furthermore , it is a logical fallacy to suggest that agreement in one direction implies prediction in the other direction .
observational studies are not scientifically capable of proving or disproving any hypothesis . with this in mind
, it is interesting to note that one of the main observations of messori and colleagues ' survey ( n = 56 for sepsis ) is at variance with the results from a very much larger rct reported in the same indication by warren and colleagues ( n = 2341 for sepsis ) . whereas warren and colleagues ' rct found significantly lower mortality in a prespecified subgroup analysis of at iii - treated patients not receiving heparin ( n = 352 ) compared with placebo - treated patients not receiving heparin ( n = 346 ) , messori and colleagues ' survey observed an increase in mortality in patients not receiving heparin .
this casts serious doubt over the scientific value of a survey reporting outcomes in a treatment that was not randomised to patients , with only 56 patients with sepsis treated with at iii .
such a type of study is not scientifically or statistically capable of proving causality or supporting statements .
other crucial weaknesses of messori and colleagues ' survey lie in its observational nature and the lack of any standardisation of definitions or outcomes .
controlled studies usually indicate the 28 day mortality whereas observational studies indicate the hospital mortality , which normally is higher than the 28 day mortality .
in addition , hospital mortality may depend on local factors affecting clinical practice such as when to discharge patients from hospital , which varies among countries .
messori and colleagues also report the results of a meta - analysis of four studies of at iii in sepsis .
it is probably unwise , however , to place too much weight on meta - analyses of studies conducted in critical care settings because the spectacular failure of meta - analyses to predict the outcomes of subsequent large - scale rcts is well known .
furthemore , if confidence intervals overlap or p > 0.05 , this does not prove that there is no difference ; on the contrary , there is still a possibility that the observation suffers from a type ii error ( failure to detect a true difference due to inadequate sample size ) .
this is particularly likely to occur when the number of patients studied is small , as in the phase ii trials on at iii . in this context , it is misleading of messori and colleagues to use figure 1 of their paper to suggest , with reference to the confidence intervals , that there is no difference between the studies or that their observational study has a very similar result to the other studies cited . in the same way
, potentially misleading claims are made about the subgroup analysis of the effects of coadministered heparin . in the results section it
is suggested , erroneously , that no differences exist between results on the basis of p values alone .
surveys and audits are very important ways of documenting routine clinical practice and of confirming the implementation of guidelines based on the results of rcts . surveys and audits are also useful for generating hypotheses , but they are not a substitute for rcts when proving or disproving hypotheses .
messori and colleagues have raised interesting concerns but they have not provided answers to those concerns .
the author has participated as a clinical investigator in the kybersept trial and served as coordinator in a basic research program of the university of innsbruck on anti - inflammatory mechanisms of antithrombin , which was in part supported by a grant from aventis behring gmbh ( marburg , germany ) .
 observational studies usually agree with randomised , controlled trials . it is a logical fallacy , however , to suggest that agreement in one direction implies prediction in the other direction . 
 observational studies are not scientifically capable of proving or disproving hypotheses such as the efficacy and safety of the treatment of severe sepsis with antithrombin . 
 observational studies are difficult to analyse and interpret because of the heterogeneity of real - life patient populations , the lack of standardised treatment regimens , the lack of standardised indications for treatment , and the lack of predefined endpoints . 