Data Analysis Pitfalls

Sandro Sperandei   (2004-08-14 23:38)  Federal University of Rio de Janeiro

Only recently have I heard about this journal and it took me by surprise the excellent quality of the presented articles. I’m writing this comment to try to contribute to increase this quality even more.

The authors of this paper made an extensive and, apparently, carefull work to compare two common field methods used in body composition analysis. However, some mistakes at data analysis jeopardize the presented conclusions.

The results showed significant differences only in some age/sex groups. To the total subject pool, there was no difference. Such results could lead to believe that one can use both methods interchangeably in some age/sex groups or with the total subject pool, and can’t with other age/sex groups. But significant differences only show the existence of systematic differences between groups, telling us nothing about the random differences between methods. Additionally, using nine t-tests in a row requires some kind of correction (i.e., Bonferroni correction), since the type-I error probability becomes very high (close to 50%)[1].

In random error analysis, the authors selected the limits of agreement technique [2]. That was a great choice! However, the graph presented at figure 1 describes only the mean difference ¿ one standard deviation (SD), while Bland-Altman plots uses two SD, representing confidence intervals to 95% of the intermethods differences.

Surprisingly, the authors used Pearson correlation coefficient to evaluate agreement between methods, while the same work of Altman and Bland [2], cited in the article, exhaustively demonstrates the inadequacy of this coefficient in evaluating intermethod agreement.

Such problems in data analysis probably led to the conclusion that the results of leg-to-leg BIA were “comparable to those obtained using skinfold”. Indeed, analysis of figure 1 shows that differences in percentual body fat predicted by different methods present a variation of approximately ¿10%, even if only one SD (¿68% of results) is used. This is a considerable difference! If we use two SD, differences increase to values above 20%.

Based on the facts exposed, I recommend caution in substituting one method by the other, since the intermethod difference could be greater than the real percentual body fat. Since the authors do not use any “gold standard” method (i.e., hydrostatic weighting), it is not possible to choose the best method to use.

References

1. Shaffer JP: Modified sequentially rejective multiple test procedures. Journal American Statistical Association 1986, 81:826-831.

2. Altman DG and Bland JM: Measurement in medicine: The analysis of method comparison studies. Statistician 1983, 32:307-317.

Sandro Sperandei

Federal University of Rio de Janeiro

Program of Biomedical Engeneering (COPPE)

Rio de Janeiro/RJ - Brazil

Competing interests

None

top