Reporting and Methods in Clinical Prediction Research: A Systematic Review

English version České info

Background:
We investigated the reporting and methods of prediction studies, focusing on aims, designs, participant selection, outcomes, predictors, statistical power, statistical methods, and predictive performance measures.

Methods and Findings:
We used a full hand search to identify all prediction studies published in 2008 in six high impact general medical journals. We developed a comprehensive item list to systematically score conduct and reporting of the studies, based on recent recommendations for prediction research. Two reviewers independently scored the studies. We retrieved 71 papers for full text review: 51 were predictor finding studies, 14 were prediction model development studies, three addressed an external validation of a previously developed model, and three reported on a model's impact on participant outcome. Study design was unclear in 15% of studies, and a prospective cohort was used in most studies (60%). Descriptions of the participants and definitions of predictor and outcome were generally good. Despite many recommendations against doing so, continuous predictors were often dichotomized (32% of studies). The number of events per predictor as a measure of statistical power could not be determined in 67% of the studies; of the remainder, 53% had fewer than the commonly recommended value of ten events per predictor. Methods for a priori selection of candidate predictors were described in most studies (68%). A substantial number of studies relied on a p-value cut-off of p<0.05 to select predictors in the multivariable analyses (29%). Predictive model performance measures, i.e., calibration and discrimination, were reported in 12% and 27% of studies, respectively.

Conclusions:
The majority of prediction studies in high impact journals do not follow current methodological recommendations, limiting their reliability and applicability.

: Please see later in the article for the Editors' Summary

Vyšlo v časopise: Reporting and Methods in Clinical Prediction Research: A Systematic Review. PLoS Med 9(5): e32767. doi:10.1371/journal.pmed.1001221
Kategorie: Research Article
prolekare.web.journal.doi_sk: https://doi.org/10.1371/journal.pmed.1001221

Souhrn

Zdroje

1. AltmanDGRileyRD 2005 Primer: an evidence-based approach to prognostic markers. Nat Clin Pract Oncol 2 466 472

2. AltmanDG 2007 Prognostic models: a methodological framework and review of models for breast cancer. LymanGHBursteinHJ Breast cancer. Translational therapeutic strategies New York New York Informa Healthcare 11 26

3. AltmanDGLymanGH 1998 Methodological challenges in the evaluation of prognostic factors in breast cancer. Breast Cancer Res Treat 52 289 303

4. McShaneLMAltmanDGSauerbreiWTaubeSEGionM 2005 Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst 97 1180 1184

5. RothwellPM 2008 Prognostic models. Pract Neurol 8 242 253

6. MoonsKGRoystonPVergouweYGrobbeeDEAltmanDG 2009 Prognosis and prognostic research: what, why, and how? BMJ 338 b375

7. RoystonPMoonsKGAltmanDGVergouweY 2009 Prognosis and prognostic research: developing a prognostic model. BMJ 338 b604

8. AltmanDGVergouweYRoystonPMoonsKG 2009 Prognosis and prognostic research: validating a prognostic model. BMJ 338 b605

9. MoonsKGAltmanDGVergouweYRoystonP 2009 Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338 b606

10. SteyerbergEW 2009 Clinical prediction models: a practical approach to development, validation, and updating New York Springer

11. GrobbeeDEHoesAW 2007 Clinical epidemiology: principles, methods, and applications for clinical research Sudbury (Massachusetts) Jones and Bartlett Publishers

12. HarrellFE 2001 Regression modeling strategies with applications to linear models, logistic regression and survival analysis New York Springer Verlag

13. HarrellFEJrLeeKLMarkDB 1996 Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15 361 387

14. GrobbeeDE 2004 Epidemiology in the right direction: the importance of descriptive research. Eur J Epidemiol 19 741 744

15. LaupacisASekarNStiellIG 1997 Clinical prediction rules. a review and suggested modifications of methodological standards. JAMA 277 488 494

16. McGinnTGGuyattGHWyerPCNaylorCDStiellIG 2000 Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 284 79 84

17. LijmerJGMolBWHeisterkampSBonselGJPrinsMH 1999 Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 282 1061 1066

18. RutjesAWReitsmaJBVandenbrouckeJPGlasASBossuytPM 2005 Case-control and two-gate designs in diagnostic accuracy studies. Clin Chem 51 1335 1341

19. RutjesAWReitsmaJBDiNMSmidtNvan RijnJC 2006 Evidence of bias and variation in diagnostic accuracy studies. CMAJ 174 469 476

20. MallettSRoystonPWatersRDuttonSAltmanDG 2010 Reporting performance of prognostic models in cancer: a review. BMC Med 8 21

21. MallettSRoystonPDuttonSWatersRAltmanDG 2010 Reporting methods in studies developing prognostic models in cancer: a review. BMC Med 8 20

22. ConcatoJFeinsteinARHolfordTR 1993 The risk of determining risk with multivariable models. Ann Intern Med 118 201 210

23. DondersARvan der HeijdenGJStijnenTMoonsKG 2006 Review: a gentle introduction to imputation of missing values. J Clin Epidemiol 59 1087 1091

24. GreenlandSFinkleWD 1995 A critical look at methods for handling missing covariates in epidemiologic regression analyses. Am J Epidemiol 142 1255 1264

25. PeduzziPConcatoJKemperEHolfordTRFeinsteinAR 1996 A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 49 1373 1379

26. PeduzziPConcatoJFeinsteinARHolfordTR 1995 Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. J Clin Epidemiol 48 1503 1510

27. HaydenJACotePBombardierC 2006 Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med 144 427 437

28. HaydenJACotePSteenstraIABombardierC 2008 Identifying phases of investigation helps planning, appraising, and applying the results of explanatory prognosis studies. J Clin Epidemiol 61 552 560

29. WassonJHSoxHCNeffRKGoldmanL 1985 Clinical prediction rules. Applications and methodological standards. N Engl J Med 313 793 799

30. SteyerbergEWEijkemansMJHabbemaJD 1999 Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. J Clin Epidemiol 52 935 942

31. SteyerbergEWBleekerSEMollHAGrobbeeDEMoonsKG 2003 Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol 56 441 447

32. BleekerSEMollHASteyerbergEWDondersARDerksen-LubsenG 2003 External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 56 826 832

33. OttenbacherKJOttenbacherHRToothLOstirGV 2004 A review of two journals found that articles using multivariable logistic regression frequently did not report commonly recommended assumptions. J Clin Epidemiol 57 1147 1152

34. MackinnonA 2010 The use and reporting of multiple imputation in medical research—a review. J Intern Med 268 586 593

35. MushkudianiNAHukkelhovenCWHernandezAVMurrayGDChoiSC 2008 A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol 61 331 343

36. PerelPEdwardsPWentzRRobertsI 2006 Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 6 38

37. LeushuisEvan der SteegJWSteuresPBossuytPMEijkemansMJ 2009 Prediction models in reproductive medicine: a critical appraisal. Hum Reprod Update 15 537 552

38. Von ElmEAltmanDGEggerMPocockSJGotzschePC 2007 The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 370 1453 1457

39. BossuytPMReitsmaJBBrunsDEGatsonisCAGlasziouPP 2003 Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. Ann Intern Med 138 40 44

40. WhitingPRutjesAWReitsmaJBBossuytPMKleijnenJ 2003 The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 3 25

41. MoherDHopewellSSchulzKFMontoriVGotzschePC 2010 CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 c869

42. SteyerbergEWVergouweYKeizerHJHabbemaJD 2001 Residual mass histology in testicular cancer: development and validation of a clinical prediction rule. Stat Med 20 3847 3859

43. SteyerbergEWHarrellFEJrBorsboomGJEijkemansMJVergouweY 2001 Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54 774 781

44. StiellIGWellsGA 1999 Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med 33 437 447

45. TollDBJanssenKJVergouweYMoonsKG 2008 Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 61 1085 1094

46. SteyerbergEWBorsboomGJvan HouwelingenHCEijkemansMJHabbemaJD 2004 Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 23 2567 2586

47. JanssenKJMoonsKGKalkmanCJGrobbeeDEVergouweY 2008 Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61 76 86

48. ReillyBMEvansAT 2006 Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144 201 209

49. BiesheuvelCJVergouweYOudegaRHoesAWGrobbeeDE 2008 Advantages of the nested case-control design in diagnostic research. BMC Med Res Methodol 8 48

50. RoystonPAltmanDGSauerbreiW 2006 Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 25 127 141

51. KirkhamJJDwanKMAltmanDGGambleCDoddS 2010 The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340 c365

52. TomlinsonGDetskyAS 2010 Composite end points in randomized trials: there is no free lunch. JAMA 303 267 268

53. LimEBrownAHelmyAMussaSAltmanDG 2008 Composite outcomes in cardiovascular research: a survey of randomized trials. Ann Intern Med 149 612 617

54. VittinghoffEMcCullochCE 2007 Relaxing the rule of ten events per variable in logistic and Cox regression. Am J Epidemiol 165 710 718

55. VergouweYSteyerbergEWEijkemansMJHabbemaJD 2005 Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58 475 483

56. SunGWShookTLKayGL 1996 Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 49 907 916

57. BurtonAAltmanDG 2004 Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines. Br J Cancer 91 4 8

58. GorelickMH 2006 Bias arising from missing data in predictive models. J Clin Epidemiol 59 1115 1123

59. MarshallAAltmanDGHolderRL 2010 Comparison of imputation methods for handling missing covariate data when fitting a Cox proportional hazards model: a resampling study. BMC Med Res Methodol 10 112

60. KnolMJJanssenKJDondersAREgbertsACHeerdinkER 2010 Unpredictable bias when using the missing indicator method or complete case analysis for missing confounder values: an empirical example. J Clin Epidemiol 63 728 736

61. WhiteIRThompsonSG 2005 Adjusting for partially missing baseline measurements in randomized trials. Stat Med 24 993 1007

62. MiettinenOS 1985 Theoretical epidemiology. Principles of occurrence research in medicine New York Wiley

63. VandenbrouckeJPvon ElmEAltmanDGGotzschePCMulrowCD 2007 Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med 4 e297 doi:10.1371/journal.pmed.0040297

64. AltmanDGRoystonP 2000 What do we mean by validating a prognostic model? Stat Med 19 453 473

65. JusticeACCovinskyKEBerlinJA 1999 Assessing the generalizability of prognostic information. Ann Intern Med 130 515 524

66. SauerbreiW 1999 The use of resampling methods to simplify regression models in medical statistics. Appl Stat 48 313 329

67. VergouweYSteyerbergEWEijkemansMJHabbemaJD 2002 Validity of prognostic models: when is a model clinically useful? Semin Urol Oncol 20 96 107

68. BachmannLMPuhanMAter RietGBossuytPM 2006 Sample sizes of studies on diagnostic accuracy: literature survey. BMJ 332 1127 1129

69. MallettSTimmerASauerbreiWAltmanDG 2010 Reporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines. Br J Cancer 102 173 180

70. Van HouwelingenJCLe CessieS 1990 Predictive value of statistical models. Stat Med 9 1303 1325

71. JanssensACIoannidisJPvan DuijnCMLittleJKhouryMJ 2011 Strengthening the reporting of Genetic Risk Prediction Studies: the GRIPS Statement. PLoS Med 8 e1000420 doi:10.1371/journal.pmed.1000420

72. PletcherMJBibbins-DomingoKLewisCEWeiGSSidneyS 2008 Prehypertension during young adulthood and coronary calcium later in life. Ann Intern Med 149 91 99

73. ImperialeTFGlowinskiEALin-CooperCLarkinGNRoggeJD 2008 Five-year risk of colorectal neoplasia after negative screening colonoscopy. N Engl J Med 359 1218 1224

74. Van VeenMSteyerbergEWRuigeMvan MeursAHRoukemaJ 2008 Manchester triage system in paediatric emergency care: prospective observational study. BMJ 337 a1501

75. SassonCHeggAJMacyMParkAKellermannA 2008 Prehospital termination of resuscitation in cases of refractory out-of-hospital cardiac arrest. JAMA 300 1432 1438