Neuroimaging modality fusion in Alzheimer’s classification using convolutional neural networks

Autoři: Arjun Punjabi aff001;  Adam Martersteck aff002;  Yanran Wang aff001;  Todd B. Parrish aff002;  Aggelos K. Katsaggelos aff001
Působiště autorů: Department of Electrical Engineering and Computer Science/McCormick School of Engineering, Northwestern University, Evanston, Illinois, United States of America aff001;  Department of Radiology/Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States of America aff002;  Mesulam Cognitive Neurology and Alzheimer’s Disease Center, Northwestern University, Chicago, Illinois, United States of America aff003
Vyšlo v časopise: PLoS ONE 14(12)
Kategorie: Research Article
prolekare.web.journal.doi_sk: 10.1371/journal.pone.0225759


Automated methods for Alzheimer’s disease (AD) classification have the potential for great clinical benefits and may provide insight for combating the disease. Machine learning, and more specifically deep neural networks, have been shown to have great efficacy in this domain. These algorithms often use neurological imaging data such as MRI and FDG PET, but a comprehensive and balanced comparison of the MRI and amyloid PET modalities has not been performed. In order to accurately determine the relative strength of each imaging variant, this work performs a comparison study in the context of Alzheimer’s dementia classification using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset with identical neural network architectures. Furthermore, this work analyzes the benefits of using both modalities in a fusion setting and discusses how these data types may be leveraged in future AD studies using deep learning.

Klíčová slova:

Alzheimer's disease – Longitudinal studies – Magnetic resonance imaging – Neural networks – Neuroimaging – Positron emission tomography – Support vector machines


1. Hebert L, Scherr P, Bienias J, Bennett D, Evans D. State-specific projections through 2025 of Alzheimer disease prevalence. Neurology. 2004;62(9):1645–1645. doi: 10.1212/01.wnl.0000123018.01306.10 15136705

2. Cuingnet R, Gerardin E, Tessieras J, Auzias G, Lehéricy S, Habert MO, et al. Automatic classification of patients with Alzheimer’s disease from structural MRI: a comparison of ten methods using the ADNI database. NeuroImage. 2011;56(2):766–781. doi: 10.1016/j.neuroimage.2010.06.013 20542124

3. Ashburner J. A fast diffeomorphic image registration algorithm. NeuroImage. 2007;38(1):95–113. doi: 10.1016/j.neuroimage.2007.07.007 17761438

4. Klöppel S, Stonnington CM, Chu C, Draganski B, Scahill RI, Rohrer JD, et al. Automatic classification of MR scans in Alzheimer’s disease. Brain. 2008;131(3):681–689. doi: 10.1093/brain/awm319 18202106

5. Liu M, Zhang D, Adeli E, Shen D. Inherent Structure-Based Multiview Learning With Multitemplate Feature Representation for Alzheimer’s Disease Diagnosis. IEEE Transactions on Biomedical Engineering. 2016;63(7):1473–1482. doi: 10.1109/TBME.2015.2496233 26540666

6. Gerardin E, Chételat G, Chupin M, Cuingnet R, Desgranges B, Kim HS, et al. Multidimensional classification of hippocampal shape features discriminates Alzheimer’s disease and mild cognitive impairment from normal aging. Neuroimage. 2009;47(4):1476–1486. doi: 10.1016/j.neuroimage.2009.05.036 19463957

7. Sabuncu MR, Konukoglu E, Initiative ADN, et al. Clinical prediction from structural brain MRI scans: a large-scale empirical study. Neuroinformatics. 2015;13(1):31–46. doi: 10.1007/s12021-014-9238-1 25048627

8. Zu C, Jie B, Liu M, Chen S, Shen D, Zhang D, et al. Label-aligned multi-task feature learning for multimodal classification of Alzheimer’s disease and mild cognitive impairment. Brain imaging and behavior. 2016;10(4):1148–1159. doi: 10.1007/s11682-015-9480-7 26572145

9. Zhu X, Suk HI, Shen D. A novel matrix-similarity based loss function for joint regression and classification in AD diagnosis. NeuroImage. 2014;100:91–105. doi: 10.1016/j.neuroimage.2014.05.078 24911377

10. Suk HI, Shen D. Deep learning-based feature representation for AD/MCI classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2013. p. 583–590.

11. Suk HI, Lee SW, Shen D, Initiative ADN, et al. Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis. NeuroImage. 2014;101:569–582. doi: 10.1016/j.neuroimage.2014.06.077 25042445

12. Li F, Tran L, Thung KH, Ji S, Shen D, Li J. A robust deep model for improved classification of AD/MCI patients. IEEE journal of biomedical and health informatics. 2015;19(5):1610–1616. doi: 10.1109/JBHI.2015.2429556 25955998

13. Yang X, Wu Q, Hong D, Zou J. Spatial regularization for neural network and application in Alzheimer’s disease classification. In: Future Technologies Conference (FTC). IEEE; 2016. p. 831–837.

14. Sarraf S, Tofighi G. Classification of Alzheimer’s disease using fMRI data and deep learning convolutional neural networks. arXiv preprint arXiv:160308631. 2016;.

15. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998;86(11):2278–2324. doi: 10.1109/5.726791

16. Sarraf S, Tofighi G, et al. DeepAD: Alzheimer′ s Disease Classification via Deep Convolutional Neural Networks using MRI and fMRI. bioRxiv. 2016; p. 070441.

17. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2015. p. 1–9.

18. Li R, Zhang W, Suk HI, Wang L, Li J, Shen D, et al. Deep learning based imaging data completion for improved brain disease diagnosis. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer; 2014. p. 305–312.

19. Hinton GE, Zemel RS. Autoencoders, minimum description length and Helmholtz free energy. In: Advances in neural information processing systems; 1994. p. 3–10.

20. Liu S, Liu S, Cai W, Che H, Pujol S, Kikinis R, et al. Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer’s disease. IEEE Transactions on Biomedical Engineering. 2015;62(4):1132–1140. doi: 10.1109/TBME.2014.2372011 25423647

21. Gupta A, Ayhan M, Maida A. Natural image bases to represent neuroimaging data. In: International Conference on Machine Learning; 2013. p. 987–994.

22. Payan A, Montana G. Predicting Alzheimer’s disease: a neuroimaging study with 3D convolutional neural networks. arXiv preprint arXiv:150202506. 2015;.

23. Hosseini-Asl E, Keynton R, El-Baz A. Alzheimer’s disease diagnostics by adaptation of 3D convolutional network. In: Image Processing (ICIP), 2016 IEEE International Conference on. IEEE; 2016. p. 126–130.

24. Vu TD, Yang HJ, Nguyen VQ, Oh AR, Kim MS. Multimodal learning using convolution neural network and Sparse Autoencoder. In: Big Data and Smart Computing (BigComp), 2017 IEEE International Conference on. IEEE; 2017. p. 309–312.

25. Shi J, Zheng X, Li Y, Zhang Q, Ying S. Multimodal neuroimaging feature learning with multimodal stacked deep polynomial networks for diagnosis of Alzheimer’s disease. IEEE journal of biomedical and health informatics. 2018;22(1):173–183. doi: 10.1109/JBHI.2017.2655720 28113353

26. Shi J, Xue Z, Dai Y, Peng B, Dong Y, Zhang Q, et al. Cascaded Multi-Column RVFL+ Classifier for Single-Modal Neuroimaging-Based Diagnosis of Parkinson’s Disease. IEEE Transactions on Biomedical Engineering. 2018;.

27. Gong B, Shi J, Ying S, Dai Y, Zhang Q, Dong Y, et al. Neuroimaging-based diagnosis of Parkinson’s disease with deep neural mapping large margin distribution machine. Neurocomputing. 2018;320:141–149. doi: 10.1016/j.neucom.2018.09.025

28. Jack CR Jr, Knopman DS, Jagust WJ, Petersen RC, Weiner MW, Aisen PS, et al. Tracking pathophysiological processes in Alzheimer’s disease: an updated hypothetical model of dynamic biomarkers. The Lancet Neurology. 2013;12(2):207–216. doi: 10.1016/S1474-4422(12)70291-0

29. McVeigh E, Bronskill M, Henkelman R. Phase and sensitivity of receiver coils in magnetic resonance imaging. Medical physics. 1986;13(6):806–814. doi: 10.1118/1.595967 3796476

30. Sled JG, Zijdenbos AP, Evans AC. A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE transactions on medical imaging. 1998;17(1):87–97. doi: 10.1109/42.668698 9617910

31. Tustison NJ, Avants BB, Cook PA, Zheng Y, Egan A, Yushkevich PA, et al. N4ITK: improved N3 bias correction. IEEE transactions on medical imaging. 2010;29(6):1310–1320. doi: 10.1109/TMI.2010.2046908 20378467

32. Fonov V, Evans AC, Botteron K, Almli CR, McKinstry RC, Collins DL, et al. Unbiased average age-appropriate atlases for pediatric studies. Neuroimage. 2011;54(1):313–327. doi: 10.1016/j.neuroimage.2010.07.033 20656036

33. Jenkinson M, Bannister P, Brady M, Smith S. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage. 2002;17(2):825–841. doi: 10.1006/nimg.2002.1132

34. Smith SM. Fast robust automated brain extraction. Human brain mapping. 2002;17(3):143–155. doi: 10.1002/hbm.10062 12391568

35. Hahnloser RH, Sarpeshkar R, Mahowald MA, Douglas RJ, Seung HS. Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature. 2000;405(6789):947–951. doi: 10.1038/35016072 10879535

36. Jack CR, Bernstein MA, Fox NC, Thompson P, Alexander G, Harvey D, et al. The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. Journal of magnetic resonance imaging. 2008;27(4):685–691. doi: 10.1002/jmri.21049 18302232

37. Chollet F, et al. Keras; 2015.

38. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. Tensorflow: a system for large-scale machine learning. In: OSDI. vol. 16; 2016. p. 265–283.

39. Gordon BA, Blazey TM, Su Y, Hari-Raj A, Dincer A, Flores S, et al. Spatial patterns of neuroimaging biomarker change in individuals from families with autosomal dominant Alzheimer’s disease: a longitudinal study. The Lancet Neurology. 2018;17(3):241–250. doi: 10.1016/S1474-4422(18)30028-0 29397305

40. Landau SM, Mintun MA, Joshi AD, Koeppe RA, Petersen RC, Aisen PS, et al. Amyloid deposition, hypometabolism, and longitudinal cognitive decline. Annals of neurology. 2012;72(4):578–586. doi: 10.1002/ana.23650 23109153

41. Fischl B. FreeSurfer. Neuroimage. 2012;62(2):774–781. doi: 10.1016/j.neuroimage.2012.01.021 22248573

42. Fischl B, Liu A, Dale AM. Automated manifold surgery: constructing geometrically accurate and topologically correct models of the human cerebral cortex. IEEE transactions on medical imaging. 2001;20(1):70–80. doi: 10.1109/42.906426 11293693

43. Jack CR, Bennett DA, Blennow K, Carrillo MC, Dunn B, Haeberlein SB, et al. NIA-AA Research Framework: Toward a biological definition of Alzheimer’s disease. Alzheimer’s & Dementia. 2018;14(4):535–562. doi: 10.1016/j.jalz.2018.02.018

44. Nelson PT, Head E, Schmitt FA, Davis PR, Neltner JH, Jicha GA, et al. Alzheimer’s disease is not brain aging: neuropathological, genetic, and epidemiological human studies. Acta neuropathologica. 2011;121(5):571–587. doi: 10.1007/s00401-011-0826-y 21516511

45. Bennett D, Schneider J, Arvanitakis Z, Kelly J, Aggarwal N, Shah R, et al. Neuropathology of older persons without cognitive impairment from two community-based studies. Neurology. 2006;66(12):1837–1844. doi: 10.1212/01.wnl.0000219668.47116.e6 16801647

46. Price JL, Davis P, Morris J, White D. The distribution of tangles, plaques and related immunohistochemical markers in healthy aging and Alzheimer’s disease. Neurobiology of aging. 1991;12(4):295–312. doi: 10.1016/0197-4580(91)90006-6 1961359

Článok vyšiel v časopise


2019 Číslo 12