Skip to main content

Multimodal neuroimaging computing: a review of the applications in neuropsychiatric disorders


Multimodal neuroimaging is increasingly used in neuroscience research, as it overcomes the limitations of individual modalities. One of the most important applications of multimodal neuroimaging is the provision of vital diagnostic data for neuropsychiatric disorders. Multimodal neuroimaging computing enables the visualization and quantitative analysis of the alterations in brain structure and function, and has reshaped how neuroscience research is carried out. Research in this area is growing exponentially, and so it is an appropriate time to review the current and future development of this emerging area. Hence, in this paper, we review the recent advances in multimodal neuroimaging (MRI, PET) and electrophysiological (EEG, MEG) technologies, and their applications to the neuropsychiatric disorders. We also outline some future directions for multimodal neuroimaging where researchers will design more advanced methods and models for neuropsychiatric research.


Neuroimaging has advanced rapidly in the past two decades. The advanced non-invasive neuroimaging techniques, e.g., magnetic resonance imaging (MRI), positron emission tomography (PET), electroencephalography (EEG), and magnetoencephalography (MEG), have enabled the visualization and analysis of the brain function and structure in unprecedented detail and transformed the way we study the nervous system under normal and pathological conditions  [1], particularly neuropsychiatric disorders including neurological and psychiatric disorders that affect the nervous system  [24].

In the US, President Obama’s announcement of the ‘Brain Research through Advancing Innovative Neurotechnologies (BRAIN) Initiative’ on his state of the union address on April 2013 fueled resurgent interest in the neuroscience with a bold commitment to better understand the brain over the forthcoming decade [4]. Similar projects have been undertaken in the European Union [5] and Asia  [6].

Multimodal neuroimaging, which we declare as the summation of information from different neuroimaging modalities, has become one of the major drivers in neuroimaging research due to the recognition of the clinical benefits of multimodal data [7, 8], and the better access to hybrid devices, e.g., PET/CT   [9, 10], PET/MRI  [11], and PET/MRI/EEG [12]. Multimodal neuroimaging data can either be obtained from simultaneous imaging measurement (EEG/fMRI [13], PET/CT[14]), or integration of separate measurements (PET and sMRI [15], sMRI and dMRI [16], fMRI and dMRI [17]).

Multimodal neuroimaging advances neuroscience research, i.e., neurology, psychiatry, neurophysiology, and neurosurgery, by overcoming the limitation of individual modalities and by allowing a more comprehensive picture of the brain. For instance, we can jointly analyze the structure and function using the data provided by PET/CT and PET/MRI; EEG combined with functional MRI (fMRI) improves the spatiotemporal resolution that cannot be achieved by the single modality alone. Multimodal neuroimaging can also cross-validate findings from different sources and identify associations and patterns, e.g., causality of brain activity can be deduced by linking dynamics in different imaging readings. It can provide access, in an experimental setting, to determine the roles of different brain areas from multiple perspectives.

The growth of neuroimaging has spurred a parallel development of multimodal neuroimaging computing, which focuses on computational analysis of multimodal neuroimaging data, including pre-processing, feature extraction, image fusion, machine learning, visualization, and post-processing. These computational advances help to address the variations in spatiotemporal resolution and merge the biophysical/biochemical information in images  [18].

Fig. 1

The explosive growth of multimodal neuroimaging studies over the past two decades. (Color figure online)

We conducted a search on PubMed using the keywords ‘multimodal AND neuroimaging’ up to ‘31 Dec 2014.’ There were 1461 relevant publications retrieved from the database. Figure 1 illustrates how multimodal neuroimaging in neuroscience research has rapidly expanded over the past 10 years. In 2004, there were 30 publications, and in 2014, there were close to 300 (indicated by the green area). There is a wide range of applications of multimodal neuroimaging, clinical and non-clinical, including building a brain machine interface (BMI)  [19], tracing neural activities and information pathways  [20], mapping mind and behavior to brain regions [2123], evaluating the effects of pharmacological treatments  [24, 25], and image-guided therapy (IGT)  [2628].

An important clinical application is the provision of functional and anatomical data for diagnosis of neuropsychiatric disorders  [3, 4]. In another PubMed search on these 1461 publications, using the keywords ‘(multimodal AND neuroimaging) AND (neuropsychiatric OR neurological OR psychiatric),’ a substantial proportion (over 30%) of the relevant results focused on the neuropsychiatric disorders (see blue area in Fig. 1). The number of publications dramatically increased each year from 10 to 121 in the period 2004–2014.

Previous reviews mainly focused on a single neuropsychiatric disorder, and summarize the image-based findings of them. For Alzheimer’s disease (AD), for example, Perrin briefly reviewed the multimodal techniques, including PET, fMRI, structural MRI (sMRI), and biochemical examination of cerebrospinal fluid (CSF), to detect AD pathology  [29]. Ewers et al. integrated the findings on changes in cortical gray matter volume, white matter fiber tracts, and brain metabolism of patients  [30], and discussed the sequential changes in neuroimaging biomarkers during different disease stages  [31], similar to the review of Lin et al. [32]. In a more recent review, Nasrallah et al. extended a review to other forms of neurodegenerative dementia  [33]. More in-depth reviews on other neuropsychiatric disorders can be found in Sect. 3.

The goal of this review differs from those above in that our interest is to review the recent advances in multimodal neuroimaging and evaluate its applications in neuropsychiatric disorders. Such a review will provide a clearer picture of the current status and offer insights and inspiration to researchers as they design better models/methods for future research.

An extensive review of the image-based findings in neuropsychiatric disorders is beyond the scope of this paper, and we instead review recent studies with a focus on the applications of multimodal neuroimaging, and refer the readers to other reviews for the detailed findings. In Sect. 2, we provide an overview of the common multimodal neuroimaging techniques, and analyze the spatial/temporal resolution, functional/structural connectivity, sensitivity/specificity to brain changes, risks/benefits for clinical applications, computing workflows, and future potential. In Sect. 3, we discuss how these neuroimaging techniques can complement each other, and how they are applied in neuropsychiatric disorders. In Sect. 4, we outline future directions for multimodal neuroimaging in neuropsychiatric research.

An overview of neuroimaging techniques

The different neuroimaging techniques have different biophysical/biochemical mechanisms, and vary in imaging capabilities. Current neuroimaging techniques could be broadly classified into functional and structural neuroimaging. For example, sMRI reveals the detailed anatomy of the brain, and diffusion MRI (dMRI) provides information about fiber tracts. Functional modalities, including fMRI, PET, and EEG/MEG, provide data in brain metabolism and neural activity.

In the following paragraphs, we briefly summarize these neuroimaging techniques with respect to

  • spatial resolution; exploring the brain anatomy and detecting morphological changes

  • temporal resolution; monitoring neural activities and interactions, tracing information pathways

  • structural connectivity; tracing the major brain white matter pathways

  • functional connectivity; recording the neural co-activation, in the resting state

  • molecular imaging; detecting the molecular activity using agents to target specific functions

  • safety and risks

  • clinical availability, accessibility, and ease of use

  • future developments

Fig. 2

The overview of the properties of sMRI (blue), dMRI (green), fMRI (orange), PET (red), EEG (violet), and multimodal neuroimaging (gray), as indicated by the polar diagrams. Each axis in the diagram represents an attribute, and greater distance from the origin means better performance. Note the indexes in the diagrams are merely indicative and should not be interpreted in a quantitative way. (Color figure online)

Structural MRI (sMRI)

sMRI includes a range of sequences—T1, T2, FLAIR, proton density [34]—that provide detailed information of brain structure, and sMRI is critical for the management of neuropsychiatric disorders. sMRI has spatial resolution up to 0.32 mm (isotropic)  [12]. As shown in Fig. 2, sMRI, however, does not provide connectivity information. Currently, there are approximately 25,000 MRI scanners in use worldwide  [35]. MRI is generally a safe procedure in patients who do not have implanted devices, such as pacemakers and implantable defibrillators  [36], although there are new MRI compatible pacemakers/defibrillators that have been introduced  [37]. MRI uses magnetic and radio waves to generate images, rather than ionizing radiation like X-ray or gamma ray. There are no known harmful side-effects associated with temporary exposure to strong magnetic field and radio waves used by MRI scanners. The narrow bore of MRI scanners is problematic for patients who are claustrophobic or overweight.

When certain contrast agents, mainly iron-oxide-based, are used, sMRI can detect the activity of the targeted molecules with high sensitivity and specificity. Gauberti et al. recently gave a detailed review of the recent advances in ‘molecular’ MRI highlighting molecules that play an important role in neuroinflammation and which may be used as therapeutic targets and biomarkers for neurological disorders [38]. sMRI is a mature technique used in scientific and clinical applications for decades; yet there are still many new developments, i.e., new pulse sequences, new contrast agents, ultra-high magnetic field, and hybrid scanners, all of which offer new imaging opportunities.

Diffusion MRI (dMRI)

dMRI is a MRI sequence that encodes molecular diffusion effects in the nuclear magnetic resonance signal by using bipolar magnetic field gradient pulses [39]. DTI is a form of diffusion imaging where fiber tracts can be delineated based on the fractional anisotropy  [40] and is currently the only technique that allows us to trace the brain white matter pathways in vivo, as shown in Fig. 2—dMRI. By probing at many different orientations, dMRI is able to estimate the orientation of axonal fiber bundles, based on the fact that water diffuses most rapidly along the length of axons. This also leads to longer scanning time as compared to sMRI.

Currently, dMRI is used as a research tool in laboratories, and has not been evaluated in clinical trials due to the crossing-fiber problems, the differences in signal estimation models and fiber tracking algorithms, the variations in datasets, and the lack of ground truth. Nevertheless, DTI is used clinically in the pre-operative planning prior to surgical resection of gliomas which usually displace but can involve the fiber tracts. New models and methods are proposed each year, e.g., the q-space trajectory imaging (QTI)  [41]. Large-scale datasets with uniformly collected dMRI data are also growing in size, and will facilitate the evaluation of these models and methods  [42].

Functional MRI (fMRI)

fMRI is a MRI technique that can depict brain activity by detecting the associated changes in brain hemodynamics. It uses blood-oxygen-level-dependent (BOLD) contrast that is closely related to cerebral blood flow (CBF), as brain function requires blood flow to supply oxygen for energy consumption by neurons. It has relatively high spatial resolution (2mm isotropic) and medium temporal resolution (minutes) for a set of successive scans. Similar to sMRI, fMRI can be used to label specific molecules with contrast agents  [43], i.e., molecular fMRI  [44]. fMRI is used clinically to identify eloquent cortex prior to surgery, e.g., identifying the motor cortex prior to resection of a glioma in the posterior frontal lobe. Two particular strengths of fMRI are that it is able to detect brain activation induced by a task, and provide the connectivity between populations of neurons based on their co-activation at resting state. These two benefits essentially define the two categories of fMRI analyses, task-evoked fMRI and resting-state fMRI.

When the brain is performing a task, CBF usually changes as neurons work to complete the task. The primary use of task-evoked fMRI is to identify the correlation between brain activation/interaction pattern and cognitive states, such as perception, language, memory, emotion, and thought [45, 46]. Recent research based on task-evoked fMRI indicated that altered cognitive functions are related to neuropsychiatric disorders. For instance, emotion regulation capability is not sustained in depressed patients as compared to healthy control subjects [47]. Resting-state fMRI is used to detect the spontaneous activation pattern in the absence of an explicit task or stimuli [48]. Resting-state fMRI enables us to deduce the functional connectivity between dispersed brain regions, which form functional brain networks, or resting-state networks (RSNs). The default mode network (DMN) is a functional network of several brain regions that show increased activity at rest and decreased activity when performing a task  [49]. DMN has been widely used as a measure to compare individual differences in behaviors, genetics, and neuropathologies, although the use of it as a biomarker is controversial  [50, 51].

Recent improvements in spatiotemporal resolution of fMRI have led to higher statistical power to detect RSNs. Further investigation is needed to derive the neuropsychiatric biomarkers from the network and/or network dynamics, and further evaluate them for the diagnosis of individual neuropsychiatric disorders and to guide therapy.

Positron emission tomography (PET)

PET is the most powerful and versatile approach to study neurotransmitter/receptor interactions. It has lower spatiotemporal resolution when compared to MRI, and involves injection of a radioactive tracer and exposure to ionizing radiation. PET is inherently a molecular imaging technique, which is exquisitely sensitive for detecting the targeted molecules or processes. For example, 2-[\(^{18}\)F]fluoro-2-deoxy-D-glucose (FDG) is the most widely used radiotracer that can assess the glucose metabolism in brain, thus has been used for diagnosis, staging, and monitoring treatment of cancers  [52] and neurodegenerative disorders  [53]. In recent years, the percentage of FDG-PET brain studies have decreased due to the introduction of new tracers, e.g., the amyloid-binding compounds, \(^{18}\)F-BAY94-9172, \(^{11}\)C-SB-13, \(^{11}\)C-BF-227, \(^{18}\)F-AV-45, and \(^{11}\) C-Pittsburgh compound B (\(^{11}\)C-PiB). There are a number of reviews of amyloid imaging agents  [29, 5456].

Electroencephalography (EEG) and magnetoencephalography (MEG)

EEG and MEG detect the synchronized activity of an assembly of neurons by displaying their weighted sum of instantaneous neuronal electrical current or magnetic fluxes throughout the brain. EEG and MEG are widely used in neurology clinics due to the simplicity and mobility of EEG monitoring systems, both are safe procedures. EEG and MEG allow us to explore brain cortical activation pattern with ultra-high temporal resolution and record the event-evoked neural information flow in real time. However, EEG and MEG are limited by the low spatial resolution and specificity, and the inability to detect and record the signals from subcortical regions.

An important opportunity for the future is the integration of EEG and MEG with MRI, in particular, with fMRI. EEG and MEG are able to demonstrate the brain activation at much greater temporal resolution when compared to MRI. The MRI produces the anatomical template which enhances the inherent poor spatial resolution of EEG and MEG source images. However, a major challenge has been to develop EEG and MEG technology that can operate in a high magnetic field. Other challenges exist to better understand the correlation between BOLD signals and electrophysiological events via neurovascular coupling and enhance performance of EEG source imaging from simultaneously acquired fMRI data.

Multimodal neuroimaging

Multimodal neuroimaging, which we refer to as the collective information offered in multiple imaging modalities, has become a major driver for current research due to the awareness of the clinical benefits of the multimodal data. As shown in Fig. 2, multimodal data analysis could take the advantages from multiple imaging techniques, e.g., improving both spatial and temporal resolution, finding the anatomical basis for functional connectivity, targeting disease biomarkers with high specificity and sensitivity, along with many new opportunities to improve brain research. Multimodal neuroimaging is currently limited by the availability and safety of the imaging scanners, but novel neuroimaging scanners, especially the hybrid scanners, such as PET/CT and EEG/MRI, will become more widely available in the midrange future. Multimodal neuroimaging analysis is much more challenging than single modality analysis, as multimodal neuroimaging requires sophisticated computing methods, i.e., pre-processing, feature extraction, image fusion, machine learning, visualization, and post-processing, to tackle the large variations in the spatiotemporal resolution and integrate the biophysical/biochemical information of the multimodal data. Many multimodal neuroimaging computing methods have been proposed and applied to a wide range of clinical and non-clinical applications, e.g., brain computer communication  [19], information pathways tracing  [20], brain mapping  [2123], drug development and discovery  [24, 25], pre-operative surgical planning, and intra-operative surgical navigation  [28, 57].

As shown in Fig. 1, there has been explosive growth in multimodal neuroimaging approaches during the last decade, and we may foresee such growth in the following years.

Applications to neuropsychiatric disorders

Neuropsychiatric disorders represent the most disabling and costly category, based on the systematic analysis of descriptive epidemiology of 291 diseases and injuries from 1990 to 2010 for 187 countries  [58]. As shown in Fig. 3, neuropsychiatric disorders caused the largest number of years lost due to illness, disability, and early death measured by disability-adjusted life years (DALYs) in US, and the socioeconomic burden of neuropsychiatric disorders will be aggravated as people live longer.

Fig. 3

The disability-adjusted life years (DALYs) of 291 diseases and injuries based on the systematic analysis of descriptive epidemiology from 1990 to 2010 in US [58]. (Color figure online)

Neuroimaging techniques have expanded beyond a traditional diagnostic role to have a fundamental role in patient management from diagnosis, to selection and assessment of treatment and to prognosis stratification. There is a rising trend of using the multimodal neuroimaging approaches in neuropsychiatric disorders, as shown in Fig. 1. In this section, we summarize how these neuroimaging techniques can be integrated using the multimodal computing methods, and further demonstrate their applications in neuropsychiatric disorders as well as in stroke, traumatic brain injury (TBI), brain tumors, and the brain connectome (Fig. 4).

Fig. 4

The applications of the multimodal neuroimaging approaches in a variety of neuropsychiatric disorders, as well as in stroke, brain injury, brain tumor, and connectome. The color of circle indicates various neuroimaging techniques, same as in Fig. 2. The size of the circle indicates the prevalence of use the technique in specific applications. Note the sizes are only indicative and should not be interpreted in a quantitative way. (Color figure online)

These multimodal approaches can be separated into categories that include a structural–structural combination, a functional–functional combination, and a structural–functional combination. Each category has different applications, and requires different computing workflows. In brief, a structural–structural combination, e.g., sMRI-dMRI, is used to extract and fuse various morphological features and is applied to disorders that affect both gray matter and white matter, such as TBI and stroke. The functional–functional combination can be used to explore brain activation/metabolism patterns and is mainly applied to cognition and consciousness-related disorders, e.g., epilepsy and obsessive-compulsive disorder (OCD). The structural–functional combination is virtually applicable to all disorders, but more frequently used for identifying the structure–function associations in neurodegenerative disorders, neurodevelopmental disorders, multiple sclerosis, schizophrenia, bipolar disorder, brain tumors, and the brain connectome.

Structural–structural combination

sMRI-dMRI methods dominate the structural–structural category, as they take clinical benefits of sMRI and dMRI by integrating the gray matter and white matter morphometry. It has become a useful tool to detect lesions and evaluate treatments for various neuropsychiatric disorders that cause brain morphological changes. Here, we list a few examples of clinical uses of sMRI-dMRI.

Traumatic brain injury (TBI) has very high incidence, resulting in 6.8 million TBI cases every year in the US, and causes impairment of memory, information processing, attention, and executive function  [59]. Multimodal structural neuroimaging can assist neurosurgeons, intensive care specialists, neurologists, and rehabilitation specialists in the management of TBI  [60]. Conventional brain CT usually fails to detect the subtle structural abnormalities in mild TBI, and sMRI and dMRI are the methods of choice to evaluate and predict outcome in TBI. The sMRI sequences (T1, T2, FLAIR, susceptibility-weighted imaging (SWI) and gradient-recalled echo (GRE)) provide highly accurate depiction of pathological lesions, and dMRI detects the effects of TBI on brain connectivity and non-hemorrhagic diffuse axonal injury (DAI), which are not detected by CT. The sMRI-dMRI methods are widely used in TBI  [61, 62]. There are also some studies that have used dMRI and fMRI to validate the connectivity information in TBI patients in the recovery phase  [63, 64].

The sMRI-dMRI methods have been routinely used in the assessment and treatment planning for stroke. Stroke is a leading cause of death worldwide. There are different types of stroke, and each requires a different diagnostic approach and treatment. T2*-weighted sMRI, e.g., SWI and GRE, is primarily used to detect hemorrhagic stroke, and has equal sensitivity to standard CT methods. However, dMRI is 4-5 times more sensitive in detecting acute ischemic stroke than CT. Other structural imaging techniques, such as perfusion CT (PCT), CT angiography (CTA), digital subtraction angiography (DSA), perfusion-weighted imaging (PWI), and MR angiography (MRA), can also be used to evaluate suspected vascular occlusion, edema, and cerebral infarction. Tong et al. [65] recently published a comprehensive comparison of these methods in the evaluation and management of stroke. Another review on multimodal neuroimaging in stroke is given by Copen et al.  [66].

sMRI-dMRI methods have also been used to analyze the gray and white matter alterations in schizophrenia  [67] and Autism spectrum disorders (ASDs)  [16, 68], neurodegeneration simulation  [69], classification of AD and frontotemporal dementia (FTD)  [70], and Parkinson’s Disease (PD) staging  [71].

Functional–functional combination

EEG-fMRI is valued in functional brain research due to the complementary nature of EEG and fMRI. EEG-fMRI can provide simultaneous cortical and subcortical recording of brain activity with high spatiotemporal resolution.

Epilepsy is one of the most prevalent neurological disorders worldwide. EEG-fMRI is increasingly used to provide clinical support for the diagnosis of epilepsy, in addition to the routinely used sMRI  [72] and PET  [14, 73]. Researches have used EEG-fMRI to identify a set of brain functional regions that collectively form ‘consciousness,’ including contributions from the DMN, ascending arousal systems, and the thalamus, as summarized by Bagshaw et al.  [74]. The activation of these regions and the connection of the networks are important in the evaluation of epilepsy, and together may provide a more fundamental understanding of the alterations of consciousness experienced in epilepsy. Abela et al.  [75] focused on altered network compositions in epilepsy, and identified the specific connectivity pathways that characterize the underlying epilepsy syndromes, such as mesial temporal lobe epilepsy (MTLE), lateral temporal lobe epilepsy (LTLE), frontal lobe epilepsy (FLE), idiopathic generalized epilepsy (IGE), and absence epilepsy (AE). A substantial proportion of patients have refractory epilepsy and surgery offers the potential to reduce seizure frequency. Successful surgical treatments, however, require accurate localization of the seizure onset zones and an understanding of surrounding functional cortex to avoid iatrogenic disability. PET, MRI, and intracranial EEG (iEEG) are all needed for optimal surgical planning and treatment evaluation of refractory epilepsy  [76, 77].

Another important application of EEG-fMRI is to evaluate patients with obsessive-compulsive disorder (OCD). OCD is a chronic and relatively common neuropsychiatric disorder that characterized by stereotyped and repetitive behaviors. Patients with OCD feel intense need to carry out these behaviors, and have impaired ability to recognize an error and to adjust future responses. OCD may result in social disability. Two neuroimaging biomarkers of error commission, the error-related negativity (ERN) and the dorsal anterior cingulate cortex activation, have been identified using EEG and fMRI, respectively  [78]. However, Agam et al.  [79] recently suggested that these biomarkers have different neural and genetic mediation. dMRI is also increasingly being used to examine the microstructural integrity of white matter in OCD patients, since white matter abnormalities have long been suspected in OCD, but the findings are inconsistent. For example, one recent study indicated that patients with OCD had decreased fractional anisotropy in the anterior cingulum bundle  [80], but in another recent study, the OCD patients showed increased fractional anisotropy of the cingulum bundle  [81]. Further investigation on large datasets is needed to confirm these findings.

Structural–functional combination

sMRI-dMRI-fMRI has been ubiquitously used in neuropsychiatric research largely because of high clinical availability, and partially due to its capability to link brain function, structure, and connectivity. It has been increasingly used in research in attention-deficit hyperactivity disorder (ADHD), Autism spectrum disorder (ASD), bipolar disorder, schizophrenia, and clinically in multiple Sclerosis (MS).

ADHD is one of the most commonly diagnosed childhood behavioral disorders. It is characterized by persistent inattention (ADHD-I), hyperactivity-impulsivity (ADHD-H), or a combination of both (ADHD-C). ADHD affects at least 5–11% of school-age children, and symptoms may persist into adulthood  [82]. Previous studies using sMRI have reported various findings, such as decreased total brain volume and abnormalities in specific brain regions. The task-evoked and resting-state fMRI approaches were also used in ADHD studies to detect the abnormal brain activation. The use of sMRI and fMRI was reported recently in ADHD  [83, 84]. It is only quite recently that dMRI has been applied to ADHD to characterize the disrupted interconnected structural networks in the brain. Shenton et al. provided a brief summary of the latest studies  [85]. For example, Hong et al. used dMRI and whole-brain tractography to investigate the altered white matter connectivity in 71 children with ADHD, and identified a single network (comprising 23 brain regions and 25 links) that differentiates the ADHD group from the normal control group  [86].

ASDs are neurodevelopmental disorders characterized by deficits in social reciprocity, impaired communication, and restricted interests and repetitive behaviors. Previous studies using sMRI have shown that infants with ASD might have excessive brain growth followed by abnormally slow or even arrested growth as compared to normal developing control infants in early childhood  [87]. Subsequent research indicated ASD affects both gray and white matter volumes. Therefore, dMRI has been exploited to describe the microstructural integrity and orientation of white matter. fMRI has enhanced the understanding of the neural circuity of ASDs by demonstrating the convergent structural and functional changes  [88, 89]. For example, Mueller et al. used sMRI-dMRI-fMRI approach and identified three brain areas with strong correlations between the structural and functional abnormalities: right temporoparietal junction and the left frontal lobe, bilateral superior temporal gyri, and the right temporoparietal region  [90].

MS is a demyelinating disease commonly seen in young people. The cause of MS is unknown. Symptoms and signs vary across patients and can include cognitive impairment, fatigue, vertigo, diplopia, ataxia, hemiparesis, and paraparesis in severe MS patients. Histopathologic and neuroimaging examinations suggest that both white matter and gray matter are affected. In particular, the thalamus can be affected frequently in MS  [91], which can lead to impaired cognition. sMRI can detect the thalamic atrophy; dMRI can be used to demonstrate the altered thalamocortical white matter pathways, and fMRI can be used to show the association between the resting-state thalamocortical functional connectivity and cognitive impairment. Recently, sMRI-dMRI-fMRI was jointly used in several studies  [92, 93].

Bipolar disorder is a psychotic disorder that characterized states of depression and mania, and sometimes with symptoms common to schizophrenia. It is therefore difficult to conceptualize bipolar disorder and its subtypes, and differentiate it from other psychiatric disorders. The multimodal MRI methods have been applied to bipolar disorder and clearly demonstrate abnormalities in brain networks associated with emotion processing, emotion regulation, and reward processing. In a recent study, Sui et al. proposed a joint analysis model for fMRI and DTI for discriminating bipolar disorder from schizophrenia  [94]. Common abnormalities were seen in dorsolateral prefrontal cortex, thalamus, and uncinate fasciculus, whereas differences were found in medial frontal and visual cortex, as well as occipitofrontal white matter tracts. Phillips and Swartz recently published an extensive review of these neuroimaging findings and further pointed out the future directions of neuroimaging research in bipolar disorder  [95].

Schizophrenia is a major psychosis that is characterized by altered perception, thought processes, and behaviors. It can be highly heritable disorder  [96], and can be triggered by a combination of genetic factors and environmental interactions  [97]. Disconnection in white matter pathways and alteration of cortex are assumed to underlie the cognitive abnormalities in schizophrenia, although this is a hypothesis and as yet there is no direct proof. The approaches used for characterizing schizophrenia are very similar to those for bipolar disorder, primarily using sMRI-dMRI-fMRI. Various findings in schizophrenia studies have been reported, based on the investigation on microstructure of white matter  [98] or gray matter  [97], or the connectivity between different brain regions  [67, 99].

The study of brain networks, the connectome, is the focus of intense current neuroscience research [100]. Exploration on the neural systems and brain connections is critical to advance our understanding of normal brain reaction and is one of the greatest challenges of the twenty first century. The Human Connectome ProjectFootnote 1 is directed at tackling this challenge using the highest quality imaging data available today, predominantly MRI data, complemented by EEG and MEG. The information about brain anatomy, structural connectivity, and functional connectivity is being obtained using dMRI and resting-state fMRI. Additional information about brain function is being obtained using task-evoked fMRI, EEG, and MEG to record the brain activity.

sMRI-PET is a new structural–functional combination that is being applied to neurodegenerative diseases and brain tumors to improve the localization and targeting of diseased tissue with high accuracy and sensitivity. AD is the most common neurodegenerative disorder among aging people, and it accounts for close to 70% of all dementia cases. In AD, activities of daily living deteriorate over a number of years, ultimately leading to death. There is no cure [101]. AD neuroimaging biomarkers can detect the changes in brain structure (e.g., atrophy on sMRI) and function (e.g., hypometabolism, amyloid plaque, and NFT formation on PET) before there is cognitive impairment. As a result, sMRI and PET with \(^{18}\)F-FDG and amyloid tracers are being increasingly used in the evaluation of patients with early dementia in the research setting  [8, 102106]. These studies also demonstrated clear benefits of multimodal neuroimaging over any single technique alone. Recently, dMRI  [107, 108] and fMRI  [109] have also been used in the evaluation of dementia as there is evidence that suggests the functional connection between networks is disrupted  [110112]. There are many extensive reviews which summarized these imaging techniques and the image-based findings [2931, 33].

Over 200,000 individuals are diagnosed with primary or metastatic brain tumors in the US each year  [28]. The primary use of sMRI-PET in brain tumors is to accurately localize and label the lesion, e.g., tumor and edema. PET has the potential to more accurately detect the peripheral tumor boundary than using sMRI alone  [11, 113]. For brain tumor surgery, dMRI is usually combined with sMRI and PET for pre-operative surgical planning and intra-operative surgical navigation. For example, Durst et al. used dMRI to predict tumor infiltration in patients with gliomas  [114]. Tempany et al. used sMRI and dMRI tractography to display a complete brain map for surgical planning  [28]. They further demonstrated how to optimize the separation between tumor and normal brain in intrinsic brain tumors with sMRI, and how to avoid inadequate resection of the tumor.

Future directions

Multimodal neuroimaging approaches have been increasingly used in detection, diagnosis, prognosis, and treatment planning of neuropsychiatric disorders. In this paper, we have briefly summarized the recent advances in neuroimaging techniques, and reviewed their applications to neuropsychiatric disorders to provide an overview of the current status. We have also outlined some future directions for multimodal neuroimaging research.

Improved neuroimaging capabilities Neuroimaging techniques will continue to advance rapidly, with higher spatial/temporal/angular resolutions, shorter scan time, and better image contrast. In particular, hybrid scanners, e.g., PET/CT and PET/MRI, will become more clinically accessible. These technologies will enable more discoveries in the neuropsychiatric disorders. The improved imaging capabilities will offer better neuroimaging biomarkers to evaluate neuropsychiatric disorders, and various subtypes or different stages of the same disorder with higher statistical power. These biomarkers will be standardized so they can be widely used clinically and evaluated in large-scale sample sets. In addition, once the biomarkers reach a satisfactory level or the treatment, appropriate clinical guidelines must be developed to support and encourage widespread clinical testing.

Enhanced neuroimaging computing models and methods The continued growth in the complexity and dimensionality of the neuroimaging data will spur the parallel advances of computation models and methods to analyze such complex data. Future neuroimaging analysis models will integrate the longitudinal information to track the long-term changes in the biomarkers [115]. This is essential for us to understand the pathology of the disorders and its degeneration trajectory. With sufficiently large longitudinal datasets, we may be able to identify the causes and detect the early signs, as well as predict the course of the disorders. Future studies will also focus on subject-centered therapy. However, no matter how large the datasets are, they cannot include the entire population, and there will always be inter-subject variations. Personalized/patient-centered care is highly demanded and is the ultimate goal of neuroimaging studies [116]. Neuroimaging computing models and methods also need to keep increasing the degree of automation, accuracy, reproducibility, and robustness, and eventually need to be integrated into the clinical workflow to facilitate clinical testing of the new neuroimaging biomarkers.

Converged neurotechnologies Another future direction will be to combine imaging with non-imaging studies. The multidisciplinary nature of neuroimaging computing will keep bringing together clinicians, biologists, computer scientists, engineers, physicists, and other researchers. Imaging genetics is a very promising area for the future, where the aim is to identify the genetic basis of anatomical and functional abnormalities of the human brain and show how this is connected with neuropsychiatric disorders. There is a trend to use imaging findings in brain disorders to reveal the endophenotypes for various gene mutations. By converting the endophenotype data to novel genetic biomarkers, it may be possible to identify individuals at greater risk of developing brain disorders, and in the near future provide treatment options before the symptoms appear.


  1. 1.


  1. 1.

    Kikinis R, Pieper SD, Vosburgh K (2014) 3D Slicer: a platform for subject-specific image analysis, visualization, and clinical support. Intraoper Imaging Image-Guided Ther 3(19):277–289

    Article  Google Scholar 

  2. 2.

    Alzheimer’s Association (2015) Changing the trajectory of Alzheimer’s disease: how a treatment by 2025 saves lives and dollars.

  3. 3.

    Brookmeyer B, Johnson E, Ziegler-Graham K, Arrighi H (2007) Forecasting the global burden of Alzheimer’s disease. Alzheimer’s Dement 3(3):186–191

    Article  Google Scholar 

  4. 4.

    Insel TR, Landis SC, Collins FS (2013) The NIH BRAIN initiative. Science 340(6133):687–688

    Article  Google Scholar 

  5. 5.

    Amunts K, Linder A, Zilles K (2014) The human brain project: neuroscience perspectives and German contributions. e-Neuroforum 5(2):43–50

    Article  Google Scholar 

  6. 6.

    Jiang T (2013) Brainnetome: a new-ome to understand the brain and its disorders. NeuroImage 80:263–272

    Article  Google Scholar 

  7. 7.

    Hinrichs C, Singh V, Xu G, Johnson S (2011) Predictive markers for AD in a multi-modality framework: an analysis of MCI progression in the ADNI population. NeuroImage 55:574–589

    Article  Google Scholar 

  8. 8.

    Zhang D, Wang Y, Zhou L, Yuan H, Shen D (2011) Multimodal classification of Alzheimer’s disease and mild cognitive impairment. NeuroImage 55(3):856–867

    Article  Google Scholar 

  9. 9.

    Beyer T, Townsend DW, Brun T, Kinahan PE, Charron M, Robby R et al (2000) A combined PET/CT scanner for clinical oncology. J Nucl Med 41(8):1369–1379

    Google Scholar 

  10. 10.

    Townsend DW (2001) A combined PET/CT scanner: the choices. J Nucl Med 42(3):533–534

    Google Scholar 

  11. 11.

    Bisdas S, Nagele T, Schlemmer P, Boss A, Claussen C, Pichler B, Ernemann U (2010) Switching on the lights for real-time multimodality tumor neuroimaging: the integrated positron-emission tomography/MR imaging system. Am J Neuroradiol 31(4):610–614

    Article  Google Scholar 

  12. 12.

    Shah NJ, Oros-Peusquens AM, Arrbula J, Zhang K, Warbrick T et al (2013) Advances in multimodal neuroimaging: hybrid MR-PET and MR-PET-EEG at 3 T and 9.4 T. J Magn Reson 229:101–115

    Article  Google Scholar 

  13. 13.

    He B, Liu Z (2008) Multimodal functional neuroimaging: integrating functional MRI and EEG/MEG. IEEE Rev Biomed Eng 1:23–40

    Article  Google Scholar 

  14. 14.

    Knopman AA, Wong CH, Stevenson RJ et al (2015) The relationship between neuropsychological functioning and FDG-PET hypometabolism in intractable mesial temporal lobe epilepsy. Epilepsy Behav 44:136–142

    Article  Google Scholar 

  15. 15.

    Liu S, Zhang L, Cai W, Song Y, Wang Z, Wen L, Feng D (2013b) A supervised multiview spectral embedding method for neuroimaging classification. In: The 20th IEEE international conference on image processing (ICIP), IEEE, pp 601–605

  16. 16.

    Savadjiev P, Rathi Y, Bouix S, Smith AR et al (2014) Fusion of white and gray matter geometry: a framework for investigating brain development. Med Image Anal 18:1349–1360

    Article  Google Scholar 

  17. 17.

    Zhu D, Zhang T, Jiang X, Hu X et al (2014b) Fusing DTI and fMRI data: a survey of methods and applications. NeuroImage 102:184–191

    Article  Google Scholar 

  18. 18.

    Liu S, Cai W, Liu SQ, Zhang F, Fulham M, Feng D, Pujol S, Kikinis R (2015b) Multimodal neuroimaging computing: the workflows, methods and platforms. Brain Info 2(3).

  19. 19.

    Morioka H, Kanemura A, Morimoto S, Yoshioka T et al (2013) Decoding spatial attention by using cortical currents estimated from electroencephalography with near-infrared spectroscopy prior information. NeuroImage 90:128–139

    Article  Google Scholar 

  20. 20.

    Liu Z, Ding L, He B (2006) Integration of EEG/MEG with MRI and fMRI in functional neuroimaging. IEEE Eng Med Biol Mag 25(4):46–53

    Article  Google Scholar 

  21. 21.

    Binder JR, Desai RH, Graves WW, Conant LL (2009) Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies. Cereb Cortex 19(12):2767–2796

    Article  Google Scholar 

  22. 22.

    Nguyen VT, Cunnington R (2014) The superior temporal sulcus and the N170 during face processing: single trial analysis of concurrent EEG-fMRI. NeuroImage 86:492–502

    Article  Google Scholar 

  23. 23.

    Okamoto M, Dan K, Shimizu K, Takeo K et al (2004) Multimodal assessment of cortical activation during apple peeling by NIRS and fMRI. NeuroImage 21(4):1275–1288

    Article  Google Scholar 

  24. 24.

    Medhi B, Misra S, Kumar P, Kumar P, Singh B (2014) Role of neuroimaging in drug development. Rev Neurosci 25(5):663–673

    Google Scholar 

  25. 25.

    Wong DF, Tauscher J, Grunder G (2009) The role of imaging in proof of concept for CNS drug discovery and development. Neuropsychopharmacology 34:187–203

    Article  Google Scholar 

  26. 26.

    Murphy MA, O’Brien TJ, Morris K, Cook MJ (2004) Multimodality image-guided surgery for the treatment of medically refractory epilepsy. J Neurosurg 100(3):452–462

    Article  Google Scholar 

  27. 27.

    Rodionov R, Vollmar C, Nowell M, Miserocchi A et al (2013) Feasibility of multimodal 3D neuroimaging to guide implantation of intracranial EEG electrodes. Epilepsy Res 107(1–2):91–100

    Article  Google Scholar 

  28. 28.

    Tempany CM, Jayender J, Kapur T, Bueno R et al (2014) Multimodal imaging for improved diagnosis and treatment of cancers. Cancer 121(6):817–827

    Article  Google Scholar 

  29. 29.

    Perrin RJ, Fagan AM, Holtzmann DM (2009) Multimodal techniques for diagnosis and prognosis of Alzheimer’s disease. Nature 461:916–922

    Article  Google Scholar 

  30. 30.

    Ewers M, Frisoni GB, Teipel SJ, Grinberg LT Jr, Amaro E et al (2011) Staging Alzheimer’s disease progression with multimodality neuroimaging. Prog Neurobiol 95:535–546

    Article  Google Scholar 

  31. 31.

    Ewers M, Sperling RA, Klunk WE, Weiner MW, Hampel H (2011b) Neuroimaging markers for the prediction and early diagnosis of Alzheimer’s disease dementia. Trends Neurosci 34(8):430–442

    Article  Google Scholar 

  32. 32.

    Lin AL, Laird AR, Fox PT, Gao JH (2012) Multimodal MRI neuroimaging biomarkers for cognitive normal adults, amnestic mild cognitive impairment, and Alzheimer’s disease. Neurol Res Int 907409:1–17

    Article  Google Scholar 

  33. 33.

    Nasrallah IM, Wolk DA (2014) Multimodality imaging of Alzheimer’s disease and other neurodegenerative dementias. J Nucl Med 55:2004–2011

    Article  Google Scholar 

  34. 34.

    Rydberg J, Hammond C, Grimm R, Erickson B, Jack CJ et al (1994) Initial clinical experience in MR imaging of the brain with a fast fluid-attenuated inversion-recovery pulse sequence. Radiology 193(1):173–180

    Article  Google Scholar 

  35. 35.

    Rinck P (2014) Magnetic resonance: a critical peer-reviewed introduction. In: Magnetic resonance in medicine. The basic textbook of the European magnetic resonance forum, 8th edn., Chap 21.

  36. 36.

    Gotte M, Russel I, de Roest G, Germans T, Veldkamp R et al (2010) Magnetic resonance imaging, pacemakers and implantable cardioverter-defibrillators: current situation and clinical perspective. Neth Heart J 18(1):31–37

    Google Scholar 

  37. 37.

    Bovenschulte H, Schluter-Brust K, Liebig T, Erdmann E, Eysel P, Zobel C (2012) MRI in patients with pacemakers—overview and procedural management. Deutsches Arzteblatt Int 109(15):270–275

    Google Scholar 

  38. 38.

    Gauberti M, Montagne A, Quenadlt A, Viven D (2014) Molecular magnetic resonance imaging of brain–immune interactions. Front Cell Neurosci 8(389):1–20

    Google Scholar 

  39. 39.

    Bihan D, Mangin J, Poupon C, Clark C et al (2001) Diffusion tensor imaging: concepts and applications. J Magn Reson Imaging 13:534–546

    Article  Google Scholar 

  40. 40.

    Basser P, Mattiello J, LeBihan D (1994) MR diffusion tensor spectroscopy and imaging. Biophys J 66(1):259–267

    Article  Google Scholar 

  41. 41.

    Westin CF, Szczepankiewicz F, Pasternak O, Ozarslan E, Topgaard D et al (2014) Measurement tensors in diffusion MRI: generalizing the concept of diffusion encoding. Medical image computing and computer-assisted intervention (MICCAI), LNCS. Springer, Heidelberg, pp 209–216

    Google Scholar 

  42. 42.

    Essen DCV, Smith SM, Barch DM, Behrens TE, Yacoub E et al (2013) The WU-Minn human connectome project: an overview. NeuroImage 80:62–79

    Article  Google Scholar 

  43. 43.

    Jasanoff A (2007) MRI contrast agents for functional molecular imaging of brain activity. Curr Opin Neurol 17(5):593–600

    Article  Google Scholar 

  44. 44.

    Lee T, Cai LX, Lelyveld VS, Hai A, Jasanoff A (2014) Molecular-level functional magnetic resonance imaging of dopaminergic signaling. Science 344(6183):533–535

    Article  Google Scholar 

  45. 45.

    Davison EN, Schlesinger KJ, Bassett DS, Lynall ME et al (2015) Brain network adaptability across task states. PLoS Comput Biol 11(1):e1004029

    Article  Google Scholar 

  46. 46.

    Turk-Browne NB (2013) Functional interactions as big data in the human brain. Science 342(6158):580–584

    Article  Google Scholar 

  47. 47.

    Erk S, Mikschl A, Stier S et al (2010) Acute and sustained effects of cognitive emotion regulation in major depression. J Neurosci 30(47):15726–15734

    Article  Google Scholar 

  48. 48.

    Biswal B, Yetkin F, Haughton V, Hyde J (1995) Functional connectivity in motor cortex of resting human brain using echo-planar MRI. Magn Reson Med 34:537–541

    Article  Google Scholar 

  49. 49.

    Raichle M, MacLeod A, Snyder A, Powers W et al (2001) A default mode of brain function. Proc Natl Acad Sci 98(2):676–682

    Article  Google Scholar 

  50. 50.

    Buckner R, Andrews-Hanna J, Schacter D (2008) The brain’s default network: anatomy, function and relevance to disease. Ann N Y Acad Sci 1124:1–38

    Article  Google Scholar 

  51. 51.

    Jiang T, Liu Y, Shi F, Shu N, Liu B et al (2008) Multimodal magnetic resonance imaging for brain disorders: advances and perspectives. Brain Imaging Behav 2:249–257

    Article  Google Scholar 

  52. 52.

    Young H, Baum R, Cremerius U, Herholz K et al (1999) Measurement of clinical and subclinical tumour response using [18F]-fluorodeoxyglucose and positron Emission Tomography: Review and 1999 EORTC Recommendations. European organization of research and treatment of cancer (EORTC) PET study group. Eur J Cancer 35(13):1773–1782

    Article  Google Scholar 

  53. 53.

    Cai W, Liu S, Wen L, Eberl S, Fulham MJ, Feng D (2010) 3D neurological image retrieval with localized pathology-centric CMRGlc patterns. In: The 17th IEEE international conference on image processing (ICIP), IEEE, pp 3201–3204

  54. 54.

    Carpenter AJ, Pontecorvo M, Hefti F, Skovronsky D (2009) The use of the exploratory IND in the evaluation and development of 18F-PET radiopharmaceuticals for amyloid imaging in the brain: a review of one company’s experience. Q J Nucl Med Mol Imaging 53(4):387–393

    Google Scholar 

  55. 55.

    Ni R, Gillberg P, Bergfors A, Marutle A, Nordberg A (2013) Amyloid tracers detect multiple binding sites in Alzheimer’s disease brain tissue. Brain 136(7):2217–2227

    Article  Google Scholar 

  56. 56.

    Thompson PM, Ye L, Morgenstem JL, Sue L, Beach TG et al (2009) Interaction of the amyloid imaging tracer FDDNP with hallmark Alzheimer’s disease pathologies. J Neurochem 109(2):623–630

    Article  Google Scholar 

  57. 57.

    Upadhyay U, Golby A (2008) Role of pre- and intraoperative imaging and neuronavigation in neurosurgery. Exp Rev Med Dev 5:65–78

    Article  Google Scholar 

  58. 58.

    Murray C, Abraham J, Ali M, Alvarado M, Atkinson C et al (2013) The state of US Health, 1990–2010: burden of diseases, injuries, and risk factors. J Am Med Assoc 310(6):591–608

    Article  Google Scholar 

  59. 59.

    Irimia A, Wang B, Aylward SR, Prastawa MW, Pace DF et al (2012) Neuroimaging of structural pathology and connectomics in traumatic brain injury: toward personalized outcome prediction. NeuroImage Clin 1:1–17

    Article  Google Scholar 

  60. 60.

    Sundstrom T, Grande P, Juul N, Kock-Jensen C et al (2012) Management of severe traumatic brain injury: evidence, tricks, and pitfalls. Springer, Berlin

    Book  Google Scholar 

  61. 61.

    Irimia A, Chambers MC, Alger JR, Filippou M, Prastawa MW et al (2011) Comparison of acute and chronic traumatic brain injury using semi-automatic multimodal segmentation of MR volumes. J Neurotrauma 28(11):2287–2306

    Article  Google Scholar 

  62. 62.

    Turken AU, Herron TJ, Kang X, O’Connor LE, Sorenson DJ et al (2009) Multimodal surface-based morphometry reveals diffuse cortical atrophy in traumatic brain injury. BMC Med Imaging 9:20

    Article  Google Scholar 

  63. 63.

    Cherubini A, Luccichenti G, Peran P, Hagberg GE et al (2007) Multimodal fMRI tractography in normal subjects and in clinically recovered traumatic brain injury patients. NeuroImage 34(4):1331–1341

    Article  Google Scholar 

  64. 64.

    Dean PJ, Sato JR, Vieira G, McNamara A, Sterr A (2014) Multimodal imaging of mild traumatic brain injury and persistent postconcussion syndrome. Brain Behav 5(1):45–61

    Google Scholar 

  65. 65.

    Tong E, Hou Q, FFiebach JB, Wintermark M (2014) The role of imaging in acute ischemic stroke. Neurosurg Focus 36(1):E3

    Article  Google Scholar 

  66. 66.

    Copen WA (2015) Multimodal imaging in acute ischemic stroke. Curr Treat Options Cardiovasc Med 17(10):1–17

    Google Scholar 

  67. 67.

    Liu X, Lai Y, Wang X, Hao C et al (2014c) A combined DTI and structural MRI Study in medicated-naive chronic schizophrenia. Magn Reson Imaging 32(1):1–8

    Article  Google Scholar 

  68. 68.

    Hasan KM, Walimuni IS, Frye RE (2013) Global cerebral and regional multimodal neuroimaging markers of the neurobiology of autism. J Child Neurol 28(7):874–885

    Article  Google Scholar 

  69. 69.

    Modat M, Simpson I, Cardoso M, Cash D et al (2014) Simulating neurodegeneration through longitudinal population analysis of structural and diffusion weighted MRI data. Medical image computing and computer-assisted intervention (MICCAI), LNCS, vol 8675. Springer, Berlin, pp 57–64

    Google Scholar 

  70. 70.

    Avants B, Cook P, Ungar L, Gee J, Grossman M (2010) Dementia induces correlated reduction in white matter integrity and cortical thickness: a multivariate neuroimaging study with sparse canonical correlation analysis. NeuroImage 50(3):1004–1016

    Article  Google Scholar 

  71. 71.

    Aquino D, Contarino V, Albanese A et al (2013) Substantia nigra in Parkinson’s disease: a multimodal MRI comparison between early and advanced stages of the disease. Neurol Sci 35(5):753–758

    Article  Google Scholar 

  72. 72.

    Bonilha L, Keller SS (2015) Quantitative MRI in refractory temporal lobe epilepsy: relationship with surgical outcomes. Quant Imaging Med Surg 5(2):204–224

    Google Scholar 

  73. 73.

    la Fougere C, Rominger A, Forster S, Geisler J, Bartenstein P (2009) PET and SPECT in epilepsy: a critical review. Epilepsy Behav 15:50–55

    Article  Google Scholar 

  74. 74.

    Bagshaw AP, Rollings DT, Khalsa S, Cavanna AE (2014) Multimodal neuroimaging investigations of alternations to consciousness: the relationship between absence epilepsy and sleep. Epilepsy Behav 30:33–37

    Article  Google Scholar 

  75. 75.

    Abela E, Rummel C, Hauf M, Weisstanner C, Schindler K, Wiest R (2014) Neuroimaging of epilepsy: lesions, networks, oscillations. Clin Neuroradiol 24(1):5–15

    Article  Google Scholar 

  76. 76.

    Fernandez S, Donaire A, Seres E, Setoain X, Bargallo N et al (2015) PET/MRI and PET/MRI/SISCOM coregistration in the presurgical evaluation of refractory facol epilepsy. Epilepsy Res 111:1–9

    Article  Google Scholar 

  77. 77.

    Rastogi S, Lee C, Salamon N (2008) Neuroimaging in pediatric epilepsy: a multimodality approach. Radiographics 28(4):1079–1095

    Article  Google Scholar 

  78. 78.

    Taylor SF, Stern ER, Gehring WJ (2007) Neural systems for error monitoring—recent findings and theoretical perspectives. Neuroscientist 13(2):160–172

    Article  Google Scholar 

  79. 79.

    Agam Y, Vangel M, Roffman JL, Gallagher PJ et al (2014b) Dissociable genetic contributions to error processing: a multimodal neuroimaging study. PLoS One 9(7):e101784

    Article  Google Scholar 

  80. 80.

    Radua J, Grau M, van den Heuvel OA, de Schotten MT et al (2014) Multimodal voxel-based meta-analysis of white matter abnormalities in obsessive-compulsive disorder. Neuropsychopharmacology 39(7):1547–1557

    Article  Google Scholar 

  81. 81.

    Agam Y, Greenberg JL, Isom M, Falkenstein NJ et al (2014a) Aberrant error processing in relation to symptom severity in obsessive-compulsive disorder: a multimodal neuroimaging study. NeuroImage Clin 5:141–151

    Article  Google Scholar 

  82. 82.

    Visser S, Danielson M, Bitsko R, Holbrook J et al (2014) Trends in the parent-report of health care provider-diagnosed and medicated attention-deficit/hyperactivity disorder: United States, 2003–2011. J Am Acad Child Adolesc Psychiatry 53(1):34–62

    Article  Google Scholar 

  83. 83.

    Anderson A, Douglas PK, Kerr WT, Haynes VS et al (2014) Non-negative matrix factorization of multimodal MRI, fMRI and phenotypic data reveals differential changes in default mode subnetworks in ADHD. NeuroImage 102(1):207–219

    Article  Google Scholar 

  84. 84.

    Dai D, Wang J, Hua J, He H (2012) Classification of ADHD children through multimodal magnetic resonance imaging. Front Syst Neurosci 6(63):1–8

    Google Scholar 

  85. 85.

    Shenton M, Kubicki M, Makris N (2014) Understanding alterations in brain connectivity in attention-deficit/hyperactivity disorder using imaging connectomics. Biol Psychiatry 76:601–602

    Article  Google Scholar 

  86. 86.

    Hong SB, Zalesky A, Fornito A, Park S, Yang YH et al (2014) Connectomic disturbances in attention-deficit/hyperactivity disorder: a whole-brain tractography analysis. Biol Psychiatry 76(8):656–663

    Article  Google Scholar 

  87. 87.

    Courchesne E (2004) Brain development in autism: early overgrowth followed by premature arrest of growth. Ment Retard Dev Disabil Res Rev 10(2):106–111

    Article  Google Scholar 

  88. 88.

    Anagnostou E, Taylor MJ (2011) Review of neuroimaging in autism spectrum disorders: what have we learnt and where we go from here. Mol Autism 2(4):1–9

    Google Scholar 

  89. 89.

    Stigler KA, McDonald BC, Anand A et al (2011) Structural and functional magnetic resonance imaging of autism spectrum disorders. Brain Res 1380:146–161

    Article  Google Scholar 

  90. 90.

    Mueller S, Keeser D, Samson AC, Kirsch V, Blautzik J et al (2013) Convergent findings of altered functional and structural brain connectivity in individuals with high functioning autism: a multimodal MRI study. PLoS One 8(6):e67329

    Article  Google Scholar 

  91. 91.

    Achiron A, Barak Y (2003) Cognitive impairment in probable multiple sclerosis. J Neurol Neurosurg Psychiatry 74:443–446

    Article  Google Scholar 

  92. 92.

    Louapre C, Perlbarg V, Garcia-Lorenzo D, Urbanski M et al (2014) Brain networks disconnection in early multiple sclerosis cognitive deficits: an anatomofunctional study. Hum Brain Mapp 35:4706–4717

    Article  Google Scholar 

  93. 93.

    Tona F, Petsas N, Sbardella E, Prosperini L et al (2014) Multiple sclerosis: altered thalamic resting-state functional connectivity and its effect on cognitive function. Radiology 271(3):814–821

    Article  Google Scholar 

  94. 94.

    Sui J, Pearlson GD, Caprihan A, Adali T, Kiehl KA et al (2011) Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+ joint ICA model. NeuroImage 57(3):839–855

    Article  Google Scholar 

  95. 95.

    Phillips ML, Swartz HA (2014) A critical appraisal of neuroimaging studies of bipolar disorder: toward a new conceptualization of underlying neural circuitry and a road map for future research. Am J Psychiatry 171(8):829–843

    Article  Google Scholar 

  96. 96.

    Purcell S, Wray N, Stone S et al (2009) Common polygenetic variation contributes to risk of schizophrenia and bipolar disorder. Nature 460(7256):748–752

    Google Scholar 

  97. 97.

    Cooper D, Barker V, Radua J, Fusar-Poli P, Lawrie SM (2014) Multimodal voxel-based meta-analysis of structural and functional magnetic resonance imaging studies in those at elevated genetic risk of developing schizophrenia. Psychiatry Res Neuroimaging 221(1):69–77

    Article  Google Scholar 

  98. 98.

    Kochunov P, Chiappelli J, Wright SN, Rowland LM et al (2014) Multimodal white matter imaging to investigate reduced fractional anisotropy and its age-related decline in schizophrenia. Psychiatry Res Neuroimaging 223(2):148–156

    Article  Google Scholar 

  99. 99.

    Pomarol-Clotet E, Canales-Rodriguez E, Salvador R, Sarro S et al (2010) Medial prefrontal cortex pathology in schizophrenia as revealed by convergent findings from multimodal imaging. Mol Psychiatry 15:823–830

    Article  Google Scholar 

  100. 100.

    Yao Z, Hu B, Xie Y, Moore P, Zheng J (2015) A review of structural and functional brain networks: small world and atlas. Brain Inf 2(1):45–52

    Article  Google Scholar 

  101. 101.

    Kalaria RN, Maestre GE, Arizaga R, Friedland RP, Galasko D, Hall K et al (2008) Alzheimer’s disease and vascular dementia in developing countries: prevalence, management, and risk factors. Lancet Neurol 7(9):812–826

    Article  Google Scholar 

  102. 102.

    Che H, Liu S, Cai W, Pujol S, Kikinis R, Feng D (2014) Co-neighbor multi-view spectral embedding for medical content-based retrieval. In: IEEE international symposium on biomedical imaging: from nano to macro (ISBI), IEEE, pp 911–914

  103. 103.

    Liu S, Song Y, Cai W, Pujol S, Kikinis R, Wang X, Feng D (2013a) Multifold Bayesian kernelization in Alzheimer’s diagnosis. In: Mori K, Sakuma I, Sato Y, Barillot C, Navab N (eds) The 16th international conference on medical image computing and computer-assisted intervention (MICCAI), LNCS, vol 8150. Springer, Berlin, pp 303–310

    Google Scholar 

  104. 104.

    Liu S, Liu SQ, Pujol S, Kikinis R, Cai W (2014a) Propagation graph fusion for multi-modal medical content-based retrieval. In: The 13th annual international conference on control. Automation, robotics and vision (ICARCV), IEEE, pp 849–854

  105. 105.

    Liu SQ, Liu S, Cai W, Pujol S, Kikinis R, Feng D (2014b) Early diagnosis of Alzheimer’s disease with deep learning. In: IEEE international symposium on biomedical imaging: from nano to macro (ISBI), IEEE, pp 1015–1018

  106. 106.

    Liu SQ, Liu S, Cai W, Che H, Pujol S, Kikinis R, Feng D (2015c) Multi-modal neuroimaging feature learning for multi-class diagnosis of Alzheimer’s disease. IEEE Trans Biomed Eng 62(4):1132–1140

    Article  Google Scholar 

  107. 107.

    Nir TM, Jahanshad N, Villalon-Reina JE, Toga AW, Jack CR, Weiner MW, Thompson PM (2013) Effectiveness of regional DTI measures in distinguishing Alzheimer’s disease, MCI, and normal aging. NeuroImage Clin 3:180–195

    Article  Google Scholar 

  108. 108.

    Racine AM, Adluru N, Alexander AL, Christian BT et al (2014) Associations between white matter microstructure and amyloid burden in preclinical Alzheimer’s disease: a multimodal imaging investigation. NeuroImage Clin 4:604–614

    Article  Google Scholar 

  109. 109.

    He X, Qin W, Liu Y, Zhang X et al (2013) Abnormal salience network in normal aging and in amnestic mild cognitive impairment and Alzheimer’s disease. Hum Brain Mapp 35(7):3446–3464

    Article  Google Scholar 

  110. 110.

    Jacobs HI, Gronenschild EH, Evers EA et al (2015) Visuospatial processing in early Alzheimer’s disease: a multimodal neuroimaging study. Cortex 64:394–406

    Article  Google Scholar 

  111. 111.

    Wee CY, Yap PT, Zhang D, Denny K et al (2012) Identification of MCI individuals using structural and functional connectivity networks. NeuroImage 59(3):2045–2056

    Article  Google Scholar 

  112. 112.

    Zhu D, Li K, Terry DP et al (2014a) Connectome-scale assessments of structural and functional connectivity in MCI. Hum Brain Mapp 35(7):2911–2923

    Article  Google Scholar 

  113. 113.

    Neuner I, Kaffanke JB, Langen KJ, Kops ER, Tellmann L et al (2012) Multimodal imaging utilising integrated MR-PET for human brain tumor assessment. Eur Radiol 22:2568–2580

    Article  Google Scholar 

  114. 114.

    Durst CR, Raghavan P, Shaffrey ME, Schiff D et al (2014) Multimodal MR imaging model to predict tumor infiltration in patients with gliomas. Neuroradiology 56(2):107–115

    Article  Google Scholar 

  115. 115.

    Liu SQ, Liu S, Zhang F, Cai W, Pujol S, Kikinis R, Feng D (2015d) Longitudinal brain MR retrieval with diffeomorphic demons registration: what happened to those patients with similar changes? In: IEEE international symposium on biomedical imaging: from nano to macro (ISBI), IEEE

  116. 116.

    Liu S, Cai W, Liu SQ, Pujol S, Kikinis R, Feng D (2015a) Subject-centered multi-view neuroimaging analysis. In: The 22nd IEEE international conference on image processing (ICIP), IEEE

Download references


This work was supported by ARC, AADRF, NA-MIC (NIH U54EB005149), and NAC (NIH P41RR013218).

Author information



Corresponding author

Correspondence to Sidong Liu.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Cai, W., Liu, S. et al. Multimodal neuroimaging computing: a review of the applications in neuropsychiatric disorders. Brain Inf. 2, 167–180 (2015).

Download citation


  • Multimodal
  • Neuroimaging
  • Neuropsychiatric