- Original research
- Open Access
DeepNeuron: an open deep learning toolbox for neuron tracing
© The Author(s) 2018
- Received: 26 February 2018
- Accepted: 18 April 2018
- Published: 6 June 2018
Reconstructing three-dimensional (3D) morphology of neurons is essential for understanding brain structures and functions. Over the past decades, a number of neuron tracing tools including manual, semiautomatic, and fully automatic approaches have been developed to extract and analyze 3D neuronal structures. Nevertheless, most of them were developed based on coding certain rules to extract and connect structural components of a neuron, showing limited performance on complicated neuron morphology. Recently, deep learning outperforms many other machine learning methods in a wide range of image analysis and computer vision tasks. Here we developed a new Open Source toolbox, DeepNeuron, which uses deep learning networks to learn features and rules from data and trace neuron morphology in light microscopy images. DeepNeuron provides a family of modules to solve basic yet challenging problems in neuron tracing. These problems include but not limited to: (1) detecting neuron signal under different image conditions, (2) connecting neuronal signals into tree(s), (3) pruning and refining tree morphology, (4) quantifying the quality of morphology, and (5) classifying dendrites and axons in real time. We have tested DeepNeuron using light microscopy images including bright-field and confocal images of human and mouse brain, on which DeepNeuron demonstrates robustness and accuracy in neuron tracing.
- Deep learning
- Neuron tracing
- Neuron morphology
Over the past few decades, researchers have developed algorithms and tools to reconstruct (trace) 3D neuron morphology. A number of manual/semiautomatic neuron tracing software packages in both the public domain and commercial world have been developed [1–9]. To further promote the development of neuron tracing tools, the DIADEM challenge  and the BigNeuron project  were launched to compare different automated algorithms. At small or medium scales, many algorithms (base tracers) have been shown to produce meaningful reconstructions on high-quality neuron images. For large-scale image datasets, UltraTracer  provides an extendible framework to scale up the capability of these base tracers. Despite these efforts on algorithm and tool development, it remains an open question on how to faithfully reconstruct neuron morphology from challenging image datasets that have medium to low qualities and contain very complex neuron morphology.
Starting from a cell body, a neuron tracing process usually follows dendrites and axons, eventually connecting all such neuron signal as a tree that represents the morphology of the neuron. In light microscopy images, dendrites typically show continuous signal, whereas axons are often hard to trace due to their punctuated appearance and large, complex arborization patterns (; see for example the bright-field images of biocytin-labeled neurons in the Allen Cell Type Database ). In addition, the image quality varies a lot depending on sample preparation, imaging process, cell types, and the healthiness of neurons. For instance, neuron signal could be continuous in one image, but dim and broken in another. It is difficult to automatically extract all such neuron signal under different conditions.
Several important steps in neuron tracing can be formulated as a classification problem. For example, detection of neuron signal from background is essentially foreground–background classification. Reconstruction of the topology of a neuron via connecting neuron fragments can be treated as connection-separation classification. In this aspect, a few studies used traditional machine learning and recent deep learning  models to produce neuron morphology. For example, Gala et al. introduced an active learning model by combining different features to automatically trace neurites . Chen et al. proposed a self-learning-based tracing approach, which did not require substantial human annotations . Fakhry et al.  and Li et al.  used deep learning neural networks to segment electron and light microscopy neuron images. Despite these algorithmic efforts, none of these methods provides publicly available tools to use on external datasets.
Neurite signal detection automatically identifies 3D dendritic and axonal signal from background.
Neurite connection automatically connects local neurite signal to form neuronal trees.
Smart pruning filters false positive and refines automated reconstruction results.
Manual reconstruction evaluation evaluates manual reconstructions and provides quality scores.
Classification of dendrites and axons automatically classifies neurite types during real-time annotation.
2.1 Neurite signal detection
Manually reconstructed neurons were used as training samples. The 3D reconstruction of a neuron is represented as a tree, which contains a series of 3D X, Y, Z locations, radius, and topological “parent” of annotation nodes. To train the network, local 3D blocks (block size 61 × 61 × 61 was used in our experiments) centered on manually annotated nodes in neurite segments were cropped from the original images. 2D maximum intensity projections (MIPs) of theses 3D blocks were used as the positive training set, and the same number of 2D background MIPs were randomly selected as the negative training set.
Fivefold cross-validation on bright-field training sets
In testing, we first projected the original 3D image stack onto the XY plane and generated a MIP image. We then cropped 2D patches using a sliding window with n-pixel stride. These patches were classified into patches centered on foreground or background pixels using our trained CNN model. To further improve classification accuracy and exclude false positive patches, we applied mean shift  to the detected foreground patches and map them back to the actual 3D locations based on the local maximum intensity along Z. Finally, we classified these 3D detected signals using our CNN model again based on the MIPs of the local 3D blocks.
2.2 Neurite connection
Siamese networks [21, 22] are used among tasks that involve finding similarity or the relationship between two subjects being compared. Our revised Siamese model in this work includes two identical arms. Each consists of two convolutional layers with max pooling, followed by three fully connected layers. The two arms are then fed to a contrastive loss function to produce a binary decision.
In training (Fig. 4a), we used pairs of patches generated from two consecutive annotation nodes as positive training samples, and pairs of patches generated from two spatially separated annotation nodes as negative training samples. We used ~ 919 K training pairs, ~ 460 K of them being positive pairs and ~ 459 K being negative pairs.
In connection (Fig. 4c), a 1 × M feature vector is extracted from individual input patch. (M can be defined by the user, and we used M = 200 in our experiment.) The Euclidean distance between two feature vectors is calculated as the dissimilarity score of a patch pair, which is multiplied by the distance to form the weight in our proposed DMST graph.
2.3 Smart pruning
2.4 Manual reconstruction evaluation
Fivefold cross-validation on 122 manual reconstructions of biocytin-labeled mouse neuron dataset
First, all annotation nodes in test subset were classified into two categories: foreground and background.
All classified foreground nodes formed an initial prediction.
Based on the orientation, tip location, and distance, fragments in the initial prediction were automatically connected to produce a refined prediction. In our experiment, we only connected terminal tips between two segments whose orientation differs less than 30 degrees and distance is smaller than 30 voxels.
The test subset is evaluated by the consistency score c:
Table 2 shows the fivefold cross-validation results on 122 manual reconstructions for the bright-field biocytin-labeled mouse neuron dataset from Allen Cell Type Database . The high consistency scores indicate that manual reconstructions are very consistent across different annotators and different subsets of data. In addition, thicker and more continuous dendrites (> 99%) have higher consistency scores than dim and discontinuous axons (> 96%), which are harder to reconstruct.
Comparison results of consistency scores on human and mouse neuron reconstructions
2.5 Classification of dendrites and axons
Dendrites and axons have their own functions and play different roles in the nervous system. Distinguishing these two types of neurites can help us gain insight into the brain circuitry. Although dendrites and axons show different shapes and intensity properties in light microscopy images, such a general rule of thumb, however, is not always guaranteed. Due to variant image quality, axons can also appear continuous and look more like dendrites. This makes them difficult to be correctly labeled in most of tracing algorithms. Here we present a deep learning module serving as a vehicle for the networks that are trained for this purpose. This tool allows to automatically classify dendrites and axons on real-time manual annotation, and potentially save time for annotators.
Comparison of AlexNet with a revised model
Deep learning network
Averaged forward–backward time (s)
Deemed axon (%)
Deemed dendrite (%)
Deemed background (%)
A revised model
For a neuron image stack, it can be used to automatically detect neurite signals.
For a neuron image stack with detected 3D signals, it can automatically connect signals to generate local segments.
For a neuron image stack with its associated automated reconstruction, it can be used as a filter to clean up all false positive tracing and generate a refined result.
For a neuron image stack with its associated manual reconstructions, it can evaluate how consistent and reliable the reconstructions are.
For a neuron image stack with interactive human annotation via the user interface, it can label neurite types in real time.
DeepNeuron has been implemented as an Open Source plugin in Vaa3D (http://vaa3d.org) [7, 8]. DeepNeuron toolbox is a highly flexible vehicle allowing investigators to take advantage of deep learning to facilitate neuron tracing in their research. As mentioned in this article, researchers can freely replace different network models that suit their needs. Combined with other related features in Vaa3D including 30+ automatic neuron tracing plugins, semiautomatic neuron annotation, annotation utilities, neuron image/reconstruction visualization, DeepNeuron works as a smart artificial intelligence engine which offers great help to biologists in exploring neuronal morphology.
The DeepNeuron toolbox was written in C++ as a plugin to Vaa3D. DeepNeuron source code is available at https://github.com/Vaa3D/vaa3d_tools/tree/master/hackathon/MK/DeepNeuron. In addition, the DeepNeuron plugin is also included as a plugin in binary releases of Vaa3D, which can be downloaded at https://github.com/Vaa3D/Vaa3D_Data/releases/tag/1.0.
HP conceived and managed the project. FL proposed the overall technical framework. ZZ developed the toolbox and conducted the experiments. HK implemented a plugin for the dendrite and axon classification and assisted in several other experiments. All authors edited the manuscript. All authors read and approved the final manuscript
We thank Allen Institute for Brain Science for providing neuron datasets and manual annotations. The authors wish to thank the Allen Institute founders, P. G. Allen and J. Allen, for their vision, encouragement, and support.
On behalf of all authors, the corresponding author states that there is no competing interests.
Ethics approval and consent to participate
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Choromanska A, Chang S-F, Yuste R (2012) Automatic reconstruction of neural morphologies with multi-scale tracking. Front Neural Circuits 6:25View ArticleGoogle Scholar
- Donohue DE, Ascoli GA (2011) Automated reconstruction of neuronal morphology: an overview. Brain Res Rev 67:94–102View ArticleGoogle Scholar
- Feng L, Zhao T, Kim J (2015) neuTube 1.0: a new design for efficient neuron reconstruction software based on the SWC format. eNeuro 2:ENEURO-0049View ArticleGoogle Scholar
- Longair MH, Baker DA, Armstrong JD (2011) Simple neurite tracer: open source software for reconstruction, visualization and analysis of neuronal processes. Bioinformatics 27:2453–2454View ArticleGoogle Scholar
- Luisi J, Narayanaswamy A, Galbreath Z, Roysam B (2011) The FARSIGHT trace editor: an open source tool for 3-D inspection and efficient pattern analysis aided editing of automated neuronal reconstructions. Neuroinformatics 9:305–315View ArticleGoogle Scholar
- Meijering E, Jacob M, Sarria JC, Steiner P, Hirling H, Unser M (2004) Design and validation of a tool for neurite tracing and analysis in fluorescence microscopy images. Cytom Part A 58:167–176View ArticleGoogle Scholar
- Peng H, Bria A, Zhou Z, Iannello G, Long F (2014) Extensible visualization and analysis for multidimensional images using Vaa3D. Nat Protoc 9:193–208View ArticleGoogle Scholar
- Peng H, Ruan Z, Long F, Simpson JH, Myers EW (2010) V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nat Biotechnol 28:348–353View ArticleGoogle Scholar
- Peng H, Tang J, Xiao H, Bria A, Zhou J, Butler V, Zhou Z, Gonzalez-Bellido PT, Oh SW, Chen J, Mitra A, Tsien RW, Zeng H, Ascoli GA, Iannello G, Hawrylycz M, Myers E, Long F (2014) Virtual finger boosts three-dimensional imaging and microsurgery as well as terabyte volume image visualization and analysis. Nat Commun 5:4342View ArticleGoogle Scholar
- Liu Y (2011) The DIADEM and beyond. Neuroinformatics 9:99–102View ArticleGoogle Scholar
- Peng H, Hawrylycz M, Roskams J, Hill S, Spruston N, Meijering E, Ascoli GA (2015) BigNeuron: large-scale 3D neuron reconstruction from optical microscopy images. Neuron 87:252–256View ArticleGoogle Scholar
- Peng H, Zhou Z, Meijering E, Zhao T, Ascoli GA, Hawrylycz M (2017) Automatic tracing of ultra-volumes of neuronal images. Nat Methods 14:332View ArticleGoogle Scholar
- LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436View ArticleGoogle Scholar
- Gala R, Chapeton J, Jitesh J, Bhavsar C, Stepanyants A (2014) Active learning of neuron morphology for accurate automated tracing of neurites. Front Neuroanat 8:37View ArticleGoogle Scholar
- Chen H, Xiao H, Liu T, Peng H (2015) SmartTracing: self-learning-based neuron reconstruction. Brain Inform 2:135–144View ArticleGoogle Scholar
- Fakhry A, Peng H, Ji S (2016) Deep models for brain EM image segmentation: novel insights and improved performance. Bioinformatics 32:2352–2358View ArticleGoogle Scholar
- Li R, Zeng T, Ji S (2017) Deep learning segmentation of optical microscopy images improves 3D neuron reconstruction. IEEE Trans Med Imaging 36:1533–1541View ArticleGoogle Scholar
- Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
- Cheng Y (1995) Mean shift, mode seeking, and clustering. IEEE Trans Pattern Anal Mach Intell 17:790–799View ArticleGoogle Scholar
- Gong H, Xu D, Yuan J, Li X, Guo C, Peng J, Li Y, Schwarz LA, Li A, Hu B (2016) High-throughput dual-colour precision imaging for brain-wide connectome with cytoarchitectonic landmarks at the cellular level. Nat Commun 7:12142View ArticleGoogle Scholar
- Bromley J, Guyon I, LeCun Y, Säckinger E, Shah R (1994) Signature verification using a” siamese” time delay neural network. In: Advances in neural information processing systems, pp 737–744Google Scholar
- Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: IEEE computer society conference on computer vision and pattern recognition. CVPR 2005. IEEE, pp 539–546Google Scholar
- Wan Y, Long F, Qu L, Xiao H, Hawrylycz M, Myers EW, Peng H (2015) BlastNeuron for automated comparison, retrieval and clustering of 3D neuron morphologies. Neuroinformatics. https://doi.org/10.1007/s12021-12015-19272-12027 Google Scholar
- Allen Institute for Brain Science (2015) Allen cell types database. http://celltypes.brain-map.org/. Accessed Jan 2018