Release notes
Updated required versions of tensorflow and numpy. cellfinder now requires CUDA 11.0.
A csv of every detected cell is now exported with coordinates and brain region
Fixed a bug causing cell distribution analysis to be rerun if already completed.
Updated cell export to be compatible with brainrender v0.2.0
Testing and deployment has moved from TravisCI to GitHub actions
​
Fixes a bug that prevents cells from being detected if the background intensity of the image is zero.
Fixed a bug in cellfinder_curate
Updated data export to be compatible with brainrender 1.1
Fixed a bug in the final data export
Minor bug fixes
Minor bug fixes
cellfinder is now part of the BrainGlobe suite of computational neuroanatomy tools, and the documentation has moved here.
The registration in cellfinder has moved from amap to brainreg, which uses the BrainGlobe atlas API to suppport many more atlases.
The default atlas is now the 25um Allen Mouse Brain atlas, but these can be chosen by using the
--atlasflag. Runbrainglobe listto see which atlases are available.cellfinder has now adopted the bg-space convention for inputting the orientation of your data, using the
--orientationflag. For more details see Image definition​The
-x,-y,-zand--metadataflags have now been removed. See Image definition for how to enter this information.manual_seghas now been replaced with a new (brainreg-based) tool, brainreg-segment.Support for Python 3.6 has been dropped, and support for Python 3.8 added (this is now the recommended Python version).
Data visualisation has now moved to napari plugins, see Visualisation.
--save-progressflag added for training. Saves training progress to a .csv file (output_directory/training.csv).Added an easy way to find the currently installed version (
cellfinder --version)
During training, the model will be saved after each epoch by default. Use
--no-save-checkpointsto suppress this behaviour and save disk space. Each model file can be large, and if you don't have much training data, they can be generated quickly.
Removed many standalone tools due to the behaviour being included within the main cellfinder program (or no longer have a use):
cellfinder_cell_standardcellfinder_count_summarycellfinder_region_summarycellfinder_xml_cropcellfinder_xml_scalecellfinder_gen_region_volcellfinder_cells_to_brainrender
New combined cell detection and registration viewer (see Visualisation)
There is a new and improved GUI for manually segmenting linear tracks and volumes in standard space (see Manually segment in standard space).
Registration can now deal with data in a non-standard orientation, see Image orientation​
Detected cell positions can now be manually curated and saved (see Visualisation).
Bug causing data to occasionally be loaded in the incorrect order for registration is fixed
amap_vishas been replaced withcellfinder_viewcellfinder_view_cellshas been replaced withcellfinder view
Support is now officially for Windows and Linux. macOS should generally work, but there may be issues.
Bugs left in
cellfinder_cell_standardafter refactoring main code are fixed.Compatible with napari v0.3.0
cellfinder_view_3Dhas been removed
Windows bug fix thanks to @EmanPaoli​
Prebuilt wheels now released for Linux, Windows and macOS (untested)
Update dependencies & refactor usage of amap api
Update dependencies
Minor refactoring
Training can now be saved every N epochs using --checkpoint-interval
Bug fixed in
cellfinder_xml_cropPandas dependencies updated thanks to @amedyukhina​
Fix memory leak in training
Update curation interface
New all python training data generation interface,
cellfinder_curate. Details here.
Heatmap and converting cells to brainrender format moved to neuro​
Lots of utility functions moved to imlib​
Automatic deployment with Travis
Very basic Windows support (allthough some issues remain.)
Pixel sizes are now defined in um (rather than mm). Users specifying a metadata file will not be affected. Those specifying the pixel sizes manually will need to change their usage. Flags such as
--x-pixel-mmare now--x-pixel-um.All cell detection parameters (e.g.
--soma-diameter) are now defined in um or um3 (rather than mm)Registration parameters are now defined via command-line arguments (e.g.
--freeform-use-n-steps) rather than a config file.Installation is simplified (no need to install tensorflow separately)
Updated default parameters to better work on noisy data.
Memory consumption during inference reduced
Registration code is now in amap​
Compatibility with tensorflow 2.1.0
Cube extraction is now carried out on the fly during classification (and not saved to disk). This prevents file system issues due to the generation of (potentially) hundreds of thousands of cells.
Likelihood during training of each individual augmentation transformation reduced to 10% (from 50%)
Bug fix in
cellfinder_view_cellsto deal with Napari API change
New viewer to inspect batches of generated cubes
(
cellfinder.classify.tools.batch_viewer)Removed some old, unnecessary tests
Fixed a bug in
tests/tests/test_integration/test_detection.py
Removed NiftyNet dependency. Classification is now implemented in
tf.keras.THIS MEANS YOU NEED TO TRAIN A NEW MODEL. OLD MODELS ARE INCOMPATIBLE!
New
cellfinder_view_cellscommand which uses napari to view cells overlaid on raw data.New
cellfinder_gen_region_volcommand line tool to generate images of specific brain regions (for figures etc)roi_transformnow supports ROIs generated in the original, full-size imagesNew
cellfinder_cells_to_brainrenderto allow classified cells to be viewed in BrainRender​
cellfinder_view_cells2D function is removed, and replaced with cellfinder_view_cells.
Cube scaling is now 3D (theoretically leading to better classification
performance)
cellfinder_view_3Dnow uses napari.
Brain image IO moved to brainio​
Reading a slurm environment (if present) moved to
​slurmio.
Logging moved to fancylog​
Added roi_transform command line tool to transform ImageJ ROI sets into images in standard space.
Fixes a bug causing heatmaps to be scaled incorrectly
Only minor changes to documentation and testing.
New
--max-ramto specify a maximum amount of RAM to use (in GB). Currently only affects processes that will scale to use as much RAM as is available (such as cube extraction).New
cellfinder_xml_scalecommand line tool to rescale the cell positions within an XML file. For compatibility with other software, or if your data has been scaled after cell detection.
THIS VERSION HAS NOT BEEN FULLY TESTED
Instructions for installing CUDA 9 and cuDNN via conda added.
cellfinder_region_summarynow has a--sum-regionsflag combine child regions.
Bug which caused heatmaps to be calculated incorrectly is fixed.
Bug which causes
--regions-listinput tocellfinder_region_summaryto be parsed incorrectly is fixed.Bug which crashed summary csv generation if a cell is transformed incorrectly due to rounding errors is fixed.
Bug which crashed volume calculation in registration if a region is missing is fixed.
Bug causing registration to crash if a custom registration file without a
base_folderentry is fixed.urllib3version is specified to prevent atlas downloading from failing.
A bug which caused the atlas not be downloaded when using
cellfinder_registeris fixed.Registration now respects the
--n-free-cpusflag
Python 3.5 is no longer supported.
Detected cell positions can now be transformed into standard space. By default, if cell classification and registration has been performed, an additional
cells_in_standard_space.xmlfile will be generated. Use--no-standard_spaceto prevent this behaviour. This functionality can also be run using the standalone scriptcellfinder_cell_standard.
Pixel sizes must still be specified, but instead of using the
-x,-y, and-zflags, a--metadataflag can be used instead. Supported formats include BakingTray recipe files, mesoSPIM metadata files or cellfinder custom metadata files (see below). If both pixel sizes and metadata are provided, the command line arguments will take priority.
An installation bug (due to new requirements of gputools is fixed.)
The main command-line interface to cellfinder is now
cellfinder,cellfinder_runhas been removed (and may call an old version of the software). Please usecellfinderfrom now on.
Bug in
cellfinder_gen_cubeshas been fixed
The main command-line interface to cellfinder will be
cellfinder,cellfinder_runwill stop working soon (and may call an old version of the software). Please usecellfinderfrom now on.
THIS VERSION HAS NOT BEEN FULLY TESTED Use Version 0.2.3a if you don't need the new features
Pixel sizes must now be specified with
-x,-y, and-z.
cellfinder.IO.cells.get_cells()can now read the.ymlfiles output from the cell counting plugin of MaSIV.Registration will now calculate a
volumes.csvfile which contains the volumes of each brain area in the atlas, in the sample brain.The
--summariseflag will now save more information, including:Cell counts combined across hemispheres
Cell densities in cells per mm3.
Registration will also calculate the reverse transforms (i.e. sample -> atlas)
A new
--figuresflag will create a subdirectory of 3D images in the coordinate space of the downsampled raw data (and registered atlas) that can be used to generate figures e.g. in FIJI. Currently three files will be generated by default.heatmap.nii - This is a 3D histogram, showing cell densities
throughout the brain. Likely to be more useful than visualising cellpositions for dense areas. By default, this is smoothed, and masked (so that the smoothing doesn't "bleed" outside of the brain).
glass_brain.nii This is a binary image, just showing the brain surface. Plotting this in 3D along with heatmap.nii gives a nice sense of cell density throughout the brain. ClearVolume works well for this.
boundaries.nii Another binary image, but showing the divisions between each region in the atlas. More useful for smaller, 3D figures to delineate region boundaries. Also useful to assess registration by overlaying on the raw, downsampled data.
Bug on certain systems where cube extraction could use too much RAM, and crash the system has been fixed.
Output directory structure has been reorganised, and simplified slightly.
Code coverage increased to 58%.
A lot of the code has been refactored, and (hopefully simplified).
Log files will now print more information (e.g. information about the git repository, and internal parameters).
Registration can now handle data in any orientation. The default is coronal, and in the NifTI standard orientation (origin is the most ventral, posterior, left voxel). If your data does not match this, use the
--orientation,--flip-x,--flip-yand--flip-ztags.
There is now a standalone registration command,
cellfinder_register. This will be the maintained python version of aMAP going forward.There is now an option
--remove-intermediateto remove all intermediate files generated by cellfinder. Use with caution.
All downsampled data, registered atlases and cell locations are in the same coordinate space as the raw data (although possibly scaled). This fixes issues #22 and #23, and allows overlay of the cell positions on the downsampled data from the registration step.
Cube extraction will now use as many CPU cores as available, slightly speeding up this step. However, if you don't have much RAM (or a lot of CPU cores) see #36.
Input data paths no longer need to end in the channel number. For channel referencing purposes, this will be set automatically, or can be overridden with
--signal-channel-idsand--background-channel-id(or--channel-idsforcellfinder_gen_cubes).By default, registration will now save downsampled versions of all channels. if you don't want these files, use
--no-save-downsampled.Cube extraction will now default to extracting cubes of 50um x 50um x 100um. This can be overridden with
--x_pixel_mm_networkand--y_pixel_mm_network. Currently there is no rescaling in z.Cube extraction now supports the output of ROI sorter, rather than just xml.
Code coverage increased to 32%.
Logging now includes diagnostic information.
Cellfinder can now be installed with a single command:
pip install git+https://github.com/adamltyson/cellfinderThe atlas will now no longer be downloaded automatically upon installation of cellfinder, but on the first use of the atlas. To download in advance,
cellfinder_downloadcan be used.
New function to load
.nrrdfiles (incellfinder.tools.brain.io.load_any).Data can now be passed as a directory of 2D .tif files (rather than just a text file of file paths).
If a network was trained with a cubes of a different pixel size to that of the input data, then the pixel spacing of the output cubes can be changed (e.g. using:
--x-pixel-mm-network).Cell (and candidate) positions can be saved as
.csv(as well as.xml), by using--save-csv.cellfinder_xml_crophas been added. This curates an input.xmlfile and outputs a file with only those positions within given anatomical areas. Currently, the cells, and the atlas need to be in the same anatomical space.
Bug which caused cellfinder to hang when using part of a node managed by
SLURM is fixed.
If a cube cannot be extracted for any reason (such as proximity to the edge of an image), the default is now to not save it (rather than save an empty cube in it's place). This can be overridden with
--save-empty-cubes.Error messages when a cube cannot be extracted (due to proximity to the edge of the image) have been clarified.
Code is now automatically formatted with Black, which requires Python >= 3.6 to run.
Testing has been added (although currently there is only one test).
All commits & pull requests will be build by Travis.
For development, cellfinder should be installed using
pip install -e .[dev]in the repository.
Simple summary information can be generated with the
--summariseflag. This will run registration (if it wasn't run already) and associate each cell output from the cell detection step with a brain region. This info is saved as a csv file.The mouse brain atlas can be downloaded to any directory when installing cellfinder using the
--atlas-install-pathflag. This flag can also be used to point to an existing atlas installation (e.g. from aMAP).This will update the default config file so the correct atlas is sourced (and a second isn't downloaded.)
cellfinder should know what parts have already been run, so if you re-run a
cellfinder_runcommand, it won't repeat any parts of the pipeline.
PDF documentation
cellfinder_gen_cubesfunction to generate tiff cubes from an xml file independent of the rest of cellfinder. Useful for generating training data.cellfinder_count_summaryfunction to combine a brain registered to the allen atlas with cell counts. Generates a csv file of cells per region.cellfinder_region_summaryfunction to organise (align by brain area) csv files of summary cell counts from multiple brains.cellfinder_view_cells2Dfunction to view cells overlaid on raw data. Uses hardlyany RAM, but I recommend ROI sorterinstead.cellfinder_view_3Dfunction. A very rudimentary 3D viewer, but can load any file type in use by cellfinder.Some API docs
Logging that interfered with multiprocessing when the
--verbosetag was used has been emoved. This was an issue in case cellfinder was stopped in the middle of cell classification as it could cause GPU memory to not be released (and not show up innvidia-smi).
Documentation moved from pydocmd to sphinx.
Installation requires configobj (as well as Cython).
Install with
--devif you want to build the documentation yourself.
Can run cell detection on N-channels
Streamlined training of cell classification network via
cellfinder_trainIncludes (rudimentary) integration of aMAP for registration to the Allen brain atlas
Further options to only run parts of the analysis
Option
--tensorboardto launch tensorboard automatically during trainingEntry point
cellfinder_viewto launch the (very simple) image/cell viewerDocs generated via pydocmd
Output of cell candidate detection step
cells.xmlcan now be opened in Roi Sorter​
Docs generated by pydocmd means that installation info is currently here​
First version compatible with NiftyNet 0.5.0, TensorFlow 1.12.0 and therefore CUDA 9/10.
Option to specify external configuration file for cell candidate classification
All results saved to a single directory
Option to keep or discard bright spots classified as artifact during cell candidate detection
Log file
Z positions of cell candidates when analysing a z-subset of the data fixed in the resulting
.xmlfileCell candidate detection .xml file only provides positive candidate types
<Type>1</Type>for compatibility with Roi Sorter​
No longer relies on NiftyNet model zoo (although this may return)
Much simplified installation (no manual cp/mv)
Moving towards a single API, calls to n_count_python etc. broken
Multiprocessing compatible with shared resources (e.g. HPC job schedulers)
Developed with Python 3.5, but should work with 3.6/3.7. no Python 2 compatibility
HPC compatibility only tested with SLURM, but should work with LSF & SGE
Final version compatible with niftynet 0.2.2 and tensorflow 1.4.1 (and therefore CUDA 8).
