Idea: Gold Standards for Microscopy Data - List
About
The idea of the list below is to help is to create a set of standards by which software and image files in microscopy can be measured.
List of All Standards
I encourage you to add to, edit and discuss the list of standards below. Note that I've tried to separate out these standards into sections: depending on what a program does, not all these standards will apply. I've tried to arrange standards roughly in order of importance, but I'm sure it will all get reordered, and that we'll also add many more (eg: standards for databases etc). Something to keep in mind though: it's important this list doesn't become so long it gets unmanageable. Only the most important standards should make the final cut - certain program attributes such as "easy-to-use" we can talk about in guidelines, but unfortunately are probably too subjective to measure and thus list as a standard.
TIP: When reading this list it's pretty normal to be defensive and think: "damn, my program doesn't meet this standard" and/or want to bias these standards to suit your software. I'd urge you to try and be objective! These standards are there to help people create an "ideal" piece of software. If we could redesign a piece of software there are always definitely things we'd want to do differently!
A) Standards for All Software
NOTE: Standards in this section apply to ALL software
- a1) Actively maintained - without the commitment of a primary author to maintain and promote his software over many years, that software is destined to die, hence this is top of the list.
- Gold: Software is funded by a grant for several years (for at least primary programmer).
- Silver: Software has been established >2 years, and continues to be maintained by committed programmer.
- a2) Easy to install - if you can't install in a couple of clicks, most biologists won't bother.
- Gold: Features one-click install or runs in browsers.
- Silver: Requires additional third party plugins unlikely to be pre-installed (eg: Java).
- a3) Cross-platform - many biologists use Mac, many use Window and many labs use Linux.. hence working on more than one platform is very important.
- Gold: Will work on almost all platforms (Windows, Mac and some breeds of Linux) and includes 64 bit versions.
- Silver: Will work on Windows and Mac (without emulators).
- a4) Well documented - '.
- Gold: Includes video documentation.
- Silver: Includes hierarchy of help files within program.
- a5) Good support - even if documentation is great, biologists like to know there are people they can rely on when questions arise.
- Gold: Has mailing list and/or online forum where main authors answer questions.
- Silver: Contains online help (preferably as a wiki other can add too).
- a6) Open source - '.
- Gold: Can download up-to-date source code from a repository such as source forge or google code.
- Silver: Can download source code (with GNU license) from a site.
- a7) Plugin friendly - '.
- Gold: Features special structures where new GUI and code can be contributed.
- Silver: Contains documentation showing how you can contribute your own code.
B) Standards for Image Files
NOTE: These standards apply to any programs that or loads or saves image files.
- b1) Metadata Meets Min Requirements
- Gold: Primary/default files includes scale, microscopy, author and providence standards.
- Silver: Primary files includes all relevant scale information and some providence.
- b2) Supports Recommended Image Standards
- Gold: Primary file is saved (by default) to recommended standard (JPEG 2000/BigTIFF).
- Silver: Software can import AND export recommended standard (JPEG 2000/BigTIFF).
- b3) Intelligent Loading
- Gold: Slices at relevant scale are loaded dynamically at relevant zoom, and option exist to pre-load/fill memory.
- Silver: Software can detect file size and knows to open (>4GB) images by binning and/or loading subareas (as chosen by user).
C) Standards for Segmentation Files / Analysis
NOTE: These standards apply to any program that performs analysis on, or that load/saves any type of segmentation (image segmentation/contours/surfaces/etc)
- c1) Metadata Meets Min Requirements
- Gold: Primary file includes scale, microscopy, author and providence standards.
- Silver: Primary file includes all relevant scale information (and hopefully some link to image file).
- c2) Object Tied to Ontology
- Gold: Each "object" MUST be given a uniqueID which ties to a INCF approved ontology (eg: neurolex.org or geneontology.org).
- Silver: Object can be given uniqueIDs but this is not enforced.
- c3) Supports Object Hierarchy
- Gold: Objects exist in hierarchy, and each object can also have tags (eg: "uniqueID") to identify one or more layers it belongs to.
- Silver: Objects can be positioned/rearranged in hierarchy/tree of any depth (where properties can be inherited).
- c4) Spatial Data Structure
- Gold: In addition to maintaining a bounding box, each geometry is indexed by a 3D spatial data structure such as an oct-tree or uniform grid.
- Silver: Each geometry maintains its own bounding box around points for timely visualization and analysis.
D) Standards for Databases
NOTE: These standards apply to anything system that is or uses a relational database... and someone else might have to complete it!
- d1) Metadata Meets Min Requirements
- Gold: Primary file includes scale, microscopy, author and providence standards.
- Silver: Primary file includes all relevant scale information (and hopefully some link to image file).
... PLEASE ADD MORE (ALL THIS IS UNDER DEVELOPMENT)
Software Evaluation Table
The table below is hopefully something which will present inventive for new and existing software to adhere to a common set of standards. In the table below each column represents a standard (roughly in ascending order of importance) by which we hope to measure software. The actual list of standards and their order is up for debate, but for now I've put this up as an example!
|
Evaluation of Popular Segmentation Tools | |||
|---|---|---|---|
| Standard | IMOD | Program2 | Comment |
|
A) Standards for All Software | |||
| a1) Actively maintained |  |
 |
|
| a2) Easy to install | |
 |
|
| a3) Cross-platform | |
|
|
| a4) Well documented | |
|
|
| a5) Good support | |
|
|
| a6) Open source | |
|
|
| a7) Plugin friendly | |
|
|
|
B) Standards for Image Files | |||
| b1) Metadata Meets Min Requirements | |
|
|
| b2) Supports Recommended Image Standards | |
|
|
| b3) Intelligent Loading | |
|
|
|
C) Standards for Segmentation Files / Analysis | |||
| c1) Metadata Meets Min Requirements | |
|
|
| c2) Object Tied to Ontology | |
|
|
| c3) Supports Object Hierarchy | |
|
|
| c4) Spatial Data Structure | |
|
|
|
SCORE | |||
| TOTAL SCORE | 5 gold 2 silver =43% (6/14) |
0 gold 5 silver =18% (2.5/14) |
NOTE: In this scheme silver stars are counted as half a gold star. |
Standards Table
Note that this table is the same as the table above, but transposed. In my opinion the first table is a little easier to read, but I've left this here anyway.
|
A) All Software |
B) Image Files |
C) Segmentation |
D) Other | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Software | Gold: Primary file includes scale, microscopy, author and providence standards.
Silver: Primary file includes all relevant scale information (and hopefully some link to image file). |
blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | blah blah | Total Score |
| IMOD |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
N/A | 5 gold + 2 half-stars = 43% (6/14) |
| Program2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
N/A | 0 gold + 5 half-stars = 18% (2.5/14) |
