Idea: Gold Standards for Microscopy Data - List

From NoskeWiki
Jump to: navigation, search

About

NOTE: This page is a daughter page of: Gold Standards for Microscopy Data


The idea of the list below is to help is to create a set of standards by which software and image files in microscopy can be measured.


List of All Standards

I encourage you to add to, edit and discuss the list of standards below. Note that I've tried to separate out these standards into sections: depending on what a program does, not all these standards will apply. I've tried to arrange standards roughly in order of importance, but I'm sure it will all get reordered, and that we'll also add many more (eg: standards for databases etc). Something to keep in mind though: it's important this list doesn't become so long it gets unmanageable. Only the most important standards should make the final cut - certain program attributes such as "easy-to-use" we can talk about in guidelines, but unfortunately are probably too subjective to measure and thus list as a standard.


TIP: When reading this list it's pretty normal to be defensive and think: "damn, my program doesn't meet this standard" and/or want to bias these standards to suit your software. I'd urge you to try and be objective! These standards are there to help people create an "ideal" piece of software. If we could redesign a piece of software there are always definitely things we'd want to do differently!



A) Standards for All Software

NOTE: Standards in this section apply to ALL software

  • a1) Actively maintained - without the commitment of a primary author to maintain and promote his software over many years, that software is destined to die, hence this is top of the list.
    • Gold: Software is funded by a grant for several years (for at least primary programmer).
    • Silver: Software has been established >2 years, and continues to be maintained by committed programmer.
  • a2) Easy to install - if you can't install in a couple of clicks, most biologists won't bother.
    • Gold: Features one-click install or runs in browsers.
    • Silver: Requires additional third party plugins unlikely to be pre-installed (eg: Java).
  • a3) Cross-platform - many biologists use Mac, many use Window and many labs use Linux.. hence working on more than one platform is very important.
    • Gold: Will work on almost all platforms (Windows, Mac and some breeds of Linux) and includes 64 bit versions.
    • Silver: Will work on Windows and Mac (without emulators).
  • a4) Well documented - '.
    • Gold: Includes video documentation.
    • Silver: Includes hierarchy of help files within program.
  • a5) Good support - even if documentation is great, biologists like to know there are people they can rely on when questions arise.
    • Gold: Has mailing list and/or online forum where main authors answer questions.
    • Silver: Contains online help (preferably as a wiki other can add too).
  • a6) Open source - '.
    • Gold: Can download up-to-date source code from a repository such as source forge or google code.
    • Silver: Can download source code (with GNU license) from a site.
  • a7) Plugin friendly - '.
    • Gold: Features special structures where new GUI and code can be contributed.
    • Silver: Contains documentation showing how you can contribute your own code.


B) Standards for Image Files

NOTE: These standards apply to any programs that or loads or saves image files.

  • b1) Metadata Meets Min Requirements
    • Gold: Primary/default files includes scale, microscopy, author and providence standards.
    • Silver: Primary files includes all relevant scale information and some providence.
  • b2) Supports Recommended Image Standards
    • Gold: Primary file is saved (by default) to recommended standard (JPEG 2000/BigTIFF).
    • Silver: Software can import AND export recommended standard (JPEG 2000/BigTIFF).
  • b3) Intelligent Loading
    • Gold: Slices at relevant scale are loaded dynamically at relevant zoom, and option exist to pre-load/fill memory.
    • Silver: Software can detect file size and knows to open (>4GB) images by binning and/or loading subareas (as chosen by user).


C) Standards for Segmentation Files / Analysis

NOTE: These standards apply to any program that performs analysis on, or that load/saves any type of segmentation (image segmentation/contours/surfaces/etc)

  • c1) Metadata Meets Min Requirements
    • Gold: Primary file includes scale, microscopy, author and providence standards.
    • Silver: Primary file includes all relevant scale information (and hopefully some link to image file).
  • c2) Object Tied to Ontology
    • Gold: Each "object" MUST be given a uniqueID which ties to a INCF approved ontology (eg: neurolex.org or geneontology.org).
    • Silver: Object can be given uniqueIDs but this is not enforced.
  • c3) Supports Object Hierarchy
    • Gold: Objects exist in hierarchy, and each object can also have tags (eg: "uniqueID") to identify one or more layers it belongs to.
    • Silver: Objects can be positioned/rearranged in hierarchy/tree of any depth (where properties can be inherited).
  • c4) Spatial Data Structure
    • Gold: In addition to maintaining a bounding box, each geometry is indexed by a 3D spatial data structure such as an oct-tree or uniform grid.
    • Silver: Each geometry maintains its own bounding box around points for timely visualization and analysis.


D) Standards for Databases

NOTE: These standards apply to anything system that is or uses a relational database... and someone else might have to complete it!

  • d1) Metadata Meets Min Requirements
    • Gold: Primary file includes scale, microscopy, author and providence standards.
    • Silver: Primary file includes all relevant scale information (and hopefully some link to image file).

... PLEASE ADD MORE (ALL THIS IS UNDER DEVELOPMENT)



Software Evaluation Table

The table below is hopefully something which will present inventive for new and existing software to adhere to a common set of standards. In the table below each column represents a standard (roughly in ascending order of importance) by which we hope to measure software. The actual list of standards and their order is up for debate, but for now I've put this up as an example!


Evaluation of Popular Segmentation Tools

Standard IMOD Program2 Comment

A) Standards for All Software

a1) Actively maintained a1 a1
  • Gold: Software is funded by a grant for several years (for at least primary programmer).
  • Silver: Software has been established >2 years, and continues to be maintained by committed programmer.
a2) Easy to install a2 a2
  • Gold: Features one-click install or runs in browsers.
  • Silver: Requires additional third party plugins unlikely to be pre-installed (eg: Java).
a3) Cross-platform a3 a3
  • Gold: Work on almost all platforms (Windows, Mac and some breeds of Linux) and includes 64 bit versions.
  • Silver: Work on Windows and Mac (without emulators).
a4) Well documented a4 a4
  • Gold: Includes video documentation.
  • Silver: Includes hierarchy of help files within program.
a5) Good support a5 a5
  • Gold: Has mailing list and/or online forum where main authors answer questions.
  • Silver: Contains online help (preferably as a wiki other can add too).
a6) Open source a6 a6
  • Gold: Can download up-to-date source code from a repository such as source forge or google code.
  • Silver: Can download source code (with GNU license) from a site.
a7) Plugin friendly a7 a7
  • Gold: Features special structures where new GUI and code can be contributed.
  • Silver: Contains documentation showing how you can contribute your own code.

B) Standards for Image Files

b1) Metadata Meets Min Requirements b1 b1
  • Gold: Primary/default files includes scale, microscopy, author and providence standards.
  • Silver: Primary files includes all relevant scale information and some providence.
b2) Supports Recommended Image Standards b2 b2
  • Gold: Primary file is saved (by default) to recommended standard (JPEG 2000/BigTIFF).
  • Silver: Software can import AND export recommended standard (JPEG 2000/BigTIFF).
b3) Intelligent Loading b3 b3
  • Gold: Slices at relevant scale are loaded dynamically at relevant zoom, and option exist to pre-load/fill memory.
  • Silver: Software can detect file size and knows to open (>4GB) images by binning and/or loading subareas (as chosen by user).

C) Standards for Segmentation Files / Analysis

c1) Metadata Meets Min Requirements c1 c1
  • Gold: Primary file includes scale, microscopy, author and providence standards.
  • Silver: Primary file includes all relevant scale information (and hopefully some link to image file).
c2) Object Tied to Ontology c2 c2
  • Gold: Each "object" MUST be given a uniqueID which ties to a INCF approved ontology (eg: neurolex.org or geneontology.org).
  • Silver: Object can be given uniqueIDs but this is not enforced.
c3) Supports Object Hierarchy c3 c3
  • Gold: Objects exist in hierarchy, and each object can also have tags (eg: "uniqueID") to identify one or more layers it belongs to.
  • Silver: Objects can be positioned/rearranged in hierarchy/tree of any depth (where properties can be inherited).
c4) Spatial Data Structure c4 c4
  • Gold: In addition to maintaining a bounding box, each geometry is indexed by a 3D spatial data structure such as an oct-tree or uniform grid.
  • Silver: Each geometry maintains its own bounding box around points for timely visualization and analysis.

SCORE

TOTAL SCORE 5 gold
2 silver

=43% (6/14)
0 gold
5 silver

=18% (2.5/14)
NOTE: In this scheme silver stars are counted as half a gold star.



Standards Table

Note that this table is the same as the table above, but transposed. In my opinion the first table is a little easier to read, but I've left this here anyway.


A) All Software

B) Image Files

C) Segmentation

D) Other

Software
Gold: Primary file includes scale, microscopy, author and providence standards.
Silver: Primary file includes all relevant scale information (and hopefully some link to image file).
blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah blah Total Score
IMOD a1 a2 a3 a4 a5 a6 a7 b1 b2 b3 c1 c2 c3 c4 N/A 5 gold + 2 half-stars = 43% (6/14)
Program2 a1 a2 a3 a4 a5 a6 a7 b1 b2 b3 c1 c2 c3 c4 N/A 0 gold + 5 half-stars = 18% (2.5/14)