Data repository

OME Tuesday meeting

15th September 2015

S├ębastien Besson

Bio-Formats metrics

Presentation Outline

  • Data repository structure
  • Bio-Formats automated tests
  • Future work

OME repository

Squig mounted as /ome/

Folder Description
apache_repo QA file store
ci Continuous Integration repository (reference DBs)
data_repo Data repository
Documentation Model documentation
team Miscellaneous internal content
www Static Web content (downloads, schemas)

QA file store

  • File uploaded via QA application
  • Stored under /ome/apache_repo/11378/Example_MJ.dm4

Data repository

Folder Description Symlinks
ci / config Continuous Integration no
from_* File format datasets no
from_skyking Curated datasets no
test_images_* Testing subsets of from_skyking yes
test_images_good Testing subset yes
public Public datasets yes

Curated datasets

  • Root directory /ome/data_repo/from_skyking
    • Datasets primarily organized by format
    • Subfolders contains one of multiple filesets
  • Group readable/not writeable
  • QA datasets deep copied
  • Tested daily by the Bio-Formats automated tests

Bio-Formats automated tests

  • See the Bio-Formats developer documentation
  • Ant automated-tests target
    • Scans files under root directory testng.repository
    • Scans matching configuration files under root configuration directory testng.configRepository
    • Runs series of tests on each configured fileset including metadata reading, bytes opening

Bio-Formats automated tests

Continuous Integration repository jobs

Bio-Formats automated tests

  • Full repository jobs
    • Testing the curated datasets repository
    • Both Linux and Windows
    • Running daily (or every 2 days for Windows)
    • 12K files, 300K tests
  • Test images good jobs
    • Testing the test_images_good datasets repository
    • 675 files, 22K tests
  • Repository subset jobs
    • Merges Bio-Formats PRs
    • Logic to test one or multiple formats
    • Primarily used for development

Bio-Formats automated tests

Run automated tests against one (or multiple) file formats

Configuration repository

  • Private repository hosting the .bioformats configuration files for the automated tests
  • Configuration files can be modified by PR workflow
  • Merged as part of the daily job suite - see DATA_REPO_CONFIG-merge
  • Tagged as part of the release process
  • Improves process during significant reader modifications (Leica LIF)

Future work

  • Data repository
    • Towards more parallelization using new IDR hardware
    • Background maintenance (curation, cleanup...)
  • Configuration repository
    • More metadata support
    • Usage for stats
    • A tool for searching data?

Thank you