Managing Imports with OMERO 5

Paris 2014

Mark Carroll, Colin Blackburn & Josh Moore

Outline

  • Introduction to OMERO 5
  • Advanced import
  • Advanced repository layouts
  • Discussion

Introduction to OMERO 5

  • OMERO 5 Import
  • ManagedRepository
  • Filesets
  • Discussion

OMERO 5 Import

OMERO 5 Data out

ManagedRepository

Filesets: One to one

Filesets: One to many

Filesets: Many to many

Discussion

Advanced import

  • In-place import
  • DropBox import
  • Future import
    • Container creation
    • Cron job import
    • Polling DropBox
  • Discussion

In-place import: soft link

$ bin/omero import path/to/image -- --transfer=ln_s

In-place import: hard link

$ bin/omero import path/to/image -- --transfer=ln

In-place import: move

$ bin/omero import path/to/image -- --transfer=ln_rm

In-place import: move

$ bin/omero import path/to/image -- --transfer=ln_rm

In-place import: Options

  • $ bin/omero import -- --advanced-help
  • transfer and checksum options
    • --transfer=ARG
      • upload, ln, ln_s, ln_rm, some.class.Name
      • default: upload
      • User-provided Java class: some.class.Name
    • --checksum_algorithm=ARG
      • Adler-32, CRC-32, MD5-128, Murmur3-32,
        Murmur3-128, SHA1-160
      • default (configurable): SHA1-160

In-place import: Options

  • $ bin/omero import -- --advanced-help
  • background import options
    • --minutes_wait=ARG
      • Minutes to wait after upload before disconnecting
      • <0 : don't disconnect (default)
      • 0 : disconnect immediately
      • >0
    • --close_completed
    • --wait_completed

DropBox: in-place import

Import: users, groups and containers

Extending DropBox file monitor

  • Point at any visible directory
    • Run in polling mode
    • Use file notification if available

Discussion

  • Questions and comments on advanced import:
    • Any further import options?
    • What DropBox improvements would be useful?

Advanced repository layouts

  • Template basics
  • Template uses
    • Quotas
    • Samba & LDAP mapping
    • Adding disks
  • Discussion

Default template

Template diagram

Shared system portion

Shared template portion

User portion

User template portion

Template configuration

in etc/omero.properties

# Allowable elements are:
# %user%       bob
# %userId%     4
# %group%      bobLab
# %groupId%    3
# %year%       2011
# %month%      01
# %monthname%  january
# %day%        01
# %time%       15-13-54.014
# /            path separator
                        

Template configuration

in etc/omero.properties

...
# The template path is created below <omero.managed.dir>,
# e.g. /OMERO/ManagedRepository/<omero.fs.repo.path>/
omero.fs.repo.path=%user%_%userId%//%year%-%month%/%day%/%time%
                        

Template configuration

in etc/omero.properties

...
# The template path is created below <omero.managed.dir>,
# e.g. /OMERO/ManagedRepository/<omero.fs.repo.path>/
omero.fs.repo.path=%user%_%userId%//%year%-%month%/%day%/%time%
                        

Quotas

Template starting with groupId Related forum thread

Samba: per group

Samba usage diagram

Samba: per user

Samba usage diagram

Adding disks

Template starting with drive

Repository Caveats

  • Moving data between groups doesn't update the repository
  • Archiving support is minimal; no UI or detection.
  • Paths are stored in the DB. Don't move files!

Discussion

  • We welcome questions and comments on FS:
    • Other import scenarios?
    • What further work on FS would be most useful?
  • Thanks!

Extra Material

Other advanced tasks

  • Find your managed repository
  • pre-FS -> FS migration
  • Java-less import

Finding your ManagedRepository

$ bin/omero fs repos
 # | Id   | UUID                                 | Type    | Path
---+------+--------------------------------------+---------+----------------------------------
 0 | 1    | c8db8f5c-b6ba-472f-b0d5-3dac4f31be66 | Public  | /home/data/ome2
 1 | 2    | ScriptRepo                           | Script  | /opt/ome2/dist/lib/scripts
 2 | 3    | c4df3508-cffe-4bf2-a1c8-b3c1395f737f | Managed | /home/data/ome2/ManagedRepository
(3 rows)
                        

pre-FS -> FS migration

  • OMERO4 and before allowed archiving original files
  • bin/omero fs images --archived
  • Preparing user script to help conversion
  • Limitations

Java-less import