Managing Imports with OMERO 5
Paris 2014
Mark Carroll, Colin Blackburn & Josh Moore
Outline
- Introduction to OMERO 5
- Advanced import
- Advanced repository layouts
- Discussion
Introduction to OMERO 5
- OMERO 5 Import
- ManagedRepository
- Filesets
- Discussion
OMERO 5 Import
OMERO 5 Data out
ManagedRepository
Filesets: One to one
Filesets: One to many
Filesets: Many to many
Discussion
- Any questions so far?
- For more details of FS see last year's workshop.
Advanced import
- In-place import
- DropBox import
- Future import
- Container creation
- Cron job import
- Polling DropBox
- Discussion
In-place import: soft link
$ bin/omero import path/to/image -- --transfer=ln_s
In-place import: hard link
$ bin/omero import path/to/image -- --transfer=ln
In-place import: move
$ bin/omero import path/to/image -- --transfer=ln_rm
In-place import: move
$ bin/omero import path/to/image -- --transfer=ln_rm
In-place import: Options
$ bin/omero import -- --advanced-help- transfer and checksum options
--transfer=ARG-
upload, ln, ln_s, ln_rm, some.class.Name - default:
upload - User-provided Java class:
some.class.Name --checksum_algorithm=ARG-
Adler-32, CRC-32, MD5-128, Murmur3-32,
Murmur3-128, SHA1-160 - default (configurable):
SHA1-160
In-place import: Options
$ bin/omero import -- --advanced-help- background import options
--minutes_wait=ARG- Minutes to wait after upload before disconnecting
- <0 : don't disconnect (default)
- 0 : disconnect immediately
- >0
--close_completed--wait_completed
DropBox: in-place import
Import: users, groups and containers
Extending DropBox file monitor
- Point at any visible directory
- Run in polling mode
- Use file notification if available
Discussion
- Questions and comments on advanced import:
- Any further import options?
- What DropBox improvements would be useful?
Advanced repository layouts
- Template basics
- Template uses
- Quotas
- Samba & LDAP mapping
- Adding disks
- Discussion
Default template
Shared system portion
User portion
Template configuration
in etc/omero.properties
# Allowable elements are:
# %user% bob
# %userId% 4
# %group% bobLab
# %groupId% 3
# %year% 2011
# %month% 01
# %monthname% january
# %day% 01
# %time% 15-13-54.014
# / path separator
|
Template configuration
in etc/omero.properties
...
# The template path is created below <omero.managed.dir>,
# e.g. /OMERO/ManagedRepository/<omero.fs.repo.path>/
omero.fs.repo.path=%user%_%userId%//%year%-%month%/%day%/%time%
|
Template configuration
in etc/omero.properties
...
# The template path is created below <omero.managed.dir>,
# e.g. /OMERO/ManagedRepository/<omero.fs.repo.path>/
omero.fs.repo.path=%user%_%userId%//%year%-%month%/%day%/%time%
|
Quotas
Related forum thread
Samba: per group
Samba: per user
Adding disks
Repository Caveats
- Moving data between groups doesn't update the repository
- Archiving support is minimal; no UI or detection.
- Paths are stored in the DB. Don't move files!
Discussion
- We welcome questions and comments on FS:
- Other import scenarios?
- What further work on FS would be most useful?
- Thanks!
Extra Material
Other advanced tasks
- Find your managed repository
- pre-FS -> FS migration
- Java-less import
Finding your ManagedRepository
$ bin/omero fs repos
|
# | Id | UUID | Type | Path
---+------+--------------------------------------+---------+----------------------------------
0 | 1 | c8db8f5c-b6ba-472f-b0d5-3dac4f31be66 | Public | /home/data/ome2
1 | 2 | ScriptRepo | Script | /opt/ome2/dist/lib/scripts
2 | 3 | c4df3508-cffe-4bf2-a1c8-b3c1395f737f | Managed | /home/data/ome2/ManagedRepository
(3 rows)
|
pre-FS -> FS migration
- OMERO4 and before allowed archiving original files
bin/omero fs images --archived- Preparing user script to help conversion
- Limitations
Java-less import
