Large Import: Lessons from IDR

OME Community Meeting
Dundee May 2020
OME Team

Import possibilities

1. Desktop Client
2. CLI:
   • In-place Import - saves space
   • Bulk Import - repeatable, easy automation

Import into OMERO

In-place import: symlink

ManagedRepository

For Demo, user importing is...

• A Facility Manager: importer1 i.e. an OMERO user with some restricted admin privileges
• who has Write access to the ManagedRepository

Bulk Import CLI

• How to import: bulk.yml

continue: "true"
transfer: "ln_s"
checksum_algorithm: "File-Size-64"
logprefix: "logs/"
output: "yaml"
path: "idr0021.tsv"
columns:
    - target
    - path

• What to import: idr0021.tsv

Dataset:name:CDK5RAP2-C /OMERO/in-place-import/idr0021-lawo-pericentriolarmaterial/Raw-files/CDK5RAP2-C/Centrin_PCNT_Cep215_20110506/Centrin_PCNT_Cep215_20110506_Fri-1545_0_SIR_PRJ.dv
Dataset:name:CDK5RAP2-C /OMERO/in-place-import/idr0021-lawo-pericentriolarmaterial/Raw-files/CDK5RAP2-C/Centrin_PCNT_Cep215_20110506/Centrin_PCNT_Cep215_20110506_Fri-1559_0_SIR_PRJ.dv
...

Post Import Data Manipulation: Demo...

• CLI: Render data
  • How to render: renderingdef.yml & renderingdef2.yml

  
  # Specification of rendering settings
  
  channels:
    1:
      # red
      color: "FF0000"
      label: "CEP192-M"
      min: 30
      max: 255
    2:
      # blue
      color: "0000FF"
      label: "CENT2"
      min: 45
      max: 5000
                          

  • What to render: renderingMapping.tsv

  
  CDK5RAP2-C_     renderingdef.yml
  CENT2_  renderingdef2.yml
  CEP120/20111106_        renderingdef.yml
  CEP120/20111209_        renderingdef2.yml
  ...