OMERO.web API update

Monday 9th January, 2017

Will Moore

Talk outline

Current state of OMERO.web API
What do we want for the 'new' API?
Work done so far
- New 'api' app
- Updates to Blitz Gateway
Questions still to answer

Current state of OMERO.web API (early 2016)

webgateway - "imgData", plate, list Projects, Datasets etc.
webclient, mixture between html and unstructured json data
jsTree 3 (Douglas) added several webclient/api/ urls for json data
- Used iQuery queries built in webclient/tree.py
- Filter by group, owner, pagination etc
- Supports 'childCount' for display in jsTree
- Uses 'projection' queries (no omero.model objects)
- Lots of tests

What do we want for the 'new' API?

Current webclient/api/ would satisfy many user needs
Also want pixel data to build full client (E.g. ICY)
Use omero-marshal
Full CRUD functionality
Cover more objects - not just P/D/I S/P/W etc

Work done so far (Design)

Cleanup of webclient/api Design Issue #24 (March 2016)
API draft doc
omero-marshal based api Design PR #43 (June 2016)
API draft doc
Web meeting notes

Work done so far (Code)

Initial attempts to port webclint/api to webgateway (not merged) #4357 #4572
Re-focus on using omero_marshal. Support CRUD for Projects only: #4708
Move to new /api/ app: #4920
Add Datasets support (not merged) #4945
Move more iQuery logic to Blitz Gateway buildQuery() #4950
- Filter Datasets by Project
- Add childCount (needs to use 'projection' query)

Questions still to answer...

1) Objects to support for 5.3.0... Design #65

Currently support:
```
Project
```
webclient/api supports:
```
Project, Dataset, Image, Screen, Plate
```

Blitz Gateway wrappers: (not ROIs, Shapes, Channels etc)

project, dataset, image, screen, plate,
plateacquisition, acquisition, well,
experimenter, experimentergroup,
originalfile, fileset, annotations

omero_marshal support (not PlateAcquisition, Wells)


>>> omero_marshal.ENCODERS
{omero.model.DoubleAnnotationI: ,
 omero.model.DetailsI: ,
 omero.model.AcquisitionModeI: ,
 omero.model.AffineTransformI: ,
 omero.model.Annotation: ,
 omero.model.BooleanAnnotationI: ,
 omero.model.ChannelI: ,
 omero.model.TextAnnotation: ,
 omero.model.CommentAnnotationI: ,
 omero.model.ContrastMethodI: ,
 omero.model.DatasetI: ,
 omero.model.DimensionOrderI: ,
 omero.model.Shape: ,
 omero.model.EllipseI: ,
 omero.model.ExperimenterI: ,
 omero.model.ExperimenterGroupI: ,
 omero.model.ExternalInfoI: ,
 omero.model.FormatI: ,
 omero.model.IlluminationI: ,
 omero.model.ImageI: ,
 omero.model.LabelI: ,
 omero.model.LineI: ,
 omero.model.LongAnnotationI: ,
 omero.model.MapAnnotationI: ,
 omero.model.PhotometricInterpretationI: ,
 omero.model.PixelsI: ,
 omero.model.PixelsTypeI: ,
 omero.model.PlateI: ,
 omero.model.PointI: ,
 omero.model.PolygonI: ,
 omero.model.PolylineI: ,
 omero.model.ProjectI: ,
 omero.model.RectangleI: ,
 omero.model.RoiI: ,
 omero.model.ScreenI: ,
 omero.model.TagAnnotationI: ,
 omero.model.TermAnnotationI: ,
 omero.model.TimestampAnnotationI: ,
 omero.model.XmlAnnotationI: ,
 omero.model.PermissionsI: }

1b) What query params to support for each Object?

Project supports: owner, group, page, limit, child_count
Dataset also filters by: project, orphaned?
Annotations to filter by: project etc?
Shapes to filter by: roi? image?

2) Versioning

See Design Issues #64
How do we support multiple versions at once?
Do we distinguish minor from breaking changes? E.g. /v1.0/ or /v1/

3) Graph traversal

What to load by default?
How do users specify what else to load?
E.g. api/images/1/?embed=Pixels,omero:format.value
What do we support?
- load Datasets AND Images? (probably not)
- load Wells AND Images? (probably?)
- load Images AND Pixels? (almost certainly)

4) Marshalling details, etc. Issue #22

Project: 6 lines vv 50 lines with perms:

{
    "omero:details": {
        "owner": {
            "UserName": "will",
            "FirstName": "Will",
            "MiddleName": "",
            "omero:details": {
                "@type": "TBD#Details",
                "permissions": {
                    "canDelete": false,
                    "perm": "rw----",
                    "canEdit": false,
                    "canAnnotate": false,
                    "canLink": false,
                    "@type": "TBD#Permissions"
                }
            },
            "Email": "",
            "LastName": "Moore",
            "@id": 3,
            "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Experimenter"
        },
        "group": {
            "omero:details": {
                "@type": "TBD#Details",
                "permissions": {
                    "canDelete": false,
                    "perm": "rwra--",
                    "canEdit": false,
                    "canAnnotate": false,
                    "canLink": false,
                    "@type": "TBD#Permissions"
                }
            },
            "@id": 5,
            "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#ExperimenterGroup",
            "Name": "read-ann"
        },
        "@type": "TBD#Details",
        "permissions": {
            "canDelete": true,
            "perm": "rwra--",
            "canEdit": true,
            "canAnnotate": true,
            "canLink": true,
            "@type": "TBD#Permissions"
        }
    },
    "Description": "",
    "@id": 13578,
    "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Project",
    "Name": "API TEST foo"
}

Image (with Pixels, Format) 50 lines vv 130 lines with perms

{
    "Name": "438_01_R3D_D3D.dv",
    "omero:details": {},
    "Pixels": {
        "SizeT": 99,
        "Type": {
            "omero:details": {},
            "@id": 3,
            "@type": "TBD#PixelsType",
            "value": "int16"
        },
        "SizeC": 2,
        "omero:sha1": "a296bee3c52ab4eb960adfb31442261ad683204f",
        "PhysicalSizeY": {
            "Symbol": "µm",
            "Value": 0.13262000679969788,
            "@type": "TBD#LengthI",
            "Unit": "MICROMETER"
        },
        "omero:details": {},
        "PhysicalSizeZ": {
            "Symbol": "µm",
            "Value": 0.4000000059604645,
            "@type": "TBD#LengthI",
            "Unit": "MICROMETER"
        },
        "SizeX": 512,
        "SizeY": 512,
        "SizeZ": 4,
        "PhysicalSizeX": {
            "Symbol": "µm",
            "Value": 0.13262000679969788,
            "@type": "TBD#LengthI",
            "Unit": "MICROMETER"
        },
        "@id": 4101,
        "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Pixels",
        "SignificantBits": 16
    },
    "omero:series": 0,
    "omero:format": {
        "omero:details": {},
        "@id": 19,
        "@type": "TBD#Format",
        "value": "Deltavision"
    },
    "@id": 4401,
    "@type": "http://www.openmicroscopy.org/Schemas/OME/2016-06#Image"
}

5) Architecture

Web api and conn.getObjects() to use conn.buildQuery()?
Couples web api and Blitz Gateway #4950

conn.buildQuery("Dataset", opts={'child_count':True, 'project': 401})

# **NB: need to use queryService.projection() instead of findAllByQuery()
""" select obj, (select count(id) from DatasetImageLink chl where chl.parent=obj.id)
 from Dataset obj
    join fetch obj.details.owner as owner
    join fetch obj.details.creationEvent
    join obj.projectLinks plink
 where plink.parent.id = :pid"""

6) Read-Only for 5.3.0?

Saving Projects will unlink Datasets
Temp fix in #4930
Now we can't remove Datasets from a Project via API
This will get much more complex with other objects E.g. Images

7) Pagination, sorting

Probably should NEVER turn off pagination.
E.g. /webclient/api/images/?page=0 lists ALL available images
Instead use ?limit=10000 to return a known limit of results
Use page=0 as first page instead of page=1
Use offset instead of page? (same as ParametersI API)
Support sorting by named attributes?

8) Use of Stateful services (for imgData json)?

Do we try to support everything that's in the current webgateway/imgData/id json?
E.g. use RenderingEngine for channel colors
Or maybe /api/channels/?image=:id

9) Documentation

Probably will have to be hand-written