pytest

(& flake8)

Colin Blackburn & Sébastien Besson

Outline

flake8
pytest

running tests
writing tests

flake8

Style and (passive) code correctness enforcement used by Travis. Code in the following areas is currently checked by default:

OmeroPy/test & OmeroPy/src

OmeroFS/test & OmeroFS/src

OmeroWeb/test

The intention is to have most Python code checked eventually.

See: flake8 docs

flake8

flake8 is a wrapper around:

`pyflakes`	correctness
`pep8`	style
`mccabe`	complexity

mccabe is turned off by default

pyflakes

pyflakes checks correctness, including:

missing & unused imports
unused variables
duplicate parameter names

pyflakes error codes

pep8

pep8 checks style, including:

whitespace: too much, too little, vertical, horizontal,...
indentation: too much, too little,...
line length
deprecated usage: e.g. .has_key() for in

pep8 error codes

Installing flake8

pip install flake8
plugins for most editors/IDEs
hooks for git (and other VCS)

The hook for git is quite buggy and needs an empty setup.cfg file at the root of the repository. It also turns on mccabe which you really don't want to do!

Using flake8

flake8 test/integration/test_admin.py
files can be excluded by setup.cfg
flake8 -v test/integration
files can also be excluded using # flake8: noqa
lines can be excluded using # noqa
flake8 -h

Excluding anything from flake8 by whatever means is a last resort, really frowned upon and is unlikely to get past code review without good reason!

pytest

Simple test discovery
Set up at module, class and test level
No significant boilerplate
Powerful features if needed
Integrates with other frameworks
Extensible - plugins

See: pytest docs

Installing pytest

pip install pytest
pip install <pytest plugin>

Running pytest

However it is run, pytest does three things:

discovers tests
runs those tests
reports on those tests

Running Python tests

Top-level build

./build.py -f components/<component>/build.xml test

Component level build

Using setup.py

cd components/tools/<component> && ./setup.py test -s test/unit

Directly using py.test

cd components/tools/<component> && py.test test/unit

See: OMERO docs

Top-level build.py

./build.py -f components/<component>/build.xml test
./build.py -f components/<component>/build.xml integration
./build.py -f components/tools/OmeroPy/build.xml test -DTEST=test/integration/test_admin.py
./build.py -f components/tools/OmeroPy/build.xml integration -DMARK="not long_running"

Component-level setup.py

./setup.py test -s test/unit
./setup.py test -s test/integration
./setup.py test -s test/integration/test_admin.py
./setup.py test -s test/integration -m "not long_running"

Component-level setup.py

./setup.py test -s test/integration -k permissions
./setup.py test -s test/integration/test_admin.py -k testGetGroup
./setup.py test -h

Using py.test

py.test test/unit
py.test test/integration
py.test test/integration/test_admin.py
py.test test/integration -m "not long_running"

Using py.test

py.test test/integration -k permissions
py.test test/integration/test_admin.py -k testGetGroup
py.test test/integration -s
py.test --repeat 20 test/unit/fstest
py.test --collect-only test/integration/ -m long_running
py.test -h

Writing tests

Naming tests
The basics
Exceptions
Fixtures
Parameters
Markers
Extending

Naming conventions

pytest discovers what tests to run using naming conventions for modules, functions, classes and methods. We use the default:

modules called test_*

functions called test*
classes called Test*

methods called test*

Naming example

components/tools/OmeroPy/test/integration/test_admin.py

class TestAdmin(lib.ITest):

    def testGetGroup(self):
        a = self.client.getSession().getAdminService()
        l = a.lookupGroups()
        g = a.getGroup(l[0].getId().val)
        assert 0 != g.sizeOfGroupExperimenterMap()

Here the method testGetGroup will be run.

A basic test

components/tools/OmeroPy/test/integration/test_admin.py

import test.integration.library as lib


class TestAdmin(lib.ITest):

    def testGetGroup(self):
        a = self.client.getSession().getAdminService()
        l = a.lookupGroups()
        g = a.getGroup(l[0].getId().val)
        assert 0 != g.sizeOfGroupExperimenterMap()

Python assert is used, but comparisons are context-sensitive and can be user-defined.

Note: lib.ITest contains general set-up and useful helpers.

Fixtures

No fixture

Adapted from: components/tools/OmeroPy/test/unit/clitest/test_tag.py

class TestTag(object):

    def testHelp(self):
        self.cli = CLI()
        self.cli.register("tag", TagControl, "TEST")
        self.args = ["tag"]
        self.args += ["-h"]
        self.cli.invoke(self.args, strict=True)

A simple setup fixture

components/tools/OmeroPy/test/unit/clitest/test_tag.py

class TestTag(object):

    def setup_method(self, method):
        self.cli = CLI()
        self.cli.register("tag", TagControl, "TEST")
        self.args = ["tag"]

    def testHelp(self):
        self.args += ["-h"]
        self.cli.invoke(self.args, strict=True)

Setup and teardown

components/tools/OmeroPy/test/integration/clitest/test_tag.py

class TestTag(CLITest):

    def setup_method(self, method):
        super(TestTag, self).setup_method(method)
        self.cli.register("tag", TagControl, "TEST")
        self.args += ["tag"]
        self.setup_mock()

    def teardown_method(self, method):
        self.teardown_mock()
        super(TestTag, self).teardown_method(method)

Decorator fixtures

components/tools/OmeroWeb/test/integration/test_show.py

@pytest.fixture(scope='module')
def path():
    """Returns the root OMERO.web webclient path."""
    return '/webclient'

@pytest.fixture(scope='module')
def request_factory():
    """Returns a fresh Django request factory."""
    return RequestFactory()

@pytest.fixture(scope='function')
def empty_request(request, request_factory, path):
    """
    Returns a simple GET request object with no 'path' query string.
    """
    return {
        'request': request_factory.get(path),
        'initially_select': list(),
        'initially_open': None
    }

class TestShow(object):

    def test_empty_path(self, empty_request):
        show = Show(conn, empty_request['request'], None)
        self.assert_instantiation(show, empty_request, conn)
        first_selected = show.first_selected
        assert first_selected is None

More decorator fixtures

components/tools/OmeroWeb/test/integration/test_show.py

@pytest.fixture(scope='function')
def itest(request):
    """
    Returns a new L{test.integration.library.ITest} instance.  With
    attached finalizer so that pytest will clean it up.
    """
    o = lib.ITest()
    o.setup_method(None)

    def finalizer():
        o.teardown_method(None)
    request.addfinalizer(finalizer)
    return o

@pytest.fixture(scope='function', params=[1, 2])
def tag(request, itest, update_service):
    """Returns a new OMERO TagAnnotation with required fields set."""
    name = rstring(itest.uuid())
    for index in range(request.param):
        tag = TagAnnotationI()
        tag.textValue = name
        tag = update_service.saveAndReturnObject(tag)
    return tag

Built-in fixtures

See section of the py.test documentation

capsys/capfd: stdout/stderr capture
monkeypatch
tmpdir: temporary paths

Exceptions

Exceptions using with

components/tools/OmeroPy/test/unit/clitest/test_tag.py

import pytest
from omero.cli import CLI, NonZeroReturnCode

class TestTag(object):

    def testCreateTagsetFails(self):
        self.args += ["createset", "--tag", "A"]
        with pytest.raises(NonZeroReturnCode):
            self.cli.invoke(self.args, strict=True)

    def testListFails(self):
        self.args += ["list", "--tagset", "tagset"]
        with pytest.raises(NonZeroReturnCode):
            self.cli.invoke(self.args, strict=True)

Exceptions

components/tools/OmeroPy/test/integration/test_repository.py

import pytest

class TestRecursiveDelete(AbstractRepoTest):

    def setup_method(self, method):
        super(TestRecursiveDelete, self).setup_method(method)
        self.filename = self.unique_dir + "/file.txt"
        self.mrepo = self.getManagedRepo()
        # ...

    def testDoubleDot(self):
        naughty = self.unique_dir + "/" + ".." + "/" + ".." + "/" + ".."
        pytest.raises(omero.ValidationException,
                      self.mrepo.deletePaths, [naughty], True, True)

This approach is compatible with Python 2.4 and so should probably be deprecated.

Parameters

Single parameter

components/tools/OmeroPy/test/unit/clitest/test_tag.py

import pytest
subcommands = [
    "create", "createset", "list", "listsets", "link", "load"]


class TestTag(object):

    @pytest.mark.parametrize('subcommand', subcommands)
    def testSubcommandHelp(self, subcommand):
        self.args += [subcommand, "-h"]
        self.cli.invoke(self.args, strict=True)

Multiple parameters

components/tools/OmeroPy/test/integration/clitest/test_tag.py

import pytest

class TestTag(object):

    @pytest.mark.parametrize(
        ('object_arg', 'tag_arg'),
        [('Image:1', 'test'), ('Image', '1'), ('Image:image', '1'),
         ('1', '1')])
    def testLinkFails(self, object_arg, tag_arg):
        self.args += ["link", object_arg, tag_arg]
        with pytest.raises(NonZeroReturnCode):
            self.cli.invoke(self.args, strict=True)

Nested multiple parameters

components/tools/OmeroPy/test/integration/clitest/test_chgrp.py

object_types = ["Image", "Dataset", "Project", "Plate", "Screen"]
permissions = ["rw----", "rwr---", "rwra--", "rwrw--"]
group_prefixes = ["", "Group:", "ExperimenterGroup:"]


class TestChgrp(CLITest):

    @pytest.mark.parametrize("object_type", object_types)
    @pytest.mark.parametrize("target_group_perms", permissions)
    @pytest.mark.parametrize("group_prefix", group_prefixes)
    def testChgrpMyData(self, object_type, target_group_perms, group_prefix):
        oid = self.create_object(object_type)

        # create a new group and move the object to the new group
        group = self.add_new_group(perms=target_group_perms)
        self.args += ['%s%s' % (group_prefix, group.id.val),
                      '/%s:%s' % (object_type, oid)]
        self.cli.invoke(self.args, strict=True)

        # change the session context and check the object has been moved
        self.set_context(self.client, group.id.val)
        new_object = self.query.get(object_type, oid)
        assert new_object.id.val == oid

Markers

xfail marker

components/tools/OmeroPy/test/integration/clitest/test_chgrp.py

class TestChgrp(CLITest):

    @pytest.mark.xfail(reason="CLI  does not wrap all chgrps in 1 DoAll")
    def testFilesetAllImages(self):
        images = self.importMIF(2)  # 2 images sharing a fileset

        # create a new group and try to move only one image to the new group
        group = self.add_new_group()
        self.args += ['%s' % group.id.val, '/Image:%s' % images[0].id.val,
                      '/Image:%s' % images[1].id.val]
        self.cli.invoke(self.args, strict=True)

        # check the images have been moved
        ctx = {'omero.group': '-1'}  # query across groups
        for i in images:
            img = self.query.get('Image', i.id.val, ctx)
            assert img.details.group.id.val == group.id.val

xfail marker results

$ py.test test/integration/test_permissions.py
========================= test session starts ================================
platform darwin -- Python 2.7.8 -- py-1.4.25 -- pytest-2.6.3
plugins: pythonpath, rerunfailures, xdist
collected 102 items

test/integration/test_permissions.py ....x.....x.xx.........xxxXxxXxx.......

=========== 61 passed, 17 xfailed, 24 xpassed in 85.58 seconds ===============


$ py.test --runxfail test/integration/test_permissions.py
========================= test session starts ================================
platform darwin -- Python 2.7.8 -- py-1.4.25 -- pytest-2.6.3
plugins: pythonpath, rerunfailures, xdist
collected 102 items

test/integration/test_permissions.py ....F.....F.FF..

...

$ py.test test/integration/test_permissions.py -m xfail --runxfail

...

User-defined markers

components/tools/OmeroPy/test/integration/test_rawpixelsstore.py

class TestRPS(lib.ITest):

    @pytest.mark.long_running
    def testRomioToPyramid(self):
        """
        Here we create a pixels that is not big,
        then modify its metadata so that it IS big,
        in order to trick the service into throwing
        us a MissingPyramidException
        """
        from omero.util import concurrency
        pix = self.missing_pyramid(self.root)
        rps = self.root.sf.createRawPixelsStore()
        ...

Markers applied to a class

components/tools/OmeroPy/test/integration/test_thumbs.py

@pytest.mark.long_running
class TestThumbs(lib.ITest):

    def testCreateThumbnails(self):
        tb = self.pyr_tb()
        try:
            tb.createThumbnails()
        finally:
            tb.close()

Pre-defining markers

components/tools/OmeroPy/pytest.ini

[pytest]
markers =
    long_running: mark the test as long-running, i.e. typically requiring more than 10 minutes to complete



$ py.test --markers
@pytest.mark.long_running: mark the test as long-running, i.e. typically requiring more than 10 minutes to complete
...
@pytest.mark.skipif(condition): skip the given test function if eval(condition) results in a True value.  Evaluation happens within the module global context. Example: skipif('sys.platform == "win32"') skips the test if we are on the win32 platform. see http://pytest.org/latest/skipping.html
...

Extending pytest

Adding an option

components/tools/OmeroPy/test/conftest.py

def pytest_addoption(parser):
    parser.addoption(
        '--repeat', action='store',
        help='Number of times to repeat each test')


def pytest_generate_tests(metafunc):
    if metafunc.config.option.repeat is not None:
        count = int(metafunc.config.option.repeat)

        # We're going to duplicate these tests by parametrizing them,
        # which requires that each test has a fixture to accept the parameter.
        # We can add a new fixture like so:
        metafunc.fixturenames.append('tmp_ct')

        # Now we parametrize. This is what happens when we do e.g.,
        # @pytest.mark.parametrize('tmp_ct', range(count))
        # def test_foo(): pass
        metafunc.parametrize('tmp_ct', range(count))

Plugins


$ py.test --version
This is pytest version 2.6.3, imported from /usr/local/lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
  pytest-pythonpath-0.3 at /usr/local/lib/python2.7/site-packages/pytest_pythonpath.pyc
  pytest-rerunfailures-0.05 at /usr/local/lib/python2.7/site-packages/rerunfailures/plugin.pyc
  pytest-xdist-1.10 at /usr/local/lib/python2.7/site-packages/xdist/plugin.pyc

Lots more available at List of Third-Party Plugins

Useful guide at pytest Plugins Compatibility