pytest

(& flake8)

Colin Blackburn & S├ębastien Besson

Outline

  • flake8
  • pytest
    • running tests
    • writing tests

flake8

Style and (passive) code correctness enforcement used by Travis. Code in the following areas is currently checked by default:

OmeroPy/test & OmeroPy/src
OmeroFS/test & OmeroFS/src
OmeroWeb/test

The intention is to have most Python code checked eventually.

See: flake8 docs

flake8

flake8 is a wrapper around:

pyflakescorrectness
pep8style
mccabecomplexity

mccabe is turned off by default

pyflakes

pyflakes checks correctness, including:

  • missing & unused imports
  • unused variables
  • duplicate parameter names

pyflakes error codes

pep8

pep8 checks style, including:

  • whitespace: too much, too little, vertical, horizontal,...
  • indentation: too much, too little,...
  • line length
  • deprecated usage: e.g. .has_key() for in

pep8 error codes

Installing flake8

  • pip install flake8
  • plugins for most editors/IDEs
  • hooks for git (and other VCS)

The hook for git is quite buggy and needs an empty setup.cfg file at the root of the repository. It also turns on mccabe which you really don't want to do!

Using flake8

  • flake8 test/integration/test_admin.py
  • files can be excluded by setup.cfg
  • flake8 -v test/integration
  • files can also be excluded using # flake8: noqa
  • lines can be excluded using # noqa
  • flake8 -h

Excluding anything from flake8 by whatever means is a last resort, really frowned upon and is unlikely to get past code review without good reason!

pytest

  • Simple test discovery
  • Set up at module, class and test level
  • No significant boilerplate
  • Powerful features if needed
  • Integrates with other frameworks
  • Extensible - plugins

See: pytest docs

Installing pytest

  • pip install pytest
  • pip install <pytest plugin>

Running pytest

However it is run, pytest does three things:

  • discovers tests
  • runs those tests
  • reports on those tests

Running Python tests

  • Top-level build
  • ./build.py -f components/<component>/build.xml test
  • Component level build
    • Using setup.py
    • cd components/tools/<component> && ./setup.py test -s test/unit
    • Directly using py.test
    • cd components/tools/<component> && py.test test/unit

See: OMERO docs

Top-level build.py

./build.py -f components/<component>/build.xml test
./build.py -f components/<component>/build.xml integration
./build.py -f components/tools/OmeroPy/build.xml test -DTEST=test/integration/test_admin.py
./build.py -f components/tools/OmeroPy/build.xml integration -DMARK="not long_running"

Component-level setup.py

./setup.py test -s test/unit
./setup.py test -s test/integration
./setup.py test -s test/integration/test_admin.py
./setup.py test -s test/integration -m "not long_running"

Component-level setup.py

./setup.py test -s test/integration -k permissions
./setup.py test -s test/integration/test_admin.py -k testGetGroup
./setup.py test -h

Using py.test

py.test test/unit
py.test test/integration
py.test test/integration/test_admin.py
py.test test/integration -m "not long_running"

Using py.test

py.test test/integration -k permissions
py.test test/integration/test_admin.py -k testGetGroup
py.test test/integration -s
py.test --repeat 20 test/unit/fstest
py.test --collect-only test/integration/ -m long_running
py.test -h

Writing tests

  • Naming tests
  • The basics
  • Exceptions
  • Fixtures
  • Parameters
  • Markers
  • Extending

Naming conventions

pytest discovers what tests to run using naming conventions for modules, functions, classes and methods. We use the default:

  • modules called test_*
    • functions called test*
    • classes called Test*
      • methods called test*

Naming example

components/tools/OmeroPy/test/integration/test_admin.py

class TestAdmin(lib.ITest):

    def testGetGroup(self):
        a = self.client.getSession().getAdminService()
        l = a.lookupGroups()
        g = a.getGroup(l[0].getId().val)
        assert 0 != g.sizeOfGroupExperimenterMap()
            

Here the method testGetGroup will be run.

A basic test

components/tools/OmeroPy/test/integration/test_admin.py

import test.integration.library as lib


class TestAdmin(lib.ITest):

    def testGetGroup(self):
        a = self.client.getSession().getAdminService()
        l = a.lookupGroups()
        g = a.getGroup(l[0].getId().val)
        assert 0 != g.sizeOfGroupExperimenterMap()
            

Python assert is used, but comparisons are context-sensitive and can be user-defined.

Note: lib.ITest contains general set-up and useful helpers.

Fixtures

No fixture

Adapted from: components/tools/OmeroPy/test/unit/clitest/test_tag.py

class TestTag(object):

    def testHelp(self):
        self.cli = CLI()
        self.cli.register("tag", TagControl, "TEST")
        self.args = ["tag"]
        self.args += ["-h"]
        self.cli.invoke(self.args, strict=True)
            

A simple setup fixture

components/tools/OmeroPy/test/unit/clitest/test_tag.py

class TestTag(object):

    def setup_method(self, method):
        self.cli = CLI()
        self.cli.register("tag", TagControl, "TEST")
        self.args = ["tag"]

    def testHelp(self):
        self.args += ["-h"]
        self.cli.invoke(self.args, strict=True)
            

Setup and teardown

components/tools/OmeroPy/test/integration/clitest/test_tag.py

class TestTag(CLITest):

    def setup_method(self, method):
        super(TestTag, self).setup_method(method)
        self.cli.register("tag", TagControl, "TEST")
        self.args += ["tag"]
        self.setup_mock()

    def teardown_method(self, method):
        self.teardown_mock()
        super(TestTag, self).teardown_method(method)
            

Decorator fixtures

components/tools/OmeroWeb/test/integration/test_show.py

@pytest.fixture(scope='module')
def path():
    """Returns the root OMERO.web webclient path."""
    return '/webclient'

@pytest.fixture(scope='module')
def request_factory():
    """Returns a fresh Django request factory."""
    return RequestFactory()

@pytest.fixture(scope='function')
def empty_request(request, request_factory, path):
    """
    Returns a simple GET request object with no 'path' query string.
    """
    return {
        'request': request_factory.get(path),
        'initially_select': list(),
        'initially_open': None
    }

class TestShow(object):

    def test_empty_path(self, empty_request):
        show = Show(conn, empty_request['request'], None)
        self.assert_instantiation(show, empty_request, conn)
        first_selected = show.first_selected
        assert first_selected is None
            

More decorator fixtures

components/tools/OmeroWeb/test/integration/test_show.py

@pytest.fixture(scope='function')
def itest(request):
    """
    Returns a new L{test.integration.library.ITest} instance.  With
    attached finalizer so that pytest will clean it up.
    """
    o = lib.ITest()
    o.setup_method(None)

    def finalizer():
        o.teardown_method(None)
    request.addfinalizer(finalizer)
    return o

@pytest.fixture(scope='function', params=[1, 2])
def tag(request, itest, update_service):
    """Returns a new OMERO TagAnnotation with required fields set."""
    name = rstring(itest.uuid())
    for index in range(request.param):
        tag = TagAnnotationI()
        tag.textValue = name
        tag = update_service.saveAndReturnObject(tag)
    return tag
            

Built-in fixtures

See section of the py.test documentation

  • capsys/capfd: stdout/stderr capture
  • monkeypatch
  • tmpdir: temporary paths

Exceptions

Exceptions using with

components/tools/OmeroPy/test/unit/clitest/test_tag.py

import pytest
from omero.cli import CLI, NonZeroReturnCode

class TestTag(object):

    def testCreateTagsetFails(self):
        self.args += ["createset", "--tag", "A"]
        with pytest.raises(NonZeroReturnCode):
            self.cli.invoke(self.args, strict=True)

    def testListFails(self):
        self.args += ["list", "--tagset", "tagset"]
        with pytest.raises(NonZeroReturnCode):
            self.cli.invoke(self.args, strict=True)
            

Exceptions

components/tools/OmeroPy/test/integration/test_repository.py

import pytest

class TestRecursiveDelete(AbstractRepoTest):

    def setup_method(self, method):
        super(TestRecursiveDelete, self).setup_method(method)
        self.filename = self.unique_dir + "/file.txt"
        self.mrepo = self.getManagedRepo()
        # ...

    def testDoubleDot(self):
        naughty = self.unique_dir + "/" + ".." + "/" + ".." + "/" + ".."
        pytest.raises(omero.ValidationException,
                      self.mrepo.deletePaths, [naughty], True, True)
            

This approach is compatible with Python 2.4 and so should probably be deprecated.

Parameters

Single parameter

components/tools/OmeroPy/test/unit/clitest/test_tag.py

import pytest
subcommands = [
    "create", "createset", "list", "listsets", "link", "load"]


class TestTag(object):

    @pytest.mark.parametrize('subcommand', subcommands)
    def testSubcommandHelp(self, subcommand):
        self.args += [subcommand, "-h"]
        self.cli.invoke(self.args, strict=True)
            

Multiple parameters

components/tools/OmeroPy/test/integration/clitest/test_tag.py

import pytest

class TestTag(object):

    @pytest.mark.parametrize(
        ('object_arg', 'tag_arg'),
        [('Image:1', 'test'), ('Image', '1'), ('Image:image', '1'),
         ('1', '1')])
    def testLinkFails(self, object_arg, tag_arg):
        self.args += ["link", object_arg, tag_arg]
        with pytest.raises(NonZeroReturnCode):
            self.cli.invoke(self.args, strict=True)
            

Nested multiple parameters

components/tools/OmeroPy/test/integration/clitest/test_chgrp.py

object_types = ["Image", "Dataset", "Project", "Plate", "Screen"]
permissions = ["rw----", "rwr---", "rwra--", "rwrw--"]
group_prefixes = ["", "Group:", "ExperimenterGroup:"]


class TestChgrp(CLITest):

    @pytest.mark.parametrize("object_type", object_types)
    @pytest.mark.parametrize("target_group_perms", permissions)
    @pytest.mark.parametrize("group_prefix", group_prefixes)
    def testChgrpMyData(self, object_type, target_group_perms, group_prefix):
        oid = self.create_object(object_type)

        # create a new group and move the object to the new group
        group = self.add_new_group(perms=target_group_perms)
        self.args += ['%s%s' % (group_prefix, group.id.val),
                      '/%s:%s' % (object_type, oid)]
        self.cli.invoke(self.args, strict=True)

        # change the session context and check the object has been moved
        self.set_context(self.client, group.id.val)
        new_object = self.query.get(object_type, oid)
        assert new_object.id.val == oid
            

Markers

xfail marker

components/tools/OmeroPy/test/integration/clitest/test_chgrp.py

class TestChgrp(CLITest):

    @pytest.mark.xfail(reason="CLI  does not wrap all chgrps in 1 DoAll")
    def testFilesetAllImages(self):
        images = self.importMIF(2)  # 2 images sharing a fileset

        # create a new group and try to move only one image to the new group
        group = self.add_new_group()
        self.args += ['%s' % group.id.val, '/Image:%s' % images[0].id.val,
                      '/Image:%s' % images[1].id.val]
        self.cli.invoke(self.args, strict=True)

        # check the images have been moved
        ctx = {'omero.group': '-1'}  # query across groups
        for i in images:
            img = self.query.get('Image', i.id.val, ctx)
            assert img.details.group.id.val == group.id.val
            

xfail marker results

$ py.test test/integration/test_permissions.py
========================= test session starts ================================
platform darwin -- Python 2.7.8 -- py-1.4.25 -- pytest-2.6.3
plugins: pythonpath, rerunfailures, xdist
collected 102 items

test/integration/test_permissions.py ....x.....x.xx.........xxxXxxXxx.......

=========== 61 passed, 17 xfailed, 24 xpassed in 85.58 seconds ===============


$ py.test --runxfail test/integration/test_permissions.py
========================= test session starts ================================
platform darwin -- Python 2.7.8 -- py-1.4.25 -- pytest-2.6.3
plugins: pythonpath, rerunfailures, xdist
collected 102 items

test/integration/test_permissions.py ....F.....F.FF..

...

$ py.test test/integration/test_permissions.py -m xfail --runxfail

...
            

User-defined markers

components/tools/OmeroPy/test/integration/test_rawpixelsstore.py

class TestRPS(lib.ITest):

    @pytest.mark.long_running
    def testRomioToPyramid(self):
        """
        Here we create a pixels that is not big,
        then modify its metadata so that it IS big,
        in order to trick the service into throwing
        us a MissingPyramidException
        """
        from omero.util import concurrency
        pix = self.missing_pyramid(self.root)
        rps = self.root.sf.createRawPixelsStore()
        ...
            

Markers applied to a class

components/tools/OmeroPy/test/integration/test_thumbs.py

@pytest.mark.long_running
class TestThumbs(lib.ITest):

    def testCreateThumbnails(self):
        tb = self.pyr_tb()
        try:
            tb.createThumbnails()
        finally:
            tb.close()
            

Pre-defining markers

components/tools/OmeroPy/pytest.ini

[pytest]
markers =
    long_running: mark the test as long-running, i.e. typically requiring more than 10 minutes to complete



$ py.test --markers
@pytest.mark.long_running: mark the test as long-running, i.e. typically requiring more than 10 minutes to complete
...
@pytest.mark.skipif(condition): skip the given test function if eval(condition) results in a True value.  Evaluation happens within the module global context. Example: skipif('sys.platform == "win32"') skips the test if we are on the win32 platform. see http://pytest.org/latest/skipping.html
...
            

Extending pytest

Adding an option

components/tools/OmeroPy/test/conftest.py

def pytest_addoption(parser):
    parser.addoption(
        '--repeat', action='store',
        help='Number of times to repeat each test')


def pytest_generate_tests(metafunc):
    if metafunc.config.option.repeat is not None:
        count = int(metafunc.config.option.repeat)

        # We're going to duplicate these tests by parametrizing them,
        # which requires that each test has a fixture to accept the parameter.
        # We can add a new fixture like so:
        metafunc.fixturenames.append('tmp_ct')

        # Now we parametrize. This is what happens when we do e.g.,
        # @pytest.mark.parametrize('tmp_ct', range(count))
        # def test_foo(): pass
        metafunc.parametrize('tmp_ct', range(count))
            

Plugins


$ py.test --version
This is pytest version 2.6.3, imported from /usr/local/lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
  pytest-pythonpath-0.3 at /usr/local/lib/python2.7/site-packages/pytest_pythonpath.pyc
  pytest-rerunfailures-0.05 at /usr/local/lib/python2.7/site-packages/rerunfailures/plugin.pyc
  pytest-xdist-1.10 at /usr/local/lib/python2.7/site-packages/xdist/plugin.pyc
            

Lots more available at List of Third-Party Plugins

Useful guide at pytest Plugins Compatibility