Good Integration Practises¶
Work with virtual environments¶
We recommend to use virtualenv environments and use pip
(or easy_install) for installing your application and any dependencies
as well as the pytest
package itself. This way you will get an isolated
and reproducible environment. Given you have installed virtualenv
and execute it from the command line, here is an example session for unix
or windows:
virtualenv . # create a virtualenv directory in the current directory
source bin/activate # on unix
scripts/activate # on Windows
We can now install pytest:
pip install pytest
Due to the activate
step above the pip
will come from
the virtualenv directory and install any package into the isolated
virtual environment.
Choosing a test layout / import rules¶
pytest
supports two common test layouts:
putting tests into an extra directory outside your actual application code, useful if you have many functional tests or for other reasons want to keep tests separate from actual application code (often a good idea):
setup.py # your distutils/setuptools Python package metadata mypkg/ __init__.py appmodule.py tests/ test_app.py ...
inlining test directories into your application package, useful if you have direct relation between (unit-)test and application modules and want to distribute your tests along with your application:
setup.py # your distutils/setuptools Python package metadata mypkg/ __init__.py appmodule.py ... test/ test_app.py ...
Important notes relating to both schemes:
make sure that “mypkg” is importable, for example by typing once:
pip install -e . # install package using setup.py in editable mode
avoid “__init__.py” files in your test directories. This way your tests can run easily against an installed version of
mypkg
, independently from the installed package if it contains the tests or not.With inlined tests you might put
__init__.py
into test directories and make them installable as part of your application. Using thepy.test --pyargs mypkg
invocation pytest will discover where mypkg is installed and collect tests from there. With the “external” test you can still distribute tests but they will not be installed or become importable.
Typically you can run tests by pointing to test directories or modules:
py.test tests/test_app.py # for external test dirs
py.test mypkg/test/test_app.py # for inlined test dirs
py.test mypkg # run tests in all below test directories
py.test # run all tests below current dir
...
Because of the above editable install
mode you can change your
source code (both tests and the app) and rerun tests at will.
Once you are done with your work, you can use tox to make sure
that the package is really correct and tests pass in all
required configurations.
Note
You can use Python3 namespace packages (PEP420) for your application
but pytest will still perform test package name discovery based on the
presence of __init__.py
files. If you use one of the
two recommended file system layouts above but leave away the __init__.py
files from your directories it should just work on Python3.3 and above. From
“inlined tests”, however, you will need to use absolute imports for
getting at your application code.
Note
If pytest
finds a “a/b/test_module.py” test file while
recursing into the filesystem it determines the import name
as follows:
- determine
basedir
: this is the first “upward” (towards the root) directory not containing an__init__.py
. If e.g. botha
andb
contain an__init__.py
file then the parent directory ofa
will become thebasedir
. - perform
sys.path.insert(0, basedir)
to make the test module importable under the fully qualified import name. import a.b.test_module
where the path is determined by converting path separators/
into ”.” characters. This means you must follow the convention of having directory and file names map directly to the import names.
The reason for this somewhat evolved importing technique is that in larger projects multiple test modules might import from each other and thus deriving a canonical import name helps to avoid surprises such as a test modules getting imported twice.
Use tox and Continuous Integration servers¶
If you frequently release code and want to make sure that your actual package passes all tests you may want to look into tox, the virtualenv test automation tool and its pytest support. Tox helps you to setup virtualenv environments with pre-defined dependencies and then executing a pre-configured test command with options. It will run tests against the installed package and not against your source code checkout, helping to detect packaging glitches.
If you want to use Jenkins you can use the --junitxml=PATH
option
to create a JUnitXML file that Jenkins can pick up and generate reports.
Create a pytest standalone script¶
If you are a maintainer or application developer and want people
who don’t deal with python much to easily run tests you may generate
a standalone pytest
script:
py.test --genscript=runtests.py
This generates a runtests.py
script which is a fully functional basic
pytest
script, running unchanged under Python2 and Python3.
You can tell people to download the script and then e.g. run it like this:
python runtests.py
Integrating with distutils / python setup.py test
¶
You can integrate test runs into your distutils or
setuptools based project. Use the genscript method
to generate a standalone pytest
script:
py.test --genscript=runtests.py
and make this script part of your distribution and then add
this to your setup.py
file:
from distutils.core import setup, Command
# you can also import from setuptools
class PyTest(Command):
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
import sys,subprocess
errno = subprocess.call([sys.executable, 'runtests.py'])
raise SystemExit(errno)
setup(
#...,
cmdclass = {'test': PyTest},
#...,
)
If you now type:
python setup.py test
this will execute your tests using runtests.py
. As this is a
standalone version of pytest
no prior installation whatsoever is
required for calling the test command. You can also pass additional
arguments to the subprocess-calls such as your test directory or other
options.
Integration with setuptools test commands¶
Setuptools supports writing our own Test command for invoking pytest. Most often it is better to use tox instead, but here is how you can get started with setuptools integration:
import sys
from setuptools.command.test import test as TestCommand
class PyTest(TestCommand):
user_options = [('pytest-args=', 'a', "Arguments to pass to py.test")]
def initialize_options(self):
TestCommand.initialize_options(self)
self.pytest_args = None
def finalize_options(self):
TestCommand.finalize_options(self)
self.test_args = []
self.test_suite = True
def run_tests(self):
#import here, cause outside the eggs aren't loaded
import pytest
errno = pytest.main(self.pytest_args)
sys.exit(errno)
setup(
#...,
tests_require=['pytest'],
cmdclass = {'test': PyTest},
)
Now if you run:
python setup.py test
this will download pytest
if needed and then run your tests
as you would expect it to. You can pass a single string of arguments
using the --pytest-args
or -a
command-line option. For example:
python setup.py test -a "--durations=5"
is equivalent to running py.test --durations=5
.
Conventions for Python test discovery¶
pytest
implements the following standard test discovery:
- collection starts from the initial command line arguments which may be directories, filenames or test ids.
- recurse into directories, unless they match
norecursedirs
test_*.py
or*_test.py
files, imported by their `package name`_.Test
prefixed test classes (without an__init__
method)test_
prefixed test functions or methods are test items
For examples of how to customize your test discovery Changing standard (Python) test discovery.
Within Python modules, pytest
also discovers tests using the standard
unittest.TestCase subclassing technique.