You are on page 1of 79

pytest advanced testing tutorial

Holger Krekel
http://merlinux.eu
Pycon US 2009, Chicago

March 26, 2009

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 1 / 78
If you have time, please install

svn checkout http://codespeak.net/svn/py/dist


run “python setup.py” with “install” or “develop”
if need be: easy install “py”
slides: http://tinyurl.com/c76gve

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 1 / 78
my technical background

programming since 20 years


Python since around 2000
released projects: pypy, py.test/py lib, rlcompleter2
other: mailwitness, shpy, vadm, codespeak, ...
merlinux GmbH since 2004
PyPy EU-project 2004-2007

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 2 / 78
my current background

http://marikaz.deviantart.com/ CC 3.0 AN-ND


H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 3 / 78
my testing background

learned Python 2001


“test-driven developer” (TDD) since 2002
founded PyPy, based on TDD principles
developed utest/stdtest, now py.test
consulted to help with testing

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 4 / 78
What’s your background?

what test tools do you use?


work test-driven?
have test-suites with >1000 tests?
need/want to run tests on >1 platforms?
have “non-python” tests?

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 5 / 78
Python

Python has no compile-time type security.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 6 / 78
Testing to the rescue!

automated tests are better than declaring types.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 7 / 78
The test tool question

what is the job of automated testing tools?

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 8 / 78
my current answer

verify that my code changes work out


be helpful when test scenarios fail

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 9 / 78
If failures are not helpful ...

improve the test tool or


write more (different) tests

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 10 / 78
The Automated Test question

what are automated tests there for?

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 11 / 78
my current answer

to make sure that


units react well to input.
components co-operate nicely
code changes work out in the end

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 12 / 78
What does “code changes work out” mean?

http://marikaz.deviantart.com/ CC 3.0 AN-ND


H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 13 / 78
my current answer

various operating systems


various Python Interpreters
cloud environments
web browsers etc.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 14 / 78
Common Test terminology

developer and customer tests


unit tests
functional tests
acceptance tests
integration tests
you may discuss a long time about categorizations ...

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 15 / 78
py.test strives to test it all

current focus:
unittesting
functional
integration tests

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 16 / 78
A pragmatic view on test types

let’s talk about small, medium or large tests.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 17 / 78
Small Tests: one aspect

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 18 / 78
Medium Tests: two aspects

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 19 / 78
Large Tests: end-to-end

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 20 / 78
generally speaking

what is the vision of automated testing?

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 21 / 78
my current answer

merge with real-life deployment.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 22 / 78
pytest advanced features (30 minutes)

http://marikaz.deviantart.com/ CC 3.0 AN-ND


H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 23 / 78
Python test function viewed abstractly

from app.pkg import SomeClass


def test_something():
inst1 = SomeClass("somevalue")
...
assert "things are ok"

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 24 / 78
observations

test configuration mixes with code instantiation


importing ’app’ may fail
the “app.pkg.SomeClass” reference may change

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 25 / 78
setting up state close to code

from app.pkg import SomeClass


class TestGroup:
def setup_method(self, method):
self.inst1 = SomeClass("somevalue")

def test_something(self):
... use self.inst1 ...
assert "things are ok"

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 26 / 78
observations

test configuration mixes with code instantiation


importing ’app’ may fail
the “app.pkg.SomeClass” reference may change
functions tend to group by setup methods
multiple methods can reuse the same setup

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 27 / 78
meet py.test “funcargs”

goals:
fully separate test setup from test code
setup app bootstraps in one place
allow to group tests in classes or files logically

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 28 / 78
basic funcarg mechanism

have a test function specify its needed setup


lookup and call a function that provides setup

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 29 / 78
Example test function

def test_somefunction(myarg):
assert myarg == ’test_somefunction’

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 30 / 78
Example conftest.py

class ConftestPlugin:
def pytest_funcarg__myarg(self, py-
funcitem):
return pyfuncitem.name

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 31 / 78
observations

funcargs:
test functions receive value by specifying a name
makers are registered after command line parsing
makers have meta-access to“collected function item”

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 32 / 78
notes on “named based lookup”

py.test often looks up names:


discover test_*.py test files
discover test_ functions or Test classes
and now: discover test function arguments
automatic discovery avoids boilerplate

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 33 / 78
Exercise

write a new package “mypkg”


add mypkg/ init and mypkg/test url.py
add a test path function that needs an “url”
argument
write the provider in mypkg/conftest.py
run your test
Bonus: add more tests, play with wrongly named args.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 34 / 78
Using Plugins and Extensions

http://marikaz.deviantart.com/ CC 3.0 AN-ND


H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 35 / 78
Historic view

0.9.x uses conftest’s for extension and configuration


1.0 uses “plugins” for extending and conftest.py for
configuration
we do “smooth” transition because of existing test
code base

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 36 / 78
Customizing py.test

configuration values go to conftest.py files


write local or global plugins
provide funcargs

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 37 / 78
Conftest.py

can be put into test directory or higher up


contains test configuration values
specifies plugins to use
can provide default values for command line options

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 38 / 78
Specifying plugins

-p NAME: load comma-separated list of plugins


plugins are always named “pytest NAME” and can
be anywhere in your import path

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 39 / 78
Writing a local conftest plugin

you can write a local conftest.py based plugin:


class ConftestPlugin:
def pytest_addoption(self, parser):
parser.addoption("--
myworld", action="store_true")
...

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 40 / 78
Exercise

add an option for specifying url on the command line


look at “py.test -h”

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 41 / 78
other Plugin Examples

integrate collection/run of traditional unit-tests


run functions in their own tempdir
testing ReST documents
running Prolog tests (old: conftest based)
running Javascript tests (old: conftest based)
html reporting for nightly runs (old: conftest-based)

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 42 / 78
if time permits ...

let’s play with “pytest unittest” plugin

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 43 / 78
Break

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 44 / 78
Writing Plugins (30 minutes)

test collection objects


plugin hooks and events
event system for custom reporting
test collection hooks
test running hooks

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 45 / 78
py.test collection objects

collectors, collect() returns list of “colitems”


items, implement runtest() for running a test

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 46 / 78
test collection tree

collection tree is built iteratively


test collection starts from directories or files (via
cmdline)

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 47 / 78
py.test.collect.* filesystem objects

Directory
File

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 48 / 78
always available Attributes

parent a reference to the collector that produced us


name: a name that is unique within the scope of our
parent
config: test configuration

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 49 / 78
Python object collectors

obj points to the underlying python object.


Module
Class/Instance
Generator
Function

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 50 / 78
Exercise “collection tree”

inspecting the test collection tree:


py.test --collectonly

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 51 / 78
Hooks and Events

Plugin hook methods implement interaction with


configuration/collection/running of tests
Events are called for notification purposes, are not
asked for return values.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 52 / 78
Configuration Hooks
def pytest_addoption(self, parser):
""" called before commandline pars-
ing. """

def pytest_configure(self, config):


""" called after command line op-
tions have been parsed. "

def pytest_unconfigure(self, config):


""" called before test process is ex-
ited. """

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 53 / 78
Collection Hooks
def pytest_collect_file(self, path, parent):
""" return Collection node or None. """

def pytest_collect_directory(self, path, par-


ent):
""" return Collection node or None. """

def pytest_collect_recurse(self, path, par-


ent):
""" return True/False to cause/prevent re-
cursion. """

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 54 / 78
Python collection hooks

::
def pytest pymodule makeitem(self, modcol, nam
“”“ return custom item/collector for
a python object in a module, or
None. ”“”

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 55 / 78
function test run hooks

::
def pytest pyfunc call(self, pyfuncitem, args, kwa
“”“ return True if we consumed/did
the call to the python function
item. ”“”
def pytest item makereport(self, item, excinfo, w
“”“ return ItemTestReport event for
the given test outcome. ”“”

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 56 / 78
Reporting hooks

::
def pytest report teststatus(self, event): “”“
return shortletter and verbose
word. ”“”
def pytest terminal summary(self, terminalreporte
“”“ add additional section in
terminal summary reporting. ”“”

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 57 / 78
Event methods

are only called, no interaction.


see py/test/plugin/pytest_terminal.py for a full
selection of events.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 58 / 78
Warning

naming changes might take place before 1.0 final

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 59 / 78
Writing cross-project plugins

put plugin into pytest name.py or package


make a “NamePlugin” class available
release or copy to somewhere importable

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 60 / 78
Exercise

port conftest plugin to global “pytest myapp” plugin.


put pytest myapp somewhere where it’s importable
(or release it :)
put “pytest plugins = ’pytest myapp”’ into your test
module

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 61 / 78
Distributed Testing (45 minutes)

http://marikaz.deviantart.com/ CC 3.0 AN-ND


H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 62 / 78
the basic idea

collect tests locally, run tests in separated test execution


processes.

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 63 / 78
test distribution modes

--dist=load # load-balance tests to exec


environments
--dist=each # send each test to each exec
environment

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 64 / 78
send test to one other python interpreter

py.test --dist=each --
tx popen//python=python2.4

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 65 / 78
send test to three different interpreters

py.test --dist=each \
--tx=popen//python=python2.4 \
--tx=popen//python=python2.5 \
--tx=popen//python=python2.6

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 66 / 78
Example: Speed up test runs

To send tests to multiple CPUs, type:


py.test --dist=load --tx popen --tx popen --
tx popen

or in short:
py.test -n 3
Especially for longer running tests or tests requiring a lot
of IO this can lead to considerable speed ups!

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 67 / 78
send tests to remote SSH accout

py.test --d --tx ssh=myhostpopen --


rsyncdir mypkg mypkg

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 68 / 78
Sending tests to remote socket servers

Download and start


http://codespeak.net/svn/py/dist/py/execnet/script/socket
Assuming an IP address, you can now tell py.test to
distribute: :
py.test -d --tx socket=192.168.1.102:8888
--rsyncdir mypkg mypkg

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 69 / 78
Exercise: Move settings into conftest

mypkg/conftest.py:: rsyncdirs = [’.’] pytest option tx =


[’ssh=myhost’, ’socket=...’]
mypkg/conftest.py:: rsyncdirs = [’.’]

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 70 / 78
Distribution modes

py.test --dist=each mypkg

py.test --dist=load mypkg

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 71 / 78
how does py.test do all this?

it uses py.execnet!

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 72 / 78
py.execnet

instantiates local or remote Python Processes


send code for execution in one or many processes
asynchronously send and receive data between
processes through channels
completely avoid manual installation steps on
remote places

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 73 / 78
xspecs: Exec environment specs

general form:
key1=value1//key2=value2//key3=value3

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 74 / 78
xspec key meanings

popen for a PopenGateway


ssh=host for a SshGateway
socket=address:port for a SocketGateway
python=executable for specifying Python
executables
chdir=path change remote working dir to given
relative or absolute path
nice=value decrease remote nice level if platforms
supports it

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 75 / 78
xspec examples

::
ssh=wyvern//python=python2.4//chdir=mycach
popen//python=2.5//nice=20
socket=192.168.1.4:8888

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 76 / 78
Exercise

Play with runing your tests in multiple interpreters


or processes
see if you can get access to your neighbour’s PC

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 77 / 78
Feedback round

How did you like the tutorial?


Thanks for taking part!
holger krekel at merlinux eu

H. Krekel (Pycon US 2009, Chicago) pytest advanced testing tutorial March 26, 2009 78 / 78