Text File
testrunner-coverage.txt

Code Coverage

If the --coverage option is used, test coverage reports will be generated. The directory name given as the parameter will be used to hold the reports.

>>> import os.path, sys
>>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')
>>> defaults = [
...     '--path', directory_with_tests,
...     '--tests-pattern', '^sampletestsf?$',
...     ]
>>> sys.argv = 'test --coverage=coverage_dir'.split()
>>> from zope.testing import testrunner
>>> testrunner.run(defaults)
Running unit tests:
  Ran 192 tests with 0 failures and 0 errors in 0.125 seconds.
Running samplelayers.Layer1 tests:
  Set up samplelayers.Layer1 in 0.000 seconds.
  Ran 9 tests with 0 failures and 0 errors in 0.003 seconds.
Running samplelayers.Layer11 tests:
  Set up samplelayers.Layer11 in 0.000 seconds.
  Ran 34 tests with 0 failures and 0 errors in 0.029 seconds.
Running samplelayers.Layer111 tests:
  Set up samplelayers.Layerx in 0.000 seconds.
  Set up samplelayers.Layer111 in 0.000 seconds.
  Ran 34 tests with 0 failures and 0 errors in 0.024 seconds.
Running samplelayers.Layer112 tests:
  Tear down samplelayers.Layer111 in 0.000 seconds.
  Set up samplelayers.Layer112 in 0.000 seconds.
  Ran 34 tests with 0 failures and 0 errors in 0.024 seconds.
Running samplelayers.Layer12 tests:
  Tear down samplelayers.Layer112 in 0.000 seconds.
  Tear down samplelayers.Layerx in 0.000 seconds.
  Tear down samplelayers.Layer11 in 0.000 seconds.
  Set up samplelayers.Layer12 in 0.000 seconds.
  Ran 34 tests with 0 failures and 0 errors in 0.026 seconds.
Running samplelayers.Layer121 tests:
  Set up samplelayers.Layer121 in 0.000 seconds.
  Ran 34 tests with 0 failures and 0 errors in 0.025 seconds.
Running samplelayers.Layer122 tests:
  Tear down samplelayers.Layer121 in 0.000 seconds.
  Set up samplelayers.Layer122 in 0.000 seconds.
  Ran 34 tests with 0 failures and 0 errors in 0.025 seconds.
Tearing down left over layers:
  Tear down samplelayers.Layer122 in 0.000 seconds.
  Tear down samplelayers.Layer12 in 0.000 seconds.
  Tear down samplelayers.Layer1 in 0.000 seconds.
Total: 405 tests, 0 failures, 0 errors
lines   cov%   module   (path)
...
   52    92%   sample1.sampletests.test1   (testrunner-ex/sample1/sampletests/test1.py)
   78    94%   sample1.sampletests.test11   (testrunner-ex/sample1/sampletests/test11.py)
   78    94%   sample1.sampletests.test111   (testrunner-ex/sample1/sampletests/test111.py)
   78    94%   sample1.sampletests.test112   (testrunner-ex/sample1/sampletests/test112.py)
   78    94%   sample1.sampletests.test12   (testrunner-ex/sample1/sampletests/test12.py)
   78    94%   sample1.sampletests.test121   (testrunner-ex/sample1/sampletests/test121.py)
   78    94%   sample1.sampletests.test122   (testrunner-ex/sample1/sampletests/test122.py)
...
False

The directory specified with the --coverage option will have been created and will hold the coverage reports.

>>> os.path.exists('coverage_dir')
True
>>> os.listdir('coverage_dir')
[...]

(We should clean up after ourselves.)

>>> import shutil
>>> shutil.rmtree('coverage_dir')

Ignoring Tests

The trace module supports ignoring directories and modules based the test selection. Only directories selected for testing should report coverage. The test runner provides a custom implementation of the relevant API.

The TestIgnore class, the class managing the ignoring, is initialized by passing the command line options. It uses the options to determine the directories that should be covered.

>>> class FauxOptions(object):
...   package = None
...   test_path = [('/myproject/src/blah/foo', ''),
...                ('/myproject/src/blah/bar', '')]
>>> from zope.testing import testrunner
>>> ignore = testrunner.TestIgnore(FauxOptions())
>>> ignore._test_dirs
['/myproject/src/blah/foo/', '/myproject/src/blah/bar/']

We can now ask whether a particular module should be ignored:

>>> ignore.names('/myproject/src/blah/foo/baz.py', 'baz')
False
>>> ignore.names('/myproject/src/blah/bar/mine.py', 'mine')
False
>>> ignore.names('/myproject/src/blah/foo/__init__.py', 'foo')
False
>>> ignore.names('/myproject/src/blah/hello.py', 'hello')
True

When running the test runner, modules are sometimes created from text strings. Those should always be ignored:

>>> ignore.names('/myproject/src/blah/hello.txt', '<string>')
True

To make this check fast, the class implements a cache. In an early implementation, the result was cached by the module name, which was a problem, since a lot of modules carry the same name (not the Python dotted name here!). So just because a module has the same name in an ignored and tested directory, does not mean it is always ignored:

>>> ignore.names('/myproject/src/blah/module.py', 'module')
True
>>> ignore.names('/myproject/src/blah/foo/module.py', 'module')
False