Text File
testrunner-edge-cases.txt

testrunner Edge Cases

This document has some edge-case examples to test various aspects of the test runner.

Separating Python path and test directories

The --path option defines a directory to be searched for tests and a directory to be added to Python's search path. The --test-path option can be used when you want to set a test search path without also affecting the Python path:

>>> import os, sys
>>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')
>>> from zope.testing import testrunner
>>> defaults = [
...     '--test-path', directory_with_tests,
...     '--tests-pattern', '^sampletestsf?$',
...     ]
>>> sys.argv = ['test']
>>> testrunner.run(defaults)
... # doctest: +ELLIPSIS
Test-module import failures:
<BLANKLINE>
Module: sampletestsf
<BLANKLINE>
ImportError: No module named sampletestsf
...
>>> sys.path.append(directory_with_tests)
>>> sys.argv = ['test']
>>> testrunner.run(defaults)
... # doctest: +ELLIPSIS
Running unit tests:
  Ran 192 tests with 0 failures and 0 errors in 0.028 seconds.
Running samplelayers.Layer1 tests:
  Set up samplelayers.Layer1 in 0.000 seconds.
  Ran 9 tests with 0 failures and 0 errors in 0.000 seconds.
...

Debugging Edge Cases

>>> class Input:
...     def __init__(self, src):
...         self.lines = src.split('\n')
...     def readline(self):
...         line = self.lines.pop(0)
...         print line
...         return line+'\n'
>>> real_stdin = sys.stdin

Using pdb.set_trace in a function called by an ordinary test:

>>> if sys.version_info[:2] == (2, 3):
...     sys.stdin = Input('n\np x\nc')
... else:
...     sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t set_trace2').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +ELLIPSIS
Running unit tests:...
> testrunner-ex/sample3/sampletests_d.py(47)f()
-> y = x
(Pdb) p x
1
(Pdb) c
  Ran 1 tests with 0 failures and 0 errors in 0.001 seconds.
False

Using pdb.set_trace in a function called by a doctest in a doc string:

>>> sys.stdin = Input('n\np x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t set_trace4').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
Running unit tests:
--Return--
> doctest.py(351)set_trace()->None
-> pdb.Pdb.set_trace(self)
(Pdb) n
> testrunner-ex/sample3/sampletests_d.py(42)f()
-> y = x
(Pdb) p x
1
(Pdb) c
  Ran 1 tests with 0 failures and 0 errors in 0.002 seconds.
False

Using pdb in a docstring-based doctest

>>> sys.stdin = Input('n\np x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t set_trace3').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
Running unit tests:
--Return--
> doctest.py(351)set_trace()->None
-> pdb.Pdb.set_trace(self)
(Pdb) n
> <doctest sample3.sampletests_d.set_trace3[1]>(3)...()
-> y = x
(Pdb) p x
1
(Pdb) c
  Ran 1 tests with 0 failures and 0 errors in 0.002 seconds.
False

Using pdb.set_trace in a doc file:

>>> sys.stdin = Input('n\np x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t set_trace5').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
Running unit tests:
--Return--
> doctest.py(351)set_trace()->None
-> pdb.Pdb.set_trace(self)
(Pdb) n
> <doctest set_trace5.txt[1]>(3)...()
-> y = x
(Pdb) p x
1
(Pdb) c
  Ran 1 tests with 0 failures and 0 errors in 0.002 seconds.
False

Using pdb.set_trace in a function called by a doctest in a doc file:

>>> sys.stdin = Input('n\np x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t set_trace6').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
Running unit tests:
--Return--
> doctest.py(351)set_trace()->None
-> pdb.Pdb.set_trace(self)
(Pdb) n
> testrunner-ex/sample3/sampletests_d.py(42)f()
-> y = x
(Pdb) p x
1
(Pdb) c
  Ran 1 tests with 0 failures and 0 errors in 0.002 seconds.
False

Post-mortem debugging function called from ordinary test:

>>> sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t post_mortem2 -D').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +NORMALIZE_WHITESPACE
Running unit tests:
<BLANKLINE>
<BLANKLINE>
Error in test test_post_mortem2 (sample3.sampletests_d.TestSomething)
Traceback (most recent call last):
  File "testrunner-ex/sample3/sampletests_d.py",
       line 37, in test_post_mortem2
    g()
  File "testrunner-ex/sample3/sampletests_d.py", line 46, in g
    raise ValueError
ValueError
<BLANKLINE>
exceptions.ValueError:
<BLANKLINE>
> testrunner-ex/sample3/sampletests_d.py(46)g()
-> raise ValueError
(Pdb) p x
1
(Pdb) c
True

Post-mortem debugging docstring-based doctest:

>>> sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t post_mortem3 -D').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +NORMALIZE_WHITESPACE
Running unit tests:
<BLANKLINE>
<BLANKLINE>
Error in test post_mortem3 (sample3.sampletests_d)
Traceback (most recent call last):
  File "zope/testing/doctest.py", line 2276, in debug
    runner.run(self._dt_test)
  File "zope/testing/doctest.py", line 1731, in run
    r = DocTestRunner.run(self, test, compileflags, out, False)
  File "zope/testing/doctest.py", line 1389, in run
    return self.__run(test, compileflags, out)
  File "zope/testing/doctest.py", line 1310, in __run
    exc_info)
  File "zope/testing/doctest.py", line 1737, in report_unexpected_exception
    raise UnexpectedException(test, example, exc_info)
UnexpectedException:
   from testrunner-ex/sample3/sampletests_d.py:61 (2 examples)>
<BLANKLINE>
exceptions.ValueError:
<BLANKLINE>
> <doctest sample3.sampletests_d.post_mortem3[1]>(1)...()
(Pdb) p x
1
(Pdb) c
True

Post-mortem debugging function called from docstring-based doctest:

>>> sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t post_mortem4 -D').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +NORMALIZE_WHITESPACE
Running unit tests:
<BLANKLINE>
<BLANKLINE>
Error in test post_mortem4 (sample3.sampletests_d)
Traceback (most recent call last):
  File "zope/testing/doctest.py", line 2276, in debug
    runner.run(self._dt_test)
  File "zope/testing/doctest.py", line 1731, in run
    r = DocTestRunner.run(self, test, compileflags, out, False)
  File "zope/testing/doctest.py", line 1389, in run
    return self.__run(test, compileflags, out)
  File "zope/testing/doctest.py", line 1310, in __run
    exc_info)
  File "zope/testing/doctest.py", line 1737, in report_unexpected_exception
    raise UnexpectedException(test, example, exc_info)
UnexpectedException: testrunner-ex/sample3/sampletests_d.py:67 (1 example)>
<BLANKLINE>
exceptions.ValueError:
<BLANKLINE>
> testrunner-ex/sample3/sampletests_d.py(46)g()
-> raise ValueError
(Pdb) p x
1
(Pdb) c
True

Post-mortem debugging file-based doctest:

>>> sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t post_mortem5 -D').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +NORMALIZE_WHITESPACE
Running unit tests:
<BLANKLINE>
<BLANKLINE>
Error in test zope/testing/testrunner-ex/sample3/post_mortem5.txt
Traceback (most recent call last):
  File "zope/testing/doctest.py", line 2276, in debug
    runner.run(self._dt_test)
  File "zope/testing/doctest.py", line 1731, in run
    r = DocTestRunner.run(self, test, compileflags, out, False)
  File "zope/testing/doctest.py", line 1389, in run
    return self.__run(test, compileflags, out)
  File "zope/testing/doctest.py", line 1310, in __run
    exc_info)
  File "zope/testing/doctest.py", line 1737, in report_unexpected_exception
    raise UnexpectedException(test, example, exc_info)
UnexpectedException: testrunner-ex/sample3/post_mortem5.txt:0 (2 examples)>
<BLANKLINE>
exceptions.ValueError:
<BLANKLINE>
> <doctest post_mortem5.txt[1]>(1)...()
(Pdb) p x
1
(Pdb) c
True

Post-mortem debugging function called from file-based doctest:

>>> sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t post_mortem6 -D').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +NORMALIZE_WHITESPACE
Running unit tests:
<BLANKLINE>
<BLANKLINE>
Error in test zope/testing/testrunner-ex/sample3/post_mortem6.txt
Traceback (most recent call last):
  File "zope/testing/doctest.py", line 2276, in debug
    runner.run(self._dt_test)
  File "zope/testing/doctest.py", line 1731, in run
    r = DocTestRunner.run(self, test, compileflags, out, False)
  File "zope/testing/doctest.py", line 1389, in run
    return self.__run(test, compileflags, out)
  File "zope/testing/doctest.py", line 1310, in __run
    exc_info)
  File "zope/testing/doctest.py", line 1737, in report_unexpected_exception
    raise UnexpectedException(test, example, exc_info)
UnexpectedException: testrunner-ex/sample3/post_mortem6.txt:0 (2 examples)>
<BLANKLINE>
exceptions.ValueError:
<BLANKLINE>
> testrunner-ex/sample3/sampletests_d.py(46)g()
-> raise ValueError
(Pdb) p x
1
(Pdb) c
True

Post-mortem debugging of a docstring doctest failure:

>>> sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t post_mortem_failure2 -D').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +NORMALIZE_WHITESPACE
Running unit tests:
<BLANKLINE>
<BLANKLINE>
Error in test post_mortem_failure2 (sample3.sampletests_d)
<BLANKLINE>
File "testrunner-ex/sample3/sampletests_d.py",
               line 81, in sample3.sampletests_d.post_mortem_failure2
<BLANKLINE>
x
<BLANKLINE>
Want:
2
<BLANKLINE>
Got:
1
<BLANKLINE>
<BLANKLINE>
> testrunner-ex/sample3/sampletests_d.py(81)_()
exceptions.ValueError:
Expected and actual output are different
> <string>(1)...()
(Pdb) p x
1
(Pdb) c
True

Post-mortem debugging of a docfile doctest failure:

>>> sys.stdin = Input('p x\nc')
>>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$'
...             ' -t post_mortem_failure.txt -D').split()
>>> try: testrunner.run(defaults)
... finally: sys.stdin = real_stdin
... # doctest: +NORMALIZE_WHITESPACE
Running unit tests:
<BLANKLINE>
<BLANKLINE>
Error in test /home/jim/z3/zope.testing/src/zope/testing/testrunner-ex/sample3/post_mortem_failure.txt
<BLANKLINE>
File "testrunner-ex/sample3/post_mortem_failure.txt",
                                  line 2, in post_mortem_failure.txt
<BLANKLINE>
x
<BLANKLINE>
Want:
2
<BLANKLINE>
Got:
1
<BLANKLINE>
<BLANKLINE>
> testrunner-ex/sample3/post_mortem_failure.txt(2)_()
exceptions.ValueError:
Expected and actual output are different
> <string>(1)...()
(Pdb) p x
1
(Pdb) c
True

Post-mortem debugging with triple verbosity

>>> sys.argv = 'test --layer samplelayers.Layer1$ -vvv -D'.split()
>>> testrunner.run(defaults)
Running tests at level 1
Running samplelayers.Layer1 tests:
  Set up samplelayers.Layer1 in 0.000 seconds.
  Running:
    test_x1 (sampletestsf.TestA1) (0.000 s)
    test_y0 (sampletestsf.TestA1) (0.000 s)
    test_z0 (sampletestsf.TestA1) (0.000 s)
    test_x0 (sampletestsf.TestB1) (0.000 s)
    test_y1 (sampletestsf.TestB1) (0.000 s)
    test_z0 (sampletestsf.TestB1) (0.000 s)
    test_1 (sampletestsf.TestNotMuch1) (0.000 s)
    test_2 (sampletestsf.TestNotMuch1) (0.000 s)
    test_3 (sampletestsf.TestNotMuch1) (0.000 s)
  Ran 9 tests with 0 failures and 0 errors in 0.001 seconds.
Tearing down left over layers:
  Tear down samplelayers.Layer1 in 0.000 seconds.
False

Test Suites with None for suites or tests

>>> sys.argv = ['test',
...             '--tests-pattern', '^sampletests_none_suite$',
...     ]
>>> testrunner.run(defaults)
Test-module import failures:
<BLANKLINE>
Module: sample1.sampletests_none_suite
<BLANKLINE>
TypeError: Invalid test_suite, None, in sample1.sampletests_none_suite
<BLANKLINE>
<BLANKLINE>
Total: 0 tests, 0 failures, 0 errors
<BLANKLINE>
Test-modules with import problems:
  sample1.sampletests_none_suite
True
>>> sys.argv = ['test',
...             '--tests-pattern', '^sampletests_none_test$',
...     ]
>>> testrunner.run(defaults)
Test-module import failures:
<BLANKLINE>
Module: sample1.sampletests_none_test
<BLANKLINE>
TypeError: Invalid test, None,
in test_suite from sample1.sampletests_none_test
<BLANKLINE>
<BLANKLINE>
Total: 0 tests, 0 failures, 0 errors
<BLANKLINE>
Test-modules with import problems:
  sample1.sampletests_none_test
True

You must use --repeat with --report-refcounts

It is an error to specify --report-refcounts (-r) without specifying a repeat count greater than 1

>>> sys.argv = 'test -r'.split()
>>> testrunner.run(defaults)
        You must use the --repeat (-N) option to specify a repeat
        count greater than 1 when using the --report_refcounts (-r)
        option.
<BLANKLINE>
True
>>> sys.argv = 'test -r -N1'.split()
>>> testrunner.run(defaults)
        You must use the --repeat (-N) option to specify a repeat
        count greater than 1 when using the --report_refcounts (-r)
        option.
<BLANKLINE>
True