This document has some edge-case examples to test various aspects of the test runner.
The --path option defines a directory to be searched for tests and a directory to be added to Python's search path. The --test-path option can be used when you want to set a test search path without also affecting the Python path:
>>> import os, sys >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')>>> from zope.testing import testrunner>>> defaults = [ ... '--test-path', directory_with_tests, ... '--tests-pattern', '^sampletestsf?$', ... ] >>> sys.argv = ['test'] >>> testrunner.run(defaults) ... # doctest: +ELLIPSIS Test-module import failures: <BLANKLINE> Module: sampletestsf <BLANKLINE> ImportError: No module named sampletestsf ...>>> sys.path.append(directory_with_tests) >>> sys.argv = ['test'] >>> testrunner.run(defaults) ... # doctest: +ELLIPSIS Running unit tests: Ran 192 tests with 0 failures and 0 errors in 0.028 seconds. Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Ran 9 tests with 0 failures and 0 errors in 0.000 seconds. ...
>>> class Input: ... def __init__(self, src): ... self.lines = src.split('\n') ... def readline(self): ... line = self.lines.pop(0) ... print line ... return line+'\n'>>> real_stdin = sys.stdin
Using pdb.set_trace in a function called by an ordinary test:
>>> if sys.version_info[:2] == (2, 3): ... sys.stdin = Input('n\np x\nc') ... else: ... sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace2').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +ELLIPSIS Running unit tests:... > testrunner-ex/sample3/sampletests_d.py(47)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.001 seconds. False
Using pdb.set_trace in a function called by a doctest in a doc string:
>>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace4').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin Running unit tests: --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > testrunner-ex/sample3/sampletests_d.py(42)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. False
Using pdb in a docstring-based doctest
>>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace3').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin Running unit tests: --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > <doctest sample3.sampletests_d.set_trace3[1]>(3)...() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. False
Using pdb.set_trace in a doc file:
>>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace5').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin Running unit tests: --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > <doctest set_trace5.txt[1]>(3)...() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. False
Using pdb.set_trace in a function called by a doctest in a doc file:
>>> sys.stdin = Input('n\np x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t set_trace6').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin Running unit tests: --Return-- > doctest.py(351)set_trace()->None -> pdb.Pdb.set_trace(self) (Pdb) n > testrunner-ex/sample3/sampletests_d.py(42)f() -> y = x (Pdb) p x 1 (Pdb) c Ran 1 tests with 0 failures and 0 errors in 0.002 seconds. False
Post-mortem debugging function called from ordinary test:
>>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem2 -D').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running unit tests: <BLANKLINE> <BLANKLINE> Error in test test_post_mortem2 (sample3.sampletests_d.TestSomething) Traceback (most recent call last): File "testrunner-ex/sample3/sampletests_d.py", line 37, in test_post_mortem2 g() File "testrunner-ex/sample3/sampletests_d.py", line 46, in g raise ValueError ValueError <BLANKLINE> exceptions.ValueError: <BLANKLINE> > testrunner-ex/sample3/sampletests_d.py(46)g() -> raise ValueError (Pdb) p x 1 (Pdb) c True
Post-mortem debugging docstring-based doctest:
>>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem3 -D').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running unit tests: <BLANKLINE> <BLANKLINE> Error in test post_mortem3 (sample3.sampletests_d) Traceback (most recent call last): File "zope/testing/doctest.py", line 2276, in debug runner.run(self._dt_test) File "zope/testing/doctest.py", line 1731, in run r = DocTestRunner.run(self, test, compileflags, out, False) File "zope/testing/doctest.py", line 1389, in run return self.__run(test, compileflags, out) File "zope/testing/doctest.py", line 1310, in __run exc_info) File "zope/testing/doctest.py", line 1737, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: from testrunner-ex/sample3/sampletests_d.py:61 (2 examples)> <BLANKLINE> exceptions.ValueError: <BLANKLINE> > <doctest sample3.sampletests_d.post_mortem3[1]>(1)...() (Pdb) p x 1 (Pdb) c True
Post-mortem debugging function called from docstring-based doctest:
>>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem4 -D').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running unit tests: <BLANKLINE> <BLANKLINE> Error in test post_mortem4 (sample3.sampletests_d) Traceback (most recent call last): File "zope/testing/doctest.py", line 2276, in debug runner.run(self._dt_test) File "zope/testing/doctest.py", line 1731, in run r = DocTestRunner.run(self, test, compileflags, out, False) File "zope/testing/doctest.py", line 1389, in run return self.__run(test, compileflags, out) File "zope/testing/doctest.py", line 1310, in __run exc_info) File "zope/testing/doctest.py", line 1737, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: testrunner-ex/sample3/sampletests_d.py:67 (1 example)> <BLANKLINE> exceptions.ValueError: <BLANKLINE> > testrunner-ex/sample3/sampletests_d.py(46)g() -> raise ValueError (Pdb) p x 1 (Pdb) c True
Post-mortem debugging file-based doctest:
>>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem5 -D').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running unit tests: <BLANKLINE> <BLANKLINE> Error in test zope/testing/testrunner-ex/sample3/post_mortem5.txt Traceback (most recent call last): File "zope/testing/doctest.py", line 2276, in debug runner.run(self._dt_test) File "zope/testing/doctest.py", line 1731, in run r = DocTestRunner.run(self, test, compileflags, out, False) File "zope/testing/doctest.py", line 1389, in run return self.__run(test, compileflags, out) File "zope/testing/doctest.py", line 1310, in __run exc_info) File "zope/testing/doctest.py", line 1737, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: testrunner-ex/sample3/post_mortem5.txt:0 (2 examples)> <BLANKLINE> exceptions.ValueError: <BLANKLINE> > <doctest post_mortem5.txt[1]>(1)...() (Pdb) p x 1 (Pdb) c True
Post-mortem debugging function called from file-based doctest:
>>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem6 -D').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running unit tests: <BLANKLINE> <BLANKLINE> Error in test zope/testing/testrunner-ex/sample3/post_mortem6.txt Traceback (most recent call last): File "zope/testing/doctest.py", line 2276, in debug runner.run(self._dt_test) File "zope/testing/doctest.py", line 1731, in run r = DocTestRunner.run(self, test, compileflags, out, False) File "zope/testing/doctest.py", line 1389, in run return self.__run(test, compileflags, out) File "zope/testing/doctest.py", line 1310, in __run exc_info) File "zope/testing/doctest.py", line 1737, in report_unexpected_exception raise UnexpectedException(test, example, exc_info) UnexpectedException: testrunner-ex/sample3/post_mortem6.txt:0 (2 examples)> <BLANKLINE> exceptions.ValueError: <BLANKLINE> > testrunner-ex/sample3/sampletests_d.py(46)g() -> raise ValueError (Pdb) p x 1 (Pdb) c True
Post-mortem debugging of a docstring doctest failure:
>>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure2 -D').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running unit tests: <BLANKLINE> <BLANKLINE> Error in test post_mortem_failure2 (sample3.sampletests_d) <BLANKLINE> File "testrunner-ex/sample3/sampletests_d.py", line 81, in sample3.sampletests_d.post_mortem_failure2 <BLANKLINE> x <BLANKLINE> Want: 2 <BLANKLINE> Got: 1 <BLANKLINE> <BLANKLINE> > testrunner-ex/sample3/sampletests_d.py(81)_() exceptions.ValueError: Expected and actual output are different > <string>(1)...() (Pdb) p x 1 (Pdb) c True
Post-mortem debugging of a docfile doctest failure:
>>> sys.stdin = Input('p x\nc') >>> sys.argv = ('test -ssample3 --tests-pattern ^sampletests_d$' ... ' -t post_mortem_failure.txt -D').split() >>> try: testrunner.run(defaults) ... finally: sys.stdin = real_stdin ... # doctest: +NORMALIZE_WHITESPACE Running unit tests: <BLANKLINE> <BLANKLINE> Error in test /home/jim/z3/zope.testing/src/zope/testing/testrunner-ex/sample3/post_mortem_failure.txt <BLANKLINE> File "testrunner-ex/sample3/post_mortem_failure.txt", line 2, in post_mortem_failure.txt <BLANKLINE> x <BLANKLINE> Want: 2 <BLANKLINE> Got: 1 <BLANKLINE> <BLANKLINE> > testrunner-ex/sample3/post_mortem_failure.txt(2)_() exceptions.ValueError: Expected and actual output are different > <string>(1)...() (Pdb) p x 1 (Pdb) c True
Post-mortem debugging with triple verbosity
>>> sys.argv = 'test --layer samplelayers.Layer1$ -vvv -D'.split() >>> testrunner.run(defaults) Running tests at level 1 Running samplelayers.Layer1 tests: Set up samplelayers.Layer1 in 0.000 seconds. Running: test_x1 (sampletestsf.TestA1) (0.000 s) test_y0 (sampletestsf.TestA1) (0.000 s) test_z0 (sampletestsf.TestA1) (0.000 s) test_x0 (sampletestsf.TestB1) (0.000 s) test_y1 (sampletestsf.TestB1) (0.000 s) test_z0 (sampletestsf.TestB1) (0.000 s) test_1 (sampletestsf.TestNotMuch1) (0.000 s) test_2 (sampletestsf.TestNotMuch1) (0.000 s) test_3 (sampletestsf.TestNotMuch1) (0.000 s) Ran 9 tests with 0 failures and 0 errors in 0.001 seconds. Tearing down left over layers: Tear down samplelayers.Layer1 in 0.000 seconds. False
>>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_suite$', ... ] >>> testrunner.run(defaults) Test-module import failures: <BLANKLINE> Module: sample1.sampletests_none_suite <BLANKLINE> TypeError: Invalid test_suite, None, in sample1.sampletests_none_suite <BLANKLINE> <BLANKLINE> Total: 0 tests, 0 failures, 0 errors <BLANKLINE> Test-modules with import problems: sample1.sampletests_none_suite True>>> sys.argv = ['test', ... '--tests-pattern', '^sampletests_none_test$', ... ] >>> testrunner.run(defaults) Test-module import failures: <BLANKLINE> Module: sample1.sampletests_none_test <BLANKLINE> TypeError: Invalid test, None, in test_suite from sample1.sampletests_none_test <BLANKLINE> <BLANKLINE> Total: 0 tests, 0 failures, 0 errors <BLANKLINE> Test-modules with import problems: sample1.sampletests_none_test True
It is an error to specify --report-refcounts (-r) without specifying a repeat count greater than 1
>>> sys.argv = 'test -r'.split() >>> testrunner.run(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. <BLANKLINE> True>>> sys.argv = 'test -r -N1'.split() >>> testrunner.run(defaults) You must use the --repeat (-N) option to specify a repeat count greater than 1 when using the --report_refcounts (-r) option. <BLANKLINE> True