8.7. Function Doctest

8.7.1. Rationale

8.7.2. Syntax

  • Docstring is a first multiline comment in: File/Module, Class, Method/Function

  • Used for generating help() documentation

  • It is accessible in __doc__ property of an object

  • Used for doctest

  • PEP 257 -- Docstring Conventions: For multiline str always use three double quote (""") characters

>>> def add(a, b):
...     """
...     >>> add(1, 2)
...     3
...     >>> add(-1, 1)
...     0
...     >>> add(0, 0)
...     0
...     """
...     return a + b
>>> # doctest: +SKIP
... """
... >>> add(1, 2)
... 3
... >>> add(-1, 1)
... 0
... >>> add(0, 0)
... 0
... """
>>>
>>> def add(a, b):
...     return a + b

8.7.3. Running Tests

Running tests in Pycharm IDE (either option):

  • Right click on source code with doctests -> Run 'Doctest for ...'

  • View menu -> Run... -> Doctest in myfunction

  • Note, that doctests are not discovered in scratch files in PyCharm

Running Tests from Python Code:

>>> if __name__ == "__main__":
...     from doctest import testmod
...     testmod()

Running tests from command line (displays errors only):

$ python -m doctest myfile.py

Add -v to display more verbose output.

$ python -m doctest -v myfile.py

8.7.4. Test Int, Float

int values:

>>> # doctest: +SKIP
... """
... >>> add(1, 2)
... 3
... >>> add(-1, 1)
... 0
... >>> add(0, 0)
... 0
... """
>>>
>>> def add(a, b):
...     return a + b

float values:

>>> # doctest: +SKIP
... """
... >>> add(1.0, 2.0)
... 3.0
...
... >>> add(0.1, 0.2)
... 0.30000000000000004
...
... >>> add(0.1, 0.2)   # doctest: +ELLIPSIS
... 0.3000...
... """
>>>
>>> def add(a, b):
...     return a + b

This is due to the floating point arithmetic in IEEE 754 standard:

>>> 0.1 + 0.2
0.30000000000000004
>>> 0.1 + 0.2  # doctest: +ELLIPSIS
0.3000...
>>> round(0.1+0.2, 16)
0.3
>>> round(0.1+0.2, 17)
0.30000000000000004

More information in Math Precision

8.7.5. Test Bool

>>> # doctest: +SKIP
... """
... Function checks if person is adult.
... Adult person is over 18 years old.
...
... >>> is_adult(18)
... True
...
... >>> is_adult(17.9)
... False
... """
>>>
>>> AGE_ADULT = 18
>>>
>>> def is_adult(age):
...     if age >= AGE_ADULT:
...         return True
...     else:
...         return False

8.7.6. Test Str

  • Python will change to single quotes in most cases

  • Python will change to double quotes to avoid escapes

  • print() function output, don't have quotes

Returning str. Python will change to single quotes in most cases:

>>> # doctest: +SKIP
... """
... >>> echo('hello')
... 'hello'
...
... # Python will change to single quotes in most cases
... >>> echo("hello")
... 'hello'
...
... Following test will fail
... >>> echo('hello')
... "hello"
...
... Python will change to double quotes to avoid escapes
... >>> echo('It\\'s Twardowski\\'s Moon')
... "It's Twardowski's Moon"
... """
>>>
>>> def echo(data):
...     return data

There are no quotes in print() function output:

>>> # doctest: +SKIP
... """
... >>> echo('hello')
... hello
... """
>>>
>>> def echo(data):
...     print(data)

Testing print(str) with newlines:

>>> # doctest: +SKIP
... """
... >>> echo('hello')
... hello
... hello
... hello
... <BLANKLINE>
... """
>>>
>>> def echo(data):
...     print(f'{data}\n' * 3)

8.7.7. Test Ordered Sequence

>>> # doctest: +SKIP
... """
... >>> echo([1,2,3])
... [1, 2, 3]
...
... >>> echo((1,2,3))
... (1, 2, 3)
... """
>>>
>>> def echo(data):
...     return data
>>> # doctest: +SKIP
... """
... >>> echo([1,2,3])
... [1, 2, 3]
...
... >>> echo((1,2,3))
... [1, 2, 3]
... """
>>>
>>> def echo(data):
...     return [x for x in data]
>>> # doctest: +SKIP
... """
... >>> echo([1,2,3])
... [274.15, 275.15, 276.15]
...
... >>> echo((1,2,3))
... (274.15, 275.15, 276.15)
... """
>>>
>>> def echo(data):
...     cls = type(data)
...     return cls(x+273.15 for x in data)

8.7.8. Test Unordered Sequence

Hash from numbers are constant:

>>> # doctest: +SKIP
... """
... >>> echo({1})
... {1}
... >>> echo({1,2})
... {1, 2}
... """
>>>
>>> def echo(data):
...     return data

However hash from str elements changes at every run:

>>> # doctest: +SKIP
... """
... >>> echo({'a', 'b'})
... {'b', 'a'}
... """
>>>
>>> def echo(data):
...     return data

Therefore you should test if element is in the result, rather than comparing output:

>>> # doctest: +SKIP
... """
... >>> result = echo({'a', 'b'})
... >>> 'a' in result
... True
... >>> 'b' in result
... True
... """
>>>
>>> def echo(data):
...     return data

8.7.9. Test Mapping

>>> # doctest: +SKIP
... """
... >>> result = echo({'a': 1, 'b': 2})
... >>> result
... {'a': 1, 'b': 2}
... >>> 'a' in result.keys()
... True
... >>> 1 in result.values()
... True
... >>> ('a', 1) in result.items()
... True
... >>> result['a']
... 1
... """
>>>
>>> def echo(data):
...     return data

8.7.10. Test Nested

>>> # doctest: +SKIP
... """
... >>> DATA = [('Sepal length', 'Sepal width', 'Petal length', 'Petal width', 'Species'),
... ...         (5.8, 2.7, 5.1, 1.9, 'virginica'),
... ...         (5.1, 3.5, 1.4, 0.2, 'setosa'),
... ...         (5.7, 2.8, 4.1, 1.3, 'versicolor'),
... ...         (6.3, 2.9, 5.6, 1.8, 'virginica'),
... ...         (6.4, 3.2, 4.5, 1.5, 'versicolor'),
... ...         (4.7, 3.2, 1.3, 0.2, 'setosa')]
...
... >>> echo(DATA)
... [('Sepal length', 'Sepal width', 'Petal length', 'Petal width', 'Species'), (5.8, 2.7, 5.1, 1.9, 'virginica'), (5.1, 3.5, 1.4, 0.2, 'setosa'), (5.7, 2.8, 4.1, 1.3, 'versicolor'), (6.3, 2.9, 5.6, 1.8, 'virginica'), (6.4, 3.2, 4.5, 1.5, 'versicolor'), (4.7, 3.2, 1.3, 0.2, 'setosa')]
...
... >>> echo(DATA)  # doctest: +NORMALIZE_WHITESPACE
... [('Sepal length', 'Sepal width', 'Petal length', 'Petal width', 'Species'),
...  (5.8, 2.7, 5.1, 1.9, 'virginica'),
...  (5.1, 3.5, 1.4, 0.2, 'setosa'),
...  (5.7, 2.8, 4.1, 1.3, 'versicolor'),
...  (6.3, 2.9, 5.6, 1.8, 'virginica'),
...  (6.4, 3.2, 4.5, 1.5, 'versicolor'),
...  (4.7, 3.2, 1.3, 0.2, 'setosa')]
... """
>>>
>>> def echo(data):
...     return data

8.7.11. Test Exceptions

>>> # doctest: +SKIP
... """
... >>> echo()
... Traceback (most recent call last):
... NotImplementedError
... """
>>>
>>> def echo():
...     raise NotImplementedError
>>> # doctest: +SKIP
... """
... >>> echo()
... Traceback (most recent call last):
... NotImplementedError: This will work in future
... """
>>>
>>> def echo():
...     raise NotImplementedError('This will work in future')

8.7.12. Test Type

>>> # doctest: +SKIP
... """
... >>> result = echo(1)
... >>> type(result)
... <class 'int'>
...
... >>> result = echo(1.1)
... >>> type(result)
... <class 'float'>
...
... >>> result = echo(True)
... >>> type(result)
... <class 'bool'>
...
... >>> result = echo([1, 2])
... >>> type(result)
... <class 'list'>
...
... >>> result = echo([1, 2])
... >>> any(type(x) is int
... ...     for x in result)
... True
... """
>>>
>>> def echo(data):
...     return data

The following doctest will fail:

>>> # doctest: +SKIP
... """
... >>> add_numbers('one', 'two')
... Traceback (most recent call last):
... TypeError: not a number
...
... >>> add_numbers(True, 1)
... Traceback (most recent call last):
... ValueError: not a number
... """
>>>
>>> def add_numbers(a, b):
...     if not isinstance(a, (int, float)):
...         raise ValueError('c')
...     if not isinstance(b, (int, float)):
...         raise ValueError('not a number')
...     return a + b

Expected exception, got 2.0:

Expected:

Traceback (most recent call last): ValueError: not a number

Got:

2.0

This test will pass:

>>> # doctest: +SKIP
... """
... >>> add_numbers('one', 'two')
... Traceback (most recent call last):
... TypeError: not a number
...
... >>> add_numbers(True, 1)
... Traceback (most recent call last):
... ValueError: not a number
... """
>>>
>>> def add_numbers(a, b):
...     if type(a) not in (int, float):
...         raise ValueError('not a number')
...     if type(b) not in (int, float):
...         raise ValueError('not a number')
...     return a + b

8.7.13. Test Python Expressions

Using python statements in doctest:

>>> def echo(text):
...     """
...     >>> name = 'Mark Watney'
...     >>> print(name)
...     Mark Watney
...     """
...     return text
>>> def when(date):
...     """
...     >>> from datetime import datetime, timezone
...     >>> moon = datetime(1969, 7, 21, 17, 54, tzinfo=timezone.utc)
...     >>> when(moon)
...     1969-07-21 17:54 UTC
...     """
...     print(f'{date:%Y-%m-%d %H:%M %Z}')

8.7.14. Flags

  • DONT_ACCEPT_TRUE_FOR_1

  • DONT_ACCEPT_BLANKLINE

  • NORMALIZE_WHITESPACE

  • ELLIPSIS

  • IGNORE_EXCEPTION_DETAIL

  • SKIP

  • COMPARISON_FLAGS

  • REPORT_UDIFF

  • REPORT_CDIFF

  • REPORT_NDIFF

  • REPORT_ONLY_FIRST_FAILURE

  • FAIL_FAST

  • REPORTING_FLAGS

8.7.15. Case Studies

Docstring used for doctest:

>>> def apollo_dsky(noun, verb):
...     """
...     This is the Apollo Display Keyboard
...     It takes noun and verb
...
...     >>> apollo_dsky(6, 61)
...     Program selected. Noun: 06, verb: 61
...
...     >>> apollo_dsky(16, 68)
...     Program selected. Noun: 16, verb: 68
...     """
...     print(f'Program selected. Noun: {noun:02}, verb: {verb:02}')

Celsius to Kelvin conversion:

>>> def celsius_to_kelvin(data):
...     """
...     >>> celsius_to_kelvin([1,2,3])
...     [274.15, 275.15, 276.15]
...
...     >>> celsius_to_kelvin((1,2,3))
...     [274.15, 275.15, 276.15]
...     """
...     return [x+273.15 for x in data]
>>> def celsius_to_kelvin(data):
...     """
...     >>> celsius_to_kelvin([1,2,3])
...     [274.15, 275.15, 276.15]
...
...     >>> celsius_to_kelvin((1,2,3))
...     (274.15, 275.15, 276.15)
...     """
...     cls = type(data)
...     return cls(x+273.15 for x in data)

Adding two numbers:

>>> def add_numbers(a, b):
...     """
...     >>> add_numbers(1, 2)
...     3.0
...     >>> add_numbers(-1, 1)
...     0.0
...     >>> add_numbers(0.1, 0.2)  # doctest: +ELLIPSIS
...     0.3000...
...     >>> add_numbers(1.5, 2.5)
...     4.0
...     >>> add_numbers(1, 1.5)
...     2.5
...     >>> add_numbers([1, 2], 3)
...     Traceback (most recent call last):
...     ValueError: not a number
...
...     >>> add_numbers(0, [1, 2])
...     Traceback (most recent call last):
...     ValueError: not a number
...
...     >>> add_numbers('one', 'two')
...     Traceback (most recent call last):
...     ValueError: not a number
...
...     >>> add_numbers(True, 1)
...     Traceback (most recent call last):
...     ValueError: not a number
...     """
...     if type(a) not in (int, float):
...         raise ValueError('not a number')
...
...     if type(b) not in (int, float):
...         raise ValueError('not a number')
...
...     return float(a + b)

Celsius to Kelvin temperature conversion:

>>> def celsius_to_kelvin(celsius):
...     """
...     >>> celsius_to_kelvin(0)
...     273.15
...
...     >>> celsius_to_kelvin(1)
...     274.15
...
...     >>> celsius_to_kelvin(-1)
...     272.15
...
...     >>> celsius_to_kelvin(-273.15)
...     0.0
...
...     >>> celsius_to_kelvin(-273.16)
...     Traceback (most recent call last):
...     ValueError: Negative Kelvin
...
...     >>> celsius_to_kelvin(-300)
...     Traceback (most recent call last):
...     ValueError: Negative Kelvin
...
...     >>> celsius_to_kelvin(True)
...     Traceback (most recent call last):
...     TypeError: Argument must be: int, float or Sequence[int, float]
...
...     >>> celsius_to_kelvin([0, 1, 2, 3])
...     [273.15, 274.15, 275.15, 276.15]
...
...     >>> celsius_to_kelvin({0, 1, 2, 3})
...     {273.15, 274.15, 275.15, 276.15}
...
...     >>> celsius_to_kelvin([0, 1, 2, -300])
...     Traceback (most recent call last):
...     ValueError: Negative Kelvin
...
...     >>> celsius_to_kelvin([0, 1, [2, 3], 3])
...     [273.15, 274.15, [275.15, 276.15], 276.15]
...     """
...     datatype = type(celsius)
...
...     if type(celsius) in {list, tuple, set, frozenset}:
...         return datatype(celsius_to_kelvin(x) for x in celsius)
...
...     if datatype not in {int, float}:
...         raise TypeError('Argument must be: int, float or Sequence[int, float]')
...
...     kelvin = celsius + 273.15
...
...     if kelvin < 0.0:
...         raise ValueError('Negative Kelvin')
...
...     return float(kelvin)

8.7.16. Assignments

Code 8.18. Solution
"""
* Assignment: Function Doctest Temperature
* Complexity: easy
* Lines of code: 5 lines
* Time: 13 min

English:
    1. Use data from "Given" section (see below)
    2. Write implementation of a function `celsius_to_kelvin`
    3. All tests must pass

Polish:
    1. Użyj danych z sekcji "Given" (patrz poniżej)
    2. Napisz implementację funkcji `celsius_to_kelvin`
    3. Wszystkie testy muszą przechodzić

Tests:
    >>> from inspect import isfunction
    >>> isfunction(celsius_to_kelvin)
    True
    >>> celsius_to_kelvin(0)
    273.15
    >>> celsius_to_kelvin(1)
    274.15
    >>> celsius_to_kelvin(-1)
    272.15
    >>> celsius_to_kelvin('a')
    Traceback (most recent call last):
    TypeError: Invalid argument
    >>> celsius_to_kelvin([0, 1])
    [273.15, 274.15]
    >>> celsius_to_kelvin((0, 1))
    (273.15, 274.15)
    >>> celsius_to_kelvin({0, 1})
    {273.15, 274.15}
"""


Code 8.19. Solution
"""
* Assignment: Function Doctest Distance
* Complexity: easy
* Lines of code: 21 lines
* Time: 13 min

English:
    1. Write functions which convert distance given in kilometers to meters
    2. 1 km = 1000 m
    3. Distance cannot be negative
    4. Returned value must by float
    5. Write doctests
    6. All tests must pass

Polish:
    1. Napisz funkcję, która przeliczy dystans podany w kilometrach na metry
    2. 1 km = 1000 m
    3. Dystans nie może być ujemny
    4. Zwracany dystans musi być float
    5. Napisz doctesty
    6. Wszystkie testy muszą przechodzić

Tests:
    1. Valid arguments:
        a. `int`
        b. `float`
    2. Invalid argument -> expect `TypeError`
        a. `bool`
        b. `str`
        c. `list[int]`
        d. `list[float]`
        e. any other type
"""


# Given
def km_to_meters(kilometers):
    ...


Code 8.20. Solution
"""
* Assignment: Function Doctest Regexp
* Complexity: almost impossible
* Lines of code: 0 lines
* Time: 5 min

English:
    1. Do not write any code, this is discussion only!
    2. Use data from "Given" section (see below)
    3. Pattern incorrectly classifies `https://foo_bar.example.com/` as invalid
    4. Fix pattern without writing automated tests
    5. Don't break classification of the other cases
    6. Discuss whether this is sane and how tests are helping?

Polish:
    1. Nie pisz żadnego kodu, to jest tylko dyskusja!
    2. Użyj danych z sekcji "Given" (patrz poniżej)
    3. Wyrażenie niepoprawnie klasyfikuje `https://foo_bar.example.com/` jako nieprawidłowy
    4. Popraw wyrażenie bez pisania testów automatycznych
    5. Nie zepsuj klasyfikacji pozostałych przypadków
    6. Przedyskutuj czy ma to sens i czy jak pomagają?

Tests:
    >>> type(PATTERN)
    <class 'str'>
    >>> import re
    >>> url = re.compile(PATTERN)
    # Note that (?:[a-z\x{00a1} will fail in Python
"""

# Author: @diegoperini
# Source: https://mathiasbynens.be/demo/url-regex
PATTERN = r'_^(?:(?:https?|ftp)://)(?:\S+(?::\S*)?@)?(?:(?!10(?:\.\d{1,3}){3})' \
          r'(?!127(?:\.\d{1,3}){3})(?!169\.254(?:\.\d{1,3}){2})(?!192\.168(?:\.\d{1,3}){2})' \
          r'(?!172\.(?:1[6-9]|2\d|3[0-1])(?:\.\d{1,3}){2})(?:[1-9]\d?|1\d\d|2[01]\d|22[0-3])' \
          r'(?:\.(?:1?\d{1,2}|2[0-4]\d|25[0-5])){2}(?:\.(?:[1-9]\d?|1\d\d|2[0-4]\d|25[0-4]))' \
          r'|(?:(?:[a-z\x{00a1}-\x{ffff}0-9]+-?)*[a-z\x{00a1}-\x{ffff}0-9]+)' \
          r'(?:\.(?:[a-z\x{00a1}-\x{ffff}0-9]+-?)*[a-z\x{00a1}-\x{ffff}0-9]+)*' \
          r'(?:\.(?:[a-z\x{00a1}-\x{ffff}]{2,})))(?::\d{2,5})?(?:/[^\s]*)?$_iuS'

VALID = [
    'http://foo.com/blah_blah',
    'http://foo.com/blah_blah/',
    'http://foo.com/blah_blah_(wikipedia)',
    'http://foo.com/blah_blah_(wikipedia)_(again)',
    'http://www.example.com/wpstyle/?p=364',
    'https://www.example.com/foo/?bar=baz&inga=42&quux',
    'http://✪df.ws/123',
    'http://myusername:mypassword@example.com:8080',
    'http://myusername:mypassword@example.com:8080/',
    'http://myusername@example.com',
    'http://myusername@example.com/',
    'http://myusername@example.com:8080',
    'http://myusername@example.com:8080/',
    'http://myusername:mypassword@example.com',
    'http://myusername:mypassword@example.com/',
    'http://142.42.1.1/',
    'http://142.42.1.1:8080/',
    'http://➡.ws/䨹',
    'http://⌘.ws',
    'http://⌘.ws/',
    'http://foo.com/blah_(wikipedia)#cite-1',
    'http://foo.com/blah_(wikipedia)_blah#cite-1',
    'http://foo.com/unicode_(✪)_in_parens',
    'http://foo.com/(something)?after=parens',
    'http://☺.damowmow.com/',
    'http://code.google.com/events/#&product=browser',
    'http://j.mp',
    'ftp://foo.bar/baz',
    'http://foo.bar/?q=Test%20URL-encoded%20stuff',
    'http://مثال.إختبار',
    'http://例子.测试',
    'http://उदाहरण.परीक्षा',
    'http://-.~_!$&\'()*+,;=:%40:80%2f::::::@example.com',
    'http://1337.net',
    'http://a.b-c.de',
    'http://223.255.255.254',
    'https://foo_bar.example.com/']

INVALID = [
    'http://',
    'http://.',
    'http://..',
    'http://../',
    'http://?',
    'http://??',
    'http://??/',
    'http://#',
    'http://##',
    'http://##/',
    'http://foo.bar?q=Spaces',
    '//',
    '//a',
    '///a',
    '///',
    'http:///a',
    'foo.com',
    'rdar://1234',
    'h://test',
    'http:// shouldfail.com',
    ':// should fail',
    'http://foo.bar/foo(bar)baz quux',
    'ftps://foo.bar/',
    'http://-error-.invalid/',
    'http://a.b--c.de/',
    'http://-a.b.co',
    'http://a.b-.co',
    'http://0.0.0.0',
    'http://10.1.1.0',
    'http://10.1.1.255',
    'http://224.1.1.1',
    'http://1.1.1.1.1',
    'http://123.123.123',
    'http://3628126748',
    'http://.www.foo.bar/',
    'http://www.foo.bar./',
    'http://.www.foo.bar./',
    'http://10.1.1.1',
    'http://10.1.1.254']