Skip to content

Unit tests and code analysis

Unit tests, as well as more-involved functional ones, are implemented under the test/ folder of the declearn gitlab repository. Functional tests are isolated in the test/functional/ subfolder to enable one to easily exclude them in order to run unit tests only.

Tests are implemented using the PyTest framework, as well as some third-party plug-ins that are automatically installed with the package when using pip install declearn[tests].

Additionally, code analysis tools are configured through the pyproject.toml file, and used to control code quality upon merging to the main branch. These tools are black for code formatting, pylint for overall static code analysis and mypy for static type-cheking.

Running the unit tests suite

Running the test suite using tox

The third-party tox tool may be used to run the entire test suite within a dedicated virtual environment. Simply run tox from the commandline with the root repo folder as working directory. You may optionally specify the python version(s) with which you want to run tests.

tox           # run with default python 3.8
tox -e py310  # override to use python 3.10

Note that additional parameters for pytest may be passed as well, by adding -- followed by any set of options you want at the end of the tox command. For example, to use the declearn-specific --fulltest option (see the section below), run:

tox [tox options] -- --fulltest

The tests pipeline specified under the tox.ini file runs the following:

  • install declearn in an isolated environment
  • run unit tests
  • run functional tests
  • run pylint on declearn, then on the tests' code
  • run mypy on declearn
  • run black on declearn, in check mode

Running unit tests using pytest

To run all the tests, simply use:

pytest test

To run the tests under a given module (here, "model"):

pytest test/model

To run the tests under a given file (here, "test_regression.py"):

pytest test/functional/test_regression.py

Note that by default, some test scenarios that are considered somewhat superfluous~redundant will be skipped in order to save time. To avoid skipping these, and therefore run a more complete test suite, add the --fulltest option to pytest:

pytest --fulltest test  # or any more-specific target you want

For more details on how to run targetted tests, please refer to the pytest documentation.

You may also arguments to compute and export coverage statistics, using the pytest-cov plug-in:

# Run all tests and export coverage information in HTML format.
pytest --cov=declearn --cov-report=html tests/

Running black to format the code

The black code formatter is used to enforce uniformity of the source code's formatting style. It is configured to have a maximum line length of 79 (as per PEP 8) and ignore auto-generated protobuf files, but will otherwise modify files in-place when executing the following commands from the repository's root folder:

black declearn  # reformat the package
black test      # reformat the tests

Note that it may also be called on individual files or folders. One may "blindly" run black, however it is actually advised to have a look at the reformatting operated, and act on any readability loss due to it. A couple of advice:

  1. Use #fmt: off / #fmt: on comments sparingly, but use them.
    It is totally okay to protect some (limited) code blocks from reformatting if you already spent some time and effort in achieving a readable code that black would disrupt. Please consider refactoring as an alternative (e.g. limiting the nest-depth of a statement).

  2. Pre-format functions and methods' signature to ensure style homogeneity.
    When a signature is short enough, black may attempt to flatten it as a one-liner, whereas the norm in declearn is to have one line per argument, all of which end with a trailing comma (for diff minimization purposes). It may sometimes be necessary to manually write the code in the latter style for black not to reformat it.

Finally, note that the test suite run with tox comprises code-checking by black, and will fail if some code is deemed to require alteration by that tool. You may run this check manually:

black --check declearn  # or any specific file or folder

Running pylint to check the code

The pylint linter is expected to be used for static code analysis. As a consequence, # pylint: disable=[some-warning] comments can be found (and added) to the source code, preferably with some indication as to the rationale for silencing the warning (or error).

A minimal amount of non-standard hyper-parameters are configured via the pyproject.toml file and will automatically be used by pylint when run from within the repository's folder.

Most code editors enable integrating the linter to analyze the code as it is being edited. To lint the entire package (or some specific files or folders) one may simply run pylint:

pylint declearn  # analyze the package
pylint test      # analyze the tests

Note that the test suite run with tox comprises the previous two commands, which both result in a score associated with the analyzed code. If the score does not equal 10/10, the test suite will fail - notably preventing acceptance of merge requests.

Running mypy to type-check the code

The mypy linter is expected to be used for static type-checking code analysis. As a consequence, # type: ignore comments can be found (and added) to the source code, as sparingly as possible (mostly, to silence warnings about untyped third-party dependencies, false-positives, or locally on closure functions that are obvious enough to read from context).

Code should be type-hinted as much and as precisely as possible - so that mypy actually provides help in identifying (potential) errors and mistakes, with code clarity as final purpose, rather than being another linter to silence off.

A minimal amount of parameters are configured via the pyproject.toml file, and some of the strictest rules are disabled as per their default value (e.g. Any expressions are authorized - but should be used sparingly).

Most code editors enable integrating the linter to analyze the code as it is being edited. To lint the entire package (or some specific files or folders) one may simply run mypy:

mypy declearn

Note that the test suite run with tox comprises the previous command. If mypy identifies errors, the test suite will fail - notably preventing acceptance of merge requests.