Thorough testing: DebTest¶
Ideally software comes with an exhaustive test suite that can be used to determine whether this particular software works as expected on the Debian platform. However, especially for complex software, these test suites are often resource hungry (CPU time, memory, disk space, network bandwidth) and cannot be ran at package build time by Debian’s buildds. Consequently, test suites are typically utilized manually and only by the respective packager on a particular machine, before uploading a new version to the archive.
However, Debian is an integrated system and packaged software typically relies on functionality provided by other Debian packages (e.g. shared libraries) instead of shipping duplicates with different versions in every package – for many good reasons. Unfortunately, there is also a downside to this: Debian packages often use versions of 3rd-party tools that are different from those tested by upstream, and moreover, the actual versions of dependencies might change frequently between subsequent uploads of a dependent package. Currently a change in a dependency that introduces an incompatibility cannot be detected reliably even if upstream provides a test suite that would have caught the breakage. Therefore integration testing heavily relies on users to detect incorrect functioning and file bug reports. Although there are archive-wide QA efforts (e.g. constantly rebuilding all packages) these tests can only detect API/ABI breakage or functionality tested during build-time checks – they are not exhaustive for the aforementioned reasons.
This is a proposal to, first of all, package upstream test suites in a way that they can be used to run expensive archive-wide QA tests. However, this is also a proposal to establish means to test interactions between software from multiple Debian packages to provide more thorough continued integration and regression testing for the Debian systems.
To address these open issues we are working on DebTest – a framework with conventions and tools that allow Debian to distribute test batteries developed by upstream or Debian developers. It aims at complementing existing QA efforts by going beyond single-package, build-time tests and cover interactions between software from multiple Debian packages to provide more thorough continued integration and regression testing for the Debian systems DebTest aims to enable developers and users to perform extensive testing of a deployed Debian system or a particular software of interest in a uniform fashion.
Status¶
The project is still in an early conceptual stage. We are currently working on a SPEC draft that will be submitted to the Debian community for further discussion of the desired properties of a more comprehensive testing framework. Furthermore, we started looking for existing (free) software solutions that might be used to implement such a framework.
We have already started packaging versatile datasets that can be used to develop test suites.
Todo
DebTest
Finish the SPEC.
Initiate discussion.
Identify and package relevant neuroscience datasets that can be used to develop multi-software regression/pipeline tests.
References¶
pkgs-by_purpose-neuroscience_datasets