The testing framework can be used to make build reproducible and ensure that no regression is introduced when components are updated.
Support for the 'test' target should be ideally added to any component at creation or at update if it is not the case.
Makefile variables 'COMPONENT_TEST_*' affecting the execution of the test suite are declared in shared-macros.mk: most of these variables need not be changed.
After the component is built and installed test suites can be run using 'gmake test' if the 'test' target is declared in the component Makefile.
Example: tests for 32-bit and 64-bit versions
The environment of the test suite can be changed with the variable 'COMPONENT_TEST_ENV'.
If a master result file exists in the component test directory, the output produced by the test run is compared against master test results.
The default directory for master test results is 'test' as defined in the variable 'COMPONENT_TEST_RESULTS_DIR'.
Default names for the master files are defined in the variable 'COMPONENT_TEST_MASTER' as:
and correspond to 32-bit and 64-bit builds respectively.
The first time a master result file is added to the component, an empty file should be created prior to running 'gmake test' to trigger the comparison stage.
Example: initial creation of a master results file for a 32-bit build
The comparison will fail as the master results file is empty: the actual test output should be copied to the test result directory (do not forget to commit the newly created file).
If the results are architecture independent, the variable may be set to:
Test results may be processed to retain relevant machine-independent output only.
Transforms of the output are controlled by two variables:
Only test results are retained:
Timestamps should be removed: