15.1 Simple Tests

If the variable TESTS is defined, its value is taken to be a list of programs or scripts to run in order to do the testing. Programs needing data files should look for them in srcdir (which is both an environment variable and a make variable) so they work when building in a separate directory (see (autoconf)Build Directories section `Build Directories' in The Autoconf Manual), and in particular for the distcheck rule (see section Checking the Distribution).

For each of the TESTS, the result of execution is printed along with the test name, where PASS denotes a successful test, FAIL denotes a failed test, XFAIL an expected failure, XPASS an unexpected pass for a test that is supposed to fail, and SKIP denotes a skipped test.

The number of failures will be printed at the end of the run. If a given test program exits with a status of 77, then its result is ignored in the final count. This feature allows non-portable tests to be ignored in environments where they don't make sense.

If the Automake option color-tests is used (see section Changing Automake's Behavior) and standard output is connected to a capable terminal, then the test results and the summary are colored appropriately. The user can disable colored output by setting the make variable ‘AM_COLOR_TESTS=no’, or force colored output even without a connecting terminal with ‘AM_COLOR_TESTS=always’.

The variable TESTS_ENVIRONMENT can be used to set environment variables for the test run; the environment variable srcdir is set in the rule. If all your test programs are scripts, you can also set TESTS_ENVIRONMENT to an invocation of the shell (e.g. ‘$(SHELL) -x’ can be useful for debugging the tests), or any other interpreter. For instance the following setup is used by the Automake package to run four tests in Perl.

TESTS_ENVIRONMENT = $(PERL) -Mstrict -I $(top_srcdir)/lib -w
TESTS = Condition.pl DisjConditions.pl Version.pl Wrap.pl

You may define the variable XFAIL_TESTS to a list of tests (usually a subset of TESTS) that are expected to fail. This will reverse the result of those tests.

Automake ensures that each file listed in TESTS is built before any tests are run; you can list both source and derived programs (or scripts) in TESTS; the generated rule will look both in srcdir and ‘.’. For instance, you might want to run a C program as a test. To do this you would list its name in TESTS and also in check_PROGRAMS, and then specify it as you would any other program.

Programs listed in check_PROGRAMS (and check_LIBRARIES, check_LTLIBRARIES...) are only built during make check, not during make all. You should list there any program needed by your tests that does not need to be built by make all. Note that check_PROGRAMS are not automatically added to TESTS because check_PROGRAMS usually lists programs used by the tests, not the tests themselves. Of course you can set TESTS = $(check_PROGRAMS) if all your programs are test cases.