2.4 A POSIX conforming test framework

DejaGnu conforms to the POSIX standard for test frameworks.

POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET.(3)

The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says “The sun shall shine”, a corresponding assertion might be “The sun is shining.” A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.

As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. DejaGnu supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:

PASS

A test has succeeded. That is, it demonstrated that the assertion is true.

FAIL

A test has produced the bug it was intended to capture. That is, it has demonstrated that the assertion is false. The FAIL message is based on the test case only. Other messages are used to indicate a failure of the framework.

UNRESOLVED

A test produced indeterminate results. Usually, this means the test executed in an unexpected fashion; this outcome requires that a human being go over results, to determine if the test should have passed or failed. This message is also used for any test that requires human intervention because it is beyond the abilities of the testing framework. Any unresolved test should resolved to PASS or FAIL before a test run can be considered finished.

Note that for POSIX, each assertion must produce a test result code. If the test isn’t actually run, it must produce UNRESOLVED rather than just leaving that test out of the output. This means that you have to be careful when writing tests, to not carelessly use tcl statements like return—if you alter the flow of control of the tcl code you must insure that every test still produces some result code.

Here are some of the ways a test may wind up UNRESOLVED:

• A test’s execution is interrupted.
• A test does not produce a clear result. This is usually because there was an ERROR from DejaGnu while processing the test, or because there were three or more WARNING messages. Any WARNING or ERROR messages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened—and to improve the test case.
• A test depends on a previous test, which fails.
• The test was set up incorrectly.

UNTESTED

A test was not run. This is a placeholder, used when there is no real test case yet.

The only remaining POSIX output message left is intended to test features that are specified by the applicable POSIX standard as conditional:

UNSUPPORTED

There is no support for the tested case. This may mean that a conditional feature of an operating system, or of a compiler, is not implemented. DejaGnu also uses this message when a testing environment (often a “bare board” target) lacks basic support for compiling or running the test case. For example, a test for the system subroutine gethostname would never work on a target board running only a boot monitor.

DejaGnu uses the same output procedures to produce these messages for all test suites, and these procedures are already known to conform to POSIX 1003.3. For a DejaGnu test suite to conform to POSIX 1003.3, you must avoid the setup_xfail and setup_kfail procedures (see below), and you must be careful to return UNRESOLVED where appropriate, as described in the UNRESOLVED section above.

Besides the POSIX messages, DejaGnu provides for variations of the PASS and FAIL messages that can be helpful for the tool maintainers. It must be noted, however, that this feature is not POSIX 1003.3 compliant, so its use should be avoided if compliance is necessary.

The additional messages are:

XFAIL

A test is expected to fail in some environment(s) due to some bug in the environment that we hope is fixed someday (but we can’t do nothing about as it is not a bug in the tool that we are testing). The procedure setup_xfail is used to indicate that a failure is expected.

POSIX 1003.3 does not incorporate the notion of expected failures, so POSIX tests must return FAIL rather than XFAIL even if a failure was expected.

KFAIL

A test is known to fail in some environment(s) due to a known bug in the tool being tested (identified by a bug id string). This exists so that, after a bug is identified and properly registered in a bug tracking database (Gnats, for instance), the count of failures can be kept as zero. Having zero as a baseline in all platforms allow the tool developers to immediately detect regressions caused by changes (which may affect some platforms and not others). The connection with a bug tracking database allows for automatic generation of the BUGS section of man pages or Release Notes, as well as a "Bugs Fixed this Release" section (by comparing to a previous release set of known failures). The procedure setup_kfail is used to indicate a failure is known to exist.

As with XFAIL, POSIX tests must return FAIL rather than KFAIL even if a failure was due to a known bug.

XPASS

A test was expected to fail with XFAIL but passed instead. Whatever problem that used to exist in the environment was corrected The test may also be failing to detect the failure due to some environment or output changes, so this possibility must be investigated as well.

KPASS

A test was expected to fail with KFAIL but passed instead. Someone may have fixed the bug and failed to unmark the test. As for XPASS, the test may also be failing to detect the failure due to some environment or output changes, so this possibility must also be checked.

PASS, instead of XPASS or KPASS, must also be returned for test cases which were expected to fail and did not, if POSIX 1003.3 compliance is required. This means that PASS is in some sense more ambiguous than if XPASS and KPASS are also used.

See also Using runtest. For information on how to mark tests as expected/known to fail by using setup_xfail and setup_kfail, see Core Internal Procedures.