5.3.1 Core Internal Procedures

See section A POSIX conforming test framework, for more detailed explanations of the test outcomes (‘FAIL’, ‘PASS’, ‘UNTESTED’, ‘UNRESOLVED’, ‘UNSUPPORTED’).

perror "string number"

Declares a severe error in the testing framework itself. perror writes in the log files a message beginning with ‘ERROR’, appending the argument string. If the optional number is supplied, then this is used to set the internal count of errors to that value.

As a side effect, perror also changes the effect of the next pass or fail command: the test outcome becomes ‘UNRESOLVED’, since an automatic ‘PASS’ or ‘FAIL’ cannot be trusted after a severe error in the test framework. If the optional numeric value is ‘0’, then there are no further side effects to calling this function, and the following test outcome doesn’t become ‘UNRESOLVED’. This can be used for errors with no known side effects.

warning "string number"

Declares detection of a minor error in the test case itself. warning writes in the log files a message beginning with ‘WARNING’, appending the argument string. Use warning rather than error for cases (such as communication failure to be followed by a retry) where the test case can recover from the error. If the optional number is supplied, then this is used to set the internal count of warnings to that value.

As a side effect, warning_threshold or more calls to warning in a single test case also changes the effect of the next pass or fail command: the test outcome becomes ‘UNRESOLVED’ since an automatic ‘PASS’ or ‘FAIL’ may not be trustworthy after many warnings. If the optional numeric value is ‘0’, then there are no further side effects to calling this function, and the following test outcome doesn’t become ‘UNRESOLVED’. This can be used for errors with no known side effects.

note "string"

Appends an informational message to the log file. note writes in the log files a message beginning with ‘NOTE’, appending the argument string. Use note sparingly. verbose should be used for most such messages, but in cases where a message is needed in the log file regardless of the verbosity level use note.

pass "string"

Declares a test to have passed. pass writes in the log files a message beginning with ‘PASS’ (or XPASS/KPASS, if failure was expected), appending the argument string.

fail "string"

Declares a test to have failed. fail writes in the log files a message beginning with ‘FAIL’ (or XFAIL, if failure was expected), appending the argument string.

unresolved "string"

Declares a test to have an unresolved outcome. unresolved writes in the log file a message beginning with ‘UNRESOLVED’, appending the argument string. This usually means the test did not execute as expected, and a human being must go over results to determine if it passed or failed (and to improve the test case).

untested "string"

Declares a test was not run. untested writes in the log file a message beginning with ‘UNTESTED’, appending the argument string. For example, you might use this in a dummy test whose only role is to record that a test does not yet exist for some feature.

unsupported "string"

Declares that a test case depends on some facility that does not exist in the testing environment. unsupported writes in the log file a message beginning with ‘UNSUPPORTED’, appending the argument string.

get_warning_threshold

Returns the current value of warning_threshold. The default value is 3.

set_warning_threshold threshold

Sets the value of warning_threshold. A value of 0 disables it: calls to warning will not turn a ‘PASS’ or ‘FAIL’ into an ‘UNRESOLVED’.

transform "toolname"

Generates a string for the name of a tool as it was configured and installed, given its native name (as the argument toolname). This makes the assumption that all tools are installed using the same naming conventions: it extrapolates from the invocation name for ‘runtest’. For example, if you call runtest as ‘m68k-vxworks-runtest’, the result of ‘ transform "gcc" ’ is ‘m68k-vxworks-gcc’.

ishost "host"

Tests for a particular host environment. If the currently configured host matches the argument string, the result is 1; otherwise the result is 0. host must be a full three-part configure host name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters, using shell syntax, to specify sets of names).

istarget "target"

Tests for a particular target environment. If the currently configured target matches the argument string, the result is 1; otherwise the result is 0. target must be a full three-part configure target name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters, using shell syntax, to specify sets of names). If it is passed a NULL string, then it returns the name of the build canonical configuration.

isbuild "host"

Tests for a particular build host environment. If the currently configured host matches the argument string, the result is 1; otherwise the result is 0. host must be a full three-part configure host name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters, using shell syntax, to specify sets of names). If it is passed a NULL string, then it returns the name of the build canonical configuration.

item is3way "host" Tests for a canadian cross. This is when the tests will be run on a remotly hosted cross compiler. If it is a canadian cross, then the result is 1; otherwise the result is 0.

isnative

Tests whether the current configuration has the same host and target. When it runs in a native configuration this procedure returns a 1; otherwise it returns a 0.

load_lib "library-file"

Loads the file library-file by searching a fixed path built into runtest. If DejaGnu has been installed, it looks in a path starting with the installed library directory. If you are running DejaGnu directly from a source directory, without first running ‘make install’, this path defaults to the current directory. In either case, it then looks in the current directory for a directory called lib. If there are duplicate definitions, the last one loaded takes precedence over the earlier ones.

setup_xfail "config [bugid]"

Declares that the test is expected to fail on a particular set of configurations. The config argument must be a list of full three-part configure target name; in particular, you may not use the shorter nicknames supported by configure (but you can use the common shell wildcard characters to specify sets of names). The bugid argument is optional, and used only in the logging file output; use it as a link to a bug-tracking system such as GNATS (see (gnats.info)Overview section ‘Overview’ in Tracking Bugs With GNATS).

See notes under setup_kfail (below).

setup_kfail "config [bugid]"

Declares that the test is known to fail on a particular set of configurations. The config argument must be a list of full three-part configure target name; in particular, you may not use the shorter nicknames supported by configure (but you can use the common shell wildcard characters to specify sets of names). The bugid argument is mandatory, and used only in the logging file output; use it as a link to a bug-tracking system such as GNATS (see (gnats.info)Overview section ‘Overview’ in Tracking Bugs With GNATS).

Once you use setup_xfail or setup_kfail, the fail and pass procedures produce the messages ‘XFAIL’ or ‘KFAIL’ and ‘XPASS’ or ‘KPASS’ respectively, allowing you to distinguish expected/known failures (and unexpected success!) from other test outcomes.

If a test is marked as both expected to fail and known to fail for a certain configuration, a ‘KFAIL’ message will be generated. As ‘KFAIL’ messages are expected to draw more attention than the ‘XFAIL’ ones this will hopefuly ensure the test result is not overlooked.

Warning: you must clear the expected/known failure after using setup_xfail or setup_kfail in a test case. Any call to pass or fail clears the expectedknown failure implicitly; if the test has some other outcome, e.g. an error, you can call clear_xfail to clear the expected failure or clear_kfail to clear the known failure explicitly. Otherwise, the expected-failure declaration applies to whatever test runs next, leading to surprising results.

check_conditional_xfail message targets includes excludes

This procedure adds a conditional xfail, based on compiler options used to create a test case executable. If an include options is found in the compiler flags, and it’s the right architecture, it’ll trigger an XFAIL. Otherwise it’ll produce an ordinary FAIL. You can also specify flags to exclude. This makes a result be a FAIL, even if the included options are found. To set the conditional, set the variable compiler_conditional_xfail_data to the fields "[message string] [targets list] [includes list] [excludes list]" (descriptions below). This is the checked at pass/fail decision time, so there is no need to call the procedure yourself, unless you wish to know if it gets triggered. After a pass/fail, the variable is reset, so it doesn’t effect other tests.

The parameters are:

message

is the message to print with the normal test result

targets

is a string with the targets to activate this conditional on.

includes

is a list of sets of options to search for in the compiler options to activate this conditional. If the list of sets of options is empty or if any set of the options matches, then this conditional is true. (It may be useful to specify an empty list of include sets if the conditional is always true unless one of the exclude sets matches.)

excludes

is a list of sets of options to search for in the compiler options to activate this conditional. If any set of the options matches, (regardless of whether any of the include sets match) then this conditional is de-activated.

returns:

1

if the conditional is true

0

if the conditional is false

An example of setting the variable would be:

set compiler_conditional_xfail_data { \ "I sure wish I knew why this was hosed" \ "sparc*-sun*-* *-pc-*-*" \ {-"Wall -v" "-O3"} \ {-"O1" "-Map" } \ }

What this does is it matches only for these two targets if "-Wall -v" or "-O3" is set, but neither "-O1" or "-Map" is set.

For a set to match, the options specified are searched for independantly of each other, so a "-Wall -v" matches either "-Wall -v" or "-v -Wall". A space seperates the options in the string. Glob-style regular expressions are also permitted.

clear_xfail config

Cancel an expected failure (previously declared with setup_xfail) for a particular set of configurations. The config argument is a list of configuration target names. It is only necessary to call clear_xfail if a test case ends without calling either pass or fail, after calling setup_xfail.

clear_kfail config

Cancel a known failure (previously declared with setup_kfail) for a particular set of configurations. The config argument is a list of configuration target names. It is only necessary to call clear_kfail if a test case ends without calling either pass or fail, after calling setup_kfail.

verbose [-log] [-n] [--] [--] "string" number

Test cases can use this function to issue helpful messages depending on the number of ‘--verbose’ options on the runtest command line. It prints string if the value of the variable verbose is higher than or equal to the optional number. The default value for number is 1. Use the optional ‘-log’ argument to cause string to always be added to the log file, even if it won’t be printed. Use the optional ‘-n’ argument to print string without a trailing newline. Use the optional ‘--’ argument if string begins with "-". To log test results to a XML file use the ‘-x’ argument.