3. Using runtest

runtest is the executable test driver for DejaGnu. You can specify two kinds of things on the runtest command line: command line options, and Tcl variables for the test scripts. The options are listed alphabetically below.

runtest returns an exit code of 1 if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns 0 as the exit code.

runtest flags the outcome of each test as one of these cases. (See section A POSIX conforming test framework, for a discussion of how POSIX specifies the meanings of these cases.)

PASS

The most desirable outcome: the test succeeded, and was expected to succeed.

XPASS

A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.

KPASS

A pleasant kind of failure: a test was known to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.

FAIL

A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.

XFAIL

A test failed, but it was expected to fail. This result indicates no change in a known environment bug. If a test fails because the operating system where the test runs lacks some facility required by the test (i.e. failure is due to the lack of a feature, not the existence of a bug), the outcome is UNSUPPORTED instead.

KFAIL

A test failed, but it was known to fail. This result indicates no change in a known bug. If a test fails because of a problem in the environment, not in the tool being tested, that is expected to be fixed one day, the outcome is XFAIL instead.

UNRESOLVED

Output from a test requires manual inspection; the test suite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.

UNTESTED

A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy “tests” that note explicitly the absence of a real test case for a particular property.

UNSUPPORTED

A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.

runtest may also display the following messages:

ERROR

Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX test suites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.)

WARNING

Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.

NOTE

An informational message about the test case.

This is the full set of command line options that runtest recognizes. Arguments may be abbreviated to the shortest unique string.

runtest --tool tool  [ testsuite.exp … ]
[ testsuite.exp="testfile1 …" ]
[ tclvar=value… ]
[ --all ]  [ --baud baud-rate ]  [ --connect type ]
[ --debug ]  [ --help ]  [ --host string ]
[ --mail "name …" ]  [ --name string ]
[ --name name ]  [ --outdir path ]
[ --objdir path ]  [ --reboot ]
[ --srcdir path ]  [ --strace n ]
[ --target string --build string ]
[ -v | --verbose ]  [ -V | --version ]  [ --Dn ]

--tool tool

tool specifies what set of tests to run, and what initialization module to use. tool is used only for these two purposes: it is not used to name the executable program to test. Executable tool names (and paths) are recorded in ‘site.exp’ (see section Configuration dependent values), and you can override them by specifying Tcl variables on the command line.

For example, including ‘--tool gcc’ on the runtest command line runs tests from all test subdirectories whose names match ‘gcc.*’, and uses one of the initialization modules named ‘config/*-gcc.exp’. To specify the name of the compiler (perhaps as an alternative path to what runtest would use by default), use ‘GCC=binname’ on the runtest command line.

testsuite.exp …

Specify the names of testsuites to run. By default, runtest runs all tests for the tool, but you can restrict it to particular testsuites by giving the names of the ‘.exp’ expect scripts that control them.

testsuite.exp may not include path information; use plain filenames.

testfile.exp="testfile1 …"

Specify a subset of tests in a suite to run. For compiler or assembler tests, which often use a single ‘.exp’ script covering many different source files, this option allows you to further restrict the tests by listing particular source files to compile. Some tools even support wildcards here. The wildcards supported depend upon the tool, but typically they are ?, *, and [chars].

tclvar=value

You can define Tcl variables for use by your test scripts in the same style used with make for environment variables. For example, ‘runtest GDB=gdb.old’ defines a variable called ‘GDB’; when your scripts refer to ‘$GDB’ in this run, they use the value ‘gdb.old’.

The default Tcl variables used for most tools are defined in the main DejaGnu Makefile; their values are captured in the ‘site.exp’ file. See section Configuration dependent values.

--all

Display all test output. By default, runtest shows only the output of tests that produce unexpected results; that is, tests with status ‘FAIL’ (unexpected failure), ‘XPASS’ or ‘KPASS’ (unexpected success), or ‘ERROR’ (a severe error in the test case itself). Specify ‘--all’ to see output for tests with status ‘PASS’ (success, as expected) ‘XFAIL’ (failure, as expected), or ‘WARNING’ (minor error in the test case itself).

--baud baud-rate

-b baud-rate

Set the default baud rate to something other than 9600. (Some serial interface programs, like tip, use a separate initialization file instead of this value.)

--connect type

Connect to a target testing environment as specified by type, if the target is not the computer running runtest. For example, use ‘--connect’ to change the program used to connect to a “bare board” boot monitor. The choices for type in the DejaGnu 1.0 distribution are ‘rlogin’, ‘telnet’, ‘rsh’, ‘tip’, ‘kermit’, and ‘mondfe’.

The default for this option depends on the configuration (see section Remote targets supported). The default is chosen to be the most convenient communication method available, but often other alternatives work as well; you may find it useful to try alternative connect methods if you suspect a communication problem with your testing target.

--debug

Turns on the expect internal debugging output. Debugging output is displayed as part of the runtest output, and logged to a file called ‘dbg.log’. The extra debugging output does not appear on standard output, unless the verbose level is greater than 2 (for instance, to see debug output immediately, specify ‘--debug -v -v’). The debugging output shows all attempts at matching the test output of the tool with the scripted patterns describing expected output. The output generated with ‘--strace’ also goes into ‘dbg.log’.

--help

-he

Prints out a short summary of the runtest options, then exits (even if you also specify other options).

--host string

string is a full configuration “triple” name as used by configure. Use this option to override the default string recorded by your configuration’s choice of host. This choice does not change how anything is actually configured unless –build is also specified; it affects only DejaGnu procedures that compare the host string with particular values. The procedures ishost, istarget, isnative, setup_xfail and setup_kfail are affected by ‘--host’. In this usage, host refers to the machine that the tests are to be run on, which may not be the same as the build machine. If --build is also specified, then --host refers to the machine that the tests wil, be run on, not the machine DejaGnu is run on.

--build string

string is a full configuration “triple” name as used by configure. This is the type of machine DejaGnu and the tools to be tested are built on. For a normal cross this is the same as the host, but for a canadian cross, they are seperate.

--name name

name is a name for the particular testing target machine (for cross testing). If the testing target has IP network support (for example, RPC or NFS), this is the network name for the target itself. (name is not the configuration string you specify as a target with configure; the ‘--name’ option names a particular target, rather than describing a class of targets.) For targets that connect in other ways, the meaning of the name string depends on the connection method. See section Remote targets supported.

--name string

Specify a network name of testing target or its host. The particular names that are meaningful with ‘--name’ will depend on your site configuration, and on the connection protocol: for example, tip connections require names from a serial line configuration file (usually called ‘/etc/remote’), while telnet connections use IP hostnames.

--objdir path

Use path as the top directory containing any auxiliary compiled test code. This defaults to ‘.’. Use this option to locate pre-compiled test code. You can normally prepare any auxiliary files needed with make.

--outdir path

Write output logs in directory path. The default is ‘.’, the directory where you start runtest. This option affects only the summary and the detailed log files ‘tool.sum’ and ‘tool.log’. The DejaGnu debug log ‘dbg.log’ always appears (when requested) in the local directory.

--reboot

Reboot the target board when runtest initializes. Usually, when running tests on a separate target board, it is safer to reboot the target to be certain of its state. However, when developing test scripts, rebooting takes a lot of time.

--srcdir path

Use path as the top directory for test scripts to run. runtest looks in this directory for any subdirectory whose name begins with the toolname (specified with ‘--tool’). For instance, with ‘--tool gdb’, runtest uses tests in subdirectories ‘gdb.*’ (with the usual shell-like filename expansion). If you do not use ‘--srcdir’, runtest looks for test directories under the current working directory.

--strace n

Turn on internal tracing for expect, to n levels deep. By adjusting the level, you can control the extent to which your output expands multi-level Tcl statements. This allows you to ignore some levels of case or if statements. Each procedure call or control structure counts as one “level”.

The output is recorded in the same file, ‘dbg.log’, used for output from ‘--debug’.

--target string

Use this option to override the default setting (running native tests). string is a full configuration “triple” name(4) as used by configure. This option changes the configuration runtest uses for the default tool names, and other setup information. See (configure.info)Using configure section ‘Using configure’ in Cygnus configure, for details about configure names.

--verbose

-v

Turns on more output. Repeating this option increases the amount of output displayed. Level one (‘-v’) is simply test output. Level two (‘-v -v’) shows messages on options, configuration, and process control. Verbose messages appear in the detailed (‘*.log’) log file, but not in the summary (‘*.sum’) log file.

--version

-V

Prints out the version numbers of DejaGnu, expect and Tcl, and exits without running any tests.

-D0

-D1

Start the internal Tcl debugger. The Tcl debugger supports breakpoints, single stepping, and other common debugging activities. (See A Debugger for Tcl Applications by Don Libes. (5))

If you specify ‘-D1’, the expect shell stops at a breakpoint as soon as DejaGnu invokes it.

If you specify ‘-D0’, DejaGnu starts as usual, but you can enter the debugger by sending an interrupt (e.g. by typing <C-c>).