*** UNIX MANUAL PAGE BROWSER ***

KYUA-ATF-INTERFACE(1) FreeBSD General Commands Manual KYUA-ATF-INTERFACE(1) NAME atf-interface - Description of the ATF test program interface DESCRIPTION The interface of ATF test programs is the interface of the test programs linked to the atf-c, atf-c++ and atf-sh libraries provided by ATF. The ATF interface can be understood as the mechanisms used by test programs to communicate with the runtime engine as well as the assumptions that test programs and test cases can make while running. A test case is the most basic part of a test suite. A test case is supposed to reproduce one, and only one, scenario. For example: if the item under test was a function, the test case would provide a single set of input parameters to the function and check its output; If the item under test was a binary, the test case would provide a single set of arguments to the program and check its behavior. Test case parts Test cases have three parts: Head Programmatically defines metadata properties. The head must not perform any other thing than defining such properties. In particular, no testing whatsoever can happen in the head. (Ideally the definition of metadata properties would not happen programmatically.) Body The actual test case which performs any desired testing and reports a result. The body is executed by the runtime engine in a deterministic way; see the isolation section below. Cleanup An optional cleanup routine. Note that the runtime engine will attempt to clean up the work directory automatically, so this routine should only be provided in cases where the test modifies system-wide state not known by the runtime engine. The cleanup part is executed in the same directory as the body. However, the body and the cleanup parts do not share the same process space; the only way to pass data around from the body to the cleanup is by means of files in the work directory. Metadata properties The following test case metadata properties must be exported in the test case list for every test case: ident Single-word string. The name of the test case. Must be unique within the test program. The following test case metadata properties may be exported in the test case list for every test case: descr Multi-word string. A textual description for the test case. Usually, providing a descriptive identifier is better than providing a textual description. has.cleanup Boolean. Whether the test case defines a cleanup routine or not. require.arch Whitespace separated list of the architectures required by the test case. If defined, the test case is skipped unless the host architecture matches any of the values defined in this property. require.config Whitespace separated list of configuration variable names. The list of configuration variables that must be defined. The test is skipped if any of these is missing. require.files Whitespace separated list of absolute paths to installed files. If any of these files is not found, the test case is skipped. require.machine Whitespace separated list of the machine types required by the test case. If defined, the test case is skipped unless the host machine type matches any of the values defined in this property. require.progs Whitespace separated list of program names (either absolute names or base names). If any of these programs is not found, the test case is skipped. require.user Either `root' or `unprivileged'. If `root', the test case must be run as the superuser or otherwise it is skipped. If `unprivileged', the test case must be run as an unprivileged user or else it is skipped. timeout Integer. The amount of seconds the test case can run for before it is killed by the runtime engine. Configuration properties The following properties may be defined by the runtime engine and are propagated to the test cases: unprivileged-user String, optional. Specifies the name of the user under which tests that set `require.user=unprivileged' are executed. Results A test case must always report a result by creating the results file specified through the -r flag. For convenience when running test cases without the runtime engine, this file may point to /dev/stdout or /dev/stderr in which case the file must not be created (because the creation will fail). Aside from creating the results file, the process in which the test case runs must terminate in particular ways for the test result to be considered valid. If the test case fails to create the test result, if the test result is created but contains an invalid syntax, or if the termination status of the process does not match the requirements of the test result, the runtime engine marks the test case as `broken'. Note that the `broken' state is decided by the runtime engine; a test case cannot report itself as `broken'. The general syntax for the results file is as follows: <status>[[(int)]: reason] The following results are allowed: expected_death The process is expected to terminate either due to a clean call to exit(3) or due to the reception of a signal. The contents of the file are `expected_death: <reason>'. Example: `expected_death: Calling libdofoo breaks due to bug xyz'. expected_exit The process is expected to terminate cleanly. The contents of the file are `expected_exit: <reason>' if the exit code is irrelevant or `expected_exit(<exitcode>): <reason>' if the process must terminate with a given exit code. Examples: `expected_exit: Calling bar exits but it should not' or `expected_exit(123): Calling bar exits with an unexpected code'. expected_failure The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are `expected_failure: <reason>' Example: `expected_failure: 2 + 2 = 3'. expected_signal The process is expected to terminate due to the reception of a signal. The contents of the file are `expected_signal: <reason>' if the signal number is irrelevant or `expected_signal(<signalno>): <reason>' if the process must terminate due to a particular signal. Examples: `expected_signal: Calling bar crashes' or `expected_signal(1): Calling bar kills ourselves due to unhandled SIGHUP'. expected_timeout The process is expected to hang for longer than its timeout metadata property. Only the runtime engine can control this situation because the runtime engine is the one implementing the timeout functionality. failed The process must exit cleanly with an EXIT_FAILURE exit code. The contents of the file are `failed: <reason>'. Example: `failed: Failed on purpose'. passed The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are `passed'. skipped The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are `skipped: <reason>'. Example: `skipped: Skipped because the foo is not present'. Isolation The runtime engine attempts to isolate test cases from other test cases in the same test program and from the rest of the system by performing what is called test case isolation. Whenever the user runs a test program binary by hand (i.e. not through kyua(1)), the test program will print a warning message stating that test case isolation does not work and therefore the program may cause side- effects and/or report invalid values. The runtime engine must set the __RUNNING_INSIDE_ATF_RUN environment variable to the magic value `internal-yes-value' to tell the test programs that they are being run with isolation enabled. The test case isolation performs the following: Process space Each test case body and cleanup routines are executed in independent processes. Corollary: the test case can do whatever it wants to the current process (such as modifying global variables) without having to undo such changes. Process group The test case body and cleanup are executed in their own process groups. Should they spawn any children, such children should maintain the same process group. This is done to allow the runtime engine to kill the whole process subtree once the test case finishes (or if the test case hangs). Work directory The test case body and its cleanup (if any) are executed in a temporary directory automatically created by the runtime engine. This temporary directory is shared among the body and cleanup parts of a single test case but is completely separate from the temporary directories of other tests. Corollary: the test case body and cleanup routines can write to their current directory without bothering to clean any files and/or directories they create. The runtime engine takes care to recursively delete the temporary directories after the execution of a test case. Any file systems mounted within the temporary directory will be unmounted if possible. Home directory The HOME environment variable is set to the absolute path of the work directory. Umask The value of the umask is set to 0022. Environment The LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC and LC_TIME variables are unset. The TZ variable is set to `UTC'. Process limits The maximum soft core size limit is raised to its corresponding hard limit. This is a simple, best-effort attempt at allowing test cases to dump core for further diagnostic purposes. Test programs A test program is, simply put, a collection of related test cases. The test program can be seen as a command-line dispatcher for the test cases. A test program must provide one or more test cases. If it does not contain any test case, the runtime system will report it as invalid. Test programs expose their list of test cases in a machine parseable format. The runtime engine obtains the list of test cases to know what tests to run and to know how to set up the environment of each test prior execution. The test program must not do any test when asked to dump its test case list. The generic syntax to obtain the list of test cases included in a test program is: <test-program> -l The list of test cases follows the following format: LIST ::= HEADER NEWLINE TEST_CASES HEADER ::= 'Content-Type: application/X-atf-tp; version="1"' NEWLINE ::= '\n' TEST_CASES ::= TEST_CASE | TEST_CASE NEWLINE TEST_CASES TEST_CASE ::= IDENT_PROPERTY PROPERTIES IDENT_PROPERTY ::= 'ident' DELIM STRING NEWLINE DELIM ::= ': ' PROPERTIES ::= PROPERTY | PROPERTY PROPERTIES PROPERTY ::= PROPERTY_NAME DELIM STRING NEWLINE PROPERTY_NAME ::= (see below) An example: Content-Type: application/X-atf-tp; version="1" ident: addition descr: Tests that the addition function works ident: subtraction descr: Tests that the subtraction function works ident: remove descr: Tests removing files require.root: true timeout: 50 has.cleanup: true The syntax to run a test case body part is: <test-program> [-r resfile] [-s srcdir] [-v var=value]* <test-case>[:body] This must run the test case body "as is", without any attempt of isolating it from the rest of the system. It is the responsibility of the runtime engine to do such isolation. The runtime engine always passes the path of a nonexistent file to -r, which must be created by the test case; and always passes an absolute path to the -s flag pointing to the directory containing the test program executable. The runtime engine shall pass any configuration variables it wants through the -v flag, and these can be later inspected by the test case at will. A note to users: if you run the test case by hand (not through kyua(1) nor atf-run(1)) from the command line, none of the isolation features described in the isolation section apply. This means that the test case can (and probably will) write to the current directory and leave garbage behind. Also, given that the test case is executed without e.g. clearing the environment, the results of the test case may differ from those obtained when running the test case inside the runtime engine. Only use this for debugging purposes (i.e. to run the test case code under GDB). The syntax to run a test case cleanup part is: <test-program> [-s srcdir] [-v var=value]* <test-case>:cleanup This can only be performed if and only if the test case sets the has.cleanup property to true. Otherwise the behavior of executing the cleanup part is undefined. The same rules for -s and -v apply as to when running the body. The cleanup part must be executed in the same directory as the body but in a separate process space. The only way for test cases to transfer state (if any) from the body to the cleanup routine is by means of files in the current directory. The cleanup part does not have to worry about deleting temporary files created in the current directory. The runtime engine does this automatically. SEE ALSO kyua-test(1), kyuafile(5) FreeBSD 14.1-RELEASE-p8 September 9, 2012 FreeBSD 14.1-RELEASE-p8