Date: Mon, 11 May 2009 09:29:28 -0700 From: Zachary Loafman <zachary.loafman@isilon.com> To: arch@freebsd.org Subject: FAIL: kernel fault injection Message-ID: <20090511162928.GD17203@isilon.com>
next in thread | raw e-mail | index | archive | help
--rmUrFcWP4LYae1gV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Arch -
I'd like to contribute the kernel fault injection system that Isilon
uses. Before contributing it, I'd like to get approval for the APIs
involved.
Testing errors is hard. Let's say you have:
int foo(void) {
[...]
error = bar();
if (error) {
/* do stuff */
}
}
.. but some_func() can't reliably be made to fail. How do you test it?
We added error injection macros that look like this:
int foo(void) {
[...]
error = bar();
KFAIL_POINT_CODE(FP_KERN, bar_fails_foo, error = RETURN_VALUE);
if (error) {
/* do stuff */
}
}
The KFAIL_POINT_CODE macro adds a sysctl MIB that allows
you to inject errors into the above code. For example:
# sysctl fail_point.kern.bar_fails_foo=".1%return(5)"
This says, ".1% of the time, evaluate the fail point code with 5 as
the RETURN_VALUE". If this were a standard errno, you could read the
above setting as "1/1000th of the time, pretend bar() returned EIO".
We also have a few wrappers around KFAIL_POINT_CODE that essentially
wrap common uses. For example, the above use can be shorthanded to:
KFAIL_POINT_ERROR(FP_KERN, bar_fails_foo, error)
Currently, the sysctl parser accepts the following variants:
return(x) - triggers the code with RETURN_VALUE set to x
sleep(t) - sleep t milliseconds,
panic/break - panic or break into the debugger
print - print that the fail point was hit
In addition to the commands, we have a syntax to express the
when to evaluate those commands:
p%<command> - evaluate command p% of the time (example above)
5*<command> - evaluate command 5 times, then disable the expression
And you can compound with expr1->expr2, so, e.g.:
5%return(5)->1%return(22):
5% of the time, return 5, 1% of the remaining time, return 22
5*return(0)->10*return(5)->1%return(19)
return 0 for 5 times, then 5 for 10 times, and after those,
return 19 1% of the time.
1%5*return(22):
1/100th of the time, return 22, but only do it 5 times total.
I've also attached an ascii rendering of a (rough draft) man page that
goes into more detail.
Comments?
...Zach
--rmUrFcWP4LYae1gV
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="fail.man.txt"
FAIL(9) FreeBSD Kernel Developer's Manual FAIL(9)
NAME
KFAIL_POINT_CODE, KFAIL_POINT_RETURN, KFAIL_POINT_RETURN_VOID,
KFAIL_POINT_ERROR, KFAIL_POINT_GOTO, fail_point, FP_KERN -- fail points
SYNOPSIS
#include <sys/fail.h>
KFAIL_POINT_CODE(parent, name, code);
KFAIL_POINT_RETURN(parent, name);
KFAIL_POINT_RETURN_VOID(parent, name);
KFAIL_POINT_ERROR(parent, name, error_var);
KFAIL_POINT_GOTO(parent, name, error_var, label);
DESCRIPTION
Fail points are used to add code points where errors may be injected in a
user controlled fashion. Fail points provide a convenient wrapper around
user provided error injection code, providing a sysctl(9) MIB, and a
parser for that MIB that describes how the error injection code should
fire.
The base fail point macro is KFAIL_POINT_CODE() where parent is a sysctl
tree (frequently FP_KERN for kernel fail points, but various subsystems
may wish to provide their own fail point trees), and name is the name of
the MIB in that tree, and code is the error injection code. The code
argument does not require braces, but it is considered good style to use
braces for any multi-line code arguments. Inside the code argument, the
evaluation of RETURN_VALUE is derived from the return() value set in the
sysctl MIB. See SYSCTL SETTINGS below.
The remaining KFAIL_POINT_*() macros are wrappers around common error
injection paths:
KFAIL_POINT_RETURN(parent, name)
is the equivalent of KFAIL_POINT_CODE(..., return RETURN_VALUE)
KFAIL_POINT_RETURN_VOID(parent, name)
is the equivalent of KFAIL_POINT_CODE(..., return)
KFAIL_POINT_ERROR(parent, name, error_var)
is the equivalent of KFAIL_POINT_CODE(..., error_var = RETURN_VALUE)
KFAIL_POINT_GOTO(parent, name, error_var, label)
is the equivalent of KFAIL_POINT_CODE(...,
{ error_var = RETURN_VALUE; goto label;})
SYSCTL VARIABLES
The KFAIL_POINT_*() macros add sysctl MIBs where specified. Many base
kernel MIBs can be found in the fail_point.kern tree (referenced in code
by FP_KERN ).
The sysctl setting recognizes the following grammar:
<fail_point> ::
<term> ( "->" <term> )*
<term> ::
( (<float> "%") | (<integer> "*" ) )*
<type>
[ "(" <integer> ")" ]
<float> ::
<integer> [ "." <integer> ] |
"." <integer>
<type> ::
"off" | "return" | "sleep" | "panic" | "break" | "print"
The <type> argument specifies which action to take:
off Take no action (does not trigger fail point code)
return Trigger fail point code with specified argument
panic Panic
break Break into the debugger.
print Print that the fail point executed
The <float>% and <integer>* modifiers prior to <type> control when <type>
is executed. The <float>% form (e.g. "1.2%") can be used to specify a
probability that <type> will execute. The <integer>* form (e.g. "5*") can
be used to specify the number of times <type> should be executed before
this <term> is disabled. Only the last probability and the last count are
used if multiple are specified, i.e. "1.2%2%" is the same as "2%". When
both a probability and a count are specified, the probability is evalu-
ated before the count, i.e. "2%5*" means "2% of the time, but only exe-
cute it 5 times total".
The operator -> can be used to express cascading terms. If you specify
<term1>-><term2>, it means that if <term1> doesn't 'execute', <term2> is
evaluated. For the purpose of this operator, the return() and print()
operators are the only types that cascade. A return() term only cascades
if the code executes, and a print() term only cascades when passed a non-
zero argument.
EXAMPLES
sysctl fail_point.kern.foobar="2.1%return(5)"
21/1000ths of the time, execute code with RETURN_VALUE set to 5.
sysctl fail_point.kern.foobar="2%return(5)->5%return(22)"
2/100th of the time, execute code with RETURN_VALUE set to 5. If
that doesn't happen, 5% of the time execute code with
RETURN_VALUE set to 22.
sysctl fail_point.kern.foobar="5*return(5)->0.1%return(22)"
For 5 times, return 5. After that, 1/1000ths of the time, return
22.
sysctl fail_point.kern.foobar="0.1%5*return(5)"
Return 5 for 1 in 1000 executions, but only execute 5 times
total.
CAVEATS
It's easy to shoot yourself in the foot by setting fail points too
aggressively or setting too many in combination. For example, forcing
malloc() to fail consistently is potentially harmful to uptime.
The sleep() sysctl setting may not be appropriate in all situations. Cur-
rently, fail_point_eval() does not verify whether the context is appro-
priate for calling msleep().
FreeBSD 8.0 May 10, 2009 FreeBSD 8.0
--rmUrFcWP4LYae1gV--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090511162928.GD17203>