Date: Fri, 22 Mar 2002 22:45:56 -0500 (EST) From: Robert Watson <rwatson@FreeBSD.org> To: John Baldwin <jhb@FreeBSD.org> Cc: smp@FreeBSD.org Subject: Re: Giant instrumentation Message-ID: <Pine.NEB.3.96L.1020322213642.47668A-100000@fledge.watson.org> In-Reply-To: <XFMail.20020321172542.jhb@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 21 Mar 2002, John Baldwin wrote: > Kelly Yancey implemented most of the syscall Giant instrumentation stuff > I outlined in an earlier e-mail. I have since mostly finished it and > changed a few things. Most of the changes were design changes so that > the instrumentation stuff was as separated from the rest of the kernel > (and thus easily condionitionally compiled out with minimal impact) as > possible. Also, it now provides better support for handling other ABIs. > It's not complete yet (it only has instrumentation for the native ABI at > the moment) but I wanted to post this patch for feedback. I have tested > it some and it is being tested some more on alpha and x86 at the moment. > The patch can be found at: > > http://www.FreeBSD.org/~jhb/patches/giantvars.patch Some general thoughts-- (1) One thing I particularly like about placing the instrumentation outside of the system call code itself, since it means avoiding restructuring the system call code to handle consitional lock release at every return point. For things like proc-locking, it may not substantially change the code flow, but for some of the more complicated VFS calls with many return points, that may be useful. Adding more components to an "unwind" tends to complicate system call code a great deal. One important thing we'd like to do for these "complicated" calls is avoid changing the code flow when we add instrumentation, since that will potentially introduce bugs associated with control flow changes. Also, when we later remove the instrumentation, we end up potentially wanting to restore the old structure, introducing new bugs or changing code flow. Where possible, it would be nice if the pre-instrumention code flow, instrumented code flow, and post-instrumentation flow were visually (and actually) the same. That is, adding instrumentation should be non-intrusive, which this certainly accomplishes. (2) Question: how easily does this framework extend to trap handling, such as VM traps where we might similarly want to be able to instrument entry to traps that might or might not want Giant? The current stuff is pretty system-call specific in my reading, at least, in where it gets the information. Similarly, could this framework be adapted for use with interrupt/driver entry points? (3) For the td_ucred changes, it seems to me that introducing instrumentation may be gratuitous, regardless of the instrumentation approach taken. If anything, the *setting* of td_ucred is something that should be instrumented as a use of the proc lock. One hesitates to say it, but "it should just work". We've (in the alfred/siego sense) actually already successfully pushed down locks for some relatively major subsystems with very little problem. If anything, experience so far has suggested that the most important thing is having effective locking assertions is the most powerful debugging tool, be it via various mtx_assert(), etc, things, or the ability to use Witness to enforce more general lock order and locking relationship constraints. (4) One down-side to this approach is that it's not possible to push the instrumentation point down below the system call level. For example, suppose we're confident that the first level code in the various generic file descriptor-based calls is correct, but we want to continue to instrument Giant over the VFS but not over pipes. In such a scenario, you could imagine something like the following: system call (giant instrumentation: proc, filedesc) struct file indirection (giant instrumentation: filedesc, file) file_ops pipe (giant instrumentation: file, pipes) pipe implementation (giant instrumentation: pipes) If we had a mechanism permitting us to identify that file and pipe locking were suspect, then Giant would be grabbed when we hit the struct file indirection code, and it would presumably be retained until we walked back up the stack to there. That said, I'm not convinced this sort of spl-like mechanism is actually desirable. If the goal of instrumentation is to provide an "escape hatch" during the integration of a particular phase of locking, simply locking at the system call level is sufficient to protect the average developer from suffering. Anything else is icing. (5) Another advantage of this model is that it centralizes the instrumentation information. I can easily look at the list of instrumented entry points in a single place, and walk through to drop in a new instrumentation type where necessary. I can walk the list of socket calls and mark them as IPv4-instrumented, if I'm pushing down Giant on the IPv4 stack without having to walk a large number of files. Likewise, I can easily strip an instrumentation type from the table in one place. This does, however, point again to the desirability of having similar tables for drivers, interrupt bits, etc. Likewise, if I've added MAC hooks for VFS entry points, I can easily add a new MAC Giant instrumentation for all the relevant system calls. Paying the development cost up-front for a centralized instrumentation mechanism reduces the later cost of changing the instrumentation as we move forward with fine-grained locking, and easily allows developers to modify instrumentation in a local tree without generating diffs involving hundreds of files, especially diffs that may be mixed in with the locking changes themselves. Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1020322213642.47668A-100000>