From owner-freebsd-current@FreeBSD.ORG Mon Apr 14 13:53:40 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AA05C37B401 for ; Mon, 14 Apr 2003 13:53:40 -0700 (PDT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id ED6F743FBF for ; Mon, 14 Apr 2003 13:53:39 -0700 (PDT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.12.9/8.12.9) with SMTP id h3EKrwrE003324 for ; Mon, 14 Apr 2003 16:53:58 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Mon, 14 Apr 2003 16:53:57 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: current@FreeBSD.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: m_pkthdr.label now moved to m_tag X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Apr 2003 20:53:41 -0000 Per a discussion here last year some time, I've gone ahead and moved the MAC label from the mbuf packet header structure to optional meta-data in an m_tag. This should improve the cleanliness of the mbuf header, which is possible now we have m_tags (thanks Sam!), which of course weren't available when we originally did the MAC adaptation of the network stack. As we now use the common meta-data infrastructure, it makes the failure modes for IPsec, MAC, etc, the same when it comes to poor handling of meta-data on mbufs, so hopefully that will improve the time to correct meta-data handling problems in the stack. Because the MAC m_tags are "deep", they required tweaks in the m_tag handling code to properly copy and free the tags--Sam and I have talked a bit about possible generalizations and slippery slopes on generality. I'd like to let this stuff settle a bit before we take on generalizing :-). I do have some initial performance results from a bit of experimentation; there was a small but measurable performance improvement for GENERIC that I believe corresponds to reducing the bzero() overhead on the mbuf header. If I unconditionally allocated m_tags with MAC enabled, I saw a slight additional performance overhead against m_pkthdr.label. However, I took this opportunity to move to conditionally allocating MAC m_tags, and in general observed a substantial performance improvement for un-labeled MAC policies, eliminating that overhead (and in fact improving performance substantially) in most cases. Labeled network policies actually see a per-byte throughput improvement with large packet sizes; they do see a per-packet overhead increase, however. Two graphs, for the bored, are available at: http://www.watson.org/~robert/freebsd/label-to-tag/ They explore the impact of three labeling approaches (m_pkthdr.label, m_tag always alloc, m_tag conditional alloc) across five kernel configurations: GENERIC, GENERIC+MAC, GENERIC+MAC with three policies: mac_none (stub), mac_bsdextended (file system firewall), and mac_biba (comprehensive system integrity policy). Of these, mac_biba is the only labeled network policy; mac_none does implement per-packet stub entry points, however, so does pay a per-packet overhead compared to no policy or mac_bsdextended. All measurements were over the loopback interface on an 800mhz box (SMP turned off, I should re-do them sometime with SMP). Incidentally, this should also fix a number of WITNESS warnings when running with labeled MAC policies because it cleans up some bugs in the mapping of mbuf allocator flags to malloc flags, and also more cleanly avoids blocking allocations during certain classes of mbuf head copies. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Network Associates Laboratories ---------- Forwarded message ---------- Date: Mon, 14 Apr 2003 13:39:06 -0700 (PDT) From: Robert Watson To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org Subject: cvs commit: src/sys/kern kern_mac.c subr_mbuf.c uipc_mbuf.c uipc_mbuf2.c src/sys/net if_loop.c src/sys/sys mac.h mac_policy.h mbuf.h rwatson 2003/04/14 13:39:06 PDT FreeBSD src repository Modified files: sys/kern kern_mac.c subr_mbuf.c uipc_mbuf.c uipc_mbuf2.c sys/net if_loop.c sys/sys mac.h mac_policy.h mbuf.h Log: Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories Revision Changes Path 1.85 +118 -17 src/sys/kern/kern_mac.c 1.45 +9 -17 src/sys/kern/subr_mbuf.c 1.115 +8 -15 src/sys/kern/uipc_mbuf.c 1.19 +22 -1 src/sys/kern/uipc_mbuf2.c 1.82 +0 -8 src/sys/net/if_loop.c 1.39 +5 -2 src/sys/sys/mac.h 1.38 +3 -0 src/sys/sys/mac_policy.h 1.120 +1 -2 src/sys/sys/mbuf.h