Date: Wed, 26 Apr 2006 18:06:45 +0100 (BST) From: Robert Watson <rwatson@FreeBSD.org> To: Stephen Clark <Stephen.Clark@seclark.us> Cc: stable@freebsd.org Subject: Re: FreeBSD 4.9 losing mbufs!!! Message-ID: <20060426180246.O16859@fledge.watson.org> In-Reply-To: <444F75FE.3010101@seclark.us> References: <4444EE93.9050003@seclark.us> <44459286.1000008@seclark.us> <20060424141346.O44099@fledge.watson.org> <444F75FE.3010101@seclark.us>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 26 Apr 2006, Stephen Clark wrote: >> Sorry not to have caught this thread earlier; I've been on travel for the >> last few weeks. My general suggestion would be to try to narrow the code >> paths traversed to try to eliminate as much code as possible from the >> search. It sounds like you've done that pretty effectively :-). >> >> Typically, memory leaks occur in edge error cases, where the memory is not >> properly released, or ownership is unclear. My suggestion would be to add >> counters (or look at existing counters where already present) and see if >> there's an error case being triggered in about the same quantity that mbuf >> leakage is occuring. Chances are, there's an error being returned and a >> missing m_freem(). >> >> Based on your comments above, I might also pay attention to the routing >> socket path -- the rate of leak could correspond to the routing daemons >> talking to the network stack, rather than the rate of traffic. For >> example, it could be that one of the routing messages is handled improperly >> resulting in a leak. >> >> Unfortunately, tracking down memory leaks can be quite difficult, and tends >> to require a combination of dogged persistence and luck... > > Good news and bad news. > > I managed to get enough of our system running on 6.x stable to test and it > does not appear to lose mbufs. Bad news my ipsec transfer rate dropped from > 54mbits/sec to 39mbits/sec. We need to be able to handle a t3 (45mbits/sec). > Any ideas as to why this drop off in 6.x? Well, that's good about the good news, but not so great about the bad news. Are you using IPv6? If not, could you look at how FAST_IPSEC performs instead of the default IPSEC package, if you're not already doing so? The KAME IPSEC implementation is not multi-processor compatible, so when IPSEC support is compiled into the kernel, execution of the network stack and related components is limited to a single CPU (you'll see a warning about this at boot if this is the case). This is something we hope to fix in a future release, but in the mean time FAST_IPSEC offers improved performance and SMP scalability, as well as support for hardware crypto acceleration. The downside to FAST_IPSEC is that it doesn't currently support IPv6, which is also something we'd like to fix in the future. Aside from the above, there are a number of other things we can look at to decide what the source of the performance problem is. First, I'd encourage you to make sure any system debugging features, such as INVARIANTS, INVARIANT_SUPPORT, and WITNESS, are disabled. Next, I would recommend generating a high level of your load on the system, and using systat -vmstat 1 and top -S to grab some information about it in the steady state. I recommend letting both run for a couple of minutes, then grabbing the output from both of them. This will give us an idea of where CPU usage is going in the kernel, which is where I assume most of your workload ends up being handled. Thanks, Robert N M Watson
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060426180246.O16859>