Date: Wed, 26 Apr 2006 15:40:31 -0400 From: Stephen Clark <Stephen.Clark@seclark.us> To: Robert Watson <rwatson@FreeBSD.org> Cc: stable@freebsd.org Subject: Re: FreeBSD 4.9 losing mbufs!!! Message-ID: <444FCCAF.8000006@seclark.us> In-Reply-To: <20060426180246.O16859@fledge.watson.org> References: <4444EE93.9050003@seclark.us> <44459286.1000008@seclark.us> <20060424141346.O44099@fledge.watson.org> <444F75FE.3010101@seclark.us> <20060426180246.O16859@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Robert Watson wrote: >On Wed, 26 Apr 2006, Stephen Clark wrote: > > > >>>Sorry not to have caught this thread earlier; I've been on travel for the >>>last few weeks. My general suggestion would be to try to narrow the code >>>paths traversed to try to eliminate as much code as possible from the >>>search. It sounds like you've done that pretty effectively :-). >>> >>>Typically, memory leaks occur in edge error cases, where the memory is not >>>properly released, or ownership is unclear. My suggestion would be to add >>>counters (or look at existing counters where already present) and see if >>>there's an error case being triggered in about the same quantity that mbuf >>>leakage is occuring. Chances are, there's an error being returned and a >>>missing m_freem(). >>> >>>Based on your comments above, I might also pay attention to the routing >>>socket path -- the rate of leak could correspond to the routing daemons >>>talking to the network stack, rather than the rate of traffic. For >>>example, it could be that one of the routing messages is handled improperly >>>resulting in a leak. >>> >>>Unfortunately, tracking down memory leaks can be quite difficult, and tends >>>to require a combination of dogged persistence and luck... >>> >>> >>Good news and bad news. >> >>I managed to get enough of our system running on 6.x stable to test and it >>does not appear to lose mbufs. Bad news my ipsec transfer rate dropped from >>54mbits/sec to 39mbits/sec. We need to be able to handle a t3 (45mbits/sec). >>Any ideas as to why this drop off in 6.x? >> >> > >Well, that's good about the good news, but not so great about the bad news. >Are you using IPv6? If not, could you look at how FAST_IPSEC performs instead >of the default IPSEC package, if you're not already doing so? The KAME IPSEC >implementation is not multi-processor compatible, so when IPSEC support is >compiled into the kernel, execution of the network stack and related >components is limited to a single CPU (you'll see a warning about this at boot >if this is the case). This is something we hope to fix in a future release, >but in the mean time FAST_IPSEC offers improved performance and SMP >scalability, as well as support for hardware crypto acceleration. The >downside to FAST_IPSEC is that it doesn't currently support IPv6, which is >also something we'd like to fix in the future. > >Aside from the above, there are a number of other things we can look at to >decide what the source of the performance problem is. First, I'd encourage >you to make sure any system debugging features, such as INVARIANTS, >INVARIANT_SUPPORT, and WITNESS, are disabled. Next, I would recommend >generating a high level of your load on the system, and using systat -vmstat 1 >and top -S to grab some information about it in the steady state. I recommend >letting both run for a couple of minutes, then grabbing the output from both >of them. This will give us an idea of where CPU usage is going in the kernel, >which is where I assume most of your workload ends up being handled. > >Thanks, > >Robert N M Watson >_______________________________________________ >freebsd-stable@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-stable >To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > Thanks Robert, I'll give that a try. Regards, Steve -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?444FCCAF.8000006>