From owner-freebsd-current@FreeBSD.ORG Sun Nov 29 07:45:56 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0E6B106568B; Sun, 29 Nov 2009 07:45:56 +0000 (UTC) (envelope-from ltning@anduin.net) Received: from mail.anduin.net (mail.anduin.net [213.225.74.249]) by mx1.freebsd.org (Postfix) with ESMTP id 6A3878FC1A; Sun, 29 Nov 2009 07:45:55 +0000 (UTC) Received: from [212.62.248.150] (helo=[192.168.2.110]) by mail.anduin.net with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1NEeTg-0007Gk-Cz; Sun, 29 Nov 2009 08:45:53 +0100 Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: =?iso-8859-1?Q?Eirik_=D8verby?= In-Reply-To: <20091129013026.GA1355@michelle.cdnetworks.com> Date: Sun, 29 Nov 2009 08:45:54 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <74BFE523-4BB3-4748-98BA-71FBD9829CD5@anduin.net> References: <20091129013026.GA1355@michelle.cdnetworks.com> To: pyunyh@gmail.com, freebsd-current@freebsd.org X-Mailer: Apple Mail (2.1077) Cc: weldon@excelsusphoto.com, Gavin Atkinson Subject: Re: FreeBSD 8.0 - network stack crashes? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Nov 2009 07:45:57 -0000 On 29. nov. 2009, at 02.30, Pyun YongHyeon wrote: > On Sat, Nov 28, 2009 at 08:46:12AM +0100, Eirik ??verby wrote: >> Hi, >>=20 >> Gavin Atkinson wrote: >>> On Tue, 2009-11-03 at 08:32 -0500, Weldon S Godfrey 3 wrote: >>>>=20 >>>> If memory serves me right, sometime around Yesterday, Gavin = Atkinson told me: >>>>=20 >>>> Gavin, thank you A LOT for helping us with this, I have answered as = much=20 >>>> as I can from the most recent crash below. We did hit max mbufs. = It is=20 >>>> at 25Kclusters, which is the default. I have upped it to 32K = because a=20 >>>> rather old article mentioned that as the top end and I need to get = into=20 >>>> work so I am not trying to do this with a remote console to go = higher. I=20 >>>> have already set it to reboot next with 64K clusters. I already = have kmem=20 >>>> maxed to what is bootable (or at least at one time) in 8.0, 4GB, = how high=20 >>>> can I safely go? This is a NFS server running ZFS with sustained 5 = min=20 >>>> averages of 120-200Mb/s running as a store for a mail system. >>>>=20 >>>>> Some things that would be useful: >>>>>=20 >>>>> - Does "arp -da" fix things? >>>>=20 >>>> no, it hangs like ssh, route add, etc >>>>=20 >>>>> - What's the output of "netstat -m" while the networking is = broken? >>>> Tue Nov 3 07:02:11 CST 2009 >>>> 36971/2033/39004 mbufs in use (current/cache/total) >>>> 24869/731/25600/25600 mbuf clusters in use = (current/cache/total/max) >>>> 24314/731 mbuf+clusters out of packet secondary zone in use=20 >>>> (current/cache) >>>> 0/35/35/12800 4k (page size) jumbo clusters in use=20 >>>> (current/cache/total/max) >>>> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) >>>> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) >>>> 58980K/2110K/61091K bytes allocated to network = (current/cache/total) >>>> 0/201276/90662 requests for mbufs denied = (mbufs/clusters/mbuf+clusters) >>>> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >>>> 0/0/0 sfbufs in use (current/peak/max) >>>> 0 requests for sfbufs denied >>>> 0 requests for sfbufs delayed >>>> 0 requests for I/O initiated by sendfile >>>> 0 calls to protocol drain routines >>>=20 >>> OK, at least we've figured out what is going wrong then. As a >>> workaround to get the machine to stay up longer, you should be able = to >>> set kern.ipc.nmbclusters=3D256000 in /boot/loader.conf -but = hopefully we >>> can resolve this soon. >>=20 >> I'll chip in with a report of exactly the same situation, and I'm on = 8.0-RELEASE. >> We've been struggling with this for some time, and latest yesterday = the box was rebooted, and already last night it wedged again. We're at a = whopping=20 >> kern.ipc.nmbclusters: 524288 >> and I've just doubled it once more, which means we're allocating 2GB = to networking.. >>=20 >> Much like the original poster, we're seeing this on a amd64 storage = server with a large ZFS array shared through NFS, and network interfaces = are two em(4) combined in a lagg(4) interface (lacp). Using either of = the two em interfaces without lagg shows the same problem, just lower = performance.. >>=20 >>=20 >>> Firstly, what kernel was the above output from? And what network = card >>> are you using? In your initial post you mentioned testing both = bce(4) >>> and em(4) cards, be aware that em(4) had an issue that would cause >>> exactly this issue, which was fixed with a commit on September 11th >>> (r197093). Make sure your kernel is from after that date if you are >>> using em(4). I guess it is also possible that bce(4) has the same >>> issue, I'm not aware of any fixes to it recently. >>=20 >> We're on GENERIC . >>=20 >>=20 >>> So, from here, I think the best thing would be to just use the em(4) = NIC >>> and an up-to-date kernel, and see if you can reproduce the issue. >>=20 >> em(4) and 8.0-RELEASE still shows this problem. >>=20 >>=20 >>> How important is this machine? If em(4) works, are you able to help >>> debug the issues with the bce(4) driver? >>=20 >> We have no bce(4), but we have the problem on em(4) so can help debug = there. The server is important, but making it stable is more important.. = See below the sig for some debug info. >>=20 >=20 > How about disabling TSO/Tx checksum offloading of em(4)? > Last time I checked the driver, em(4) seems to assume it can access > IP/TCP header in mbuf chains without computing required header size. Hi, I just did that (-rxcsum -txcsum -tso), but the numbers still keep = rising. I'll wait and see if it goes down again, then reboot with those = values to see how it behaves. But right away it doesn't look too good .. /Eirik=