From owner-freebsd-current@FreeBSD.ORG Mon Nov 30 19:13:53 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20C471065694; Mon, 30 Nov 2009 19:13:53 +0000 (UTC) (envelope-from ltning@anduin.net) Received: from mail.anduin.net (mail.anduin.net [213.225.74.249]) by mx1.freebsd.org (Postfix) with ESMTP id B8AF38FC18; Mon, 30 Nov 2009 19:13:52 +0000 (UTC) Received: from [212.62.248.150] (helo=[192.168.2.110]) by mail.anduin.net with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1NFBgz-000ACv-Vz; Mon, 30 Nov 2009 20:13:50 +0100 Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=iso-8859-1 From: =?iso-8859-1?Q?Eirik_=D8verby?= In-Reply-To: Date: Mon, 30 Nov 2009 20:13:49 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20091129013026.GA1355@michelle.cdnetworks.com> <74BFE523-4BB3-4748-98BA-71FBD9829CD5@anduin.net> <34AD565D-814A-446A-B9CA-AC16DD762E1B@anduin.net> <1DFC4992-E136-4674-BC0E-A6B1DAE12AF4@anduin.net> To: Robert N. M. Watson X-Mailer: Apple Mail (2.1077) Cc: pyunyh@gmail.com, weldon@excelsusphoto.com, freebsd-current@freebsd.org, Gavin Atkinson Subject: Re: FreeBSD 8.0 - network stack crashes? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Nov 2009 19:13:53 -0000 On 30. nov. 2009, at 16.36, Robert N. M. Watson wrote: > On 30 Nov 2009, at 08:40, Eirik =D8verby wrote: >=20 >>>> Short follow-up: Making OpenBSD use TCP mounts (it defaults to UDP) = seems to solve the issue. >>>>=20 >>>> So this is a UDP-NFS-related problem, it would seem? >>>=20 >>> Could well be. Let's try another debugging tactic -- there are two = possible things going on here: resource leak, and resource exhaustion = leading to deadlock. If you shut down to single user mode from = multi-user, and let the system quiesce for a few minutes, then run = netstat -m, what does it look like? Do vast numbers of mbufs+clusters = get freed, or do they remain accounted for as allocated? >>=20 >> It's been sitting in single-user mode for about 15 minutes now, no = change in allocation. >> I'll reboot in about 15 minutes, then try to mount from a FreeBSD box = using UDP - if that causes the same issues, I guess it's not an OpenBSD = specific issue but a UDP issue "in general". Next step would be to try = to reproduce the same between two VMs on my own box, as this box needs = to return to production soonish - if we manage to reproduce elsewhere.. >=20 > This sounds like a good plan -- especially reproducing it on a = non-production box :-). I agree it's most likely that the OpenBSD NFS = client simply does something a little differently than the other NFS = clients you are dealing with, triggering an edge case in our NFS server = code. But, to be clear, I think it's much more likely that the bug is in = the NFS over UDP code than UDP itself, given the complexity of the NFS = code (although a UDP bug can't be ruled out). I meant NFS-UDP ... However I was wrong even there; Using NFS over UDP = from FreeBSD boxes does not cause the same issue. So OpenBSD seems to be = a special case here. I'm no Wireshark expert (to be fair, I've seen it a few times and tried = it once or twice, and that's so long ago it's almost no longer true), so = I'd need some input on how to gather useful data. I assume tcpdump, = which options? And would it be OK if I made the dump available for = download somewhere, so you or someone else can take a look with = whichever tools you'd like? Thanks for your time, /Eirik > Robert >=20 >>=20 >> Other ideas/suggestions? >>=20 >> /Eirik >>=20 >>> (If they remain allocated, they were likely leaked, since most/all = sockets will have been closed, releasing their resources on shutdown to = single user when all processes are killed) >>>=20 >>> The theory of an mbuf leak in NFS isn't an unlikely theory -- the = socket code there continues to change, and rare edge cases frequently = lead to leaks (per my earlier e-mail). Perhaps there's a case the = OpenBSD client is triggering that other NFS clients normally don't. If = we think that's the case, the next step is usually to narrow down what = causes the leak to trigger a lot (i.e., the backup starting), and then = grab a packet trace that we can analyze with wireshark. We'll want to = look at the types of errors being returned for RPCs and, in particular, = if there's one that happens about the same number of times as the = resource has leaked over the same window, look at the code and see if = that error case is handled properly. >>>=20 >>> If this is definitely an NFS leak bug, we should get the NFS folks = attention by sticking "NFS mbuf leak" in the subject line and CC'ing = rmacklem/dfr. :-) >>>=20 >>> Robert >>>=20 >>>=20 >>>=20 >>>=20 >>>> /Eirik >>>>=20 >>>> On 30. nov. 2009, at 11.22, Eirik =D8verby wrote: >>>>=20 >>>>> Hi, >>>>>=20 >>>>> I have something that might be more interesting than any counter = ... >>>>> It seems to me as if the problem *only* manifests itself when an = OpenBSD box is backing up to this FreeBSD 8.0-NFS-ZFS server. All other = boxes are FreeBSD, and I have so far today been unable to reproduce the = problem from any of those. As soon as I interrupted the backup running = from OpenBSD, the mbuf cluster usage stabilized. >>>>>=20 >>>>> How's that for a mystery in the morning? >>>>>=20 >>>>> /Eirik >>>>>=20 >>>>> On 29. nov. 2009, at 15.29, Robert Watson wrote: >>>>>=20 >>>>>> On Sun, 29 Nov 2009, Eirik =D8verby wrote: >>>>>>=20 >>>>>>> I just did that (-rxcsum -txcsum -tso), but the numbers still = keep rising. I'll wait and see if it goes down again, then reboot with = those values to see how it behaves. But right away it doesn't look too = good .. >>>>>>=20 >>>>>> It would be interesting to know if any of the counters in the = output of netstat -s grow linearly with the allocation count in netstat = -m. Often times leaks are associated with edge cases in the stack = (typically because if they are in common cases the bug is detected = really quickly!) -- usually error handling, where in some error case the = unwinding fails to free an mbuf that it should free. These are = notoriously hard to track down, unfortunately, but the stats output = (especially where delta alloc is linear to delta stat) may inform the = situation some more. >>>>>>=20 >>>>>> Robert N M Watson >>>>>> Computer Laboratory >>>>>> University of Cambridge >>>>>=20 >>>>> _______________________________________________ >>>>> freebsd-current@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current >>>>> To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org" >>>>>=20 >>>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-current@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-current >>> To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org" >>>=20 >>=20 >=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org" >=20