Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 30 Nov 2009 20:13:49 +0100
From:      =?iso-8859-1?Q?Eirik_=D8verby?= <ltning@anduin.net>
To:        Robert N. M. Watson <rwatson@freebsd.org>
Cc:        pyunyh@gmail.com, weldon@excelsusphoto.com, freebsd-current@freebsd.org, Gavin Atkinson <gavin@freebsd.org>
Subject:   Re: FreeBSD 8.0 - network stack crashes?
Message-ID:  <E8883290-0D35-44AC-9ED9-8464C4F7FF62@anduin.net>
In-Reply-To: <EE2AA268-2309-4924-A3AD-1EC256E7BB2A@freebsd.org>
References:  <A1648B95-F36D-459D-BBC4-FFCA63FC1E4C@anduin.net> <20091129013026.GA1355@michelle.cdnetworks.com> <74BFE523-4BB3-4748-98BA-71FBD9829CD5@anduin.net> <alpine.BSF.2.00.0911291427240.80654@fledge.watson.org> <34AD565D-814A-446A-B9CA-AC16DD762E1B@anduin.net> <A0C9ED20-5536-44E2-B26B-0F1AEC2AF79C@anduin.net> <BA47FDA1-1097-4C43-AF71-51E7227795B5@FreeBSD.org> <1DFC4992-E136-4674-BC0E-A6B1DAE12AF4@anduin.net> <EE2AA268-2309-4924-A3AD-1EC256E7BB2A@freebsd.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On 30. nov. 2009, at 16.36, Robert N. M. Watson wrote:
> On 30 Nov 2009, at 08:40, Eirik =D8verby wrote:
>=20
>>>> Short follow-up: Making OpenBSD use TCP mounts (it defaults to UDP) =
seems to solve the issue.
>>>>=20
>>>> So this is a UDP-NFS-related problem, it would seem?
>>>=20
>>> Could well be. Let's try another debugging tactic -- there are two =
possible things going on here: resource leak, and resource exhaustion =
leading to deadlock. If you shut down to single user mode from =
multi-user, and let the system quiesce for a few minutes, then run =
netstat -m, what does it look like? Do vast numbers of mbufs+clusters =
get freed, or do they remain accounted for as allocated?
>>=20
>> It's been sitting in single-user mode for about 15 minutes now, no =
change in allocation.
>> I'll reboot in about 15 minutes, then try to mount from a FreeBSD box =
using UDP - if that causes the same issues, I guess it's not an OpenBSD =
specific issue but a UDP issue "in general". Next step would be to try =
to reproduce the same between two VMs on my own box, as this box needs =
to return to production soonish - if we manage to reproduce elsewhere..
>=20
> This sounds like a good plan -- especially reproducing it on a =
non-production box :-). I agree it's most likely that the OpenBSD NFS =
client simply does something a little differently than the other NFS =
clients you are dealing with, triggering an edge case in our NFS server =
code. But, to be clear, I think it's much more likely that the bug is in =
the NFS over UDP code than UDP itself, given the complexity of the NFS =
code (although a UDP bug can't be ruled out).

I meant NFS-UDP ... However I was wrong even there; Using NFS over UDP =
from FreeBSD boxes does not cause the same issue. So OpenBSD seems to be =
a special case here.

I'm no Wireshark expert (to be fair, I've seen it a few times and tried =
it once or twice, and that's so long ago it's almost no longer true), so =
I'd need some input on how to gather useful data. I assume tcpdump, =
which options? And would it be OK if I made the dump available for =
download somewhere, so you or someone else can take a look with =
whichever tools you'd like?

Thanks for your time,
/Eirik


> Robert
>=20
>>=20
>> Other ideas/suggestions?
>>=20
>> /Eirik
>>=20
>>> (If they remain allocated, they were likely leaked, since most/all =
sockets will have been closed, releasing their resources on shutdown to =
single user when all processes are killed)
>>>=20
>>> The theory of an mbuf leak in NFS isn't an unlikely theory -- the =
socket code there continues to change, and rare edge cases frequently =
lead to leaks (per my earlier e-mail). Perhaps there's a case the =
OpenBSD client is triggering that other NFS clients normally don't. If =
we think that's the case, the next step is usually to narrow down what =
causes the leak to trigger a lot (i.e., the backup starting), and then =
grab a packet trace that we can analyze with wireshark. We'll want to =
look at the types of errors being returned for RPCs and, in particular, =
if there's one that happens about the same number of times as the =
resource has leaked over the same window, look at the code and see if =
that error case is handled properly.
>>>=20
>>> If this is definitely an NFS leak bug, we should get the NFS folks =
attention by sticking "NFS mbuf leak" in the subject line and CC'ing =
rmacklem/dfr. :-)
>>>=20
>>> Robert
>>>=20
>>>=20
>>>=20
>>>=20
>>>> /Eirik
>>>>=20
>>>> On 30. nov. 2009, at 11.22, Eirik =D8verby wrote:
>>>>=20
>>>>> Hi,
>>>>>=20
>>>>> I have something that might be more interesting than any counter =
...
>>>>> It seems to me as if the problem *only* manifests itself when an =
OpenBSD box is backing up to this FreeBSD 8.0-NFS-ZFS server. All other =
boxes are FreeBSD, and I have so far today been unable to reproduce the =
problem from any of those. As soon as I interrupted the backup running =
from OpenBSD, the mbuf cluster usage stabilized.
>>>>>=20
>>>>> How's that for a mystery in the morning?
>>>>>=20
>>>>> /Eirik
>>>>>=20
>>>>> On 29. nov. 2009, at 15.29, Robert Watson wrote:
>>>>>=20
>>>>>> On Sun, 29 Nov 2009, Eirik =D8verby wrote:
>>>>>>=20
>>>>>>> I just did that (-rxcsum -txcsum -tso), but the numbers still =
keep rising. I'll wait and see if it goes down again, then reboot with =
those values to see how it behaves. But right away it doesn't look too =
good ..
>>>>>>=20
>>>>>> It would be interesting to know if any of the counters in the =
output of netstat -s grow linearly with the allocation count in netstat =
-m.  Often times leaks are associated with edge cases in the stack =
(typically because if they are in common cases the bug is detected =
really quickly!) -- usually error handling, where in some error case the =
unwinding fails to free an mbuf that it should free.  These are =
notoriously hard to track down, unfortunately, but the stats output =
(especially where delta alloc is linear to delta stat) may inform the =
situation some more.
>>>>>>=20
>>>>>> Robert N M Watson
>>>>>> Computer Laboratory
>>>>>> University of Cambridge
>>>>>=20
>>>>> _______________________________________________
>>>>> freebsd-current@freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>>>> To unsubscribe, send any mail to =
"freebsd-current-unsubscribe@freebsd.org"
>>>>>=20
>>>>=20
>>>=20
>>> _______________________________________________
>>> freebsd-current@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>> To unsubscribe, send any mail to =
"freebsd-current-unsubscribe@freebsd.org"
>>>=20
>>=20
>=20
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to =
"freebsd-current-unsubscribe@freebsd.org"
>=20




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E8883290-0D35-44AC-9ED9-8464C4F7FF62>