Date: Sat, 18 Jan 2014 08:13:22 -0800 From: Adrian Chadd <adrian@freebsd.org> To: Daniel Braniss <danny@cs.huji.ac.il> Cc: Rick Macklem <rmacklem@uoguelph.ca>, FreeBSD stable <freebsd-stable@freebsd.org> Subject: Re: on 9.2-stable nfs/zfs and 10g hang Message-ID: <CAJ-VmomgV=W6O2fMXiaJnLopMBDV-=N6XDF17mSWe2Tok96Jkg@mail.gmail.com> In-Reply-To: <2C287272-7B57-4AAD-B22F-6A65D9F8677B@cs.huji.ac.il> References: <588564685.11730322.1389970076386.JavaMail.root@uoguelph.ca> <2C287272-7B57-4AAD-B22F-6A65D9F8677B@cs.huji.ac.il>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi!
Please try reducing the size down to 32k but leave TSO enabled.
It's 9.2, so there may be some bugfixes that haven't been backported
from 10 or -HEAD. Would you be able to try a -HEAD snapshot here?
What's the NFS server and hosts? I saw the core.txt.16 that says
"ix0/ix1" so I can glean the basic chipset family but which NIC in
particular is it? What would people need to try and reproduce it?
-a
On 18 January 2014 03:24, Daniel Braniss <danny@cs.huji.ac.il> wrote:
>
> On Jan 17, 2014, at 4:47 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
>
>> Daniel Braniss wrote:
>>> hi all,
>>>
>>> All was going ok till I decided to connect this host via a 10g nic
>>> and very soon it started
>>> to hang. Running multiple make buildworlds from other hosts connected
>>> via 10g and
>>> using both src and obj on the server via tcp/nfs did ok. but running
>>> find =85 -exec md5 {} + (the find finds over 6M files)
>>> from another host (at 10g) will hang it very quickly.
>>>
>>> If I wait a while (can=92t be more specific) it sometimes recovers -
>>> but my users are not very
>>> patient :-)
>>>
>> This suggests that an RPC request/reply gets dropped in a way that TCP
>> doesn't recover. Eventually (after up to about 15min, I think?) the TCP
>> connection will be shut down and a new TCP connection started, with a
>> retry of outstanding RPCs.
>>
>>> I will soon try the same experiment using the old 1G nic, but in the
>>> meantime, if someone
>>> could shed some light would be very helpful
>>>
>>> I=92m attaching core.txt, but if it doesn=92t make it, it=92s also
>>> available at:
>>> ftp://ftp.cs.huji.ac.il/users/danny/freebsd/core.txt.16
>>>
>> You might try disabling TSO on the net interface. There are been issues
>> with TSO for segments around 64K in the past (or use rsize=3D32768,wsize=
=3D32768
>> options on the client mount, to avoid RPCs over about 32K in size).
>>
> BINGO! disabling tso did it. I=92ll try reducing the packet size later.
> some numbers:
> there where some 7*10^6 files
> doing it locally (the find + md5) took about 3hs,
> via nfs at 1g took 11 hrs.
> at 10g it took 4 hrs.
>
> thanks!
> danny
>
>
>> Beyond that, capturing a packet trace for the case that hangs easily and
>> looking at what goes on near the end of it in wireshark might give you
>> a hint about what is going on.
>>
>> rick
>>
>>> thanks,
>>> danny
>>> _______________________________________________
>>> freebsd-stable@freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>> To unsubscribe, send any mail to
>>> "freebsd-stable-unsubscribe@freebsd.org"
>>>
>
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmomgV=W6O2fMXiaJnLopMBDV-=N6XDF17mSWe2Tok96Jkg>