Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 28 Nov 2015 16:53:37 -0500 (EST)
From:      Rick Macklem <rmacklem@uoguelph.ca>
To:        "Mikhail T." <mi+thun@aldan.algebra.com>
Cc:        stable@freebsd.org, freebsd-fs <freebsd-fs@freebsd.org>
Subject:   Re: cp from NFS to ZFS hung in "fifoor"
Message-ID:  <1797738664.110049243.1448747617558.JavaMail.zimbra@uoguelph.ca>
In-Reply-To: <5659CB64.5020105@aldan.algebra.com>
References:  <5659CB64.5020105@aldan.algebra.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Mikhail T. wrote:
> I was copying /home from an old server (narawntapu) to a new one
> (aldan). The narawntapu:/home is mounted on aldan as /mnt with flags
> ro,intr. On narawntapu /home was simply located on an SSD, but on aldan
> I created a ZFS filesystem for it.
> 
> The copying was started thus:
> 
>     root@aldan:/home (435) cp -Rpn /mnt/* .
> 
> for a while this was proceeding at a decent clip with cp making
> newnfsreq-uests:
> 
>     load: 0.78  cmd: cp 38711 [newnfsreq] 802.84r 1.57u 140.63s 20% 10768k
>     /mnt/mi/.kde/share/apps/kmail/dimap/.42838394.directory/sent/cur/1219621413.32392.hd8cl:2,S
>     ->
>     ./mi/.kde/share/apps/kmail/dimap/.42838394.directory/sent/cur/1219621413.32392.hd8cl:2,S
>     100%
>     load: 1.23  cmd: cp 38711 [newnfsreq] 874.19r 1.66u 154.74s 17% 4576k
>     /mnt/mi/.kde/share/apps/kmail/dimap/.42838394.directory/ML/cur/1219595347.32392.rMDFf:2,S
>     ->
>     ./mi/.kde/share/apps/kmail/dimap/.42838394.directory/ML/cur/1219595347.32392.rMDFf:2,S
>     100%
> 
> ZFS on the destination compressing and writing stuff out and the traffic
> between the two ranging from 30 to 50Mb/s (according to systat), but
> then something happened and the cp-process is now hung:
> 
>     load: 0.55  cmd: cp 38711 [fifoor] 1107.67r 2.09u 194.12s 0% 3300k
>     load: 0.50  cmd: cp 38711 [fifoor] 1112.66r 2.09u 194.12s 0% 3300k
>     load: 0.22  cmd: cp 38711 [fifoor] 1642.37r 2.09u 194.12s 0% 3300k
> 
Doing `ps axHl` will show you what the ``cp`` process is stuck on (WCHAN).
If it is down inside ZFS, then I suspect it is ZFS resource related. If
it is stuck somewhere in NFS or the kernel RPC, then I`d suspect a net
driver issue:
- The number 1 issue for net drivers vs NFS is TSO, so disabling TSO is
  the first thing to try (if the processes aren`t stuck inside zfs).
  (In the machine that is sending data, since it is a transmit segment limit
   problem. If I understood what you were doing, that would be the NFS server,
   but I`d disable it on both server and client.)
  - If that doesn`t fix it, try rsize=32768,wsize=32768 mount options for the
    NFS mount.

These TSO issues are slowly getting resolved, but some drivers may still be
broken, especially if you aren`t running head. (For example, only a very recent
em(4) driver is fixed.)

You can also do things like `netstat -m` to look for mbuf cluster exhaustion
and look at the stats for your net driver (usually a sysctl).

Good luck with it, rick

> There is nothing in the logs on the new system, but the old one has a
> number of entries like:
> 
>     Nov 28 10:28:45 narawntapu kernel: sonewconn: pcb
>     0xfffff80086231930: Listen queue overflow: 8 already in queue
>     awaiting acceptance (62 occurrences)
>     Nov 28 10:29:45 narawntapu kernel: sonewconn: pcb
>     0xfffff80086231930: Listen queue overflow: 8 already in queue
>     awaiting acceptance (50 occurrences)
>     Nov 28 10:30:46 narawntapu kernel: sonewconn: pcb
>     0xfffff80086231930: Listen queue overflow: 8 already in queue
>     awaiting acceptance (59 occurrences)
>     Nov 28 10:31:46 narawntapu kernel: sonewconn: pcb
>     0xfffff80086231930: Listen queue overflow: 8 already in queue
>     awaiting acceptance (57 occurrences)
>     Nov 28 10:32:46 narawntapu kernel: sonewconn: pcb
>     0xfffff80086231930: Listen queue overflow: 8 already in queue
>     awaiting acceptance (68 occurrences)
> 
> Both systems are largely idle now. I'm not in a hurry -- is anybody
> interested in investigating it in situ? What is "fifoor" -- does this
> point to a trouble in the ZFS, the NFS-client, or the NFS-server? Both
> systems run FreeBSD/amd64 of recent 10.x-vintage.
> 
> Thanks!
> 
>     -mi
> 
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
> 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1797738664.110049243.1448747617558.JavaMail.zimbra>