From owner-freebsd-stable@freebsd.org Sat Nov 28 16:42:59 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EF820A3B2D1 for ; Sat, 28 Nov 2015 16:42:58 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D501411FE for ; Sat, 28 Nov 2015 16:42:58 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: by mailman.ysv.freebsd.org (Postfix) id D1EC3A3B2CF; Sat, 28 Nov 2015 16:42:58 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5ADBA3B2CE; Sat, 28 Nov 2015 16:42:58 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from vms173003pub.verizon.net (vms173003pub.verizon.net [206.46.173.3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8E2D611FD; Sat, 28 Nov 2015 16:42:58 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) MIME-version: 1.0 Received: from [192.168.1.8] ([100.1.236.52]) by vms173003.mailsrvcs.net (Oracle Communications Messaging Server 7.0.5.34.0 64bit (built Oct 14 2014)) with ESMTPA id <0NYJ000H57MS5Q10@vms173003.mailsrvcs.net>; Sat, 28 Nov 2015 09:42:29 -0600 (CST) X-CMAE-Score: 0 X-CMAE-Analysis: v=2.1 cv=WpDWSorv c=1 sm=1 tr=0 a=UorMnhrCY2jH/mPejITChw==:117 a=LaogzpLLAAAA:8 a=oR5dmqMzAAAA:8 a=qtqOOiqGOCEA:10 a=r77TgQKjGQsHNAKrUKIA:9 a=weWSkvkYGQq-ntKvi4AA:9 a=QEXdDO2ut3YA:10 a=DyZeXvobOD24KcgYWNcA:9 a=G3rUhnM3Q6f7ZNui:21 a=_W_S_7VecoQA:10 To: stable@freebsd.org, freebsd-fs From: "Mikhail T." Subject: cp from NFS to ZFS hung in "fifoor" X-Enigmail-Draft-Status: N1110 Message-id: <5659CB64.5020105@aldan.algebra.com> Date: Sat, 28 Nov 2015 10:42:28 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 Content-Type: text/plain; CHARSET=US-ASCII Content-Transfer-Encoding: 7BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Nov 2015 16:42:59 -0000 I was copying /home from an old server (narawntapu) to a new one (aldan). The narawntapu:/home is mounted on aldan as /mnt with flags ro,intr. On narawntapu /home was simply located on an SSD, but on aldan I created a ZFS filesystem for it. The copying was started thus: root@aldan:/home (435) cp -Rpn /mnt/* . for a while this was proceeding at a decent clip with cp making newnfsreq-uests: load: 0.78 cmd: cp 38711 [newnfsreq] 802.84r 1.57u 140.63s 20% 10768k /mnt/mi/.kde/share/apps/kmail/dimap/.42838394.directory/sent/cur/1219621413.32392.hd8cl:2,S -> ./mi/.kde/share/apps/kmail/dimap/.42838394.directory/sent/cur/1219621413.32392.hd8cl:2,S 100% load: 1.23 cmd: cp 38711 [newnfsreq] 874.19r 1.66u 154.74s 17% 4576k /mnt/mi/.kde/share/apps/kmail/dimap/.42838394.directory/ML/cur/1219595347.32392.rMDFf:2,S -> ./mi/.kde/share/apps/kmail/dimap/.42838394.directory/ML/cur/1219595347.32392.rMDFf:2,S 100% ZFS on the destination compressing and writing stuff out and the traffic between the two ranging from 30 to 50Mb/s (according to systat), but then something happened and the cp-process is now hung: load: 0.55 cmd: cp 38711 [fifoor] 1107.67r 2.09u 194.12s 0% 3300k load: 0.50 cmd: cp 38711 [fifoor] 1112.66r 2.09u 194.12s 0% 3300k load: 0.22 cmd: cp 38711 [fifoor] 1642.37r 2.09u 194.12s 0% 3300k There is nothing in the logs on the new system, but the old one has a number of entries like: Nov 28 10:28:45 narawntapu kernel: sonewconn: pcb 0xfffff80086231930: Listen queue overflow: 8 already in queue awaiting acceptance (62 occurrences) Nov 28 10:29:45 narawntapu kernel: sonewconn: pcb 0xfffff80086231930: Listen queue overflow: 8 already in queue awaiting acceptance (50 occurrences) Nov 28 10:30:46 narawntapu kernel: sonewconn: pcb 0xfffff80086231930: Listen queue overflow: 8 already in queue awaiting acceptance (59 occurrences) Nov 28 10:31:46 narawntapu kernel: sonewconn: pcb 0xfffff80086231930: Listen queue overflow: 8 already in queue awaiting acceptance (57 occurrences) Nov 28 10:32:46 narawntapu kernel: sonewconn: pcb 0xfffff80086231930: Listen queue overflow: 8 already in queue awaiting acceptance (68 occurrences) Both systems are largely idle now. I'm not in a hurry -- is anybody interested in investigating it in situ? What is "fifoor" -- does this point to a trouble in the ZFS, the NFS-client, or the NFS-server? Both systems run FreeBSD/amd64 of recent 10.x-vintage. Thanks! -mi