From owner-freebsd-fs@freebsd.org Mon Aug 21 17:26:23 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42C07DE9641 for ; Mon, 21 Aug 2017 17:26:23 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [46.17.166.21]) by mx1.freebsd.org (Postfix) with ESMTP id C70087C8E0 for ; Mon, 21 Aug 2017 17:26:21 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (higson.cam.lispworks.com [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.15.2/8.15.2) with ESMTP id v7LHErAb046174; Mon, 21 Aug 2017 18:14:53 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id v7LHEr0F010633; Mon, 21 Aug 2017 18:14:53 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id v7LHErBa010629; Mon, 21 Aug 2017 18:14:53 +0100 Date: Mon, 21 Aug 2017 18:14:53 +0100 Message-Id: <201708211714.v7LHErBa010629@higson.cam.lispworks.com> From: Martin Simmons To: Chris Ross CC: freebsd-fs@freebsd.org In-reply-to: <4BCBA5DE-6AC7-4B0A-BF33-CFFE5A54B2DF@distal.com> (message from Chris Ross on Fri, 18 Aug 2017 23:30:49 -0400) Subject: Re: zfs recv went idle after 20+ hours References: <4BCBA5DE-6AC7-4B0A-BF33-CFFE5A54B2DF@distal.com> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Aug 2017 17:26:23 -0000 >>>>> On Fri, 18 Aug 2017 23:30:49 -0400, Chris Ross said: > > From earlier posts on this list, many know I was dealing with an issue on a zfs pool. I eventually got it running in an off way, and backed it up (via zfs send to a file in UFS). I then destroyed and recreated the pool as I wanted it. For the last 23 hours or so, I’ve been running a “zfs recv” from the large send output (~5.2T). It was cooking along, more slowly than I’d expected, but moving. > > I looked in on it a short while ago, and the pv I’m using on the input file was reporting 0.0 B/s, 77% complete. Looking at the iostat I had running, I could see the raidz disks still seeing activity, but less than a minute later that stopped. Now, all disks on the system seem idle. The “zfs recv” process is idle according to ps (STAT I+), and a ktrace on the zfs process shows no activity either. The window in which I was running the commands look as follows: > > cross@hyrule[~](511): zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > tank 18.8T 949K 18.8T - 0% 0% 1.00x ONLINE - > zroot 50.5G 34.7G 15.8G - 55% 68% 1.00x ONLINE - > cross@hyrule[~](512): pv /mnt/tank@20170815| sudo zfs recv -Fvu tank > receiving full stream of tank@20170815 into tank@20170815 > received 46.3KB stream in 1 seconds (46.3KB/sec) > receiving full stream of tank/deluge@20170815 into tank/deluge@20170815 > received 2.87TB stream in 45452 seconds (66.3MB/sec) ] 55% ETA 10:18:14 > receiving full stream of tank/timecapsule@20170815 into tank/timecapsule@20170815 > 3.99TiB 22:54:18 [0.00 B/s] [========================> ] 76% ETA 7:04:49 > > The time-of-day and ETA value keep increasing, but other than that, it’s just sitting. zfs list shows: > > hyrule# zfs list -t all | grep tank > tank 3.89T 8.28T 73.3K none > tank@20170815 0 - 73.3K - > tank/deluge 2.86T 8.28T 2.86T /data/deluge > tank/deluge@20170815 0 - 2.86T - > tank/timecapsule 1024G 8.28T 1024G /data/timecapsule > > I’m going to leave it this way for a while, but it’s already been about 10 minutes since it started, so I don’t expect that it will free itself. Does anyone have a suggestion, or something I should look at? Did I do something wrong? Check that top and/or ps show nothing else using CPU. You could use procstat -kk to see what each unexpectedly idle process is waiting for in the kernel. __Martin