Date: Mon, 21 Aug 2017 18:14:53 +0100 From: Martin Simmons <martin@lispworks.com> To: Chris Ross <cross+freebsd@distal.com> Cc: freebsd-fs@freebsd.org Subject: Re: zfs recv went idle after 20+ hours Message-ID: <201708211714.v7LHErBa010629@higson.cam.lispworks.com> In-Reply-To: <4BCBA5DE-6AC7-4B0A-BF33-CFFE5A54B2DF@distal.com> (message from Chris Ross on Fri, 18 Aug 2017 23:30:49 -0400) References: <4BCBA5DE-6AC7-4B0A-BF33-CFFE5A54B2DF@distal.com>
next in thread | previous in thread | raw e-mail | index | archive | help
>>>>> On Fri, 18 Aug 2017 23:30:49 -0400, Chris Ross said: > > From earlier posts on this list, many know I was dealing with an issue on a zfs pool. I eventually got it running in an off way, and backed it up (via zfs send to a file in UFS). I then destroyed and recreated the pool as I wanted it. For the last 23 hours or so, I’ve been running a “zfs recv” from the large send output (~5.2T). It was cooking along, more slowly than I’d expected, but moving. > > I looked in on it a short while ago, and the pv I’m using on the input file was reporting 0.0 B/s, 77% complete. Looking at the iostat I had running, I could see the raidz disks still seeing activity, but less than a minute later that stopped. Now, all disks on the system seem idle. The “zfs recv” process is idle according to ps (STAT I+), and a ktrace on the zfs process shows no activity either. The window in which I was running the commands look as follows: > > cross@hyrule[~](511): zpool list > NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > tank 18.8T 949K 18.8T - 0% 0% 1.00x ONLINE - > zroot 50.5G 34.7G 15.8G - 55% 68% 1.00x ONLINE - > cross@hyrule[~](512): pv /mnt/tank@20170815| sudo zfs recv -Fvu tank > receiving full stream of tank@20170815 into tank@20170815 > received 46.3KB stream in 1 seconds (46.3KB/sec) > receiving full stream of tank/deluge@20170815 into tank/deluge@20170815 > received 2.87TB stream in 45452 seconds (66.3MB/sec) ] 55% ETA 10:18:14 > receiving full stream of tank/timecapsule@20170815 into tank/timecapsule@20170815 > 3.99TiB 22:54:18 [0.00 B/s] [========================> ] 76% ETA 7:04:49 > > The time-of-day and ETA value keep increasing, but other than that, it’s just sitting. zfs list shows: > > hyrule# zfs list -t all | grep tank > tank 3.89T 8.28T 73.3K none > tank@20170815 0 - 73.3K - > tank/deluge 2.86T 8.28T 2.86T /data/deluge > tank/deluge@20170815 0 - 2.86T - > tank/timecapsule 1024G 8.28T 1024G /data/timecapsule > > I’m going to leave it this way for a while, but it’s already been about 10 minutes since it started, so I don’t expect that it will free itself. Does anyone have a suggestion, or something I should look at? Did I do something wrong? Check that top and/or ps show nothing else using CPU. You could use procstat -kk to see what each unexpectedly idle process is waiting for in the kernel. __Martin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201708211714.v7LHErBa010629>