From owner-freebsd-fs@FreeBSD.ORG Sat Nov 10 22:32:59 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 74C8D56A for ; Sat, 10 Nov 2012 22:32:59 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 0373A8FC08 for ; Sat, 10 Nov 2012 22:32:58 +0000 (UTC) Received: from server.rulingia.com (c220-239-241-202.belrs5.nsw.optusnet.com.au [220.239.241.202]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id qAAMWupG059984 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Sun, 11 Nov 2012 09:32:56 +1100 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id qAAMWotV096126 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 11 Nov 2012 09:32:50 +1100 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id qAAMWnrF096121 for freebsd-fs@freebsd.org; Sun, 11 Nov 2012 09:32:49 +1100 (EST) (envelope-from peter) Date: Sun, 11 Nov 2012 09:32:49 +1100 From: Peter Jeremy To: freebsd-fs@freebsd.org Subject: zfs diff deadlock Message-ID: <20121110223249.GB506@server.rulingia.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0ntfKIWw70PvrIHh" Content-Disposition: inline X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2012 22:32:59 -0000 --0ntfKIWw70PvrIHh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I recently decided to do a "zfs diff" between two snapshots to try and identify why there was so much "USED" space in the snapshot. The diff ran for a while (though with very little IO) but has now wedged unkillably. There's nothing on the console or in any logs, the pool reports no problems and there are no other visible FS issues. Any ideas on tracking this down? root@server:/root # ps -lp 53146 UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 53146 53092 0 76 0 18096 948 ffffff0149cc6100 D+ 33 4:18.= 76 zfs diff -F tank/beckett/home@20121108 tank/beckett/home@20121109 root@server:/root # procstat -kk 53146 PID TID COMM TDNAME KSTACK = =20 53146 100490 zfs - mi_switch+0x18e sleepq_wait+= 0x44 _sx_slock_hard+0x22c _sx_slock+0x7a zap_get_leaf_byblk+0xbd zap_deref_= leaf+0x68 fzap_cursor_retrieve+0xe7 zap_cursor_retrieve+0x155 zap_value_sea= rch+0x7f zfs_obj_to_path_impl+0x292 zfs_obj_to_stats+0x175 zfs_ioc_obj_to_s= tats+0x7e zfsdev_ioctl+0xe6 devfs_ioctl_f+0x7a kern_ioctl+0xb6 ioctl+0xf0 a= md64_syscall+0x109 Xfast_syscall+0xfc=20 53146 100643 zfs initial thread mi_switch+0x18e sleepq_catch= _signals+0x247 sleepq_wait_sig+0x16 _sleep+0x2be pipe_write+0x6c8 write_rec= ord+0x79 diff_cb+0x166 traverse_visitbp+0x21c traverse_visitbp+0x316 traver= se_visitbp+0x316 traverse_visitbp+0x316 traverse_visitbp+0x316 traverse_vis= itbp+0x316 traverse_visitbp+0x316 traverse_dnode+0x7c traverse_visitbp+0x48= c traverse_impl+0x188 traverse_dataset+0x32=20 root@server:/root # zpool status tank pool: tank state: ONLINE scan: resilvered 1.58G in 0h10m with 0 errors on Tue Oct 30 06:55:29 2012 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 ada2p5 ONLINE 0 0 0 ada3p5 ONLINE 0 0 0 ada1p5 ONLINE 0 0 0 ada5p5 ONLINE 0 0 0 ada0p5 ONLINE 0 0 0 ada4p5 ONLINE 0 0 0 logs ada6p3 ONLINE 0 0 0 cache ada6p4 ONLINE 0 0 0 errors: No known data errors The systems is running a 4-month old 8-stable (r237444) --=20 Peter Jeremy --0ntfKIWw70PvrIHh Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlCe1hEACgkQ/opHv/APuIc8cgCeM46iryYjF5fu86dHSlptsm9p 47UAoMC43Ujh6w51gsXNk5mKF367zFUN =dEyt -----END PGP SIGNATURE----- --0ntfKIWw70PvrIHh--