From owner-freebsd-hackers@freebsd.org Wed Dec 14 17:17:17 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6676AC778E2 for ; Wed, 14 Dec 2016 17:17:17 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4F12D6CA for ; Wed, 14 Dec 2016 17:17:17 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: by mailman.ysv.freebsd.org (Postfix) id 4E58FC778E1; Wed, 14 Dec 2016 17:17:17 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DFACC778E0 for ; Wed, 14 Dec 2016 17:17:17 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from weser.webweaving.org (weser.webweaving.org [148.251.234.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.webweaving.org", Issuer "RapidSSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE6936C9 for ; Wed, 14 Dec 2016 17:17:16 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from beeb.leiden.webweaving.org (5ED29A06.cm-7-3c.dynamic.ziggo.nl [94.210.154.6]) (authenticated bits=0) by weser.webweaving.org (8.15.2/8.15.2) with ESMTPSA id uBEHFeJf095674 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 14 Dec 2016 18:15:55 +0100 (CET) (envelope-from dirkx@webweaving.org) X-Authentication-Warning: weser.webweaving.org: Host 5ED29A06.cm-7-3c.dynamic.ziggo.nl [94.210.154.6] claimed to be beeb.leiden.webweaving.org Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: Re: ZFS - directory entry From: Dirk-Willem van Gulik In-Reply-To: Date: Wed, 14 Dec 2016 18:15:40 +0100 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Alan Somers X-Mailer: Apple Mail (2.3259) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (weser.webweaving.org [148.251.234.232]); Wed, 14 Dec 2016 18:15:56 +0100 (CET) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2016 17:17:17 -0000 > On 14 Dec 2016, at 17:14, Alan Somers wrote: >=20 > On Wed, Dec 14, 2016 at 8:27 AM, Dirk-Willem van Gulik > wrote: >> A rather odd directory entry (in /root, the home dir of root/toor) = appeared on a bog standard FreeBSD 10.2 (p18) lightly loaded machine = under ZFS during/post a backup: >>=20 >> $ ls -la /root | tail -q >> ---------- 1 root wheel 9223372036854775807 Jan 1 1970 = ?%+?kD?H???x,?5?Dh;*s!?h???jw??????\h?:????????``?13?@?????OA????????Puux?= ???>=20 >> OS and ZFS is installed with a bog standard sysinstall. =E2=80=98SMART=E2= =80=99 nor smartd have reported anything. nothing in dmesg, syslog of = boot log. Any suggestions as how to debug or get to the root of this ? >>=20 >> And in particular - what is a risk of a reboot (to get a kernel with = debug, etc) causing the issue to =E2=80=98go away=E2=80=99 - and hence = stopping the forensic ? >>=20 >> Dw. >>=20 >> sudo zpool list -v >> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT >> tank 25.2T 9.27T 16.0T - 17% 36% 1.53x = ONLINE - >> raidz3 25.2T 9.27T 16.0T - 17% 36% >> ada0p3 - - - - - - >> ada1p3 - - - - - - >> ada2p3 - - - - - - >> ada3p3 - - - - - - >> ada4p3 - - - - - - >> ada5p3 - - - - - - >> ada6p3 - - - - - - Most wonderful - I did not know about the inode/zdb magic. Thanks! > Two things to try: > 1) zpool scrub. This will reveal any corrupt metadata objects Ok - some 300 hours to go :) So am now trying figure out why it is = running at just 8M/s (prefetch_disable=3D1, vfs.zfs.scrub_delay=3D0). > 2) Maybe the filename is created in an encoding not supported by your > current terminal. Try "LANG=3Den_US.UTF-8 ls -l=E2=80=9D No cookie (and not overly likely - barebone install which is not visible = to anything =E2=80=98modern=E2=80=99 but ssh et.al.). > 3) Use zdb to examine the file. First, do "ls -li /root" to get the > object id. It's the same as the inode number. Then, assuming /root > is in the tank/root filesystem, do "zdb -ddddd tank/root ". > That might reveal some clues. A: zdb -ddddd tank/root 7426414 gives; after a 50 second pause (pre/during =E2=80=98zpool scrub=E2=80=99)= : Dataset tank/root [ZPL], ID 40, cr_txg 6, 902M, 14669 objects, = rootbp DVA[0]=3D<0:4c000:4000> DVA[1]=3D<0:4c00004c000:4000> [L0 DMU = objset] fletcher4 uncompressed LE contiguous unique double = size=3D800L/800P birth=3D225L/225P fill=3D14669 = cksum=3D9c7252c3b:ad096bfa68f:7b6298f1d2648:4235b444c02eba0 Object lvl iblk dblk dsize lsize %full type zdb: dmu_bonus_hold(7426414) failed, errno 2 So I guess I should wait for the scrub to complete. Cannot recall scrub = to be this slow. Dw.