Date: Wed, 14 Dec 2016 18:15:40 +0100 From: Dirk-Willem van Gulik <dirkx@webweaving.org> To: Alan Somers <asomers@freebsd.org> Cc: FreeBSD Hackers <hackers@freebsd.org> Subject: Re: ZFS - directory entry Message-ID: <C3EE2E11-92FA-4CA0-A193-F6CECB3E232F@webweaving.org> In-Reply-To: <CAOtMX2i5_tL=jXcq2T5F2CkiQUVK9DoYofUsktNGXOHkmLaYRg@mail.gmail.com> References: <BEAC6EE9-C50F-4FB9-B215-D5A6691E2DD9@webweaving.org> <CAOtMX2i5_tL=jXcq2T5F2CkiQUVK9DoYofUsktNGXOHkmLaYRg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 14 Dec 2016, at 17:14, Alan Somers <asomers@freebsd.org> wrote: > > On Wed, Dec 14, 2016 at 8:27 AM, Dirk-Willem van Gulik > <dirkx@webweaving.org> wrote: >> A rather odd directory entry (in /root, the home dir of root/toor) appeared on a bog standard FreeBSD 10.2 (p18) lightly loaded machine under ZFS during/post a backup: >> >> $ ls -la /root | tail -q >> ---------- 1 root wheel 9223372036854775807 Jan 1 1970 ?%+?kD?H???x,?5?Dh;*s!?h???jw??????\h?:????????``?13?@?????OA????????Puux????<T]???R??Qv?g???]??%?R? >> >> OS and ZFS is installed with a bog standard sysinstall. ‘SMART’ nor smartd have reported anything. nothing in dmesg, syslog of boot log. Any suggestions as how to debug or get to the root of this ? >> >> And in particular - what is a risk of a reboot (to get a kernel with debug, etc) causing the issue to ‘go away’ - and hence stopping the forensic ? >> >> Dw. >> >> sudo zpool list -v >> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT >> tank 25.2T 9.27T 16.0T - 17% 36% 1.53x ONLINE - >> raidz3 25.2T 9.27T 16.0T - 17% 36% >> ada0p3 - - - - - - >> ada1p3 - - - - - - >> ada2p3 - - - - - - >> ada3p3 - - - - - - >> ada4p3 - - - - - - >> ada5p3 - - - - - - >> ada6p3 - - - - - - Most wonderful - I did not know about the inode/zdb magic. Thanks! > Two things to try: > 1) zpool scrub. This will reveal any corrupt metadata objects Ok - some 300 hours to go :) So am now trying figure out why it is running at just 8M/s (prefetch_disable=1, vfs.zfs.scrub_delay=0). > 2) Maybe the filename is created in an encoding not supported by your > current terminal. Try "LANG=en_US.UTF-8 ls -l” No cookie (and not overly likely - barebone install which is not visible to anything ‘modern’ but ssh et.al.). > 3) Use zdb to examine the file. First, do "ls -li /root" to get the > object id. It's the same as the inode number. Then, assuming /root > is in the tank/root filesystem, do "zdb -ddddd tank/root <object id>". > That might reveal some clues. A: zdb -ddddd tank/root 7426414 gives; after a 50 second pause (pre/during ‘zpool scrub’): Dataset tank/root [ZPL], ID 40, cr_txg 6, 902M, 14669 objects, rootbp DVA[0]=<0:4c000:4000> DVA[1]=<0:4c00004c000:4000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=225L/225P fill=14669 cksum=9c7252c3b:ad096bfa68f:7b6298f1d2648:4235b444c02eba0 Object lvl iblk dblk dsize lsize %full type zdb: dmu_bonus_hold(7426414) failed, errno 2 So I guess I should wait for the scrub to complete. Cannot recall scrub to be this slow. Dw.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C3EE2E11-92FA-4CA0-A193-F6CECB3E232F>