From owner-freebsd-hackers@freebsd.org Wed Dec 14 19:05:56 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DE6CFC8071B for ; Wed, 14 Dec 2016 19:05:56 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BC6BE300 for ; Wed, 14 Dec 2016 19:05:56 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B8C4CC8071A; Wed, 14 Dec 2016 19:05:56 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8632C80719 for ; Wed, 14 Dec 2016 19:05:56 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qk0-x231.google.com (mail-qk0-x231.google.com [IPv6:2607:f8b0:400d:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 73A122FF for ; Wed, 14 Dec 2016 19:05:56 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-qk0-x231.google.com with SMTP id n21so31793372qka.3 for ; Wed, 14 Dec 2016 11:05:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=bmTYpi5X129Ty2PwOvcDKZ4AyA1jcef6q6XqOtRXFS8=; b=smzm76x6CMjW5lC9qfAMCwLgJOPBDrtOji4Atb/f6yKGYfN7MmBUCDG8TxJSLvI87/ IQuQ1MoRtgf5LqBe7MkdeyZr7lni/7qJfpAeKjuuDstwqo3olJJ5SaxHzmVRmP0RN05/ ZzUBZzo+Fp8OeUyCppN1xgiYACo4BjTs8kI4+j/H2LyhJWO+jobhmlr9cMk1kKPhsDoF SjffL50rliPzO3L4Eq7WS/aBhMuc9kLVfHq9PNzC0+QBnoaP8nO5Gbx+kRRLYZ0/xgl9 6RaxFuE5ZSCFYX80LMHXZGcl76HQvgQe0CkkMeZfoudo/P46mDzR4WgBp0ofrjtCd+YA eT4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=bmTYpi5X129Ty2PwOvcDKZ4AyA1jcef6q6XqOtRXFS8=; b=gxSOwMiI4siifZiJ2rMuF4SMpGf8B8NarmtueZna1pQcF920DxuzmVZxPSKVDVzKGy TQrkk/Pmd5CcvsTeiRpbLkLgdA8HY4Xv9SacDGnuoxj7Z9F9yQUlP5ShK5lJithzGsh5 ufyMfPAIpElelZEESc3UQQ1i226PyqbZJGDvgpSMMYxUpnZoI5o8hqlxXUH61hMrDrLQ g8HOWlxwh6WnVMYu4R0NGEJtxxoNDmwtbjS3wk3IqmFK1p/5wG40KOC43mA/Cg+9OBcq z+X91zKDM+NRwITy4WLLDEdNDcCd1s+P3ditH1CSxbXirmev+TOuGnkSJu52Qzq/da72 P79w== X-Gm-Message-State: AKaTC02zM392u6amOkCXVkEPDyHhyZ4LylcCvrd5HubfLAWzsq1Zj4KIceCnJU1XmuCzXw3p2uuVnKTz7ugf7w== X-Received: by 10.55.80.198 with SMTP id e189mr39201729qkb.222.1481742355670; Wed, 14 Dec 2016 11:05:55 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.12.174.145 with HTTP; Wed, 14 Dec 2016 11:05:55 -0800 (PST) In-Reply-To: <68290275-9C8D-4ACC-9E29-B6A7805512FD@webweaving.org> References: <20161214185154.GT61036@server.rulingia.com> <68290275-9C8D-4ACC-9E29-B6A7805512FD@webweaving.org> From: Alan Somers Date: Wed, 14 Dec 2016 12:05:55 -0700 X-Google-Sender-Auth: Z1mCm5iDdtSW4AAPNNJzzvLWJl0 Message-ID: Subject: Re: ZFS - directory entry To: Dirk-Willem van Gulik Cc: Peter Jeremy , FreeBSD Hackers Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2016 19:05:57 -0000 On Wed, Dec 14, 2016 at 12:01 PM, Dirk-Willem van Gulik wrote: > >> On 14 Dec 2016, at 19:51, Peter Jeremy wrote: >> >> On 2016-Dec-14 16:27:00 +0100, Dirk-Willem van Gulik wrote: >>> A rather odd directory entry (in /root, the home dir of root/toor) appe= ared on a bog standard FreeBSD 10.2 (p18) lightly loaded machine under ZFS = during/post a backup: >>> >>> $ ls -la /root | tail -q >>> ---------- 1 root wheel 9223372036854775807 Jan 1 1970 ?%+?kD?H??= ?x,?5?Dh;*s!?h???jw??????\h?:????????``?13?@?????OA????????Puux????>> >>> OS and ZFS is installed with a bog standard sysinstall. =E2=80=98SMART= =E2=80=99 nor smartd have reported anything. nothing in dmesg, syslog of bo= ot log. Any suggestions as how to debug or get to the root of this ? >>> >>> And in particular - what is a risk of a reboot (to get a kernel with de= bug, etc) causing the issue to =E2=80=98go away=E2=80=99 - and hence stoppi= ng the forensic ? >> >> Do you have ECC RAM? If not, it's possible this is an artifact of some = RAM >> corruption, rather than on-disk corruption. > > No ECC (simple 4x8 Gbyte sticks; nothing on L2 and L3 either). > >> I'm surprised by the slow scrub, though they are very slow disks. You m= ight >> like to use gstat or zpool iostat to see if one of the disks is slower t= han >> the others - indicating a possible problem with it. > > Perfectly balanced. Rocks steady last few hours according to per disk gra= phs. We=E2=80=99ll see how it goes before rebooting (and if it is indeed me= mory - that will I guess mask the issue until it appears again). > > Thanks, > > Dw If your system is mostly idle, then you can speedup scrub with the following sysctls: vfs.zfs.resilver_min_time_ms=3D5000 vfs.zfs.top_maxinflight=3D1024 # 1024 is appropriate for 8GB of ram. With more RAM, you can go higher. -Alan