From owner-freebsd-hackers@freebsd.org Sat Aug 25 01:00:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08DFC109986D for ; Sat, 25 Aug 2018 01:00:26 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CBC37F4A0 for ; Sat, 25 Aug 2018 01:00:24 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w7P10N4G078523 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 24 Aug 2018 18:00:23 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w7P10NmT078522; Fri, 24 Aug 2018 18:00:23 -0700 (PDT) (envelope-from jmg) Date: Fri, 24 Aug 2018 18:00:23 -0700 From: John-Mark Gurney To: David Cross Cc: FreeBSD Hackers Subject: Re: weird geli behavior Message-ID: <20180825010023.GD45503@funkthat.com> Mail-Followup-To: David Cross , FreeBSD Hackers References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Fri, 24 Aug 2018 18:00:23 -0700 (PDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2018 01:00:26 -0000 David Cross wrote this message on Fri, Aug 24, 2018 at 17:54 -0400: > Ok, I am seeing something truely bizzare, I am sending this out as a shot > across the bow since I am not even sure where or how to begin debugging > this. > > Some background. This in on an Intel Xeon 5520 based machine, 72G ECC > memory, 11.2, fully patched. Though this has been a problem since at least > 11.1, probably 11.0, and maybe earlier. ~4G of eli encrypted swap, which it > basically never even touches, even when problems are occuring) I assume you've applied the lazyfpu SA patch? If so, there's another patch you need to apply, see: https://docs.freebsd.org/cgi/mid.cgi?20180821081150.GU2340@kib.kiev.ua > The first symptom was (and I think these are all aspects of the same root > underlying cause) that fsck on a geli encrypted d stripe of 2 USB drives > would *randomly* error out on a corrupt entry. Upon investigating this I > discovered by watching gstat that as this happened the IO on the drives > would STOP. the L(q) would hover at 1 for a number of seconds, and then > when it returned fsck was complaining about various corrupt structures. a > ktrace of fsck shows that it got back data from the pread() that was > partially corrupted (I am guessing, but I cannot confirm that 'some part' > of the stack handed back a zeroed page, or otherwise 'not the right data' > that geli dutifully 'decrypted'. No errors are ever logged in the kernel > about da0 or da1 (the respective underlying USB disks). It *seems* this is > *always* on phase 2 of fsck (files and paths), and its never the same > inode. no data is *ever* corrupted when in the filesystem, no matter how > hard I hit the disks (all data on these devices is fully checksummed) > Devices have passed multiple SMART full diag checks, full read/write tests > with no issues. Under heavy FS IO it does occasionally lock.. but > recovers, and again data and filesystem are fully consistent. > > I was willing to live with that.. weird as it was (these are backup disks, > data is fully checksummed, and I was only fscking out of extreme paranoia > every reboot) Then I added an internal drive, configured with gmirror > (broken mirror currently, second disk hasn't been added) and geli. On this > disk I have a postgres 10 database in WAL replication. This was working > fine and then the other day the system just locked for a few hours. During > that time I saw the L(q) of the _internal_ disk in the 10,000+ range, and > it doing _1_ operation a second to the underlying disk... all the while > geli is logging 'error 11' to the console (nothing about the underlying > disk) After this happened a static file on the disk (a zip file) had bad > data in the middle of a page (after reboot the file was ok.. so it was > just in cache). Again, this disk fully checks ok, no corruption on the > disk, no errors from the disk itself. > > > Halp? where do I even begin with this? It really feels like there is > some massive locking going on in geli in some way? Where should I even > begin looking? I run geli on most of my systems and don't have any issues. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."