From owner-freebsd-hackers@freebsd.org Fri Aug 24 21:54:54 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B39701094B9D for ; Fri, 24 Aug 2018 21:54:54 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47D64781AF for ; Fri, 24 Aug 2018 21:54:54 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-oi0-x22e.google.com with SMTP id l202-v6so10890345oig.7 for ; Fri, 24 Aug 2018 14:54:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=/xpLAp0W+nhPb0LAp3UT3LT50kdzQ/sN/ehLDKQbquc=; b=GHRXfEaDhHaDok00ah9nk5BS22k+Run+EX1yoS5h+utw15g7ouRC5GA4GfYLlYeosg zDpAlUClkX3uGMfOU2uZDgI1e5cR7Gi9EfrTeiuSlgFX5x5Cj15qLvtJsV599wsTEuDa NwgwslGdHjYuYssigb20HNEV+uCGJgY43YfAaGaEPMsncU+VldJAzETkIvJBj7l9Srj4 NUlfNdhuveKj3ASUIQ/xuBvjP20VLkmtYZmibBpU6Ek/BtIomsgQC0Ety1CSUia+Tr2i ozaNr8Z1IHnFkYYjzw2Qyydys7eBAlF/X/240a9JmWBkLqkbn+5VxUmqelPUgkaupSR2 cggA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=/xpLAp0W+nhPb0LAp3UT3LT50kdzQ/sN/ehLDKQbquc=; b=ccdu7A/tFquAlQGLNviNfSmHxnxYJ9sXolWrQLH6xpHLXjt3xlJvJnIQqbZZp1mqWc R3qhfmUJLjg+N6TEpv7pJijWSB81f8K+pTLBpWnPsWl9N+PRU1Cz5eeYwcLLgeou1aM2 Q+R2gxETbASoykiRFkVGmDk7HzVR090ZouyJG/otBfWGNTqT6GG8aftwc10ZcK+JAFyJ H0w+WfUUWXk5aVYEY2TiN6rs84LV8DmgCvvStvtx3bPb8i35s/0C2I5YLCDkDtDsfo/g 87LtrORNg8L0BZXCDLn7FR9x2y/g44RTbYOfLxGLN7pP+hdL7NWwvCXIo/gaEU2PI/Aw L+Ig== X-Gm-Message-State: APzg51Ck8i0HB1PM9L+weNAwfOoVYzNohDmcNTEteyf/spUadtbFabkO wCZ6alpLatDLft+X3IDP1DLQJXxzFWwUkLJOEwHjbQ== X-Google-Smtp-Source: ANB0VdbdaKGwzlZbiFTlz7xtztAswSH4noajk3mToCeI8elvjjHgKCEjy/aeUqSPJt3/v0QqELiVxqf6oIjFEgdak3I= X-Received: by 2002:aca:de08:: with SMTP id v8-v6mr3317786oig.124.1535147693363; Fri, 24 Aug 2018 14:54:53 -0700 (PDT) MIME-Version: 1.0 From: David Cross Date: Fri, 24 Aug 2018 17:54:42 -0400 Message-ID: Subject: weird geli behavior To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 21:54:54 -0000 Ok, I am seeing something truely bizzare, I am sending this out as a shot across the bow since I am not even sure where or how to begin debugging this. Some background. This in on an Intel Xeon 5520 based machine, 72G ECC memory, 11.2, fully patched. Though this has been a problem since at least 11.1, probably 11.0, and maybe earlier. ~4G of eli encrypted swap, which it basically never even touches, even when problems are occuring) The first symptom was (and I think these are all aspects of the same root underlying cause) that fsck on a geli encrypted d stripe of 2 USB drives would *randomly* error out on a corrupt entry. Upon investigating this I discovered by watching gstat that as this happened the IO on the drives would STOP. the L(q) would hover at 1 for a number of seconds, and then when it returned fsck was complaining about various corrupt structures. a ktrace of fsck shows that it got back data from the pread() that was partially corrupted (I am guessing, but I cannot confirm that 'some part' of the stack handed back a zeroed page, or otherwise 'not the right data' that geli dutifully 'decrypted'. No errors are ever logged in the kernel about da0 or da1 (the respective underlying USB disks). It *seems* this is *always* on phase 2 of fsck (files and paths), and its never the same inode. no data is *ever* corrupted when in the filesystem, no matter how hard I hit the disks (all data on these devices is fully checksummed) Devices have passed multiple SMART full diag checks, full read/write tests with no issues. Under heavy FS IO it does occasionally lock.. but recovers, and again data and filesystem are fully consistent. I was willing to live with that.. weird as it was (these are backup disks, data is fully checksummed, and I was only fscking out of extreme paranoia every reboot) Then I added an internal drive, configured with gmirror (broken mirror currently, second disk hasn't been added) and geli. On this disk I have a postgres 10 database in WAL replication. This was working fine and then the other day the system just locked for a few hours. During that time I saw the L(q) of the _internal_ disk in the 10,000+ range, and it doing _1_ operation a second to the underlying disk... all the while geli is logging 'error 11' to the console (nothing about the underlying disk) After this happened a static file on the disk (a zip file) had bad data in the middle of a page (after reboot the file was ok.. so it was just in cache). Again, this disk fully checks ok, no corruption on the disk, no errors from the disk itself. Halp? where do I even begin with this? It really feels like there is some massive locking going on in geli in some way? Where should I even begin looking? I run geli on most of my systems and don't have any issues.