From owner-freebsd-stable@freebsd.org Sun Apr 14 13:01:48 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 091EF15754FA for ; Sun, 14 Apr 2019 13:01:48 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B5D2F69EDD for ; Sun, 14 Apr 2019 13:01:41 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x3ED1XvP089723; Sun, 14 Apr 2019 15:01:33 +0200 (CEST) Received: from [217.29.46.66] (unassigned [217.29.46.66] (may be forged)) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x3ED1WID013331; Sun, 14 Apr 2019 15:01:32 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o and controller resets From: "Patrick M. Hausen" In-Reply-To: Date: Sun, 14 Apr 2019 15:01:32 +0200 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> To: Warner Losh X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: B5D2F69EDD X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.80 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.981,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[mailin.pluspunkthosting.de,mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.73)[-0.733,0]; IP_SCORE(-0.28)[ipnet: 217.29.32.0/20(-0.77), asn: 16188(-0.62), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Apr 2019 13:01:48 -0000 Alrighteeee ... > Am 13.04.2019 um 02:37 schrieb Warner Losh : > > There's been some minor improvements in -current here. Any chance = you could experimentally try that with this test? You won't get as many = I/O abort errors (since we don't print those), and we have a few more = workarounds for the reset path (though honestly, it's still kinda = stinky). >=20 > HEAD or RELENG_12, too? >=20 > HEAD is preferred, but any recent snapshot will do. I could not reproduce the problem for a couple of hours with an otherwise identical system but only 4 of these Intel drives. Now the same test system with 6 drives just as our FreeNAS boxes - instantly reproducible. I=E2=80=99ll upgrade to HEAD and see if that changes anything. Kind regards Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Mon Apr 15 06:46:11 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7B26158AA69 for ; Mon, 15 Apr 2019 06:46:10 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8CBAF6BFF5 for ; Mon, 15 Apr 2019 06:46:09 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x3F6k6bM099597; Mon, 15 Apr 2019 08:46:06 +0200 (CEST) Received: from [217.29.44.36] ([217.29.44.36]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x3F6k66g034441; Mon, 15 Apr 2019 08:46:06 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o and controller resets From: "Patrick M. Hausen" In-Reply-To: <1A448217-C0FA-4B62-9A7F-5AF9A83D207B@punkt.de> Date: Mon, 15 Apr 2019 08:46:05 +0200 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> <26B9E28E-8BA5-41EE-9146-5336AA7605A6@punkt.de> <1A448217-C0FA-4B62-9A7F-5AF9A83D207B@punkt.de> To: Warner Losh X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 8CBAF6BFF5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.64 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.987,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.58)[-0.577,0]; IP_SCORE(-0.27)[ipnet: 217.29.32.0/20(-0.75), asn: 16188(-0.60), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 06:46:11 -0000 Hi! > Am 14.04.2019 um 23:33 schrieb Patrick M. Hausen : > Since the system runs well with RELENG_11 and only 4 drives > and there is this question about the cabling and shared resources > I will try to set up a system with 5 drives, each of them *without* > another one in a =E2=80=9Epair=E2=80=9C sharing the same MB connector. So much for that theory: with 5 drives arranged in that way I get the errors even during installation. https://cloud.hausen.com/s/2myrX2Jr3fgLWGj https://cloud.hausen.com/s/yryckgp56sH2CRe So I=E2=80=99ll test RELENG_12 next. If that works, I can probably craft a FreeNAS 11.2 installation with a 12 kernel. I would be hesitating to = run HEAD in production, though. Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Mon Apr 15 08:01:30 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03159156B4E5 for ; Mon, 15 Apr 2019 08:01:30 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (m.saper.info [IPv6:2a01:4f8:a0:7383::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "saper.info", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B4136DF59 for ; Mon, 15 Apr 2019 08:01:29 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (saper@m.saper.info [IPv6:2a01:4f8:a0:7383:0:0:0:0]) by m.saper.info (8.15.2/8.15.2) with ESMTPS id x3F81R5d051535 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Mon, 15 Apr 2019 08:01:28 GMT (envelope-from saper@saper.info) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=saper.info; s=Sep2014; t=1555315288; bh=PKuXLmAnPpNNQ8C+oItQJy4cQAdUQeNuDTo8flB5+TY=; h=Date:From:To:Subject; b=UV76kXJ4W39A+FrQqrUUJU+vHDxO8BRsDMV3ZLjYfI4II9xxoHAdY57JpYbgIhKZh i9MZF4GaJ3vp4i4Q/YFBpoNJX+OAbwIm7veSDHYI4W+P47JbdahzrW/wb3klctW1+0 riITT1LsIoruBwCKJYxA2sIJGNWa2xSpSgNfQkfo= Received: from localhost (saper@localhost) by m.saper.info (8.15.2/8.15.2/Submit) with ESMTP id x3F81Rcr051532 for ; Mon, 15 Apr 2019 08:01:27 GMT (envelope-from saper@saper.info) X-Authentication-Warning: m.saper.info: saper owned process doing -bs Date: Mon, 15 Apr 2019 08:01:27 +0000 From: Marcin Cieslak To: freebsd-stable@freebsd.org Subject: After 12.x upgrade: mysqld RET _umtx_op -1 errno 45 Operation not supported Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 3B4136DF59 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=saper.info header.s=Sep2014 header.b=UV76kXJ4 X-Spamd-Result: default: False [-4.04 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[saper.info:s=Sep2014]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[saper.info:+]; MX_GOOD(-0.01)[cached: m.saper.info]; DMARC_NA(0.00)[saper.info]; NEURAL_HAM_SHORT(-0.91)[-0.906,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; IP_SCORE(-0.82)[ipnet: 2a01:4f8::/29(-2.13), asn: 24940(-1.98), country: DE(-0.01)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 08:01:30 -0000 Hello, for archival purposes I am running an ancient (__FreeBSD_version 602100) jail with mysqld inside. This was working fine when the jail host was running 10.4. After upgrade to 12.x (r345375) mysqld process starts but it hangs before PID is created and the socket opened. Quick ktrace on the process shows this coming up all the time: 50766 mysqld CALL _umtx_op(0x966eaa0,UMTX_OP_RESERVED0,0x18cab,0,0) 50766 mysqld RET _umtx_op -1 errno 45 Operation not supported What has changed, can I somehow bring it up again? (I understood that we keep compatibility with very old userland). Marcin From owner-freebsd-stable@freebsd.org Mon Apr 15 08:19:53 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1291A156BB73 for ; Mon, 15 Apr 2019 08:19:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5ECA96E888 for ; Mon, 15 Apr 2019 08:19:52 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x3F8Jjqn001004 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 15 Apr 2019 11:19:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x3F8Jjqn001004 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x3F8Jij1001003; Mon, 15 Apr 2019 11:19:44 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 15 Apr 2019 11:19:44 +0300 From: Konstantin Belousov To: Marcin Cieslak Cc: freebsd-stable@freebsd.org Subject: Re: After 12.x upgrade: mysqld RET _umtx_op -1 errno 45 Operation not supported Message-ID: <20190415081944.GP1923@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 08:19:53 -0000 On Mon, Apr 15, 2019 at 08:01:27AM +0000, Marcin Cieslak wrote: > Hello, > > for archival purposes I am running an ancient (__FreeBSD_version 602100) > jail with mysqld inside. > > This was working fine when the jail host was running 10.4. After upgrade to 12.x > (r345375) mysqld process starts but it hangs before PID is created and the > socket opened. > > Quick ktrace on the process shows this coming up all the time: > > 50766 mysqld CALL _umtx_op(0x966eaa0,UMTX_OP_RESERVED0,0x18cab,0,0) > 50766 mysqld RET _umtx_op -1 errno 45 Operation not supported > > What has changed, can I somehow bring it up again? (I understood that > we keep compatibility with very old userland). I believe these ops were removed at r263318. From owner-freebsd-stable@freebsd.org Mon Apr 15 08:28:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39CF2156C05B for ; Mon, 15 Apr 2019 08:28:41 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (m.saper.info [IPv6:2a01:4f8:a0:7383::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "saper.info", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BD2756EE0F for ; Mon, 15 Apr 2019 08:28:40 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (saper@m.saper.info [IPv6:2a01:4f8:a0:7383:0:0:0:0]) by m.saper.info (8.15.2/8.15.2) with ESMTPS id x3F8SdIM052311 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 15 Apr 2019 08:28:39 GMT (envelope-from saper@saper.info) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=saper.info; s=Sep2014; t=1555316919; bh=iVjh0Lis/LauGpJICY/ifPW/+oGlhKzyQmW+e5E6ic4=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=Z7/iOsNTv/ONcEXUjBgRPgVcAMZ9EQT50KGqxTGCjLEr9OqcL66DH2luhyvO9wbpz OCsMspa20ntSVmzdXt8R+XOxh03MZ9BEOs4ukQk3NLmvsk6R4ytX2Gh6Nb6v68WLsn oNOzoEjjcWZw1W//G8CviFunfJF1dsEjuzKejTeM= Received: from localhost (saper@localhost) by m.saper.info (8.15.2/8.15.2/Submit) with ESMTP id x3F8SdCw052308; Mon, 15 Apr 2019 08:28:39 GMT (envelope-from saper@saper.info) X-Authentication-Warning: m.saper.info: saper owned process doing -bs Date: Mon, 15 Apr 2019 08:28:39 +0000 From: Marcin Cieslak To: Konstantin Belousov cc: freebsd-stable@freebsd.org Subject: Re: After 12.x upgrade: mysqld RET _umtx_op -1 errno 45 Operation not supported In-Reply-To: <20190415081944.GP1923@kib.kiev.ua> Message-ID: References: <20190415081944.GP1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="1563967779-1364000654-1555316919=:84924" X-Rspamd-Queue-Id: BD2756EE0F X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.992,0] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 08:28:41 -0000 --1563967779-1364000654-1555316919=:84924 Content-Type: text/plain; charset=US-ASCII On Mon, 15 Apr 2019, Konstantin Belousov wrote: > On Mon, Apr 15, 2019 at 08:01:27AM +0000, Marcin Cieslak wrote: > > > > 50766 mysqld CALL _umtx_op(0x966eaa0,UMTX_OP_RESERVED0,0x18cab,0,0) > > 50766 mysqld RET _umtx_op -1 errno 45 Operation not supported > I believe these ops were removed at r263318. That was quick, thank you! Is there any way this could be fixed on the library level? --1563967779-1364000654-1555316919=:84924 Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: BASE64 Content-Description: S/MIME Cryptographic Signature Content-Disposition: attachment; filename=smime.p7s MIIOSwYJKoZIhvcNAQcCoIIOPDCCDjgCAQExDzANBglghkgBZQMEAgEFADAL BgkqhkiG9w0BBwGgggqQMIIElzCCA3+gAwIBAgIOSBtqCKJEiNNcmz3JSA0w DQYJKoZIhvcNAQELBQAwTDEgMB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENB IC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMTCkdsb2JhbFNp Z24wHhcNMTYwNjE1MDAwMDAwWhcNMjQwNjE1MDAwMDAwWjBdMQswCQYDVQQG EwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEzMDEGA1UEAxMqR2xv YmFsU2lnbiBQZXJzb25hbFNpZ24gMSBDQSAtIFNIQTI1NiAtIEczMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyrCba00KOKyGuwh9h+/MAcZm ZUF9OxGKA56AADHaDE08rB0WEbgm6J4XvJP3OGQ7cgHdVJu6XMZkRd6EcfjD yRrIwE6oAVWJe57co3gKk/XxvuubSZuUahrcOiv3D2qaHwva4zumubxQQI4f unEzRIJHPiNjaq0cCcZsMcp5pxsEz8aG0sr8Oh80sxKNnzPmuUETLESktfMC pQKHUGmWXLsG6sgCZOezUjDjKpPKW7l4PUt0TEBEyqLhifv9/YPn5C4o10PP daDazZPeKNif2PVQ5u0HRnkFrHh4wmmrMtY22Mse3eR01gD6rEEGWf+gdzuy EQE+ZVlNhCP4gXjdBQIDAQABo4IBZDCCAWAwDgYDVR0PAQH/BAQDAgEGMCcG A1UdJQQgMB4GCCsGAQUFBwMCBggrBgEFBQcDBAYIKwYBBQUHAwkwEgYDVR0T AQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUlifCwqX3HPgCenpkr2NvMtKYwrEw HwYDVR0jBBgwFoAUj/BLf6guRSSuTVD6Y5qL3uLdG7wwPgYIKwYBBQUHAQEE MjAwMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcDIuZ2xvYmFsc2lnbi5jb20v cm9vdHIzMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFsc2ln bi5jb20vcm9vdC1yMy5jcmwwWQYDVR0gBFIwUDALBgkrBgEEAaAyASgwQQYJ KwYBBAGgMgFfMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8vd3d3Lmdsb2JhbHNp Z24uY29tL3JlcG9zaXRvcnkvMA0GCSqGSIb3DQEBCwUAA4IBAQCxh3ekjKKy RrUdfI6D1U7qUggdFLksiU+KiIqJzJG6GXcQ2KiBy2tF3+KYb0IixXMpIVli VXlcD5Vh4tiMxJ4WONMFt3f7/53gSXLf24WMwErubc+mGMzgUGE5HKC98PcK UV/5pPggQdzPxCBNeiXnLU1tCGYhPatFTDhUBGaVhBeuUCbgR9gpXJ9guqrD OVwouKvovdIeI5KEAcoAAiSL6naeLk/GbKUaBFa2RxXC17e+YyBWtWlWDEM3 1V8pUIx76lkO8IJYREhLcg/LnyoYy5wcrzI6pbX2vw1x/jR3GHSC1AEdoqbE xui2XLLlSa6y9yQNgdkPz7GTLmpwIT+dMIIF8TCCBNmgAwIBAgIMGk4Oe/1h 2+wMOby/MA0GCSqGSIb3DQEBCwUAMF0xCzAJBgNVBAYTAkJFMRkwFwYDVQQK ExBHbG9iYWxTaWduIG52LXNhMTMwMQYDVQQDEypHbG9iYWxTaWduIFBlcnNv bmFsU2lnbiAxIENBIC0gU0hBMjU2IC0gRzMwHhcNMTcwNTI1MDg0NDE2WhcN MjAwNTI1MDg0NDE2WjA8MRkwFwYDVQQDDBBzYXBlckBzYXBlci5pbmZvMR8w HQYJKoZIhvcNAQkBFhBzYXBlckBzYXBlci5pbmZvMIICIjANBgkqhkiG9w0B AQEFAAOCAg8AMIICCgKCAgEA2sO3aQNus/oe4ZBZ4fu1Y1mzxnUYAkb4k/dw gMFc2Kd0eRoOY0AHj4rTEi/vVzzizxjLbEwXzQ9cBEAu/PqS8WsOmhZXtlfi szPDmP7ZpOwmNTWKSd9O7jHu9uTCGfEOsocQNYH2ULD1gVFkgKb8jHf+3u9d uCzh6qMomTtwLrCGEP70Lq385xUzRaD6qbOeIB99tpzgvMR6Z0GPTt4z8tLM kfdtohq5llwZ5vYnj/hJohVS9iLMQMHW4nuLj/mLZNaYE1CWJBT1rBwn5YPJ uR6811O9eAP7aX4iG8k1jkiBh+QNgGRBIK4GIdqy7IVRhA7v2OlpLYHMk4zP 9Fs3M+56QromVKBnxfzLhuYMUK6ugj9jwskNVitqlEFUeyfgvmR1jnPRp1Nd XGJllTNwGicR8wkaRj14RxfrvTZfwXs8OBODKFupqun/tNzdpOgyHMGQACss 9yv2SnLGCJvJK3rGIdRZEiUhLZH/Ct4L92dBhev+SjUqWKbHb4yIlGMgLdoh nwqatuWw7iyOeInjcinX7ghiIKDWhulUN493Fzl6kaUBtIIcrb7jzZ2pHAQT WUmuVnCTHk6NtoWB09lvuK77fw4GfxLWDFWkBQiJYPVBrmxlrkCKzrWdTMfS W9BiEC10jT1sSimUBIjDz22RkfsApeBJoAIWjiOZogILu9MCAwEAAaOCAdAw ggHMMA4GA1UdDwEB/wQEAwIFoDCBngYIKwYBBQUHAQEEgZEwgY4wTQYIKwYB BQUHMAKGQWh0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5jb20vY2FjZXJ0L2dz cGVyc29uYWxzaWduMXNoYTJnM29jc3AuY3J0MD0GCCsGAQUFBzABhjFodHRw Oi8vb2NzcDIuZ2xvYmFsc2lnbi5jb20vZ3NwZXJzb25hbHNpZ24xc2hhMmcz MEwGA1UdIARFMEMwQQYJKwYBBAGgMgEoMDQwMgYIKwYBBQUHAgEWJmh0dHBz Oi8vd3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMAkGA1UdEwQCMAAw RAYDVR0fBD0wOzA5oDegNYYzaHR0cDovL2NybC5nbG9iYWxzaWduLmNvbS9n c3BlcnNvbmFsc2lnbjFzaGEyZzMuY3JsMBsGA1UdEQQUMBKBEHNhcGVyQHNh cGVyLmluZm8wHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1Ud DgQWBBReBINaGUKUo7HCrIjsKLKERu6ooTAfBgNVHSMEGDAWgBSWJ8LCpfcc +AJ6emSvY28y0pjCsTANBgkqhkiG9w0BAQsFAAOCAQEAC0VK968ySq/6B+Kd ecjVThQOKtVXuG17Krfk0xz7OPYR/V+qZtBFm2Uc6tkUEmAmq3Tyf+SE3TTX Q58eJFq0uCTUhIY714ioJs1uVWBz8rPyJ3swkOfDaUXUxkQsBsf73VfKjUk4 kB5MTrApLYUe35NmEY3FqyyX13elhW1tp864vOKM2Git61cYoRn/bwd/z2JM Zkxwkd5JgvmM+p4Da+WO4CUsGzdrZEH8X/8NQIzWtUDIh7VEQZFX5fot/KvH Am8AajtpmNqTfMyg6LfcfJUXSFqXn/KEWu4Td62vX6Pd70dYKUZxnLwYvGqG A4Ktrp9zyrUzxLbmdaPln7CstjGCA38wggN7AgEBMG0wXTELMAkGA1UEBhMC QkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExMzAxBgNVBAMTKkdsb2Jh bFNpZ24gUGVyc29uYWxTaWduIDEgQ0EgLSBTSEEyNTYgLSBHMwIMGk4Oe/1h 2+wMOby/MA0GCWCGSAFlAwQCAQUAoIHkMBgGCSqGSIb3DQEJAzELBgkqhkiG 9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE5MDQxNTA4MjgzOVowLwYJKoZIhvcN AQkEMSIEIFM5g5aGqf4uXJUvVmz3he+c4TRbLXzz53Za9oTJGxPLMHkGCSqG SIb3DQEJDzFsMGowCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBFjALBglghkgB ZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMC AgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMA0GCSqGSIb3DQEBAQUABIIC AFEzxevjlschD1PL3zclEaFY+e1U5qJ4TMsZa60wn8nEdxMSJ2EgfgIXafYj ZnyoRcMzQHQgItqf7956aF+SuYVYDbO+3lPMpVaxiayNjLISgm4OE7N+4St5 MVifIjMp8qDnXNDkp1VfUJZjgR5wwlrB40tDwihd/VtSyrzSIjsOrou79j/1 eAcsPfSHD/XdIF1Cr+f8E5/AeJAo+miC0+E5I30VLSiG/otRt0H1sw4/mNJX 38TTMdoDqfrMPsZ9bibKq/GILp54AW346+ohy7Cl+bYxTbio9qv995hlewOT BqM0AwY3Wnv+HCsf3YKVVJ0abXF2IudaoWM7MlGjnD6beMjLRMmI3QBwt2QG yrgVrT2L5/u8hsJFt3VtIXZL5l6anxHirEL5rwFHKdnxVgaiZdu/LhGfPtXD bGPU0ekbBz+LH1796GsZLtbddjYcBe0ZQAkAFHBbFn21FNXl2BKscTSrnKyq 1UfiaR1ilbektw/WNO8cdtbIqmhvfVipNkupn8/kS+g1j/e4tOObbxFyKL87 ELNKDoTdV/kQ2bHT7GS633s5rbI+vI/uZ9RCKv+P7Xt7b7kE5rRo8RnJvRJd zGYPa1HEJyH+EZIt9+y9E/YSdq15IAU0jVyNj04qg1+5ck8O2CUodb/pFEAC 8uMdKR/pGH1/PqyKkssgjrrR --1563967779-1364000654-1555316919=:84924-- From owner-freebsd-stable@freebsd.org Mon Apr 15 08:51:49 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1D09156CCD7 for ; Mon, 15 Apr 2019 08:51:49 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 56A386FD06 for ; Mon, 15 Apr 2019 08:51:48 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x3F8pjK5002635; Mon, 15 Apr 2019 10:51:45 +0200 (CEST) Received: from [217.29.44.36] ([217.29.44.36]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x3F8piRb040559; Mon, 15 Apr 2019 10:51:44 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o and controller resets From: "Patrick M. Hausen" In-Reply-To: Date: Mon, 15 Apr 2019 10:51:44 +0200 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <63F643CA-EDB1-41EB-A27E-EB427F3886B4@punkt.de> References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> <26B9E28E-8BA5-41EE-9146-5336AA7605A6@punkt.de> <1A448217-C0FA-4B62-9A7F-5AF9A83D207B@punkt.de> To: Warner Losh X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 56A386FD06 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.80 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.987,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.74)[-0.741,0]; IP_SCORE(-0.26)[ipnet: 217.29.32.0/20(-0.73), asn: 16188(-0.59), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 08:51:50 -0000 > Am 15.04.2019 um 08:46 schrieb Patrick M. Hausen : > So I=E2=80=99ll test RELENG_12 next. If that works, I can probably = craft > a FreeNAS 11.2 installation with a 12 kernel. I would be hesitating to = run > HEAD in production, though. root@hurz:/var/tmp # uname -a FreeBSD hurz 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 = 04:32:14 UTC 2018 = root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@hurz:/var/tmp # dd if=3D/dev/urandom of=3Dhurz bs=3D10m Result: Apr 15 09:56:07 hurz kernel: nvme4: resetting controller Apr 15 09:56:07 hurz kernel: nvme3: resetting controller Apr 15 09:56:07 hurz kernel: nvme4: aborting outstanding i/o Apr 15 09:56:07 hurz kernel: nvme4: WRITE sqid:5 cid:126 nsid:1 = lba:188361216 len:208 Apr 15 09:56:07 hurz kernel: nvme4: ABORTED - BY REQUEST (00/07) sqid:5 = cid:126 cdw0:0 Apr 15 09:56:07 hurz kernel: nvme4: aborting outstanding i/o Apr 15 09:56:07 hurz kernel: nvme4: WRITE sqid:5 cid:127 nsid:1 = lba:188368784 len:64 Apr 15 09:56:07 hurz kernel: nvme4: ABORTED - BY REQUEST (00/07) sqid:5 = cid:127 cdw0:0 Apr 15 09:56:07 hurz kernel: nvme4: aborting outstanding i/o Apr 15 09:56:07 hurz kernel: nvme4: WRITE sqid:5 cid:125 nsid:1 = lba:188371408 len:48 Apr 15 09:56:07 hurz kernel: nvme4: ABORTED - BY REQUEST (00/07) sqid:5 = cid:125 cdw0:0 Apr 15 09:56:07 hurz kernel: nvme4: aborting outstanding i/o Apr 15 09:56:07 hurz kernel: nvme4: WRITE sqid:5 cid:124 nsid:1 = lba:188371456 len:16 Apr 15 09:56:07 hurz kernel: nvme4: ABORTED - BY REQUEST (00/07) sqid:5 = cid:124 cdw0:0 [=E2=80=A6] Now, RELENG_12 kernel, 11.2-RELEASE userland: root@hurz:/var/tmp # uname -a FreeBSD hurz 12.0-STABLE FreeBSD 12.0-STABLE r346220 GENERIC amd64 root@hurz:/var/tmp # dd if=3D/dev/urandom of=3Dhurz bs=3D10m Result: no problems, not with two of these jobs running in parallel, not with a = zpool scrub at the same time =E2=80=A6 I uploaded a complete dmesg of the system running RELENG_12: https://cloud.hausen.com/s/5dRMsewCtDFHRYA Is there anything else I should send? pciconf, nvmecontrol =E2=80=A6? Kind regards Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Mon Apr 15 08:52:59 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7534C156CD7A for ; Mon, 15 Apr 2019 08:52:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BAC366FF49 for ; Mon, 15 Apr 2019 08:52:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x3F8qlvD008975 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 15 Apr 2019 11:52:50 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x3F8qlvD008975 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x3F8qlsr008974; Mon, 15 Apr 2019 11:52:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 15 Apr 2019 11:52:47 +0300 From: Konstantin Belousov To: Marcin Cieslak Cc: freebsd-stable@freebsd.org Subject: Re: After 12.x upgrade: mysqld RET _umtx_op -1 errno 45 Operation not supported Message-ID: <20190415085247.GQ1923@kib.kiev.ua> References: <20190415081944.GP1923@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 08:52:59 -0000 On Mon, Apr 15, 2019 at 08:28:39AM +0000, Marcin Cieslak wrote: > On Mon, 15 Apr 2019, Konstantin Belousov wrote: > > > On Mon, Apr 15, 2019 at 08:01:27AM +0000, Marcin Cieslak wrote: > > > > > > 50766 mysqld CALL _umtx_op(0x966eaa0,UMTX_OP_RESERVED0,0x18cab,0,0) > > > 50766 mysqld RET _umtx_op -1 errno 45 Operation not supported > > > I believe these ops were removed at r263318. > > That was quick, thank you! > Is there any way this could be fixed on the library level? You probably can write a LD_PRELOAD'ed library which would intercept _umtx_op and emulate the call. From owner-freebsd-stable@freebsd.org Mon Apr 15 09:15:56 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 33D06156D849 for ; Mon, 15 Apr 2019 09:15:56 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 342DF70D63 for ; Mon, 15 Apr 2019 09:15:54 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x3F9FrOW003296; Mon, 15 Apr 2019 11:15:53 +0200 (CEST) Received: from [217.29.44.36] ([217.29.44.36]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x3F9FqQL042117; Mon, 15 Apr 2019 11:15:52 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o and controller resets From: "Patrick M. Hausen" In-Reply-To: <63F643CA-EDB1-41EB-A27E-EB427F3886B4@punkt.de> Date: Mon, 15 Apr 2019 11:15:52 +0200 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <947E0513-6449-464E-A84D-068F4E3233EF@punkt.de> References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> <26B9E28E-8BA5-41EE-9146-5336AA7605A6@punkt.de> <1A448217-C0FA-4B62-9A7F-5AF9A83D207B@punkt.de> <63F643CA-EDB1-41EB-A27E-EB427F3886B4@punkt.de> To: Warner Losh X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 342DF70D63 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.89 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.985,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.84)[-0.837,0]; IP_SCORE(-0.26)[ipnet: 217.29.32.0/20(-0.71), asn: 16188(-0.57), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 09:15:56 -0000 Hi! > Am 15.04.2019 um 10:51 schrieb Patrick M. Hausen : > Now, RELENG_12 kernel, 11.2-RELEASE userland: >=20 > root@hurz:/var/tmp # uname -a > FreeBSD hurz 12.0-STABLE FreeBSD 12.0-STABLE r346220 GENERIC amd64 > root@hurz:/var/tmp # dd if=3D/dev/urandom of=3Dhurz bs=3D10m >=20 > Result: >=20 > no problems, not with two of these jobs running in parallel, not with = a zpool scrub at the same time =E2=80=A6 After they ran for half an hour I find these in /var/log/messages: Apr 15 11:03:54 hurz kernel: nvme2: Missing interrupt Apr 15 11:07:07 hurz kernel: nvme3: Missing interrupt Apr 15 11:09:47 hurz kernel: nvme4: Missing interrupt They are the only occurrences. The system does not seem to hang or = otherwise misbehave ... Kind regards Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Mon Apr 15 13:24:23 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6966E1574143 for ; Mon, 15 Apr 2019 13:24:23 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BD1E081408 for ; Mon, 15 Apr 2019 13:24:22 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTP id x3FDOFhB046995 for ; Mon, 15 Apr 2019 15:24:15 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id x3FDOFkR046992 for ; Mon, 15 Apr 2019 15:24:15 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Mon, 15 Apr 2019 15:24:15 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable Subject: ZFS parallel mounting gone wrong? Message-ID: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 13:24:23 -0000 Hi, I upgraded a non-critical system running amd64 stable/12 to r346220. During multiuser boot not all ZFS filesystems below zroot/usr/local was mounted. Out of these filesystems, only zroot/usr/local/etc/shellkonfig3 was mounted: zroot/usr/local zroot/usr/local/certs zroot/usr/local/etc zroot/usr/local/etc/shellkonfig3 zroot/usr/local/etc/shellkonfig3/head zroot/usr/local/etc/shellkonfig3/stable-10 zroot/usr/local/etc/shellkonfig3/stable-11 zroot/usr/local/etc/shellkonfig3/stable-8 zroot/usr/local/etc/shellkonfig3/stable-9 zroot/usr/local/info zroot/usr/local/var It's not a one-off bug, it can be replicated simply by rebooting this system. There is no M appended to my revision number. Booting into singleuser mode and correctly mount all filesystems manually before returning to multiuser mode is not something I want to repeat. I will create a shell script to ease the pain anyway. -- Trond. From owner-freebsd-stable@freebsd.org Mon Apr 15 13:31:18 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AEDC157451E for ; Mon, 15 Apr 2019 13:31:18 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 932CD81702 for ; Mon, 15 Apr 2019 13:31:17 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTP id x3FDVElb047058 for ; Mon, 15 Apr 2019 15:31:14 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id x3FDVEYp047055 for ; Mon, 15 Apr 2019 15:31:14 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Mon, 15 Apr 2019 15:31:14 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable Subject: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() Message-ID: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 13:31:18 -0000 Hi, Has anyone else witnessed a panic during reboot involving softclock_call_cc(), nd6_timer(), and nd6_dad_start()? The stack trace goes more or less like this: db_trace_self_wrapper() vpanic() panic() trap_fatal() trap() calltrap() nd6_dad_start() nd6_timer() softclock_call_cc() softclock() ithread_loop() fork_exit() fork_trampoline() This was last seen while transitioning from r345628 to r346220 on amd64 stable/12. -- Trond. From owner-freebsd-stable@freebsd.org Mon Apr 15 14:15:37 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58859157505D for ; Mon, 15 Apr 2019 14:15:37 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 95F67832CB for ; Mon, 15 Apr 2019 14:15:36 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTP id x3FEFXhN047366 for ; Mon, 15 Apr 2019 16:15:33 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id x3FEFXVD047363 for ; Mon, 15 Apr 2019 16:15:33 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Mon, 15 Apr 2019 16:15:32 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable Subject: Re: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 14:15:37 -0000 On Mon, 15 Apr 2019 15:31+0200, Trond Endrestøl wrote: > Has anyone else witnessed a panic during reboot involving > softclock_call_cc(), nd6_timer(), and nd6_dad_start()? > > The stack trace goes more or less like this: > > db_trace_self_wrapper() > vpanic() > panic() > trap_fatal() > trap() > calltrap() > nd6_dad_start() > nd6_timer() > softclock_call_cc() > softclock() > ithread_loop() > fork_exit() > fork_trampoline() > > This was last seen while transitioning from r345628 to r346220 on > amd64 stable/12. The NIC in question is a Chelsio T6225-CR, cxgbe(4), using the cc0 port only. -- Trond. From owner-freebsd-stable@freebsd.org Mon Apr 15 15:55:18 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2DFBE15776D0 for ; Mon, 15 Apr 2019 15:55:18 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-ua1-x935.google.com (mail-ua1-x935.google.com [IPv6:2607:f8b0:4864:20::935]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C30EC8714D for ; Mon, 15 Apr 2019 15:55:16 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-ua1-x935.google.com with SMTP id k32so5684961uae.3 for ; Mon, 15 Apr 2019 08:55:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:from:subject:date:importance; bh=T7xv9+s4KsmQvf7fPT8QvJlAATKdOAZqorTruQkKElk=; b=UF3jxmaIAKkZj1qLP9iyhWylLWjoGN0BAftwk9JhMi6d4VaLzrWu9475yYGv840aFn gTVmvtpF+JC8ImuNIXwnMkAb2ojZn7H5yp5sokDZIzRxAmLCRDxnW+urFlw1ZOGWGmey f3dCnr6OtoXsobgHR0DC1QecDjrU0epA1cGmCuisMZwYTQL7sv86IqLzbT5JtvIw3rxU m0mzhY/aleAZmn3Ew8/A1EERluGv51ch+LSyL684uP8E9SHNAkmHHTNO/Xqs30ppADHT S/mdMp9CWItN2eqZc1IKCXJDlyofXUtgfxvivXOg3BfhlfhUL6UGD9WwC1mzo+VWxzdd E+SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:from:subject:date :importance; bh=T7xv9+s4KsmQvf7fPT8QvJlAATKdOAZqorTruQkKElk=; b=JeVcfh3UdiZEHGbcCdA9eL5ZBxwgJrmcsB417xUmsOdZQRKBs0MChYjHLYbzTiK5fP BpH0X15Fne44ZminVck0ljOF/q+L/JOTmzqosTxvGhuzkvkTRxq32zbjl0Vj1oIfp9Kp PAhVIsnl7cwQMRhyHWmPJiMoHkLC3iPAOCmOm2JcM8dkTc6LC4yDgMYm+bO+SfuQX3+G ylgHzeyybvjaNnCQynHQInjx0VEY4O7qzqUjEQSTWz4hjzYRq9ePi2q9r7zGRZ6dtl7A Gi6Dv2nhmLsNBTHO08JbRNOirxtOJ3q6D/JxM1SEHT6sa661s9Ln4M85hFNjbYYTGwIe C98w== X-Gm-Message-State: APjAAAW2miX2kj9dKyAUAhylFUYbaipkZqmyoY87MFGTrm+TwCSZ52tF uoGufhHquZCqxLGGzh+AtkUrjb3b X-Google-Smtp-Source: APXvYqzIIefUBW5VK61kF/Vb4mO2hcmxkpdrLYm4QWW9pE7w3B/NpK4N4BAOCbZQq+Qt2ER8hJNMqQ== X-Received: by 2002:ab0:702a:: with SMTP id u10mr16932651ual.5.1555343715973; Mon, 15 Apr 2019 08:55:15 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.129? (git.mayberryinv.com. [63.143.111.202]) by smtp.gmail.com with ESMTPSA id y64sm11961077vkd.26.2019.04.15.08.55.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 Apr 2019 08:55:15 -0700 (PDT) Message-ID: <5cb4a963.1c69fb81.a03c.f9af@mx.google.com> MIME-Version: 1.0 To: "freebsd-stable@freebsd.org" From: Software Info Subject: Cron MAILFROM Date: Mon, 15 Apr 2019 10:55:15 -0500 Importance: normal X-Priority: 3 X-Rspamd-Queue-Id: C30EC8714D X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UF3jxmaI; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::935 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-6.95 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.95)[ip: (-9.54), ipnet: 2607:f8b0::/32(-2.97), asn: 15169(-2.20), country: US(-0.06)]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[5.3.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_DN_EQ_ADDR_ALL(0.00)[] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 15:55:18 -0000 Hi All I saw an RFC here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D140= 304. I was just wondering if the MAILFROM feature was ever implemented in C= ron. I would love to use it myself. Regards SI Sent from Mail for Windows 10 From owner-freebsd-stable@freebsd.org Mon Apr 15 16:02:38 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1DBA41577AED for ; Mon, 15 Apr 2019 16:02:38 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4965876A0 for ; Mon, 15 Apr 2019 16:02:37 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 548CAA89F for ; Mon, 15 Apr 2019 16:02:37 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-lj1-f169.google.com with SMTP id p14so16167751ljg.5 for ; Mon, 15 Apr 2019 09:02:37 -0700 (PDT) X-Gm-Message-State: APjAAAX7RvX9RinR0k8uLPZBh7rHL+YfWAmA7MseaJfyf84r5zNskJvu mt0zrLaQeBWLN+fI2b/q3PnuS3PazLLnFSkVJ6I= X-Google-Smtp-Source: APXvYqwnk/P0trZY46U8hvY5zgfnmPwz92Ko1fj0Q4UCINVWXH1WYzcGp9SqLCP+CkN6jso12b9WoyvC3EwTpmvZ0SI= X-Received: by 2002:a2e:9194:: with SMTP id f20mr1249455ljg.10.1555344156057; Mon, 15 Apr 2019 09:02:36 -0700 (PDT) MIME-Version: 1.0 References: <5cb4a963.1c69fb81.a03c.f9af@mx.google.com> In-Reply-To: <5cb4a963.1c69fb81.a03c.f9af@mx.google.com> From: Kyle Evans Date: Mon, 15 Apr 2019 11:02:18 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Cron MAILFROM To: Software Info Cc: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: B4965876A0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 16:02:38 -0000 On Mon, Apr 15, 2019 at 10:55 AM Software Info wrote: > > Hi All > I saw an RFC here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=140304. I was just wondering if the MAILFROM feature was ever implemented in Cron. I would love to use it myself. > Hi, This would indeed be useful and support for it has not been merged; I've taken the PR. Thanks, Kyle Evans From owner-freebsd-stable@freebsd.org Mon Apr 15 17:55:08 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F26F3157A081 for ; Mon, 15 Apr 2019 17:55:07 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BF7A68BD08 for ; Mon, 15 Apr 2019 17:55:01 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x3FHsrEd010568 for ; Mon, 15 Apr 2019 19:54:53 +0200 (CEST) Received: from [217.29.46.66] (unassigned [217.29.46.66] (may be forged)) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x3FHsrvC062816 for ; Mon, 15 Apr 2019 19:54:53 +0200 (CEST) (envelope-from hausen@punkt.de) From: "Patrick M. Hausen" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o and controller resets Date: Mon, 15 Apr 2019 19:54:53 +0200 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> <26B9E28E-8BA5-41EE-9146-5336AA7605A6@punkt.de> <1A448217-C0FA-4B62-9A7F-5AF9A83D207B@punkt.de> <63F643CA-EDB1-41EB-A27E-EB427F3886B4@punkt.de> <947E0513-6449-464E-A84D-068F4E3233EF@punkt.de> To: FreeBSD-STABLE Mailing List In-Reply-To: <947E0513-6449-464E-A84D-068F4E3233EF@punkt.de> Message-Id: X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: BF7A68BD08 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.86 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.953,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[punkt.de]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[mailin.pluspunkthosting.de,mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.84)[-0.845,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-0.25)[ipnet: 217.29.32.0/20(-0.70), asn: 16188(-0.56), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Apr 2019 17:55:08 -0000 Some updates: = https://www.ixsystems.com/community/threads/nvme-problems-are-there-nightl= ies-based-on-12-stable-already.75685 https://jira.ixsystems.com/browse/NAS-101427 Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Tue Apr 16 15:12:26 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A97AC157515E for ; Tue, 16 Apr 2019 15:12:26 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-ua1-x932.google.com (mail-ua1-x932.google.com [IPv6:2607:f8b0:4864:20::932]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 87C85682E2; Tue, 16 Apr 2019 15:12:25 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-ua1-x932.google.com with SMTP id t15so6830768uao.5; Tue, 16 Apr 2019 08:12:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:cc:from:subject:date:importance :in-reply-to:references; bh=HVRkYJnYOdV7rC7jY7wHYdhO1ia6VVIHTr0Qet/+C5g=; b=Jef/3IfiQple2LEDnxpReUvoSRu+/5aYX9p8yZVznGLbGLX19bEqx5GcXFwu4E1D4p YrI0P0ZcExNrQ3t5o+DOA/NcgLHZWGF7q0wjRRlkkA+ZFPKXwlCg1NBPEZTvwmyhEn7e Gx1AcqBAUdq8G+yZr8HdFpMTMvbSDE6yzE7v1Y9RDjIAQg/gtt0jyaYk9piYD8glVLZP jke31CqeJ8bUnCw5kXXOyeyoS0DUmgUCjCfmlny9+Y/6/M5aGvm6YoWivtmK/CIitcBt vVyqQYbBgyxNilxBHGcQlXcbFuHXyDk9+sZeVQLm02veL5mHenF4eoU3dMKCNGrNb7XD HrqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:cc:from:subject:date :importance:in-reply-to:references; bh=HVRkYJnYOdV7rC7jY7wHYdhO1ia6VVIHTr0Qet/+C5g=; b=oBA01x14CN9MzsH0XdDqLLFH2pUHrjcXlUROPpVvK5XdnLWp9XWCvlTaNZ4g0ScQw6 IL1ESPwx3y5hw6jh3gCqwXtRMV44NhNPG3HRo2UoJdsTW7f6kAB7UCulkbiBvhxjOzz0 tPr4pXhE3kZoKicrbDhBZ1GauQdJSaVvxvEey3Shn6vUBVAcs0n774ZoVyms7rTCmjMV jRiQsAW9XtCY520k6HNkgeHWp4USr3Amm3fnw5bNOe9iGzOwiMZJMR4Rp3Q0/isTlvll P3/j0SylHFOJwkV7huED3+W45ItpMZ7MDn9BItw2Ko51Jd1PFEQcpuunVvY6k1ibA/i2 LkCg== X-Gm-Message-State: APjAAAXfN/9Shfe9gavgVY1MxcuNfZ+Z1eaGG+5Xk4jkFPXYJCIB+0h3 mzPObEYhPfkMljF+/y0WismkHnBo X-Google-Smtp-Source: APXvYqxMF7fwgPndDduvWotC8m/IZgbKH8/o3jJ6maeW3szUQtUiPMosJprdFQ33p+7Ci9Qo6ixHKQ== X-Received: by 2002:ab0:2b98:: with SMTP id q24mr5990428uar.122.1555427544634; Tue, 16 Apr 2019 08:12:24 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.129? (git.mayberryinv.com. [63.143.111.202]) by smtp.gmail.com with ESMTPSA id 4sm9488208uap.18.2019.04.16.08.12.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 08:12:23 -0700 (PDT) Message-ID: <5cb5f0d7.1c69fb81.2326c.85ad@mx.google.com> MIME-Version: 1.0 To: Kyle Evans Cc: "freebsd-stable@freebsd.org" From: Software Info Subject: RE: Cron MAILFROM Date: Tue, 16 Apr 2019 10:12:24 -0500 Importance: normal X-Priority: 3 In-Reply-To: References: <5cb4a963.1c69fb81.a03c.f9af@mx.google.com> X-Rspamd-Queue-Id: 87C85682E2 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Jef/3Ifi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::932 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-6.92 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.88)[-0.880,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-3.03)[ip: (-9.86), ipnet: 2607:f8b0::/32(-3.00), asn: 15169(-2.21), country: US(-0.06)]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.3.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Apr 2019 15:12:26 -0000 Thanks so much for that. Regards SI Sent from Mail for Windows 10 From: Kyle Evans Sent: Monday, April 15, 2019 11:02 AM To: Software Info Cc: freebsd-stable@freebsd.org Subject: Re: Cron MAILFROM On Mon, Apr 15, 2019 at 10:55 AM Software Info wrote: > > Hi All > I saw an RFC here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D1= 40304. I was just wondering if the MAILFROM feature was ever implemented in= Cron. I would love to use it myself. > Hi, This would indeed be useful and support for it has not been merged; I've taken the PR. Thanks, Kyle Evans From owner-freebsd-stable@freebsd.org Wed Apr 17 09:12:42 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20AB4158B17D for ; Wed, 17 Apr 2019 09:12:42 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7ECAE6F524 for ; Wed, 17 Apr 2019 09:12:41 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTP id x3H9CYTX030479 for ; Wed, 17 Apr 2019 11:12:34 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id x3H9CY0m030476 for ; Wed, 17 Apr 2019 11:12:34 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Wed, 17 Apr 2019 11:12:34 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable Subject: Re: ZFS parallel mounting gone wrong? In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 09:12:42 -0000 On Mon, 15 Apr 2019 15:24+0200, Trond Endrestøl wrote: > I upgraded a non-critical system running amd64 stable/12 to r346220. > > During multiuser boot not all ZFS filesystems below zroot/usr/local > was mounted. Some more explaining is in order: This system has two 7 year old pools that complement each other. /usr/local comes mostly from the zroot pool, but other children comes from a zdata pool. The intermediary filesystems have their canmount property set to off and mountpoints are specified at the top level only. The same goes for other parts of the filesystem hierarchy, such as /var/db and /var/spool. I just upgraded to stable/12 global r346269, local r346268. During a routine "zfs mount -av" performed in single user mode, the kernel proceeded to mount a child filesystem (enterprise_zdata/var/db/mysql) without the parent filesystems being mounted first. I rebooted back to r345628 from March 28th, and this kernel has no problems correctly mounting the ZFS filesystems in parallel. That BE used LLVM 7.0.1 from base as its system compiler. Rebooting into r346220 (April 15th) or r346269 (April 17th) clearly shows problems mounting filesystems in the correct order. These BEs was compiled using LLVM 8.0.0 from base. Maybe the system compiler is irrelevant. The name of the pools might also be a factor, the zdata pool preceedes the zroot pool in alphanumerical order. Maybe there is a bug in the code, or the code breaks when parts of the filesystem hierarchy is being built from multiple pools. Here's an attempt at explaining how this fits together: zfs list -ro name,canmount,mountpoint enterprise_zroot/usr enterprise_zdata/usr enterprise_zroot/var enterprise_zdata/var [the list has been slightly edited, moving zdata below zroot and adding an empty line] NAME CANMOUNT MOUNTPOINT enterprise_zroot/usr off /usr enterprise_zroot/usr/compat on /usr/compat enterprise_zroot/usr/local on /usr/local enterprise_zroot/usr/local/certs on /usr/local/certs enterprise_zroot/usr/local/etc on /usr/local/etc enterprise_zroot/usr/local/etc/shellkonfig3 on /usr/local/etc/shellkonfig3 enterprise_zroot/usr/local/etc/shellkonfig3/head on /usr/local/etc/shellkonfig3/head enterprise_zroot/usr/local/etc/shellkonfig3/stable-10 on /usr/local/etc/shellkonfig3/stable-10 enterprise_zroot/usr/local/etc/shellkonfig3/stable-11 on /usr/local/etc/shellkonfig3/stable-11 enterprise_zroot/usr/local/etc/shellkonfig3/stable-8 on /usr/local/etc/shellkonfig3/stable-8 enterprise_zroot/usr/local/etc/shellkonfig3/stable-9 on /usr/local/etc/shellkonfig3/stable-9 enterprise_zroot/usr/local/info on /usr/local/info enterprise_zroot/usr/local/var on /usr/local/var enterprise_zroot/usr/obj on /usr/obj enterprise_zroot/usr/ports on /usr/ports enterprise_zroot/usr/ports/distfiles on /usr/ports/distfiles enterprise_zroot/usr/ports/local off /usr/ports/local enterprise_zroot/usr/ports/packages on /usr/ports/packages enterprise_zroot/usr/ports/workdirs on /usr/ports/workdirs enterprise_zroot/usr/src on /usr/src enterprise_zdata/usr off /usr enterprise_zdata/usr/local off /usr/local enterprise_zdata/usr/local/moodledata on /usr/local/moodledata enterprise_zdata/usr/local/pgsql on /usr/local/pgsql enterprise_zdata/usr/local/restaurering on /usr/local/restaurering enterprise_zdata/usr/local/www on /usr/local/www enterprise_zdata/usr/local/www/moodle on /usr/local/www/moodle enterprise_zroot/var off /var enterprise_zroot/var/Named on /var/Named enterprise_zroot/var/account on /var/account enterprise_zroot/var/audit on /var/audit enterprise_zroot/var/cache off /var/cache enterprise_zroot/var/cache/ccache on /var/cache/ccache enterprise_zroot/var/cache/synth on /var/cache/synth enterprise_zroot/var/crash on /var/crash enterprise_zroot/var/db on /var/db enterprise_zroot/var/db/darkstat on /var/db/darkstat enterprise_zroot/var/db/dkim on /var/db/dkim enterprise_zroot/var/db/etcupdate on /var/db/etcupdate enterprise_zroot/var/db/hyperv on /var/db/hyperv enterprise_zroot/var/db/ntp on /var/db/ntp enterprise_zroot/var/db/pkg on /var/db/pkg enterprise_zroot/var/db/ports on /var/db/ports enterprise_zroot/var/db/sup on /var/db/sup enterprise_zroot/var/empty on /var/empty enterprise_zroot/var/log on /var/log enterprise_zroot/var/mail on /var/mail enterprise_zroot/var/munin on /var/munin enterprise_zroot/var/run on /var/run enterprise_zroot/var/spool on /var/spool enterprise_zroot/var/spool/cvsup on /var/spool/cvsup enterprise_zroot/var/synth on /var/synth enterprise_zroot/var/synth/builders on /var/synth/builders enterprise_zroot/var/synth/live_packages on /var/synth/live_packages enterprise_zroot/var/tmp on /var/tmp enterprise_zroot/var/unbound on /var/unbound enterprise_zdata/var off /var enterprise_zdata/var/db off /var/db enterprise_zdata/var/db/mysql on /var/db/mysql enterprise_zdata/var/db/mysql_secure on /var/db/mysql_secure enterprise_zdata/var/db/mysql_tmpdir on /var/db/mysql_tmpdir enterprise_zdata/var/db/postgres on /var/db/postgres enterprise_zdata/var/db/postgres/data11 on /var/db/postgres/data11 enterprise_zdata/var/db/postgres/data11/base on /var/db/postgres/data11/base enterprise_zdata/var/db/postgres/data11/pg_wal on /var/db/postgres/data11/pg_wal enterprise_zdata/var/db/postgres/data96 on /var/db/postgres/data96 enterprise_zdata/var/db/postgres/data96/base on /var/db/postgres/data96/base enterprise_zdata/var/db/postgres/data96/pg_xlog on /var/db/postgres/data96/pg_xlog enterprise_zdata/var/db/prometheus on /var/db/prometheus enterprise_zdata/var/db/prometheus/data on /var/db/prometheus/data enterprise_zdata/var/db/prometheus/data/wal on /var/db/prometheus/data/wal enterprise_zdata/var/spool off /var/spool enterprise_zdata/var/spool/bareos on /var/spool/bareos enterprise_zdata/var/spool/ftp on /var/spool/ftp Using this remount script in singleuser mode, brings order to chaos: #!/bin/sh # To be run while in singleuser mode, # preferably (re)booted directly to singleuser mode. PATH="/bin:/sbin:/usr/bin:/usr/sbin:/rescue" export PATH killall devd killall moused umount /usr/compat/linux/dev/fd umount /usr/compat/linux/dev umount /usr/compat/linux/proc umount /usr/compat/linux/sys zfs unmount -a for fs in `zfs list -Hro canmount,name enterprise_zroot | grep -v '^off' | grep -v 'enterprise_zroot$' | grep -v 'enterprise_zroot/ROOT' | grep -v 'enterprise_zroot/do-not-destroy' | awk '{print $2}'`; do zfs mount ${fs} done for fs in `zfs list -Hro canmount,name enterprise_zdata | grep -v '^off' | grep -v 'enterprise_zdata$' | grep -v 'enterprise_zdata/do-not-destroy' | awk '{print $2}'`; do zfs mount ${fs} done mount -al echo "You may now attempt to exit to multiuser mode ..." # EOF -- Trond. From owner-freebsd-stable@freebsd.org Wed Apr 17 09:42:42 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8793A158C056 for ; Wed, 17 Apr 2019 09:42:42 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from butcher-nb.yandex.net (unknown [IPv6:2a02:6b8:b010:d003::1:13]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "butcher-nb.yandex.net", Issuer "butcher-nb.yandex.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BF2697091F for ; Wed, 17 Apr 2019 09:42:41 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from butcher-nb.yandex.net (localhost [127.0.0.1]) by butcher-nb.yandex.net (8.15.2/8.15.2) with ESMTP id x3H9fGY9030277; Wed, 17 Apr 2019 12:41:22 +0300 (MSK) (envelope-from ae@FreeBSD.org) Subject: Re: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() To: =?UTF-8?Q?Trond_Endrest=c3=b8l?= , FreeBSD stable References: From: "Andrey V. Elsukov" Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Autocrypt: addr=ae@FreeBSD.org; prefer-encrypt=mutual; keydata= mQENBEwBF1kBCADB9sXFhBEUy8qQ4X63Y8eBatYMHGEFWN9ypS5lI3RE6qQW2EYbxNk7qUC5 21YIIS1mMFVBEfvR7J9uc7yaYgFCEb6Sce1RSO4ULN2mRKGHP3/Sl0ijZEjWHV91hY1YTHEF ZW/0GYinDf56sYpDDehaBF5wkWIo1+QK5nmj3vl0DIDCMNd7QEiWpyLVwECgLX2eOAXByT8B bCqVhJGcG6iFP7/B9Ll6uX5gb8thM9LM+ibwErDBVDGiOgvfxqidab7fdkh893IBCXa82H9N CNwnEtcgzh+BSKK5BgvPohFMgRwjti37TSxwLu63QejRGbZWSz3OK3jMOoF63tCgn7FvABEB AAG0IkFuZHJleSBWLiBFbHN1a292IDxhZUBmcmVlYnNkLm9yZz6JATsEEwECACUCGwMGCwkI BwMCBhUIAgkKCwQWAgMBAh4BAheABQJMB/ruAhkBAAoJEAHF6gQQyKF6MLwH/3Ri/TZl9uo0 SepYWXOnxL6EaDVXDA+dLb1eLKC4PRBBjX29ttQ0KaWapiE6y5/AfzOPmRtHLrHYHjd/aiHX GMLHcYRXD+5GvdkK8iMALrZ28X0JXyuuZa8rAxWIWmCbYHNSBy2unqWgTI04Erodk90IALgM 9JeHN9sFqTM6zalrMnTzlcmel4kcjT3lyYw3vOKgoYLtsLhKZSbJoVVVlvRlGBpHFJI5AoYJ SyfXoN0rcX6k9X7Isp2K50YjqxV4v78xluh1puhwZyC0p8IShPrmrp9Oy9JkMX90o6UAXdGU KfdExJuGJfUZOFBTtNIMNIAKfMTjhpRhxONIr0emxxC5AQ0ETAEXWQEIAJ2p6l9LBoqdH/0J PEFDY2t2gTvAuzz+8zs3R03dFuHcNbOwjvWCG0aOmVpAzkRa8egn5JB4sZaFUtKPYJEQ1Iu+ LUBwgvtXf4vWpzC67zs2dDuiW4LamH5p6xkTD61aHR7mCB3bg2TUjrDWn2Jt44cvoYxj3dz4 S49U1rc9ZPgD5axCNv45j72tggWlZvpefThP7xT1OlNTUqye2gAwQravXpZkl5JG4eOqJVIU X316iE3qso0iXRUtO7OseBf0PiVmk+wCahdreHOeOxK5jMhYkPKVn7z1sZiB7W2H2TojbmcK HZC22sz7Z/H36Lhg1+/RCnGzdEcjGc8oFHXHCxUAEQEAAYkBHwQYAQIACQUCTAEXWQIbDAAK CRABxeoEEMihegkYCAC3ivGYNe2taNm/4Nx5GPdzuaAJGKWksV+w9mo7dQvU+NmI2az5w8vw 98OmX7G0OV9snxMW+6cyNqBrVFTu33VVNzz9pnqNCHxGvj5dL5ltP160JV2zw2bUwJBYsgYQ WfyJJIM7l3gv5ZS3DGqaGIm9gOK1ANxfrR5PgPzvI9VxDhlr2juEVMZYAqPLEJe+SSxbwLoz BcFCNdDAyXcaAzXsx/E02YWm1hIWNRxanAe7Vlg7OL+gvLpdtrYCMg28PNqKNyrQ87LQ49O9 50IIZDOtNFeR0FGucjcLPdS9PiEqCoH7/waJxWp6ydJ+g4OYRBYNM0EmMgy1N85JJrV1mi5i Message-ID: Date: Wed, 17 Apr 2019 12:41:16 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc" X-Rspamd-Queue-Id: BF2697091F X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; ASN(0.00)[asn:13238, ipnet:2a02:6b8::/32, country:RU]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 09:42:42 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc Content-Type: multipart/mixed; boundary="zDSNDJ1vDRQQMOQzD6czJNOhKP2EyxQmF"; protected-headers="v1" From: "Andrey V. Elsukov" To: =?UTF-8?Q?Trond_Endrest=c3=b8l?= , FreeBSD stable Message-ID: Subject: Re: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() References: In-Reply-To: --zDSNDJ1vDRQQMOQzD6czJNOhKP2EyxQmF Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 15.04.2019 16:31, Trond Endrest=C3=B8l wrote: > Has anyone else witnessed a panic during reboot involving=20 > softclock_call_cc(), nd6_timer(), and nd6_dad_start()? >=20 > The stack trace goes more or less like this: >=20 > db_trace_self_wrapper() > vpanic() > panic() > trap_fatal() > trap() > calltrap() > nd6_dad_start() > nd6_timer() > softclock_call_cc() > softclock() > ithread_loop() > fork_exit() > fork_trampoline() >=20 > This was last seen while transitioning from r345628 to r346220 on=20 > amd64 stable/12. Hi, do you have exact panic message and/or backtrace from core dump? It would be good to submit PR about such problems. --=20 WBR, Andrey V. Elsukov --zDSNDJ1vDRQQMOQzD6czJNOhKP2EyxQmF-- --lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAly29LwACgkQAcXqBBDI oXraEgf+LdzH0ZILT9jNUW19NuSCLHH+TRvEMdAC5HrYODOxaNwW6rwGFpOBzjXG 3JdTwGzevnOj00aRkVaBfkt+gM49QFgXeVuRl4NxSMgG2RaXYpiB0kIpcc8Erx7R e2IBg4vpFlEqqOkB6ESKquE9cA4XR+1BqdMH05NrXoQlUi4vm2pUw8YERKhBAypb xJyU7IoALMS/4uA/fAlPXx5A7lzRuCZ+HmqjVrAMcmBkHTUfivEZkIafGAxypb5V Kg3UdVJMV8ttKFA0jBtLxwzbvaLa+h9ORUxtrsL2N1dOEXjHGfQpGOwWfrKw5GO6 BkIFfvGHfYZ7xe8CL4c8L1nfavIk0A== =4XV1 -----END PGP SIGNATURE----- --lKLtm7DNWRtYUeCc0Lw8PZEWgm4ew9agc-- From owner-freebsd-stable@freebsd.org Wed Apr 17 10:05:23 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 448B4158C62E for ; Wed, 17 Apr 2019 10:05:23 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8C83B716C2; Wed, 17 Apr 2019 10:05:22 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTP id x3HA5D61030753; Wed, 17 Apr 2019 12:05:13 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id x3HA5D7o030750; Wed, 17 Apr 2019 12:05:13 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Wed, 17 Apr 2019 12:05:13 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable cc: "Andrey V. Elsukov" Subject: Re: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 Content-ID: X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no Content-Type: text/plain; CHARSET=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 10:05:23 -0000 On Wed, 17 Apr 2019 12:41+0300, Andrey V. Elsukov wrote: > On 15.04.2019 16:31, Trond Endrestøl wrote: > > Has anyone else witnessed a panic during reboot involving > > softclock_call_cc(), nd6_timer(), and nd6_dad_start()? > > > > The stack trace goes more or less like this: > > > > db_trace_self_wrapper() > > vpanic() > > panic() > > trap_fatal() > > trap() > > calltrap() > > nd6_dad_start() > > nd6_timer() > > softclock_call_cc() > > softclock() > > ithread_loop() > > fork_exit() > > fork_trampoline() > > > > This was last seen while transitioning from r345628 to r346220 on > > amd64 stable/12. > > Hi, > > do you have exact panic message and/or backtrace from core dump? Here's another system I had to shut down recently: root@HOSTNAME:/var/crash # kgdb /boot/kernel/kernel vmcore.0 [...] Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x410 fault code = supervisor read data , page not present instruction pointer = 0x20:0xffffffff807ea33d stack pointer = 0x28:0xfffffe005ad3c8d0 frame pointer = 0x28:0xfffffe005ad3c960 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 12 (swi4: clock (0)) trap number = 12 panic: page fault cpuid = 0 time = 1555402802 KDB: stack backtrace: db_trace_self_wrapper() at 0xffffffff8054125b = db_trace_self_wrapper+0x2b/frame 0xfffffe005ad3c570 vpanic() at 0xffffffff8080aae4 = vpanic+0x1b4/frame 0xfffffe005ad3c5d0 panic() at 0xffffffff8080a923 = panic+0x43/frame 0xfffffe005ad3c630 trap_fatal() at 0xffffffff80b76244 = trap_fatal+0x394/frame 0xfffffe005ad3c690 trap_pfault() at 0xffffffff80b762a9 = trap_pfault+0x49/frame 0xfffffe005ad3c6f0 trap() at 0xffffffff80b7588f = trap+0x29f/frame 0xfffffe005ad3c800 calltrap() at 0xffffffff80b514c5 = calltrap+0x8/frame 0xfffffe005ad3c800 --- trap 0xc, rip = 0xffffffff807ea33d, rsp = 0xfffffe005ad3c8d0, rbp = 0xfffffe005ad3c960 --- __mtx_lock_sleep() at 0xffffffff807ea33d = __mtx_lock_sleep+0xbd/frame 0xfffffe005ad3c960 mld_fasttimo() at 0xffffffff80a3ae32 = mld_fasttimo+0x492/frame 0xfffffe005ad3ca50 pffasttimo() at 0xffffffff80899fa4 = pffasttimo+0x54/frame 0xfffffe005ad3ca80 softclock_call_cc() at 0xffffffff80824e0e = softclock_call_cc+0x12e/frame 0xfffffe005ad3cb30 softclock() at 0xffffffff808252f9 = softclock+0x79/frame 0xfffffe005ad3cb50 ithread_loop() at 0xffffffff807cd824 = ithread_loop+0x1d4/frame 0xfffffe005ad3cbb0 fork_exit() at 0xffffffff807ca2d3 = fork_exit+0x83/frame 0xfffffe005ad3cbf0 fork_trampoline() at 0xffffffff80b524be = fork_trampoline+0xe/frame 0xfffffe005ad3cbf0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- Uptime: 34d16h8m2s Dumping 4593 out of 12258 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% This particular system uses lagg0 comprised of bce0, bce1, em0, and em1. Also, it runs a custom kernel. > It would be good to submit PR about such problems. I'll submit the details in a PR. -- Trond. From owner-freebsd-stable@freebsd.org Wed Apr 17 10:17:20 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C4CC158CAE4 for ; Wed, 17 Apr 2019 10:17:20 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C5F9871D7B; Wed, 17 Apr 2019 10:17:19 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTP id x3HAHFEx030821; Wed, 17 Apr 2019 12:17:16 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id x3HAHFa3030818; Wed, 17 Apr 2019 12:17:15 +0200 (CEST) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Wed, 17 Apr 2019 12:17:15 +0200 (CEST) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD stable cc: "Andrey V. Elsukov" Subject: Re: Panic during reboot involving softclock_call_cc(), nd6_timer() and nd6_dad_start() In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21.9999 (BSF 287 2018-06-16) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 X-Spam-Status: No, score=-2.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.fig.ol.no Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8BIT X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 10:17:20 -0000 On Wed, 17 Apr 2019 12:05+0200, Trond Endrestøl wrote: > On Wed, 17 Apr 2019 12:41+0300, Andrey V. Elsukov wrote: > > > On 15.04.2019 16:31, Trond Endrestøl wrote: > > > Has anyone else witnessed a panic during reboot involving > > > softclock_call_cc(), nd6_timer(), and nd6_dad_start()? > > > > > > The stack trace goes more or less like this: > > > > > > db_trace_self_wrapper() > > > vpanic() > > > panic() > > > trap_fatal() > > > trap() > > > calltrap() > > > nd6_dad_start() > > > nd6_timer() > > > softclock_call_cc() > > > softclock() > > > ithread_loop() > > > fork_exit() > > > fork_trampoline() > > > > > > This was last seen while transitioning from r345628 to r346220 on > > > amd64 stable/12. > > > > Hi, > > > > do you have exact panic message and/or backtrace from core dump? > > Here's another system I had to shut down recently: > > root@HOSTNAME:/var/crash # kgdb /boot/kernel/kernel vmcore.0 > [...] > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x410 > fault code = supervisor read data , page not present > instruction pointer = 0x20:0xffffffff807ea33d > stack pointer = 0x28:0xfffffe005ad3c8d0 > frame pointer = 0x28:0xfffffe005ad3c960 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 12 (swi4: clock (0)) > trap number = 12 > panic: page fault > cpuid = 0 > time = 1555402802 > KDB: stack backtrace: > db_trace_self_wrapper() at 0xffffffff8054125b = db_trace_self_wrapper+0x2b/frame 0xfffffe005ad3c570 > vpanic() at 0xffffffff8080aae4 = vpanic+0x1b4/frame 0xfffffe005ad3c5d0 > panic() at 0xffffffff8080a923 = panic+0x43/frame 0xfffffe005ad3c630 > trap_fatal() at 0xffffffff80b76244 = trap_fatal+0x394/frame 0xfffffe005ad3c690 > trap_pfault() at 0xffffffff80b762a9 = trap_pfault+0x49/frame 0xfffffe005ad3c6f0 > trap() at 0xffffffff80b7588f = trap+0x29f/frame 0xfffffe005ad3c800 > calltrap() at 0xffffffff80b514c5 = calltrap+0x8/frame 0xfffffe005ad3c800 > --- trap 0xc, rip = 0xffffffff807ea33d, rsp = 0xfffffe005ad3c8d0, rbp = 0xfffffe005ad3c960 --- > __mtx_lock_sleep() at 0xffffffff807ea33d = __mtx_lock_sleep+0xbd/frame 0xfffffe005ad3c960 > mld_fasttimo() at 0xffffffff80a3ae32 = mld_fasttimo+0x492/frame 0xfffffe005ad3ca50 > pffasttimo() at 0xffffffff80899fa4 = pffasttimo+0x54/frame 0xfffffe005ad3ca80 > softclock_call_cc() at 0xffffffff80824e0e = softclock_call_cc+0x12e/frame 0xfffffe005ad3cb30 > softclock() at 0xffffffff808252f9 = softclock+0x79/frame 0xfffffe005ad3cb50 > ithread_loop() at 0xffffffff807cd824 = ithread_loop+0x1d4/frame 0xfffffe005ad3cbb0 > fork_exit() at 0xffffffff807ca2d3 = fork_exit+0x83/frame 0xfffffe005ad3cbf0 > fork_trampoline() at 0xffffffff80b524be = fork_trampoline+0xe/frame 0xfffffe005ad3cbf0 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > Uptime: 34d16h8m2s > Dumping 4593 out of 12258 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > This particular system uses lagg0 comprised of bce0, bce1, em0, and > em1. Also, it runs a custom kernel. > > > It would be good to submit PR about such problems. > > I'll submit the details in a PR. PR is 237329. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237329 -- Trond. From owner-freebsd-stable@freebsd.org Wed Apr 17 13:44:22 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D865156C4B3 for ; Wed, 17 Apr 2019 13:44:22 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A19A580113 for ; Wed, 17 Apr 2019 13:44:21 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: by mailman.ysv.freebsd.org (Postfix) id 61CC4156C4B2; Wed, 17 Apr 2019 13:44:21 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DCF5156C4B0 for ; Wed, 17 Apr 2019 13:44:21 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: from sonic317-32.consmr.mail.ne1.yahoo.com (sonic317-32.consmr.mail.ne1.yahoo.com [66.163.184.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E26EC80112 for ; Wed, 17 Apr 2019 13:44:19 +0000 (UTC) (envelope-from filippomore@yahoo.com) X-YMail-OSG: fiWQODgVM1nsVpb8pH9dwHV8v6.oUygWQqzn9aZtkNoB7nIorQkB.mgj5MjyWft Zrd7J6qOqaX.on4uAOQTD0gI1773y7FWGFvOKSM0NPbbYpob0iU503hAV3LNVpMOzuWteySGzJrs B5VBNNoKVzPAjlerZMJjTeszPKHKVdQeW_rVFS1iaPgVTCgqzd2UV1Y33DdfezW9AwqhurMr8OHF IuTpCrCLUM_.EZoA_ao64JRoh5JQaeANJcDWLTavUTy8KEJ3w9mWWQ1YSz42AtbN0N2Js58cg_lG mvrGPympqSutNfGO7q9tWIEP5UAEsg5Ax3DF7CoLHWUZxeGBUdfSUSEutb7N12YVqxClk3aIPuZQ f63llEpPoHW1rjxBpxrI9nyw5QnTeFwsqEKoCQpeGbRyDwnLIsVgExHPVAjnHOZrwIJydI144I2B hU.05m0Eq210mTpX9tW0z7SjF81_Zt8NsF2aInqdgLT.Rs..qv6I1_2bxM0Al3elyuzYmMCwuRlE OyNyivtQjhYcVhPEYfMiOkuvoyJHCOCTM4lnchSqA8jjIeKjqD6ZW0I011kjc6ebXpK3b6cO3gux rYuZYHy0yUKnCe5qUqtvqaXRFYhn4Q9vTtYun6JrlxrmwfW8avdtSjWEL2tv44lsk7UYGgR_0Gmu V.BLDkQd3OzMK4Fgp4LuwHOLkSIjh8gw00BPHT3YDWCBFcGqBn15FfE5sXrnfeNooYLYhaO_dQnm L8NOc1ads4rwL1Rz.a5nmh68xZXEkUa1OAh4WywNQT_IRhtCMMUXdReq1O_goYrmbjtFpmO7TD1G W_2OFn3PbpaAjfXQqMMaAnvi5n8n0QdOCB1RliwUMi7eBTuT8J2EFaUtxMVtxJIbCfuRTtZQ9R6T ScjLabFeYkRuC3TWLA2IcgbBewIj49a1zZ6ZX75khGCiWlzp69f9uLx38sN5kkfXtgRuVy0hxRKd VTeuJebodwbtsYclTyGR1EN0xxlZtTE8lzJ5i2B.GyF2ySST2CX1YHikB9t4di9H4Ewb_5VckDE. nqnfjx.nAjNJTmT12uxgIVIbmFrJX9aikHP4fRU6v4MH3WlLUbylSHAV.IJSUgsNWTDUT.zhZLTR j Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.ne1.yahoo.com with HTTP; Wed, 17 Apr 2019 13:44:13 +0000 Date: Wed, 17 Apr 2019 13:44:07 +0000 (UTC) From: Filippo Moretti To: FreeBSD Stable ML Message-ID: <468416908.868069.1555508647970@mail.yahoo.com> Subject: Problem with STABLE-12 MIME-Version: 1.0 References: <468416908.868069.1555508647970.ref@mail.yahoo.com> X-Mailer: WebService/1.1.13490 YMailNorrin Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0 X-Rspamd-Queue-Id: E26EC80112 X-Spamd-Bar: + X-Spamd-Result: default: False [1.74 / 15.00]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.94)[0.936,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(0.75)[ip: (1.26), ipnet: 66.163.184.0/21(1.43), asn: 36646(1.14), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.67)[0.665,0]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_LONG(0.39)[0.393,0]; RCVD_IN_DNSWL_NONE(0.00)[43.184.163.66.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 13:44:22 -0000 I previously reported that I was unable to boot my system with STABLE-12 of april 13 the system stop at Loading kernel modules.I installed stable-12 snapshot of april 11 on a second disc to try to rescue the first one.I could go on with my plan until I installed drm kmodand addedd kld_list=/boot/modules/radeonkms.ko in /etc/rc.conf now also this second disk no longer boots,it stops in Loading kernel modules.Further how can I mount ada0p3 partition rw on this second disc considering that bot are zfs?(I could not figure this out mysel)I plan to reinstall on the second disksincerelyFilippo From owner-freebsd-stable@freebsd.org Wed Apr 17 14:07:50 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DC5C156CEC0 for ; Wed, 17 Apr 2019 14:07:50 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D80A580BDD for ; Wed, 17 Apr 2019 14:07:49 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: by mailman.ysv.freebsd.org (Postfix) id 94FB0156CEBF; Wed, 17 Apr 2019 14:07:49 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80F08156CEBE for ; Wed, 17 Apr 2019 14:07:49 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.49.70]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "gromit.dlib.vt.edu", Issuer "Chumby Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E82D80BDC for ; Wed, 17 Apr 2019 14:07:49 +0000 (UTC) (envelope-from paul@gromit.dlib.vt.edu) Received: from mather.gromit23.net (c-98-244-101-97.hsd1.va.comcast.net [98.244.101.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id 5212C3C6; Wed, 17 Apr 2019 10:07:42 -0400 (EDT) Content-Type: text/plain; charset=us-ascii; delsp=yes; format=flowed Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: Problem with STABLE-12 From: Paul Mather In-Reply-To: <468416908.868069.1555508647970@mail.yahoo.com> Date: Wed, 17 Apr 2019 10:07:41 -0400 Cc: stable@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: References: <468416908.868069.1555508647970.ref@mail.yahoo.com> <468416908.868069.1555508647970@mail.yahoo.com> To: Filippo Moretti X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 1E82D80BDC X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.94 / 15.00]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.94)[-0.937,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 14:07:50 -0000 On Apr 17, 2019, at 9:44 AM, Filippo Moretti via freebsd-stable wrote: > I previously reported that I was unable to boot my system with STABLE-12 > of april 13 the system stop at Loading kernel modules.I installed > stable-12 snapshot of april 11 on a second disc to try to rescue the > first one.I could go on with my plan until I installed drm kmodand addedd > kld_list=/boot/modules/radeonkms.ko in /etc/rc.conf I have a radeon graphics card, too. Recently, I had a problem with the graphics/drm-kmod hanging on boot in multi-user. My fix was to switch to graphics/drm-legacy-kmod, which at least lets the system boot again. I figure the newer graphics/drm-kmod no longer supports my old radeon card. > now also this second disk no longer boots,it stops in Loading kernel > modules.Further how can I mount ada0p3 partition rw on this second disc > considering that bot are zfs?(I could not figure this out mysel)I plan to > reinstall on the second disksincerelyFilippo The easiest way to be able to edit the configuration of the existing system to fix things is to boot it in single user mode (press "2" from the loader menu). When you get to the single user prompt, you can then mount your ZFS file systems for writing as follows: mount -uw / /etc/rc.d/zfsbe start /etc/rc.d/zfs start You can then, e.g., edit /etc/rc.conf and remove the problematic kld_list entry. Cheers, Paul. From owner-freebsd-stable@freebsd.org Wed Apr 17 19:51:37 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 705B215777A9 for ; Wed, 17 Apr 2019 19:51:37 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yw1-f67.google.com (mail-yw1-f67.google.com [209.85.161.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F45E908B3 for ; Wed, 17 Apr 2019 19:51:36 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yw1-f67.google.com with SMTP id l15so9025818ywe.8 for ; Wed, 17 Apr 2019 12:51:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ilnpqffpf96Z6vE10Aq8gbtRWw7o882AABqo+go0pNE=; b=F7sVE+zZlwxluQ0TlcPb7WPSlu8S0CyfV46Nm39QNQ3CYd8dTj2OGl/Yeg8SQYe6BJ /y4yS2gIzeddDX5qi/k/eQuuuEd+X9jn0mfDyUh9IXqi11IrnirlkItwqPN+PIkPaIMG Bgh6YVkgQ5yw88lKBBI7Xr7olDdkLqWjse3+iDUUBo/RCFMEXy20tCa8nwKZSb7c6hET q6MKcYTG4kccE5BBwJKC38XgWu6sPNWvOGHqdf4UNtMwK0fmG0wM1eakk1Vx+W+KTK3r 4M05DOunsFsrj74tRu741VDRxMvZF70fyNi1XsveDEs+MA/+mBi/Mk3p/KkB6qpoUdmk pn6Q== X-Gm-Message-State: APjAAAUQNOdzfp3MuRvcjMrIIPYdwzBUXurAYfTd6/Adu8BhsLKBPsev JHudxY22qqg6sMc1sFjm2fRh4MAbm6YVpFF9lWc= X-Google-Smtp-Source: APXvYqw9O1CUUv/9QiVlexbNhY5JnNMYQR5Wky68QcH/2G3sdzCnW1L3uob+oz6lntHt9Fa9GL74are+kqZWbRR4Bns= X-Received: by 2002:a0d:e585:: with SMTP id o127mr72369387ywe.364.1555529131627; Wed, 17 Apr 2019 12:25:31 -0700 (PDT) MIME-Version: 1.0 From: Li-Wen Hsu Date: Thu, 18 Apr 2019 04:25:19 +0900 Message-ID: Subject: FreeBSD CI Weekly Report 2019-04-14 To: freebsd-testing@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4F45E908B3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of lwhsufreebsd@gmail.com designates 209.85.161.67 as permitted sender) smtp.mailfrom=lwhsufreebsd@gmail.com X-Spamd-Result: default: False [-3.72 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[67.161.85.209.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.48)[-0.483,0]; DMARC_NA(0.00)[freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[67.161.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.23)[ipnet: 209.85.128.0/17(-3.88), asn: 15169(-2.22), country: US(-0.06)]; FORGED_SENDER(0.30)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FORGED_RECIPIENTS(0.00)[freebsd-testing@freebsd.org,freebsd-stable@freebsd.org]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Mailman-Approved-At: Wed, 17 Apr 2019 20:05:29 +0000 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 19:51:37 -0000 (bcc -current and -stable for more audience) FreeBSD CI Weekly Report 2019-04-14 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2019-04-08 to 2019-04-14. During this period, we have: * 1702 builds (95.7% passed, 4.3% failed) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 274 test runs (47.1% passed, 45.6% unstable, 7.3% exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 13 doc buils (100% passed) (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/s/r1EE3jotE and archive is available at http://hackfoldr.org/freebsd-ci-report/, any help is welcome. ## Fixed tests * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ * sys.geom.class.eli.online_resize_test.online_resize (at clean up stage) Fixed in https://svnweb.freebsd.org/changeset/base/346057 * https://ci.freebsd.org/job/FreeBSD-head-riscv64-test/ * Because Python default version switched 3, fixed in the test code. ## Failing Tests * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ * sys.opencrypto.runtests.main * sys.kern.coredump_phnum_test.coredump_phnum WIP: https://reviews.freebsd.org/D18495 * lib.libc.sys.sendfile_test.fd_positive_shm_v4 * lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v4 * lib.libc.gen.floatunditf_test.floatunditf * lib.libc.stdio.printfloat_test.hexadecimal_rounding * lib.msun.ctrig_test.test_small_inputs * lib.msun.precision_test.t_precision https://bugs.freebsd.org/236936 (fixed when this report published) * https://ci.freebsd.org/job/FreeBSD-stable-12-i386-test/ * sys.netmap.ctrl-api-test.main * sys.opencrypto.runtests.main * lib.libc.regex.exhaust_test.regcomp_too_big * lib.libregex.exhaust_test.regcomp_too_big * sys.kern.coredump_phnum_test.coredump_phnum WIP: https://reviews.freebsd.org/D18495 * https://ci.freebsd.org/job/FreeBSD-stable-11-amd64-test/ * usr.bin.procstat.procstat_test.kernel_stacks * https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/ * sys.netmap.ctrl-api-test.main * sys.opencrypto.runtests.main * usr.bin.procstat.procstat_test.kernel_stacks * local.kyua.* (31 cases) * local.lutok.* (3 cases) * lib.libc.sys.sendfile_test.fd_positive_shm_v4 * lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v4 ## Failing Tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * common.ip.t_dtrace_contrib.tst_ipv4localsctp_ksh * common.ip.t_dtrace_contrib.tst_localsctpstate_ksh * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ There are ~60 failing cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details ## Disabled Tests * lib.libc.sys.mmap_test.mmap_truncate_signal https://bugs.freebsd.org/211924 * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * usr.bin.procstat.procstat_test.command_line_arguments https://bugs.freebsd.org/233587 * usr.bin.procstat.procstat_test.environment https://bugs.freebsd.org/233588 ## Closed Issues * https://bugs.freebsd.org/237128 sys/geom/class/eli:online_resize_test fails to clean up cleanly, causing false positives * https://bugs.freebsd.org/237129 sys.netmap.ctrl-api-test.main fails on ^/stable/11 and ^/stable/12 i386 because the kernel lacks netmap support ## Oepn Issues * https://bugs.freebsd.org/237077 possible race in build: /usr/src/sys/amd64/linux/linux_support.s:38:2: error: expected relocatable expression ### In progress * https://bugs.freebsd.org/236936 4 test cases failing on i386 after r345562 ### Cause build fails * [233735: Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory](https://bugs.freebsd.org/233735) * [233769: Possible build race: ld: error: unable to find library -lgcc_s](https://bugs.freebsd.org/233769) ### Others [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-stable@freebsd.org Thu Apr 18 12:02:05 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87103156EE90 for ; Thu, 18 Apr 2019 12:02:05 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [31.24.6.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 284878ACBF for ; Thu, 18 Apr 2019 12:02:04 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6]) by constantine.ingresso.co.uk with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1hH5jn-000KO2-W4 for freebsd-stable@freebsd.org; Thu, 18 Apr 2019 12:01:55 +0000 Subject: Re: Problem with STABLE-12 To: freebsd-stable@freebsd.org References: <468416908.868069.1555508647970.ref@mail.yahoo.com> <468416908.868069.1555508647970@mail.yahoo.com> From: Pete French Message-ID: Date: Thu, 18 Apr 2019 13:01:55 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 284878ACBF X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.995,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 12:02:05 -0000 > I have a radeon graphics card, too.  Recently, I had a problem with the > graphics/drm-kmod hanging on boot in multi-user.  My fix was to switch > to graphics/drm-legacy-kmod, which at least lets the system boot again. > I figure the newer graphics/drm-kmod no longer supports my old radeon card. So, I was about to post about this, and then saw this thread. I too ave had to go back to the legacy-kmod because something broke it in the last couple of weeks. Was using the new one fine until then. Unfortunately I didnt know it was broken as have been working remotely and thus not in front of the machine. But I doubt that the Radeon stopped being supported soehow - that would be mentioned somewhere surely ? It looks like a hang on loading the module to me. -pete. From owner-freebsd-stable@freebsd.org Thu Apr 18 21:05:05 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66512157C6D2 for ; Thu, 18 Apr 2019 21:05:05 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [IPv6:2001:19f0:300:2185:123::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D250D810A6 for ; Thu, 18 Apr 2019 21:05:04 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [104.207.135.49]) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5) with ESMTP id x3IL514t095013; Thu, 18 Apr 2019 22:05:01 +0100 (BST) (envelope-from jamie@donotpassgo.dyslexicfish.net) Received: (from jamie@localhost) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5/Submit) id x3IL50FW095012; Thu, 18 Apr 2019 22:05:00 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <201904182105.x3IL50FW095012@donotpassgo.dyslexicfish.net> Date: Thu, 18 Apr 2019 22:05:00 +0100 Organization: Dyslexic Fish To: peter@holm.cc, mckusick@mckusick.com Cc: jamie@catflap.org, jamie@catflap.dyslexicfish.net, imp@bsdimp.com, freebsd-stable@freebsd.org Subject: Re: Replicable file-system corruption due to fsck/ufs References: <201904131341.x3DDfjWf090153@chez.mckusick.com> In-Reply-To: <201904131341.x3DDfjWf090153@chez.mckusick.com> User-Agent: Heirloom mailx 12.4 7/29/08 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (donotpassgo.dyslexicfish.net [104.207.135.49]); Thu, 18 Apr 2019 22:05:02 +0100 (BST) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2019 21:05:05 -0000 Kirk McKusick wrote: > > From: Peter Holm > > > > On Fri, Apr 12, 2019 at 04:13:00PM -0700, Kirk McKusick wrote: > > > >> This is indeed a bug in the calculation of the location of the last > >> block of a file. I believe that the following patch to head will > >> fix it. > >> > >> Peter, can you please test and let me know. > >> > >> If Peter confirms that it fixes the bug, I will check it into head > >> and MFC it to 12-stable and 11-stable after a 2-week settle-in time. > >> > >> Kirk McKusick > > > > Yes, this patch works for me. > > > > -- > > Peter > > Great, thanks for the quick test. Now committed to head as -r346185. Apologies for the delay in replying. Thanks Kirk for fixing this, and thanks Peter for testing it. The patch also applied cleanly to 12-stable, and is working there too. Cheers, Jamie From owner-freebsd-stable@freebsd.org Fri Apr 19 08:54:04 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B68F41589EDB for ; Fri, 19 Apr 2019 08:54:04 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from echo.brtsvcs.net (echo.brtsvcs.net [IPv6:2607:f740:c::4ae]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 90DA36D770 for ; Fri, 19 Apr 2019 08:54:03 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from chombo.houseloki.net (chombo [IPv6:2601:1c2:1402:1770:ae1f:6bff:fe6b:9e1c]) by echo.brtsvcs.net (Postfix) with ESMTPS id CF53238D14 for ; Fri, 19 Apr 2019 01:53:55 -0700 (PDT) Received: from [10.26.25.100] (unknown [10.26.25.100]) by chombo.houseloki.net (Postfix) with ESMTPSA id 6AFEB5C54 for ; Fri, 19 Apr 2019 01:53:55 -0700 (PDT) Subject: Re: Can someone MFC usb/234503 From: Mel Pilgrim To: freebsd-stable@freebsd.org References: <748ac1e6-53f0-53fa-c9e6-f19110dd90eb@bluerosetech.com> Message-ID: <4c18879a-624c-5d16-bddc-696b235b7f91@bluerosetech.com> Date: Fri, 19 Apr 2019 01:53:53 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <748ac1e6-53f0-53fa-c9e6-f19110dd90eb@bluerosetech.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 90DA36D770 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of list_freebsd@bluerosetech.com designates 2607:f740:c::4ae as permitted sender) smtp.mailfrom=list_freebsd@bluerosetech.com X-Spamd-Result: default: False [-6.80 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[bluerosetech.com]; MX_GOOD(-0.01)[echo.brtsvcs.net,foxtrot.brtsvcs.net]; NEURAL_HAM_SHORT(-0.99)[-0.992,0]; IP_SCORE(-3.49)[ip: (-8.99), ipnet: 2607:f740:c::/48(-4.52), asn: 36236(-3.89), country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:36236, ipnet:2607:f740:c::/48, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 08:54:04 -0000 This is still waiting for MFC to stable/11 and stable/12. Would someone please have a look at it? Warner Losh has timed out. From owner-freebsd-stable@freebsd.org Fri Apr 19 11:46:04 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ACCA0158D71D for ; Fri, 19 Apr 2019 11:46:04 +0000 (UTC) (envelope-from kris@ixsystems.com) Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 302017238E for ; Fri, 19 Apr 2019 11:46:03 +0000 (UTC) (envelope-from kris@ixsystems.com) Received: by mail-oi1-x233.google.com with SMTP id y84so3822363oia.12 for ; Fri, 19 Apr 2019 04:46:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:mime-version:thread-index :content-language; bh=3ZC3zNn7rUCmh76RtR2/O14MNo/SI99vYZ1RtVIRCAM=; b=AgYSi96aZ5JKwf3VhKNkh/bupmOkJLICLBpkM3xXfifYUSWe5KeC1Ms0tdQrCAVcYd am1ZzDPOhiWonfdXIe3JxvcFcXkafAanEGy/muzrCAR7fhG8EWcPygIbLIAAnj8/zTwH u0pZ2smCLEhlUPg5+CMUgp3GPbLneSjBoqqa2upYDxb6Qx7aEMe3beEKl+mNS98HdHYS DPM2L2W7iq30N5rTxEvAuk7gu3ZEzk1OorEIny6BuMEnYGbvJ+fxnL9XbBitxfScGUgX FIDhk9hcENl7KDIRzR0g7Jao6wlvtuJn0GjP25rHxFj2F1SuQSpZiuM9bj6i1cEmGqOM 1yqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :thread-index:content-language; bh=3ZC3zNn7rUCmh76RtR2/O14MNo/SI99vYZ1RtVIRCAM=; b=I0a8qDOwS1eU6tucB2t4tVHZSKT42RC3W9jEqR+52tjDHwyD4q3Eo8WUNK87GILFaY qM89uM//nuALwS8yaN7vE8XZR719w+FD4hpqkrBXcVVD5RsHNP/owTw8zWOVare9zTZW gXLauBqnR3XiU59ukz3EfMl3v1XDCwaLjBqCDEt3u7nkzc4PAIQc2e2qI6knU+pHc5HS AGOn7ODIobzWBw5yA+Hdd14xZfOyenSeEp4mBl6sQWiX8aKB68v83kdJ58pxKurcRXyM cNmLyHzOkfqrbcHyGthqZvKvqufL/BZRz04fo9UXZKpmCm3vMxJ3nHP0Iag4+ozKasSP uyOw== X-Gm-Message-State: APjAAAVTkLZ1/GEPfymOsrtI76OTHpSobURtyO109DwVAmDG3mn+FnbW lOdm0fLmnk9FS62ta2WpC7xX5g== X-Google-Smtp-Source: APXvYqx7i3tduqp1yo+vZJ7qEjVUleH3uPWhYNuDq63xen+K/8hgAlMptuvihVPdZqJV49Endby1vA== X-Received: by 2002:aca:5241:: with SMTP id g62mr1669470oib.103.1555674361499; Fri, 19 Apr 2019 04:46:01 -0700 (PDT) Received: from KrisWindows (71-136-150-27.lightspeed.knvltn.sbcglobal.net. [71.136.150.27]) by smtp.gmail.com with ESMTPSA id g21sm1832371otk.12.2019.04.19.04.46.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 19 Apr 2019 04:46:00 -0700 (PDT) From: To: , , Subject: CFT for FreeBSD + ZoL Date: Fri, 19 Apr 2019 07:46:00 -0400 Message-ID: <2cb101d4f6a5$763ad1b0$62b07510$@ixsystems.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: AdT2pOuihq3FlN3IRvmhc3q+vkx7Ug== Content-Language: en-us X-Rspamd-Queue-Id: 302017238E X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ixsystems-com.20150623.gappssmtp.com header.s=20150623 header.b=AgYSi96a; dmarc=pass (policy=none) header.from=ixsystems.com; spf=pass (mx1.freebsd.org: domain of kris@ixsystems.com designates 2607:f8b0:4864:20::233 as permitted sender) smtp.mailfrom=kris@ixsystems.com X-Spamd-Result: default: False [-5.86 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[3]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: ALT3.ASPMX.L.GOOGLE.com]; DKIM_TRACE(0.00)[ixsystems-com.20150623.gappssmtp.com:+]; DMARC_POLICY_ALLOW(-0.50)[ixsystems.com,none]; NEURAL_HAM_SHORT(-0.96)[-0.956,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[ixsystems-com.20150623.gappssmtp.com:s=20150623]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NO_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.3.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.90)[ip: (-9.08), ipnet: 2607:f8b0::/32(-3.09), asn: 15169(-2.24), country: US(-0.06)] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 11:46:04 -0000 FreeBSD Developers, We're pleased to make available images allowing testing of FreeBSD using ZFS on Linux. During this development cycle, the ZoL code has been made portable, and available in the ports tree as sysutils/zol and sysutils/zol-kmod, for userland/kernel bits respectively. While some have used these for testing, we felt it necessary to generate some installation images which are an easier method of getting up and started using ZoL. These images are built against FreeBSD 12-stable and 13-HEAD and will install a world / kernel with the base system ZFS disabled and the sysutils/zol ports pre-installed. It is possible to these with both UFS or ZFS on root, and we're looking for feedback on any stability issues or other regressions that you see vs the legacy ZFS in base. FreeBSD 12-Stable https://pkg.trueos.org/iso/freebsd12-zol/ FreeBSD HEAD https://pkg.trueos.org/iso/freebsd13-zol/ Please report issues on our GitHub tracker at: https://github.com/zfsonfreebsd/ZoF Thanks and happy testing! -- Kris Moore Vice President of Engineering iXsystems, Inc Ph: (408) 943-4100 Ph: (408) 943-4101 The Groundbreaking TrueNAS M-Series - Enterprise Storage & Servers Driven By Open Source From owner-freebsd-stable@freebsd.org Fri Apr 19 20:32:51 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03534157844F for ; Fri, 19 Apr 2019 20:32:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E83D06AB74 for ; Fri, 19 Apr 2019 20:32:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82a.google.com with SMTP id h39so6578878qte.2 for ; Fri, 19 Apr 2019 13:32:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=DeagXxt5s3AORaZ+7QyngDC6et6WkhZRUQvh3qNX+eM=; b=P7TFSgb+uoC3iVn6zNs+aNLfo7BtLaiqMnzrRK57dIlIY6/dwwMTk0CLU+v15j0VCC ekVhQbniC6alwRL/sMypcmD9qdK+XtXs2hW5n/7/z0IDjagWSvORZeT28JiJkJawKlx2 EHTXKAcJF/KTcQx1mga59q2Dsq7lq/V4eyvNnVnaa/tj9MBOI6Ro88EiCOcWvlHj3ra3 V8teh+1cU0KUjE80964hfc/SfQIOc5Inykqnz/p1JTLvbGnRYWSeNKz2lYGG1Kab0sX/ t37UHd3n4JHWQYrAiTQlwUqj3wktCjGdX1Z3AZ3VgmSDqLkfT8uqmTAPWXmp0GYzBWHi DRHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=DeagXxt5s3AORaZ+7QyngDC6et6WkhZRUQvh3qNX+eM=; b=YvBjsuEyoLtTZRdQVplmdgzAC0rNPkKIrJFbqJUDA1WJSVr0trA5ZhNIwo6oI7/H/X VN81Fxk9l/8lqKayJYL4cGjqYhqGW+6SwiD2RbZO3l9c5lThDvdlRRVl98rqCal6dx+c yL1wmO2TLkC1jtSd6NBmvJchbop4SwlB2HP+rLwS2kMhl4MODW/94H9bawLxnbleoxnb GCw02BhAd3JlTIpN4yOGY/ARO4YI/g6ihzqh+1NZmjWjei0OYgn+JzWbyU0/J0yOax8k VW+ZyhOfemRZkQhCVSJsOUYcOWl2/zEeVsD2UZ1tU6dQxEBQ7kHFRbsTWV18d6um/TG5 wR9Q== X-Gm-Message-State: APjAAAUZxRHdp4iPBx4i6Vu3Abk56fy5FKX+7uCZ0srNqBOtJrZiaVad qil/L7I7TjhP0csgmHxrszlHNLJvOD9h3zd0oF+CCc1P X-Google-Smtp-Source: APXvYqwd3DEdefyrHbVKPSwOotflIzUBA17/26rrm3LqcKTPufP2ANaEKlhB/ejPAsqb5qbV4K8uoZbF4NFEynz1ieE= X-Received: by 2002:ac8:28d0:: with SMTP id j16mr5125544qtj.15.1555705969415; Fri, 19 Apr 2019 13:32:49 -0700 (PDT) MIME-Version: 1.0 References: <748ac1e6-53f0-53fa-c9e6-f19110dd90eb@bluerosetech.com> <4c18879a-624c-5d16-bddc-696b235b7f91@bluerosetech.com> In-Reply-To: <4c18879a-624c-5d16-bddc-696b235b7f91@bluerosetech.com> From: Warner Losh Date: Fri, 19 Apr 2019 14:32:38 -0600 Message-ID: Subject: Re: Can someone MFC usb/234503 To: Mel Pilgrim Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: E83D06AB74 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=P7TFSgb+ X-Spamd-Result: default: False [-5.81 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.86)[-0.862,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[a.2.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-2.94)[ip: (-9.25), ipnet: 2607:f8b0::/32(-3.12), asn: 15169(-2.26), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Apr 2019 20:32:51 -0000 On Fri, Apr 19, 2019 at 2:55 AM Mel Pilgrim wrote: > This is still waiting for MFC to stable/11 and stable/12. Would someone > please have a look at it? Warner Losh has timed out. > Just merged it. Warner From owner-freebsd-stable@freebsd.org Sat Apr 20 12:00:38 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44198158BE35 for ; Sat, 20 Apr 2019 12:00:38 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [31.24.6.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0B078E960 for ; Sat, 20 Apr 2019 12:00:36 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from [82.47.240.30] (helo=foula.local) by constantine.ingresso.co.uk with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1hHofU-000LPo-Vb for freebsd-stable@freebsd.org; Sat, 20 Apr 2019 12:00:29 +0000 Subject: Re: CFT for FreeBSD + ZoL To: freebsd-stable@freebsd.org References: <2cb101d4f6a5$763ad1b0$62b07510$@ixsystems.com> From: Pete French Message-ID: Date: Sat, 20 Apr 2019 13:00:29 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:67.0) Gecko/20100101 Thunderbird/67.0 MIME-Version: 1.0 In-Reply-To: <2cb101d4f6a5$763ad1b0$62b07510$@ixsystems.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: A0B078E960 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=ingresso.co.uk; spf=pass (mx1.freebsd.org: domain of petefrench@ingresso.co.uk designates 31.24.6.74 as permitted sender) smtp.mailfrom=petefrench@ingresso.co.uk X-Spamd-Result: default: False [-3.84 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:31.24.6.74]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.16)[asn: 16082(-0.70), country: GB(-0.09)]; MX_GOOD(-0.01)[us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mimecast.com, us-smtp-inbound-1.mimecast.com, us-smtp-inbound-2.mi mecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com,us-smtp-inbound-1.mimecast.com,us-smtp-inbound-2.mimecast.com]; DMARC_POLICY_ALLOW(-0.50)[ingresso.co.uk,none]; NEURAL_HAM_SHORT(-0.87)[-0.874,0]; RECEIVED_SPAMHAUS_PBL(0.00)[30.240.47.82.zen.spamhaus.org : 127.0.0.11]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16082, ipnet:31.24.0.0/21, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 12:00:38 -0000 On 19/04/2019 12:46, kris@ixsystems.com wrote: > FreeBSD Developers, > > > > We're pleased to make available images allowing testing of FreeBSD using ZFS > on Linux. During this development cycle, the ZoL code has been made > portable, and available in the ports tree as sysutils/zol and > sysutils/zol-kmod, for userland/kernel bits respectively. While some have > used these for testing, we felt it necessary to generate some installation > images which are an easier method of getting up and started using ZoL. These > images are built against FreeBSD 12-stable and 13-HEAD and will install a > world / kernel with the base system ZFS disabled and the sysutils/zol ports > pre-installed. Ah, this is excellent, thankyou for all the work on this. A question though - is the intnet to keep these as ports, or will the ZoL code be merged back into the base, replacing the existing ZFS implementation? cheers, -pete. [who will give this a test next week if he can] From owner-freebsd-stable@freebsd.org Sat Apr 20 14:39:29 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88047158F347 for ; Sat, 20 Apr 2019 14:39:29 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 521C4931B6 for ; Sat, 20 Apr 2019 14:39:28 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 7B70421109F for ; Sat, 20 Apr 2019 10:39:21 -0400 (EDT) Received: from [192.168.10.17] (D7.Denninger.Net [192.168.10.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id E2CFAC64CE for ; Sat, 20 Apr 2019 09:39:20 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> Date: Sat, 20 Apr 2019 09:39:21 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms070906010904010606030101" X-Rspamd-Queue-Id: 521C4931B6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.59 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[px.denninger.net]; NEURAL_HAM_SHORT(-0.92)[-0.921,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.46)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.97), asn: 14061(1.60), country: US(-0.06)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 14:39:29 -0000 This is a cryptographically signed message in MIME format. --------------ms070906010904010606030101 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/13/2019 06:00, Karl Denninger wrote: > On 4/11/2019 13:57, Karl Denninger wrote: >> On 4/11/2019 13:52, Zaphod Beeblebrox wrote: >>> On Wed, Apr 10, 2019 at 10:41 AM Karl Denninger = wrote: >>> >>> >>>> In this specific case the adapter in question is... >>>> >>>> mps0: port 0xc000-0xc0ff mem >>>> 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on = pci3 >>>> mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd >>>> mps0: IOCCapabilities: >>>> 1285c >>>> >>>> Which is indeed a "dumb" HBA (in IT mode), and Zeephod says he conne= cts >>>> his drives via dumb on-MoBo direct SATA connections. >>>> >>> Maybe I'm in good company. My current setup has 8 of the disks conne= cted >>> to: >>> >>> mps0: port 0xb000-0xb0ff mem >>> 0xfe240000-0xfe24ffff,0xfe200000-0xfe23ffff irq 32 at device 0.0 on p= ci6 >>> mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd >>> mps0: IOCCapabilities: >>> 5a85c >>> >>> ... just with a cable that breaks out each of the 2 connectors into 4= >>> SATA-style connectors, and the other 8 disks (plus boot disks and SSD= >>> cache/log) connected to ports on... >>> >>> - ahci0: port >>> 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f= mem >>> 0xfe900000-0xfe9001ff irq 44 at device 0.0 on pci2 >>> - ahci2: port >>> 0xa050-0xa057,0xa040-0xa043,0xa030-0xa037,0xa020-0xa023,0xa000-0xa01f= mem >>> 0xfe610000-0xfe6107ff irq 40 at device 0.0 on pci7 >>> - ahci3: port >>> 0xf040-0xf047,0xf030-0xf033,0xf020-0xf027,0xf010-0xf013,0xf000-0xf00f= mem >>> 0xfea07000-0xfea073ff irq 19 at device 17.0 on pci0 >>> >>> ... each drive connected to a single port. >>> >>> I can actually reproduce this at will. Because I have 16 drives, whe= n one >>> fails, I need to find it. I pull the sata cable for a drive, determi= ne if >>> it's the drive in question, if not, reconnect, "ONLINE" it and wait f= or >>> resilver to stop... usually only a minute or two. >>> >>> ... if I do this 4 to 6 odd times to find a drive (I can tell, in gen= eral, >>> that a drive is part of the SAS controller or the SATA controllers...= so >>> I'm only looking among 8, ever) ... then I "REPLACE" the problem driv= e. >>> More often than not, the a scrub will find a few problems. In fact, = it >>> appears that the most recent scrub is an example: >>> >>> [1:7:306]dgilbert@vr:~> zpool status >>> pool: vr1 >>> state: ONLINE >>> scan: scrub repaired 32K in 47h16m with 0 errors on Mon Apr 1 23:1= 2:03 >>> 2019 >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> vr1 ONLINE 0 0 0 >>> raidz2-0 ONLINE 0 0 0 >>> gpt/v1-d0 ONLINE 0 0 0 >>> gpt/v1-d1 ONLINE 0 0 0 >>> gpt/v1-d2 ONLINE 0 0 0 >>> gpt/v1-d3 ONLINE 0 0 0 >>> gpt/v1-d4 ONLINE 0 0 0 >>> gpt/v1-d5 ONLINE 0 0 0 >>> gpt/v1-d6 ONLINE 0 0 0 >>> gpt/v1-d7 ONLINE 0 0 0 >>> raidz2-2 ONLINE 0 0 0 >>> gpt/v1-e0c ONLINE 0 0 0 >>> gpt/v1-e1b ONLINE 0 0 0 >>> gpt/v1-e2b ONLINE 0 0 0 >>> gpt/v1-e3b ONLINE 0 0 0 >>> gpt/v1-e4b ONLINE 0 0 0 >>> gpt/v1-e5a ONLINE 0 0 0 >>> gpt/v1-e6a ONLINE 0 0 0 >>> gpt/v1-e7c ONLINE 0 0 0 >>> logs >>> gpt/vr1log ONLINE 0 0 0 >>> cache >>> gpt/vr1cache ONLINE 0 0 0 >>> >>> errors: No known data errors >>> >>> ... it doesn't say it now, but there were 5 CKSUM errors on one of th= e >>> drives that I had trial-removed (and not on the one replaced). >>> _______________________________________________ >> That is EXACTLY what I'm seeing; the "OFFLINE'd" drive is the one that= , >> after a scrub, comes up with the checksum errors.=C2=A0 It does *not* = flag >> any errors during the resilver and the drives *not* taken offline do n= ot >> (ever) show checksum errors either. >> >> Interestingly enough you have 19.00.00.00 firmware on your card as wel= l >> -- which is what was on mine. >> >> I have flashed my card forward to 20.00.07.00 -- we'll see if it still= >> does it when I do the next swap of the backup set. > Verrrrrryyyyy interesting. > > This drive was last written/read under 19.00.00.00.=C2=A0 Yesterday I s= wapped > it back in.=C2=A0 Note that right now I am running: > > mps0: port 0xc000-0xc0ff mem > 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci= 3 > mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd > mps0: IOCCapabilities: > 1285c > > And, after the scrub completed overnight.... > > [karl@NewFS ~]$ zpool status backup > =C2=A0 pool: backup > =C2=A0state: DEGRADED > status: One or more devices has experienced an unrecoverable error.=C2=A0= An > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to correct = the error.=C2=A0 Applications are unaffected. > action: Determine if the device needs to be replaced, and clear the err= ors > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear' or repla= ce the device with 'zpool replace'. > =C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P > =C2=A0 scan: scrub repaired 4K in 0 days 06:30:55 with 0 errors on Sat = Apr 13 > 01:42:04 2019 > config: > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSUM > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0 0 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0 0 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/= backup61.eli=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2650= 799076683778414=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0= =C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was > /dev/gpt/backup62-1.eli > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/= backup62-2.eli=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 1 > > errors: No known data errors > > The OTHER interesting data point is that the resilver *also* posted one= > checksum error, which I cleared before doing the scrub.=C2=A0 Both on t= he > 62-2 device. > > That would be one block in both cases.=C2=A0 The expected was several (= maybe > a half-dozen) checksum errors on 19.00.00.00 during the scrub but *zero= * > during the resilver. > > The unit which was put *into* the vault and is now offline was written > and scrubbed under 20.00.07.00.=C2=A0 The behavior change certainly imp= lies > that there are some differences and again, none of these OFFLINE state > situations are uncontrolled -- in each case the drive is taken offline > intentionally, the geli provider is detached and then the unit has > "camcontrol standby" executed against it before it is yanked, so in > theory at least there should be no way for a unflushed but write-cached= > block to be lost or damaged. > > I smell a rat but it may well be in the 19.00.00.00 firmware on the car= d... I can confirm that 20.00.07.00 does *not* stop this. The previous write/scrub on this device was on 20.00.07.00.=C2=A0 It was swapped back in from the vault yesterday, resilvered without incident, but a scrub says.... root@NewFS:/home/karl # zpool status backup =C2=A0 pool: backup =C2=A0state: DEGRADED status: One or more devices has experienced an unrecoverable error.=C2=A0= An =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to correct th= e error.=C2=A0 Applications are unaffected. action: Determine if the device needs to be replaced, and clear the error= s =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear' or replace= the device with 'zpool replace'. =C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P =C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on Sat = Apr 20 08:45:09 2019 config: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSUM =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 47 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 132828= 12295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was /dev/gpt/backup62-2.eli errors: No known data errors So this is firmware-invariant (at least between 19.00.00.00 and 20.00.07.00); the issue persists. Again, in my instance these devices are never removed "unsolicited" so there can't be (or at least shouldn't be able to) unflushed data in the device or kernel cache.=C2=A0 The procedure is and remains: zpool offline ..... geli detach ..... camcontrol standby ... Wait a few seconds for the spindle to spin down. Remove disk. Then of course on the other side after insertion and the kernel has reported "finding" the device: geli attach ... zpool online .... Wait... If this is a boogered TXG that's held in the metadata for the "offline"'d device (maybe "off by one"?) that's potentially bad in that if there is an unknown failure in the other mirror component the resilver will complete but data has been irrevocably destroyed. Granted, this is a very low probability scenario (the area where the bad checksums are has to be where the corruption hits, and it has to happen between the resilver and access to that data.)=C2=A0 Those are long odds = but nonetheless a window of "you're hosed" does appear to exist. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms070906010904010606030101 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDIwMTQzOTIx WjBPBgkqhkiG9w0BCQQxQgRASwZdol2oQ6cIII6ZuyV8UWL46NapIjpaYQ93Tyw/rmR4StST nnmRUqn/LiljOQKwdBAOoXplkPaVuSkZUPoMITBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgCz6rmwBBY/dQ0YTtrGwCMybyApZBRkWR4RgT2ePg8t8jfqvRZtgOArY3OD51LkCo5i ieug3e7HLL8kayRbb7Y6g1eQOnj3rapDQN/miwyOeY5kuTe3BGLF71nRmsrRDzXbG+GSAT5x q+3A+QEQJ8J+GlHNMLqejcJBsiVIj7kO7AX9uMvB4VzO6WxXiUKuoESRD7T0HsR6GOcGX4V3 BwtnN7z8c0ZDwDsvVi8JoTorVZc8RfpmHFGJA65KVQOlf1hAP2wLsarXoVx/uuxaNZG1TBJz MNKdNpZJrrzIo+SEzOtQfwvBa+RbQqjfmBccFX3vEdUebrnhivUcraiFxjXyISpUnImH9fU9 JHvGGkzP01GHKSJeXfgcdrRXL08KOvYggg6mLf9VATgIN8xbjP4jaan4Z8QS2ndTxPdxTb7g yfGQEB1/osoCKY6SmPDuf0r83Gk6surZYXTAQcWpKGswC1DEZH1qHR6b8xCmzJpeZ1TQO8eW D7Wfkwf+0yXOt3ikT8UKcNd9zqebNSewVYpCUW0N2WjFYhx3+ejoyZpFABHDEwu2mFZGEP7r m3Gv8dOuveqcukdYBUOjRjhzbE/2AhgHjXVLXlCsIhRp0hp7HlVTyjcvcDSUZMImMMIPtOq5 fmTfYbxsIXda09SMZ7Fh2sv6soymTFN+sjTEpWrl0QAAAAAAAA== --------------ms070906010904010606030101-- From owner-freebsd-stable@freebsd.org Sat Apr 20 15:50:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEC3B1590E11 for ; Sat, 20 Apr 2019 15:50:40 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76B4C95EC0 for ; Sat, 20 Apr 2019 15:50:39 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x52f.google.com with SMTP id g6so6552530edc.8 for ; Sat, 20 Apr 2019 08:50:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=Z+j1odoeDC4IpR/s8eCr8f/+BAWdTInwmy8cpngnMWE=; b=sKyqVzy6N/SU9UMsrkROJpPCI3T8SeP/yis8XaPjpYFZkvZ6guJOsysmZMnKhwDgWS c17wgaM3YWUiTB5f7DE7xduzbru1W9YLM7wES2lmtYapYMpC2+55F5jU949bs7W/Nuep G+DuR62kK0TNvSbb/7lJS/z47wudVg9IQbiF/wNAKnPFiwIteOk6Zw1F0oQvxbq1Nk8o 38WBqCpRATPMCbTQ50X7yibzWxObdxBZO3exPfBYA7Ug8sQV8boakNZMQrW0QxefkrQT Ug3EA6zPv6TrX3d7dGL8aInzdnlcDHqhdyq6O5wo2mSjwZeOPFqknLf+Y8EMxwrCUQVl AueQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=Z+j1odoeDC4IpR/s8eCr8f/+BAWdTInwmy8cpngnMWE=; b=giPlQrEPKtRgEmZVyLM8YVC9LOc4nAM/ZX1nmIsx2qdhYo1P7X7duCEsbWUhs+mN/i 9+9hEz7ANxfAgMMdehq7NB6nGuNFTcu+nV/Z2lHnkdOJLCW6xFVWdUJCoZwEiuCo5EV7 7nyQShFG/Lzjy9/cfgVq72waToVC3n08s6OMxmicRvgkInJ2qLvCSE/8tx3FBjO+xBf1 brRKQZT5Yw+eZcbA8VuxDGvqw4nuQyEpkARxhrsue3QEvX3BZA68KbxFmmpgFSX0o8w8 pq+g1crsl8q0oKHKdO9OLlIT/RJcDL606gKsGkD99dHiUAnWAYx2zERog5cKY2pCIFOR VKJg== X-Gm-Message-State: APjAAAUMi1VgJj3OvEF6wk+kZ9si5tKPVMHNqpQwjpoVbZGr3SFBRTcX SQB1jjWP4YC2zWGsANT87KCQW+ute/8= X-Google-Smtp-Source: APXvYqynX15dAnwwqVjsGAgJp5lNOkQUPx7vGe2HdsuJSLAkjB5Ni8UAZMzwR69ymJfR8D5sCQg9aQ== X-Received: by 2002:a50:a915:: with SMTP id l21mr6338715edc.164.1555775437671; Sat, 20 Apr 2019 08:50:37 -0700 (PDT) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id s13sm1369794eju.89.2019.04.20.08.50.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 20 Apr 2019 08:50:36 -0700 (PDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Karl Denninger , freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> From: Steven Hartland Message-ID: <758d5611-c3cf-82dd-220f-a775a57bdd0b@multiplay.co.uk> Date: Sat, 20 Apr 2019 16:50:38 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 76B4C95EC0 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=sKyqVzy6; spf=pass (mx1.freebsd.org: domain of killing@multiplay.co.uk designates 2a00:1450:4864:20::52f as permitted sender) smtp.mailfrom=killing@multiplay.co.uk X-Spamd-Result: default: False [-6.26 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[f.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[ASPMX.L.GOOGLE.COM,ALT2.ASPMX.L.GOOGLE.COM,ALT1.ASPMX.L.GOOGLE.COM,ASPMX3.GOOGLEMAIL.COM,ASPMX2.GOOGLEMAIL.COM]; IP_SCORE(-2.76)[ip: (-9.09), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-2.26), country: US(-0.06)]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 15:50:41 -0000 Have you eliminated geli as possible source? I've just setup an old server which has a LSI 2008 running and old FW (11.0) so was going to have a go at reproducing this. Apart from the disconnect steps below is there anything else needed e.g. read / write workload during disconnect? mps0: port 0xe000-0xe0ff mem 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 26 at device 0.0 on pci3 mps0: Firmware: 11.00.00.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 185c     Regards     Steve On 20/04/2019 15:39, Karl Denninger wrote: > I can confirm that 20.00.07.00 does *not* stop this. > The previous write/scrub on this device was on 20.00.07.00.  It was > swapped back in from the vault yesterday, resilvered without incident, > but a scrub says.... > > root@NewFS:/home/karl # zpool status backup >   pool: backup >  state: DEGRADED > status: One or more devices has experienced an unrecoverable error.  An >         attempt was made to correct the error.  Applications are unaffected. > action: Determine if the device needs to be replaced, and clear the errors >         using 'zpool clear' or replace the device with 'zpool replace'. >    see: http://illumos.org/msg/ZFS-8000-9P >   scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on Sat Apr > 20 08:45:09 2019 > config: > >         NAME                      STATE     READ WRITE CKSUM >         backup                    DEGRADED     0     0     0 >           mirror-0                DEGRADED     0     0     0 >             gpt/backup61.eli      ONLINE       0     0     0 >             gpt/backup62-1.eli    ONLINE       0     0    47 >             13282812295755460479  OFFLINE      0     0     0  was > /dev/gpt/backup62-2.eli > > errors: No known data errors > > So this is firmware-invariant (at least between 19.00.00.00 and > 20.00.07.00); the issue persists. > > Again, in my instance these devices are never removed "unsolicited" so > there can't be (or at least shouldn't be able to) unflushed data in the > device or kernel cache.  The procedure is and remains: > > zpool offline ..... > geli detach ..... > camcontrol standby ... > > Wait a few seconds for the spindle to spin down. > > Remove disk. > > Then of course on the other side after insertion and the kernel has > reported "finding" the device: > > geli attach ... > zpool online .... > > Wait... > > If this is a boogered TXG that's held in the metadata for the > "offline"'d device (maybe "off by one"?) that's potentially bad in that > if there is an unknown failure in the other mirror component the > resilver will complete but data has been irrevocably destroyed. > > Granted, this is a very low probability scenario (the area where the bad > checksums are has to be where the corruption hits, and it has to happen > between the resilver and access to that data.)  Those are long odds but > nonetheless a window of "you're hosed" does appear to exist. > From owner-freebsd-stable@freebsd.org Sat Apr 20 17:36:06 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 440501571254 for ; Sat, 20 Apr 2019 17:36:06 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9FD266BE3D for ; Sat, 20 Apr 2019 17:36:04 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 33D4821109F; Sat, 20 Apr 2019 13:35:33 -0400 (EDT) Received: from [192.168.10.17] (D7.Denninger.Net [192.168.10.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id A8BF6C69A4; Sat, 20 Apr 2019 12:35:31 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Steven Hartland , freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> <758d5611-c3cf-82dd-220f-a775a57bdd0b@multiplay.co.uk> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <3f53389a-0cb5-d106-1f64-bbc2123e975c@denninger.net> Date: Sat, 20 Apr 2019 12:35:31 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <758d5611-c3cf-82dd-220f-a775a57bdd0b@multiplay.co.uk> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020001040304000308080205" X-Rspamd-Queue-Id: 9FD266BE3D X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-5.93 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(-2.47)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.99), asn: 14061(1.60), country: US(-0.06)]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; DMARC_NA(0.00)[denninger.net]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: px.denninger.net]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.26)[-0.256,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 17:36:06 -0000 This is a cryptographically signed message in MIME format. --------------ms020001040304000308080205 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/20/2019 10:50, Steven Hartland wrote: > Have you eliminated geli as possible source? No; I could conceivably do so by re-creating another backup volume set without geli-encrypting the drives, but I do not have an extra set of drives of the capacity required laying around to do that.=C2=A0 I would h= ave to do it with lower-capacity disks, which I can attempt if you think it would help.=C2=A0 I *do* have open slots in the drive backplane to set up= a second "test" unit of this sort.=C2=A0 For reasons below it will take at least a couple of weeks to get good data on whether the problem exists without geli, however. > > I've just setup an old server which has a LSI 2008 running and old FW > (11.0) so was going to have a go at reproducing this. > > Apart from the disconnect steps below is there anything else needed > e.g. read / write workload during disconnect? Yes.=C2=A0 An attempt to recreate this on my sandbox machine using smalle= r disks (WD RE-320s) and a decent amount of read/write activity (tens to ~100 gigabytes) on a root mirror of three disks with one taken offline did not succeed.=C2=A0 It *reliably* appears, however, on my backup volum= es with every drive swap.=C2=A0 The sandbox machine is physically identical other than the physical disks; both are Xeons with ECC RAM in them. The only operational difference is that the backup volume sets have a *lot* of data written to them via zfs send|zfs recv over the intervening period where with "ordinary" activity from I/O (which was the case on my sandbox) the I/O pattern is materially different.=C2=A0 The root pool on = the sandbox where I tried to reproduce it synthetically *is* using geli (in fact it boots native-encrypted.) The "ordinary" resilver on a disk swap typically covers ~2-3Tb and is a ~6-8 hour process. The usual process for the backup pool looks like this: Have 2 of the 3 physical disks mounted; the third is in the bank vault. Over the space of a week, the backup script is run daily.=C2=A0 It first imports the pool and then for each zfs filesystem it is backing up (which is not all of them; I have a few volatile ones that I don't care if I lose, such as object directories for builds and such, plus some that are R/O data sets that are backed up separately) it does: If there is no "...@zfs-base": zfs snapshot -r ...@zfs-base; zfs send -R =2E..@zfs-base | zfs receive -Fuvd $BACKUP else zfs rename -r ...@zfs-base ...@zfs-old zfs snapshot -r ...@zfs-base zfs send -RI ...@zfs-old ...@zfs-base |zfs recv -Fudv $BACKUP =2E... if ok then zfs destroy -vr ...@zfs-old otherwise print a complaint= and stop. When all are complete it then does a "zpool export backup" to detach the pool in order to reduce the risk of "stupid root user" (me) accidents. In short I send an incremental of the changes since the last backup, which in many cases includes a bunch of automatic snapshots that are taken on frequent basis out of the cron.=C2=A0 Typically there are a week= 's worth of these that accumulate between swaps of the disk to the vault, and the offline'd disk remains that way for a week.=C2=A0 I also wait for= the zpool destroy on each of the targets to drain before continuing, as not doing so back in the 9 and 10.x days was a good way to stimulate an instant panic on re-import the next day due to kernel stack page exhaustion if the previous operation destroyed hundreds of gigabytes of snapshots (which does routinely happen as part of the backed up data is Macrium images from PCs, so when a new month comes around the PC's backup routine removes a huge amount of old data from the filesystem.) Trying to simulate the checksum errors in a few hours' time thus far has failed.=C2=A0 But every time I swap the disks on a weekly basis I get a handful of checksum errors on the scrub.=C2=A0 If I export and re-import = the backup mirror after that the counters are zeroed -- the checksum error count does *not* remain across an export/import cycle although the "scrub repaired" line remains. For example after the scrub completed this morning I exported the pool (the script expects the pool exported before it begins) and ran the backup.=C2=A0 When it was complete: root@NewFS:~/backup-zfs # zpool status backup =C2=A0 pool: backup =C2=A0state: DEGRADED status: One or more devices has been taken offline by the administrator. =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Sufficient replicas exist for = the pool to continue functioning in a =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 degraded state. action: Online the device using 'zpool online' or replace the device with= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 'zpool replace'. =C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on Sat = Apr 20 08:45:09 2019 config: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSUM =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2= =A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 132828= 12295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was /dev/gpt/backup62-2.eli errors: No known data errors It knows it fixed the checksums but the error count is zero -- I did NOT "zpool clear". This may have been present in 11.2; I didn't run that long enough in this environment to know.=C2=A0 It definitely was *not* present in 11.1 a= nd before; the same data structure and script for backups has been in use for a very long time without any changes and this first appeared when I upgraded from 11.1 to 12.0 on this specific machine, with the exact same physical disks being used for over a year (they're currently 6Tb units; the last change out for those was ~1.5 years ago when I went from 4Tb to 6Tb volumes.)=C2=A0 I have both HGST-NAS and He-Enterprise disks in the rotation and both show identical behavior so it doesn't appear to be related to a firmware problem in one disk .vs. the other (e.g. firmware that fails to flush the on-drive cache before going to standby even though it was told to.) > > mps0: port 0xe000-0xe0ff mem > 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 26 at device 0.0 on pci= 3 > mps0: Firmware: 11.00.00.00, Driver: 21.02.00.00-fbsd > mps0: IOCCapabilities: > 185c > > =C2=A0=C2=A0=C2=A0 Regards > =C2=A0=C2=A0=C2=A0 Steve > > On 20/04/2019 15:39, Karl Denninger wrote: >> I can confirm that 20.00.07.00 does *not* stop this. >> The previous write/scrub on this device was on 20.00.07.00.=C2=A0 It w= as >> swapped back in from the vault yesterday, resilvered without incident,= >> but a scrub says.... >> >> root@NewFS:/home/karl # zpool status backup >> =C2=A0=C2=A0 pool: backup >> =C2=A0=C2=A0state: DEGRADED >> status: One or more devices has experienced an unrecoverable error.=C2= =A0 An >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to c= orrect the error.=C2=A0 Applications are >> unaffected. >> action: Determine if the device needs to be replaced, and clear the >> errors >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear' o= r replace the device with 'zpool replace'. >> =C2=A0=C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P >> =C2=A0=C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 error= s on Sat Apr >> 20 08:45:09 2019 >> config: >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRIT= E CKSUM >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0= =C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= gpt/backup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= gpt/backup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 47 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 13282812295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was >> /dev/gpt/backup62-2.eli >> >> errors: No known data errors >> >> So this is firmware-invariant (at least between 19.00.00.00 and >> 20.00.07.00); the issue persists. >> >> Again, in my instance these devices are never removed "unsolicited" so= >> there can't be (or at least shouldn't be able to) unflushed data in th= e >> device or kernel cache.=C2=A0 The procedure is and remains: >> >> zpool offline ..... >> geli detach ..... >> camcontrol standby ... >> >> Wait a few seconds for the spindle to spin down. >> >> Remove disk. >> >> Then of course on the other side after insertion and the kernel has >> reported "finding" the device: >> >> geli attach ... >> zpool online .... >> >> Wait... >> >> If this is a boogered TXG that's held in the metadata for the >> "offline"'d device (maybe "off by one"?) that's potentially bad in tha= t >> if there is an unknown failure in the other mirror component the >> resilver will complete but data has been irrevocably destroyed. >> >> Granted, this is a very low probability scenario (the area where the b= ad >> checksums are has to be where the corruption hits, and it has to happe= n >> between the resilver and access to that data.)=C2=A0 Those are long od= ds but >> nonetheless a window of "you're hosed" does appear to exist. >> > --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020001040304000308080205 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDIwMTczNTMx WjBPBgkqhkiG9w0BCQQxQgRATpkGJK6sJ0iZw/PgzmFGRWATiGfd4pd4wr/vpRGSAUNJR3l4 y1dQJv0vJ/Vx8pzztupUuUCh9VLjGKwrVIbXgTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAwpE340/dt2YXvRUi6Vq4Ehbs1DpKL/bYpjrDoam4C/MHfXVM92XufPJkcM2JRXTe5 PNekyDlEHyrk1Xw9cfTpI5rSiMsU0gaAwwujn7PztN4j07/Dx+6ZDSctA74geT5jo6kagkDb sEypDtwuLHrMmHKklnWg3rN4MXOLIeyeLZ3zGCXXsekm33Nh9zhrelOhpXYUZY0K8EwbZwrC /sMW+W2X9mS7E+3UPClY/4qogXV5aD3B3NxN56aNgJ7nLO35IDJ8QMaREpzbMEGM5Z4gXcrL ATsq8PhBwl5dsppau5NyfEn97f2OZnhr8zRPscu6tyBf8WjVF7aXwmpZRrsvW6bEh4s+AYKh yNlaXWqEbQLQ+SSKSLCugyHPm9T+gbugkgepDR6bq6lUaXSvtiU3GqUCNByNWyDe8savT8XM i6OeWwwlYtEWgXNLqU5+ybyQdBu9KnsWXHO0G9AgUSi7JIjbvCo8vJLWwmGV5aygPzJLKK1T MYFatN63C5fhALERo/GsugoUrl23F3UxWLDJVvs/iL2YdtJjVOMxZ5Dyv7Xnask1k96boQW3 0CuPOZkChmp6i5fxeao6HUa0MDV+uWYBb2j8ejYwtMm7FoaVvMRjb4g3BR7lFa6ckN27IV0l YlHM3Hn0/KalV9UmIVtW/ruBuiconAo8ow9TGwezvQAAAAAAAA== --------------ms020001040304000308080205-- From owner-freebsd-stable@freebsd.org Sat Apr 20 20:56:49 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9957C1575BC7 for ; Sat, 20 Apr 2019 20:56:48 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 79C3F7195E for ; Sat, 20 Apr 2019 20:56:47 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x541.google.com with SMTP id j20so6859441edq.10 for ; Sat, 20 Apr 2019 13:56:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=+ehUGQF8ap7GkDIsHR+/Dgsoa187H0nI3UdT1cVcNzk=; b=XWcoyFMKNVi/b/AT1yIComq+LM7fU2J8XTE7nv99VjBjvgQKTFigbdFtDIbsMw5mb4 lIoumLYt2SgCOq9jASFDapbKaPqVbPs0c0qB3rCIjoDbnZI/8VmM313gUWYk86/3vN9T rfA8Q5h2NwPdE6ftAq8IeORm4HC4FgAWhPR4BOw1woX46hxSJ2TXnE8PFp9SQdPoYNPP Ff7h8mLAarxM9a7xuePr+5RYXgmZGayiSLjwWPfBnzwA5OEKNSkn72US9s2zgWSwpTpo dRQv+CSRcKBzNa72CMCC/ag6Gdhe6jOEI52exQhUnOOyMHFTN8zAXMewGvJ9UWZSEK5H R9OQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=+ehUGQF8ap7GkDIsHR+/Dgsoa187H0nI3UdT1cVcNzk=; b=n4YWze0EsQdziloi929iT3Vb3/i4YnJ4o1wJgmV9JAFrv+75uAzoj1cNd4yLdiSjNZ w/nXMjO+1cqzHTWnGOiDAOKU8+Nqq1NNvFVrZT5gtHKYgAYe4HPPpUa4MTxvSnXl1lkl MB1W/Df39mNC4fGZFsVE9HPyx3PG+Oyu15uFuVZ2H06QakiIf3Wgec+hMZKRFPVWvFb/ 1DKBK6hq2s4zcdWgliN9MVf8w9/oITzbyrK0gKhgCk7AsUnI4lQObLh5w+5G1uWp84rg p3i+bW2ccTR6w3jxop2YACrZXQTbhY7PhJzlBd8J8wdUs/oIWKrqW0mW/plefU+A7qvS du6g== X-Gm-Message-State: APjAAAUdycbhVhsWvU+ya3fpqmvEiwxQeN0UXk8AVUt/MGs3SFapJy/6 0V6e3iH7HKvHpm2/ywCg1ZQaSOg2b9E= X-Google-Smtp-Source: APXvYqyBkPdPoBnfB4OEJL6lvtLLdZ1a9VHGPEezPTtJqUvJCyWDYXoPSW3V1kaUOx83P6JFbqLBVA== X-Received: by 2002:aa7:c88a:: with SMTP id p10mr7103307eds.145.1555793805602; Sat, 20 Apr 2019 13:56:45 -0700 (PDT) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id w10sm2356737edh.62.2019.04.20.13.56.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 20 Apr 2019 13:56:44 -0700 (PDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Karl Denninger , freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> <758d5611-c3cf-82dd-220f-a775a57bdd0b@multiplay.co.uk> <3f53389a-0cb5-d106-1f64-bbc2123e975c@denninger.net> From: Steven Hartland Message-ID: <8108da18-2cdd-fa29-983c-3ae7be6be412@multiplay.co.uk> Date: Sat, 20 Apr 2019 21:56:45 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <3f53389a-0cb5-d106-1f64-bbc2123e975c@denninger.net> Content-Language: en-US X-Rspamd-Queue-Id: 79C3F7195E X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=XWcoyFMK; spf=pass (mx1.freebsd.org: domain of killing@multiplay.co.uk designates 2a00:1450:4864:20::541 as permitted sender) smtp.mailfrom=killing@multiplay.co.uk X-Spamd-Result: default: False [-4.49 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: ASPMX.L.GOOGLE.COM]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.99)[ip: (-0.28), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-2.26), country: US(-0.06)] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 20:56:49 -0000 Thanks for extra info, the next question would be have you eliminated that corruption exists before the disk is removed? Would be interesting to add a zpool scrub to confirm this isn't the case before the disk removal is attempted.     Regards     Steve On 20/04/2019 18:35, Karl Denninger wrote: > > On 4/20/2019 10:50, Steven Hartland wrote: >> Have you eliminated geli as possible source? > No; I could conceivably do so by re-creating another backup volume set > without geli-encrypting the drives, but I do not have an extra set of > drives of the capacity required laying around to do that. I would have > to do it with lower-capacity disks, which I can attempt if you think > it would help.  I *do* have open slots in the drive backplane to set > up a second "test" unit of this sort.  For reasons below it will take > at least a couple of weeks to get good data on whether the problem > exists without geli, however. >> >> I've just setup an old server which has a LSI 2008 running and old FW >> (11.0) so was going to have a go at reproducing this. >> >> Apart from the disconnect steps below is there anything else needed >> e.g. read / write workload during disconnect? > > Yes.  An attempt to recreate this on my sandbox machine using smaller > disks (WD RE-320s) and a decent amount of read/write activity (tens to > ~100 gigabytes) on a root mirror of three disks with one taken offline > did not succeed.  It *reliably* appears, however, on my backup volumes > with every drive swap. The sandbox machine is physically identical > other than the physical disks; both are Xeons with ECC RAM in them. > > The only operational difference is that the backup volume sets have a > *lot* of data written to them via zfs send|zfs recv over the > intervening period where with "ordinary" activity from I/O (which was > the case on my sandbox) the I/O pattern is materially different.  The > root pool on the sandbox where I tried to reproduce it synthetically > *is* using geli (in fact it boots native-encrypted.) > > The "ordinary" resilver on a disk swap typically covers ~2-3Tb and is > a ~6-8 hour process. > > The usual process for the backup pool looks like this: > > Have 2 of the 3 physical disks mounted; the third is in the bank vault. > > Over the space of a week, the backup script is run daily.  It first > imports the pool and then for each zfs filesystem it is backing up > (which is not all of them; I have a few volatile ones that I don't > care if I lose, such as object directories for builds and such, plus > some that are R/O data sets that are backed up separately) it does: > > If there is no "...@zfs-base": zfs snapshot -r ...@zfs-base; zfs send > -R ...@zfs-base | zfs receive -Fuvd $BACKUP > > else > > zfs rename -r ...@zfs-base ...@zfs-old > zfs snapshot -r ...@zfs-base > > zfs send -RI ...@zfs-old ...@zfs-base |zfs recv -Fudv $BACKUP > > .... if ok then zfs destroy -vr ...@zfs-old otherwise print a > complaint and stop. > > When all are complete it then does a "zpool export backup" to detach > the pool in order to reduce the risk of "stupid root user" (me) accidents. > > In short I send an incremental of the changes since the last backup, > which in many cases includes a bunch of automatic snapshots that are > taken on frequent basis out of the cron. Typically there are a week's > worth of these that accumulate between swaps of the disk to the vault, > and the offline'd disk remains that way for a week.  I also wait for > the zpool destroy on each of the targets to drain before continuing, > as not doing so back in the 9 and 10.x days was a good way to > stimulate an instant panic on re-import the next day due to kernel > stack page exhaustion if the previous operation destroyed hundreds of > gigabytes of snapshots (which does routinely happen as part of the > backed up data is Macrium images from PCs, so when a new month comes > around the PC's backup routine removes a huge amount of old data from > the filesystem.) > > Trying to simulate the checksum errors in a few hours' time thus far > has failed.  But every time I swap the disks on a weekly basis I get a > handful of checksum errors on the scrub. If I export and re-import the > backup mirror after that the counters are zeroed -- the checksum error > count does *not* remain across an export/import cycle although the > "scrub repaired" line remains. > > For example after the scrub completed this morning I exported the pool > (the script expects the pool exported before it begins) and ran the > backup.  When it was complete: > > root@NewFS:~/backup-zfs # zpool status backup >   pool: backup >  state: DEGRADED > status: One or more devices has been taken offline by the administrator. >         Sufficient replicas exist for the pool to continue functioning > in a >         degraded state. > action: Online the device using 'zpool online' or replace the device with >         'zpool replace'. >   scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on Sat > Apr 20 08:45:09 2019 > config: > >         NAME                      STATE     READ WRITE CKSUM >         backup                    DEGRADED     0 0     0 >           mirror-0                DEGRADED     0 0     0 >             gpt/backup61.eli      ONLINE       0 0     0 >             gpt/backup62-1.eli    ONLINE       0 0     0 >             13282812295755460479  OFFLINE      0 0     0  was > /dev/gpt/backup62-2.eli > > errors: No known data errors > > It knows it fixed the checksums but the error count is zero -- I did > NOT "zpool clear". > > This may have been present in 11.2; I didn't run that long enough in > this environment to know.  It definitely was *not* present in 11.1 and > before; the same data structure and script for backups has been in use > for a very long time without any changes and this first appeared when > I upgraded from 11.1 to 12.0 on this specific machine, with the exact > same physical disks being used for over a year (they're currently 6Tb > units; the last change out for those was ~1.5 years ago when I went > from 4Tb to 6Tb volumes.)  I have both HGST-NAS and He-Enterprise > disks in the rotation and both show identical behavior so it doesn't > appear to be related to a firmware problem in one disk .vs. the other > (e.g. firmware that fails to flush the on-drive cache before going to > standby even though it was told to.) > >> >> mps0: port 0xe000-0xe0ff mem >> 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 26 at device 0.0 on pci3 >> mps0: Firmware: 11.00.00.00, Driver: 21.02.00.00-fbsd >> mps0: IOCCapabilities: >> 185c >> >>     Regards >>     Steve >> >> On 20/04/2019 15:39, Karl Denninger wrote: >>> I can confirm that 20.00.07.00 does *not* stop this. >>> The previous write/scrub on this device was on 20.00.07.00. It was >>> swapped back in from the vault yesterday, resilvered without incident, >>> but a scrub says.... >>> >>> root@NewFS:/home/karl # zpool status backup >>>    pool: backup >>>   state: DEGRADED >>> status: One or more devices has experienced an unrecoverable error.  An >>>          attempt was made to correct the error.  Applications are >>> unaffected. >>> action: Determine if the device needs to be replaced, and clear the >>> errors >>>          using 'zpool clear' or replace the device with 'zpool >>> replace'. >>>     see: http://illumos.org/msg/ZFS-8000-9P >>>    scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on Sat >>> Apr >>> 20 08:45:09 2019 >>> config: >>> >>>          NAME                      STATE     READ WRITE CKSUM >>>          backup                    DEGRADED     0     0     0 >>>            mirror-0                DEGRADED     0     0     0 >>>              gpt/backup61.eli      ONLINE       0     0     0 >>>              gpt/backup62-1.eli    ONLINE       0     0    47 >>>              13282812295755460479  OFFLINE      0     0     0 was >>> /dev/gpt/backup62-2.eli >>> >>> errors: No known data errors >>> >>> So this is firmware-invariant (at least between 19.00.00.00 and >>> 20.00.07.00); the issue persists. >>> >>> Again, in my instance these devices are never removed "unsolicited" so >>> there can't be (or at least shouldn't be able to) unflushed data in the >>> device or kernel cache.  The procedure is and remains: >>> >>> zpool offline ..... >>> geli detach ..... >>> camcontrol standby ... >>> >>> Wait a few seconds for the spindle to spin down. >>> >>> Remove disk. >>> >>> Then of course on the other side after insertion and the kernel has >>> reported "finding" the device: >>> >>> geli attach ... >>> zpool online .... >>> >>> Wait... >>> >>> If this is a boogered TXG that's held in the metadata for the >>> "offline"'d device (maybe "off by one"?) that's potentially bad in that >>> if there is an unknown failure in the other mirror component the >>> resilver will complete but data has been irrevocably destroyed. >>> >>> Granted, this is a very low probability scenario (the area where the >>> bad >>> checksums are has to be where the corruption hits, and it has to happen >>> between the resilver and access to that data.)  Those are long odds but >>> nonetheless a window of "you're hosed" does appear to exist. >>> >> > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ From owner-freebsd-stable@freebsd.org Sat Apr 20 21:26:06 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 836651576716 for ; Sat, 20 Apr 2019 21:26:06 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 94C1972AD8 for ; Sat, 20 Apr 2019 21:26:05 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id E56C621109F for ; Sat, 20 Apr 2019 17:26:03 -0400 (EDT) Received: from [192.168.10.17] (D7.Denninger.Net [192.168.10.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 70D6BC6FE2 for ; Sat, 20 Apr 2019 16:26:02 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> <6dc1bad1-05b8-2c65-99d3-61c547007dfe@denninger.net> <758d5611-c3cf-82dd-220f-a775a57bdd0b@multiplay.co.uk> <3f53389a-0cb5-d106-1f64-bbc2123e975c@denninger.net> <8108da18-2cdd-fa29-983c-3ae7be6be412@multiplay.co.uk> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: Date: Sat, 20 Apr 2019 16:26:01 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <8108da18-2cdd-fa29-983c-3ae7be6be412@multiplay.co.uk> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms050604040905000400030000" X-Rspamd-Queue-Id: 94C1972AD8 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.54 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: px.denninger.net]; NEURAL_HAM_SHORT(-0.86)[-0.858,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.47)[ip: (-9.88), ipnet: 104.236.64.0/18(-4.01), asn: 14061(1.60), country: US(-0.06)]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Apr 2019 21:26:07 -0000 This is a cryptographically signed message in MIME format. --------------ms050604040905000400030000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable No; I can, but of course that's another ~8 hour (overnight) delay between swaps. That's not a bad idea however.... On 4/20/2019 15:56, Steven Hartland wrote: > Thanks for extra info, the next question would be have you eliminated > that corruption exists before the disk is removed? > > Would be interesting to add a zpool scrub to confirm this isn't the > case before the disk removal is attempted. > > =C2=A0=C2=A0=C2=A0 Regards > =C2=A0=C2=A0=C2=A0 Steve > > On 20/04/2019 18:35, Karl Denninger wrote: >> >> On 4/20/2019 10:50, Steven Hartland wrote: >>> Have you eliminated geli as possible source? >> No; I could conceivably do so by re-creating another backup volume >> set without geli-encrypting the drives, but I do not have an extra >> set of drives of the capacity required laying around to do that. I >> would have to do it with lower-capacity disks, which I can attempt if >> you think it would help.=C2=A0 I *do* have open slots in the drive >> backplane to set up a second "test" unit of this sort.=C2=A0 For reaso= ns >> below it will take at least a couple of weeks to get good data on >> whether the problem exists without geli, however. >>> >>> I've just setup an old server which has a LSI 2008 running and old >>> FW (11.0) so was going to have a go at reproducing this. >>> >>> Apart from the disconnect steps below is there anything else needed >>> e.g. read / write workload during disconnect? >> >> Yes.=C2=A0 An attempt to recreate this on my sandbox machine using sma= ller >> disks (WD RE-320s) and a decent amount of read/write activity (tens >> to ~100 gigabytes) on a root mirror of three disks with one taken >> offline did not succeed.=C2=A0 It *reliably* appears, however, on my >> backup volumes with every drive swap. The sandbox machine is >> physically identical other than the physical disks; both are Xeons >> with ECC RAM in them. >> >> The only operational difference is that the backup volume sets have a >> *lot* of data written to them via zfs send|zfs recv over the >> intervening period where with "ordinary" activity from I/O (which was >> the case on my sandbox) the I/O pattern is materially different.=C2=A0= The >> root pool on the sandbox where I tried to reproduce it synthetically >> *is* using geli (in fact it boots native-encrypted.) >> >> The "ordinary" resilver on a disk swap typically covers ~2-3Tb and is >> a ~6-8 hour process. >> >> The usual process for the backup pool looks like this: >> >> Have 2 of the 3 physical disks mounted; the third is in the bank vault= =2E >> >> Over the space of a week, the backup script is run daily.=C2=A0 It fir= st >> imports the pool and then for each zfs filesystem it is backing up >> (which is not all of them; I have a few volatile ones that I don't >> care if I lose, such as object directories for builds and such, plus >> some that are R/O data sets that are backed up separately) it does: >> >> If there is no "...@zfs-base": zfs snapshot -r ...@zfs-base; zfs send >> -R ...@zfs-base | zfs receive -Fuvd $BACKUP >> >> else >> >> zfs rename -r ...@zfs-base ...@zfs-old >> zfs snapshot -r ...@zfs-base >> >> zfs send -RI ...@zfs-old ...@zfs-base |zfs recv -Fudv $BACKUP >> >> .... if ok then zfs destroy -vr ...@zfs-old otherwise print a >> complaint and stop. >> >> When all are complete it then does a "zpool export backup" to detach >> the pool in order to reduce the risk of "stupid root user" (me) >> accidents. >> >> In short I send an incremental of the changes since the last backup, >> which in many cases includes a bunch of automatic snapshots that are >> taken on frequent basis out of the cron. Typically there are a week's >> worth of these that accumulate between swaps of the disk to the >> vault, and the offline'd disk remains that way for a week.=C2=A0 I als= o >> wait for the zpool destroy on each of the targets to drain before >> continuing, as not doing so back in the 9 and 10.x days was a good >> way to stimulate an instant panic on re-import the next day due to >> kernel stack page exhaustion if the previous operation destroyed >> hundreds of gigabytes of snapshots (which does routinely happen as >> part of the backed up data is Macrium images from PCs, so when a new >> month comes around the PC's backup routine removes a huge amount of >> old data from the filesystem.) >> >> Trying to simulate the checksum errors in a few hours' time thus far >> has failed.=C2=A0 But every time I swap the disks on a weekly basis I = get >> a handful of checksum errors on the scrub. If I export and re-import >> the backup mirror after that the counters are zeroed -- the checksum >> error count does *not* remain across an export/import cycle although >> the "scrub repaired" line remains. >> >> For example after the scrub completed this morning I exported the >> pool (the script expects the pool exported before it begins) and ran >> the backup.=C2=A0 When it was complete: >> >> root@NewFS:~/backup-zfs # zpool status backup >> =C2=A0 pool: backup >> =C2=A0state: DEGRADED >> status: One or more devices has been taken offline by the administrato= r. >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Sufficient replicas exist f= or the pool to continue >> functioning in a >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 degraded state. >> action: Online the device using 'zpool online' or replace the device >> with >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 'zpool replace'. >> =C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 errors on S= at >> Apr 20 08:45:09 2019 >> config: >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSU= M >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0 0=C2=A0=C2=A0=C2=A0= =C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt= /backup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt= /backup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 132= 82812295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0 0=C2=A0=C2= =A0=C2=A0=C2=A0 0=C2=A0 was >> /dev/gpt/backup62-2.eli >> >> errors: No known data errors >> >> It knows it fixed the checksums but the error count is zero -- I did >> NOT "zpool clear". >> >> This may have been present in 11.2; I didn't run that long enough in >> this environment to know.=C2=A0 It definitely was *not* present in 11.= 1 >> and before; the same data structure and script for backups has been >> in use for a very long time without any changes and this first >> appeared when I upgraded from 11.1 to 12.0 on this specific machine, >> with the exact same physical disks being used for over a year >> (they're currently 6Tb units; the last change out for those was ~1.5 >> years ago when I went from 4Tb to 6Tb volumes.)=C2=A0 I have both HGST= -NAS >> and He-Enterprise disks in the rotation and both show identical >> behavior so it doesn't appear to be related to a firmware problem in >> one disk .vs. the other (e.g. firmware that fails to flush the >> on-drive cache before going to standby even though it was told to.) >> >>> >>> mps0: port 0xe000-0xe0ff mem >>> 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 26 at device 0.0 on >>> pci3 >>> mps0: Firmware: 11.00.00.00, Driver: 21.02.00.00-fbsd >>> mps0: IOCCapabilities: >>> 185c >>> >>> =C2=A0=C2=A0=C2=A0 Regards >>> =C2=A0=C2=A0=C2=A0 Steve >>> >>> On 20/04/2019 15:39, Karl Denninger wrote: >>>> I can confirm that 20.00.07.00 does *not* stop this. >>>> The previous write/scrub on this device was on 20.00.07.00. It was >>>> swapped back in from the vault yesterday, resilvered without inciden= t, >>>> but a scrub says.... >>>> >>>> root@NewFS:/home/karl # zpool status backup >>>> =C2=A0=C2=A0 pool: backup >>>> =C2=A0=C2=A0state: DEGRADED >>>> status: One or more devices has experienced an unrecoverable >>>> error.=C2=A0 An >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to= correct the error.=C2=A0 Applications are >>>> unaffected. >>>> action: Determine if the device needs to be replaced, and clear the >>>> errors >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear'= or replace the device with 'zpool >>>> replace'. >>>> =C2=A0=C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P >>>> =C2=A0=C2=A0 scan: scrub repaired 188K in 0 days 09:40:18 with 0 err= ors on >>>> Sat Apr >>>> 20 08:45:09 2019 >>>> config: >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ W= RITE CKSUM >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-= 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 gpt/backup61.eli=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 gpt/backup62-1.eli=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 47 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 13282812295755460479=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 was >>>> /dev/gpt/backup62-2.eli >>>> >>>> errors: No known data errors >>>> >>>> So this is firmware-invariant (at least between 19.00.00.00 and >>>> 20.00.07.00); the issue persists. >>>> >>>> Again, in my instance these devices are never removed "unsolicited" = so >>>> there can't be (or at least shouldn't be able to) unflushed data in >>>> the >>>> device or kernel cache.=C2=A0 The procedure is and remains: >>>> >>>> zpool offline ..... >>>> geli detach ..... >>>> camcontrol standby ... >>>> >>>> Wait a few seconds for the spindle to spin down. >>>> >>>> Remove disk. >>>> >>>> Then of course on the other side after insertion and the kernel has >>>> reported "finding" the device: >>>> >>>> geli attach ... >>>> zpool online .... >>>> >>>> Wait... >>>> >>>> If this is a boogered TXG that's held in the metadata for the >>>> "offline"'d device (maybe "off by one"?) that's potentially bad in >>>> that >>>> if there is an unknown failure in the other mirror component the >>>> resilver will complete but data has been irrevocably destroyed. >>>> >>>> Granted, this is a very low probability scenario (the area where >>>> the bad >>>> checksums are has to be where the corruption hits, and it has to >>>> happen >>>> between the resilver and access to that data.)=C2=A0 Those are long = odds >>>> but >>>> nonetheless a window of "you're hosed" does appear to exist. >>>> >>> >> --=C2=A0 >> Karl Denninger >> karl@denninger.net >> /The Market Ticker/ >> /[S/MIME encrypted email preferred]/ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g" --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms050604040905000400030000 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDIwMjEyNjAx WjBPBgkqhkiG9w0BCQQxQgRApM1ZSXalW0sFaLNthNU5f/k+2Z1ruYlH/ZZDmYdJxghokLjq NY4ki9TVRYJulJHlkos7fCQcORTCqZxhyCW2oTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgBTjiHmXJGAIUHx3z0UqQd8lrBXyzNkSy6MIDs5dGJUD+odpe0L8Dtu351k0xkRgur5 g+9chTSCU+aYJBIaD6n6ScHe1d5GqTlRh2N5zz4ZSsJSubd83rGKRh9FY9NIa9Lqpnh5zWyx DYU4J4+B9GKvQA79nJ6XmwUl7HxQJZMX3yo63wkbagwJEo647KKT47L0PeQs1SIyMRGOlO/E 8uJKXjqfEm1dlbevhoUa0ltV/bCc5r9MWnpPAQZLnJFEhxnL7p9jXbSuxe/kmIQZ8ikjoy2s CW9kkUYsZNnuRKzoJbxsMh3+1GCRTPnHrLo7v896FJtDms248+jHLY4vPTjYb+4dvUmYmIPl AjCtBqnhts454IGrhbAdS8YdfYcTH/imSk/swGTGOwSvniYRLATXGH0U7HH/a5eHPcjyRUq+ +5Lupnb1S8jEgVB+LpmuQE7aVMvEEB2OGhzBE2OZID/CCSaDhHCMj9A5TA+v34jKIY7CeiQ3 smvoHkiD1RYvD7Ftw+EJfUbLdpIF/8UPv7RlJmApCrtIvXablY1BrULEi+sGBWuhFT0gsj9M w6FjIscvv4rn5wt6Es2w+R3NkJUCuT+FVto8otQD30xFkubON7cPU0PieGNYn2+ZT9T3LKUn dd24nM4Q8r0UU7a0E/K/tmWUrvdpZkD3OKq0egrGsQAAAAAAAA== --------------ms050604040905000400030000--