From owner-freebsd-stable@freebsd.org Wed Oct 14 10:00:46 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 17EE5433AFD for ; Wed, 14 Oct 2020 10:00:46 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4CB7G56PFyz3Vdp for ; Wed, 14 Oct 2020 10:00:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.nyi.freebsd.org (Postfix) id DB9E1433D5D; Wed, 14 Oct 2020 10:00:45 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DB651433AFC for ; Wed, 14 Oct 2020 10:00:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10G.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CB7G341cmz3Vxm for ; Wed, 14 Oct 2020 10:00:43 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10g.digiware.nl (localhost.digiware.nl [127.0.0.1]) by router10G.digiware.nl (Postfix) with ESMTP id 8B5C569997; Wed, 14 Oct 2020 12:00:34 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.com Received: from router10G.digiware.nl ([127.0.0.1]) by router10g.digiware.nl (router10g.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n5tRklKNTFRh; Wed, 14 Oct 2020 12:00:33 +0200 (CEST) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by router10G.digiware.nl (Postfix) with ESMTPSA id BF9B169996 for ; Wed, 14 Oct 2020 12:00:33 +0200 (CEST) To: "stable@freebsd.org" From: Willem Jan Withagen Subject: 12.2-RC2 crashing on leftover ZFS cache/log on SSD Message-ID: <8467b594-2f27-4b6c-b980-56b67860843a@digiware.nl> Date: Wed, 14 Oct 2020 12:00:31 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.12.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 4CB7G341cmz3Vxm X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-1.39 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; HFILTER_HELO_IP_A(1.00)[router10g.digiware.nl]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.03)[-1.034]; RCVD_COUNT_THREE(0.00)[4]; HFILTER_HELO_NORES_A_OR_MX(0.30)[router10g.digiware.nl]; NEURAL_HAM_SHORT(-0.38)[-0.382]; DMARC_NA(0.00)[digiware.nl]; TO_DN_EQ_ADDR_ALL(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.972]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 10:00:46 -0000 Hi, I managed myself in a somewhat weird situation on a new Epyc server. It has 2x 4T spinning HDD and 2x intel SSD. This is what I did:  - Installed 12.1-RELASE from a USB stick    - Put the zroot on the 2 HHDs in mirror  - Downloaded 12/stable, compiled and installed that.  - Then I added cache and log from  the SSDs (2 gparts)    to the zroot pool. caches as stripe and log in mirror. That worked fine. Only I found out that I had stable, instead of RC2 which I actually wanted since upgrading with freebsd-upgrade is a lot easiers then when 12.2-RELASE is out. So I fetch 12.2-RC2 and installed that on the 2 rusty spinners. But now the SSDs have leftover ZFS stuff on it that panics the 12.2-RC2. Simple soluction is to remover the SSDs from their trays, boot and reinsert the SSDs. That works. I'll be able to clean the SSDs and redo cache and log stuff. There are 2 things with the panic:  - It flies by so fast that I had to film it to actually get the panic message.    no kernel debugger    No 30/60 secs before autoboot    Is this a seeting that has changed?     debug.debugger_on_panic=1 does not seem to work....  - The panic itself:    panic: solaris assert: nvlist_lookup_uint64(configs[i], ZPOOL_CONFIG_POOL_TXG,      , &txg) == 0, file /usr/src/sys/cddl/contribute/opensolaris/uts/common/fs/zfs/spa.c      line 5636 Now this could be a case of wrong order and 12/stable has tings in the pool that 12.2-RC2 does not like. And should just reuse the SSDs. Or it is a real bug, and somebody wants to take a look at it? When we get into the debugger we could look at more data.... I have a picture of the stack trace, and will leave the system for a day as it is. It there are no takers, than I'll fix things tomorrow and put the system into production. --WjW