From owner-freebsd-stable@freebsd.org Sat Mar 20 03:00:45 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 51A655B7858 for ; Sat, 20 Mar 2021 03:00:45 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F2QW110lYz4pXh for ; Sat, 20 Mar 2021 03:00:45 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: by mailman.nyi.freebsd.org (Postfix) id 226345B79CC; Sat, 20 Mar 2021 03:00:45 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 222A35B7A18 for ; Sat, 20 Mar 2021 03:00:45 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from mail02.asahi-net.or.jp (mail02.asahi-net.or.jp [202.224.55.14]) by mx1.freebsd.org (Postfix) with ESMTP id 4F2QVz3dm8z4pHC; Sat, 20 Mar 2021 03:00:42 +0000 (UTC) (envelope-from ota@j.email.ne.jp) Received: from vmware12.advok.com (cpe-184-152-96-96.nj.res.rr.com [184.152.96.96]) (Authenticated sender: NR2Y-OOT) by mail02.asahi-net.or.jp (Postfix) with ESMTPSA id 039D3797E3; Sat, 20 Mar 2021 12:00:40 +0900 (JST) Date: Fri, 19 Mar 2021 23:01:12 -0400 From: Yoshihiro Ota To: Andriy Gapon Cc: stable@freebsd.org Subject: Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA Message-Id: <20210319230112.e327d1c69197b73244fd89d6@j.email.ne.jp> In-Reply-To: References: <20210306130913.dae1fb546e68bcaec882cdbe@j.email.ne.jp> <9f9db150-ee5d-e74e-7f24-74d1fa687a48@FreeBSD.org> <20210307222441.02755e7396cd329edd15df15@j.email.ne.jp> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.32; i386-portbld-freebsd12.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4F2QVz3dm8z4pHC X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ota@j.email.ne.jp designates 202.224.55.14 as permitted sender) smtp.mailfrom=ota@j.email.ne.jp X-Spamd-Result: default: False [-2.80 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:202.224.55.0/24]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; RCVD_NO_TLS_LAST(0.10)[]; RECEIVED_SPAMHAUS_PBL(0.00)[184.152.96.96:received]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[202.224.55.14:from]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:4685, ipnet:202.224.32.0/19, country:JP]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_IN_DNSWL_LOW(-0.10)[202.224.55.14:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[email.ne.jp]; SPAMHAUS_ZRD(0.00)[202.224.55.14:from:127.0.2.255]; RWL_MAILSPIKE_POSSIBLE(0.00)[202.224.55.14:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2021 03:00:45 -0000 On Tue, 9 Mar 2021 11:24:53 +0200 Andriy Gapon wrote: > On 08/03/2021 05:24, Yoshihiro Ota wrote: > > On Sun, 7 Mar 2021 00:09:33 +0200 > > Andriy Gapon wrote: > > > >> On 06/03/2021 20:09, Yoshihiro Ota wrote: > >>> Hi all, > >>> > >>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one. > >>> > >>> After upgrading one in VMWare, 'zfs mount -a' hangs the system. > >>> I don't have boottime zfs mount on nor don't have zfsroot. > >>> I just simply ran install world/kernel and mergemaster. > >> > >> Please use procstat -kk to capture a kernel stack trace of the hung process. > > > > Actually, spining was 'kldload zfs'. > > Console doesn't response but ping and sshd sessions still work. > > procstat output is below. > > In addition, this doesn't happen to systems that I've been following 13-CURRENT > > but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC. > > > > > > # procstat -kk 1049 > > PID TID COMM TDNAME KSTACK > > 1049 100215 kldload - spa_init+0xc6 zfs_kmod_init+0x1a > > zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab kern_kldload+0xc1 > > sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 > > > > If you could use kgdb to find out what source code line spa_init+0xc6 > corresponds to that may help to see what's going on. > It look me a while to get kgdb working properly. At last, I got the output. It looks it is spining on a mutex. I have few other machines run the same kernel but they can load zfs.ko. It is only vmware VM that spins with 'kldload zfs'. vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-portbld-freebsd13.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "helpType "apropos word" to search for commands related to "word"... Reading symbols from zfs.ko.debug... (kgdb) info line *spa_init+0xc6 Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c" starts at address 0x2b0461 and ends at 0x2b0467 . (kgdb) void spa_init(spa_mode_t mode) { mutex_init(&spa_namespace_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&spa_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 2345 Hiro