Date: Fri, 19 Mar 2021 23:01:12 -0400 From: Yoshihiro Ota <ota@j.email.ne.jp> To: Andriy Gapon <avg@FreeBSD.org> Cc: stable@freebsd.org Subject: Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA Message-ID: <20210319230112.e327d1c69197b73244fd89d6@j.email.ne.jp> In-Reply-To: <a6e7fd23-34b4-19bb-a57c-034a8abc3433@FreeBSD.org> References: <20210306130913.dae1fb546e68bcaec882cdbe@j.email.ne.jp> <9f9db150-ee5d-e74e-7f24-74d1fa687a48@FreeBSD.org> <20210307222441.02755e7396cd329edd15df15@j.email.ne.jp> <a6e7fd23-34b4-19bb-a57c-034a8abc3433@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 9 Mar 2021 11:24:53 +0200 Andriy Gapon <avg@FreeBSD.org> wrote: > On 08/03/2021 05:24, Yoshihiro Ota wrote: > > On Sun, 7 Mar 2021 00:09:33 +0200 > > Andriy Gapon <avg@FreeBSD.org> wrote: > > > >> On 06/03/2021 20:09, Yoshihiro Ota wrote: > >>> Hi all, > >>> > >>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one. > >>> > >>> After upgrading one in VMWare, 'zfs mount -a' hangs the system. > >>> I don't have boottime zfs mount on nor don't have zfsroot. > >>> I just simply ran install world/kernel and mergemaster. > >> > >> Please use procstat -kk to capture a kernel stack trace of the hung process. > > > > Actually, spining was 'kldload zfs'. > > Console doesn't response but ping and sshd sessions still work. > > procstat output is below. > > In addition, this doesn't happen to systems that I've been following 13-CURRENT > > but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC. > > > > > > # procstat -kk 1049 > > PID TID COMM TDNAME KSTACK > > 1049 100215 kldload - spa_init+0xc6 zfs_kmod_init+0x1a > > zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab kern_kldload+0xc1 > > sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 > > > > If you could use kgdb to find out what source code line spa_init+0xc6 > corresponds to that may help to see what's going on. > It look me a while to get kgdb working properly. At last, I got the output. It looks it is spining on a mutex. I have few other machines run the same kernel but they can load zfs.ko. It is only vmware VM that spins with 'kldload zfs'. vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>; This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-portbld-freebsd13.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "helpType "apropos word" to search for commands related to "word"... Reading symbols from zfs.ko.debug... (kgdb) info line *spa_init+0xc6 Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c" starts at address 0x2b0461 <spa_init+193> and ends at 0x2b0467 <spa_init+199>. (kgdb) void spa_init(spa_mode_t mode) { mutex_init(&spa_namespace_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&spa_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 2345 Hiro
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20210319230112.e327d1c69197b73244fd89d6>