Date: Sat, 20 Mar 2021 18:27:38 +0200 From: Andriy Gapon <avg@FreeBSD.org> To: Yoshihiro Ota <ota@j.email.ne.jp> Cc: stable@freebsd.org Subject: Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA Message-ID: <8b11fad4-5b6e-cf4a-0cde-8a8feff9705e@FreeBSD.org> In-Reply-To: <20210319230112.e327d1c69197b73244fd89d6@j.email.ne.jp> References: <20210306130913.dae1fb546e68bcaec882cdbe@j.email.ne.jp> <9f9db150-ee5d-e74e-7f24-74d1fa687a48@FreeBSD.org> <20210307222441.02755e7396cd329edd15df15@j.email.ne.jp> <a6e7fd23-34b4-19bb-a57c-034a8abc3433@FreeBSD.org> <20210319230112.e327d1c69197b73244fd89d6@j.email.ne.jp>
next in thread | previous in thread | raw e-mail | index | archive | help
On 20/03/2021 05:01, Yoshihiro Ota wrote: > On Tue, 9 Mar 2021 11:24:53 +0200 > Andriy Gapon <avg@FreeBSD.org> wrote: > >> On 08/03/2021 05:24, Yoshihiro Ota wrote: >>> On Sun, 7 Mar 2021 00:09:33 +0200 >>> Andriy Gapon <avg@FreeBSD.org> wrote: >>> >>>> On 06/03/2021 20:09, Yoshihiro Ota wrote: >>>>> Hi all, >>>>> >>>>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one. >>>>> >>>>> After upgrading one in VMWare, 'zfs mount -a' hangs the system. >>>>> I don't have boottime zfs mount on nor don't have zfsroot. >>>>> I just simply ran install world/kernel and mergemaster. >>>> >>>> Please use procstat -kk to capture a kernel stack trace of the hung process. >>> >>> Actually, spining was 'kldload zfs'. >>> Console doesn't response but ping and sshd sessions still work. >>> procstat output is below. >>> In addition, this doesn't happen to systems that I've been following 13-CURRENT >>> but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC. >>> >>> >>> # procstat -kk 1049 >>> PID TID COMM TDNAME KSTACK >>> 1049 100215 kldload - spa_init+0xc6 zfs_kmod_init+0x1a >>> zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab kern_kldload+0xc1 >>> sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 >>> >> >> If you could use kgdb to find out what source code line spa_init+0xc6 >> corresponds to that may help to see what's going on. >> > > It look me a while to get kgdb working properly. > At last, I got the output. > It looks it is spining on a mutex. > > I have few other machines run the same kernel but they can load zfs.ko. > It is only vmware VM that spins with 'kldload zfs'. > > vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug > GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] > Copyright (C) 2020 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>; > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "i386-portbld-freebsd13.0". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > <https://www.gnu.org/software/gdb/bugs/>. > Find the GDB manual and other documentation resources online at: > <http://www.gnu.org/software/gdb/documentation/>. > > For help, type "helpType "apropos word" to search for commands related to "word"... > Reading symbols from zfs.ko.debug... > (kgdb) info line *spa_init+0xc6 > Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c" > starts at address 0x2b0461 <spa_init+193> > and ends at 0x2b0467 <spa_init+199>. > (kgdb) > > void > spa_init(spa_mode_t mode) > { > mutex_init(&spa_namespace_lock, NULL, MUTEX_DEFAULT, NULL); > mutex_init(&spa_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 2345 > It would highly unusual (and unexplainable) for a thread to get stuck here. Are you sure that your source tree matches the binary? Can you disassemble the function to check instructions at and near 0xc6 offset? -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8b11fad4-5b6e-cf4a-0cde-8a8feff9705e>