From owner-freebsd-stable@freebsd.org Sat Mar 20 16:27:44 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 84B7E5ADFB2 for ; Sat, 20 Mar 2021 16:27:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4F2mQ83L1zz4jps for ; Sat, 20 Mar 2021 16:27:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 70AEE5ADF57; Sat, 20 Mar 2021 16:27:44 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 707485AE1A2 for ; Sat, 20 Mar 2021 16:27:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F2mQ811zNz4jhZ for ; Sat, 20 Mar 2021 16:27:43 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from [192.168.0.88] (unknown [195.64.148.76]) (Authenticated sender: andriy.gapon@uabsd.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 1C660200003; Sat, 20 Mar 2021 16:27:39 +0000 (UTC) Subject: Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA To: Yoshihiro Ota Cc: stable@freebsd.org References: <20210306130913.dae1fb546e68bcaec882cdbe@j.email.ne.jp> <9f9db150-ee5d-e74e-7f24-74d1fa687a48@FreeBSD.org> <20210307222441.02755e7396cd329edd15df15@j.email.ne.jp> <20210319230112.e327d1c69197b73244fd89d6@j.email.ne.jp> From: Andriy Gapon Message-ID: <8b11fad4-5b6e-cf4a-0cde-8a8feff9705e@FreeBSD.org> Date: Sat, 20 Mar 2021 18:27:38 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: <20210319230112.e327d1c69197b73244fd89d6@j.email.ne.jp> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4F2mQ811zNz4jhZ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2021 16:27:44 -0000 On 20/03/2021 05:01, Yoshihiro Ota wrote: > On Tue, 9 Mar 2021 11:24:53 +0200 > Andriy Gapon wrote: > >> On 08/03/2021 05:24, Yoshihiro Ota wrote: >>> On Sun, 7 Mar 2021 00:09:33 +0200 >>> Andriy Gapon wrote: >>> >>>> On 06/03/2021 20:09, Yoshihiro Ota wrote: >>>>> Hi all, >>>>> >>>>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one. >>>>> >>>>> After upgrading one in VMWare, 'zfs mount -a' hangs the system. >>>>> I don't have boottime zfs mount on nor don't have zfsroot. >>>>> I just simply ran install world/kernel and mergemaster. >>>> >>>> Please use procstat -kk to capture a kernel stack trace of the hung process. >>> >>> Actually, spining was 'kldload zfs'. >>> Console doesn't response but ping and sshd sessions still work. >>> procstat output is below. >>> In addition, this doesn't happen to systems that I've been following 13-CURRENT >>> but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC. >>> >>> >>> # procstat -kk 1049 >>> PID TID COMM TDNAME KSTACK >>> 1049 100215 kldload - spa_init+0xc6 zfs_kmod_init+0x1a >>> zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab kern_kldload+0xc1 >>> sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 >>> >> >> If you could use kgdb to find out what source code line spa_init+0xc6 >> corresponds to that may help to see what's going on. >> > > It look me a while to get kgdb working properly. > At last, I got the output. > It looks it is spining on a mutex. > > I have few other machines run the same kernel but they can load zfs.ko. > It is only vmware VM that spins with 'kldload zfs'. > > vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug > GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD] > Copyright (C) 2020 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "i386-portbld-freebsd13.0". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . > > For help, type "helpType "apropos word" to search for commands related to "word"... > Reading symbols from zfs.ko.debug... > (kgdb) info line *spa_init+0xc6 > Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c" > starts at address 0x2b0461 > and ends at 0x2b0467 . > (kgdb) > > void > spa_init(spa_mode_t mode) > { > mutex_init(&spa_namespace_lock, NULL, MUTEX_DEFAULT, NULL); > mutex_init(&spa_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 2345 > It would highly unusual (and unexplainable) for a thread to get stuck here. Are you sure that your source tree matches the binary? Can you disassemble the function to check instructions at and near 0xc6 offset? -- Andriy Gapon