From owner-freebsd-stable@freebsd.org Sat Nov 30 09:21:35 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DD6461CD8EF for ; Sat, 30 Nov 2019 09:21:35 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47Q5V70YGtz3PnX for ; Sat, 30 Nov 2019 09:21:34 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xAU9HV2I086497 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sat, 30 Nov 2019 09:17:31 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xAU9HStX066035 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Sat, 30 Nov 2019 16:17:28 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: ZFS deadlock To: FreeBSD stable References: <2a1d0c98-03a9-66e4-4602-7f81c28cd34e@grosbein.net> From: Eugene Grosbein Message-ID: Date: Sat, 30 Nov 2019 16:17:23 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <2a1d0c98-03a9-66e4-4602-7f81c28cd34e@grosbein.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Flag: YES X-Spam-Status: Yes, score=6.0 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, HELO_MISC_IP,LOCAL_FROM,RDNS_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.7 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after Received: * date * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains * 1.1 HELO_MISC_IP Looking for more Dynamic IP Relays X-Spam-Level: ****** X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on eg.sd.rdtc.ru X-Rspamd-Queue-Id: 47Q5V70YGtz3PnX X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [1.31 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.93)[-0.933,0]; SPAM_FLAG(5.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.981,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[grosbein.net]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-1.68)[ip: (-4.44), ipnet: 2a01:4f8::/29(-2.35), asn: 24940(-1.58), country: DE(-0.01)]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Nov 2019 09:21:35 -0000 30.11.2019 15:48, Eugene Grosbein wrote: > Hi! > > I have RAIDZ1 with five GELI-encrypted SSDs da[2-6].eli (non-boot pool). > > I've exported the pool, destroyed da2.eli then successfully imported pool back in degraded state. > Then I've mounted some file systems successfully but zfs mount for next one hung on [tx->tx_sync_done_cv] > for 4400 seconds and counting. > > # procstat -kk -L 55464 > PID TID COMM TDNAME KSTACK > 55464 102422 zfs - mi_switch+0xeb sleepq_wait+0x2c _cv_wait+0x16e txg_wait_synced+0xa5 dmu_tx_assign+0x48 zfs_rmnode+0x122 zfs_freebsd_reclaim+0x4e VOP_RECLAIM_APV+0x80 vgonel+0x213 vrecycle+0x46 zfs_freebsd_inactive+0xd VOP_INACTIVE_APV+0x80 vinactive+0xf0 vputx+0x2c3 zfs_unlinked_drain+0x1b8 zfsvfs_setup+0x5e zfs_mount+0x5f5 vfs_domount+0x573 > > It looks like deadlock for me. > > What can I do to resolve this? FreeBSD 11.3-STABLE/amd64 r354667. "zfs mount" just completed successfully after trim -f /dev/da2 (3.5TB) finished in 76 minutes.