From owner-freebsd-stable@freebsd.org Wed Sep 9 12:59:20 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 710493D04BF for ; Wed, 9 Sep 2020 12:59:20 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [128.127.146.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.norma.perm.ru", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmhtH2x9Cz4Dqc for ; Wed, 9 Sep 2020 12:59:19 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (net147.234.188-221.ertelecom.ru [188.234.147.221] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id 089CxjC4039238 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Wed, 9 Sep 2020 17:59:45 +0500 (+05) (envelope-from emz@norma.perm.ru) Subject: Re: spa_namespace_lock and concurrent zfs commands To: freebsd-stable@freebsd.org References: From: "Eugene M. Zheganin" Message-ID: <0d113463-be6d-5c6b-3cfc-867a0891e7d8@norma.perm.ru> Date: Wed, 9 Sep 2020 17:59:10 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Server: elf.hq.norma.perm.ru X-Rspamd-Queue-Id: 4BmhtH2x9Cz4Dqc X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=perm.ru; spf=pass (mx1.freebsd.org: domain of emz@norma.perm.ru designates 128.127.146.8 as permitted sender) smtp.mailfrom=emz@norma.perm.ru X-Spamd-Result: default: False [5.56 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; NEURAL_SPAM_SHORT(0.27)[0.271]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(0.00)[+a]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; NEURAL_SPAM_MEDIUM(0.99)[0.995]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; HFILTER_HELO_IP_A(1.00)[elf.hq.norma.perm.ru]; RBL_VIRUSFREE_BOTNET(2.00)[128.127.146.8:from]; DMARC_POLICY_ALLOW(0.00)[perm.ru,none]; HFILTER_HELO_NORES_A_OR_MX(0.30)[elf.hq.norma.perm.ru]; NEURAL_SPAM_LONG(0.99)[0.992]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:57401, ipnet:128.127.146.0/24, country:RU]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-stable]; RECEIVED_SPAMHAUS_PBL(0.00)[188.234.147.221:received] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 12:59:20 -0000 On 09.09.2020 17:29, Eugene M. Zheganin wrote: > Hello, > > I'm using sort of FreeBSD ZFS appliance with custom API, and I'm > suffering from huge timeouts when large (dozens, actually) of > concurrent zfs/zpool commands are issued > (get/create/destroy/snapshot/clone mostly). > > Are there any tunables that could help mitigate this ? > > Once I took part in reporting the > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 , but the > issue that time got resolved somehow. Now I have another set of > FreeBSD SANs and it;'s back. I've read the > https://wiki.freebsd.org/AndriyGapon/AvgZFSLocking and I realize this > probably doesn't have a quick solution, but still... > This actually looks like this (sometime it takes several [dozens of] minutes): root@cg-mr-prod-stg09:/usr/ports/sysutils/smartmontools # zfs get volmode load: 3.58š cmd: zfs 70231 [spa_namespace_lock] 16.38r 0.00u 0.00s 0% 3872k load: 3.58š cmd: zfs 70231 [spa_namespace_lock] 16.59r 0.00u 0.00s 0% 3872k load: 3.58š cmd: zfs 70231 [spa_namespace_lock] 16.76r 0.00u 0.00s 0% 3872k load: 3.58š cmd: zfs 70231 [spa_namespace_lock] 16.90r 0.00u 0.00s 0% 3872k load: 3.58š cmd: zfs 70231 [spa_namespace_lock] 17.04r 0.00u 0.00s 0% 3872k load: 3.58š cmd: zfs 70231 [spa_namespace_lock] 17.17r 0.00u 0.00s 0% 3872k root@cg-mr-prod-stg09:~ # ps ax | grep volmode 70231š 5š D+ššššššš 0:00.00 zfs get volmode 70233š 6š S+ššššššš 0:00.00 grep volmode root@cg-mr-prod-stg09:~ # procstat -kk 70231 š PIDššš TID COMMššššššššššššššš TDNAME KSTACK 70231 101598 zfsšššššššššššššššš - mi_switch+0xe2 sleepq_wait+0x2c _sx_xlock_hard+0x459 spa_all_configs+0x1aa zfs_ioc_pool_configs+0x19 zfsdev_ioctl+0x72e devfs_ioctl+0xad VOP_IOCTL_APV+0x7c vn_ioctl+0x16a devfs_ioctl_f+0x1f kern_ioctl+0x2be sys_ioctl+0x15d amd64_syscall+0x364 fast_syscall_common+0x101 root@cg-mr-prod-stg09:~ # Thanks. Eugene.