From owner-freebsd-stable@freebsd.org Fri Aug 11 06:28:30 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 607A9DC563C; Fri, 11 Aug 2017 06:28:30 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BB5953538; Fri, 11 Aug 2017 06:28:29 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (net206-94.perm.ertelecom.ru [46.146.206.94] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id v7B6SMEH008220 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 11 Aug 2017 11:28:23 +0500 (YEKT) (envelope-from emz@norma.perm.ru) To: freebsd-fs@freebsd.org Cc: freebsd-stable From: "Eugene M. Zheganin" Subject: zfs listing and CPU Message-ID: Date: Fri, 11 Aug 2017 11:28:21 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Spamd-Result: default: False [1.50 / 25.00] RBL_SPAMHAUS_PBL(2.00)[94.206.146.46.zen.spamhaus.org : 127.0.0.10] HFILTER_HOSTNAME_UNKNOWN(2.50)[] BAYES_HAM(-3.00)[99.99%] DMARC_NA(0.00)[norma.perm.ru] MIME_GOOD(-0.10)[text/plain] R_DKIM_NA(0.00)[] R_SPF_SOFTFAIL(0.00)[~all] MID_RHS_MATCH_FROM(0.00)[] RECEIVED_SPAMHAUS(0.00)[94.206.146.46.zen.spamhaus.org] TO_DN_SOME(0.00)[] RCPT_COUNT_2(0.00)[] TO_MATCH_ENVRCPT_ALL(0.00)[] FROM_HAS_DN(0.00)[] FROM_EQ_ENVFROM(0.00)[] RCVD_COUNT_1(0.00)[] ONCE_RECEIVED(0.10)[] X-Rspamd-Server: localhost X-Rspamd-Scan-Time: 0.93 X-Rspamd-Queue-ID: v7B6SMEH008220 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Aug 2017 06:28:30 -0000 Hi, Why does the zfs listing eat so much of the CPU ? last pid: 47151; load averages: 3.97, 6.35, 6.13 up 1+23:21:18 09:15:13 146 processes: 3 running, 142 sleeping, 1 waiting CPU: 0.0% user, 0.0% nice, 30.5% system, 0.3% interrupt, 69.2% idle Mem: 44M Active, 360M Inact, 37G Wired, 25G Free ARC: 32G Total, 14G MFU, 17G MRU, 312M Anon, 803M Header, 523M Other Swap: 32G Total, 185M Used, 32G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 32 155 ki31 0K 512K CPU0 0 1104.1 2666.23% idle 0 root 9880 -16 - 0K 154M swapin 11 314.0H 281.29% kernel 13 root 3 -8 - 0K 48K gread 0 20.9H 28.71% geom 47114 root 1 20 0 40432K 3840K db->db 4 0:05 26.84% zfs 47099 root 1 20 0 40432K 3840K zio->i 17 0:05 26.83% zfs 47106 root 1 20 0 40432K 3840K db->db 21 0:05 26.81% zfs 47150 root 1 20 0 40432K 3428K db->db 13 0:03 26.31% zfs 47141 root 1 20 0 40432K 3428K zio->i 28 0:03 26.31% zfs 47135 root 1 20 0 40432K 3312K g_wait 9 0:03 25.51% zfs 4 root 7 -16 - 0K 112K - 20 975:01 19.73% cam 5 root 2494 -8 - 0K 39952K arc_re 18 20.2H 18.58% zfskern 12 root 65 -60 - 0K 1040K WAIT 0 17.8H 15.64% intr 22 root 2 -16 - 0K 32K psleep 3 66:34 7.31% pagedaemon 590 root 10 -16 - 0K 160K - 21 177:02 2.96% ctl [...] This is from winter 2017 11-STABLE (r310734), one of the 'zfs'es is cloning, and all the others are 'zfs list -t all'. I have like 25 gigs of free RAM, do I have any chance of speeding this up using may be some caching or some sysctl tuning ? We are using a simple ZFS web API that may issue concurrent or sequential listing requests, so as you can see they sometimes do stack. Thanks. Eugene.