From owner-freebsd-stable@freebsd.org Fri Jul 31 15:44:21 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DD8EA37BEE3 for ; Fri, 31 Jul 2020 15:44:21 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BJBR85WSrz4tN1 for ; Fri, 31 Jul 2020 15:44:20 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:2c4b:9d7c:5c5a:c05a] ([IPv6:2607:f3e0:0:4:2c4b:9d7c:5c5a:c05a]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 06VFiJOq023441 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 31 Jul 2020 11:44:20 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: zfs meta data slowness To: Eugene Grosbein , FreeBSD-STABLE Mailing List References: <1949194763.1.1595250243575@localhost> <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> <9ddce60a-9d91-fa69-0e11-54342cb67890@grosbein.net> <2267c96f-b988-df1e-887b-77dc9b2a3e32@sentex.net> <097a63f1-3b25-3916-cdb9-1d4cf0d088fd@grosbein.net> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: <306e7adc-10a0-8f1a-d2b7-12d16ba91557@sentex.net> Date: Fri, 31 Jul 2020 11:44:20 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <097a63f1-3b25-3916-cdb9-1d4cf0d088fd@grosbein.net> Content-Language: en-US X-Rspamd-Queue-Id: 4BJBR85WSrz4tN1 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.57 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.02)[-1.024]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; NEURAL_HAM_LONG(-1.01)[-1.010]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; HAS_ATTACHMENT(0.00)[]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[sentex.net]; NEURAL_SPAM_SHORT(0.46)[0.465]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2020 15:44:21 -0000 On 7/22/2020 9:26 AM, Eugene Grosbein wrote: > > It's hard to read due to wrapping plus, it is truncated. > Maybe additional flag -d3 or similar will help, combined with dedirection < /dev/null > top.out > so it won't use size of your terminal to wrap/truncate output. > > Also, make sure you invoke top while "zfs" command is running. > Also, procstat -kk for pid of "zfs" command would be useful (but may occur pretty long). > > I suppose it blocks waiting for some kernel lock and procstat would show details. .txt file is attached. I also ran procstat -kk in a loop for a bit while it was running. In this case the list too just under 5min to run. time /sbin/zfs list -Hp -oname,creation -S creation -t snapshot > /var/log/snapshots.out 2.311u 24.963s 4:55.13 9.2%     71+179k 2706576+26io 0pf+0w Ideally, I would like to bias slightly ARC to hold this meta data a little longer than evict it out of cache so fast.  I put in an nvme disk hoping that would help, but it makes no difference or doesnt seem to anyways.     ---Mike