From owner-freebsd-stable@freebsd.org Wed Jul 22 13:02:30 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CC2D937C381 for ; Wed, 22 Jul 2020 13:02:30 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [IPv6:2607:f3e0:0:3::19]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "pyroxene.sentex.ca", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBbGZ1tKhz49JN for ; Wed, 22 Jul 2020 13:02:30 +0000 (UTC) (envelope-from mike@sentex.net) Received: from [IPv6:2607:f3e0:0:4:8125:6e09:8b8c:7875] ([IPv6:2607:f3e0:0:4:8125:6e09:8b8c:7875]) by pyroxene2a.sentex.ca (8.15.2/8.15.2) with ESMTPS id 06MD2RsQ069415 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 22 Jul 2020 09:02:27 -0400 (EDT) (envelope-from mike@sentex.net) Subject: Re: zfs meta data slowness To: Eugene Grosbein , Ronald Klop , FreeBSD-STABLE Mailing List References: <1949194763.1.1595250243575@localhost> <975657af-ccac-bbd1-e22b-86270c624226@sentex.net> <9ddce60a-9d91-fa69-0e11-54342cb67890@grosbein.net> From: mike tancsa Autocrypt: addr=mike@sentex.net; keydata= mQENBFywzOMBCACoNFpwi5MeyEREiCeHtbm6pZJI/HnO+wXdCAWtZkS49weOoVyUj5BEXRZP xflV2ib2hflX4nXqhenaNiia4iaZ9ft3I1ebd7GEbGnsWCvAnob5MvDZyStDAuRxPJK1ya/s +6rOvr+eQiXYNVvfBhrCfrtR/esSkitBGxhUkBjOti8QwzD71JVF5YaOjBAs7jZUKyLGj0kW yDg4jUndudWU7G2yc9GwpHJ9aRSUN8e/mWdIogK0v+QBHfv/dsI6zVB7YuxCC9Fx8WPwfhDH VZC4kdYCQWKXrm7yb4TiVdBh5kgvlO9q3js1yYdfR1x8mjK2bH2RSv4bV3zkNmsDCIxjABEB AAG0HW1pa2UgdGFuY3NhIDxtaWtlQHNlbnRleC5uZXQ+iQFUBBMBCAA+FiEEmuvCXT0aY6hs 4SbWeVOEFl5WrMgFAlywzOYCGwMFCQHhM4AFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ eVOEFl5WrMhnPAf7Bf+ola0V9t4i8rwCMGvzkssGaxY/5zNSZO9BgSgfN0WzgmBEOy/3R4km Yn5KH94NltJYAAE5hqkFmAwK6psOqAR9cxHrRfU+gV2KO8pCDc6K/htkQcd/mclJYpCHp6Eq EVJOiAxcNaYuHZkeMdXDuvvI5Rk82VHk84BGgxIqIrhLlkguoPbXOOa+8c/Mpb1sRAGZEOuX EzKNC49+GS9gKW6ISbanyPsGEcFyP7GKMzcHBPf3cPrewZQZ6gBoNscasL6IJeAQDqzQAxbU GjO0qBSMRgnLXK7+DJlxrYdHGXqNbV6AYsmHJ6c2WWWiuRviFBqXinlgJ2FnYebZPAfWiQ== Message-ID: <2267c96f-b988-df1e-887b-77dc9b2a3e32@sentex.net> Date: Wed, 22 Jul 2020 09:02:27 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <9ddce60a-9d91-fa69-0e11-54342cb67890@grosbein.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4BBbGZ1tKhz49JN X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:3::19 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-0.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.91)[-0.911]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; NEURAL_HAM_LONG(-1.00)[-0.996]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[pyroxene2a.sentex.ca]; HFILTER_HELO_NORES_A_OR_MX(0.30)[pyroxene2a.sentex.ca]; DMARC_NA(0.00)[sentex.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.09)[-0.088]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 13:02:30 -0000 On 7/22/2020 1:29 AM, Eugene Grosbein wrote: > 22.07.2020 2:37, mike tancsa wrote: > >>> Something else special about the setup. >>> output of "top -b" >>> >> ports are right now being built in a VM, but the problem (zrepl hanging) >> and zfs list -t snapshots taking forever happens regardless >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve >> 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs >> 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrepl >> 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs >> 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs > You need top -SHPI last pid: 66981;  load averages:  3.44,  3.25,  3.34  up 1+22:49:34    08:58:06 2056 threads:  12 running, 1988 sleeping, 3 zombie, 53 waiting CPU 0:  7.7% user,  0.1% nice, 15.0% system,  0.7% interrupt, 76.5% idle CPU 1:  9.9% user,  0.1% nice, 16.6% system,  0.1% interrupt, 73.2% idle CPU 2: 10.0% user,  0.1% nice, 17.5% system,  0.4% interrupt, 71.9% idle CPU 3: 10.3% user,  0.1% nice, 21.2% system,  0.1% interrupt, 68.2% idle CPU 4:  9.7% user,  0.1% nice, 15.6% system,  0.4% interrupt, 74.3% idle CPU 5: 10.2% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.3% idle CPU 6:  9.7% user,  0.1% nice, 16.6% system,  0.5% interrupt, 73.1% idle CPU 7: 10.1% user,  0.1% nice, 21.3% system,  0.1% interrupt, 68.4% idle Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other      20G Compressed, 29G Uncompressed, 1.43:1 Ratio Swap: 20G Total, 20G Free   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND  4439 root        123   20  6167M  5856M CPU2     2  20.2H 100.00% bhyve{vcpu 0}  4439 root        122   20  6167M  5856M CPU3     3  20.2H  99.17% bhyve{vcpu 1}    11 root        155 ki31     0B   128K CPU0     0  35.9H  65.38% idle{idle: cpu0}    11 root        155 ki31     0B   128K CPU4     4  34.8H  63.38% idle{idle: cpu4} 66956 root         90    0    61M    54M CPU5     5   0:09  62.70% zfs    11 root        155 ki31     0B   128K CPU6     6  34.3H  58.06% idle{idle: cpu6}    11 root        155 ki31     0B   128K RUN      2  33.7H  54.49% idle{idle: cpu2}    11 root        155 ki31     0B   128K RUN      1  34.3H  53.76% idle{idle: cpu1}    11 root        155 ki31     0B   128K RUN      3  32.0H  53.47% idle{idle: cpu3}    11 root        155 ki31     0B   128K CPU7     7  32.0H  50.68% idle{idle: cpu7}    11 root        155 ki31     0B   128K RUN      5  32.0H  48.29% idle{idle: cpu5}    56 root        -12    -     0B  5168K -        1   5:49   9.67% zpool-zbackup2{zio_write_issue_3}    56 root        -12    -     0B  5168K -        3   5:48   9.57% zpool-zbackup2{zio_write_issue_5}    56 root        -12    -     0B  5168K -        4   5:48   9.47% zpool-zbackup2{zio_write_issue_4}    56 root        -12    -     0B  5168K -        7   5:48   9.47% zpool-zbackup2{zio_write_issue_2}    56 root        -12    -     0B  5168K -        0   5:49   9.38% zpool-zbackup2{zio_write_issue_1}    56 root        -12    -     0B  5168K -        1   5:48   9.38% zpool-zbackup2{zio_write_issue_0} 63392 root         23    0   729M    32M uwait    0   0:01   4.20% zrepl{zrepl}