From owner-freebsd-fs@freebsd.org Mon Apr 20 11:11:45 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF69A2BF962 for ; Mon, 20 Apr 2020 11:11:45 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 495PCj5ryYz48v8 for ; Mon, 20 Apr 2020 11:11:45 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 9BE508175 for ; Mon, 20 Apr 2020 11:11:45 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.230] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 65DFB8095 for ; Mon, 20 Apr 2020 14:11:44 +0300 (MSK) To: freebsd-fs Reply-To: lev@FreeBSD.org From: Lev Serebryakov Subject: Two ZFS pools on one system: load on one pool badly affects other one Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGwBBMBCABDAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEWIQT5bRygtfQxi2dLMwrqsDxYv9xHjwUCW/03kQUJDwW3xgAh CRDqsDxYv9xHjxYhBPltHKC19DGLZ0szCuqwPFi/3EePHxkP+wWNrAyks2fQctY/Gl7TMh+Y Q9uX0hAuZ2Vvi0LswBl/R85SsS7IvI9b3ogOWA8CAlHAxkvgH6sWrwRTNcCPS1MzulYxS914 0CSkdwwbv1JyDOOWYU6s8PfT9+BZr+9eNXStmEdEL5XcA1k2YncQtlR3m+oLkqlAOtteZWti pitMIX9BGYIVKyl0t0RnIx+m/QPVGU9gu02j0I3NSRnKQPyFxZqYK0nPBu+FKaEhIAqdKPOv GL4/ijansdiWO3mXy18G0Mkr8yYRSidpGgXGY6lmGzQ3R6ZS30bLI8DkskOOvfErwhZv5dH5 w4+JH5sQ7bIL5HEXs//ZU9UzMdQwcURMjcFfKGyfL0hSLRqzP8m7SL1k9ZL161OQ6C5zVO/M bSCmeeLkbfOj1NW1ZIv6UjVVWE/LS4+gqg/04C+Y24vj+7vMpBVEevdwmIEdmVciFudklcnN omuocb29GKbquRZRDGiE+mhqkwmp5e59AnePp3+AvkewSCsXlR1sfjEP/Tn5OsYerJ7eAAOj DjxO374TAqJG5ftW4BA/nVmx9FGKV1/A9Yc1UuH6LdQfLf7pmTck1Cxg4kdH+3qKGD63sAR0 Wh27XDjnBKXJUN7J+nctWMZJMvw4OhTXdTyVhWt6USKEzw8M5plY4sFqxBEAe8igQXlq1Xjd ISV7wYhT4l3FzsFNBFKbGksBEAC0a9wfjo2P3JyT7Lc+QlbFVshGbSbazb4ma7QYG5IZZD5v fLBFkePoG6cnrn3WCXp4A43hszAynCwe4eXyAkv4+gPF3ZSeNE5Wz3zYG+jh2nm2iGCkyaVy kfbA+2chor2DKH5tHpuNMBlF+wSJHZKJmlo/sFIktAnV1NBVg4/cL+9/hIpvl82cl3hYCD7/ e7/qRE+w38CpAAzn65FvbODn7xlY3fsJt+cHPBJ4EBM9KnTwcce+F+72RQMZQEl7vIAwSRmL dgZHN0MFC533l62SVoKjT0eaOOIBrvesmojhWjfwugibXr+WRF/tGcW77Bxwe2eQLbEVESqW eMORxRxocx7Q7aACoHmf4G4U1Vzx7zUEfNfHjfjZeQVfAURf/MoUelZSW/BmMIfKCg3lRlWA t+Pq2h2UADPVqAZze45beE/c8z8LZsOZiGoRhYL8NSg6+ziLTdmYLWdtFGAuZhqOtNp5h6tG j21OksBotcaIa5YjbCmmnImIjGlSBkUKvIhq/RXth5b2gNwaQdu+Yv4AlZVHRsuVywL/skDF L5+We11bDK6MQ5PzvmntRJcgbyoisn1hiV04OV1LpJJMkJn1j8VlBqDQNT/z+BjB0ru/0anv +5uLj7v0ck06rEo4yiXT/ZAcBM76j7V7FaGbkoba6bUUCQ2H5YYBOKpikjCnpwARAQABwsGT BBgBCAAmAhsMFiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlv9N7IFCQ8Ft+cAIQkQ6rA8WL/c R48WIQT5bRygtfQxi2dLMwrqsDxYv9xHj3CnD/9btCtkcphRYRUe08tUyVwzV/syDCdiUhF7 8jqDKTC+3zuyrFJi7t4fF9follHYz1Ri5RixxJHnuDFcq7ZTOprPYqO8QhckLAJOy5dmORDX 2guEA+y5zDYBwwjpio9dtnuE7QyHyMx4nMPq8O/HfO+6dDEZChkrGvcG9FTI7s0JhsDs3xxw jcROZ2OP0lNu2571ZpR4YuzMUOIhOaQBIF2wrTvLjKUsAnNQYK9gsFTeDHRsE4HZLxJvEdiZ CWN7COi9un4xtP4Khc3Fmn6ANEyh0bIgx1Eii2RGINuA2XRVYhPRJLUZRSVQcrND9k9S+m+T oaqz9JgFLusFA1KhdeYnE1bojpq1U1bsmEicLW2QfEGVumKTgUrTsno0cVPH73KDILFvHA0D 8t4UaQveRTRUVdHZ02IBVt655Q8Xq1TkHJ7l+2Ckso5IBujWD74QpSRzzffn/ihhEExwYSTj FSs0C/OgU+EDZbcq2SWu4n1OGsW337/80HnJKVWBPAZYy4EmiyQSY05MG/fj9RA9Qi4TjFLD LrIf6dFAmiiIwWjlAKiyyUk+XDJXrc1L2VhcHqfdBY4I/qwV1YAI1QI4W/i6TstB1j0GwKa3 ZORwu4eahL5+9R6xBedhXZpCL0dyKuI8iPaC8npaOCJoL8+l4+KXR/PKt8b8kzIcvSpyCZii PQ== Organization: FreeBSD Message-ID: <802e3424-7edc-8e4a-b3e6-706c2b4a197e@FreeBSD.org> Date: Mon, 20 Apr 2020 14:11:43 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="Zet3zJYk89Xq3CLGp7azNq6UCA78T7C8e" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Apr 2020 11:11:46 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Zet3zJYk89Xq3CLGp7azNq6UCA78T7C8e Content-Type: multipart/mixed; boundary="iYes23KYdsk3U8P2CUBHNQPpD9CxedtYg" --iYes23KYdsk3U8P2CUBHNQPpD9CxedtYg Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable I have system with two ZFS pools. Both pools are RADIZ on HDDs (not SSDs= ), and don't share any storage hardware. One pool is "my private space" a= nd other one is storage for torrents. Private space contains mirrors of FreeBSD SVN, my home directory, etc. S= torage for torrents contains torrents. My home directory contains, among other data, working copies of FreeBSD = src and ports. When torrent-client is stopped, disk-depended operations on PRIVATE pool= like "svn up ~/FreeBSD/ports" starts immediately on cold caches. When torrent-client are running (and reads about 5-6MB/s in small chunks= ), "svn up ~/FreeBSD/ports" could wait up to 5 minutes before start outpu= t of updated paths. ^T shows this: load: 0.48 cmd: svn 18126 [zio->io_cv] 275.95r 3.14u 6.14s 2% 21388k Is it normal? I understand, that ARC is shared between pools, but it loo= ks like too much interaction: there are NO shared I/O resources (disks, c= ontrollers) between pools and these operations, but "svn" waits 60-300 se= conds for some condition variable! --=20 // Lev Serebryakov --iYes23KYdsk3U8P2CUBHNQPpD9CxedtYg-- --Zet3zJYk89Xq3CLGp7azNq6UCA78T7C8e Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAl6dg29fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R493Mg//fP1KvZSSdM1/O9ULUHlHSaC8NytA8GcZAkGKJ0/bnlXXPRkDBl0Yc4b5 7KSoK8b0k92rTPoFJHjOnVHA3un+1jCWunLmrU25dKL2yTBzefjcA3uOJVzgpTW5 6pqjecIN49An+iaFsRm7htrL2JTvCRVenhltd+Mwl02Uh0JPJinCGaJwAdWj+X3p qGSC0EjdjEaKp0sP6ReX8H6qgtEc9OnAUC79HjuOX9EDRXBKwKK9Rl8c9dYDL5VB Sw2IbETcKriHEMn+/Jwl0OsWsaE0McvY+c3MRdUejPH63R8qs0okS1LIwlsZuZFx 1HQ5vG45bVghQqGRrmeCRsB5q26/+/oKcQxGj/uItWRnlHqhoORtABAR/mlmFPWC v56WYf/1B02gKCOfSrV6s/AF+HYPn0rg67WDKXjeudBOtQxW9kzA4u9UZFaHlqDN N7NQ/PEYTPzrU2E+eYfyAO5/TCNtD1HJ9kt2lUBte2tLtF/0m9qN1xwjkPgNOLBZ dMDQJeVq3I6MiGtTYu6who1NAcq2Blg9fk45pn8x8d5RPGMPR4MUF4ZlscpTUiz0 ChP7GOlBBqR8UMGxavWMk9pptSAY1k0tJ5C+v/6oW/h1O8bqtwAwXTLm7YxMhpMJ 8s86e9D1i/cKtVHkGFDvK3hX6dVVGJeFoNiM0Uk5nf3wE5T/Heg= =HaJc -----END PGP SIGNATURE----- --Zet3zJYk89Xq3CLGp7azNq6UCA78T7C8e-- From owner-freebsd-fs@freebsd.org Tue Apr 21 16:39:50 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 93D912AB1D9 for ; Tue, 21 Apr 2020 16:39:50 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4968Rn1J9Jz4YBC; Tue, 21 Apr 2020 16:39:48 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-ej1-x634.google.com with SMTP id rh22so11478445ejb.12; Tue, 21 Apr 2020 09:39:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ePmGHeCRLAAFutEOdx4xXLIR17Ytdsd/uh9nEYtBBY0=; b=EM09cyUgMMkOn0eRHfbtdJOyGprJh1wuhuyvJlqMdxhpD/tdCs8gj+HDrJhGrsRLXk +NBOcS7M2WB4J/a4OqYYg6F8hA++pXakI7pZZcrdgZ/3aCbjkh7tBxfhqIDKQqbuF0Gp 1N1nqxEP9U5h+9Ajk+S7+oBd7RhtP6MrLRrQ6BXQtf66b7N8FSDpU+X+0epg+y1MtqML jpwjgzLYKu+7eWjQ0t+dTy/abgu+wH//EGg9ZL0RUN68rDfqwq70rz+CvDDo7FhgN8IJ aXCiCtZAubY+ICnKRrHYBtgJBq+42bjSdtnCF9H0Kmy0MZUDMcnST6jxQ+98czlhDLT/ PdKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ePmGHeCRLAAFutEOdx4xXLIR17Ytdsd/uh9nEYtBBY0=; b=WY+9sdLBqaVTfIgtwbi5z295gEb9qVdzcJRDe0ipvEVvaTosmkNuKP6JBahS5RdHKC 7D2vm13m5n3VqahFxEKVNoE+GwhDvdy0v7QMDTierhbnRqmHS/PhrLBjQfscOrqP0pYJ jo+7V6Czf4SyX/gZq7HXKlE+HaB2C/2GOhh4TMugGEwxRx54gIex6ktrpULmpELW8Cn9 wl0k95wY2oNt2/8UscuxA29/WqFmy90Gq3BwsAT4PcSd9PFccLeqpN2XwzDNSIJa807n tl1hgczhUpAE+2wjTf1RF9RlGXQPPl4PeA5g5hZEQy+aIp9SAberWxFEDeC1BTy9dv49 kfCg== X-Gm-Message-State: AGi0PuYvYtR1CFx0xa6DYa5nU33I/y1VoAXHtq+sK7d60i6pmxIiHnZg YAs8xh/4hey7ix8C3nAwPjOvdgibQTmGkhh61NV4CPk= X-Google-Smtp-Source: APiQypIGHoBJh+mbzKDAOMcZoGT65fXKTccjsG/iJztwvK/jxfJ2axoNusrSYysQVij4quUPaMhOuQXC+6K8IFqqkJg= X-Received: by 2002:a17:906:7804:: with SMTP id u4mr22616480ejm.328.1587487181917; Tue, 21 Apr 2020 09:39:41 -0700 (PDT) MIME-Version: 1.0 References: <802e3424-7edc-8e4a-b3e6-706c2b4a197e@FreeBSD.org> In-Reply-To: <802e3424-7edc-8e4a-b3e6-706c2b4a197e@FreeBSD.org> From: Zaphod Beeblebrox Date: Tue, 21 Apr 2020 12:39:30 -0400 Message-ID: Subject: Re: Two ZFS pools on one system: load on one pool badly affects other one To: lev@freebsd.org Cc: freebsd-fs X-Rspamd-Queue-Id: 4968Rn1J9Jz4YBC X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=EM09cyUg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2a00:1450:4864:20::634 as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ipnet: 2a00:1450::/32(-2.34), asn: 15169(-0.43), country: US(-0.05)]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2020 16:39:50 -0000 [ regarding zfs arc cache on two pools, where one pool can clear the cache of the other ] ... have you tried introducing an l2 cache on an SSD for the "private" pool? It seems to me that size of the l2 ARC are pool specific (where the division of memory in the l1 ARC is not). It doesn't surprise me that this happens. I have a large RAIZ2 pool for media (currently 40T, expanding to 60T shortly) on an otherwise general home server with 32G RAM. There is also a mirrored pool (root, home, a few things) on this machine for various reasons (like not booting from such a large raid) and things got overall better when I added a few hundred gig of l2 cache to each of the two pools. On Mon, Apr 20, 2020 at 7:11 AM Lev Serebryakov wrote: > > I have system with two ZFS pools. Both pools are RADIZ on HDDs (not > SSDs), and don't share any storage hardware. One pool is "my private space" > and other one is storage for torrents. > > Private space contains mirrors of FreeBSD SVN, my home directory, etc. > Storage for torrents contains torrents. > > My home directory contains, among other data, working copies of FreeBSD > src and ports. > > When torrent-client is stopped, disk-depended operations on PRIVATE pool > like "svn up ~/FreeBSD/ports" starts immediately on cold caches. > > When torrent-client are running (and reads about 5-6MB/s in small > chunks), "svn up ~/FreeBSD/ports" could wait up to 5 minutes before start > output of updated paths. ^T shows this: > > load: 0.48 cmd: svn 18126 [zio->io_cv] 275.95r 3.14u 6.14s 2% 21388k > > Is it normal? I understand, that ARC is shared between pools, but it > looks like too much interaction: there are NO shared I/O resources (disks, > controllers) between pools and these operations, but "svn" waits 60-300 > seconds for some condition variable! > > -- > // Lev Serebryakov > > From owner-freebsd-fs@freebsd.org Tue Apr 21 16:57:25 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 626622ACB68 for ; Tue, 21 Apr 2020 16:57:25 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4968r46HXgz4bfp; Tue, 21 Apr 2020 16:57:24 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.not-for.work (onlyone.not-for.work [148.251.9.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: lev/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 5CBB617A82; Tue, 21 Apr 2020 16:57:21 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from [192.168.23.230] (unknown [89.113.128.32]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.not-for.work (Postfix) with ESMTPSA id 3606582B4; Tue, 21 Apr 2020 19:57:19 +0300 (MSK) Reply-To: lev@FreeBSD.org Subject: Re: Two ZFS pools on one system: load on one pool badly affects other one To: Zaphod Beeblebrox Cc: freebsd-fs References: <802e3424-7edc-8e4a-b3e6-706c2b4a197e@FreeBSD.org> From: Lev Serebryakov Autocrypt: addr=lev@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFKbGksBEADeguVs+XyJc3mL3iiOBqDd16wSk97YTJYOi4VsHsINzJr09oFvNDiaDBIi fLn2p8XcJvehcsF2GSgrfXfw+uK4O1jyNIKJmiYA0EtE+ZbRtvDrrE0w6Q8+SDeKA21SWh3Y vSQ0DJUontbgW55ER2CbEiIUTIn34uQ0kmESAaw/v5p/9ue8yPTmURvv130FqPFz8VPzltqL NxyGt54TxPfKAzAHEIwxlEZ63JOwzloKh1UDBExcsf9nJO08/TAVgR5UZ5njFBPzaaquhRoP qPJLEQQDqxPIlvMNtHKf7iIebE4BHeqgCdJA0BoiR6gpa0wlsZtdrTPK3n4wYSphLvGbhfOZ YW/hbcu7HYS/FImkVxB3iY17kcC1UTnx4ZaYeASPBGOOPbXky1lLfmDGWIFT//70yx+G17qD OZzF1SvJJhGvh6ilFYaWMX7T+nIp6Mcafc4D7AakXM+XdubNXOMlCJhzPcZ0skgAEnYV587w V7em5fDVwQccwvtfezzqKeJAU5TGiywBHSR5Svzk2FwRNf6M//hWkpq0SRR63iOhkHGOAEBi 69GfEIwH2/w24rLxP0E+Hqq8n+EWNkPatw1Mhcl5PKkdvGCjJUaGNMkpBffjyYo254JXRscR eEnwdIkJt4ErDvjb2/UrOFq31wWMOiLzJeVchAgvTHBMRfP9aQARAQABzShMZXYgU2VyZWJy eWFrb3YgPGxldkBzZXJlYnJ5YWtvdi5zcGIucnU+wsGwBBMBCABDAhsDBwsJCAcDAgEGFQgC CQoLBBYCAwECHgECF4ACGQEWIQT5bRygtfQxi2dLMwrqsDxYv9xHjwUCW/03kQUJDwW3xgAh CRDqsDxYv9xHjxYhBPltHKC19DGLZ0szCuqwPFi/3EePHxkP+wWNrAyks2fQctY/Gl7TMh+Y Q9uX0hAuZ2Vvi0LswBl/R85SsS7IvI9b3ogOWA8CAlHAxkvgH6sWrwRTNcCPS1MzulYxS914 0CSkdwwbv1JyDOOWYU6s8PfT9+BZr+9eNXStmEdEL5XcA1k2YncQtlR3m+oLkqlAOtteZWti pitMIX9BGYIVKyl0t0RnIx+m/QPVGU9gu02j0I3NSRnKQPyFxZqYK0nPBu+FKaEhIAqdKPOv GL4/ijansdiWO3mXy18G0Mkr8yYRSidpGgXGY6lmGzQ3R6ZS30bLI8DkskOOvfErwhZv5dH5 w4+JH5sQ7bIL5HEXs//ZU9UzMdQwcURMjcFfKGyfL0hSLRqzP8m7SL1k9ZL161OQ6C5zVO/M bSCmeeLkbfOj1NW1ZIv6UjVVWE/LS4+gqg/04C+Y24vj+7vMpBVEevdwmIEdmVciFudklcnN omuocb29GKbquRZRDGiE+mhqkwmp5e59AnePp3+AvkewSCsXlR1sfjEP/Tn5OsYerJ7eAAOj DjxO374TAqJG5ftW4BA/nVmx9FGKV1/A9Yc1UuH6LdQfLf7pmTck1Cxg4kdH+3qKGD63sAR0 Wh27XDjnBKXJUN7J+nctWMZJMvw4OhTXdTyVhWt6USKEzw8M5plY4sFqxBEAe8igQXlq1Xjd ISV7wYhT4l3FzsFNBFKbGksBEAC0a9wfjo2P3JyT7Lc+QlbFVshGbSbazb4ma7QYG5IZZD5v fLBFkePoG6cnrn3WCXp4A43hszAynCwe4eXyAkv4+gPF3ZSeNE5Wz3zYG+jh2nm2iGCkyaVy kfbA+2chor2DKH5tHpuNMBlF+wSJHZKJmlo/sFIktAnV1NBVg4/cL+9/hIpvl82cl3hYCD7/ e7/qRE+w38CpAAzn65FvbODn7xlY3fsJt+cHPBJ4EBM9KnTwcce+F+72RQMZQEl7vIAwSRmL dgZHN0MFC533l62SVoKjT0eaOOIBrvesmojhWjfwugibXr+WRF/tGcW77Bxwe2eQLbEVESqW eMORxRxocx7Q7aACoHmf4G4U1Vzx7zUEfNfHjfjZeQVfAURf/MoUelZSW/BmMIfKCg3lRlWA t+Pq2h2UADPVqAZze45beE/c8z8LZsOZiGoRhYL8NSg6+ziLTdmYLWdtFGAuZhqOtNp5h6tG j21OksBotcaIa5YjbCmmnImIjGlSBkUKvIhq/RXth5b2gNwaQdu+Yv4AlZVHRsuVywL/skDF L5+We11bDK6MQ5PzvmntRJcgbyoisn1hiV04OV1LpJJMkJn1j8VlBqDQNT/z+BjB0ru/0anv +5uLj7v0ck06rEo4yiXT/ZAcBM76j7V7FaGbkoba6bUUCQ2H5YYBOKpikjCnpwARAQABwsGT BBgBCAAmAhsMFiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAlv9N7IFCQ8Ft+cAIQkQ6rA8WL/c R48WIQT5bRygtfQxi2dLMwrqsDxYv9xHj3CnD/9btCtkcphRYRUe08tUyVwzV/syDCdiUhF7 8jqDKTC+3zuyrFJi7t4fF9follHYz1Ri5RixxJHnuDFcq7ZTOprPYqO8QhckLAJOy5dmORDX 2guEA+y5zDYBwwjpio9dtnuE7QyHyMx4nMPq8O/HfO+6dDEZChkrGvcG9FTI7s0JhsDs3xxw jcROZ2OP0lNu2571ZpR4YuzMUOIhOaQBIF2wrTvLjKUsAnNQYK9gsFTeDHRsE4HZLxJvEdiZ CWN7COi9un4xtP4Khc3Fmn6ANEyh0bIgx1Eii2RGINuA2XRVYhPRJLUZRSVQcrND9k9S+m+T oaqz9JgFLusFA1KhdeYnE1bojpq1U1bsmEicLW2QfEGVumKTgUrTsno0cVPH73KDILFvHA0D 8t4UaQveRTRUVdHZ02IBVt655Q8Xq1TkHJ7l+2Ckso5IBujWD74QpSRzzffn/ihhEExwYSTj FSs0C/OgU+EDZbcq2SWu4n1OGsW337/80HnJKVWBPAZYy4EmiyQSY05MG/fj9RA9Qi4TjFLD LrIf6dFAmiiIwWjlAKiyyUk+XDJXrc1L2VhcHqfdBY4I/qwV1YAI1QI4W/i6TstB1j0GwKa3 ZORwu4eahL5+9R6xBedhXZpCL0dyKuI8iPaC8npaOCJoL8+l4+KXR/PKt8b8kzIcvSpyCZii PQ== Organization: FreeBSD Message-ID: <2b26b2c2-a658-5660-95c5-07fb9867ffb7@FreeBSD.org> Date: Tue, 21 Apr 2020 19:57:18 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="0uvtJF5xMRUqupdJDM1s8MOjiutDNMkDR" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2020 16:57:25 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --0uvtJF5xMRUqupdJDM1s8MOjiutDNMkDR Content-Type: multipart/mixed; boundary="MMcjXjSayJB37fMgXCTvpecUWfZGRpAAF" --MMcjXjSayJB37fMgXCTvpecUWfZGRpAAF Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 21.04.2020 19:39, Zaphod Beeblebrox wrote: > [ regarding zfs arc cache on two pools, where one pool can clear the ca= che of the other ] >=20 > ... have you tried introducing an l2 cache on an SSD for the "private" = pool? I've tried to add L2ARC to torrents pool (as "private" pool is more-or-= less sequential read, it contains a lot of photos, for example), and it w= as mistake: SATA SSD was destroyed in 6 months, and hitrate was about 10%= (10% of 7% of ARC misses!). I didn't thinks about adding L2ARC for "private" pool as I don't have P= CIe lines for NMVe left (it is old E3-1220v3 based system), and SATA SSD = is slower than 5 fast HDDs which comprise "private" pool in terms of line= ar read. I understand, that SSD is much better in terms of latency, of co= urse, but as most real access to "private" pool is linear access to large= files, I didn't think tat it could help. > It seems to me that size of the l2 ARC are pool specific (where the div= ision of memory in the l1 ARC is not).=C2=A0 It doesn't surprise me that = this happens.=C2=A0 I have a large RAIZ2 pool for media (currently 40T, e= xpanding to 60T shortly) on an otherwise general home server with 32G RAM= =2E=C2=A0 There is also a mirrored pool (root, home, a few things) on thi= s machine for various reasons (like not booting from such a large raid) a= nd things got overall better when I added a few hundred gig of l2 cache t= o each of the two pools. It doesn't look like cache wash-out, as I tried this "svn up" "test" on = cold caches, not over-and-opver again. Looks like it is some lock content= ion... But I'll try to find some SATA SSD and add it as L2ARC for "private" cac= he (as I said, L2ARC cache for torrents was bad idea!). --=20 // Lev Serebryakov --MMcjXjSayJB37fMgXCTvpecUWfZGRpAAF-- --0uvtJF5xMRUqupdJDM1s8MOjiutDNMkDR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE+W0coLX0MYtnSzMK6rA8WL/cR48FAl6fJe5fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 NkQxQ0EwQjVGNDMxOEI2NzRCMzMwQUVBQjAzQzU4QkZEQzQ3OEYACgkQ6rA8WL/c R4/PgRAAu+QdK/wOcZ3HlBDca/F2SBQAVwXyWESg+nVCJRagYwiGqv49FLrg8/fg RLntnwIW1Q0ClZAr09PrLE7Pl6242QBuFhvKIbUZ5e64BJAdPsT6j1UivgH56hfb Ibmc2++OGYYElj3BWJVvJ5BAL6V3LZR8q+74/ZjLzXRUPTE9pvVJmhA/ARiWaXF0 HgfImSKyQvURJDSg9i0cGX9dhebJX8pfsTHr/CkyUraiFK76p3vzn7R/MVAhQc5a 0GKZmepmMh4PtH+x7cEM+/1PJRRp3CUF7AgtbmCgyhvPceQbiBkXvMr8rEUy5xdU ICZYfRZTeAGe8MDYSWVxCt4W+f+NaK+ngBh/KNic74Fh6QVMq2k4Wjwku0IAqLbR 0dZOLcyXHOwCy3QT1Rez8DfleTFlAOTcH2KFxtlBtdqaIWMYE1db1k9CnX17vGT9 BGXLj15arWjrf51lMJxBn8QMsG4kTMVFoySU9jBC8ilzDqg7CGkjg7jrbUcpn72F /ptedGYz48Ve+y3QFdmhZTWL4MlElksY/DCx3SU6gHvccDHYuZV+SCqUlas6GmCh +bSFNEl/2qEBlk8g5bnn1ixDraiLmdHdDb2SMn8/+LBeNLpexA+314bTImQ2J8uK Z8SCj8j33OY/SMhqmp63M8IMnebiex5dQyPdc3japiXnMBxzHmE= =hVe6 -----END PGP SIGNATURE----- --0uvtJF5xMRUqupdJDM1s8MOjiutDNMkDR--