From owner-freebsd-fs@freebsd.org Tue Apr 21 16:39:50 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 93D912AB1D9 for ; Tue, 21 Apr 2020 16:39:50 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4968Rn1J9Jz4YBC; Tue, 21 Apr 2020 16:39:48 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-ej1-x634.google.com with SMTP id rh22so11478445ejb.12; Tue, 21 Apr 2020 09:39:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ePmGHeCRLAAFutEOdx4xXLIR17Ytdsd/uh9nEYtBBY0=; b=EM09cyUgMMkOn0eRHfbtdJOyGprJh1wuhuyvJlqMdxhpD/tdCs8gj+HDrJhGrsRLXk +NBOcS7M2WB4J/a4OqYYg6F8hA++pXakI7pZZcrdgZ/3aCbjkh7tBxfhqIDKQqbuF0Gp 1N1nqxEP9U5h+9Ajk+S7+oBd7RhtP6MrLRrQ6BXQtf66b7N8FSDpU+X+0epg+y1MtqML jpwjgzLYKu+7eWjQ0t+dTy/abgu+wH//EGg9ZL0RUN68rDfqwq70rz+CvDDo7FhgN8IJ aXCiCtZAubY+ICnKRrHYBtgJBq+42bjSdtnCF9H0Kmy0MZUDMcnST6jxQ+98czlhDLT/ PdKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ePmGHeCRLAAFutEOdx4xXLIR17Ytdsd/uh9nEYtBBY0=; b=WY+9sdLBqaVTfIgtwbi5z295gEb9qVdzcJRDe0ipvEVvaTosmkNuKP6JBahS5RdHKC 7D2vm13m5n3VqahFxEKVNoE+GwhDvdy0v7QMDTierhbnRqmHS/PhrLBjQfscOrqP0pYJ jo+7V6Czf4SyX/gZq7HXKlE+HaB2C/2GOhh4TMugGEwxRx54gIex6ktrpULmpELW8Cn9 wl0k95wY2oNt2/8UscuxA29/WqFmy90Gq3BwsAT4PcSd9PFccLeqpN2XwzDNSIJa807n tl1hgczhUpAE+2wjTf1RF9RlGXQPPl4PeA5g5hZEQy+aIp9SAberWxFEDeC1BTy9dv49 kfCg== X-Gm-Message-State: AGi0PuYvYtR1CFx0xa6DYa5nU33I/y1VoAXHtq+sK7d60i6pmxIiHnZg YAs8xh/4hey7ix8C3nAwPjOvdgibQTmGkhh61NV4CPk= X-Google-Smtp-Source: APiQypIGHoBJh+mbzKDAOMcZoGT65fXKTccjsG/iJztwvK/jxfJ2axoNusrSYysQVij4quUPaMhOuQXC+6K8IFqqkJg= X-Received: by 2002:a17:906:7804:: with SMTP id u4mr22616480ejm.328.1587487181917; Tue, 21 Apr 2020 09:39:41 -0700 (PDT) MIME-Version: 1.0 References: <802e3424-7edc-8e4a-b3e6-706c2b4a197e@FreeBSD.org> In-Reply-To: <802e3424-7edc-8e4a-b3e6-706c2b4a197e@FreeBSD.org> From: Zaphod Beeblebrox Date: Tue, 21 Apr 2020 12:39:30 -0400 Message-ID: Subject: Re: Two ZFS pools on one system: load on one pool badly affects other one To: lev@freebsd.org Cc: freebsd-fs X-Rspamd-Queue-Id: 4968Rn1J9Jz4YBC X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=EM09cyUg; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2a00:1450:4864:20::634 as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ipnet: 2a00:1450::/32(-2.34), asn: 15169(-0.43), country: US(-0.05)]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2020 16:39:50 -0000 [ regarding zfs arc cache on two pools, where one pool can clear the cache of the other ] ... have you tried introducing an l2 cache on an SSD for the "private" pool? It seems to me that size of the l2 ARC are pool specific (where the division of memory in the l1 ARC is not). It doesn't surprise me that this happens. I have a large RAIZ2 pool for media (currently 40T, expanding to 60T shortly) on an otherwise general home server with 32G RAM. There is also a mirrored pool (root, home, a few things) on this machine for various reasons (like not booting from such a large raid) and things got overall better when I added a few hundred gig of l2 cache to each of the two pools. On Mon, Apr 20, 2020 at 7:11 AM Lev Serebryakov wrote: > > I have system with two ZFS pools. Both pools are RADIZ on HDDs (not > SSDs), and don't share any storage hardware. One pool is "my private space" > and other one is storage for torrents. > > Private space contains mirrors of FreeBSD SVN, my home directory, etc. > Storage for torrents contains torrents. > > My home directory contains, among other data, working copies of FreeBSD > src and ports. > > When torrent-client is stopped, disk-depended operations on PRIVATE pool > like "svn up ~/FreeBSD/ports" starts immediately on cold caches. > > When torrent-client are running (and reads about 5-6MB/s in small > chunks), "svn up ~/FreeBSD/ports" could wait up to 5 minutes before start > output of updated paths. ^T shows this: > > load: 0.48 cmd: svn 18126 [zio->io_cv] 275.95r 3.14u 6.14s 2% 21388k > > Is it normal? I understand, that ARC is shared between pools, but it > looks like too much interaction: there are NO shared I/O resources (disks, > controllers) between pools and these operations, but "svn" waits 60-300 > seconds for some condition variable! > > -- > // Lev Serebryakov > >