From owner-freebsd-stable@freebsd.org Mon Mar 18 14:31:55 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC3BE15410EE for ; Mon, 18 Mar 2019 14:31:54 +0000 (UTC) (envelope-from peter.blok@bsd4all.org) Received: from smtpq1.tb.mail.iss.as9143.net (smtpq1.tb.mail.iss.as9143.net [212.54.42.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 00FC48B7C1 for ; Mon, 18 Mar 2019 14:31:52 +0000 (UTC) (envelope-from peter.blok@bsd4all.org) Received: from [212.54.42.135] (helo=smtp11.tb.mail.iss.as9143.net) by smtpq1.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1h5tIl-0002yI-0H for freebsd-stable@freebsd.org; Mon, 18 Mar 2019 15:31:43 +0100 Received: from 5ed17ad9.cm-7-2b.dynamic.ziggo.nl ([94.209.122.217] helo=wan0.bsd4all.org) by smtp11.tb.mail.iss.as9143.net with esmtp (Exim 4.86_2) (envelope-from ) id 1h5tIk-0008JQ-Sv for freebsd-stable@freebsd.org; Mon, 18 Mar 2019 15:31:42 +0100 Received: from newnas (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 89D8C319 for ; Mon, 18 Mar 2019 15:31:42 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8u9plNvnXwS3 for ; Mon, 18 Mar 2019 15:31:41 +0100 (CET) Received: from [192.168.1.65] (unknown [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 603BD314 for ; Mon, 18 Mar 2019 15:31:41 +0100 (CET) From: peter.blok@bsd4all.org Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: Observations from a ZFS reorganization on 12-STABLE Date: Mon, 18 Mar 2019 15:31:40 +0100 References: <58eb1994-41bd-cd22-be66-0024bcbc36e6@denninger.net> <2baf16fd-3767-1dda-d519-995f7ebaf0cb@ingresso.co.uk> <20190318091431.W52549@mulder.mintsol.com> <76f222db-8b75-80ef-ce48-a43217f10e60@denninger.net> To: FreeBSD Stable In-Reply-To: <76f222db-8b75-80ef-ce48-a43217f10e60@denninger.net> Message-Id: X-Mailer: Apple Mail (2.3445.102.3) X-SourceIP: 94.209.122.217 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.3 cv=T6TysMCQ c=1 sm=1 tr=0 a=0XONDDbZk2SpjknwKA3Xxg==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=NTGMnVQrEZIA:10 a=C6q32NwkAAAA:8 a=HDP82zNOIxMqSwr-1SMA:9 a=CjuIK1q_8ugA:10 a=Omqe0uy41MGbrpyyOKEA:9 a=8o6lKlSoSwwGiz_K:21 a=_W_S_7VecoQA:10 a=d0GpIgV8JInD5zQLk0HG:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 00FC48B7C1 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of peter.blok@bsd4all.org designates 212.54.42.164 as permitted sender) smtp.mailfrom=peter.blok@bsd4all.org X-Spamd-Result: default: False [-2.15 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; NEURAL_HAM_MEDIUM(-0.82)[-0.818,0]; MIME_TRACE(0.00)[0:+,1:+]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsd4all.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.99)[-0.986,0]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[smtp.bsd4all.org]; NEURAL_HAM_SHORT(-0.35)[-0.346,0]; FROM_NO_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-0.09)[asn: 33915(-0.49), country: NL(0.01)]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[164.42.54.212.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[217.122.209.94.zen.spamhaus.org : 127.0.0.11] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2019 14:31:55 -0000 Same here using mfsbsd from 11-RELEASE. First attempt I forgot to add = swap - it killed the ssh I was using to issue a zfs send on the remote = system. Next attempt I added swap, but ssh got killed too. Third attempt I used mfsbsd from 12-RELEASE. It succeeded. Now I am using mfsbsd 11-RELEASE with added swap and vis.zfs.arc_min and = arc_max to 128Mb (it is a 4GB system) and it succeeds > On 18 Mar 2019, at 15:14, Karl Denninger wrote: >=20 > On 3/18/2019 08:37, Walter Cramer wrote: >> I suggest caution in raising vm.v_free_min, at least on 11.2-RELEASE >> systems with less RAM. I tried "65536" (256MB) on a 4GB mini-server, >> with vfs.zfs.arc_max of 2.5GB. Bad things happened when the cron >> daemon merely tried to run `periodic daily`. >>=20 >> A few more details - ARC was mostly full, and "bad things" was 1: >> `pagedaemon` seemed to be thrashing memory - using 100% of CPU, with >> little disk activity, and 2: many normal processes seemed unable to >> run. The latter is probably explained by `man 3 sysctl` (see entry = for >> "VM_V_FREE_MIN"). >>=20 >>=20 >> On Mon, 18 Mar 2019, Pete French wrote: >>=20 >>> On 17/03/2019 21:57, Eugene Grosbein wrote: >>>> I agree. Recently I've found kind-of-workaround for this problem: >>>> increase vm.v_free_min so when "FREE" memory goes low, >>>> page daemon wakes earlier and shrinks UMA (and ZFS ARC too) moving >>>> some memory >>>> from WIRED to FREE quick enough so it can be re-used before bad >>>> things happen. >>>>=20 >>>> But avoid increasing vm.v_free_min too much (e.g. over 1/4 of total >>>> RAM) >>>> because kernel may start behaving strange. For 16Gb system it = should >>>> be enough >>>> to raise vm.v_free_min upto 262144 (1GB) or 131072 (512M). >>>>=20 >>>> This is not permanent solution in any way but it really helps. >>>=20 >>> Ah, thats very interesting, thankyou for that! I;ve been bitten by >>> this issue too in the past, and it is (as mentioned) much improved = on >>> 12, but the act it could still cause issues worries me. >>>=20 > Raising free_target should *not* result in that sort of thrashing.=20 > However, that's not really a fix standing alone either since the > underlying problem is not being addressed by either change. It is > especially dangerous to raise the pager wakeup thresholds if you still > run into UMA allocated-but-not-in-use not being cleared out issues as > there's a risk of severe pathological behavior arising that's worse = than > the original problem. >=20 > 11.1 and before (I didn't have enough operational experience with 11.2 > to know, as I went to 12.x from mostly-11.1 installs around here) were > essentially unusable in my workload without either my patch set or the > Phabricator one. >=20 > This is *very* workload-specific however, or nobody would use ZFS on > earlier releases, and many do without significant problems. >=20 > --=20 > Karl Denninger > karl@denninger.net = > > /The Market Ticker/ > /[S/MIME encrypted email preferred]/