From eugen@grosbein.net Wed Apr 6 16:14:51 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7848B1A8F2A7; Wed, 6 Apr 2022 16:15:28 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KYV3f0Bzhz4bLk; Wed, 6 Apr 2022 16:15:25 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.16.1/8.16.1) with ESMTPS id 236GFMI2057370 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 6 Apr 2022 16:15:23 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: egoitz@ramattack.net Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.16.1/8.16.1) with ESMTPS id 236GEvnk092344 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Wed, 6 Apr 2022 23:15:22 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Desperate with 870 QVO and ZFS To: egoitz@ramattack.net, freebsd-fs@freebsd.org, freebsd-hackers@freebsd.org, Freebsd performance References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> <665236B1-8F61-4B0E-BD9B-7B501B8BD617@ultra-secure.de> <0ef282aee34b441f1991334e2edbcaec@ramattack.net> <28e11d7ec0ac5dbea45f9f271fc28f06@ramattack.net> From: Eugene Grosbein Message-ID: Date: Wed, 6 Apr 2022 23:14:51 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 In-Reply-To: <28e11d7ec0ac5dbea45f9f271fc28f06@ramattack.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.2 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 4KYV3f0Bzhz4bLk X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=fail (mx1.freebsd.org: domain of eugen@grosbein.net does not designate 2a01:4f8:c2c:26d8::2 as permitted sender) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-2.10 / 15.00]; ARC_NA(0.00)[]; R_SPF_FAIL(1.00)[-all]; FREEFALL_USER(0.00)[eugen]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-performance,freebsd-hackers,freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N 06.04.2022 22:30, egoitz@ramattack.net пишет: > One perhaps important note!! > > > When this happens... almost all processes appear in top in the following state: > > > txg state or > > txg-> > > bio.... > > > perhaps should the the vfs.zfs.dirty_data_max, vfs.zfs.txg.timeout, vfs.zfs.vdev.async_write_active_max_dirty_percent be increased, decreased.... I'm afraid of doing some chage ana finally ending up with an inestable server.... I'm not an expert in handling these values.... > > > Any recommendation?. 1) Make sure the pool has enough free space because ZFS can became crawling slow otherwise. 2) Increase recordsize upto 1MB for file systems located in the pool so ZFS is allowed to use bigger request sizes for read/write operations 3) If you use compression, look if achieved compressratio worth it and if not (<1.4 f.e.) then better disable compression to avoid its overhead; 4) try "zfs set redundant_metadata=most" to decrease amount of small writes to the file systems; 5) If you have good power supply and stable (non-crashing) OS, try increasing sysctl vfs.zfs.txg.timeout from defaule 5sec, but do not be extreme (f.e. upto 10sec). Maybe it will increase amount of long writes and decrease amount of short writes, that is good.