From nobody Wed Apr 6 14:57:14 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AA4301A9DA96 for ; Wed, 6 Apr 2022 14:57:55 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from outgoing-exchange-7.mit.edu (outgoing-exchange-7.mit.edu [18.9.28.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.outgoing-exchange.mit.edu", Issuer "InCommon RSA Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KYSLB5GDQz3pGZ for ; Wed, 6 Apr 2022 14:57:51 +0000 (UTC) (envelope-from jfc@mit.edu) Received: from w92exedge3.exchange.mit.edu (W92EXEDGE3.EXCHANGE.MIT.EDU [18.7.73.15]) by outgoing-exchange-7.mit.edu (8.14.7/8.12.4) with ESMTP id 236EvWTG014723; Wed, 6 Apr 2022 10:57:43 -0400 Received: from OC11EXPO29.exchange.mit.edu (18.9.4.102) by w92exedge3.exchange.mit.edu (18.7.73.15) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Wed, 6 Apr 2022 10:56:26 -0400 Received: from OC11EXPO29.exchange.mit.edu (18.9.4.102) by oc11expo29.exchange.mit.edu (18.9.4.102) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Wed, 6 Apr 2022 10:57:14 -0400 Received: from OC11EXPO29.exchange.mit.edu ([18.9.4.102]) by oc11expo29.exchange.mit.edu ([18.9.4.102]) with mapi id 15.00.1497.023; Wed, 6 Apr 2022 10:57:14 -0400 From: John F Carr To: "egoitz@ramattack.net" CC: "freebsd-fs@freebsd.org" Subject: Re: Desperate with 870 QVO and ZFS Thread-Topic: Desperate with 870 QVO and ZFS Thread-Index: AQHYSae2PxESj506kky2T4intWV7dqzjPPIA Date: Wed, 6 Apr 2022 14:57:14 +0000 Message-ID: References: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> In-Reply-To: <4e98275152e23141eae40dbe7ba5571f@ramattack.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [108.7.221.50] Content-Type: text/plain; charset="us-ascii" Content-ID: <881C66B0C8A23F45B966E1A2A58135CB@exchange.mit.edu> Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4KYSLB5GDQz3pGZ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=mit.edu; spf=pass (mx1.freebsd.org: domain of jfc@mit.edu designates 18.9.28.58 as permitted sender) smtp.mailfrom=jfc@mit.edu X-Spamd-Result: default: False [-3.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_COUNT_FIVE(0.00)[5]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:18.9.28.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_EXCELLENT(0.00)[18.9.28.58:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[18.9.28.58:from]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[mit.edu,none]; NEURAL_HAM_SHORT(-0.98)[-0.976]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US]; RCVD_TLS_LAST(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On Apr 6, 2022, at 07:15 , egoitz@ramattack.net wrote: >=20 > Good morning, >=20 > I write this post with the expectation that perhaps someone could help me= >=20 > I am running some mail servers with FreeBSD and ZFS. They use 870 QVO (no= t EVO or other Samsung SSD disks) disks as storage. They can easily have fr= om 1500 to 2000 concurrent connections. The machines have 128GB of ram and = the CPU is almost absolutely idle. The disk IO is normally at 30 or 40% per= cent at most. >=20 > The problem I'm facing is that they could be running just fine and sudden= ly at some peak hour, the IO goes to 60 or 70% and the machine becomes extr= emely slow. ZFS is all by default, except the sync parameter which is set d= isabled. Apart from that the ARC is limited to 64GB. But even this is extre= mely odd. The used ARC is near 20GB. I have seen, that meta cache in arc is= very near to the limit that FreeBSD automatically sets depending on the si= ze of the ARC you set. It seems that almost all ARC is used by meta cache. = I have seen this effect in all my mail servers with this hardware and softw= are config. >=20 My system with nvd0: NVMe namespace has a problem with high write volume. If I build llvm with debugging symbo= ls, which writes about 70 GB, the filesystem nearly grinds to a halt. I ha= ve to use a spinning disk to get decent performance on this workload. Ther= e is some old talk on the mailing lists about certain drives handling TRIM = commands badly. See comment by Ted Ts'o here: https://forums.freebsd.org/t= hreads/ssd-trim-maintenance.56951/ Unfortunately the documentation for adjusting trim settings is out of date.