From owner-freebsd-stable@freebsd.org Fri Apr 6 09:05:36 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71B6DF9452E for ; Fri, 6 Apr 2018 09:05:35 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu1176c.smtpx.saremail.com (cu1176c.smtpx.saremail.com [195.16.148.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A51D76EB9B for ; Fri, 6 Apr 2018 09:05:34 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.141] (unknown [192.148.167.11]) by proxypop02.sare.net (Postfix) with ESMTPA id BFF359DC8CE; Fri, 6 Apr 2018 10:56:12 +0200 (CEST) From: Borja Marcos Message-Id: <9AC3F9CE-B715-4894-B047-38758ACD913C@sarenet.es> Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) Subject: Re: TRIM, iSCSI and %busy waves Date: Fri, 6 Apr 2018 10:56:11 +0200 In-Reply-To: Cc: "Eugene M. Zheganin" , FreeBSD-STABLE Mailing List , Warner Losh To: Steven Hartland References: <92b92a3d-3262-c006-ed5a-dc2f9f4a5cb9@zhegan.in> <8935E1F4-10DE-4621-8153-26AF851CC26E@sarenet.es> X-Mailer: Apple Mail (2.3445.6.18) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 09:05:36 -0000 > On 6 Apr 2018, at 10:41, Steven Hartland = wrote: >=20 > That is very hw and use case dependent. >=20 > The reason we originally sponsored the project to add TRIM to ZFS was = that in our case without TRIM the performance got so bad that we had to = secure erase disks every couple of weeks as they simply became so slow = they where unusable. >=20 > Now admittedly that was a fair few years ago and hw has moved on since = then, but the point remains that it=E2=80=99s not totally true that just = not TRIMing is an option. As far as I know, letting the device know that you have released a block = should be a Good Thing.=20 So I prefer to TRIM. I have also seen cases where not doing TRIMs = resulted in a terrible performance although modern SSDs should have better algorithms to deal with it.=20 But I have also seen bad cases of TRIM requests piling up and clogging = the queues for long periods of time. Admittedly it was a purely synthetic situation (running = bonnie++) but, again, performing operations like destroying a huge snapshot or dataset could trigger a = similar condition. It was especially nasty when I tried NVMEs. I mentioned this here: https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023134.html That=E2=80=99s why I think there is a mid point between the radical = approach of not doing TRIMs at all and clogging the queues with too many of them.=20 If the TRIM queue gets very large and the =E2=80=9Cmandatory=E2=80=9D = operations (read, writes, etc) begin to get large delays I think it=E2=80=99s safe to assume that *there is* a = problem and discarding at least part of those TRIMs could help to solve it.=20 A =E2=80=9CTRIM when you can=E2=80=9D should still be better than a = =E2=80=9CDon=E2=80=99t TRIM at all=E2=80=9D.=20 Cheers, Borja. P.S: Attaching the graphs that were lost.