From owner-svn-src-all@freebsd.org Tue Dec 17 02:53:28 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CC5F91D8DAC for ; Tue, 17 Dec 2019 02:53:28 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47cN4R4GVkz3CxD for ; Tue, 17 Dec 2019 02:53:27 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qv1-xf30.google.com with SMTP id t9so3603106qvh.13 for ; Mon, 16 Dec 2019 18:53:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=auxWG4zOX89zBsVORtjijiDwKXg+AuDHwJO7eT79kWk=; b=wswElnWQJWlVjSUUwcps2+dbNyrc51CalCFAzdfODm9zSfIqI32kQdXS+6ExYX5zGH jgBEnFXa5aLagd5xh8tD4AxyGVeNVOFQ4gBn4qQ6Z/aLBm82BnHMT4AJi7VgQmaZ/Rsb GSdlottWSoObj8o9twY72reLGuNnAWtjSbJ9IobjGrhP4+/BCcIP9C7Fj68DnVcxReL1 NMeMC97UUupTHzw3WEzvd//WnNPoE7ekxOVUepVooFnQtpSiozGHzs3Dqj1qaxzreQmo awn8bdOrYtBPqSQMJC7M+FOpkIXmb4X72K6OKDZ2boJzfNOaUyAFR/2sKz9juh48EPIt n29g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=auxWG4zOX89zBsVORtjijiDwKXg+AuDHwJO7eT79kWk=; b=TqJEsLZINaPoZvcOFJuuUNoBo8r02rdRPrOMggDwXzKoO5ZgnPkD5C2puYcZwqe6Y2 ZADeJ0HPM0fpSldlj+sMkNq/gG+5WKNltLpu/51UEEeB5bECfDcsnPk/fEXGajuptbh8 yY1DzG1wy+CwDx9T91FVfVrGp22F0ZAdp05/DkgRR/Ag3vXV9oUzNJPPRcO1fB7QTyeN ZE1kcvKaIxkSOHRAPmiI74ybdUAAu4ywT80QGYT4l5UCgNbjnonxXWg3fv8f3B+1l5ex AGAxr32JKxWGmNhz3IXMDFrIzXPRLltg59GBMi8UKpnmKQF0LXWWf6K69WQQaARNnk4j LWtg== X-Gm-Message-State: APjAAAVDLjARalgonyM4j+GyJJfV3vxTJcOBd73XPpA+qWGLBExvDewC kerZec0WFgfv0RI535OXY0HXWKgCj7DGzn1kVRprSg== X-Google-Smtp-Source: APXvYqyG+Kx/qGBLVOWLr5rTN/7/PWVvV3oo0Q4GYm4BVp9QTekB7f0dWUV9E6tHVqpO9UAPS7edRuOtTGhJ2fPDiX0= X-Received: by 2002:a0c:e74d:: with SMTP id g13mr2589781qvn.125.1576551201415; Mon, 16 Dec 2019 18:53:21 -0800 (PST) MIME-Version: 1.0 References: <201912170011.xBH0Bm5I088826@repo.freebsd.org> <4c5ce3c8-d074-f907-af03-20f4752f428c@multiplay.co.uk> In-Reply-To: <4c5ce3c8-d074-f907-af03-20f4752f428c@multiplay.co.uk> From: Warner Losh Date: Mon, 16 Dec 2019 19:53:10 -0700 Message-ID: Subject: Re: svn commit: r355831 - head/sys/cam/nvme To: Steven Hartland Cc: Warner Losh , src-committers , svn-src-all , svn-src-head X-Rspamd-Queue-Id: 47cN4R4GVkz3CxD X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=wswElnWQ; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::f30) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-1.83 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.998,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[svn-src-all@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; URI_COUNT_ODD(1.00)[3]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.83)[ipnet: 2607:f8b0::/32(-2.19), asn: 15169(-1.90), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Dec 2019 02:53:28 -0000 On Mon, Dec 16, 2019, 5:28 PM Steven Hartland < steven.hartland@multiplay.co.uk> wrote: > Be aware that ZFS already does a pretty decent job of this already, so > the statement > about upper layers isn't true for all. It even has different priorities > for different request > types so I'm a little concerned that doing it at both layers could cause > issues. > ZFS' BIO_DELETE scheduling works well for enterprise drives, but needs tuning the further away you get from enterprise performance. I don't anticipate any effect on performance here since this is not enabled by default, unless I've messed something up (and if I have screwed this up, please let me know). I've honestly not tried to enable these things on ZFS. In addition to this if its anything like SSD's numbers of requests are > only a small part > of the story with total trim size being the other one. I this case you > could hit total > desired size with just one BIO_DELETE request. > > With this code what's the impact of this? > You're correct. It tends to be the number of segments and/or the size of the segment. This steers cases where the number of segments dominates. For cases where total size dominates, you're often better off using the I/O scheduler to rate limit the size of the trims. This feature is designed to allow a large number of files to be deleted at once while doing the trims from them a little at a time to even the load out. Warner > On 17/12/2019 00:11, Warner Losh wrote: > > Author: imp > > Date: Tue Dec 17 00:11:48 2019 > > New Revision: 355831 > > URL: https://svnweb.freebsd.org/changeset/base/355831 > > > > Log: > > NVME trim stuff. > > > > Add two sysctls to control pacing of nvme > > trims. kern.cam.nda.X.goal_trim is the number of upper layer > > BIO_DEELETE requests to try to collecet before sending TRIM down too > > the nvme drive. trim_ticks is the number of ticks, at mosot, to wait > > for at least goal_trim BIOS_DELEETE requests to come in. > > > > Trim pacing is useful when a large number off disjoint trims are > > comoing in from the upper layers. Since we have no way to chain > > toogether trims from the upper layers that are sent down, this acts as > > a hueristic to group trims into reasonable sized chunks. What's > > reasonable varies from drive to drive. > > > > Sponsored by: Netflix > > > > Modified: > > head/sys/cam/nvme/nvme_da.c > > > > Modified: head/sys/cam/nvme/nvme_da.c > > > ============================================================================== > > --- head/sys/cam/nvme/nvme_da.c Tue Dec 17 00:10:19 2019 > (r355830) > > +++ head/sys/cam/nvme/nvme_da.c Tue Dec 17 00:11:48 2019 > (r355831) > > @@ -177,6 +177,14 @@ static int nda_max_trim_entries = > NDA_MAX_TRIM_ENTRIES > > SYSCTL_INT(_kern_cam_nda, OID_AUTO, max_trim, CTLFLAG_RDTUN, > > &nda_max_trim_entries, NDA_MAX_TRIM_ENTRIES, > > "Maximum number of BIO_DELETE to send down as a DSM TRIM."); > > +static int nda_goal_trim_entries = NDA_MAX_TRIM_ENTRIES / 2; > > +SYSCTL_INT(_kern_cam_nda, OID_AUTO, goal_trim, CTLFLAG_RDTUN, > > + &nda_goal_trim_entries, NDA_MAX_TRIM_ENTRIES / 2, > > + "Number of BIO_DELETE to try to accumulate before sending a DSM > TRIM."); > > +static int nda_trim_ticks = 50; /* 50ms ~ 1000 Hz */ > > +SYSCTL_INT(_kern_cam_nda, OID_AUTO, trim_ticks, CTLFLAG_RDTUN, > > + &nda_trim_ticks, 50, > > + "Number of ticks to hold BIO_DELETEs before sending down a trim"); > > > > /* > > * All NVMe media is non-rotational, so all nvme device instances > > @@ -741,6 +749,9 @@ ndaregister(struct cam_periph *periph, void *arg) > > free(softc, M_DEVBUF); > > return(CAM_REQ_CMP_ERR); > > } > > + /* Statically set these for the moment */ > > + cam_iosched_set_trim_goal(softc->cam_iosched, > nda_goal_trim_entries); > > + cam_iosched_set_trim_ticks(softc->cam_iosched, nda_trim_ticks); > > > > /* ident_data parsing */ > > > >