From owner-svn-src-all@freebsd.org Tue Dec 17 00:28:20 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 01B9D1D3055 for ; Tue, 17 Dec 2019 00:28:20 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47cJrz056Tz4WqP for ; Tue, 17 Dec 2019 00:28:18 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: by mail-wm1-x32e.google.com with SMTP id q9so1151189wmj.5 for ; Mon, 16 Dec 2019 16:28:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=17DXskGW4LtVR5PNsMRbaHMe62krSpBAAu6p40Y/pO0=; b=IBoqyea1WYSoaubsnsjP6YT7M0Mdg7g4TimYk3G9HzMBM4dJPZ/TvQVCBSDb7vck5e AdJS9AULr8vGJqCv1zh+3JfT2A1UV4ToJmoD0WAR74ru/I9dYbeIQw3zDHUt0UojFGaG 6Dlby+XFRLUwAJ5oYbwi2vfJJfKlSNhNKI9GEuQh5bNW36QtyM0Q/YpQyLCA2m/mqSNe FtKYHsxEUe8gRgC4SjknAljdYK7eYqdjgn8XGSTOwAbZ1L3LNpWvRnSY357dErz1jBvA VwWpAH3+FYNPplpY4GLblxCioGUjdGG0zlAluaed2QUfSlcxVxHpCtN69P26sGN95DGG or4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=17DXskGW4LtVR5PNsMRbaHMe62krSpBAAu6p40Y/pO0=; b=FxxpdfjMfc9w6MWYUTwO8Ak5x6XsY6M3/+/Xyb8b0e2PHB3faZKOJz4aPGX2SaKFtm 7zqHo8DofwnOEkvnFgPa2vzCSTFFjr7/XnD6tQPnKTaGM/O8D/39vQpLmLNf2zchbOhH XUZjI7OFL/xZvXGLKOsCALLct+XWsA0eldt9Mi5IindoWg5uogo+G+hzFC6Os9k+xyZO omzX924B8dTpcVDW4nycXLQcsco1UpRD7X5ToNgG/U4meUxgHTYuAg6yRKDkN9uUfdyG /6kUefNKcnKA5OFpb2JSLNrZrLoZ3DSHXIiiWm9Ln9JezjbadD9KgmUso8wsWHqwIMHu bw0g== X-Gm-Message-State: APjAAAWfneMiuAZzoeCIKlcQHTipWT10D7J2w4uNDZ1dxcJtYgoKMwmH eEhzTam1Fz1lNgCxhzYPx7KE5+6bryw= X-Google-Smtp-Source: APXvYqxFncI1nBIFoEl2jGaKGNRs8Smyewy83Xd4EFAlB6FVCFDJVIbNQ8rzsja2aAKsJn8qkPur9g== X-Received: by 2002:a7b:cd91:: with SMTP id y17mr1887412wmj.140.1576542496943; Mon, 16 Dec 2019 16:28:16 -0800 (PST) Received: from [10.44.128.75] ([193.117.175.106]) by smtp.gmail.com with ESMTPSA id r5sm22799612wrt.43.2019.12.16.16.28.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 16 Dec 2019 16:28:16 -0800 (PST) Subject: Re: svn commit: r355831 - head/sys/cam/nvme To: Warner Losh , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201912170011.xBH0Bm5I088826@repo.freebsd.org> From: Steven Hartland Message-ID: <4c5ce3c8-d074-f907-af03-20f4752f428c@multiplay.co.uk> Date: Tue, 17 Dec 2019 00:28:15 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <201912170011.xBH0Bm5I088826@repo.freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-Rspamd-Queue-Id: 47cJrz056Tz4WqP X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=IBoqyea1; dmarc=pass (policy=none) header.from=multiplay.co.uk; spf=pass (mx1.freebsd.org: domain of steven.hartland@multiplay.co.uk designates 2a00:1450:4864:20::32e as permitted sender) smtp.mailfrom=steven.hartland@multiplay.co.uk X-Spamd-Result: default: False [-5.77 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[svn-src-all@freebsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; DMARC_POLICY_ALLOW(-0.50)[multiplay.co.uk,none]; RCVD_IN_DNSWL_NONE(0.00)[e.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.77)[ip: (-9.25), ipnet: 2a00:1450::/32(-2.67), asn: 15169(-1.90), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Dec 2019 00:28:20 -0000 Be aware that ZFS already does a pretty decent job of this already, so the statement about upper layers isn't true for all. It even has different priorities for different request types so I'm a little concerned that doing it at both layers could cause issues. In addition to this if its anything like SSD's numbers of requests are only a small part of the story with total trim size being the other one. I this case you could hit total desired size with just one BIO_DELETE request. With this code what's the impact of this? On 17/12/2019 00:11, Warner Losh wrote: > Author: imp > Date: Tue Dec 17 00:11:48 2019 > New Revision: 355831 > URL: https://svnweb.freebsd.org/changeset/base/355831 > > Log: > NVME trim stuff. > > Add two sysctls to control pacing of nvme > trims. kern.cam.nda.X.goal_trim is the number of upper layer > BIO_DEELETE requests to try to collecet before sending TRIM down too > the nvme drive. trim_ticks is the number of ticks, at mosot, to wait > for at least goal_trim BIOS_DELEETE requests to come in. > > Trim pacing is useful when a large number off disjoint trims are > comoing in from the upper layers. Since we have no way to chain > toogether trims from the upper layers that are sent down, this acts as > a hueristic to group trims into reasonable sized chunks. What's > reasonable varies from drive to drive. > > Sponsored by: Netflix > > Modified: > head/sys/cam/nvme/nvme_da.c > > Modified: head/sys/cam/nvme/nvme_da.c > ============================================================================== > --- head/sys/cam/nvme/nvme_da.c Tue Dec 17 00:10:19 2019 (r355830) > +++ head/sys/cam/nvme/nvme_da.c Tue Dec 17 00:11:48 2019 (r355831) > @@ -177,6 +177,14 @@ static int nda_max_trim_entries = NDA_MAX_TRIM_ENTRIES > SYSCTL_INT(_kern_cam_nda, OID_AUTO, max_trim, CTLFLAG_RDTUN, > &nda_max_trim_entries, NDA_MAX_TRIM_ENTRIES, > "Maximum number of BIO_DELETE to send down as a DSM TRIM."); > +static int nda_goal_trim_entries = NDA_MAX_TRIM_ENTRIES / 2; > +SYSCTL_INT(_kern_cam_nda, OID_AUTO, goal_trim, CTLFLAG_RDTUN, > + &nda_goal_trim_entries, NDA_MAX_TRIM_ENTRIES / 2, > + "Number of BIO_DELETE to try to accumulate before sending a DSM TRIM."); > +static int nda_trim_ticks = 50; /* 50ms ~ 1000 Hz */ > +SYSCTL_INT(_kern_cam_nda, OID_AUTO, trim_ticks, CTLFLAG_RDTUN, > + &nda_trim_ticks, 50, > + "Number of ticks to hold BIO_DELETEs before sending down a trim"); > > /* > * All NVMe media is non-rotational, so all nvme device instances > @@ -741,6 +749,9 @@ ndaregister(struct cam_periph *periph, void *arg) > free(softc, M_DEVBUF); > return(CAM_REQ_CMP_ERR); > } > + /* Statically set these for the moment */ > + cam_iosched_set_trim_goal(softc->cam_iosched, nda_goal_trim_entries); > + cam_iosched_set_trim_ticks(softc->cam_iosched, nda_trim_ticks); > > /* ident_data parsing */ >