From owner-svn-src-head@freebsd.org Tue Dec 17 00:32:20 2019 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 02A4D1D325C for ; Tue, 17 Dec 2019 00:32:20 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47cJxb18zHz4XBH for ; Tue, 17 Dec 2019 00:32:19 +0000 (UTC) (envelope-from steven.hartland@multiplay.co.uk) Received: by mail-wr1-x441.google.com with SMTP id t2so9450824wrr.1 for ; Mon, 16 Dec 2019 16:32:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=6Ojk3vXOJr7FWRbBqhnu7l1PQhTyOFNnxQu94Gh1UXw=; b=xRlHCgYjb0rKTER/vmJYPaOgQNrj6OclWV6dG7jMQ/q47NivFeTVgf6Tz9GG+znRTc c9//wLfX7VUR+cdxM5bQK+H5ILwFiXEH2ENPjPXCg8V0cfzTTw5Mk71wG/GoV9Eupzja YmztPDZb4q7tkiE6EDz9kganTGSO0sf1yp26nkr3u2MmCQkQ/gKtrmaYQJ62O/qYlnbe 1BbYvSEkr7DmNXV0rDb9ZY1uexoCt2lw3+fAd84Kn9EWiimCwU6KddgkBl3JD0osZZTH EAIcB7EJTJdFCU+tVFCwH0riiu93FdwL+RDPhvYPoceeBx4Odg0al2of1y1Zs+kPZJoA N4/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=6Ojk3vXOJr7FWRbBqhnu7l1PQhTyOFNnxQu94Gh1UXw=; b=Rep95glj1lZQcKzolP8JHsj5rLd3bUgWG6aQH2fb60D0lKnJQaaNSPIlPR519aMeay Eo7aHrTBDNJO2swB/d7oQ33R0T718sLW227wNILjbw2vBX0FlxIindFQmZzP/QFOXBm/ hfD55E8e05PSAA+YxhTLbuBKVWOAc3CuuH6Di0Ix/k2jUnXURSYF++V+2o6OCz3yeeiH 1FYsQDKq1xRABqAb9ebhrLlgXEk2m8rzOpQ3TK6wtrJmK+iCWn+97WJfqN45lTaWxw/U 3HiKatchAWEuyYmG7Ey/01rD0+81cA4R/DwCH0tR1oY+GqdBdLOBvv+Pq+bS3JhI5SEp i60g== X-Gm-Message-State: APjAAAXwjO5wTMA3CMaNvqLF/gmSPObTUZXEcuiYg/fqR7NumtAUgEyC kqyeKE/I9YjA1tlFzNtTky5gBWSIE1o= X-Google-Smtp-Source: APXvYqzIoUlNNWiAxCXRLHrgM0As0Q20EudFuDlnPdLrJRmk6m0ymgVnkNpwpRSjVZO4H1WSiEwgDA== X-Received: by 2002:adf:f10a:: with SMTP id r10mr33408512wro.202.1576542736895; Mon, 16 Dec 2019 16:32:16 -0800 (PST) Received: from [10.44.128.75] ([193.117.175.106]) by smtp.gmail.com with ESMTPSA id e18sm23371357wrr.95.2019.12.16.16.32.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 16 Dec 2019 16:32:16 -0800 (PST) Subject: Re: svn commit: r355832 - head/sys/cam To: Warner Losh , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201912170013.xBH0DLnh090458@repo.freebsd.org> From: Steven Hartland Message-ID: <135a3dcc-5e32-153d-6398-822e106bd79d@multiplay.co.uk> Date: Tue, 17 Dec 2019 00:32:15 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: <201912170013.xBH0DLnh090458@repo.freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-Rspamd-Queue-Id: 47cJxb18zHz4XBH X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=xRlHCgYj; dmarc=pass (policy=none) header.from=multiplay.co.uk; spf=pass (mx1.freebsd.org: domain of steven.hartland@multiplay.co.uk designates 2a00:1450:4864:20::441 as permitted sender) smtp.mailfrom=steven.hartland@multiplay.co.uk X-Spamd-Result: default: False [-3.45 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[svn-src-head@freebsd.org]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; DMARC_POLICY_ALLOW(-0.50)[multiplay.co.uk,none]; RCVD_IN_DNSWL_NONE(0.00)[1.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.45)[ip: (2.37), ipnet: 2a00:1450::/32(-2.67), asn: 15169(-1.90), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Dec 2019 00:32:20 -0000 What if any is the impact on request ordering with this new delayed TRIM? On 17/12/2019 00:13, Warner Losh wrote: > Author: imp > Date: Tue Dec 17 00:13:21 2019 > New Revision: 355832 > URL: https://svnweb.freebsd.org/changeset/base/355832 > > Log: > Add rate limiters to TRIM. > > Add rate limiters to trims. Trims are a bit different than reads or > writes in that they can be combined, so some care needs to be taken > where we rate limit them. Additional work will be needed to push the > working rate limit below the I/O quanta rate for things like IOPS. > > Sponsored by: Netflix > > Modified: > head/sys/cam/cam_iosched.c > > Modified: head/sys/cam/cam_iosched.c > ============================================================================== > --- head/sys/cam/cam_iosched.c Tue Dec 17 00:11:48 2019 (r355831) > +++ head/sys/cam/cam_iosched.c Tue Dec 17 00:13:21 2019 (r355832) > @@ -755,7 +755,20 @@ cam_iosched_has_io(struct cam_iosched_softc *isc) > static inline bool > cam_iosched_has_more_trim(struct cam_iosched_softc *isc) > { > + struct bio *bp; > > + bp = bioq_first(&isc->trim_queue); > +#ifdef CAM_IOSCHED_DYNAMIC > + if (do_dynamic_iosched) { > + /* > + * If we're limiting trims, then defer action on trims > + * for a bit. > + */ > + if (bp == NULL || cam_iosched_limiter_caniop(&isc->trim_stats, bp) != 0) > + return false; > + } > +#endif > + > /* > * If we've set a trim_goal, then if we exceed that allow trims > * to be passed back to the driver. If we've also set a tick timeout > @@ -771,8 +784,8 @@ cam_iosched_has_more_trim(struct cam_iosched_softc *is > return false; > } > > - return !(isc->flags & CAM_IOSCHED_FLAG_TRIM_ACTIVE) && > - bioq_first(&isc->trim_queue); > + /* NB: Should perhaps have a max trim active independent of I/O limiters */ > + return !(isc->flags & CAM_IOSCHED_FLAG_TRIM_ACTIVE) && bp != NULL; > } > > #define cam_iosched_sort_queue(isc) ((isc)->sort_io_queue >= 0 ? \ > @@ -1389,10 +1402,17 @@ cam_iosched_next_trim(struct cam_iosched_softc *isc) > struct bio * > cam_iosched_get_trim(struct cam_iosched_softc *isc) > { > +#ifdef CAM_IOSCHED_DYNAMIC > + struct bio *bp; > +#endif > > if (!cam_iosched_has_more_trim(isc)) > return NULL; > #ifdef CAM_IOSCHED_DYNAMIC > + bp = bioq_first(&isc->trim_queue); > + if (bp == NULL) > + return NULL; > + > /* > * If pending read, prefer that based on current read bias setting. The > * read bias is shared for both writes and TRIMs, but on TRIMs the bias > @@ -1414,6 +1434,26 @@ cam_iosched_get_trim(struct cam_iosched_softc *isc) > */ > isc->current_read_bias = isc->read_bias; > } > + > + /* > + * See if our current limiter allows this I/O. Because we only call this > + * here, and not in next_trim, the 'bandwidth' limits for trims won't > + * work, while the iops or max queued limits will work. It's tricky > + * because we want the limits to be from the perspective of the > + * "commands sent to the device." To make iops work, we need to check > + * only here (since we want all the ops we combine to count as one). To > + * make bw limits work, we'd need to check in next_trim, but that would > + * have the effect of limiting the iops as seen from the upper layers. > + */ > + if (cam_iosched_limiter_iop(&isc->trim_stats, bp) != 0) { > + if (iosched_debug) > + printf("Can't trim because limiter says no.\n"); > + isc->trim_stats.state_flags |= IOP_RATE_LIMITED; > + return NULL; > + } > + isc->current_read_bias = isc->read_bias; > + isc->trim_stats.state_flags &= ~IOP_RATE_LIMITED; > + /* cam_iosched_next_trim below keeps proper book */ > #endif > return cam_iosched_next_trim(isc); > }