From nobody Sun May 1 17:27:37 2022 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 327601ABAAAB; Sun, 1 May 2022 17:27:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KrtTP6wrCz4rf4; Sun, 1 May 2022 17:27:37 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1651426058; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Lc+EoGsgvGOv+/ABIzUYxKtpz6GuQsLvhgLB40tSU2c=; b=OBYkTAKfy7aey7o+gRPOFhMYGzBO6vCwet1KSdTCfw6KVhC0U8RyRBV4KOhRUx1ZSYOaKR iMcoeI7ItdnHzTK5Cbf5pBqxM6dLhucrDKFd3v82xKP7mlrPLp91968sJ/wDlQt9e2qqxq u3PFouVkqyaqHwDT6BlGCnRCrCWQOBFpfm5GsfKJ0DnMf7cYuKjhoX+XtTvZCOg5gaDinB dNBs1wJrBTsWCHomv8vH5sC/JenW5ImPz6NQY6PhqMoTs5hZeSUsOXVEf/uF1r5Bm2MfG6 Js4p13rwHpvDtlsqGmMhJ6DdU0FUZqpRMtC7lXh1i3LF6aDhuGyXagMHltcNXg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CA7B519DD0; Sun, 1 May 2022 17:27:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 241HRbiO061956; Sun, 1 May 2022 17:27:37 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 241HRblT061955; Sun, 1 May 2022 17:27:37 GMT (envelope-from git) Date: Sun, 1 May 2022 17:27:37 GMT Message-Id: <202205011727.241HRblT061955@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Warner Losh Subject: git: cc1572ddeb8c - main - cam iosched: Remove write bias when read bias = 0 List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: imp X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: cc1572ddeb8cd82879ce0cca634bf6a8830c0f40 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1651426058; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Lc+EoGsgvGOv+/ABIzUYxKtpz6GuQsLvhgLB40tSU2c=; b=BN5+H0L4/YAZvr8moGUUe7wijgpu74jG+xSV9QoboHvZk41IvjHNpnLXXhZt1w6iPIafUh tENmS4UDHY9odJXZe0zwEsd3hAo2H5rNEIo6TiKn7HIeDIgk4BBaMAs8r+S4Ezt4atrxdo 6TsuuWK5180WstsHoSqfcTrsGvwK/ligNDu8CCLr1cZrvrRAs70KNVm8C3fT0O3GsyStH0 o9QfW84V7bU1Px9cqgcfGMiJm75Nm5y8DRqvqHmhbJpPf2Hs07ejnei7XZN+ndDRntD8io X6Tp5uBXGYCXpwqQmtWQUmkKZxJiJ+Nm3CDGSuJCovbPtdMys8JvCBjyKzNUAQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1651426058; a=rsa-sha256; cv=none; b=g9VifIQ8sJQNsHsrkAdJZd+KPRUviyBMUN5qtH8gpgid7nNONjzXWcXpGoviLgFC9QKa+X P4ZEOPjJlYFl/kmuVKBljqPoAIYunnHHVP/zBfcoJ8LLyv60OSenGP1fGHNWris0oVcin+ qiHK4QFr5/zr3Hxw/24EHySu5qQSTxOpbIjFnZ+pkNjKBzt/PYmKfIw1y+nrQ1iNh158Lp t5yXyaOqhdIMsxvaVKdE2EF+RhwEf+yDCVw0FGnT1mQj65sWIgd3xEmu6ZGkW0L7Vi21Se FWlAjYC2TiBa0FEUpxDNq3EkYfN/CULap4r5hgogD4pmrhe4QlxmUiSe18aXUw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/src/commit/?id=cc1572ddeb8cd82879ce0cca634bf6a8830c0f40 commit cc1572ddeb8cd82879ce0cca634bf6a8830c0f40 Author: Warner Losh AuthorDate: 2022-05-01 17:18:18 +0000 Commit: Warner Losh CommitDate: 2022-05-01 17:27:34 +0000 cam iosched: Remove write bias when read bias = 0 Change the meaning of read bias == 0 in the dynamic I/O scheduler. Prior to this change, a read bias of 0 would mean prefer writes. Now, when read bias is 0, we queue all requests to the same queue removing the bias. When it's non-zero, we still separate the queues we use so we can bias reads vs writes for workloads that are read centric. These changes restore the typical bias you get from disksort or ordered insertion at the end of the list. Sponsored by: Netflix --- sys/cam/cam_iosched.c | 84 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 69 insertions(+), 15 deletions(-) diff --git a/sys/cam/cam_iosched.c b/sys/cam/cam_iosched.c index 5b5e375cc8e2..7ddfabbe5125 100644 --- a/sys/cam/cam_iosched.c +++ b/sys/cam/cam_iosched.c @@ -1495,6 +1495,28 @@ cam_iosched_get_trim(struct cam_iosched_softc *isc) return cam_iosched_next_trim(isc); } + +static struct bio * +bio_next(struct bio *bp) +{ + bp = TAILQ_NEXT(bp, bio_queue); + /* + * After the first commands, the ordered bit terminates + * our search because BIO_ORDERED acts like a barrier. + */ + if (bp == NULL || bp->bio_flags & BIO_ORDERED) + return NULL; + return bp; +} + +#ifdef CAM_IOSCHED_DYNAMIC +static bool +cam_iosched_rate_limited(struct iop_stats *ios) +{ + return ios->state_flags & IOP_RATE_LIMITED; +} +#endif + /* * Determine what the next bit of work to do is for the periph. The * default implementation looks to see if we have trims to do, but no @@ -1527,27 +1549,54 @@ cam_iosched_next_bio(struct cam_iosched_softc *isc) return bp; } #endif - /* * next, see if there's other, normal I/O waiting. If so return that. */ - if ((bp = bioq_first(&isc->bio_queue)) == NULL) - return NULL; - #ifdef CAM_IOSCHED_DYNAMIC - /* - * For the dynamic scheduler, bio_queue is only for reads, so enforce - * the limits here. Enforce only for reads. - */ if (do_dynamic_iosched) { - if (bp->bio_cmd == BIO_READ && - cam_iosched_limiter_iop(&isc->read_stats, bp) != 0) { - isc->read_stats.state_flags |= IOP_RATE_LIMITED; - return NULL; + for (bp = bioq_first(&isc->bio_queue); bp != NULL; + bp = bio_next(bp)) { + /* + * For the dynamic scheduler with a read bias, bio_queue + * is only for reads. However, without one, all + * operations are queued. Enforce limits here for any + * operation we find here. + */ + if (bp->bio_cmd == BIO_READ) { + if (cam_iosched_rate_limited(&isc->read_stats) || + cam_iosched_limiter_iop(&isc->read_stats, bp) != 0) { + isc->read_stats.state_flags |= IOP_RATE_LIMITED; + continue; + } + isc->read_stats.state_flags &= ~IOP_RATE_LIMITED; + } + /* + * There can only be write requests on the queue when + * the read bias is 0, but we need to process them + * here. We do not assert for read bias == 0, however, + * since it is dynamic and we can have WRITE operations + * in the queue after we transition from 0 to non-zero. + */ + if (bp->bio_cmd == BIO_WRITE) { + if (cam_iosched_rate_limited(&isc->write_stats) || + cam_iosched_limiter_iop(&isc->write_stats, bp) != 0) { + isc->write_stats.state_flags |= IOP_RATE_LIMITED; + continue; + } + isc->write_stats.state_flags &= ~IOP_RATE_LIMITED; + } + /* + * here we know we have a bp that's != NULL, that's not rate limited + * and can be the next I/O. + */ + break; } - } - isc->read_stats.state_flags &= ~IOP_RATE_LIMITED; + } else #endif + bp = bioq_first(&isc->bio_queue); + + if (bp == NULL) + return (NULL); bioq_remove(&isc->bio_queue, bp); #ifdef CAM_IOSCHED_DYNAMIC if (do_dynamic_iosched) { @@ -1555,6 +1604,10 @@ cam_iosched_next_bio(struct cam_iosched_softc *isc) isc->read_stats.queued--; isc->read_stats.total++; isc->read_stats.pending++; + } else if (bp->bio_cmd == BIO_WRITE) { + isc->write_stats.queued--; + isc->write_stats.total++; + isc->write_stats.pending++; } else printf("Found bio_cmd = %#x\n", bp->bio_cmd); } @@ -1632,7 +1685,8 @@ cam_iosched_queue_work(struct cam_iosched_softc *isc, struct bio *bp) #endif } #ifdef CAM_IOSCHED_DYNAMIC - else if (do_dynamic_iosched && (bp->bio_cmd != BIO_READ)) { + else if (do_dynamic_iosched && isc->read_bias != 0 && + (bp->bio_cmd != BIO_READ)) { if (cam_iosched_sort_queue(isc)) bioq_disksort(&isc->write_queue, bp); else