From owner-svn-src-all@freebsd.org Mon Nov 9 01:13:28 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADDE92E3B6C; Mon, 9 Nov 2020 01:13:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CTtKh4YGCz3N2j; Mon, 9 Nov 2020 01:13:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8EE13E7C; Mon, 9 Nov 2020 01:13:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0A91DSV2010221; Mon, 9 Nov 2020 01:13:28 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0A91DSg5010220; Mon, 9 Nov 2020 01:13:28 GMT (envelope-from mav@FreeBSD.org) Message-Id: <202011090113.0A91DSg5010220@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 9 Nov 2020 01:13:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r367505 - stable/12/sys/kern X-SVN-Group: stable-12 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: stable/12/sys/kern X-SVN-Commit-Revision: 367505 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Nov 2020 01:13:28 -0000 Author: mav Date: Mon Nov 9 01:13:28 2020 New Revision: 367505 URL: https://svnweb.freebsd.org/changeset/base/367505 Log: MFC r367052: Enable bioq 'car limit' added at r335066 at 128 bios. Without the 'car limit' enabled (before this), running sequential ZFS scrub on HDD without command queuing support, I've measured latency on concurrent random reads reaching 4 seconds (surprised that not more). Enabling this reduced the latency to 65 milliseconds, while scrub still doing ~180MB/s. For disks with command queuing this does not make much difference (if any), since most time all the requests are queued down to the disk or HBA, leaving nothing in the queue to sort. And even if something does not fit, staying on the queue, it is likely not for long. To not limit sorting in such bursty scenarios I've added batched counter zeroing when the queue is getting empty. The internal scheduler of the SAS HDD I was testing seems to be even more loyal to random I/O, reducing the scrub speed to ~120MB/s. So in case somebody worried this is limit is too strict -- it actually looks relaxed. Modified: stable/12/sys/kern/subr_disk.c Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/kern/subr_disk.c ============================================================================== --- stable/12/sys/kern/subr_disk.c Mon Nov 9 00:39:48 2020 (r367504) +++ stable/12/sys/kern/subr_disk.c Mon Nov 9 01:13:28 2020 (r367505) @@ -26,7 +26,7 @@ __FBSDID("$FreeBSD$"); #include #include -static int bioq_batchsize = 0; +static int bioq_batchsize = 128; SYSCTL_INT(_debug, OID_AUTO, bioq_batchsize, CTLFLAG_RW, &bioq_batchsize, 0, "BIOQ batch size"); @@ -172,6 +172,8 @@ bioq_remove(struct bio_queue_head *head, struct bio *b head->insert_point = NULL; TAILQ_REMOVE(&head->queue, bp, bio_queue); + if (TAILQ_EMPTY(&head->queue)) + head->batched = 0; head->total--; }