From owner-svn-src-head@freebsd.org Wed Jun 13 19:22:09 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7AC771009814 for ; Wed, 13 Jun 2018 19:22:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-it0-x22c.google.com (mail-it0-x22c.google.com [IPv6:2607:f8b0:4001:c0b::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0AB0275875 for ; Wed, 13 Jun 2018 19:22:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-it0-x22c.google.com with SMTP id 188-v6so5431431ita.5 for ; Wed, 13 Jun 2018 12:22:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=P4uac3RzoXw4OQ5Uj1DwCQpQzNp9waJkeTa6g6NpMYA=; b=m4EOl4j17nWTfHu8vl6bP1AGFtzItuQSk3pOffdy9WKNCJ94owgDxYVR1HFh4QRhtn wJv/zmYKDzWSMF3LkbaULaFu+GZxFw49QtPCs1uITxaHzomfrxeYJOXHiePGRm0QqawJ G/w8YbiT6t8T6mu44lonxR7wFaU/6XgNYVrL0+geD6t+qHDIOEyumgg4E+4DKiC3CL0Z jHInFeHVXnKri0bdOTOv2Co7rmad7KGfbsOljlNk2yYwVX3zLenHqzmvEYgeWpScEOmA QcqNa44AqyggQXEMZUzu9o6bux/4xAaX8DxKYGuYmHc/JdlVuePDbyxzpE+OFaIMvqda pTVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=P4uac3RzoXw4OQ5Uj1DwCQpQzNp9waJkeTa6g6NpMYA=; b=dJ0LJCrlMILLkChj+UIT/lVc6vMypSX+MoCspDZdlegcqZS1CAwZ2pG8eMnUMpCg6u vvYqZbLAHKmibol8WI8iLKJmNRLVMc6JHMQDdYqgUHNPqlAWGvGaicFuU7QophS5HFJF c6Tz+vugr4ZHgKvfUWqSmCIWLAds/oHdzfbbPj1lO6sJQSlgNNgZiGQubdBd/+R+ZtL+ 5Mu5Bq9CC4U0X9RSBu+T5AQ4rujA3Qb0oYHenkjpJzmM8LcqqHVy0MsJ8rUoQp6B+Jkx NxMQZdOCnZiw5/jAd7tuE2YMmaoAs1oJEQmMTJET/UA+AUa4oPwcxrAXTv9XadU9XedZ ryOA== X-Gm-Message-State: APt69E1iJissvZhcthI2krCP7piY2rz5qlqryp93Y/FOPiaaJuj+oLaK xeHkpHXNfc+5ktTUnooA26HiHzNpPlcvihWK2h4ANQ== X-Google-Smtp-Source: ADUXVKIqYPCMPin3RjdILALdzk+x7T4APlllcdy2P4uYUVcbyqZU9esJ5Uyl1SwprFKS6RzGGtxJfW0PmbVwRATUenY= X-Received: by 2002:a24:64ce:: with SMTP id t197-v6mr5861637itc.36.1528917728328; Wed, 13 Jun 2018 12:22:08 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 2002:a4f:d028:0:0:0:0:0 with HTTP; Wed, 13 Jun 2018 12:22:07 -0700 (PDT) X-Originating-IP: [2603:300b:6:5100:1052:acc7:f9de:2b6d] In-Reply-To: <7D695D8A-D273-4914-9980-F4BD096A80A2@panasas.com> References: <201806131648.w5DGm7i4037714@repo.freebsd.org> <7D695D8A-D273-4914-9980-F4BD096A80A2@panasas.com> From: Warner Losh Date: Wed, 13 Jun 2018 13:22:07 -0600 X-Google-Sender-Auth: JBf3wyG9OztSEb0qrJ8lMhcmU7g Message-ID: Subject: Re: svn commit: r335066 - in head/sys: kern sys To: Ravi Pokala Cc: Warner Losh , src-committers , svn-src-all@freebsd.org, svn-src-head@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jun 2018 19:22:09 -0000 On Wed, Jun 13, 2018 at 1:12 PM, Ravi Pokala wrote: > Hi Warner, > > I have many questions... > > "Car limit"? I'm not sure what you mean by that? > So we implement the elevator algorithm, but limit the number of requests that we can put in each 'car'. Elevators themselves are sometimes call 'cars' for reasons I have no clue about. So we only put N requests into the sorting algorithm. We put the next request at the end, unsorted, and that has the side effect of starting over so the net request after that is sorted against the last one, but not the first N. This is to help bound software queuing latency. > This change starts tracking the total number of BIOs on the queue, but > doesn't actually do anything with that value. > Oh, you're right. We have a sysctl that counts them, I think, in Netflix's code base (or did, I can't find it in the current sources)... I should remove that from here. I'm surprised I didn't notice that before. > When there are more requests batched than the limit, your comment say you > "start over", and the code says you bioq_insert_tail(). Why does the latter > imply "start(ing) over"? > Because that also sets a new sorting point for inserting sorted. Warner > Thanks, > > Ravi (rpokala@) > > -----Original Message----- > From: on behalf of Warner Losh > > Date: 2018-06-13, Wednesday at 12:48 > To: , , < > svn-src-head@freebsd.org> > Subject: svn commit: r335066 - in head/sys: kern sys > > Author: imp > Date: Wed Jun 13 16:48:07 2018 > New Revision: 335066 > URL: https://svnweb.freebsd.org/changeset/base/335066 > > Log: > Implement a 'car limit' for bioq. > > Allow one to implement a 'car limit' for > bioq_disksort. debug.bioq_batchsize sets the size of car limit. Every > time we queue that many requests, we start over so that we limit the > latency for requests when the software queue depths are large. A value > of '0', the default, means to revert to the old behavior. > > Sponsored by: Netflix > > Modified: > head/sys/kern/subr_disk.c > head/sys/sys/bio.h > > Modified: head/sys/kern/subr_disk.c > ============================================================ > ================== > --- head/sys/kern/subr_disk.c Wed Jun 13 15:58:33 2018 (r335065) > +++ head/sys/kern/subr_disk.c Wed Jun 13 16:48:07 2018 (r335066) > @@ -23,8 +23,13 @@ __FBSDID("$FreeBSD$"); > #include > #include > #include > +#include > #include > > +static int bioq_batchsize = 0; > +SYSCTL_INT(_debug, OID_AUTO, bioq_batchsize, CTLFLAG_RW, > + &bioq_batchsize, 0, "BIOQ batch size"); > + > /*- > * Disk error is the preface to plaintive error messages > * about failing disk transfers. It prints messages of the form > @@ -152,6 +157,8 @@ bioq_init(struct bio_queue_head *head) > TAILQ_INIT(&head->queue); > head->last_offset = 0; > head->insert_point = NULL; > + head->total = 0; > + head->batched = 0; > } > > void > @@ -165,6 +172,7 @@ bioq_remove(struct bio_queue_head *head, struct bio *b > head->insert_point = NULL; > > TAILQ_REMOVE(&head->queue, bp, bio_queue); > + head->total--; > } > > void > @@ -183,6 +191,8 @@ bioq_insert_head(struct bio_queue_head *head, struct b > if (head->insert_point == NULL) > head->last_offset = bp->bio_offset; > TAILQ_INSERT_HEAD(&head->queue, bp, bio_queue); > + head->total++; > + head->batched = 0; > } > > void > @@ -190,6 +200,7 @@ bioq_insert_tail(struct bio_queue_head *head, struct b > { > > TAILQ_INSERT_TAIL(&head->queue, bp, bio_queue); > + head->total++; > head->insert_point = bp; > head->last_offset = bp->bio_offset; > } > @@ -248,6 +259,11 @@ bioq_disksort(struct bio_queue_head *head, struct bio > return; > } > > + if (bioq_batchsize > 0 && head->batched > bioq_batchsize) { > + bioq_insert_tail(head, bp); > + return; > + } > + > prev = NULL; > key = bioq_bio_key(head, bp); > cur = TAILQ_FIRST(&head->queue); > @@ -266,4 +282,6 @@ bioq_disksort(struct bio_queue_head *head, struct bio > TAILQ_INSERT_HEAD(&head->queue, bp, bio_queue); > else > TAILQ_INSERT_AFTER(&head->queue, prev, bp, bio_queue); > + head->total++; > + head->batched++; > } > > Modified: head/sys/sys/bio.h > ============================================================ > ================== > --- head/sys/sys/bio.h Wed Jun 13 15:58:33 2018 (r335065) > +++ head/sys/sys/bio.h Wed Jun 13 16:48:07 2018 (r335066) > @@ -138,6 +138,8 @@ struct bio_queue_head { > TAILQ_HEAD(bio_queue, bio) queue; > off_t last_offset; > struct bio *insert_point; > + int total; > + int batched; > }; > > extern struct vm_map *bio_transient_map; > > > >