From owner-freebsd-arch@FreeBSD.ORG Tue Mar 23 00:05:26 2010 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03C38106564A; Tue, 23 Mar 2010 00:05:26 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 96FA18FC08; Tue, 23 Mar 2010 00:05:25 +0000 (UTC) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.14.3/8.14.3) with ESMTP id o2N05MI0056853; Mon, 22 Mar 2010 18:05:22 -0600 (MDT) (envelope-from scottl@samsco.org) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Scott Long In-Reply-To: <20100322233607.GB1767@garage.freebsd.pl> Date: Mon, 22 Mar 2010 18:05:22 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <8D465FFB-0389-4321-84B9-E45292697D26@samsco.org> References: <4BA633A0.2090108@icyb.net.ua> <5754.1269246223@critter.freebsd.dk> <20100322233607.GB1767@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1077) X-Spam-Status: No, score=-1.0 required=3.8 tests=ALL_TRUSTED autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on pooker.samsco.org Cc: freebsd-arch@FreeBSD.org, Poul-Henning Kamp , freebsd-current@FreeBSD.org, Alexander Motin , Andriy Gapon Subject: Re: Increasing MAXPHYS X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Mar 2010 00:05:26 -0000 On Mar 22, 2010, at 5:36 PM, Pawel Jakub Dawidek wrote: > On Mon, Mar 22, 2010 at 08:23:43AM +0000, Poul-Henning Kamp wrote: >> In message <4BA633A0.2090108@icyb.net.ua>, Andriy Gapon writes: >>> on 21/03/2010 16:05 Alexander Motin said the following: >>>> Ivan Voras wrote: >>>>> Hmm, it looks like it could be easy to spawn more g_* threads = (and, >>>>> barring specific class behaviour, it has a fair chance of working = out of >>>>> the box) but the incoming queue will need to also be broken up for >>>>> greater effect. >>>>=20 >>>> According to "notes", looks there is a good chance to obtain races, = as >>>> some places expect only one up and one down thread. >>>=20 >>> I haven't given any deep thought to this issue, but I remember us = discussing >>> them over beer :-) >>=20 >> The easiest way to obtain more parallelism, is to divide the mesh = into >> multiple independent meshes. >>=20 >> This will do you no good if you have five disks in a RAID-5 config, = but >> if you have two disks each mounted on its own filesystem, you can run >> a g_up & g_down for each of them. >=20 > A class is suppose to interact with other classes only via GEOM, so I > think it should be safe to choose g_up/g_down threads for each class > individually, for example: >=20 > /dev/ad0s1a (DEV) > | > g_up_0 + g_down_0 > | > ad0s1a (BSD) > | > g_up_1 + g_down_1 > | > ad0s1 (MBR) > | > g_up_2 + g_down_2 > | > ad0 (DISK) >=20 > We could easly calculate g_down thread based on bio_to->geom->class = and > g_up thread based on bio_from->geom->class, so we know I/O requests = for > our class are always coming from the same threads. >=20 > If we could make the same assumption for geoms it would allow for even > better distribution. The whole point of the discussion, sans PHK's interlude, is to reduce = the context switches and indirection, not to increase it. But if you = can show decreased latency/higher-iops benefits of increasing it, more = power to you. I would think that the results of DFly's experiment with = parallelism-via-more-queues would serve as a good warning, though. Scott