From owner-freebsd-hackers@FreeBSD.ORG Wed Jul 25 21:14:11 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9711E106564A; Wed, 25 Jul 2012 21:14:11 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 48FB28FC0A; Wed, 25 Jul 2012 21:14:11 +0000 (UTC) Received: from [127.0.0.1] (pooker.samsco.org [168.103.85.57]) (authenticated bits=0) by pooker.samsco.org (8.14.5/8.14.5) with ESMTP id q6PLE2bT012334; Wed, 25 Jul 2012 15:14:03 -0600 (MDT) (envelope-from scottl@samsco.org) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=iso-8859-1 From: Scott Long In-Reply-To: <501056C4.3080806@FreeBSD.org> Date: Wed, 25 Jul 2012 14:14:02 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <23011628-17F1-49A0-A41E-E7A8A8E3EA64@samsco.org> References: <501056C4.3080806@FreeBSD.org> To: Andriy Gapon X-Mailer: Apple Mail (2.1278) X-Spam-Status: No, score=-50.0 required=3.8 tests=ALL_TRUSTED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on pooker.samsco.org X-Mailman-Approved-At: Wed, 25 Jul 2012 22:07:55 +0000 Cc: freebsd-scsi@freebsd.org, freebsd-hackers@freebsd.org, freebsd-geom@freebsd.org Subject: Re: geom <-> cam disk X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2012 21:14:11 -0000 Once the bio is put into the bioq from da_strategy, the CAM scheduler is = called. It may or may not wind up calling dastart right away; if the = simq or devq is frozen, or if the devq has been exhausted, then the io = will be deferred until later and the call stack will unwind back into = g_down. The bioq can therefore accumulate many bio's before being = drained. Draining will usually happen from the camisr, at which point = you can potentially have i/o being initiated from both the camisr and = the g_down threads in parallel. The monolithic locking in CAM right now = prevents this from actually happening, though that's a topic that needs = to be revisited. Scott On Jul 25, 2012, at 1:27 PM, Andriy Gapon wrote: >=20 >=20 > Preamble. I am trying to understand in detail how things work at GEOM = <-> "CAM > disk" boundary. I am looking at scsi_da and ata_da which seem to be = twins in > this respect. >=20 > I got an impression that the bioq_disksort calls in the strategy = methods and the > related queues are completely useless in the GEOM single-threaded = world. > There is only one thread, g_down, that can call a strategy method, the = method > enqueues a bio, then calls a schedule function and through = xpt_schedule the call > flow continues to a start method which dequeues the bio and off it = goes. > I currently can see how a bio queue can accumulate more than one bio. >=20 > What am I missing? :-) > I will be very glad to learn more about this layer if anyone is = willing to > educate me. > Thank you in advance. >=20 > P.S. I wrote a very simple to DTrace script to my "theory" = experimentally and my > testing with various workloads didn't disprove the theory so far = (which doesn't > mean that it is correct, of course). >=20 > The script: > fbt::bioq_disksort:entry > /args[0]->queue.tqh_first =3D=3D 0/ > { > @["empty"] =3D count(); > } >=20 > fbt::bioq_disksort:entry > /args[0]->queue.tqh_first !=3D 0/ > { > @["non-empty"] =3D count(); > } >=20 > It works on all bioq_disksort calls, but I stressing only ada disks at = the moment. > --=20 > Andriy Gapon >=20