From owner-freebsd-hackers@FreeBSD.ORG Wed Jul 25 22:30:16 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A4B92106566B; Wed, 25 Jul 2012 22:30:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7206D8FC14; Wed, 25 Jul 2012 22:30:15 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id BAA24285; Thu, 26 Jul 2012 01:29:57 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1SuA5d-0009BM-28; Thu, 26 Jul 2012 01:29:57 +0300 Message-ID: <50107362.7050709@FreeBSD.org> Date: Thu, 26 Jul 2012 01:29:54 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120620 Thunderbird/13.0.1 MIME-Version: 1.0 To: Scott Long References: <501056C4.3080806@FreeBSD.org> <23011628-17F1-49A0-A41E-E7A8A8E3EA64@samsco.org> In-Reply-To: <23011628-17F1-49A0-A41E-E7A8A8E3EA64@samsco.org> X-Enigmail-Version: 1.4.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-scsi@FreeBSD.org, freebsd-hackers@FreeBSD.org, freebsd-geom@FreeBSD.org Subject: Re: geom <-> cam disk X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2012 22:30:16 -0000 on 26/07/2012 00:14 Scott Long said the following: > Once the bio is put into the bioq from da_strategy, the CAM scheduler is > called. It may or may not wind up calling dastart right away; if the simq or > devq is frozen, or if the devq has been exhausted, then the io will be > deferred until later and the call stack will unwind back into g_down. The > bioq can therefore accumulate many bio's before being drained. Draining will > usually happen from the camisr, at which point you can potentially have i/o > being initiated from both the camisr and the g_down threads in parallel. The Uh-hah. Thank you for the answer. I didn't think of the case of frozen/exhausted queues and also didn't hit in my tests. Now I am starting to understand the logic in xpt_run_dev_allocq. BTW, I think that it would be nice if the GEOM work-processing could re-use the CAM model. That is, try to execute GEOM bio transformations in the original thread as much as possible, defer work to the GEOM thread as the last resort. > monolithic locking in CAM right now prevents this from actually happening, > though that's a topic that needs to be revisited. > On Jul 25, 2012, at 1:27 PM, Andriy Gapon wrote: > >> >> >> Preamble. I am trying to understand in detail how things work at GEOM <-> >> "CAM disk" boundary. I am looking at scsi_da and ata_da which seem to be >> twins in this respect. >> >> I got an impression that the bioq_disksort calls in the strategy methods >> and the related queues are completely useless in the GEOM single-threaded >> world. There is only one thread, g_down, that can call a strategy method, >> the method enqueues a bio, then calls a schedule function and through >> xpt_schedule the call flow continues to a start method which dequeues the >> bio and off it goes. I currently can see how a bio queue can accumulate >> more than one bio. >> >> What am I missing? :-) I will be very glad to learn more about this layer >> if anyone is willing to educate me. Thank you in advance. >> >> P.S. I wrote a very simple to DTrace script to my "theory" experimentally >> and my testing with various workloads didn't disprove the theory so far >> (which doesn't mean that it is correct, of course). >> >> The script: fbt::bioq_disksort:entry /args[0]->queue.tqh_first == 0/ { >> @["empty"] = count(); } >> >> fbt::bioq_disksort:entry /args[0]->queue.tqh_first != 0/ { @["non-empty"] = >> count(); } >> >> It works on all bioq_disksort calls, but I stressing only ada disks at the >> moment. -- Andriy Gapon >> > -- Andriy Gapon