From owner-freebsd-current@FreeBSD.ORG Tue Dec 8 04:04:47 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CBC621065695; Tue, 8 Dec 2009 04:04:47 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id D072C8FC1E; Tue, 8 Dec 2009 04:04:46 +0000 (UTC) Received: from ydesk.samsco.home (ydesk.samsco.home [192.168.254.15]) (authenticated bits=0) by pooker.samsco.org (8.14.2/8.14.2) with ESMTP id nB844WGQ024360; Mon, 7 Dec 2009 21:04:37 -0700 (MST) (envelope-from scottl@samsco.org) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Scott Long In-Reply-To: <3c0b01820912072000l7ad1a67ek3514dfccb96417be@mail.gmail.com> Date: Mon, 7 Dec 2009 21:04:31 -0700 Content-Transfer-Encoding: 7bit Message-Id: <0FFC216C-E938-48E4-B0E4-351077C6088A@samsco.org> References: <3c0b01820912071342u1c722b2clf9c8413e40097279@mail.gmail.com> <200912071931.46002.jkim@FreeBSD.org> <200912072005.02662.jkim@FreeBSD.org> <3A549504-2AFE-4133-A8EF-642D53BC9F73@samsco.org> <3c0b01820912072000l7ad1a67ek3514dfccb96417be@mail.gmail.com> To: Alexander Sack X-Mailer: Apple Mail (2.1076) X-Spam-Status: No, score=-4.4 required=3.8 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: scottl@freebsd.org, freebsd-current@freebsd.org, emaste@freebsd.org, Jung-uk Kim Subject: Re: aac(4) resource FIB starvation on BUS scan revisited X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Dec 2009 04:04:48 -0000 On Dec 7, 2009, at 9:00 PM, Alexander Sack wrote: > On Mon, Dec 7, 2009 at 8:14 PM, Scott Long wrote: >> On Dec 7, 2009, at 6:05 PM, Jung-uk Kim wrote: >>> >>> On Monday 07 December 2009 07:47 pm, Scott Long wrote: >>>> >>>> On Dec 7, 2009, at 5:31 PM, Jung-uk Kim wrote: >>>>> >>>>> On Monday 07 December 2009 05:30 pm, Alexander Sack wrote: >>>>>> >>>>>> On Mon, Dec 7, 2009 at 4:42 PM, Alexander Sack >>>>>> >>>>> >>>>> wrote: >>>>>>> >>>>>>> Folks: >>>>>>> >>>>>>> I posted a similar thread on freebsd-scsi only to realize that >>>>>>> scottl had fixed my first issue during some MP CAM cleanup with >>>>>>> respect to a race during resource allocation issues on a later >>>>>>> version of the driver we are using (I believe we did the same >>>>>>> thing to resolve a lock issue on bootup). >>>>>>> >>>>>>> However on my RELENG_8 box with (2) Adaptec 5085s connected to >>>>>>> some JBODs (9TB each) I still have a FIB starvation issue >>>>>>> during the LUN scan: >>>>>>> >>>>>>> The number of FIBs allocated to this card is 512 (older cards >>>>>>> are 256). The max_target per bus is 287. On a six channel >>>>>>> controller with a BUS scan done in parallel I see a lot of >>>>>>> this: >>>>>>> >>>>>>> ... >>>>>>> (probe501:aacp1:0:214:0): Request Requeued >>>>>>> (probe501:aacp1:0:214:0): Retrying Command >>>>>>> (probe520:aacp1:0:233:0): Request Requeued >>>>>>> (probe520:aacp1:0:233:0): Retrying Command >>>>>>> (probe528:aacp1:0:241:0): Request Requeued >>>>>>> (probe528:aacp1:0:241:0): Retrying Command >>>>>>> (probe540:aacp1:0:253:0): Request Requeued >>>>>>> (probe540:aacp1:0:253:0): Retrying Command >>>>>>> (probe541:aacp1:0:254:0): Request Requeued >>>>>>> (probe541:aacp1:0:254:0): Retrying Command >>>>>>> .... >>>>>>> >>>>>>> I think the driver is much happier with the following attached >>>>>>> patch (with dmesg). >>>>>> >>>>>> Patch again but this time not base-64 encoded: >>>>> >>>>> [SNIP!] >>>>> >>>>> I want it to be little conservative here, i.e., pre-allocating >>>>> half of max_fibs. Will the attached patch work for you? >>>> >>>> The FIB allocation scheme was written when it was common for >>>> machines to only have 64MB of RAM and proportionally less KVA, so >>>> 256KB or 512KB was a lot of RAM to wire down. Those days have >>>> probably passed. >>> >>> So, what would do if you were hypothetically rewriting it today? :-) >>> >> >> Most hardware have mechanisms for probing their command queue >> depth. What I >> typically do these days is allocate a minimum number of commands so >> that >> this probing can be done, then do a single slab allocation based on >> the >> results. AAC doesn't have this capability, but the 256/512 size is >> pretty >> well understood. The page-by-page allocation of aac works, but >> adds extra >> bookkeeping and complication to the driver. >> > > Right Scott, that is what JK and I discussed this evening. I figured > the 128 macro was just historical cruft and your email confirms it. > So are we ALL okay with the original patch as it stands for now? JK I > am fine with the divide 2 change but I think raising it to 256 is > really the way to go at this point! :D If you're going to increase it, why not simply increase it to the max amount that is appropriate for each card? One other thing I forgot to mention was contiguous memory. The page- by-page allocation in aac has another benefit, and that's to not tax contigmalloc with finding 256KB of contiguous memory. That's not a big deal at boot, but is a problem if you load the driver after the system has been running for a while. It's immensely useful during development, but it's never been clear to me how useful it is in real life. Scott