Date: Mon, 7 Dec 2009 18:14:06 -0700 From: Scott Long <scottl@samsco.org> To: Jung-uk Kim <jkim@FreeBSD.org> Cc: Alexander Sack <pisymbol@gmail.com>, scottl@FreeBSD.org, freebsd-current@FreeBSD.org, emaste@FreeBSD.org Subject: Re: aac(4) resource FIB starvation on BUS scan revisited Message-ID: <3A549504-2AFE-4133-A8EF-642D53BC9F73@samsco.org> In-Reply-To: <200912072005.02662.jkim@FreeBSD.org> References: <3c0b01820912071342u1c722b2clf9c8413e40097279@mail.gmail.com> <200912071931.46002.jkim@FreeBSD.org> <D7DDDA30-44B2-4E84-9F52-42DD2C43DC62@samsco.org> <200912072005.02662.jkim@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Dec 7, 2009, at 6:05 PM, Jung-uk Kim wrote: > On Monday 07 December 2009 07:47 pm, Scott Long wrote: >> On Dec 7, 2009, at 5:31 PM, Jung-uk Kim wrote: >>> On Monday 07 December 2009 05:30 pm, Alexander Sack wrote: >>>> On Mon, Dec 7, 2009 at 4:42 PM, Alexander Sack >>>> <pisymbol@gmail.com> >>> >>> wrote: >>>>> Folks: >>>>> >>>>> I posted a similar thread on freebsd-scsi only to realize that >>>>> scottl had fixed my first issue during some MP CAM cleanup with >>>>> respect to a race during resource allocation issues on a later >>>>> version of the driver we are using (I believe we did the same >>>>> thing to resolve a lock issue on bootup). >>>>> >>>>> However on my RELENG_8 box with (2) Adaptec 5085s connected to >>>>> some JBODs (9TB each) I still have a FIB starvation issue >>>>> during the LUN scan: >>>>> >>>>> The number of FIBs allocated to this card is 512 (older cards >>>>> are 256). The max_target per bus is 287. On a six channel >>>>> controller with a BUS scan done in parallel I see a lot of >>>>> this: >>>>> >>>>> ... >>>>> (probe501:aacp1:0:214:0): Request Requeued >>>>> (probe501:aacp1:0:214:0): Retrying Command >>>>> (probe520:aacp1:0:233:0): Request Requeued >>>>> (probe520:aacp1:0:233:0): Retrying Command >>>>> (probe528:aacp1:0:241:0): Request Requeued >>>>> (probe528:aacp1:0:241:0): Retrying Command >>>>> (probe540:aacp1:0:253:0): Request Requeued >>>>> (probe540:aacp1:0:253:0): Retrying Command >>>>> (probe541:aacp1:0:254:0): Request Requeued >>>>> (probe541:aacp1:0:254:0): Retrying Command >>>>> .... >>>>> >>>>> I think the driver is much happier with the following attached >>>>> patch (with dmesg). >>>> >>>> Patch again but this time not base-64 encoded: >>> >>> [SNIP!] >>> >>> I want it to be little conservative here, i.e., pre-allocating >>> half of max_fibs. Will the attached patch work for you? >> >> The FIB allocation scheme was written when it was common for >> machines to only have 64MB of RAM and proportionally less KVA, so >> 256KB or 512KB was a lot of RAM to wire down. Those days have >> probably passed. > > So, what would do if you were hypothetically rewriting it today? :-) > Most hardware have mechanisms for probing their command queue depth. What I typically do these days is allocate a minimum number of commands so that this probing can be done, then do a single slab allocation based on the results. AAC doesn't have this capability, but the 256/512 size is pretty well understood. The page-by-page allocation of aac works, but adds extra bookkeeping and complication to the driver. Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A549504-2AFE-4133-A8EF-642D53BC9F73>