From owner-freebsd-current@FreeBSD.ORG Tue Dec 8 04:28:42 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7756C1065693; Tue, 8 Dec 2009 04:28:42 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from mail-yx0-f171.google.com (mail-yx0-f171.google.com [209.85.210.171]) by mx1.freebsd.org (Postfix) with ESMTP id E4D0A8FC13; Tue, 8 Dec 2009 04:28:41 +0000 (UTC) Received: by yxe1 with SMTP id 1so4700866yxe.3 for ; Mon, 07 Dec 2009 20:28:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=18RP6zZBtbHMks0ACLpGwj68uoRJP1d8yeaYb521LJY=; b=Im+Adx3Rer1RwF0OuXokLeLotDFpE2lSm4UrNWS8S2NBgvuXOzmfSL1U7dtq7N4zyH ZQgYaMzgzX67VWv+T5dMnUcvBj8W7vPpk/IMT7x+szOjIQKEJz9eBFu2x9qJlj3WJ+Rm nX5KSEw9ZJnpBf2648TknLCYjzMWEHcbiro4A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=b6hJixwNSEkVEpDgnsx08aDw+xvvKDZWq6vc9C4Su02n2CZrObAa86UmndKaIxzAMI JwfvKTlat8Y+vagycPWf4XYLwnWpWgAAycfUM2VL3mnMaYL7R3JDXgVja+MsXdnjgjnk 17BFhIqyE8MVQZMOEK9KKl5Wsj+sHEOqN6iPY= MIME-Version: 1.0 Received: by 10.101.159.30 with SMTP id l30mr6305809ano.56.1260246521005; Mon, 07 Dec 2009 20:28:41 -0800 (PST) In-Reply-To: <3c0b01820912072017x7d85c9e3t875692d7264bc05@mail.gmail.com> References: <3c0b01820912071342u1c722b2clf9c8413e40097279@mail.gmail.com> <200912071931.46002.jkim@FreeBSD.org> <200912072005.02662.jkim@FreeBSD.org> <3A549504-2AFE-4133-A8EF-642D53BC9F73@samsco.org> <3c0b01820912072000l7ad1a67ek3514dfccb96417be@mail.gmail.com> <0FFC216C-E938-48E4-B0E4-351077C6088A@samsco.org> <3c0b01820912072017x7d85c9e3t875692d7264bc05@mail.gmail.com> Date: Mon, 7 Dec 2009 23:28:40 -0500 Message-ID: <3c0b01820912072028g4146523fxf9c649913095365e@mail.gmail.com> From: Alexander Sack To: Scott Long Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: scottl@freebsd.org, freebsd-current@freebsd.org, emaste@freebsd.org, Jung-uk Kim Subject: Re: aac(4) resource FIB starvation on BUS scan revisited X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Dec 2009 04:28:42 -0000 On Mon, Dec 7, 2009 at 11:17 PM, Alexander Sack wrote: > On Mon, Dec 7, 2009 at 11:04 PM, Scott Long wrote: >> >> On Dec 7, 2009, at 9:00 PM, Alexander Sack wrote: >> >>> On Mon, Dec 7, 2009 at 8:14 PM, Scott Long wrote: >>>> >>>> On Dec 7, 2009, at 6:05 PM, Jung-uk Kim wrote: >>>>> >>>>> On Monday 07 December 2009 07:47 pm, Scott Long wrote: >>>>>> >>>>>> On Dec 7, 2009, at 5:31 PM, Jung-uk Kim wrote: >>>>>>> >>>>>>> On Monday 07 December 2009 05:30 pm, Alexander Sack wrote: >>>>>>>> >>>>>>>> On Mon, Dec 7, 2009 at 4:42 PM, Alexander Sack >>>>>>>> >>>>>>> >>>>>>> wrote: >>>>>>>>> >>>>>>>>> Folks: >>>>>>>>> >>>>>>>>> I posted a similar thread on freebsd-scsi only to realize that >>>>>>>>> scottl had fixed my first issue during some MP CAM cleanup with >>>>>>>>> respect to a race during resource allocation issues on a later >>>>>>>>> version of the driver we are using (I believe we did the same >>>>>>>>> thing to resolve a lock issue on bootup). >>>>>>>>> >>>>>>>>> However on my RELENG_8 box with (2) Adaptec 5085s connected to >>>>>>>>> some JBODs (9TB each) I still have a FIB starvation issue >>>>>>>>> during the LUN scan: >>>>>>>>> >>>>>>>>> The number of FIBs allocated to this card is 512 (older cards >>>>>>>>> are 256). =A0The max_target per bus is 287. =A0On a six channel >>>>>>>>> controller with a BUS scan done in parallel I see a lot of >>>>>>>>> this: >>>>>>>>> >>>>>>>>> ... >>>>>>>>> (probe501:aacp1:0:214:0): Request Requeued >>>>>>>>> (probe501:aacp1:0:214:0): Retrying Command >>>>>>>>> (probe520:aacp1:0:233:0): Request Requeued >>>>>>>>> (probe520:aacp1:0:233:0): Retrying Command >>>>>>>>> (probe528:aacp1:0:241:0): Request Requeued >>>>>>>>> (probe528:aacp1:0:241:0): Retrying Command >>>>>>>>> (probe540:aacp1:0:253:0): Request Requeued >>>>>>>>> (probe540:aacp1:0:253:0): Retrying Command >>>>>>>>> (probe541:aacp1:0:254:0): Request Requeued >>>>>>>>> (probe541:aacp1:0:254:0): Retrying Command >>>>>>>>> .... >>>>>>>>> >>>>>>>>> I think the driver is much happier with the following attached >>>>>>>>> patch (with dmesg). >>>>>>>> >>>>>>>> Patch again but this time not base-64 encoded: >>>>>>> >>>>>>> [SNIP!] >>>>>>> >>>>>>> I want it to be little conservative here, i.e., pre-allocating >>>>>>> half of max_fibs. =A0Will the attached patch work for you? >>>>>> >>>>>> The FIB allocation scheme was written when it was common for >>>>>> machines to only have 64MB of RAM and proportionally less KVA, so >>>>>> 256KB or 512KB was a lot of RAM to wire down. =A0Those days have >>>>>> probably passed. >>>>> >>>>> So, what would do if you were hypothetically rewriting it today? :-) >>>>> >>>> >>>> Most hardware have mechanisms for probing their command queue depth. >>>> =A0What I >>>> typically do these days is allocate a minimum number of commands so th= at >>>> this probing can be done, then do a single slab allocation based on th= e >>>> results. =A0AAC doesn't have this capability, but the 256/512 size is >>>> pretty >>>> well understood. =A0The page-by-page allocation of aac works, but adds >>>> extra >>>> bookkeeping and complication to the driver. >>>> >>> >>> Right Scott, that is what JK and I discussed this evening. =A0I figured >>> the 128 macro was just historical cruft and your email confirms it. >>> So are we ALL okay with the original patch as it stands for now? =A0JK = I >>> am fine with the divide 2 change but I think raising it to 256 is >>> really the way to go at this point! =A0:D >> >> >> If you're going to increase it, why not simply increase it to the max am= ount >> that is appropriate for each card? > > Totally right! =A0I thought though that the max fibs variable was set my > reading firmware bits up. =A0 Am I off? > > 1755 =A0 =A0 =A0 =A0 /* Check for broken hardware that does a lower numbe= r of > commands */ > 1756 =A0 =A0 =A0 =A0 sc->aac_max_fibs =3D (sc->flags & AAC_FLAGS_256FIBS = ? 256:512); > 1757 > > So checking against sc->aac_max_fibs would yield 512 up front on > modern controllers. > >> One other thing I forgot to mention was contiguous memory. =A0The page-b= y-page >> allocation in aac has another benefit, and that's to not tax contigmallo= c >> with finding 256KB of contiguous memory. That's not a big deal at boot, = but >> is a problem if you load the driver after the system has been running fo= r a >> while. =A0It's immensely useful during development, but it's never been = clear >> to me how useful it is in real life. > > True. =A0I can't imagine even today after loading it, it would be THAT > much of an issue (besides its a RAID controller, do you really think > you are going to load it so late in the game?). > > I am filing PR as we speak just to track! http://www.freebsd.org/cgi/query-pr.cgi?pr=3D141269 I botched the category though, it should probably be "scsi" please... -aps