From owner-freebsd-current@FreeBSD.ORG Tue Dec 8 17:11:21 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71470106566C; Tue, 8 Dec 2009 17:11:21 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from mail-yw0-f194.google.com (mail-yw0-f194.google.com [209.85.211.194]) by mx1.freebsd.org (Postfix) with ESMTP id E40E08FC2F; Tue, 8 Dec 2009 17:11:20 +0000 (UTC) Received: by ywh32 with SMTP id 32so5943995ywh.14 for ; Tue, 08 Dec 2009 09:11:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ml6uL4BsTBG5gpx27XJgTI7+xKqs9EHGQUdesySF4mA=; b=qW/Koc0dutbMbNylyvk4wYsuN9cpJjWXotn/qgCnvARA2OwRTcY/U6QqB/X7M+C9cx m1IUG1aazn0gU5mvpKKalIvXddnQ6UIFa+umNzxuutOfFBCQFCfGPftZRLIB8eJP3KXK ExrWexxqU6jLvxI1u3OEdwLOqT4D/DMm4Afxk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=tYcAEjM0T/xyS+4Bi4RDj6O1ExNeQSt81W5ufjGujCbMmpGdYKQS51fTUw+OEjmDaY 0VSL+ALa806vrFZx+574OqLmX2/h5rGajlOdq131W6FM4O+np9HEAdGI3m3JyVURCjwl teFHtZ3ggxF8TEc1i5tRhdoksN8lHcqRm8Jwg= MIME-Version: 1.0 Received: by 10.101.136.12 with SMTP id o12mr1127920ann.26.1260292279776; Tue, 08 Dec 2009 09:11:19 -0800 (PST) In-Reply-To: <200912081122.02870.jkim@FreeBSD.org> References: <3c0b01820912071342u1c722b2clf9c8413e40097279@mail.gmail.com> <3c0b01820912072000l7ad1a67ek3514dfccb96417be@mail.gmail.com> <0FFC216C-E938-48E4-B0E4-351077C6088A@samsco.org> <200912081122.02870.jkim@FreeBSD.org> Date: Tue, 8 Dec 2009 12:11:19 -0500 Message-ID: <3c0b01820912080911k5ea4d749qe33ab8dfb0a8c205@mail.gmail.com> From: Alexander Sack To: Jung-uk Kim Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: scottl@freebsd.org, freebsd-current@freebsd.org, emaste@freebsd.org Subject: Re: aac(4) resource FIB starvation on BUS scan revisited X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Dec 2009 17:11:21 -0000 On Tue, Dec 8, 2009 at 11:22 AM, Jung-uk Kim wrote: > On Monday 07 December 2009 11:04 pm, Scott Long wrote: >> On Dec 7, 2009, at 9:00 PM, Alexander Sack wrote: >> > On Mon, Dec 7, 2009 at 8:14 PM, Scott Long > wrote: >> >> On Dec 7, 2009, at 6:05 PM, Jung-uk Kim wrote: >> >>> On Monday 07 December 2009 07:47 pm, Scott Long wrote: >> >>>> On Dec 7, 2009, at 5:31 PM, Jung-uk Kim wrote: >> >>>>> On Monday 07 December 2009 05:30 pm, Alexander Sack wrote: >> >>>>>> On Mon, Dec 7, 2009 at 4:42 PM, Alexander Sack >> >>>>>> >> >>>>> >> >>>>> wrote: >> >>>>>>> Folks: >> >>>>>>> >> >>>>>>> I posted a similar thread on freebsd-scsi only to realize >> >>>>>>> that scottl had fixed my first issue during some MP CAM >> >>>>>>> cleanup with respect to a race during resource allocation >> >>>>>>> issues on a later version of the driver we are using (I >> >>>>>>> believe we did the same thing to resolve a lock issue on >> >>>>>>> bootup). >> >>>>>>> >> >>>>>>> However on my RELENG_8 box with (2) Adaptec 5085s connected >> >>>>>>> to some JBODs (9TB each) I still have a FIB starvation >> >>>>>>> issue during the LUN scan: >> >>>>>>> >> >>>>>>> The number of FIBs allocated to this card is 512 (older >> >>>>>>> cards are 256). =A0The max_target per bus is 287. =A0On a six >> >>>>>>> channel controller with a BUS scan done in parallel I see a >> >>>>>>> lot of this: >> >>>>>>> >> >>>>>>> ... >> >>>>>>> (probe501:aacp1:0:214:0): Request Requeued >> >>>>>>> (probe501:aacp1:0:214:0): Retrying Command >> >>>>>>> (probe520:aacp1:0:233:0): Request Requeued >> >>>>>>> (probe520:aacp1:0:233:0): Retrying Command >> >>>>>>> (probe528:aacp1:0:241:0): Request Requeued >> >>>>>>> (probe528:aacp1:0:241:0): Retrying Command >> >>>>>>> (probe540:aacp1:0:253:0): Request Requeued >> >>>>>>> (probe540:aacp1:0:253:0): Retrying Command >> >>>>>>> (probe541:aacp1:0:254:0): Request Requeued >> >>>>>>> (probe541:aacp1:0:254:0): Retrying Command >> >>>>>>> .... >> >>>>>>> >> >>>>>>> I think the driver is much happier with the following >> >>>>>>> attached patch (with dmesg). >> >>>>>> >> >>>>>> Patch again but this time not base-64 encoded: >> >>>>> >> >>>>> [SNIP!] >> >>>>> >> >>>>> I want it to be little conservative here, i.e., >> >>>>> pre-allocating half of max_fibs. =A0Will the attached patch >> >>>>> work for you? >> >>>> >> >>>> The FIB allocation scheme was written when it was common for >> >>>> machines to only have 64MB of RAM and proportionally less KVA, >> >>>> so 256KB or 512KB was a lot of RAM to wire down. =A0Those days >> >>>> have probably passed. >> >>> >> >>> So, what would do if you were hypothetically rewriting it >> >>> today? :-) >> >> >> >> Most hardware have mechanisms for probing their command queue >> >> depth. =A0What I >> >> typically do these days is allocate a minimum number of commands >> >> so that >> >> this probing can be done, then do a single slab allocation based >> >> on the >> >> results. =A0AAC doesn't have this capability, but the 256/512 size >> >> is pretty >> >> well understood. =A0The page-by-page allocation of aac works, but >> >> adds extra >> >> bookkeeping and complication to the driver. >> > >> > Right Scott, that is what JK and I discussed this evening. =A0I >> > figured the 128 macro was just historical cruft and your email >> > confirms it. So are we ALL okay with the original patch as it >> > stands for now? =A0JK I am fine with the divide 2 change but I >> > think raising it to 256 is really the way to go at this point! >> > :D >> >> If you're going to increase it, why not simply increase it to the >> max amount that is appropriate for each card? > > My intention was to minimize impact as little as possible, i.e., > > old card: max fibs =3D=3D 256, max fibs / 2 =3D=3D 128, no change > new card: max fibs =3D=3D 512, max fibs / 2 =3D=3D 256, twice > > Old cards are most likely to be used on old systems with very little > RAM (if they are still in production). =A0Hence, no change is > necessary. =A0Anyway I just committed OP's patch (with a minor comment > tweak). Thanks JK! >> One other thing I forgot to mention was contiguous memory. =A0The >> page- by-page allocation in aac has another benefit, and that's to >> not tax contigmalloc with finding 256KB of contiguous memory. >> That's not a big deal at boot, but is a problem if you load the >> driver after the system has been running for a while. =A0It's >> immensely useful during development, but it's never been clear to >> me how useful it is in real life. > > Thanks for your review and comments! Ditto to everyone! :D -aps