From owner-freebsd-current@FreeBSD.ORG Tue Dec 8 04:00:16 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 309671065676; Tue, 8 Dec 2009 04:00:16 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from mail-gx0-f218.google.com (mail-gx0-f218.google.com [209.85.217.218]) by mx1.freebsd.org (Postfix) with ESMTP id 9AEFA8FC14; Tue, 8 Dec 2009 04:00:15 +0000 (UTC) Received: by gxk10 with SMTP id 10so3893531gxk.3 for ; Mon, 07 Dec 2009 20:00:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=z/9fbOimk/AxnKnaOQJBdVozFLxLWPpCguhs4auc2yw=; b=vIQim0Z6KqRCD/laYrDEKL9zFnIvOjXqgRIO4ZsRee596SiKyLvu8ePb70StBIeU4X fc4Ok2ICi8rlbk1itS2jsGFGDo8NHhIpBoR3cSXz9QzPGaF+maoGgq6GPZHf+FyRQt9j CVsGx1b3qI7Xia53+dmrHzz0DrQbI+K41Puq4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=Avg5hCjfu7TTAhg5huKqf+eBnLvUSWLMs+WKp+mUp+BGKHLb3Dg0ABy3r98Jpm8rAa Ev4K4itiLMbXYa3reN0sz85yZK7E4h5hMNDVYxh9kReblF9wk0j1yOV6xQkVSOKNeKqQ kX5gY3sKiexni6YW6BZsQ9zdPNttWFT4GQKxY= MIME-Version: 1.0 Received: by 10.101.159.30 with SMTP id l30mr6302404ano.56.1260244814785; Mon, 07 Dec 2009 20:00:14 -0800 (PST) In-Reply-To: <3A549504-2AFE-4133-A8EF-642D53BC9F73@samsco.org> References: <3c0b01820912071342u1c722b2clf9c8413e40097279@mail.gmail.com> <200912071931.46002.jkim@FreeBSD.org> <200912072005.02662.jkim@FreeBSD.org> <3A549504-2AFE-4133-A8EF-642D53BC9F73@samsco.org> Date: Mon, 7 Dec 2009 23:00:14 -0500 Message-ID: <3c0b01820912072000l7ad1a67ek3514dfccb96417be@mail.gmail.com> From: Alexander Sack To: Scott Long Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: scottl@freebsd.org, freebsd-current@freebsd.org, emaste@freebsd.org, Jung-uk Kim Subject: Re: aac(4) resource FIB starvation on BUS scan revisited X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Dec 2009 04:00:16 -0000 On Mon, Dec 7, 2009 at 8:14 PM, Scott Long wrote: > On Dec 7, 2009, at 6:05 PM, Jung-uk Kim wrote: >> >> On Monday 07 December 2009 07:47 pm, Scott Long wrote: >>> >>> On Dec 7, 2009, at 5:31 PM, Jung-uk Kim wrote: >>>> >>>> On Monday 07 December 2009 05:30 pm, Alexander Sack wrote: >>>>> >>>>> On Mon, Dec 7, 2009 at 4:42 PM, Alexander Sack >>>>> >>>> >>>> wrote: >>>>>> >>>>>> Folks: >>>>>> >>>>>> I posted a similar thread on freebsd-scsi only to realize that >>>>>> scottl had fixed my first issue during some MP CAM cleanup with >>>>>> respect to a race during resource allocation issues on a later >>>>>> version of the driver we are using (I believe we did the same >>>>>> thing to resolve a lock issue on bootup). >>>>>> >>>>>> However on my RELENG_8 box with (2) Adaptec 5085s connected to >>>>>> some JBODs (9TB each) I still have a FIB starvation issue >>>>>> during the LUN scan: >>>>>> >>>>>> The number of FIBs allocated to this card is 512 (older cards >>>>>> are 256). =A0The max_target per bus is 287. =A0On a six channel >>>>>> controller with a BUS scan done in parallel I see a lot of >>>>>> this: >>>>>> >>>>>> ... >>>>>> (probe501:aacp1:0:214:0): Request Requeued >>>>>> (probe501:aacp1:0:214:0): Retrying Command >>>>>> (probe520:aacp1:0:233:0): Request Requeued >>>>>> (probe520:aacp1:0:233:0): Retrying Command >>>>>> (probe528:aacp1:0:241:0): Request Requeued >>>>>> (probe528:aacp1:0:241:0): Retrying Command >>>>>> (probe540:aacp1:0:253:0): Request Requeued >>>>>> (probe540:aacp1:0:253:0): Retrying Command >>>>>> (probe541:aacp1:0:254:0): Request Requeued >>>>>> (probe541:aacp1:0:254:0): Retrying Command >>>>>> .... >>>>>> >>>>>> I think the driver is much happier with the following attached >>>>>> patch (with dmesg). >>>>> >>>>> Patch again but this time not base-64 encoded: >>>> >>>> [SNIP!] >>>> >>>> I want it to be little conservative here, i.e., pre-allocating >>>> half of max_fibs. =A0Will the attached patch work for you? >>> >>> The FIB allocation scheme was written when it was common for >>> machines to only have 64MB of RAM and proportionally less KVA, so >>> 256KB or 512KB was a lot of RAM to wire down. =A0Those days have >>> probably passed. >> >> So, what would do if you were hypothetically rewriting it today? :-) >> > > Most hardware have mechanisms for probing their command queue depth. =A0W= hat I > typically do these days is allocate a minimum number of commands so that > this probing can be done, then do a single slab allocation based on the > results. =A0AAC doesn't have this capability, but the 256/512 size is pre= tty > well understood. =A0The page-by-page allocation of aac works, but adds ex= tra > bookkeeping and complication to the driver. > Right Scott, that is what JK and I discussed this evening. I figured the 128 macro was just historical cruft and your email confirms it. So are we ALL okay with the original patch as it stands for now? JK I am fine with the divide 2 change but I think raising it to 256 is really the way to go at this point! :D -aps