From owner-freebsd-amd64@FreeBSD.ORG Tue Dec 8 04:20:02 2009 Return-Path: Delivered-To: freebsd-amd64@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB25B1065696 for ; Tue, 8 Dec 2009 04:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9E2CD8FC1A for ; Tue, 8 Dec 2009 04:20:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id nB84K2sj064706 for ; Tue, 8 Dec 2009 04:20:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id nB84K2FM064705; Tue, 8 Dec 2009 04:20:02 GMT (envelope-from gnats) Resent-Date: Tue, 8 Dec 2009 04:20:02 GMT Resent-Message-Id: <200912080420.nB84K2FM064705@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-amd64@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Alexander Sack Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C7C11065693 for ; Tue, 8 Dec 2009 04:19:12 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id 405B08FC27 for ; Tue, 8 Dec 2009 04:19:12 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id nB84JBoT000793 for ; Tue, 8 Dec 2009 04:19:11 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id nB84JBOX000790; Tue, 8 Dec 2009 04:19:11 GMT (envelope-from nobody) Message-Id: <200912080419.nB84JBOX000790@www.freebsd.org> Date: Tue, 8 Dec 2009 04:19:11 GMT From: Alexander Sack To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 X-Mailman-Approved-At: Tue, 08 Dec 2009 04:46:22 +0000 Cc: Subject: amd64/141269: aac(4) FIB starvation during bus scan X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Dec 2009 04:20:02 -0000 >Number: 141269 >Category: amd64 >Synopsis: aac(4) FIB starvation during bus scan >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-amd64 >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Dec 08 04:20:02 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Alexander Sack >Release: RELENG_8, CURRENT >Organization: Niksun >Environment: >Description: aac(4) preallocates 128 FIBs during initialization time as its active command pool. Unfortunately, on modern controllers with a max targets of 287 and several buses (e.g. a 5085 controller has 6), at xpt_bus_scan time, it runs out very quickly causing lots of requeue/retry cycles increasing boot time. A sequential bus scan alleviates the problem completely. Furthermore, the 128 number is half of what the minimum amount of FIBs modern controllers can handle. As the driver is written today, the minimum number of FIBs is actually 256 (upper bound is 512). In the interest of minimum command reallocation overhead, it might be wise to preallocate the maximum amount of FIBs which can also mitigate the BUS scan starvation issue described above. >How-To-Repeat: Use (2) 5085s in a single chassis and boot -v to see request/retry cycles >Fix: My patch I have been testing with a lot of success: Index: aac.c =================================================================== RCS file: /home/ncvs/src/sys/dev/aac/aac.c,v retrieving revision 1.143.2.4 diff -u -r1.143.2.4 aac.c --- aac.c 5 Nov 2009 18:34:01 -0000 1.143.2.4 +++ aac.c 7 Dec 2009 21:23:43 -0000 @@ -604,7 +604,7 @@ TAILQ_INIT(&sc->aac_fibmap_tqh); sc->aac_commands = malloc(sc->aac_max_fibs * sizeof(struct aac_command), M_AACBUF, M_WAITOK|M_ZERO); - while (sc->total_fibs < AAC_PREALLOCATE_FIBS) { + while (sc->total_fibs < sc->aac_max_fibs) { if (aac_alloc_commands(sc) != 0) break; } Index: aac_cam.c =================================================================== RCS file: /home/ncvs/src/sys/dev/aac/aac_cam.c,v retrieving revision 1.31.2.2 diff -u -r1.31.2.2 aac_cam.c --- aac_cam.c 5 Nov 2009 18:34:01 -0000 1.31.2.2 +++ aac_cam.c 7 Dec 2009 21:23:43 -0000 @@ -261,7 +261,7 @@ cpi->target_sprt = 0; /* Resetting via the passthrough causes problems. */ - cpi->hba_misc = PIM_NOBUSRESET; + cpi->hba_misc = PIM_NOBUSRESET | PIM_SEQSCAN; cpi->hba_eng_cnt = 0; cpi->max_target = camsc->inf->TargetsPerBus; cpi->max_lun = 8; /* Per the controller spec */ Index: aacvar.h =================================================================== RCS file: /home/ncvs/src/sys/dev/aac/aacvar.h,v retrieving revision 1.52.2.2 diff -u -r1.52.2.2 aacvar.h --- aacvar.h 2 Nov 2009 16:54:23 -0000 1.52.2.2 +++ aacvar.h 7 Dec 2009 21:23:44 -0000 @@ -57,13 +57,6 @@ #define AAC_ADAPTER_FIBS 8 /* - * FIBs are allocated in page-size chunks and can grow up to the 512 - * limit imposed by the hardware. - */ -#define AAC_PREALLOCATE_FIBS 128 -#define AAC_NUM_MGT_FIB 8 - -/* * The controller reports status events in AIFs. We hang on to a number of * these in order to pass them out to user-space management tools. */ >Release-Note: >Audit-Trail: >Unformatted: