Date: Tue, 8 Dec 2009 04:19:11 GMT From: Alexander Sack <asack@niksun.com> To: freebsd-gnats-submit@FreeBSD.org Subject: amd64/141269: aac(4) FIB starvation during bus scan Message-ID: <200912080419.nB84JBOX000790@www.freebsd.org> Resent-Message-ID: <200912080420.nB84K2FM064705@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 141269
>Category: amd64
>Synopsis: aac(4) FIB starvation during bus scan
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-amd64
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Dec 08 04:20:02 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator: Alexander Sack
>Release: RELENG_8, CURRENT
>Organization:
Niksun
>Environment:
>Description:
aac(4) preallocates 128 FIBs during initialization time as its active command pool. Unfortunately, on modern controllers with a max targets of 287 and several buses (e.g. a 5085 controller has 6), at xpt_bus_scan time, it runs out very quickly causing lots of requeue/retry cycles increasing boot time. A sequential bus scan alleviates the problem completely.
Furthermore, the 128 number is half of what the minimum amount of FIBs modern controllers can handle. As the driver is written today, the minimum number of FIBs is actually 256 (upper bound is 512). In the interest of minimum command reallocation overhead, it might be wise to preallocate the maximum amount of FIBs which can also mitigate the BUS scan starvation issue described above.
>How-To-Repeat:
Use (2) 5085s in a single chassis and boot -v to see request/retry cycles
>Fix:
My patch I have been testing with a lot of success:
Index: aac.c
===================================================================
RCS file: /home/ncvs/src/sys/dev/aac/aac.c,v
retrieving revision 1.143.2.4
diff -u -r1.143.2.4 aac.c
--- aac.c 5 Nov 2009 18:34:01 -0000 1.143.2.4
+++ aac.c 7 Dec 2009 21:23:43 -0000
@@ -604,7 +604,7 @@
TAILQ_INIT(&sc->aac_fibmap_tqh);
sc->aac_commands = malloc(sc->aac_max_fibs * sizeof(struct aac_command),
M_AACBUF, M_WAITOK|M_ZERO);
- while (sc->total_fibs < AAC_PREALLOCATE_FIBS) {
+ while (sc->total_fibs < sc->aac_max_fibs) {
if (aac_alloc_commands(sc) != 0)
break;
}
Index: aac_cam.c
===================================================================
RCS file: /home/ncvs/src/sys/dev/aac/aac_cam.c,v
retrieving revision 1.31.2.2
diff -u -r1.31.2.2 aac_cam.c
--- aac_cam.c 5 Nov 2009 18:34:01 -0000 1.31.2.2
+++ aac_cam.c 7 Dec 2009 21:23:43 -0000
@@ -261,7 +261,7 @@
cpi->target_sprt = 0;
/* Resetting via the passthrough causes problems. */
- cpi->hba_misc = PIM_NOBUSRESET;
+ cpi->hba_misc = PIM_NOBUSRESET | PIM_SEQSCAN;
cpi->hba_eng_cnt = 0;
cpi->max_target = camsc->inf->TargetsPerBus;
cpi->max_lun = 8; /* Per the controller spec */
Index: aacvar.h
===================================================================
RCS file: /home/ncvs/src/sys/dev/aac/aacvar.h,v
retrieving revision 1.52.2.2
diff -u -r1.52.2.2 aacvar.h
--- aacvar.h 2 Nov 2009 16:54:23 -0000 1.52.2.2
+++ aacvar.h 7 Dec 2009 21:23:44 -0000
@@ -57,13 +57,6 @@
#define AAC_ADAPTER_FIBS 8
/*
- * FIBs are allocated in page-size chunks and can grow up to the 512
- * limit imposed by the hardware.
- */
-#define AAC_PREALLOCATE_FIBS 128
-#define AAC_NUM_MGT_FIB 8
-
-/*
* The controller reports status events in AIFs. We hang on to a number of
* these in order to pass them out to user-space management tools.
*/
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200912080419.nB84JBOX000790>