From owner-freebsd-bugs@FreeBSD.ORG Fri Aug 17 20:50:05 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84461106566C for ; Fri, 17 Aug 2012 20:50:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 568D08FC12 for ; Fri, 17 Aug 2012 20:50:05 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7HKo5ao002042 for ; Fri, 17 Aug 2012 20:50:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7HKo5ji002041; Fri, 17 Aug 2012 20:50:05 GMT (envelope-from gnats) Resent-Date: Fri, 17 Aug 2012 20:50:05 GMT Resent-Message-Id: <201208172050.q7HKo5ji002041@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Garrett Cooper Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2ED25106566C for ; Fri, 17 Aug 2012 20:45:06 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 05D438FC18 for ; Fri, 17 Aug 2012 20:45:06 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q7HKj5lh094140 for ; Fri, 17 Aug 2012 20:45:05 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q7HKj5Da094133; Fri, 17 Aug 2012 20:45:05 GMT (envelope-from nobody) Message-Id: <201208172045.q7HKj5Da094133@red.freebsd.org> Date: Fri, 17 Aug 2012 20:45:05 GMT From: Garrett Cooper To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/170713: [cxgb] Driver must be loaded after boot due to timing issues checking for kern.ipc.nmb* values set via /boot/loader.conf X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Aug 2012 20:50:05 -0000 >Number: 170713 >Category: kern >Synopsis: [cxgb] Driver must be loaded after boot due to timing issues checking for kern.ipc.nmb* values set via /boot/loader.conf >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Aug 17 20:50:04 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Garrett Cooper >Release: 7.3-RELEASE/7.4-RELEASE >Organization: EMC Isilon >Environment: FreeBSD 7.4-RELEASE-p10 FreeBSD 7.4-RELEASE-p10 #0: Fri Aug 17 07:15:01 UTC 2012 root@:/usr/obj/mnt/freebsd/releng/7.4/sys/ISI-GENERIC amd64 >Description: There's a timing issue where the following section of code in cxgb_sge.c always fails on 7.3-RELEASE/7.4-RELEASE when trying to load the driver because the values being checked are set to default values (in particular the jumbo_q_size check fails): 383 #if __FreeBSD_version >= 700111 384 if (cxgb_use_16k_clusters) 385 jumbo_q_size = min(nmbjumbo16/(3*nqsets), JUMBO_Q_SIZE); 386 else 387 jumbo_q_size = min(nmbjumbo9/(3*nqsets), JUMBO_Q_SIZE); 388 #else 389 jumbo_q_size = min(nmbjumbop/(3*nqsets), JUMBO_Q_SIZE); 390 #endif 391 while (!powerof2(jumbo_q_size)) 392 jumbo_q_size--; 393 394 if (fl_q_size < (FL_Q_SIZE / 4) || jumbo_q_size < (JUMBO_Q_SIZE / 2)) 395 device_printf(adap->dev, 396 "Insufficient clusters and/or jumbo buffers.\n"); Example (7.3-RELEASE with the patch noted below): cxgbc0: mem 0xfaf7e000-0xfaf7efff,0xfaf7f000-0xfaf7ffff irq 24 at device 0.0 on pci8 cxgbc0: Insufficient clusters and/or jumbo buffers (4096 < 1024 or 256 < 512). cxgbc0: using MSI-X interrupts (9 vectors) cxgb0: on cxgbc0 cxgb0: Ethernet address: 00:07:43:07:41:1c cxgb1: on cxgbc0 cxgb1: Ethernet address: 00:07:43:07:41:1d cxgbc0: Firmware Version 7.8.0 Example (7.4-RELEASE): cxgbc0: mem 0xfaf7e000-0xfaf7efff,0xfaf7f000-0xfaf7ffff irq 24 at device 0.0 on pci8 cxgbc0: Insufficient clusters and/or jumbo buffers. cxgbc0: using MSI-X interrupts (9 vectors) cxgb0: on cxgbc0 cxgb0: Ethernet address: 00:07:43:07:41:1c cxgb1: on cxgbc0 cxgb1: Ethernet address: 00:07:43:07:41:1d cxgbc0: Firmware Version 7.8.0 I've worked around this issue by loading the driver later in the boot process, but this chicken and egg problem should be properly resolved with the proper calls being made to kern_ipc.c before sge is attached to the kernel. Please note that this issue does not exist on 9.0-RELEASE (or at least didn't exist in the limited testing I've done.. the race still might be there but ameliorated); did not verify if this issue exists on 8.x. More details: # cat /boot/loader.conf if_cxgb_load="YES" if_em_load="YES" kern.ipc.nmbclusters="250000" kern.ipc.nmbjumbo9=262144 kern.ipc.nmbjumbo16=262144 kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 kern.ipc.maxsockbuf=2097152 >How-To-Repeat: 1. Set values for kern.ipc to values shown above. 2. Load if_cxgb at boot. 3. Boot the kernel. >Fix: Something needs to be backported from 9.x to 7.x in order to fix this chicken and egg problem, or better sequencing needs to be done, so the mbuf tunables are read in before the driver is probed and attached. >Release-Note: >Audit-Trail: >Unformatted: