Date: Fri, 17 Aug 2012 20:45:05 GMT From: Garrett Cooper <yanegomi@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/170713: [cxgb] Driver must be loaded after boot due to timing issues checking for kern.ipc.nmb* values set via /boot/loader.conf Message-ID: <201208172045.q7HKj5Da094133@red.freebsd.org> Resent-Message-ID: <201208172050.q7HKo5ji002041@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 170713
>Category: kern
>Synopsis: [cxgb] Driver must be loaded after boot due to timing issues checking for kern.ipc.nmb* values set via /boot/loader.conf
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Fri Aug 17 20:50:04 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator: Garrett Cooper
>Release: 7.3-RELEASE/7.4-RELEASE
>Organization:
EMC Isilon
>Environment:
FreeBSD 7.4-RELEASE-p10 FreeBSD 7.4-RELEASE-p10 #0: Fri Aug 17 07:15:01 UTC 2012 root@:/usr/obj/mnt/freebsd/releng/7.4/sys/ISI-GENERIC amd64
>Description:
There's a timing issue where the following section of code in cxgb_sge.c always fails on 7.3-RELEASE/7.4-RELEASE when trying to load the driver because the values being checked are set to default values (in particular the jumbo_q_size check fails):
383 #if __FreeBSD_version >= 700111
384 if (cxgb_use_16k_clusters)
385 jumbo_q_size = min(nmbjumbo16/(3*nqsets), JUMBO_Q_SIZE);
386 else
387 jumbo_q_size = min(nmbjumbo9/(3*nqsets), JUMBO_Q_SIZE);
388 #else
389 jumbo_q_size = min(nmbjumbop/(3*nqsets), JUMBO_Q_SIZE);
390 #endif
391 while (!powerof2(jumbo_q_size))
392 jumbo_q_size--;
393
394 if (fl_q_size < (FL_Q_SIZE / 4) || jumbo_q_size < (JUMBO_Q_SIZE / 2))
395 device_printf(adap->dev,
396 "Insufficient clusters and/or jumbo buffers.\n");
Example (7.3-RELEASE with the patch noted below):
cxgbc0: <Chelsio T320, 2 ports> mem 0xfaf7e000-0xfaf7efff,0xfaf7f000-0xfaf7ffff irq 24 at device 0.0 on pci8
cxgbc0: Insufficient clusters and/or jumbo buffers (4096 < 1024 or 256 < 512).
cxgbc0: using MSI-X interrupts (9 vectors)
cxgb0: <Port 0 10GBASE-R> on cxgbc0
cxgb0: Ethernet address: 00:07:43:07:41:1c
cxgb1: <Port 1 10GBASE-R> on cxgbc0
cxgb1: Ethernet address: 00:07:43:07:41:1d
cxgbc0: Firmware Version 7.8.0
Example (7.4-RELEASE):
cxgbc0: <Chelsio T320, 2 ports> mem 0xfaf7e000-0xfaf7efff,0xfaf7f000-0xfaf7ffff irq 24 at device 0.0 on pci8
cxgbc0: Insufficient clusters and/or jumbo buffers.
cxgbc0: using MSI-X interrupts (9 vectors)
cxgb0: <Port 0 10GBASE-R> on cxgbc0
cxgb0: Ethernet address: 00:07:43:07:41:1c
cxgb1: <Port 1 10GBASE-R> on cxgbc0
cxgb1: Ethernet address: 00:07:43:07:41:1d
cxgbc0: Firmware Version 7.8.0
I've worked around this issue by loading the driver later in the boot process,
but this chicken and egg problem should be properly resolved with the proper calls being made to kern_ipc.c before sge is attached to the kernel.
Please note that this issue does not exist on 9.0-RELEASE (or at least didn't exist in the limited testing I've done.. the race still might be there but ameliorated); did not verify if this issue exists on 8.x.
More details:
# cat /boot/loader.conf
if_cxgb_load="YES"
if_em_load="YES"
kern.ipc.nmbclusters="250000"
kern.ipc.nmbjumbo9=262144
kern.ipc.nmbjumbo16=262144
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144
kern.ipc.maxsockbuf=2097152
>How-To-Repeat:
1. Set values for kern.ipc to values shown above.
2. Load if_cxgb at boot.
3. Boot the kernel.
>Fix:
Something needs to be backported from 9.x to 7.x in order to fix this chicken and egg problem, or better sequencing needs to be done, so the mbuf tunables are read in before the driver is probed and attached.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201208172045.q7HKj5Da094133>