Date: Tue, 7 Aug 2001 21:33:20 -0700 From: "Brian O'Shea" <boshea@ricochet.net> To: freebsd-hackers@freebsd.org Subject: Tuning the 4.1-R kernel for networking Message-ID: <20010807213320.D529@ricochet.net>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Hello,
I am using a PIII 550MHz UP system running FreeBSD 4.1-RELEASE. It has
a 3c905B-TX Fast Etherlink XL card.
# ifconfig xl0
xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 10.34.24.62 netmask 0xfffffc00 broadcast 10.34.27.255
inet6 fe80::2c0:4fff:fe20:3926%xl0 prefixlen 64 scopeid 0x1
ether 00:c0:4f:20:39:26
media: autoselect (100baseTX <full-duplex>) status: active
supported media: autoselect 100baseTX <full-duplex> 100baseTX
10baseT/UTP <full-duplex> 10baseT/UTP 100baseTX <hw-loopback>
On this machine I run a program which simulates many (~150) simultaneous
TCP clients. This is actually a multithreaded Linux binary, and one
thread per simulated TCP client is created. After a few seconds the
system runs out of mbuf clusters:
# netstat -m
3231/3392/6144 mbufs in use (current/peak/max):
1641 mbufs allocated to data
182 mbufs allocated to packet headers
1408 mbufs allocated to socket names and addresses
1536/1536/1536 mbuf clusters in use (current/peak/max)
3920 Kbytes allocated to network (98% in use)
96993 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Also, I see a steady stream of these messages on the console:
xl0: no memory for rx list -- packet dropped!
>From the xl(4) man page:
xl%d: no memory for rx list The driver failed to allocate an mbuf
for the receiver ring.
Looking at the xl_newbuf() function in the xl driver, there are two
places where this message can be generated. It looks like the problem
is with the second case where the MCLGET fails, since we are running out
of those.
/*
* Initialize an RX descriptor and attach an MBUF cluster.
*/
static int xl_newbuf(sc, c)
struct xl_softc *sc;
struct xl_chain_onefrag *c;
{
struct mbuf *m_new = NULL;
MGETHDR(m_new, M_DONTWAIT, MT_DATA);
if (m_new == NULL) {
printf("xl%d: no memory for rx list -- "
"packet dropped!\n", sc->xl_unit);
return(ENOBUFS);
}
MCLGET(m_new, M_DONTWAIT);
if (!(m_new->m_flags & M_EXT)) {
printf("xl%d: no memory for rx list -- "
"packet dropped!\n", sc->xl_unit);
m_freem(m_new);
return(ENOBUFS);
}
m_new->m_len = m_new->m_pkthdr.len = MCLBYTES;
/* Force longword alignment for packet payload. */
m_adj(m_new, ETHER_ALIGN);
c->xl_mbuf = m_new;
c->xl_ptr->xl_frag.xl_addr = vtophys(mtod(m_new, caddr_t));
c->xl_ptr->xl_frag.xl_len = MCLBYTES | XL_LAST_FRAG;
c->xl_ptr->xl_status = 0;
return(0);
}
I increased maxusers to 128 (2560 mbuf clusters) and it ran out of mbuf
clusters again. Then I increased it to 256 (4608 mbuf clusters), with
the same results. I don't have any sense of what is reasonable mbuf
cluster usage for the application that I am running, but the system
never seems to recover from the condition, which would seem to point to
an mbuf cluster leak.
Does this sound like a problem with the driver (mbuf cluster leak), or
with the way that I have tuned this system? (the kernel config file for
this system is attached)
I compiled a debug kernel and panicked the system while it was in the
state described above, in case that is any use. I don't know how to
analyze the crash dump to determine where the problem is. Any
suggestions are welcome.
Thanks,
-brian
--
Brian O'Shea
<boshea@ricochet.net>
[-- Attachment #2 --]
#
# SHAOLIN -- Kernel based on the GENERIC configuration file for FreeBSD/i386
#
machine i386
cpu I686_CPU
ident SHAOLIN
maxusers 256
makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
#options MATH_EMULATE #Support for x87 emulation
options INET #InterNETworking
options INET6 #IPv6 communications protocols
options FFS #Berkeley Fast Filesystem
options FFS_ROOT #FFS usable as root device [keep this!]
options SOFTUPDATES #Enable FFS soft updates support
options MFS #Memory Filesystem
options MD_ROOT #MD is a potential root device
options NFS #Network Filesystem
options NFS_ROOT #NFS usable as root device, NFS required
options MSDOSFS #MSDOS Filesystem
options CD9660 #ISO 9660 Filesystem
options CD9660_ROOT #CD-ROM usable as root, CD9660 required
options PROCFS #Process filesystem
options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI
options UCONSOLE #Allow users to grab the console
options USERCONFIG #boot -c editor
options VISUAL_USERCONFIG #visual boot -c editor
options KTRACE #ktrace(1) support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options P1003_1B #Posix P1003_1B real-time extensions
options _KPOSIX_PRIORITY_SCHEDULING
options ICMP_BANDLIM #Rate limit bad replies
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options DDB #Enable the kernel debugger
device isa
device eisa
device pci
# Floppy drives
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device fd1 at fdc0 drive 1
# ATA and ATAPI devices
device ata0 at isa? port IO_WD1 irq 14
device ata1 at isa? port IO_WD2 irq 15
device ata
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID #Static device numbering
options ATA_ENABLE_ATAPI_DMA #Enable DMA on ATAPI devices
# atkbdc0 controls both the keyboard and the PS/2 mouse
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1 flags 0x1
device psm0 at atkbdc? irq 12
device vga0 at isa?
# splash screen/screen saver
pseudo-device splash
# syscons is the default console driver, resembling an SCO console
device sc0 at isa? flags 0x100
# Floating point support - do not disable.
device npx0 at nexus? port IO_NPX irq 13
# Power management support (see LINT for more options)
device apm0 at nexus? disable flags 0x20 # Advanced Power Management
# Serial (COM) ports
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
device sio2 at isa? disable port IO_COM3 irq 5
device sio3 at isa? disable port IO_COM4 irq 9
# Parallel port
device ppc0 at isa? irq 7
device ppbus # Parallel port bus (required)
device lpt # Printer
device plip # TCP/IP over parallel
device ppi # Parallel port interface device
#device vpo # Requires scbus and da
# PCI Ethernet NICs that use the common MII bus controller code.
device miibus # MII bus support
device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'')
# Sound card support
device pcm
# Pseudo devices - the number indicates how many units to allocated.
pseudo-device loop # Network loopback
pseudo-device ether # Ethernet support
pseudo-device sl 1 # Kernel SLIP
pseudo-device ppp 1 # Kernel PPP
pseudo-device tun # Packet tunnel.
pseudo-device pty # Pseudo-ttys (telnet etc)
pseudo-device md # Memory "disks"
pseudo-device gif 4 # IPv6 and IPv4 tunneling
pseudo-device faith 1 # IPv6-to-IPv4 relaying (translation)
# The `bpf' pseudo-device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
pseudo-device bpf #Berkeley packet filter
# USB support
device uhci # UHCI PCI->USB interface
device ohci # OHCI PCI->USB interface
device usb # USB Bus (required)
device ugen # Generic
device uhid # "Human Interface Devices"
device ukbd # Keyboard
device ulpt # Printer
#device umass # Disks/Mass storage - Requires scbus and da
device ums # Mouse
# USB Ethernet, requires mii
device aue # ADMtek USB ethernet
device cue # CATC USB ethernet
device kue # Kawasaki LSI USB ethernet
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010807213320.D529>