Date: Mon, 3 Nov 2003 18:05:33 -0600 From: John <strgout@unixjunkie.com> To: freebsd-stable@freebsd.org Subject: (long) high traffic syslog server. Message-ID: <20031104000519.GA31319@mail.unixjunkie.com>
next in thread | raw e-mail | index | archive | help
I have some questions about what needs tuned on a high traffic syslog box.
I seem to be dropping quite a few syslog packets.
This is a syslog server for a high usage Firewall btw.
Nic is a Compaq tl0
4.8-P13
netstat -s -p udp | grep buf
19,762,079 dropped due to full socket buffers
uptime
5:28PM up 7 days, 18:30, 2 users, load averages: 0.21, 0.23, 0.23
I though maybe syslogd was the problem, but running nc on the syslog port and
sending output to /dev/null still shows the buffer problem.
i've tried uping net.inet.udp.recvspace
if this gets too high i will no longer be able to send udp packets
and will get a socket buff full err.
net.local.dgram.recvspace This didn't do much.
i tried moving kern.ipc.maxsockbuf in by doubling each time
This didn't help
kern.ipc.maxsockbuf: 1048576 <- This is what it currently is set to.
if someone could point me in the right direction that would be great :).
here is some info on the box in question.
btw all these command were run while the system
was doing about 1500 pps (as per netstat -inb 1)
kern.maxfilesperproc: 8272
kern.openfiles: 86
btw syslogd runs at %20 cpu from top
systat -vm 1
shows disk mostly idle (1-5% usage).
this box has 6 9 gig drives in raid5 also.
Which i think show up as one drive.
/dev/idad0s2a on / (ufs, local)
/dev/idad0s2f on /tmp (ufs, local)
/dev/idad0s2e on /usr (ufs, local, soft-updates)
/dev/idad0s2g on /var (ufs, local, soft-updates)
ps -axwwj | grep syslogd
root 84 1 84 c500e740 0 Rs ?? 1601:25.44 /usr/sbin/syslogd -n
ps -axwwu | grep syslogd
root 84 18.6 0.1 972 620 ?? Rs 26Oct03 1601:30.54 /usr/sbin/syslogd -n
ifconfig tl0
tl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether 00:08:c7:9f:78:1e
media: Ethernet 100baseTX <full-duplex>
status: active
netstat -inb 1
This can peak at around 2100 pps.
low is about 600 pps.
packets errs bytes packets errs bytes colls
1568 0 226804 6 0 0 0
1274 0 200785 1 0 178 0
netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
tl0 1500 <Link#1> 00:08:c7:9f:78:1e 713151669 0 83482 0 0
netstat -s -p udp
udp:
711282523 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
1 with no checksum
306 dropped due to no socket
0 broadcast/multicast datagrams dropped due to no socket
19783694 dropped due to full socket buffers
0 not for hashed pcb
691498523 delivered
20954 datagrams output
netstat -m
66/336/81408 mbufs in use (current/peak/max):
66 mbufs allocated to data
64/220/20352 mbuf clusters in use (current/peak/max)
524 Kbytes allocated to network (0% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
I was using ipf, but now its disabled (no rules, and ipf -D).
top line of.. top
CPU states: 9.9% user, 0.0% nice, 9.3% system, 3.3% interrupt, 77.5% idle
Mem: 12M Active, 461M Inact, 64M Wired, 25M Cache, 67M Buf, 1076K Free
Swap: 768M Total, 112K Used, 768M Free
dmesg.boot
btw its a dual 400
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 4.8-RELEASE-p13 #2: Sun Oct 26 22:47:48 CST 2003
root@ME.MYDOMAIN.com:/usr/obj/usr/src/sys/SYSLOG
Timecounter "i8254" frequency 1193182 Hz
Timecounter "TSC" frequency 399072197 Hz
CPU: Pentium II/Pentium II Xeon/Celeron (399.07-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x652 Stepping = 2
Features=0x183fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CM
OV,PAT,PSE36,MMX,FXSR>
real memory = 603979776 (589824K bytes)
avail memory = 583192576 (569524K bytes)
Preloaded elf kernel "kernel" at 0xc0368000.
Pentium Pro MTRR support enabled
md0: Malloc disk
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Intel 82443BX host to PCI bridge (AGP disabled)> on motherboard
pci0: <PCI bus> on pcib0
pci0: <Cirrus Logic GD5446 SVGA controller> at 11.0
pcib1: <DEC 21150 PCI-PCI bridge> at device 13.0 on pci0
pci1: <PCI bus> on pcib1
tl0: <Compaq Netelligent 10/100 Proliant> port 0x2c00-0x2c0f mem 0xc6efcdf0-0xc6
efcdff irq 5 at device 7.0 on pci1
tl0: Ethernet address: 00:08:c7:9f:78:1e
miibus0: <MII bus> on tl0
nsphy0: <DP83840 10/100 media interface> on miibus0
nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
tlphy0: <ThunderLAN 10baseT media interface> on miibus0
tlphy0: 10base2/BNC, 10base5/AUI
sym0: <875> port 0x2000-0x20ff mem 0xc6eff000-0xc6efffff,0xc6efcf00-0xc6efcfff i
rq 9 at device 9.0 on pci1
sym0: No NVRAM, ID 7, Fast-20, SE, parity checking
sym1: <875> port 0x2400-0x24ff mem 0xc6efe000-0xc6efefff,0xc6efce00-0xc6efceff i
rq 10 at device 9.1 on pci1
sym1: No NVRAM, ID 7, Fast-20, SE, parity checking
pci1: <unknown card> (vendor=0x10b8, dev=0x0005) at 10.0 irq 15
pci0: <unknown card> (vendor=0x0e11, dev=0xa0f0) at 14.0
pcib2: <IBM 82351 PCI-PCI bridge> at device 15.0 on pci0
pci2: <PCI bus> on pcib2
ida0: <Compaq SMART-2/P array controller> port 0x3000-0x30ff mem 0xb8000000-0xbf
ffffff,0xc6ffff00-0xc6ffffff irq 11 at device 0.0 on pci2
ida0: drives=1 firm_rev=3.08
idad0: <Compaq Logical Drive> on ida0
idad0: 34707MB (71081760 sectors), blocksize=512
isab0: <Intel 82371AB PCI to ISA bridge> at device 20.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 ATA33 controller> port 0xf100-0xf10f at device 20.1 on pci
0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <Intel 82371AB/EB (PIIX4) USB controller> at 20.2 irq 0
chip1: <Intel 82371AB Power management controller> at device 20.3 on pci0
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff,0xe8000-0xedfff,0xe
e000-0xeffff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse Explorer, device ID 4
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: parallel port not found.
IP Filter: v3.4.31 initialized. Default = pass all, Logging = enabled
acd0: CDROM <CD-ROM CDU701-Q> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
Mounting root from ufs:/dev/idad0s2a
KERN CONFIG file
machine i386
cpu I686_CPU
ident SYSLOG
options INET #InterNETworking
options INET6 #IPv6 communications protocols
options FFS #Berkeley Fast Filesystem
options FFS_ROOT #FFS usable as root device [keep this!]
options SOFTUPDATES #Enable FFS soft updates support
options MFS #Memory Filesystem
options MD_ROOT #MD is a potential root device
options NFS #Network Filesystem
options NFS_ROOT #NFS usable as root device, NFS required
options MSDOSFS #MSDOS Filesystem
options CD9660 #ISO 9660 Filesystem
options CD9660_ROOT #CD-ROM usable as root, CD9660 required
options PROCFS #Process filesystem
options COMPAT_43 #Compatible with BSD 4.3 [KEEP THIS!]
options SCSI_DELAY=15000 #Delay (in ms) before probing SCSI
options UCONSOLE #Allow users to grab the console
options USERCONFIG #boot -c editor
options VISUAL_USERCONFIG #visual boot -c editor
options KTRACE #ktrace(1) support
options SYSVSHM #SYSV-style shared memory
options SYSVMSG #SYSV-style message queues
options SYSVSEM #SYSV-style semaphores
options P1003_1B #Posix P1003_1B real-time extensions
options _KPOSIX_PRIORITY_SCHEDULING
options ICMP_BANDLIM #Rate limit bad replies
options KBD_INSTALL_CDEV # install a CDEV entry in /dev
options IPFILTER #ipfilter support
options IPFILTER_LOG #ipfilter logging
options SC_NORM_ATTR="(FG_GREEN|BG_BLACK)"
options SC_NORM_REV_ATTR="(FG_YELLOW|BG_GREEN)"
options SC_KERNEL_CONS_ATTR="(FG_RED|BG_BLACK)"
options SC_KERNEL_CONS_REV_ATTR="(FG_BLACK|BG_RED)"
options UFS_DIRHASH
options INCLUDE_CONFIG_FILE
options NMBUFS=81408
options NMBCLUSTERS=20352
device isa
device pci
device fdc0 at isa? port IO_FD1 irq 6 drq 2
device fd0 at fdc0 drive 0
device fd1 at fdc0 drive 1
device ata0 at isa? port IO_WD1 irq 14
device ata1 at isa? port IO_WD2 irq 15
device ata
device atadisk # ATA disk drives
device atapicd # ATAPI CDROM drives
device atapifd # ATAPI floppy drives
device atapist # ATAPI tape drives
options ATA_STATIC_ID #Static device numbering
device sym # NCR/Symbios Logic (newer chipsets)
device scbus # SCSI bus (required)
device da # Direct Access (disks)
device pass # Passthrough device (direct SCSI access)
device ida # Compaq Smart RAID
device atkbdc0 at isa? port IO_KBD
device atkbd0 at atkbdc? irq 1 flags 0x1
device psm0 at atkbdc? irq 12
device vga0 at isa?
pseudo-device splash
device sc0 at isa? flags 0x100
device npx0 at nexus? port IO_NPX irq 13
device sio0 at isa? port IO_COM1 flags 0x10 irq 4
device sio1 at isa? port IO_COM2 irq 3
device sio2 at isa? disable port IO_COM3 irq 5
device sio3 at isa? disable port IO_COM4 irq 9
device ppc0 at isa? irq 7
device ppbus # Parallel port bus (required)
device lpt # Printer
device plip # TCP/IP over parallel
device ppi # Parallel port interface device
device miibus # MII bus support
device fxp # Intel EtherExpress PRO/100B (82557, 82558)
device tl # Texas Instruments ThunderLAN
pseudo-device loop # Network loopback
pseudo-device ether # Ethernet support
pseudo-device pty # Pseudo-ttys (telnet etc)
pseudo-device md # Memory "disks"
pseudo-device bpf #Berkeley packet filter
same random stuff from /etc/sysctl.conf
net.inet.udp.recvspace=84160
net.inet.tcp.blackhole=1
net.inet.udp.blackhole=1
net.inet.icmp.log_redirect=1
net.inet.tcp.log_in_vain=1
Is this too much info btw?
I just wanted to make sure i didn't get a, not enough info
reply, sorry if this was too much.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031104000519.GA31319>