From owner-freebsd-stable@FreeBSD.ORG Mon Nov 3 15:41:37 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5FB9E16A4DB for ; Mon, 3 Nov 2003 15:41:37 -0800 (PST) Received: from dmz2.unixjunkie.com (adsl-65-70-175-250.dsl.rcsntx.swbell.net [65.70.175.250]) by mx1.FreeBSD.org (Postfix) with ESMTP id 19F68440CF for ; Mon, 3 Nov 2003 15:41:17 -0800 (PST) (envelope-from strgout@unixjunkie.com) Received: from mail.unixjunkie.com (mail [10.253.254.36]) by dmz2.unixjunkie.com (8.12.8p2/8.12.8) with ESMTP id hA405YGf031427 for ; Mon, 3 Nov 2003 18:05:34 -0600 (CST) (envelope-from strgout@mail.unixjunkie.com) Received: from mail.unixjunkie.com (mail [10.253.254.36]) by mail.unixjunkie.com (8.12.8p2/8.12.8) with ESMTP id hA405Ymf031424 for ; Mon, 3 Nov 2003 18:05:34 -0600 (CST) (envelope-from strgout@mail.unixjunkie.com) Received: (from strgout@localhost) by mail.unixjunkie.com (8.12.8p2/8.12.8/Submit) id hA405YcO031423 for freebsd-stable@freebsd.org; Mon, 3 Nov 2003 18:05:34 -0600 (CST) (envelope-from strgout) Date: Mon, 3 Nov 2003 18:05:33 -0600 From: John To: freebsd-stable@freebsd.org Message-ID: <20031104000519.GA31319@mail.unixjunkie.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4i Subject: (long) high traffic syslog server. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Nov 2003 23:41:39 -0000 I have some questions about what needs tuned on a high traffic syslog box. I seem to be dropping quite a few syslog packets. This is a syslog server for a high usage Firewall btw. Nic is a Compaq tl0 4.8-P13 netstat -s -p udp | grep buf 19,762,079 dropped due to full socket buffers uptime 5:28PM up 7 days, 18:30, 2 users, load averages: 0.21, 0.23, 0.23 I though maybe syslogd was the problem, but running nc on the syslog port and sending output to /dev/null still shows the buffer problem. i've tried uping net.inet.udp.recvspace if this gets too high i will no longer be able to send udp packets and will get a socket buff full err. net.local.dgram.recvspace This didn't do much. i tried moving kern.ipc.maxsockbuf in by doubling each time This didn't help kern.ipc.maxsockbuf: 1048576 <- This is what it currently is set to. if someone could point me in the right direction that would be great :). here is some info on the box in question. btw all these command were run while the system was doing about 1500 pps (as per netstat -inb 1) kern.maxfilesperproc: 8272 kern.openfiles: 86 btw syslogd runs at %20 cpu from top systat -vm 1 shows disk mostly idle (1-5% usage). this box has 6 9 gig drives in raid5 also. Which i think show up as one drive. /dev/idad0s2a on / (ufs, local) /dev/idad0s2f on /tmp (ufs, local) /dev/idad0s2e on /usr (ufs, local, soft-updates) /dev/idad0s2g on /var (ufs, local, soft-updates) ps -axwwj | grep syslogd root 84 1 84 c500e740 0 Rs ?? 1601:25.44 /usr/sbin/syslogd -n ps -axwwu | grep syslogd root 84 18.6 0.1 972 620 ?? Rs 26Oct03 1601:30.54 /usr/sbin/syslogd -n ifconfig tl0 tl0: flags=8843 mtu 1500 ether 00:08:c7:9f:78:1e media: Ethernet 100baseTX status: active netstat -inb 1 This can peak at around 2100 pps. low is about 600 pps. packets errs bytes packets errs bytes colls 1568 0 226804 6 0 0 0 1274 0 200785 1 0 178 0 netstat -in Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll tl0 1500 00:08:c7:9f:78:1e 713151669 0 83482 0 0 netstat -s -p udp udp: 711282523 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 1 with no checksum 306 dropped due to no socket 0 broadcast/multicast datagrams dropped due to no socket 19783694 dropped due to full socket buffers 0 not for hashed pcb 691498523 delivered 20954 datagrams output netstat -m 66/336/81408 mbufs in use (current/peak/max): 66 mbufs allocated to data 64/220/20352 mbuf clusters in use (current/peak/max) 524 Kbytes allocated to network (0% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines I was using ipf, but now its disabled (no rules, and ipf -D). top line of.. top CPU states: 9.9% user, 0.0% nice, 9.3% system, 3.3% interrupt, 77.5% idle Mem: 12M Active, 461M Inact, 64M Wired, 25M Cache, 67M Buf, 1076K Free Swap: 768M Total, 112K Used, 768M Free dmesg.boot btw its a dual 400 Copyright (c) 1992-2003 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.8-RELEASE-p13 #2: Sun Oct 26 22:47:48 CST 2003 root@ME.MYDOMAIN.com:/usr/obj/usr/src/sys/SYSLOG Timecounter "i8254" frequency 1193182 Hz Timecounter "TSC" frequency 399072197 Hz CPU: Pentium II/Pentium II Xeon/Celeron (399.07-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x652 Stepping = 2 Features=0x183fbff real memory = 603979776 (589824K bytes) avail memory = 583192576 (569524K bytes) Preloaded elf kernel "kernel" at 0xc0368000. Pentium Pro MTRR support enabled md0: Malloc disk npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 pci0: at 11.0 pcib1: at device 13.0 on pci0 pci1: on pcib1 tl0: port 0x2c00-0x2c0f mem 0xc6efcdf0-0xc6 efcdff irq 5 at device 7.0 on pci1 tl0: Ethernet address: 00:08:c7:9f:78:1e miibus0: on tl0 nsphy0: on miibus0 nsphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto tlphy0: on miibus0 tlphy0: 10base2/BNC, 10base5/AUI sym0: <875> port 0x2000-0x20ff mem 0xc6eff000-0xc6efffff,0xc6efcf00-0xc6efcfff i rq 9 at device 9.0 on pci1 sym0: No NVRAM, ID 7, Fast-20, SE, parity checking sym1: <875> port 0x2400-0x24ff mem 0xc6efe000-0xc6efefff,0xc6efce00-0xc6efceff i rq 10 at device 9.1 on pci1 sym1: No NVRAM, ID 7, Fast-20, SE, parity checking pci1: (vendor=0x10b8, dev=0x0005) at 10.0 irq 15 pci0: (vendor=0x0e11, dev=0xa0f0) at 14.0 pcib2: at device 15.0 on pci0 pci2: on pcib2 ida0: port 0x3000-0x30ff mem 0xb8000000-0xbf ffffff,0xc6ffff00-0xc6ffffff irq 11 at device 0.0 on pci2 ida0: drives=1 firm_rev=3.08 idad0: on ida0 idad0: 34707MB (71081760 sectors), blocksize=512 isab0: at device 20.0 on pci0 isa0: on isab0 atapci0: port 0xf100-0xf10f at device 20.1 on pci 0 ata0: at 0x1f0 irq 14 on atapci0 ata1: at 0x170 irq 15 on atapci0 pci0: at 20.2 irq 0 chip1: at device 20.3 on pci0 orm0: