From owner-freebsd-performance@FreeBSD.ORG Thu Dec 8 11:49:51 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2860816A420 for ; Thu, 8 Dec 2005 11:49:51 +0000 (GMT) (envelope-from imriz@co.zahav.net.il) Received: from mrx.wan.inter.net.il (mrx.wan.inter.net.il [192.117.191.51]) by mx1.FreeBSD.org (Postfix) with SMTP id 6F55D43D8C for ; Thu, 8 Dec 2005 11:49:37 +0000 (GMT) (envelope-from imriz@co.zahav.net.il) Received: from igfe.InetGold ([172.33.1.137]) by mrx.wan.inter.net.il (SAVSMTP 3.1.0.29) with SMTP id M2005120813504605731 ; Thu, 08 Dec 2005 13:50:46 +0200 Received: from IGMAIL.InetGold ([172.31.100.164]) by igfe.InetGold with Microsoft SMTPSVC(6.0.3790.1830); Thu, 8 Dec 2005 13:49:24 +0200 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Thu, 8 Dec 2005 13:54:51 +0200 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: very busy syslog server Thread-Index: AcX77VSyFQkHCRIkRCKNV70UWt2CBwAAD+4w From: "Imri Zvik" To: "jaws" X-OriginalArrivalTime: 08 Dec 2005 11:49:24.0139 (UTC) FILETIME=[6F336FB0:01C5FBED] Cc: freebsd-performance@freebsd.org Subject: RE: very busy syslog server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2005 11:49:51 -0000 Already tried different values, even some really high arbitrary ones... I think I will try splitting the work load on a couple of instances of syslog...=20 I hope that would do the trick -- Imri Zvik PGP (2.6.3ia) Public Key: http://mariska.inter.net.il/~imriz/imriz.pgp =20 -----Original Message----- From: jaws [mailto:jaws@skyinet.net]=20 Sent: Thursday, December 08, 2005 1:48 PM To: Imri Zvik Cc: j_guojun@lbl.gov; freebsd-performance@freebsd.org Subject: Re: very busy syslog server What you need is more sockets available and a much larger sockbufs. Try=20 adjusting the following parameters: net.inet.udp.maxdgram net.inet.udp.recvspace Dont rely on the default values rather set it to a much higher values. Regards, jaws Imri Zvik wrote: >Hi, > >1. The NIC being used is "Intel(R) PRO/1000" (the em(4) driver). >2. The CPU utilization in average is between 15% and 20%. >3. This machine is being used _only_ for the sysloging - the database resides on another server. > >Meanwhile, I have added some more memory to the machine, and now it has 3GB of RAM, but I am still seeing packets being dropped due to full socket buffers. > >Thanks, > >-- >Imri Zvik >PGP (2.6.3ia) Public Key: http://mariska.inter.net.il/~imriz/imriz.pgp >=20 >________________________________________ >From: Jin Guojun [mailto:jinmtb@sbcglobal.net]=20 >Sent: Wednesday, December 07, 2005 9:56 PM >To: Sean Chittenden >Cc: Imri Zvik; freebsd-performance@freebsd.org >Subject: Re: very busy syslog server > >Sean Chittenden wrote:=20 >I'm trying to setup a syslog server to serve a large group of >servers. For the syslog daemon, I have chosen rsyslogd, and the >backend is mysql (on a different machine). > >The machine has 2 Intel Xeon 2.80GHz CPUs, and 1GB of RAM, and it is >running FreeBSD 6 (6.0-STABLE). > >The problem is, that I see a lot of UDP packets being dropped: > >udp: > 390202 datagrams received > 0 with incomplete header > 0 with bad data length field > 0 with bad checksum > 6 with no checksum > 0 dropped due to no socket > 0 broadcast/multicast datagrams dropped due to no socket >->>> 123677 dropped due to full socket buffers > 0 not for hashed pcb > 266525 delivered > 133260 datagrams output > >I have tried to increase net.inet.udp.recvspace, but it didn't solve >the problem. > >I would appreciate any hint or tips. > =20 > >When you're doing a large number of packets per second, you may want >to look into enabling device polling(4). Right now, every packet >results in an interrupt. With device polling, you can handle more >than one packet per interrupt. See the man page for details.=20 >Not quite, the interrupt interval depends on the device driver, or which NIC is used. >A number NICs are able to to interrupt coalescence, which requires to increase buffer >descriptor ring size (just for receiving buffer descriptors). Of course, polling is a simple thing >to try. > >Before we can come up a better way to alter a better solution for this case, you also need to >monitor a few things: > >What is NIC on this machine? > >What is the CUP utilization in average and in case the packet drops? You can simply write a >script to do this instead of instructing kernel to do so (since this needs no super accurate): > >run vmstat to record CPU utilization in every 1 to 3 seconds for use when following event happens: >use netstat watch UDP and pipe it to awk "netstat -udp | awk '$2=3D=3D"drooped" {print $1; exit}'" >every 3-5 seconds, and compare the result with previous one to see if any changes. If so, >grep the last couple of line from vmstat output records. > >>From your information, it seems that this machine has enough memory bandwidth for syslog needs, >since it is not clear what this machine is for rlog daemon or sql server, or both are on the same machine. >If the third case is true, then you may run out of memory bandwidth. Under this circumstance, >you need to obtain the packet rate and the average packet size in order to determine the I/O >and memory bandwidth requirements. > > -Jin Guojun >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > =20 >