From owner-freebsd-questions@FreeBSD.ORG Thu Mar 22 11:41:08 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BA531065670 for ; Thu, 22 Mar 2012 11:41:08 +0000 (UTC) (envelope-from Traiano.Welcome@mtnbusiness.co.za) Received: from smtprelay01.ops.mtnbusiness.co.za (smtprelay01.ops.mtnbusiness.co.za [41.181.93.235]) by mx1.freebsd.org (Postfix) with ESMTP id B73308FC1C for ; Thu, 22 Mar 2012 11:41:07 +0000 (UTC) Received: from [196.30.97.135] (helo=CPT-EXCH01.int.mtnbusiness.net) by smtprelay01.ops.mtnbusiness.co.za with esmtp (ULTRA Special SMTP Internal Alpha) (envelope-from ) id 1SAgO8-0009l6-0c; Thu, 22 Mar 2012 13:41:04 +0200 Received: from CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) by CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) with mapi id 14.01.0218.012; Thu, 22 Mar 2012 13:40:57 +0200 From: Traiano Welcome To: Mark Blackman Thread-Topic: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' Thread-Index: AQHNCAowmWzmAjIpD0afh6AVbHiFtJZ18VCAgAA/5oA= Date: Thu, 22 Mar 2012 11:40:57 +0000 Message-ID: In-Reply-To: <4C89080E-48D3-4C6C-8945-227713769E91@exonetric.com> Accept-Language: en-US, en-ZA Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 x-originating-ip: [196.30.72.139] Content-Type: text/plain; charset="us-ascii" Content-ID: <724DC752C574084482AAA201CD0E7BDC@int.mtnbusiness.net> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: freebsd questions Subject: Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Mar 2012 11:41:08 -0000 Hi Mark On 22/03/2012 11:52, "Mark Blackman" wrote: > >On 22 Mar 2012, at 09:00, Traiano Welcome wrote: > >>=20 >>=20 >> My question is: What does this error mean, and how can I resolve it? > > >From a very casual inspection of the problem, I'd say you're pushing out >syslog messages faster than the kernel can get them out the interface. >How many syslog messages are going in (per second) and what kind of >network interface are you trying to send them out through? That's what I thought as well, but it's the details that evade me. Almost all traffic to and from this server is UDP (syslog), the graph I sent earlier shows the kind of volumes and trends that are typical: Peak traffic during the problem periods averages at about 1 Mbps outbound and 200 Kbps inbound to/from the interface. The interface itself is a Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950. Here are a couple of netstat polls during one of the problem periods: ---- [root@syslog2]# date;netstat -p udp -s |egrep -w "(received|delivered|dropped)" Thu Mar 22 12:11:34 SAST 2012 19969 datagrams received 2 dropped due to no socket 0 dropped due to full socket buffers 19967 delivered . . . [root@syslog2~]# date;netstat -p udp -s |egrep -w "(received|delivered|dropped)" Thu Mar 22 13:36:46 SAST 2012 662385 datagrams received 118 dropped due to no socket 0 dropped due to full socket buffers 662267 delivered --- Somehow this doesn't strike me as a large volume of throughput ... > >>=20 >> I have tried to frame this as an operating system kernel resource issue, >> and experimented with increasing the freebsd kernel sysctls for UDP >> performance: > > >I think you can push nmbclusters up to about 600k, but if your input is >running faster than your output, no amount of buffering will permanently >stave off this problem. I've done that just in the last 2 hours, though I agree with you that this is probably a (very) temporary imrovement. > >- Mark > > >