From owner-freebsd-net@FreeBSD.ORG Thu Mar 22 10:03:31 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97479106566B for ; Thu, 22 Mar 2012 10:03:31 +0000 (UTC) (envelope-from Traiano.Welcome@mtnbusiness.co.za) Received: from smtprelay01.ops.mtnbusiness.co.za (smtprelay01.ops.mtnbusiness.co.za [41.181.93.235]) by mx1.freebsd.org (Postfix) with ESMTP id 9D8C28FC19 for ; Thu, 22 Mar 2012 10:03:30 +0000 (UTC) Received: from [196.30.97.135] (helo=CPT-EXCH01.int.mtnbusiness.net) by smtprelay01.ops.mtnbusiness.co.za with esmtp (ULTRA Special SMTP Internal Alpha) (envelope-from ) id 1SAerg-0007KL-84 for freebsd-net@freebsd.org; Thu, 22 Mar 2012 12:03:28 +0200 Received: from CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) by CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) with mapi id 14.01.0218.012; Thu, 22 Mar 2012 12:03:22 +0200 From: Traiano Welcome To: "freebsd-net@freebsd.org" Thread-Topic: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' Thread-Index: AQHNCBMDrRAwT2ab00myKBl/mmLfFA== Date: Thu, 22 Mar 2012 10:03:21 +0000 Message-ID: Accept-Language: en-US, en-ZA Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 x-originating-ip: [196.30.72.139] Content-Type: text/plain; charset="us-ascii" Content-ID: <81843D020080AF498B1D4D690084456D@int.mtnbusiness.net> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Mar 2012 10:03:31 -0000 Hi List I've been seeing the following in the messages log of my freebsd syslog server for quite some time now: --- Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while writing; fd=3D'12', error=3D'No buffer space available (55)' Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; time_reopen=3D'60' Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while writing; fd=3D'13', error=3D'No buffer space available (55)' Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; time_reopen=3D'60' --- These happen at a frequency of about 7 per minute on average. See attached trend graphs for an idea of the volume of traffic we're doing, as well as the memory and cpu utilisation trends on this server during this period. As can be seen from the graphs, load does not seem to be the issue. Occasionally during the week, the system freezes and requires a reboot, I think it's related to the above message, though I'm not sure. My question is: What does this error mean, and how can I resolve it? I have tried to frame this as an operating system kernel resource issue, and experimented with increasing the freebsd kernel sysctls for UDP performance: --- [root@syslog2 /var/log]# sysctl kern.ipc.nmbclusters=3D102400 kern.ipc.nmbclusters: 25600 -> 102400 [root@syslog2 /var/log]# sysctl kern.ipc.maxsockbuf=3D201326592 kern.ipc.maxsockbuf: 100663296 -> 201326592 [root@syslog2 /var/log]# sysctl net.inet.udp.recvspace=3D33554432 net.inet.udp.recvspace: 16777216 -> 33554432 --- This has reduced the frequency of the errors a little, but in general the problem still remains. Syslog version: -- [root@syslog2]# syslog-ng -V syslog-ng 2.0.10 -- FreeBSD version: -- FreeBSD 7.2-STABLE #0 -- Any help would be much appreciated! Traiano