From owner-freebsd-net@FreeBSD.ORG Fri Mar 23 12:03:31 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E4DFE106566C for ; Fri, 23 Mar 2012 12:03:31 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id B778B8FC0C for ; Fri, 23 Mar 2012 12:03:31 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id 57FEA46B2A; Fri, 23 Mar 2012 08:03:31 -0400 (EDT) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id E257DB911; Fri, 23 Mar 2012 08:03:30 -0400 (EDT) From: John Baldwin To: Traiano Welcome Date: Thu, 22 Mar 2012 13:49:26 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p10; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201203221349.26701.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 23 Mar 2012 08:03:31 -0400 (EDT) Cc: "freebsd-net@freebsd.org" Subject: Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Mar 2012 12:03:32 -0000 On Thursday, March 22, 2012 11:00:35 am Traiano Welcome wrote: > Hi John > > > > On 22/03/2012 14:16, "John Baldwin" wrote: > > >On Thursday, March 22, 2012 6:03:21 am Traiano Welcome wrote: > >> Hi List > >> > >> I've been seeing the following in the messages log of my freebsd syslog > >> server for quite some time now: > >> > >> --- > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while > >> writing; fd='12', error='No buffer space available (55)' > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; > >> time_reopen='60' > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while > >> writing; fd='13', error='No buffer space available (55)' > >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; > >> time_reopen='60' > >> --- > >> > >> These happen at a frequency of about 7 per minute on average. See > >>attached > >> trend graphs for an idea of the volume of traffic we're doing, as well > >>as > >> the memory and cpu utilisation trends on this server during this period. > >> As can be seen from the graphs, load does not seem to be the issue. > >> Occasionally during the week, the system freezes and requires a reboot, > >>I > >> think it's related to the above message, though I'm not sure. > >> > >> My question is: What does this error mean, and how can I resolve it? > >> > >> I have tried to frame this as an operating system kernel resource issue, > >> and experimented with increasing the freebsd kernel sysctls for UDP > >> performance: > > > >It means that the network driver has "filled" up with packets. Are you > >using > >igb(4)? We have to crank the number of descriptors assigned to igb to > >the max > >to workaround this at work (we get DNS timeouts during a simple boot > >otherwise). hw.igb.maxtxd is the tunable you would set. The max value > >you > >can set it to is 4096. > > > Thanks for the clue. It seems igb is indeed being used. However, I not > that when I try to list this tunable, I don't get anything: > > --- > [root@syslog2]# sysctl -a | grep igb > [root@syslog2]# > --- > > > --- > [root@syslog2 /boot]# sysctl hw.igb.maxtxd=4000 > sysctl: unknown oid 'hw.igb.maxtxd' > --- > > > Is this due to the way igb is compiled into the kernel, or am I tuning in > the wrong place :-) You need to put it in /boot/loader.conf and then reboot. Do try using 4096 rather than 4000. -- John Baldwin