From owner-freebsd-questions@FreeBSD.ORG Fri Mar 23 08:58:40 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ABB86106564A for ; Fri, 23 Mar 2012 08:58:40 +0000 (UTC) (envelope-from Traiano.Welcome@mtnbusiness.co.za) Received: from smtprelay01.ops.mtnbusiness.co.za (smtprelay01.ops.mtnbusiness.co.za [41.181.93.235]) by mx1.freebsd.org (Postfix) with ESMTP id 22F0C8FC19 for ; Fri, 23 Mar 2012 08:58:39 +0000 (UTC) Received: from [196.30.97.135] (helo=CPT-EXCH01.int.mtnbusiness.net) by smtprelay01.ops.mtnbusiness.co.za with esmtp (ULTRA Special SMTP Internal Alpha) (envelope-from ) id 1SB0KS-0009u0-N1; Fri, 23 Mar 2012 10:58:36 +0200 Received: from CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) by CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) with mapi id 14.01.0218.012; Fri, 23 Mar 2012 10:58:29 +0200 From: Traiano Welcome To: Mark Blackman Thread-Topic: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' Thread-Index: AQHNCAowmWzmAjIpD0afh6AVbHiFtJZ18VCAgAA/5oD//+IeAIABgswA Date: Fri, 23 Mar 2012 08:58:29 +0000 Message-ID: In-Reply-To: Accept-Language: en-US, en-ZA Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 x-originating-ip: [196.30.72.139] Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: freebsd questions Subject: Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Mar 2012 08:58:40 -0000 Hi Mark On 22/03/2012 13:54, "Mark Blackman" wrote: > >On 22 Mar 2012, at 11:40, Traiano Welcome wrote: > >> That's what I thought as well, but it's the details that evade me. >>Almost >> all traffic to and from this server is UDP (syslog), the graph I sent >> earlier shows the kind of volumes and trends that are typical: Peak >> traffic during the problem periods averages at about 1 Mbps outbound and >> 200 Kbps inbound to/from the interface. The interface itself is a >> Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950. >>=20 >>=20 >> Here are a couple of netstat polls during one of the problem periods: >>=20 >> ---- >> [root@syslog2]# date;netstat -p udp -s |egrep -w >> "(received|delivered|dropped)" >> Thu Mar 22 12:11:34 SAST 2012 >> 19969 datagrams received >> 2 dropped due to no socket >> 0 dropped due to full socket buffers >> 19967 delivered >> . >> . >> . >> [root@syslog2~]# date;netstat -p udp -s |egrep -w >> "(received|delivered|dropped)" >> Thu Mar 22 13:36:46 SAST 2012 >> 662385 datagrams received >> 118 dropped due to no socket >> 0 dropped due to full socket buffers >> 662267 delivered >> --- >>=20 >>=20 >> Somehow this doesn't strike me as a large volume of throughput =8A > >Ok, fair enough. You might try simulating the problem by deliberately >overloading the syslog UDP output and confirm the cause. Apparently this means that the network driver has "filled" up with packets. John Baldwin over at freebsd-net@ advises I up the number of descriptors assigned to igb to the maximum to workaround this using the hw.igb.maxtxd tunable you would set. So I've rebooted with the following in loader.conf: hw.igb.rxd=3D4096 hw.igb.txd=3D4096 This seems to be working so far. What I've noticed is that the system is using far less RAM than previously, and CPU utilisation is up to 100% of one core, load average is 1, which I would guess means that the system is now processing a lot more syslog data now that "more packets are making it through the network driver". I'll keep monitoring over a 24 hour period though, to see how effective this is. > >- Mark