Date: Fri, 23 Mar 2012 08:58:29 +0000 From: Traiano Welcome <Traiano.Welcome@mtnbusiness.co.za> To: Mark Blackman <mark@exonetric.com> Cc: freebsd questions <freebsd-questions@freebsd.org> Subject: Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' Message-ID: <CB920690.DFDE%traiano.welcome@mtnbusiness.co.za> In-Reply-To: <F4E891E9-138A-447D-8FC6-83CD2C91CCAF@exonetric.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Mark On 22/03/2012 13:54, "Mark Blackman" <mark@exonetric.com> wrote: > >On 22 Mar 2012, at 11:40, Traiano Welcome wrote: > >> That's what I thought as well, but it's the details that evade me. >>Almost >> all traffic to and from this server is UDP (syslog), the graph I sent >> earlier shows the kind of volumes and trends that are typical: Peak >> traffic during the problem periods averages at about 1 Mbps outbound and >> 200 Kbps inbound to/from the interface. The interface itself is a >> Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950. >>=20 >>=20 >> Here are a couple of netstat polls during one of the problem periods: >>=20 >> ---- >> [root@syslog2]# date;netstat -p udp -s |egrep -w >> "(received|delivered|dropped)" >> Thu Mar 22 12:11:34 SAST 2012 >> 19969 datagrams received >> 2 dropped due to no socket >> 0 dropped due to full socket buffers >> 19967 delivered >> . >> . >> . >> [root@syslog2~]# date;netstat -p udp -s |egrep -w >> "(received|delivered|dropped)" >> Thu Mar 22 13:36:46 SAST 2012 >> 662385 datagrams received >> 118 dropped due to no socket >> 0 dropped due to full socket buffers >> 662267 delivered >> --- >>=20 >>=20 >> Somehow this doesn't strike me as a large volume of throughput =8A > >Ok, fair enough. You might try simulating the problem by deliberately >overloading the syslog UDP output and confirm the cause. Apparently this means that the network driver has "filled" up with packets. John Baldwin over at freebsd-net@ advises I up the number of descriptors assigned to igb to the maximum to workaround this using the hw.igb.maxtxd tunable you would set. So I've rebooted with the following in loader.conf: hw.igb.rxd=3D4096 hw.igb.txd=3D4096 This seems to be working so far. What I've noticed is that the system is using far less RAM than previously, and CPU utilisation is up to 100% of one core, load average is 1, which I would guess means that the system is now processing a lot more syslog data now that "more packets are making it through the network driver". I'll keep monitoring over a 24 hour period though, to see how effective this is. > >- Mark
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CB920690.DFDE%traiano.welcome>