Date: Fri, 23 Mar 2012 08:59:58 +0000 From: Traiano Welcome <Traiano.Welcome@mtnbusiness.co.za> To: John Baldwin <jhb@freebsd.org>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' Message-ID: <CB9207D6.DFEA%traiano.welcome@mtnbusiness.co.za> In-Reply-To: <201203220816.58217.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--_002_CB9207D6DFEAtraianowelcomemtnbusinesscoza_ Content-Type: text/plain; charset="us-ascii" Content-ID: <5B62C4836E8E914BAFCDE4E83D3CC0CD@int.mtnbusiness.net> Content-Transfer-Encoding: quoted-printable Hi John On 22/03/2012 14:16, "John Baldwin" <jhb@freebsd.org> wrote: >On Thursday, March 22, 2012 6:03:21 am Traiano Welcome wrote: >> Hi List >>=20 >> I've been seeing the following in the messages log of my freebsd syslog >> server for quite some time now: >>=20 >> --- >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> writing; fd=3D'12', error=3D'No buffer space available (55)' >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> time_reopen=3D'60' >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> writing; fd=3D'13', error=3D'No buffer space available (55)' >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> time_reopen=3D'60' >> --- >>=20 >> These happen at a frequency of about 7 per minute on average. See >>attached >> trend graphs for an idea of the volume of traffic we're doing, as well >>as >> the memory and cpu utilisation trends on this server during this period. >> As can be seen from the graphs, load does not seem to be the issue. >> Occasionally during the week, the system freezes and requires a reboot, >>I >> think it's related to the above message, though I'm not sure. >>=20 >> My question is: What does this error mean, and how can I resolve it? >>=20 >> I have tried to frame this as an operating system kernel resource issue, >> and experimented with increasing the freebsd kernel sysctls for UDP >> performance: > >It means that the network driver has "filled" up with packets. Are you >using=20 >igb(4)? We have to crank the number of descriptors assigned to igb to >the max=20 >to workaround this at work (we get DNS timeouts during a simple boot >otherwise). hw.igb.maxtxd is the tunable you would set. The max value >you=20 >can set it to is 4096. This seems to be working so far. What I've noticed is that the system is using far less RAM than previously, and CPU utilisation is up to 100% of one core (see attached graph), load average is now 1, which I would guess means that the system is now processing a lot more syslog data now that "more packets are making it through the network driver". I'll keep monitoring over a 24 hour period though, to see how effective this is. > >--=20 >John Baldwin Many Thanks! Traiano > --_002_CB9207D6DFEAtraianowelcomemtnbusinesscoza_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CB9207D6.DFEA%traiano.welcome>