From owner-freebsd-net@FreeBSD.ORG Fri Mar 23 12:51:23 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 34F21106564A; Fri, 23 Mar 2012 12:51:23 +0000 (UTC) (envelope-from Traiano.Welcome@mtnbusiness.co.za) Received: from smtprelay01.ops.mtnbusiness.co.za (smtprelay01.ops.mtnbusiness.co.za [41.181.93.235]) by mx1.freebsd.org (Postfix) with ESMTP id 6FC9F8FC0C; Fri, 23 Mar 2012 12:51:21 +0000 (UTC) Received: from [196.30.97.135] (helo=CPT-EXCH01.int.mtnbusiness.net) by smtprelay01.ops.mtnbusiness.co.za with esmtp (ULTRA Special SMTP Internal Alpha) (envelope-from ) id 1SB3xZ-000FZM-Iq; Fri, 23 Mar 2012 14:51:13 +0200 Received: from CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) by CPT-EXCH01.int.mtnbusiness.net ([196.30.97.135]) with mapi id 14.01.0218.012; Fri, 23 Mar 2012 14:51:06 +0200 From: Traiano Welcome To: John Baldwin Thread-Topic: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' Thread-Index: AQHNCBMDrRAwT2ab00myKBl/mmLfFJZ2GayAgABPOoCAAA2rAIABYICA Date: Fri, 23 Mar 2012 12:51:05 +0000 Message-ID: In-Reply-To: <201203221349.26701.jhb@freebsd.org> Accept-Language: en-US, en-ZA Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.14.0.111121 x-originating-ip: [196.30.72.139] Content-Type: text/plain; charset="us-ascii" Content-ID: <5001690633350B4A92FAF06177A72F8A@int.mtnbusiness.net> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-net@freebsd.org" Subject: Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Mar 2012 12:51:23 -0000 Hi John On 22/03/2012 19:49, "John Baldwin" wrote: >On Thursday, March 22, 2012 11:00:35 am Traiano Welcome wrote: >> Hi John >>=20 >>=20 >>=20 >> On 22/03/2012 14:16, "John Baldwin" wrote: >>=20 >> >On Thursday, March 22, 2012 6:03:21 am Traiano Welcome wrote: >> >> Hi List >> >>=20 >> >> I've been seeing the following in the messages log of my freebsd >>syslog >> >> server for quite some time now: >> >>=20 >> >> --- >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> >> writing; fd=3D'12', error=3D'No buffer space available (55)' >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> >> time_reopen=3D'60' >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while >> >> writing; fd=3D'13', error=3D'No buffer space available (55)' >> >> Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; >> >> time_reopen=3D'60' >> >> --- >> >>=20 >> >> These happen at a frequency of about 7 per minute on average. See >> >>attached >> >> trend graphs for an idea of the volume of traffic we're doing, as >>well >> >>as >> >> the memory and cpu utilisation trends on this server during this >>period. >> >> As can be seen from the graphs, load does not seem to be the issue. >> >> Occasionally during the week, the system freezes and requires a >>reboot, >> >>I >> >> think it's related to the above message, though I'm not sure. >> >>=20 >> >> My question is: What does this error mean, and how can I resolve it? >> >>=20 >> >> I have tried to frame this as an operating system kernel resource >>issue, >> >> and experimented with increasing the freebsd kernel sysctls for UDP >> >> performance: >> > >> >It means that the network driver has "filled" up with packets. Are you >> >using=20 >> >igb(4)? We have to crank the number of descriptors assigned to igb to >> >the max=20 >> >to workaround this at work (we get DNS timeouts during a simple boot >> >otherwise). hw.igb.maxtxd is the tunable you would set. The max value >> >you=20 >> >can set it to is 4096. >>=20 >>=20 >> Thanks for the clue. It seems igb is indeed being used. However, I not >> that when I try to list this tunable, I don't get anything: >>=20 >> --- >> [root@syslog2]# sysctl -a | grep igb >> [root@syslog2]# >> --- >>=20 >>=20 >> --- >> [root@syslog2 /boot]# sysctl hw.igb.maxtxd=3D4000 >> sysctl: unknown oid 'hw.igb.maxtxd' > >> --- >>=20 >>=20 >> Is this due to the way igb is compiled into the kernel, or am I tuning >>in >> the wrong place :-) > >You need to put it in /boot/loader.conf and then reboot. Do try using >4096 >rather than 4000. I'm an idiot :-) Looks like the correct driver is bce: --- [root@syslog2]# sysctl dev.bce | grep -v stat dev.bce.0.%desc: Broadcom NetXtreme II BCM5708 1000Base-T (B2) dev.bce.0.%driver: bce dev.bce.0.%location: slot=3D0 function=3D0 dev.bce.0.%pnpinfo: vendor=3D0x14e4 device=3D0x164c subvendor=3D0x1028 subdevice=3D0x01b3 class=3D0x020000 dev.bce.0.%parent: pci7 dev.bce.0.l2fhdr_error_count: 0 dev.bce.0.mbuf_alloc_failed_count: 0 dev.bce.0.fragmented_mbuf_count: 0 dev.bce.0.dma_map_addr_rx_failed_count: 0 dev.bce.0.dma_map_addr_tx_failed_count: 0 dev.bce.0.unexpected_attention_count: 0 dev.bce.0.com_no_buffers: 0 dev.bce.1.%desc: Broadcom NetXtreme II BCM5708 1000Base-T (B2) dev.bce.1.%driver: bce dev.bce.1.%location: slot=3D0 function=3D0 dev.bce.1.%pnpinfo: vendor=3D0x14e4 device=3D0x164c subvendor=3D0x1028 subdevice=3D0x01b3 class=3D0x020000 dev.bce.1.%parent: pci3 dev.bce.1.l2fhdr_error_count: 0 dev.bce.1.mbuf_alloc_failed_count: 0 dev.bce.1.fragmented_mbuf_count: 0 dev.bce.1.dma_map_addr_rx_failed_count: 0 dev.bce.1.dma_map_addr_tx_failed_count: 0 dev.bce.1.unexpected_attention_count: 0 dev.bce.1.com_no_buffers: --- And it looks like the buffer full errors have returned, and the perceived improvement may just have been due to traffic backoff due to the reboot But I can't seem to find any tunables for the bce driver. The man page only indicates one: --- SYSCTL VARIABLES The following variables are available as both sysctl(8) variables and loader(8) tunables: hw.bce.msi_enable Whether or not MSI support is enabled in the driver. The default value is 1. --- The only other solution I can think of is an upgrade. Is there an alternative to this ? > >--=20 >John Baldwin