From owner-freebsd-net@FreeBSD.ORG Tue Jun 24 15:08:38 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69D7F106566B for ; Tue, 24 Jun 2008 15:08:38 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from mx45.mail.ru (mx45.mail.ru [194.67.23.236]) by mx1.freebsd.org (Postfix) with ESMTP id E81318FC1D for ; Tue, 24 Jun 2008 15:08:37 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from f7.mail.ru (f7.mail.ru [194.67.57.37]) by mx45.mail.ru (mPOP.Fallback_MX) with ESMTP id 2DD8EE0046D0 for ; Tue, 24 Jun 2008 18:46:39 +0400 (MSD) Received: from mail by f7.mail.ru with local id 1KB9n2-0002X3-00; Tue, 24 Jun 2008 18:46:36 +0400 Received: from [213.180.219.187] by koi.mail.ru with HTTP; Tue, 24 Jun 2008 18:46:36 +0400 From: Dmitriy <_pppp@mail.ru> To: Stefan Lambrev Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [213.180.219.187] Date: Tue, 24 Jun 2008 18:46:36 +0400 In-Reply-To: <4860EF76.1050807@moneybookers.com> References: <4860EF76.1050807@moneybookers.com> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: X-Spam: Not detected Cc: freebsd-net@FreeBSD.org Subject: Re: jboss4 on freebsd X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Dmitriy <_pppp@mail.ru> List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jun 2008 15:08:38 -0000 -----Original Message----- From: Stefan Lambrev To: freebsd-net@FreeBSD.org Date: Tue, 24 Jun 2008 15:58:30 +0300 Subject: jboss4 on freebsd > > Greetings, > > I'm experimenting with jboss4 cluster under freebsd 7 (amd64). > In my configuration I have 2 jboss instances which are in cluster and > they communicate via separate network (used only for shared data) > When I create some load on the application sometimes I see this error: > > 2008-06-24 14:46:21,602 ERROR [org.jgroups.protocols.UDP] failed sending > message to 10.50.1.1:57680 (59800 bytes) > java.io.IOException: No buffer space available > > It looks very much, that jboss can't handle properly such error as on > linux there is no such thing as no network buffers ;) - > http://wiki.freebsd.org/AvoidingLinuxisms > > But what really bothers me is that I see "No buffer space available" on > very low network IO - > > input (em2) output > packets errs bytes packets errs bytes colls > 144 0 2203390 292 0 2072771 0 > 1568 0 2329764 63 0 9099 0 > 76 0 231562 34 0 148306 0 > 563 0 1152531 1009 0 1768748 0 > 1625 0 2601502 104 0 229728 0 > 65 0 467296 85 0 441566 0 > 464 0 680082 973 0 1439442 0 > 357 0 1940361 55 0 222484 0 > 1651 0 2827932 145 0 450265 0 > > E.g. traffic between 1-3MB/s. > > I'm using: > em2: flags=8843 metric 0 mtu 9000 > options=19b > ether 00:15:17:60:04:c8 > inet 10.3.3.117 netmask 0xffffff00 broadcast 10.3.3.255 > media: Ethernet autoselect (1000baseTX ) > status: active > > em2: port 0x2020-0x203f mem > 0xb8820000-0xb883ffff,0xb8400000-0xb87fffff irq 18 at device 0.0 on pci5 > em2: Using MSI interrupt > em2: [FILTER] > > and my sysctl.conf is: > kern.maxfiles=65000 > kern.ipc.shmmax=67108864 > kern.fallback_elf_brand=3 > kern.threads.max_threads_per_proc=6000 > kern.ipc.somaxconn=512 > #jboss extra > net.inet.udp.maxdgram=73728 > kern.ipc.maxsockbuf=1048576 > net.inet.udp.recvspace=147456 > kern.ipc.maxsockets=49312 > > Any ideas how I can improve things? > I guess you get the error from the NIC driver. There 2 conditions which can trigger it: 1. if( adapter->num_tx_desc_avail < EM_TX_OP_THRESHOLD ) 2. if( nsegs > ( adapter->num_tx_desc_avail - 2 ) ) I have 2 suggestions how to fix your error: 1. try to disable TXCSUM ( #ifconfig em2 -txcsum ) 2. increase the number of transfer descriptors available if your hardware supports it: [man em(4) excerpt] Tunables can be set at the loader(8) prompt before booting the kernel or stored in loader.conf(5). hw.em.txd Number of transmit descriptors allocated by the driver. The default value is 256. The 82542 and 82543-based adapters can handle up to 256 descriptors, while others can have up to 4096. hw.em.tx_int_delay This value delays the generation of transmit interrupts in units of 1.024 microseconds. The default value is 64. hw.em.tx_abs_int_delay If hw.em.tx_int_delay is non-zero, this tunable limits the maxi- mum delay in which a transmit interrupt is generated. [/man] Regards, Dmitriy.