From owner-freebsd-stable@FreeBSD.ORG Fri Apr 6 13:18:46 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 33F3716A404 for ; Fri, 6 Apr 2007 13:18:46 +0000 (UTC) (envelope-from quetzal@zone3000.net) Received: from mx1.sitevalley.com (sitevalley.com [209.67.60.43]) by mx1.freebsd.org (Postfix) with SMTP id D561F13C48C for ; Fri, 6 Apr 2007 13:18:45 +0000 (UTC) (envelope-from quetzal@zone3000.net) Received: from zone3000.kharkov.ua (HELO localhost) (217.144.69.37) by 209.67.61.254 with SMTP; 6 Apr 2007 13:18:44 -0000 Date: Fri, 6 Apr 2007 16:18:17 +0300 From: Nikolay Pavlov To: Sven Willenberger Message-ID: <20070406131817.GA1977@zone3000.net> Mail-Followup-To: Nikolay Pavlov , Sven Willenberger , Tom Judge , freebsd-stable@freebsd.org References: <1175785136.8280.10.camel@lanshark.dmv.com> <20070405160313.GE1230@dkirhlarov.mow.oilspace.com> <46152615.1040602@tomjudge.com> <1175804135.8280.21.camel@lanshark.dmv.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1175804135.8280.21.camel@lanshark.dmv.com> User-Agent: Mutt/1.4.2.2i X-Operating-System: FreeBSD 6.1-RELEASE-p10 Cc: Tom Judge , freebsd-stable@freebsd.org Subject: Re: ggate + gmirror write performance woes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2007 13:18:46 -0000 On Thursday, 5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote: > On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: > > Dmitriy Kirhlarov wrote: > > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > > >> I am trying to set up a HA type system involving two identical boxes and > > >> have gone through the following to set up the systems: > > >> > > >> Slave server: > > >> ggated -R 196608 -S 196608 > > >> (exporting /dev/amrd1 ) > > >> net.inet.tcp.sendspace: 65536 > > >> net.inet.tcp.recvspace: 131072 > > > > > > Try > > > net.local.stream.recvspace=65535 > > > net.local.stream.sendspace=65535 > > > > > > Also, try increase this sysctls with > > > net.inet.tcp.rfc1323=1 > > > > > > I use it on FreeBSD 5.x with: > > > net.inet.tcp.sendspace=131072 > > > net.inet.tcp.recvspace=131072 > > > net.local.stream.recvspace=65535 > > > net.local.stream.sendspace=65535 > > > > > > ggated -R 1048576 -S 1048576 > > > ggatec -R 1048576 -S 1048576 > > > > > > WBR. > > > Dmitriy > > > _______________________________________________ > > > freebsd-stable@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > > > > I have seen sustained writes of 30Mb/s using the following configuration: > > > > cat /boot/loader.conf > > kern.ipc.nmbclusters="32768" > > > > cat /etc/sysctl.conf > > net.inet.tcp.sendspace=1048576 > > net.inet.tcp.recvspace=1048576 > > > > Server: > > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports > > > > Client: > > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 > > /dev/amrd0s2 > > > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with > > adaptive read ahead and write back caching. > > > > Tom > > I have tried both the settings ideas suggested above but I cannot even > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to > anything higher that 131072 results in ggated bailing with the error: > # ggated -v -a 10.10.0.19 > info: Reading exports file (/etc/gg.exports). > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. > info: Exporting 2 object(s). > error: Cannot open stream socket: No buffer space available. > error: Exiting. For values of net.inet.tcp.{send,recv}space more than 524288 you also need to adjust kern.ipc.maxsockbuf Try this configuration for example: kern.ipc.maxsockbuf=2049152 net.inet.tcp.recvspace=1024576 net.inet.tcp.sendspace=1024576 > > setting net.inet.tcp.{send,recv}space to 131072 allows me to start > ggated with the default R and S values of 131072; anything higher > results in "no buffer space" errors. At 131072 ggated starts but then I > cannot even open a new connection (like ssh) to the server as the ssh > client bails with "no buffer space available". > > more information: > # netstat -m > 514/641/1155 mbufs in use (current/cache/total) > 512/284/796/32768 mbuf clusters in use (current/cache/total/max) > 512/256 mbuf+clusters out of packet secondary zone in use > (current/cache) > 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) > 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) > 1152K/728K/1880K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/0/0 requests for jumbo clusters denied (4k/9k/16k) > 0/4/6656 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA > Raid using LSiMegaRaid. > > The odd thing is that even after I set the send and recvspace down to > values like 65536, I continue to get the no buffer error when trying to > connect to it remotely again. > > Sven > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ======================================================================