From owner-freebsd-geom@FreeBSD.ORG Wed Oct 17 19:50:49 2007 Return-Path: Delivered-To: freebsd-geom@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20FF016A49A for ; Wed, 17 Oct 2007 19:50:49 +0000 (UTC) (envelope-from kurtseel@primetime.com) Received: from mail.primetime.com (mail.primetime.com [146.145.135.164]) by mx1.freebsd.org (Postfix) with ESMTP id E390E13C46E for ; Wed, 17 Oct 2007 19:50:48 +0000 (UTC) (envelope-from kurtseel@primetime.com) Received: from [10.200.1.130] (unknown [10.200.1.130]) by mail.primetime.com (Postfix) with ESMTP id 8F98AF9C423; Wed, 17 Oct 2007 14:49:05 -0400 (EDT) Message-ID: <47166562.60803@primetime.com> Date: Wed, 17 Oct 2007 15:41:22 -0400 From: kurtseel User-Agent: Thunderbird 2.0.0.5 (X11/20070724) MIME-Version: 1.0 To: Patrick Tracanelli References: <471650AA.30903@primetime.com> <47165C0B.7080707@freebsdbrasil.com.br> In-Reply-To: <47165C0B.7080707@freebsdbrasil.com.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-geom@freebsd.org Subject: Re: gmirror + ggated question X-BeenThere: freebsd-geom@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: GEOM-specific discussions and implementations List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 19:50:49 -0000 Patrick Tracanelli wrote: > kurtseel escreveu: >> >> I built a mirror of a local drive and a ggated backed device. I ran >> iozone on it >> and it runs along fine until a certain point when it slows down to a >> near stand >> still. It doesn't break the mirror or crash the system, but it does >> slow the system >> down to a near stop. >> I kill the iozone, and a short time later I can login and then : >> >> # df >> Filesystem 1K-blocks Used Avail Capacity Mounted on >> /dev/mirror/thinkcs1a 1012974 155780 776158 17% / >> devfs 1 1 0 100% /dev >> /dev/mirror/thinkcs1e 85469448 1163474 77468420 1% /usr >> /dev/mirror/thinkcs1d 4058062 40426 3692992 1% /var >> [root@ ~/temp]# gmirror status >> Name Status Components >> mirror/thinkc COMPLETE ad0 >> ggate0 >> >> And all seems normal again. Seems like it has to do with big files ... >> This is the same configuration I used in : >> http://bsdtips.utcorp.net/mediawiki/index.php/Mirroring_over_network >> This is where the iozone gets stuck : > > Did you try raising send and receive buffers on ggated? I found myself > confortable with -S and -R around 512k-780k. I didnt, however, did an > iozone stress test, just a production test (real load) before going > production. > > Try raising the buffer and let us know about your tests. TCP_NODELAY > is also worth trying. > Makes sense. So now I get this : Test (/root/benchmarks) > ggated -v -R 262144 -S 262144 /etc/ggated.conf info: Reading exports file (/etc/ggated.conf). debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list. info: Exporting 1 object(s). error: Cannot open stream socket: No buffer space available. error: Exiting. Test (/root/benchmarks) > ggated -v -R 524288 -S 524288 /etc/ggated.conf info: Reading exports file (/etc/ggated.conf). debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list. info: Exporting 1 object(s). error: Cannot open stream socket: No buffer space available. error: Exiting. I have raised sysctl net.inet.tcp.sendspace=4194304 sysctl net.inet.tcp.recvspace=4194304 sysctl kern.ipc.maxsockbuf=2097152 Which I saw in a posting ... It even happens here : Test (/root/benchmarks) > ggated -v -R 1 -S 1 /etc/ggated.conf info: Reading exports file (/etc/ggated.conf). debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list. info: Exporting 1 object(s). error: Cannot open stream socket: No buffer space available. error: Exiting.