Date: Thu, 18 Oct 2007 11:08:34 -0200 From: Patrick Tracanelli <eksffa@freebsdbrasil.com.br> To: kurtseel <kurtseel@primetime.com> Cc: freebsd-geom@freebsd.org Subject: Re: gmirror + ggated question Message-ID: <47175AD2.5080308@freebsdbrasil.com.br> In-Reply-To: <47166562.60803@primetime.com> References: <471650AA.30903@primetime.com> <47165C0B.7080707@freebsdbrasil.com.br> <47166562.60803@primetime.com>
next in thread | previous in thread | raw e-mail | index | archive | help
kurtseel escreveu: > Patrick Tracanelli wrote: >> kurtseel escreveu: >>> >>> I built a mirror of a local drive and a ggated backed device. I ran >>> iozone on it >>> and it runs along fine until a certain point when it slows down to a >>> near stand >>> still. It doesn't break the mirror or crash the system, but it does >>> slow the system >>> down to a near stop. >>> I kill the iozone, and a short time later I can login and then : >>> >>> # df >>> Filesystem 1K-blocks Used Avail Capacity Mounted on >>> /dev/mirror/thinkcs1a 1012974 155780 776158 17% / >>> devfs 1 1 0 100% /dev >>> /dev/mirror/thinkcs1e 85469448 1163474 77468420 1% /usr >>> /dev/mirror/thinkcs1d 4058062 40426 3692992 1% /var >>> [root@ ~/temp]# gmirror status >>> Name Status Components >>> mirror/thinkc COMPLETE ad0 >>> ggate0 >>> >>> And all seems normal again. Seems like it has to do with big files ... >>> This is the same configuration I used in : >>> http://bsdtips.utcorp.net/mediawiki/index.php/Mirroring_over_network >>> This is where the iozone gets stuck : >> >> Did you try raising send and receive buffers on ggated? I found myself >> confortable with -S and -R around 512k-780k. I didnt, however, did an >> iozone stress test, just a production test (real load) before going >> production. >> >> Try raising the buffer and let us know about your tests. TCP_NODELAY >> is also worth trying. >> > Makes sense. So now I get this : > > Test (/root/benchmarks) > ggated -v -R 262144 -S 262144 /etc/ggated.conf > info: Reading exports file (/etc/ggated.conf). > debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list. > info: Exporting 1 object(s). > error: Cannot open stream socket: No buffer space available. > error: Exiting. > > Test (/root/benchmarks) > ggated -v -R 524288 -S 524288 /etc/ggated.conf > info: Reading exports file (/etc/ggated.conf). > debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list. > info: Exporting 1 object(s). > error: Cannot open stream socket: No buffer space available. > error: Exiting. > > I have raised > > sysctl net.inet.tcp.sendspace=4194304 > sysctl net.inet.tcp.recvspace=4194304 > sysctl kern.ipc.maxsockbuf=2097152 > > Which I saw in a posting ... > > It even happens here : > > Test (/root/benchmarks) > ggated -v -R 1 -S 1 /etc/ggated.conf > info: Reading exports file (/etc/ggated.conf). > debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list. > info: Exporting 1 object(s). > error: Cannot open stream socket: No buffer space available. > error: Exiting. > Seems that you are out of buffer spance and it is not related to ggated, since -R 1 and -S 1 would not demand a bounch of extra memory. In any case, tuning kern.ipc.maxsockbuf should be enough. If I raise to 512K I get out of buffer space too, on the default value. However, just raising it solves the prob: (eksffa@claire)~# sysctl -qw kern.ipc.maxsockbuf=`echo "524288*2" | bc ` kern.ipc.maxsockbuf: 262144 -> 1048576 (eksffa@claire)~# ggated -R 524288 -S 524288 -v info: Reading exports file (/etc/gg.exports). debug: Added 10.0.0.0/24 /dev/ad12 RO to exports list. info: Exporting 1 object(s). info: Listen on port: 3080. And so, I can import ggate0 on the other host. Try figuring out with netstat -m why you ran out of buffer. Also, I believe it can be related to the fact you have raised recvspace and sndspace way too high. I dont think it makes any sense raising it over 64k on 100Mbit network, or 128-512k on 1Gbit network. You have raised 'em up to 4MB :) Lower down to the default 32k (send) / 64k (recv) first. If you are on 1Gbit or 10Gbit you can, later, tray raising on multiple of 32K untill the point you see it makes sense (where it makes positive difference on your benchs). If it is anyhow relevant, I run it on 6.2-STABLE, cvsuped on Sept 24th, with the patches PJD mentioned applied. -- Patrick Tracanelli FreeBSD Brasil LTDA. (31) 3281-9633 / 3281-3547 316601@sip.freebsdbrasil.com.br http://www.freebsdbrasil.com.br "Long live Hanin Elias, Kim Deal!"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47175AD2.5080308>