From owner-freebsd-performance@FreeBSD.ORG Sun Jul 20 18:05:06 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FF80106567A for ; Sun, 20 Jul 2008 18:05:06 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id B86868FC12 for ; Sun, 20 Jul 2008 18:05:05 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1KKcb4-0004Jx-SW for freebsd-performance@freebsd.org; Sun, 20 Jul 2008 17:21:22 +0000 Received: from ms158-2-gprs01.net.vip.hr ([212.91.111.158]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 20 Jul 2008 17:21:22 +0000 Received: from ivoras by ms158-2-gprs01.net.vip.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sun, 20 Jul 2008 17:21:22 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Sun, 20 Jul 2008 19:21:12 +0200 Lines: 32 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: ms158-2-gprs01.net.vip.hr User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) In-Reply-To: Sender: news Subject: Re: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2008 18:05:06 -0000 Alexander Strange wrote: > We're running a rather high-load webserver using FreeBSD > 7-RELEASE/amd64/nginx on an Intel em gigabit connection. > Performance is good for our current bandwidth use (about 20Mbit and > ~2000 connections/sec at the moment), but a large number of HTTP > requests are being immediately dropped before getting to nginx. I see > complaints about this with earlier versions of FreeBSD - > http://forum.lighttpd.net/topic/171 - but no solutions. Does anyone know > what could be the problem, or anything we could do about it? > > There are several other servers running earlier FreeBSDs on i386 which > don't seem to have this problem, but I still haven't ruled out upstream > hardware problems or Sandvine yet. > > On the server: > -nginx's error log is full of "accept() failed (53: Software caused > connection abort)", sometimes printing three or four at the same time. > > -messages is full of: > Limiting open port RST response from 441 to 200 packets/sec > Limiting open port RST response from 488 to 200 packets/sec > Limiting open port RST response from 399 to 200 packets/sec > Limiting open port RST response from 434 to 200 packets/sec > Limiting open port RST response from 308 to 200 packets/sec > I'm not sure if that's related or not. It's almost certainly related - in addition to other suggested tuning by Istvan, set net.inet.icmp.icmplim sysctl to something high - for example 2000 in your case. Actually, in your sysctl.conf it's set to 400 - you do know you have to run "/etc/rc.d/sysctl restart" to reaload sysctl.conf? From owner-freebsd-performance@FreeBSD.ORG Sun Jul 20 18:25:41 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB13B1065676 for ; Sun, 20 Jul 2008 18:25:41 +0000 (UTC) (envelope-from astrange@ithinksw.com) Received: from fmailhost03.isp.att.net (fmailhost03.isp.att.net [207.115.11.53]) by mx1.freebsd.org (Postfix) with ESMTP id 9E2BD8FC1D for ; Sun, 20 Jul 2008 18:25:41 +0000 (UTC) (envelope-from astrange@ithinksw.com) Received: from [10.0.1.4] (adsl-176-63-128.asm.bellsouth.net[74.176.63.128]) by isp.att.net (frfwmhc03) with SMTP id <20080720182540H0300pplfme>; Sun, 20 Jul 2008 18:25:40 +0000 X-Originating-IP: [74.176.63.128] Message-Id: From: Alexander Strange To: freebsd-performance@freebsd.org In-Reply-To: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v926) Date: Sun, 20 Jul 2008 14:25:40 -0400 References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> X-Mailer: Apple Mail (2.926) Subject: Re: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2008 18:25:41 -0000 On Jul 17, 2008, at 12:44 PM, Sean Chittenden wrote: >> -messages is full of: >> Limiting open port RST response from 441 to 200 packets/sec >> Limiting open port RST response from 488 to 200 packets/sec >> Limiting open port RST response from 399 to 200 packets/sec >> Limiting open port RST response from 434 to 200 packets/sec >> Limiting open port RST response from 308 to 200 packets/sec >> I'm not sure if that's related or not. > > Likely not, but you want to set net.inet.icmp.icmplim=2000 or > something much higher. ICMP is a good thing and an important part > of TCP. For that much traffic, you need more ICMP packets. > net.inet.tcp.recvspace seems high, you probably only want it to be > 4096 or maybe double that.... unless your traffic is all HTTP > posts. Why don't you want to run with accept filters? Any > firewalls or rate filters in the way? -sc The httpready filter was just off for debugging (in case it solved our problem) - it didn't seem to affect it, so it's back on now. There are a lot of large HTTP posts happening, and we don't seem to be low on memory, so recvspace should be ok. somaxconn is also much higher than necessary, though, so maybe that could be a problem. Anyway, raising icmplim has emptied the system log, but there are still several errors per minute. I don't think any of the netstat -s counters are going up at the same rate, but I'll keep looking at those. And there's no firewalls or packet shapers in front of it. From owner-freebsd-performance@FreeBSD.ORG Mon Jul 21 19:53:41 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 508531065687 for ; Mon, 21 Jul 2008 19:53:41 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id F33DA8FC2E for ; Mon, 21 Jul 2008 19:53:40 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1KL1Ru-0004pf-UP for freebsd-performance@freebsd.org; Mon, 21 Jul 2008 19:53:34 +0000 Received: from 77.237.115.237 ([77.237.115.237]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 21 Jul 2008 19:53:34 +0000 Received: from ivoras by 77.237.115.237 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 21 Jul 2008 19:53:34 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Mon, 21 Jul 2008 21:53:29 +0200 Lines: 5 Message-ID: References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: 77.237.115.237 User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) In-Reply-To: Sender: news Subject: Re: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2008 19:53:41 -0000 Alexander Strange wrote: > And there's no firewalls or packet shapers in front of it. How about on it? Do you run ipfw? From owner-freebsd-performance@FreeBSD.ORG Wed Jul 23 19:08:12 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35205106566C for ; Wed, 23 Jul 2008 19:08:12 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from parsely.rain.com (parsely.rain.com [199.26.172.196]) by mx1.freebsd.org (Postfix) with ESMTP id C8AD68FC18 for ; Wed, 23 Jul 2008 19:08:11 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from sopwith.solgatos.com (uucp@localhost) by parsely.rain.com (8.11.4/8.11.4) with UUCP id m6NIjTg74887 for freebsd-performance@freebsd.org; Wed, 23 Jul 2008 11:45:29 -0700 (PDT) (envelope-from freebsd@sopwith.solgatos.com) Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id SAA09833; Wed, 23 Jul 2008 18:43:48 GMT Message-Id: <200807231843.SAA09833@sopwith.solgatos.com> To: freebsd-performance@freebsd.org Date: Wed, 23 Jul 2008 11:43:48 +0100 From: Dieter Subject: disk i/o unfairness with multiple processes X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd@sopwith.solgatos.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2008 19:08:12 -0000 I've been seeing unfairness in disk i/o when multiple processes compete for resources. While some unfairness can be tolerated in order to gain overall efficiency, (e.g. avoiding long seeks) there is a limit. I've seen this with various scenarios, with 6.0, 6.2 and 7.0. Here is a simple test case which demonstrates the problem, and should be easy for others to duplicate. AMD64 2 GiB memory 7200 rpm SATA connected to nforce4-ultra FreeBSD 7.0 FFS, soft-updates $ time man de > /dev/null real 0m0.013s user 0m0.011s sys 0m0.001s $ cat 9_GB_file 9_GB_file 9_GB_file 9_GB_file > /dev/null & [1] 84904 $ time man de > /dev/null [1]+ Done cat 9_GB_file 9_GB_file 9_GB_file 9_GB_file > /dev/null real 9m20.508s user 0m1.053s sys 0m44.091s $ systat -vmstat reports that cat is reading at 50-60 MB/s, which is reasonable for this disk. The 9_GB_file and /usr are both on the same disk. Accessing different disks is more likely to give the expected performance. I suspect that some scenarios bottleneck in memory. I certainly expect man to take longer if it is competing for disk i/o, but 9 minutes seems a bit much. The user and sys times are also up significantly, which seems odd?