From owner-freebsd-performance@FreeBSD.ORG Wed Jul 30 05:38:55 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB20E106566C for ; Wed, 30 Jul 2008 05:38:55 +0000 (UTC) (envelope-from astrange@ithinksw.com) Received: from fmailhost02.isp.att.net (fmailhost02.isp.att.net [207.115.11.52]) by mx1.freebsd.org (Postfix) with ESMTP id AB0A98FC12 for ; Wed, 30 Jul 2008 05:38:55 +0000 (UTC) (envelope-from astrange@ithinksw.com) Received: from [10.0.1.3] (adsl-232-6-168.asm.bellsouth.net[74.232.6.168]) by isp.att.net (frfwmhc02) with SMTP id <20080730053851H02000t566e>; Wed, 30 Jul 2008 05:38:54 +0000 X-Originating-IP: [74.232.6.168] Message-Id: From: Alexander Strange To: freebsd-performance@freebsd.org In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v926) Date: Wed, 30 Jul 2008 01:38:50 -0400 References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> X-Mailer: Apple Mail (2.926) Subject: Re: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 Jul 2008 05:38:55 -0000 On Jul 21, 2008, at 3:53 PM, Ivan Voras wrote: > Alexander Strange wrote: > >> And there's no firewalls or packet shapers in front of it. > > How about on it? Do you run ipfw? No, I wouldn't answer a question so specifically like that. We didn't see this problem after recompiling without SMP support and waiting for a day or two, but that immediately brought the load average up to around 50 and made it much slower, so that's clearly not a solution. It also really doesn't make me look forward to debugging it... (Disabling net.isr.direct and some other things didn't seem to have any effect) From owner-freebsd-performance@FreeBSD.ORG Thu Jul 31 21:30:28 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B25B71065688 for ; Thu, 31 Jul 2008 21:30:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 84A2D8FC2E for ; Thu, 31 Jul 2008 21:30:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 6934B46B88; Thu, 31 Jul 2008 17:12:58 -0400 (EDT) Date: Thu, 31 Jul 2008 22:12:58 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alexander Strange In-Reply-To: Message-ID: <20080731220701.Y22038@fledge.watson.org> References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org Subject: Re: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2008 21:30:28 -0000 On Wed, 30 Jul 2008, Alexander Strange wrote: > On Jul 21, 2008, at 3:53 PM, Ivan Voras wrote: > >> Alexander Strange wrote: >> >>> And there's no firewalls or packet shapers in front of it. >> >> How about on it? Do you run ipfw? > > No, I wouldn't answer a question so specifically like that. > > We didn't see this problem after recompiling without SMP support and waiting > for a day or two, but that immediately brought the load average up to around > 50 and made it much slower, so that's clearly not a solution. It also really > doesn't make me look forward to debugging it... > > (Disabling net.isr.direct and some other things didn't seem to have any > effect) Turning off SMP is probably slowing the transaction rate down sufficiently that you're not seeing the problem. The reason to ask the firewall question (ipfw, pf, etc) is that as the rate of TCP connections goes up, and if there are a small number of addresses involved, the reuse rate for TCP/IP port/address tuples becomes very high, which can cause connections to reuse tuples too quickly. Sometimes firewalls are more sensitive to this than the stack -- especially if those firewalls are doing things like randomizing port numbers, TCP sequence numbers, etc, so in the past there have been reports (and bug fixes) along those lines. I may have missed you answering this already, but are there a large number of remote endpoints (unique IP addresses) or a small one? Such problems have come up in the past especially when there is a load balancer or proxy in front, as that reduces what starts out as a large number of hosts to a very small number (exactly one). Robert N M Watson Computer Laboratory University of Cambridge