From owner-freebsd-performance@FreeBSD.ORG Thu Jul 31 21:30:28 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B25B71065688 for ; Thu, 31 Jul 2008 21:30:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 84A2D8FC2E for ; Thu, 31 Jul 2008 21:30:28 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 6934B46B88; Thu, 31 Jul 2008 17:12:58 -0400 (EDT) Date: Thu, 31 Jul 2008 22:12:58 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alexander Strange In-Reply-To: Message-ID: <20080731220701.Y22038@fledge.watson.org> References: <31AFE70B-CE45-42DE-97C7-AFF96383C6E2@chittenden.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org Subject: Re: Large number of http connections immediately dropped X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2008 21:30:28 -0000 On Wed, 30 Jul 2008, Alexander Strange wrote: > On Jul 21, 2008, at 3:53 PM, Ivan Voras wrote: > >> Alexander Strange wrote: >> >>> And there's no firewalls or packet shapers in front of it. >> >> How about on it? Do you run ipfw? > > No, I wouldn't answer a question so specifically like that. > > We didn't see this problem after recompiling without SMP support and waiting > for a day or two, but that immediately brought the load average up to around > 50 and made it much slower, so that's clearly not a solution. It also really > doesn't make me look forward to debugging it... > > (Disabling net.isr.direct and some other things didn't seem to have any > effect) Turning off SMP is probably slowing the transaction rate down sufficiently that you're not seeing the problem. The reason to ask the firewall question (ipfw, pf, etc) is that as the rate of TCP connections goes up, and if there are a small number of addresses involved, the reuse rate for TCP/IP port/address tuples becomes very high, which can cause connections to reuse tuples too quickly. Sometimes firewalls are more sensitive to this than the stack -- especially if those firewalls are doing things like randomizing port numbers, TCP sequence numbers, etc, so in the past there have been reports (and bug fixes) along those lines. I may have missed you answering this already, but are there a large number of remote endpoints (unique IP addresses) or a small one? Such problems have come up in the past especially when there is a load balancer or proxy in front, as that reduces what starts out as a large number of hosts to a very small number (exactly one). Robert N M Watson Computer Laboratory University of Cambridge