From owner-freebsd-stable@FreeBSD.ORG Fri Oct 13 04:03:19 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 22F1316A403 for ; Fri, 13 Oct 2006 04:03:19 +0000 (UTC) (envelope-from sullrich@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.174]) by mx1.FreeBSD.org (Postfix) with ESMTP id 49F2D43D46 for ; Fri, 13 Oct 2006 04:03:18 +0000 (GMT) (envelope-from sullrich@gmail.com) Received: by ug-out-1314.google.com with SMTP id m2so366037uge for ; Thu, 12 Oct 2006 21:03:17 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=J21NRDqDLag9FhpSKXPdP70u0b6QM/x6Z2v6dNJnpAIqL/U5aKCRhzWX+TTw77RjwlF5ASodZUu5cBbBrwHPslxmTodMVpcx4Z20m86g9rBVat8o+nU2nS8q8uRyx6jY6DUFC+E4lkHwtiFy3O9XIt3vSK//YJTwUGh2GY9u28M= Received: by 10.66.224.3 with SMTP id w3mr3751103ugg; Thu, 12 Oct 2006 21:03:17 -0700 (PDT) Received: by 10.67.28.14 with HTTP; Thu, 12 Oct 2006 21:03:17 -0700 (PDT) Message-ID: Date: Fri, 13 Oct 2006 00:03:17 -0400 From: "Scott Ullrich" To: "Chuck Swiger" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: Cc: freebsd-stable Subject: Re: Userland freezes during heavy packet forwarding workloads X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Oct 2006 04:03:19 -0000 On 10/12/06, Chuck Swiger wrote: > On Oct 12, 2006, at 2:11 PM, Scott Ullrich wrote: > > We (pfSense developers) have noticed an interesting problem where > > userland stops functioning under high packet forwarding workloads. > > Userland applications such as sshd and lighttpd freeze but userland > > resumes after the network load eases. > > [ ...note that the links you posted included the trailing square > bracket, > and thus were broken, although easily fixable by hand... ] > > > Is there anything we can do to prevent userland from stopping > > completely when under heavy load? Is this a bug? > > You're probably experiencing some form of "livelock"; your hardware > isn't horribly fast, and if the NICs are generating interrupts > rapidly enough due to the high rate of packet forwarding, you're not > going to have a lot of spare CPU available to run userland tasks. > > I notice you're compiling in support for DEVICE_POLLING; does > enabling it do anything to help the responsiveness of the userland > tasks under high network load? You might try adjusting HZ from 100 > to 250 or so...I found that made a decent tradeoff between packet > delay and scheduler overhead on similar Soerkis 4801 or VIA C3/EPIA-M > hardware to yours. When polling is enabled, userland does indeed stay responsive, but throughput also drops by 25-35% in testing on a Soekris 4801, 266 MHz. Again, this is not the case with 4.x. These slower boxes actually get as much as 20% faster with polling in 4.x, not slower as it does in 6.x. And with or without polling, userland always stays responsive (albeit much slower, of course) on 4.x when under much higher pps load. At this time, we're not sure if this same problem exists in 5.x, but if anyone thinks that information might be useful, we can test that as well. Thanks