From owner-freebsd-net@FreeBSD.ORG Tue Jan 4 15:22:10 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 94A0916A4CE for ; Tue, 4 Jan 2005 15:22:10 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 16BDE43D48 for ; Tue, 4 Jan 2005 15:22:10 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.13.1/8.13.1) with ESMTP id j04FIUpj041122; Tue, 4 Jan 2005 10:18:30 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)j04FIUUL041119; Tue, 4 Jan 2005 15:18:30 GMT (envelope-from robert@fledge.watson.org) Date: Tue, 4 Jan 2005 15:18:30 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Youlin Feng In-Reply-To: <12394E9FCB7C8441BB238D7F67B402E407E5E1@exch2.verniernetworks.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-net@freebsd.org Subject: Re: User process starvation under heavy network traffic in FreeBSD 5.3 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 15:22:10 -0000 On Mon, 3 Jan 2005, Youlin Feng wrote: > We are building a network appliance running FreeBSD 5.3 and under very > heavy network traffic the user processes don't get scheduled for an > unacceptable period of time. Marking the user process/thread real-time > class doesn't help since the real-time user threads priorities are still > lower than the interrupt threads. BTW, in our case, the CPU spends very > long time at PI_NET (16) priority level, instead of at the (PI_SOFT + 4) > soft intr level due to the packet forwarding nature. Either way, the > user processes don't have a chance. In the following discussion I'll use > PI_NET to represent the network interrupt threads priority. I've seen some similar live lock issues with large quantities of local IPC traffic on SMP systems. Could you tell me if you are using an SMP system or SMP kernel? Are you using direct dispatch or link layer bridging that's running in the ithread as opposed to the netisr swi? Robert N M Watson > > > > I am experimenting with improving the real-time threads scheduling > performance and would like to hear from you. > > > > Here is what I am doing: > > 1. Raise the minimum real-time user threads priority to a value > higher than PI_NET, e.g. > > #define PRI_MIN_REALTIME 0 > > And use the rtprio() syscall or command to set the priority to be higher > than PI_NET, e.g. "rtprio 10 " > > > > This didn't turn out to be enough, since the real-time user process > still uses system services or drivers that run at a lower priority than > PI_NET, e.g. disk, tty, etc. > > > > 2. Change the PI_NET value to be the highest of all interrupt > threads. Potential performance impact isn't a concern here since we have > relatively rare non-network events. So PI_NET is changed to > (PRI_MIN_ITHD + 32) from (PRI_MIN_ITHD + 16). This should give > preference to all other I/O interrupt threads. I am assuming the > software interrupt scheduling isn't the bottleneck. > > > > I haven't got a chance to reproduce the problem using the 2nd change > yet. I suspect that it isn't enough to get what I want, since lots of > kernel code do msleep(...,pri,...) with "pri" bigger than the changed > PI_NET. But, if this thinking of changing the priority ranges makes > sense, PI_NET can always be changed to the highest value of all kernel > priority values, ie the lowest kernel priority. > > > > Hopefully this change would work and not be intrusive to the kernel. > > > > Thank you for your comments in advance. > > > > Youlin Feng > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >