From owner-freebsd-net@FreeBSD.ORG Tue Jan 4 01:43:28 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8FF5816A4CE for ; Tue, 4 Jan 2005 01:43:28 +0000 (GMT) Received: from mail23-kan-R.bigfish.com (mail-kan.bigfish.com [63.161.60.29]) by mx1.FreeBSD.org (Postfix) with ESMTP id F07AF43D1F for ; Tue, 4 Jan 2005 01:43:27 +0000 (GMT) (envelope-from yfeng@verniernetworks.com) Received: from mail23-kan.bigfish.com (localhost.localdomain [127.0.0.1]) by mail23-kan-R.bigfish.com (Postfix) with ESMTP id 5403D2E50D6 for ; Tue, 4 Jan 2005 01:43:27 +0000 (UCT) X-BigFish: VC Received: by mail23-kan (MessageSwitch) id 1104803007264277_21350; Tue, 4 Jan 2005 01:43:27 +0000 (UCT) Received: from exch2.verniernetworks.com (dns.verniernetworks.com [65.200.185.165]) by mail23-kan.bigfish.com (Postfix) with ESMTP id F3A9E2E3031 for ; Tue, 4 Jan 2005 01:43:26 +0000 (UCT) X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Mon, 3 Jan 2005 17:43:16 -0800 Message-ID: <12394E9FCB7C8441BB238D7F67B402E407E5E1@exch2.verniernetworks.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: User process starvation under heavy network traffic in FreeBSD 5.3 Thread-Index: AcTx/sKgb83W/4tDRuOSDUpWKB83uw== From: "Youlin Feng" To: Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: User process starvation under heavy network traffic in FreeBSD 5.3 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jan 2005 01:43:28 -0000 Hello, =20 I posted this question to freebsd-arch earlier. Stefan Bethke suggested that I try freebsd-net instead. =20 We are building a network appliance running FreeBSD 5.3 and under very heavy network traffic the user processes don't get scheduled for an unacceptable period of time. Marking the user process/thread real-time class doesn't help since the real-time user threads priorities are still lower than the interrupt threads. BTW, in our case, the CPU spends very long time at PI_NET (16) priority level, instead of at the (PI_SOFT + 4) soft intr level due to the packet forwarding nature. Either way, the user processes don't have a chance. In the following discussion I'll use PI_NET to represent the network interrupt threads priority. =20 I am experimenting with improving the real-time threads scheduling performance and would like to hear from you. =20 Here is what I am doing: 1. Raise the minimum real-time user threads priority to a value higher than PI_NET, e.g.=20 #define PRI_MIN_REALTIME 0 And use the rtprio() syscall or command to set the priority to be higher than PI_NET, e.g. "rtprio 10 " =20 This didn't turn out to be enough, since the real-time user process still uses system services or drivers that run at a lower priority than PI_NET, e.g. disk, tty, etc. =20 2. Change the PI_NET value to be the highest of all interrupt threads. Potential performance impact isn't a concern here since we have relatively rare non-network events. So PI_NET is changed to (PRI_MIN_ITHD + 32) from (PRI_MIN_ITHD + 16). This should give preference to all other I/O interrupt threads. I am assuming the software interrupt scheduling isn't the bottleneck.=20 =20 I haven't got a chance to reproduce the problem using the 2nd change yet. I suspect that it isn't enough to get what I want, since lots of kernel code do msleep(...,pri,...) with "pri" bigger than the changed PI_NET. But, if this thinking of changing the priority ranges makes sense, PI_NET can always be changed to the highest value of all kernel priority values, ie the lowest kernel priority. =20 Hopefully this change would work and not be intrusive to the kernel. =20 Thank you for your comments in advance. =20 Youlin Feng =20