From owner-freebsd-arch@FreeBSD.ORG Mon Jan 3 22:23:39 2005 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E76B516A4CE for ; Mon, 3 Jan 2005 22:23:39 +0000 (GMT) Received: from mail18-ash-R.bigfish.com (mail-ash.bigfish.com [206.16.192.253]) by mx1.FreeBSD.org (Postfix) with ESMTP id 63FD643D45 for ; Mon, 3 Jan 2005 22:23:39 +0000 (GMT) (envelope-from yfeng@verniernetworks.com) Received: from mail18-ash.bigfish.com (localhost.localdomain [127.0.0.1]) by mail18-ash-R.bigfish.com (Postfix) with ESMTP id D61295E572B for ; Mon, 3 Jan 2005 22:23:36 +0000 (UCT) X-BigFish: VC Received: by mail18-ash (MessageSwitch) id 1104791016709629_16166; Mon, 3 Jan 2005 22:23:36 +0000 (UCT) Received: from exch2.verniernetworks.com (dns.verniernetworks.com [65.200.185.165]) by mail18-ash.bigfish.com (Postfix) with ESMTP id 5C9DF5E5BB5 for ; Mon, 3 Jan 2005 22:23:36 +0000 (UCT) X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Mon, 3 Jan 2005 14:23:25 -0800 Message-ID: <12394E9FCB7C8441BB238D7F67B402E407E5C6@exch2.verniernetworks.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: User process starvation in FreeBSD 5.3 Thread-Index: AcTx4teEtGwBgzB/QNOtBKDrs6TYug== From: "Youlin Feng" To: Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: User process starvation in FreeBSD 5.3 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jan 2005 22:23:40 -0000 Hello all, =20 We are building a network appliance running FreeBSD 5.3 and under very heavy network traffic the user processes don't get scheduled for an unacceptable period of time. Marking the user process/thread real-time class doesn't help since the real-time user threads priorities are still lower than the interrupt threads. BTW, in our case, the CPU spends very long time at PI_NET (16) priority level, instead of at the (PI_SOFT + 4) soft intr level due to the packet forwarding nature. Either way, the user processes don't have a chance. In the following discussion I'll use PI_NET to represent the network interrupt threads priority. =20 I am experimenting with improving the real-time threads scheduling performance and would like to hear from you. =20 Here is what I am doing: 1. Raise the minimum real-time user threads priority to a value higher than PI_NET, e.g.=20 #define PRI_MIN_REALTIME 0 And use the rtprio() syscall or command to set the priority to be higher than PI_NET, e.g. "rtprio 10 " =20 This didn't turn out to be enough, since the real-time user process still uses system services or drivers that run at a lower priority than PI_NET, e.g. disk, tty, etc. =20 2. Change the PI_NET value to be the highest of all interrupt threads. Potential performance impact isn't a concern here since we have relatively rare non-network events. So PI_NET is changed to (PRI_MIN_ITHD + 32) from (PRI_MIN_ITHD + 16). This should give preference to all other I/O interrupt threads. I am assuming the software interrupt scheduling isn't the bottleneck. =20 I haven't got a chance to reproduce the problem using the 2nd change yet. I suspect that it isn't enough to get what I want, since lots of kernel code do msleep(...,pri,...) with "pri" bigger than the changed PI_NET. But, if this thinking of changing the priority ranges makes sense, PI_NET can always be changed to the highest value of all kernel priority values, ie the lowest kernel priority. =20 Hopefully this change would work and not be intrusive to the kernel. =20 Thank you for your comments in advance. =20 Youlin Feng