From owner-freebsd-arch@FreeBSD.ORG Fri Mar 7 12:13:59 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A528B1065672; Fri, 7 Mar 2008 12:13:59 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 6D1008FC1E; Fri, 7 Mar 2008 12:13:59 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m27CDpcK074502; Fri, 7 Mar 2008 07:13:55 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Fri, 7 Mar 2008 02:16:30 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: arch@freebsd.org Message-ID: <20080307020626.G920@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Getting rid of the static msleep priority boost X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Mar 2008 12:13:59 -0000 Hello, I've been studying some problems with recent scheduler improvements that help a lot on some workloads and hurt on others. I've tracked the problem down to static priority boosts handed out by msleep/cv_broadcastpri. The basic problem is that a user thread will be woken with a kernel priority thus allowing it to preempt a thread running on any processor with a lesser priority. The lesser priority thread may in fact hold some resource that the higher priority thread requires. Thus we context switch several times and perhaps go through priority propagation as well. I have verified that disabling these static priority boosts entirely fixes the performance problem I've run into on at least one workload. There are probably others that it helps and hopefully we can discover that. I'd like to know if anyone has a strong preference to keep this feature. It is likely that it helps in some interactive situations. I'm not sure how much however. I propose that we make a sysctl that disables it and turn it off by default. If we see complaints on current@ we can suggest that they toggle the sysctl to see if it alleviates problems. Based on feedback from that experiment and some testing we can then choose a few options: 1) Disable the static boosts entirely. Leave kernel priorities for kernel threads and priority propagation. Most other kernels do this. Would make my life in ULE much easier as well. 2) Leave the support for static boosts but remove it from all but a few key locations. Leaving it in the api would give some flexibility but might confuse developers. 3) Leave things as they are. undesirable. I'm leaning towards #2 based on the information I have presently. This is almost a significant change to historic BSD behavior so we might want to tread lightly. Thanks, Jeff