From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 5 04:56:13 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93B12106566C for ; Thu, 5 Apr 2012 04:56:13 +0000 (UTC) (envelope-from listlog2011@gmail.com) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7D0BF8FC08 for ; Thu, 5 Apr 2012 04:56:13 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q354uAFm079316 for ; Thu, 5 Apr 2012 04:56:12 GMT (envelope-from listlog2011@gmail.com) Message-ID: <4F7D25E9.8030703@gmail.com> Date: Thu, 05 Apr 2012 12:56:09 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:11.0) Gecko/20120327 Thunderbird/11.0.1 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org References: <1333590846.58474.YahooMailClassic@web180011.mail.gq1.yahoo.com> In-Reply-To: <1333590846.58474.YahooMailClassic@web180011.mail.gq1.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Startvation of realtime piority threads X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: davidxu@freebsd.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2012 04:56:13 -0000 On 2012/4/5 9:54, Sushanth Rai wrote: > I have a multithreaded user space program that basically runs at realtime priority. Synchronization between threads are done using spinlock. When running this program on a SMP system under heavy memory pressure I see that thread holding the spinlock is starved out of cpu. The cpus are effectively consumed by other threads that are spinning for lock to become available. > > After instrumenting the kernel a little bit what I found was that under memory pressure, when the user thread holding the spinlock traps into the kernel due to page fault, that thread sleeps until the free pages are available. The thread sleeps PUSER priority (within vm_waitpfault()). When it is ready to run, it is queued at PUSER priority even thought it's base priority is realtime. The other siblings threads that are spinning at realtime priority to acquire the spinlock starves the owner of spinlock. > > I was wondering if the sleep in vm_waitpfault() should be a MAX(td_user_pri, PUSER) instead of just PUSER. I'm running on 7.2 and it looks like this logic is the same in the trunk. > > Thanks, > Sushanth I think 7.2 still has libkse which supports static priority scheduling, if performance is not important but correctness, you may try libkse with process-scope threads, and use priority-inherit mutex to do locking. Kernel is known to be vulnerable to support user realtime threads. I think not every-locking primitive can support priority propagation, this is an issue. In userland, internal library mutexes are not priority-inherit, so starvation may happen too. If you know what you are doing, don't call such functions which uses internal mutexes, but this is rather difficult. Regards, David Xu