From owner-freebsd-hackers@FreeBSD.ORG Fri Apr 6 01:08:31 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BBC2106566C for ; Fri, 6 Apr 2012 01:08:31 +0000 (UTC) (envelope-from sushanth_rai@yahoo.com) Received: from nm14.bullet.mail.sp2.yahoo.com (nm14.bullet.mail.sp2.yahoo.com [98.139.91.84]) by mx1.freebsd.org (Postfix) with SMTP id 201428FC15 for ; Fri, 6 Apr 2012 01:08:31 +0000 (UTC) Received: from [98.139.91.69] by nm14.bullet.mail.sp2.yahoo.com with NNFMP; 06 Apr 2012 01:08:25 -0000 Received: from [98.139.44.66] by tm9.bullet.mail.sp2.yahoo.com with NNFMP; 06 Apr 2012 01:08:25 -0000 Received: from [127.0.0.1] by omp1003.access.mail.sp2.yahoo.com with NNFMP; 06 Apr 2012 01:08:25 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 333431.3713.bm@omp1003.access.mail.sp2.yahoo.com Received: (qmail 6175 invoked by uid 60001); 6 Apr 2012 01:08:24 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1333674504; bh=ZY/jupau+wmGIoHnSpQZnDVpCcfM2clnbdlwYmmJJis=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=It6eb1GMG5wfvvBFsRPO3Ta9d0uHXu1cLu/1O1lCYcynlo+lDPo2Yn2CUJMJ+LF8EBeskot4ufpBy4WWES3fv+Z3PFuqeK/1qWDogKhiKDbWldKIfvzErm7eKLgd07aclzyJbI5oIjNBVaDBziH80/FyTRg6eBTajOMg1w3jgzY= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=vnAbMRTBN3PRiO7Q/oW+05CHH3T2h5R0Qamem+9Emz5aXF80cdPj2XJxZ+srVE9/ggkbmiapDuGPvsU1LoP7P2xcGX+b6280a3hEyr2U0qLGKI5+7FkYCdWHI5y5wHRmMW4lQFlPyrlt66jIZSjlbPljTUM8n3EgAatv2aSv0Pw=; X-YMail-OSG: NxGL5jIVM1lJjoAHdiXfgj0g8jpbBhDeZrfFYD42EBCO63r sOFstUBAnQPoTIpxf7cPerBx_dVLxi8oZG6gL1YugXudAGxTakrazJiTCgKf kgrX93Y_.ARSSMsAqVcj_mv0BSVp4.uvx16UV3lBdixDGgK.Cvj.GLkpZgi. IbCTpyftQeLMjdNn9G2xuMmU1YKcZ7DXGDtrQg6bpHr.mRtInrq.reHAxnb6 pv3ZxtfSnHiFKetK.7ouy7El2GON5lQOEwthzWAjqmWZfSTd9cQgq.tlXNuM 9Q8CHeSViQATaIViE0lRbluZELOknG9oL1tIL3jrpe47oWyaIL9L_hTlK6W. ffCk_LMrNDiXZf7VufsLwXFBRtZ6JaTy9.NpH0azOy0jhoZoxT68uHv6Hdis MgQlEBxGrbXkwHqMtijHiIVBlHeLxefYl3FOtWqDb6E0NcrOp75nYG.SWD.f DTqW3 Received: from [75.62.238.5] by web180016.mail.gq1.yahoo.com via HTTP; Thu, 05 Apr 2012 18:08:24 PDT X-Mailer: YahooMailClassic/15.0.5 YahooMailWebService/0.8.117.340979 Message-ID: <1333674504.97862.YahooMailClassic@web180016.mail.gq1.yahoo.com> Date: Thu, 5 Apr 2012 18:08:24 -0700 (PDT) From: Sushanth Rai To: John Baldwin In-Reply-To: <201204051201.58651.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Startvation of realtime piority threads X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2012 01:08:31 -0000 I understand the downside of badly written realtime app. In my case applic= ation runs in userspace without making much syscalls and by all means it is= a well behaved application. Yes, I can wire memory, change the application= to use mutex instead of spinlock and those changes should help but they ar= e still working around the problem. I still believe kernel should not lower= the realtime priority when blocking on resources. This can lead to priorit= y inversion, especially since these threads run at fixed priorities and ker= nel doesn't muck with them.=0A =0AAs you suggested _sleep() should not adju= st the priorities for realtime threads. =0A=0AThanks,=0ASushanth=0A=0A--- O= n Thu, 4/5/12, John Baldwin wrote:=0A=0A> From: John Bald= win =0A> Subject: Re: Startvation of realtime piority thre= ads=0A> To: freebsd-hackers@freebsd.org, davidxu@freebsd.org=0A> Date: Thur= sday, April 5, 2012, 9:01 AM=0A> On Thursday, April 05, 2012 1:07:55=0A> am= David Xu wrote:=0A> > On 2012/4/5 11:56, Konstantin Belousov wrote:=0A> > = > On Wed, Apr 04, 2012 at 06:54:06PM -0700, Sushanth=0A> Rai wrote:=0A> > >= > I have a multithreaded user space program that=0A> basically runs at real= time =0A> priority. Synchronization between threads are done using=0A> spin= lock. When =0A> running this program on a SMP system under heavy memory=0A>= pressure I see that =0A> thread holding the spinlock is starved out of cpu= . The cpus=0A> are effectively =0A> consumed by other threads that are spin= ning for lock to=0A> become available.=0A> > >>=0A> > >> After instrumentin= g the kernel a little bit=0A> what I found was that under =0A> memory press= ure, when the user thread holding the spinlock=0A> traps into the =0A> kern= el due to page fault, that thread sleeps until the free=0A> pages are =0A> = available. The thread sleeps PUSER priority (within=0A> vm_waitpfault()). W= hen it =0A> is ready to run, it is queued at PUSER priority even thought=0A= > it's base =0A> priority is realtime. The other siblings threads that are= =0A> spinning at realtime =0A> priority to acquire the spinlock starves the= owner of=0A> spinlock.=0A> > >>=0A> > >> I was wondering if the sleep in= =0A> vm_waitpfault() should be a =0A> MAX(td_user_pri, PUSER) instead of ju= st PUSER. I'm running=0A> on 7.2 and it looks =0A> like this logic is the s= ame in the trunk.=0A> > > It just so happen that your program stumbles upon= =0A> a single sleep point in=0A> > > the kernel. If for whatever reason the= thread in=0A> kernel is put off CPU=0A> > > due to failure to acquire any = resource without=0A> priority propagation,=0A> > > you would get the same e= ffect. Only blockable=0A> primitives do priority=0A> > > propagation, that = are mutexes and rwlocks, AFAIR.=0A> In other words, any=0A> > > sx/lockmgr/= sleep points are vulnerable to the same=0A> issue.=0A> > This is why I sugg= ested that POSIX realtime priority=0A> should not be =0A> > boosted, it sho= uld be=0A> > only higher than PRI_MIN_TIMESHARE but lower than any=0A> prio= rity all =0A> > msleep() callers=0A> > provided.=A0 The problem is userland= realtime thread=0A> 's busy looping code =0A> > can cause=0A> > starvation= a thread in kernel which holding a critical=0A> resource.=0A> > In kernel = we can avoid to write dead-loop code, but=0A> userland code is not =0A> > t= rustable.=0A> =0A> Note that you have to be root to be rtprio, and that the= re=0A> is trustable=0A> userland code (just because you haven't used any do= esn't=0A> mean it doesn't=0A> exist).=0A> =0A> > If you search "Realtime th= read priorities" in=0A> 2010-december within @arch =0A> > list.=0A> > you m= ay find the argument.=0A> =0A> I think the bug here is that sched_sleep() s= hould not lower=0A> the priority of=0A> an rtprio process.=A0 It should arg= uably not raise the=0A> priority of an idprio=0A> process either, but sched= _sleep() should probably only apply=0A> to timesharing=0A> threads.=0A> =0A= > All that said, userland rtprio code is going to have to be=0A> careful.= =A0 It should=0A> be using things like wired memory as Kostik suggested, an= d=0A> probably avoiding=0A> most system calls.=A0 You can definitely blow y= our foot=0A> off quite easily in lots =0A> of ways with rtprio.=0A> =0A> --= =0A> John Baldwin=0A> _______________________________________________=0A> = freebsd-hackers@freebsd.org=0A> mailing list=0A> http://lists.freebsd.org/m= ailman/listinfo/freebsd-hackers=0A> To unsubscribe, send any mail to "freeb= sd-hackers-unsubscribe@freebsd.org"=0A>