From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 19 07:33:55 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D8B4106564A; Wed, 19 Sep 2012 07:33:55 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4D0FD8FC08; Wed, 19 Sep 2012 07:33:53 +0000 (UTC) Received: by lbbgg13 with SMTP id gg13so716945lbb.13 for ; Wed, 19 Sep 2012 00:33:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=NC28xv6dmwd8sDkrrRh6mnWAD4BGvFF0SVCD3a8Z8l4=; b=fIXNPPLuN0jqC8+f9Gi41QY9HB7IghAOlcl99HCGX9vm4pyQj6/D0YCBfgN320Fjwz XcyJbrbBkgFuZnR8per5svK//dANm5wquc3zk+qXzXbjzQYwSzYWY0pPgU8FIZ/2IsCc J2krnnMBVbuHh8qxkWYZGu61g9j9aJ1p4talyBbPE1VlAfAKVoZSBBvQwf1SoZWQXc51 P/GQtdhMyhIhP1WIpl0G1SY0bfWZb9IIyEo3uivZFi4KoesJd8/qm17B92eWKmDXwlJl 8iiRNcF/vvZVxMfBCeqL/kl+e7uCOmYEKumSgdHYonwkQKCidd3t1p2dGadxbPrhLkrS RzOA== MIME-Version: 1.0 Received: by 10.152.112.233 with SMTP id it9mr1877561lab.40.1348040031905; Wed, 19 Sep 2012 00:33:51 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.112.102.39 with HTTP; Wed, 19 Sep 2012 00:33:51 -0700 (PDT) In-Reply-To: <50596019.5060708@FreeBSD.org> References: <50587F8D.9060102@FreeBSD.org> <5058C68B.1010508@FreeBSD.org> <50596019.5060708@FreeBSD.org> Date: Wed, 19 Sep 2012 08:33:51 +0100 X-Google-Sender-Auth: MKVOak4DfQYeJ2NWf35VvsWNbok Message-ID: From: Attilio Rao To: Andriy Gapon Content-Type: text/plain; charset=UTF-8 Cc: freebsd-hackers , Jeff Roberson Subject: Re: ule+smp: small optimization for turnstile priority lending X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Sep 2012 07:33:55 -0000 On Wed, Sep 19, 2012 at 7:03 AM, Andriy Gapon wrote: > on 19/09/2012 02:01 Attilio Rao said the following: >> On Tue, Sep 18, 2012 at 8:07 PM, Andriy Gapon wrote: >>> on 18/09/2012 19:50 Attilio Rao said the following: >>>> On 9/18/12, Andriy Gapon wrote: >>>>> >>>>> Here is a snippet that demonstrates the issue on a supposedly fully loaded >>>>> 2-processor system: >>>>> >>>>> 136794 0 3670427870244462 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>>>> state:"running", attributes: prio:122 >>>>> >>>>> 136793 0 3670427870241000 KTRGRAPH group:"thread", id:"cc1plus tid >>>>> 111916", >>>>> state:"yielding", attributes: prio:183, wmesg:"(null)", lockname:"(null)" >>>>> >>>>> 136792 1 3670427870240829 KTRGRAPH group:"thread", id:"idle: cpu1 tid >>>>> 100004", >>>>> state:"running", attributes: prio:255 >>>>> >>>>> 136791 1 3670427870239520 KTRGRAPH group:"load", id:"CPU 1 load", >>>>> counter:0, >>>>> attributes: none >>>>> >>>>> 136790 1 3670427870239248 KTRGRAPH group:"thread", id:"firefox tid >>>>> 113473", >>>>> state:"blocked", attributes: prio:122, wmesg:"(null)", lockname:"unp_mtx" >>>>> >>>>> 136789 1 3670427870237697 KTRGRAPH group:"load", id:"CPU 0 load", >>>>> counter:2, >>>>> attributes: none >>>>> >>>>> 136788 1 3670427870236394 KTRGRAPH group:"thread", id:"firefox tid >>>>> 113473", >>>>> point:"wokeup", attributes: linkedto:"Xorg tid 102818" >>>>> >>>>> 136787 1 3670427870236145 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>>>> state:"runq add", attributes: prio:122, linkedto:"firefox tid 113473" >>>>> >>>>> 136786 1 3670427870235981 KTRGRAPH group:"load", id:"CPU 1 load", >>>>> counter:1, >>>>> attributes: none >>>>> >>>>> 136785 1 3670427870235707 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>>>> state:"runq rem", attributes: prio:176 >>>>> >>>>> 136784 1 3670427870235423 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>>>> point:"prio", attributes: prio:176, new prio:122, linkedto:"firefox tid >>>>> 113473" >>>>> >>>>> 136783 1 3670427870202392 KTRGRAPH group:"thread", id:"firefox tid >>>>> 113473", >>>>> state:"running", attributes: prio:104 >>>>> >>>>> See how how the Xorg thread was forced from CPU 1 to CPU 0 where it >>>>> preempted >>>>> cc1plus thread (I do have preemption enabled) only to leave CPU 1 with zero >>>>> load. >>>> >>>> I think that the idea is bright, but I have reservations against the >>>> implementation because it seems to me there are too many layering >>>> violations. >>> >>> Just one - for a layer between tunrstile and scheduler :-) >>> But I agree. >>> >>>> What is suggest is somewhat summarized like that: >>>> - Add a new SRQ_WILLSLEEP or the name you prefer >>>> - Add a new "flags" argument to sched_lend_prio() (both ule and 4bsd) >>>> and sched_thread_priority (ule only) >>>> - sched_thread_priority() will pass down the new flag to sched_add() >>>> which passed down to sched_pickcpu(). >>>> >>>> This way sched_pickcpu() has the correct knowledge of what is going on >>>> and it can make the right decision. You likely don't need to lower the >>>> tdq_load at that time either this way, because sched_pickcpu() can >>>> just adjust it locally for its decision. >>>> >>>> What do you think? >>> >>> This sounds easy but it is not quite so given the implementation of >>> sched_pickcpu and sched_lowest. This is probably more work than I am able to >>> take now. >> >> I think actually that the attached patch should do what you need. Of >> course there is more runqueue locking, due to the tdq_load fondling, >> but I'm not sure it is really a big deal. >> I didn't test it yet, as I understand you have a test case, so maybe >> you can. However if Jeff agrees I can send the patch to flo@ for >> performance evaluation. >> >> Thoughts? > > Originally I had a similar patch with a small difference that I did tdq_load > manipulations in sched_thread_priority around sched_add call. That patch > produced a deadlock because of the need for TDQ_LOCK. It is hard for me to tell if this is subject to same issues because I do not entirely understand where the locking was happening in your patch. Can you try testing this with your already KTR instrumented test and possibly report? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein