From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 18 19:17:06 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5B28106566C for ; Tue, 18 Sep 2012 19:17:06 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9EC4E8FC0C for ; Tue, 18 Sep 2012 19:17:06 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so757505pbb.13 for ; Tue, 18 Sep 2012 12:17:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:x-x-sender:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version:content-type:x-gm-message-state; bh=b7dtbKLjbiaXxjyZ/2hvIyYcLi/Z00Al/o8eKWv1Cv4=; b=DM4NCivOCA8UQBEGK3Rj0xGrtuC82JbA8XZeW3vlGAD1PypwIqJkOJNBBiD26vy1o8 LrtQFxNxSw4F/xA7/sCEQKo4wnrKuw7x2mAb6K+I8v23OC/1OPOLsKFa/XjvXGnyPZ7S NpeKqzsMxseKoBY4OCv7Ewfk15HvU9nNf4hVZnUoFmjS3mGaWKc184trJ4I8fCPP1FCq xAtiBBRfRT2MC3/cYQW+7uF900jrTs1wtV9GDhfTIyNc39H18hsFhwemqmm9BAGNsqo/ 2ph0FYFOOCptgDgxIjBdXRSPu1BK/BOodr7ReGTal50wlws5lKX7DPQDeQzOPHJil/s3 8IKg== Received: by 10.68.202.168 with SMTP id kj8mr1363012pbc.8.1347995826344; Tue, 18 Sep 2012 12:17:06 -0700 (PDT) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by mx.google.com with ESMTPS id sr3sm410069pbc.44.2012.09.18.12.17.04 (version=SSLv3 cipher=OTHER); Tue, 18 Sep 2012 12:17:05 -0700 (PDT) Date: Tue, 18 Sep 2012 09:16:35 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Andriy Gapon In-Reply-To: <5058C68B.1010508@FreeBSD.org> Message-ID: References: <50587F8D.9060102@FreeBSD.org> <5058C68B.1010508@FreeBSD.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Gm-Message-State: ALoCoQnbQPTCmSGJ+lmnKAcjLuqVkwq1R92SoxtDZ5b98MztFpeEzac8TTCS0Fmiui5eYON+s0ok X-Mailman-Approved-At: Tue, 18 Sep 2012 20:41:01 +0000 Cc: attilio@FreeBSD.org, freebsd-hackers , Jeff Roberson Subject: Re: ule+smp: small optimization for turnstile priority lending X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Sep 2012 19:17:06 -0000 On Tue, 18 Sep 2012, Andriy Gapon wrote: > on 18/09/2012 19:50 Attilio Rao said the following: >> On 9/18/12, Andriy Gapon wrote: >>> >>> Here is a snippet that demonstrates the issue on a supposedly fully loaded >>> 2-processor system: >>> >>> 136794 0 3670427870244462 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>> state:"running", attributes: prio:122 >>> >>> 136793 0 3670427870241000 KTRGRAPH group:"thread", id:"cc1plus tid >>> 111916", >>> state:"yielding", attributes: prio:183, wmesg:"(null)", lockname:"(null)" >>> >>> 136792 1 3670427870240829 KTRGRAPH group:"thread", id:"idle: cpu1 tid >>> 100004", >>> state:"running", attributes: prio:255 >>> >>> 136791 1 3670427870239520 KTRGRAPH group:"load", id:"CPU 1 load", >>> counter:0, >>> attributes: none >>> >>> 136790 1 3670427870239248 KTRGRAPH group:"thread", id:"firefox tid >>> 113473", >>> state:"blocked", attributes: prio:122, wmesg:"(null)", lockname:"unp_mtx" >>> >>> 136789 1 3670427870237697 KTRGRAPH group:"load", id:"CPU 0 load", >>> counter:2, >>> attributes: none >>> >>> 136788 1 3670427870236394 KTRGRAPH group:"thread", id:"firefox tid >>> 113473", >>> point:"wokeup", attributes: linkedto:"Xorg tid 102818" >>> >>> 136787 1 3670427870236145 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>> state:"runq add", attributes: prio:122, linkedto:"firefox tid 113473" >>> >>> 136786 1 3670427870235981 KTRGRAPH group:"load", id:"CPU 1 load", >>> counter:1, >>> attributes: none >>> >>> 136785 1 3670427870235707 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>> state:"runq rem", attributes: prio:176 >>> >>> 136784 1 3670427870235423 KTRGRAPH group:"thread", id:"Xorg tid 102818", >>> point:"prio", attributes: prio:176, new prio:122, linkedto:"firefox tid >>> 113473" >>> >>> 136783 1 3670427870202392 KTRGRAPH group:"thread", id:"firefox tid >>> 113473", >>> state:"running", attributes: prio:104 >>> >>> See how how the Xorg thread was forced from CPU 1 to CPU 0 where it >>> preempted >>> cc1plus thread (I do have preemption enabled) only to leave CPU 1 with zero >>> load. >> >> I think that the idea is bright, but I have reservations against the >> implementation because it seems to me there are too many layering >> violations. > > Just one - for a layer between tunrstile and scheduler :-) > But I agree. > >> What is suggest is somewhat summarized like that: >> - Add a new SRQ_WILLSLEEP or the name you prefer >> - Add a new "flags" argument to sched_lend_prio() (both ule and 4bsd) >> and sched_thread_priority (ule only) >> - sched_thread_priority() will pass down the new flag to sched_add() >> which passed down to sched_pickcpu(). >> >> This way sched_pickcpu() has the correct knowledge of what is going on >> and it can make the right decision. You likely don't need to lower the >> tdq_load at that time either this way, because sched_pickcpu() can >> just adjust it locally for its decision. >> >> What do you think? > > This sounds easy but it is not quite so given the implementation of > sched_pickcpu and sched_lowest. This is probably more work than I am able to > take now. I agree with Attillio's assessment. I have tried to do similar things before for ithreads etc. I think a more generic approach would be good. I will put it on my list of things to look at and we'll see who gets time first. Thanks, Jeff > > -- > Andriy Gapon >