From owner-freebsd-current@FreeBSD.ORG Thu Apr 5 18:46:01 2012 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA18F1065680; Thu, 5 Apr 2012 18:46:01 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9F8508FC17; Thu, 5 Apr 2012 18:46:00 +0000 (UTC) Received: by wern13 with SMTP id n13so1363198wer.13 for ; Thu, 05 Apr 2012 11:45:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=d+b5ocGMfgLi5iHgcjBuONGM0pHcjjFqK6YOdy+fwXk=; b=rfliI3aIpQ9MN2qx/j+3oPf7mGyR5nCVTpqODciiEMk2SPQktZBLhl2ZqcanTqvJKF h82FzLn6NJqVWItexC+bgqEmWSGBLAQ9PrPFxllBHwU+oYS10M7lxBl88qSXIKgOQwwk xVo1Bd4Mx97TCTEzLWy87R1F1VDqhZnnO4Yy6PbDI8lJuzU08Lz3ouVRwfZR9mEozJ8E o2AVxTazkIvOIUj4mbWM52n+E3oQNN1HlnXvL9HLR++6l59cThG576unsPJGrqyLArxD mVZQN0+Nhz5dfUsXbOZb9EhFnwSUb6f8FFVLFR+WskchCvCYQznTjGv04mdpPc+d9aFN 1P4w== Received: by 10.216.134.27 with SMTP id r27mr2307338wei.107.1333651559469; Thu, 05 Apr 2012 11:45:59 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id 9sm20370826wid.2.2012.04.05.11.45.56 (version=SSLv3 cipher=OTHER); Thu, 05 Apr 2012 11:45:58 -0700 (PDT) Sender: Alexander Motin Message-ID: <4F7DE863.6080607@FreeBSD.org> Date: Thu, 05 Apr 2012 21:45:55 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20120116 Thunderbird/9.0 MIME-Version: 1.0 To: Arnaud Lacombe References: <4F2F7B7F.40508@FreeBSD.org> <4F366E8F.9060207@FreeBSD.org> <4F367965.6000602@FreeBSD.org> <4F396B24.5090602@FreeBSD.org> <4F3978BC.6090608@FreeBSD.org> <4F3990EA.1080002@FreeBSD.org> <4F3C0BB9.6050101@FreeBSD.org> <4F3E807A.60103@FreeBSD.org> <4F3E8858.4000001@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Jeff Roberson , Andriy Gapon , FreeBSD current Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2012 18:46:01 -0000 On 05.04.2012 21:12, Arnaud Lacombe wrote: > Hi, > > [Sorry for the delay, I got a bit sidetrack'ed...] > > 2012/2/17 Alexander Motin: >> On 17.02.2012 18:53, Arnaud Lacombe wrote: >>> >>> On Fri, Feb 17, 2012 at 11:29 AM, Alexander Motin wrote: >>>> >>>> On 02/15/12 21:54, Jeff Roberson wrote: >>>>> >>>>> On Wed, 15 Feb 2012, Alexander Motin wrote: >>>>>> >>>>>> I've decided to stop those cache black magic practices and focus on >>>>>> things that really exist in this world -- SMT and CPU load. I've >>>>>> dropped most of cache related things from the patch and made the rest >>>>>> of things more strict and predictable: >>>>>> http://people.freebsd.org/~mav/sched.htt34.patch >>>>> >>>>> >>>>> This looks great. I think there is value in considering the other >>>>> approach further but I would like to do this part first. It would be >>>>> nice to also add priority as a greater influence in the load balancing >>>>> as well. >>>> >>>> >>>> I haven't got good idea yet about balancing priorities, but I've >>>> rewritten >>>> balancer itself. As soon as sched_lowest() / sched_highest() are more >>>> intelligent now, they allowed to remove topology traversing from the >>>> balancer itself. That should fix double-swapping problem, allow to keep >>>> some >>>> affinity while moving threads and make balancing more fair. I did number >>>> of >>>> tests running 4, 8, 9 and 16 CPU-bound threads on 8 CPUs. With 4, 8 and >>>> 16 >>>> threads everything is stationary as it should. With 9 threads I see >>>> regular >>>> and random load move between all 8 CPUs. Measurements on 5 minutes run >>>> show >>>> deviation of only about 5 seconds. It is the same deviation as I see >>>> caused >>>> by only scheduling of 16 threads on 8 cores without any balancing needed >>>> at >>>> all. So I believe this code works as it should. >>>> >>>> Here is the patch: http://people.freebsd.org/~mav/sched.htt40.patch >>>> >>>> I plan this to be a final patch of this series (more to come :)) and if >>>> there will be no problems or objections, I am going to commit it (except >>>> some debugging KTRs) in about ten days. So now it's a good time for >>>> reviews >>>> and testing. :) >>>> >>> is there a place where all the patches are available ? >> >> >> All my scheduler patches are cumulative, so all you need is only the last >> mentioned here sched.htt40.patch. >> > You may want to have a look to the result I collected in the > `runs/freebsd-experiments' branch of: > > https://github.com/lacombar/hackbench/ > > and compare them with vanilla FreeBSD 9.0 and -CURRENT results > available in `runs/freebsd'. On the dual package platform, your patch > is not a definite win. > >> But in some cases, especially for multi-socket systems, to let it show its >> best, you may want to apply additional patch from avg@ to better detect CPU >> topology: >> https://gitorious.org/~avg/freebsd/avgbsd/commit/6bca4a2e4854ea3fc275946a023db65c483cb9dd >> > test I conducted specifically for this patch did not showed much improvement... If I understand right, this test runs thousands of threads sending and receiving data over the pipes. It is quite likely that all CPUs will be always busy and so load balancing is not really important in this test, What looks good is that more complicated new code is not slower then old one. While this test seems very scheduler-intensive, it may depend on many other factors, such as syscall performance, context switch, etc. I'll try to play more with it. -- Alexander Motin