From owner-freebsd-hackers@FreeBSD.ORG Mon Feb 13 19:57:29 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D007106566C; Mon, 13 Feb 2012 19:57:29 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 96C3E8FC13; Mon, 13 Feb 2012 19:57:28 +0000 (UTC) Received: by eekb47 with SMTP id b47so2243341eek.13 for ; Mon, 13 Feb 2012 11:57:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=a1qqtJUMOr8gjjShKuVthnvAOVm9NBbDYiHottTMAWY=; b=p1YM3bGKWS2Bn2Gch8P5/Mun8D4xxtH3V/nmI4/4yNN5gUSVpUBszh0x6Bjwzi0qY8 gc67wW8h/P/MjbNZ28u6wxLNl/1mgQIHMsup09Bzd+2vFOqKC6w7yEZqSRRxNiJ0DaDH Qpy3zbQnXqAefeFpNdWncteKKBCY5UlSXRPcw= Received: by 10.14.95.129 with SMTP id p1mr5908439eef.10.1329163047480; Mon, 13 Feb 2012 11:57:27 -0800 (PST) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua. [212.86.226.226]) by mx.google.com with ESMTPS id n52sm4957768eea.5.2012.02.13.11.57.25 (version=SSLv3 cipher=OTHER); Mon, 13 Feb 2012 11:57:26 -0800 (PST) Sender: Alexander Motin Message-ID: <4F396B24.5090602@FreeBSD.org> Date: Mon, 13 Feb 2012 21:57:24 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:9.0) Gecko/20111227 Thunderbird/9.0 MIME-Version: 1.0 To: Andriy Gapon References: <4F2F7B7F.40508@FreeBSD.org> <4F366E8F.9060207@FreeBSD.org> <4F367965.6000602@FreeBSD.org> In-Reply-To: <4F367965.6000602@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org, Florian Smeets Subject: Re: [RFT][patch] Scheduling for HTT and not only X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2012 19:57:29 -0000 On 02/11/12 16:21, Alexander Motin wrote: > I've heavily rewritten the patch already. So at least some of the ideas > are already addressed. :) At this moment I am mostly satisfied with > results and after final tests today I'll probably publish new version. It took more time, but finally I think I've put pieces together: http://people.freebsd.org/~mav/sched.htt23.patch The patch is more complicated then previous one both logically and computationally, but with growing CPU power and complexity I think we can possibly spend some more time deciding how to spend time. :) Patch formalizes several ideas of the previous code about how to select CPU for running a thread and adds some new. It's main idea is that I've moved from comparing raw integer queue lengths to higher-resolution flexible values. That additional 8-bit precision allows same time take into account many factors affecting performance. Beside just choosing best from equally-loaded CPUs, with new code it may even happen that because of SMT, cache affinity, etc, CPU with more threads on it's queue will be reported as less loaded and opposite. New code takes into account such factors: - SMT sharing penalty. - Cache sharing penalty. - Cache affinity (with separate coefficients for last-level and other level caches) to the: - other running threads of it's process, - previous CPU where it was running, - current CPU (usually where it was called from). All of these factors are configurable via sysctls, but I think reasonable defaults should fit most. Also, comparing to previous patch, I've resurrected optimized shortcut in CPU selection for the case of SMT. Comparing to original code having problems with this, I've added check for other logical cores load that should make it safe and still very fast when there are less running threads then physical cores. I've tested in on Core i7 and Atom systems, but more interesting would be to test it on multi-socket system with properly detected topology to check benefits from affinity. At this moment the main issue I see is that this patch affects only time when thread is starting. If thread runs continuously, it will stay where it was, even if due to situation change that is not very effective (causes SMT sharing, etc). I haven't looked much on periodic load balancer yet, but probably it could also be somehow improved. What is your opinion, is it too over-engineered, or it is the right way to go? -- Alexander Motin