From owner-freebsd-net@FreeBSD.ORG Mon Apr 6 12:09:10 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F6291065745; Mon, 6 Apr 2009 12:09:10 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id E62D08FC1A; Mon, 6 Apr 2009 12:09:09 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id 96DAC46B90; Mon, 6 Apr 2009 08:09:09 -0400 (EDT) Date: Mon, 6 Apr 2009 13:09:09 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Barney Cordoba In-Reply-To: <86599.63596.qm@web63904.mail.re1.yahoo.com> Message-ID: References: <86599.63596.qm@web63904.mail.re1.yahoo.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Ivan Voras Subject: Re: Advice on a multithreaded netisr patch? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Apr 2009 12:09:15 -0000 On Mon, 6 Apr 2009, Barney Cordoba wrote: > Is there a way to give a kernel thread exclusive use of a core? I know you > can pin a kernel thread with sched_bind(), but is there a way to keep other > threads from using the core? On an 8 core system it almost seems that the > randomness of more cores is a negative in some situations. > > Also, I've noticed that calling sched_bind() during bootup is a bad thing in > that it locks the system. I'm not certain but I suspect its the thread_lock > that is the culprit. Is there a clean way to determine that its safe to lock > curthread and do a cpu bind? There isn't an interface to cleanly express "Use CPUs 4-7 for only network processing". You can configure the system this way using the cpuset command (including directing the low-level interrupts to specific CPUs in 8.x), but if we think this is going to be a frequently desired policy, a bit more abstraction will be required. I'm not familiar with the problem you're seeing with sched_bind() -- I'm using it from within some of my code without a problem, and that's fairly early in the boot. A number of deadlocks are possible if one isn't very careful early in the boot though, so I might look specifically for some of those: if you migrate a thread to a CPU that isn't yet started, it won't be able to run until the CPU has started. This means it's important not to migrate threads that might lead to priority version-like deadlocks: - Be careful not to migrate threads that hold locks the system requires to get to the point where multiple CPUs run. - Be careful not to migrate threads that will signal a resource being available, such as a device driver, required to get to the point where multiple CPUs run. - Be careful not to migrate the main boot thread. Could you be running into one of those cases? Usually they're fairly easy to diagnose using DDB, if you can get into it, because you can see what the main boot thread is waiting for, and reason about what's holding it. Are you able to get into DDB when this occurs? (Perhaps using an NMI?) Robert N M Watson Computer Laboratory University of Cambridge