From owner-freebsd-net@FreeBSD.ORG Mon Apr 6 13:41:01 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B9D6910656CA for ; Mon, 6 Apr 2009 13:41:01 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63906.mail.re1.yahoo.com (web63906.mail.re1.yahoo.com [69.147.97.121]) by mx1.freebsd.org (Postfix) with SMTP id 771C68FC23 for ; Mon, 6 Apr 2009 13:41:01 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 42589 invoked by uid 60001); 6 Apr 2009 13:41:00 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1239025260; bh=4zCf0qunp1yvuBrFzFbUJrL2OUSB327jY2S0yHu6Otk=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=o6EkvyJfY851396ia3yZsEMP+sbop7DFQjpD7UwXkDRA2PrTTnPHlzKm+avcWWMPW2AUiWuSdr87dD3xM9T9q2OrahshD7btZDk9zX1FT+BuDENx0D5e/oB/TuQg1D12/ZkoP4ahJ24Fh1nBQVlFr8sQ+bgNlE3XUIZtxJo1m4o= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=pGj/cgoD2ixdz3Dw6zSWaHXfhVVJiNeF740dr+0uXNlcpXggQjEbnH5K56KOCeFammkAQ4N4aGfQDf29sN5w7rnWSmY5I237KmE3FM0n8DV/ouyOPRio29hY0FDhWLZVIa1RT+Kjti7dHKI9OuQVdal6rqu8kO3ZecSI0lCYJoA=; Message-ID: <812958.41771.qm@web63906.mail.re1.yahoo.com> X-YMail-OSG: F3zTFRsVM1kGN4FvWZRmCWc7y9S5oeB4I4iqDv.KXpzSQtqyjerSwCOvJYslesZUyl_iwQ7LIir507cIyIMo4p8DXw57bV3x_fxwOQMb3C5zh273_JfLvLSTOnoHbmjqrofeM9uFM4cbNfPJ7o7KhsSUhXqSYnqIi_wDLLxXQevZVu5Y0Qc73QsK9t348kXfBlicKrxISmpJu6Msn.SvgDyzFZKZzxyA.Yeg3gEpiuP8waiZMmVyA3K45EL4KIE461W7ieLCjD20bMDg0uuu8sf7CDHOvIu5H3UMGTN5FTJAafiYbBgtwXecRHa5fM9ceguL8Qd4Hw4o_JeU3rxvPzhI Received: from [98.242.222.229] by web63906.mail.re1.yahoo.com via HTTP; Mon, 06 Apr 2009 06:41:00 PDT X-Mailer: YahooMailWebService/0.7.289.1 Date: Mon, 6 Apr 2009 06:41:00 -0700 (PDT) From: Barney Cordoba To: Robert Watson In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-net@freebsd.org, Ivan Voras Subject: Re: Advice on a multithreaded netisr patch? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: barney_cordoba@yahoo.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Apr 2009 13:41:02 -0000 --- On Mon, 4/6/09, Robert Watson wrote: > From: Robert Watson > Subject: Re: Advice on a multithreaded netisr patch? > To: "Barney Cordoba" > Cc: freebsd-net@freebsd.org, "Ivan Voras" > Date: Monday, April 6, 2009, 8:09 AM > On Mon, 6 Apr 2009, Barney Cordoba wrote: > > > Is there a way to give a kernel thread exclusive use > of a core? I know you can pin a kernel thread with > sched_bind(), but is there a way to keep other threads from > using the core? On an 8 core system it almost seems that the > randomness of more cores is a negative in some situations. > > > > Also, I've noticed that calling sched_bind() > during bootup is a bad thing in that it locks the system. > I'm not certain but I suspect its the thread_lock that > is the culprit. Is there a clean way to determine that its > safe to lock curthread and do a cpu bind? > > There isn't an interface to cleanly express "Use > CPUs 4-7 for only network processing". You can > configure the system this way using the cpuset command > (including directing the low-level interrupts to specific > CPUs in 8.x), but if we think this is going to be a > frequently desired policy, a bit more abstraction will be > required. > > I'm not familiar with the problem you're seeing > with sched_bind() -- I'm using it from within some of my > code without a problem, and that's fairly early in the > boot. A number of deadlocks are possible if one isn't > very careful early in the boot though, so I might look > specifically for some of those: if you migrate a thread to a > CPU that isn't yet started, it won't be able to run > until the CPU has started. This means it's important > not to migrate threads that might lead to priority > version-like deadlocks: > > - Be careful not to migrate threads that hold locks the > system requires to get > to the point where multiple CPUs run. > - Be careful not to migrate threads that will signal a > resource being > available, such as a device driver, required to get to > the point where > multiple CPUs run. > - Be careful not to migrate the main boot thread. > > Could you be running into one of those cases? Usually > they're fairly easy to diagnose using DDB, if you can > get into it, because you can see what the main boot thread > is waiting for, and reason about what's holding it. Are > you able to get into DDB when this occurs? (Perhaps using > an NMI?) Yes, the cpus are launched quite late, so that must be it. I guess the mp_ncpus is set before they are launched. Is there a way to determine that a specific core has been lauched? Regarding using cpuset, John B indicated that you couldn't allocate "sets" for kernel threads; and that sched_bind() was the only function available. So that brings 2 questions: 1) How do you get the thread ID for a process from user space to use with cpuset? I don't see that ps displays it. 2) Can cpu sets be manipulated / setup from within the kernel? Barney