From owner-freebsd-arch@FreeBSD.ORG Fri Nov 21 19:59:09 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B6781065677; Fri, 21 Nov 2008 19:59:09 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id D795E8FC14; Fri, 21 Nov 2008 19:59:08 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.3/8.14.3) with ESMTP id mALJwuri081978; Fri, 21 Nov 2008 14:59:02 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Lawrence Stewart Date: Fri, 21 Nov 2008 13:48:41 -0500 User-Agent: KMail/1.9.7 References: <492412E8.3060700@freebsd.org> <200811201502.23943.jhb@freebsd.org> <4925E30B.8010709@freebsd.org> In-Reply-To: <4925E30B.8010709@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200811211348.41536.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Fri, 21 Nov 2008 14:59:03 -0500 (EST) X-Virus-Scanned: ClamAV 0.93.1/8661/Fri Nov 21 10:39:30 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00,NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-arch@freebsd.org Subject: Re: kthread_exit(9) unexpectedness X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Nov 2008 19:59:09 -0000 On Thursday 20 November 2008 05:22:03 pm Lawrence Stewart wrote: > John Baldwin wrote: > > On Wednesday 19 November 2008 08:21:44 am Lawrence Stewart wrote: > >> Hi all, > >> > >> I tracked down a deadlock in some of my code today to some weird > >> behaviour in the kthread(9) KPI. The executive summary is that > >> kthread_exit() thread termination notification using wakeup() behaves as > >> expected intuitively in 8.x, but not in 7.x. > > > > In 5.x/6.x/7.x kthreads are still processes and it has always been a wakeup on > > the proc pointer. kthread_create() in 7.x returns a proc pointer, not a > > thread pointer for example. In 8.x kthreads are actual threads and > > Yep, but the processes have a *thread in them right? The API naming is > obviously slightly misleading, but it essentially creates a new single > threaded process prior to 8.x. Yes, but you have to go explicitly use FIRST_THREAD_IN_PROC(). Most of the kernel modules I've written that use kthread's in < 8 do this: static struct proc *foo_thread; /* Called for MOD_LOAD. */ static void load(...) { error = kthread_create(..., &foo_thread); } static void unload(...) { /* set flag */ msleep(foo_thread, ...); } And never actually use the thread at all. However, if you write the code for 8.x, now you _do_ get a kthread and sleep on the thread so it becomes: static struct thread *foo_thread; static void load(...) { error = kproc_kthread_add(..., proc0, &foo_thread); } static void unload(...) { /* set flag */ msleep(foo_thread, ...); } > > kthread_add() and kproc_kthread_add() both return thread pointers. Hence in > > Yup. > > > 8.x kthread_exit() is used for exiting kernel threads and wakes up the thread > > pointer, but in 7.x kthread_exit() is used for exiting kernel processes and > > wakes up the proc pointer. I think what is probably needed is to simply > > In the code, yes. Our documented behaviour as far as I can tell is > different though, unless we equate a "thread handle" to "proc handle" > prior to 8.x, which I don't think is the case - they are still different. It has always been the case in < 8 that you sleep on the proc handle (what kthread_create() actually returns in < 8). And in fact, you even have to dig around in the proc you get from kthread_create() to even find the thread pointer as opposed to having the API hand it to you. > > document that arrangement as such. Note that the sleeping on proc pointer > > I agree that the arrangement needs to be better documented. The change > in 8.x is subtle enough that reading the kthread man page in 7.x and 8.x > doesn't immediately make it obvious what's going on. > > > has been the documented way to synchronize with kthread_exit() since 5.0. > > > > Could you please point me at this documentation? I've missed it in my > poking around thus far. It is probably only documented in numerous threads in the mail archives sadly, but there have been several of them and there have been several fixes to get this right (the randomdev thread and fdc(4) thread come to mind). -- John Baldwin