From owner-freebsd-hackers@FreeBSD.ORG Thu Jan 22 05:42:59 2009 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DFF71106564A; Thu, 22 Jan 2009 05:42:59 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id AAEB88FC16; Thu, 22 Jan 2009 05:42:59 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.3/8.14.3/NETPLEX) with ESMTP id n0M5gu8u021756; Thu, 22 Jan 2009 00:42:56 -0500 (EST) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.0 (mail.netplex.net [204.213.176.10]); Thu, 22 Jan 2009 00:42:57 -0500 (EST) Date: Thu, 22 Jan 2009 00:42:56 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Schultz In-Reply-To: <20090122045637.GA61058@zim.MIT.EDU> Message-ID: References: <20090109053117.GB2825@green.homeunix.org> <4966F81C.3070406@elischer.org> <20090109163426.GC2825@green.homeunix.org> <49678BBC.8050306@elischer.org> <20090116211959.GA12007@green.homeunix.org> <49710BD6.7040705@FreeBSD.org> <20090120004135.GB12007@green.homeunix.org> <20090121230033.GC12007@green.homeunix.org> <20090122045637.GA61058@zim.MIT.EDU> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: hackers@freebsd.org, Jason Evans , Julian Elischer Subject: Re: threaded, forked, rethreaded processes will deadlock X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jan 2009 05:43:00 -0000 On Wed, 21 Jan 2009, David Schultz wrote: > I think there *is* a real bug here, but there's two distinct ways > to fix it. When a threaded process forks, malloc acquires all its > locks so that its state is consistent after a fork. However, the > post-fork hook that's supposed to release these locks fails to do > so in the child because the child process isn't threaded, and > malloc_mutex_unlock() is optimized to be a no-op in > single-threaded processes. If the child *stays* single-threaded, > malloc() works by accident even with all the locks held because > malloc_mutex_lock() is also a no-op in single-threaded processes. > But if the child goes multi-threaded, then things break. > > Solution 1 is to actually unlock the locks in the child process, > which is what Brian is proposing. > > Solution 2 is to take the position that all of this pre- and > post-fork bloat in the fork() path is gratuitous and should be > removed. The rationale here is that if you fork with multiple > running threads, there's scads of ways in which the child's heap > could be inconsistent; fork hooks would be needed not just in > malloc(), but in stdio, third party libraries, etc. Why should > malloc() be special? It's the programmer's job to quiesce all the > threads before calling fork(), and if the programmer doesn't do > this, then POSIX only guarantees that async-signal-safe functions > will work. > > Note that Solution 2 also fixes Brian's problem if he quiesces all > of his worker threads before forking (as he should!) With the > pre-fork hook removed, all the locks will start out free in the > child. So that's what I vote for... The problem is that our own libraries (libthr included) need to malloc() for themselves, even after a fork() in the child. After a fork(), the malloc locks should be reinitialized in the child if it was threaded, so that our implementation actually works for all the async signal calls, fork(), exec(), etc. I forget the exact failure modes for very common cases, but if you remove the re-initialization of the malloc locks, I'm sure you will have problems. Perhaps much of this malloc() stuff goes away when we move to pthread locks that are not pointers to allocated objects, but instead are actual objects/structures. This needs to be done in order for mutexes/CVs/etc to be PTHREAD_PROCESS_SHARED (placed in shared memory and used by multiple processes). In other words, pthread_mutex_t goes from this: typedef struct pthread_mutex *pthread_mutex_t; to something like this: struct __pthread_mutex { uint32_t lock; ... } typedef struct __pthread_mutex pthread_mutex_t; Same thing for CVs, and we probably should convert any other locks used internally by libc/libpthread (spinlocks). So after a fork(), there is no need to reallocate anything, it can just be reinitialized if necessary. -- DE