From owner-freebsd-current@FreeBSD.ORG Thu Sep 18 21:03:48 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97B331065671 for ; Thu, 18 Sep 2008 21:03:48 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id 37C0D8FC0C for ; Thu, 18 Sep 2008 21:03:47 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.3/8.14.3/NETPLEX) with ESMTP id m8IL3kfO019172; Thu, 18 Sep 2008 17:03:46 -0400 (EDT) X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.0 (mail.netplex.net [204.213.176.10]); Thu, 18 Sep 2008 17:03:46 -0400 (EDT) Date: Thu, 18 Sep 2008 17:03:46 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: David Naylor In-Reply-To: <200809181927.52384.naylor.b.david@gmail.com> Message-ID: References: <200809180631.47071.naylor.b.david@gmail.com> <200809181857.25872.naylor.b.david@gmail.com> <200809181927.52384.naylor.b.david@gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Tom Evans , freebsd-current@freebsd.org Subject: Re: FreeBSD deadlock (with fork?) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Sep 2008 21:03:48 -0000 On Thu, 18 Sep 2008, David Naylor wrote: > On Thursday 18 September 2008 19:13:19 Daniel Eischen wrote: >> On Thu, 18 Sep 2008, David Naylor wrote: >>> On Thursday 18 September 2008 15:33:52 Tom Evans wrote: >>>> On Thu, 2008-09-18 at 06:31 +0200, David Naylor wrote: >>>>> Hi, >>>>> >>>>> I have a program that spawns a lot of subprocesses (with pipes open) >>>>> from multiple threads. The problem is the program often deadlocks, but >>>>> not consistently. Sometimes the program can run over 5 times to >>>>> competition without incidence and yet othertimes it locks within a few >>>>> seconds. >>>> >>>> Do you create threads, which then fork(), or do you fork() and then >>>> create threads? >>> >>> I have many threads that then fork. (aka forking threads). >>> >>>> I think the former will not work.. >>> >>> Is there any reason for this and is this a FreeBSD limitation or a >>> general problem? >> >> No, the former will work. You can fork() from threads, only as long >> as your forked processes only call async-signal-safe functions or >> some form of exec(). For instance, you can't fork and then create >> new threads from that child process. If you are not immediately >> exec()'ing from the child process, then you have to be careful >> and only use async-signal-safe functions. >> >> Remember that you have multiple threads in the parent, so the >> state of libthr, libc, etc may be inconsistent in a child >> process. > > If you have a look at the backtrace from my original post you will see that > all my threads are getting locked. The program I am using is python and the > call that is forking is Popen(['some', 'program'], stdout=PIPE, > stderr=STDOUT). As far as I know it just sets up the pipes and does an > execvp. (See > http://svn.python.org/view/*checkout*/python/tags/r252/Lib/subprocess.py?content-type=text%2Fplain&rev=60915 > from line 981 to 1091). [[I tried appending close_fds=True to the argument, > it appeared to make the program deadlock less often...]] > > Any ideas as to what might be causing this dead lock (and why all the threads > are stopping in amd64 specific assembler)? No, could be a bug in python though. -- DE