From owner-freebsd-current@FreeBSD.ORG Wed Feb 2 15:06:59 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DCDA1065674 for ; Wed, 2 Feb 2011 15:06:59 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id E5C898FC1D for ; Wed, 2 Feb 2011 15:06:58 +0000 (UTC) Received: by qwj9 with SMTP id 9so62862qwj.13 for ; Wed, 02 Feb 2011 07:06:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=AHgKcHHF3uQgefreV1v9lddfFcrmCpliM13zn614JBc=; b=fv3DE2FKiMz2elgsx0dUA/EUdlIwlYlvgdWWIJXZyFtFN6bhfn91kkmkTLMk/HIpzI q6SJgHKxrrNsIhFg7C/s1FJ71wc5P1/XIK7uNpgVok1p8mIgNzxGdqMKvGJBE+j4UItu FUv8JjZ6404J4PpMgbg8dbYkvB9r2rgN4YWz4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=nQ3qo4hjVIcRPsxHAqqFddE6s6HyK02yvAGlq/w4R3jjRSzUB5n734mu0sY9PlYplO 3seJO4WLLAtvR38ECOxVb7/5fXqBtgcWMHL6OXwRJYJ2N45iv9Ll0WDqHJMKlZ1TAtOQ M3Th2IlSwx4vsmQeN9K2aKtEXU56DG7rBqA1o= MIME-Version: 1.0 Received: by 10.224.176.202 with SMTP id bf10mr8732710qab.236.1296659215807; Wed, 02 Feb 2011 07:06:55 -0800 (PST) Received: by 10.229.102.87 with HTTP; Wed, 2 Feb 2011 07:06:55 -0800 (PST) In-Reply-To: <20110128121412.GA10532@stack.nl> References: <20110128121412.GA10532@stack.nl> Date: Wed, 2 Feb 2011 18:06:55 +0300 Message-ID: From: Sergey Kandaurov To: Jilles Tjoelker Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Current Subject: Re: unbounded sleep on [fifoow] while open a named pipe: is it a feature? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Feb 2011 15:06:59 -0000 On 28 January 2011 15:14, Jilles Tjoelker wrote: > On Fri, Jan 28, 2011 at 12:43:38PM +0300, Sergey Kandaurov wrote: >> That's FreeBSD 8.1-RELEASE i386 (w/o debug). > >> It's observed for bash processes which end up in unbounded sleep >> in [fifoow] wchan while executing the next script: > >> %%% >> #!/usr/local/bin/bash > >> LOG_FACILITY=3D"local7.notice" >> LOG_TOPIC_OUT=3D"$TAG-output[$$]" >> LOG_TOPIC_ERR=3D"$TAG-error[$$]" > >> cd $WORK_DIR > >> exec =A0> >(logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_OUT" ) >> exec 2> >(logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_ERR" ) >> eval exec $1 >> %%% > >> It's used to call $1 and redirect its stdout and stderr streams >> to different files. Bash implements this functionality by creating >> on every execution a new named pipe stored at /var/tmp/ directory. >> This script is used to run periodically from cron(8). > >> Note, that bash has a misfeature to not delete those named pipes. > > I think that is more to do with the 'exec'. Bash will delete process > substitution fifos when the process using them is done, but 'exec' will > prevent this. Yes, indeed. > > The original use case for process substitution was > =A0program <(command) > and it may not be known when 'program' opens the fifo so the fifo needs > to be kept around until 'program' is done. > >> With a high creation frequency of named pipes (several per minute) it >> eventually ends up with a large number [~137000] of existing named >> pipes, when it was found that a number [~44] of bash processes sleep >> on [fifoow], which is guessed to be likely caused by the number of >> fifo pipes created. > >> For example, this one sleeps for about 1.5 days: >> =A01001 78809 78803 =A0 0 =A050 =A00 =A04564 =A01952 fifoow Is =A0 =A0??= =A0 =A00:00.00 >> /usr/local/bin/bash ./logger_wrapper.sh ./sync_overlords.pl > >> # procstat -f 78809 >> [...] >> 78809 bash =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A01 f - -w------ =A0 1 =A0 =A0 = =A0 0 - =A0 /var/tmp/sh-np-[random] >> [...] > > If a process gets "stuck" in [fifoow], it means that the process that is > supposed to read from the fifo fails to do so. > > This may be because multiple use of the same pathname -- bash's random > name generation algorithm is not very good. > > How about something like this instead of the two exec commands with > process substitution (untested): > > tempdir=3D$(mktemp -d /tmp/logtmp.XXXXXXXXXX) || exit > fifo1=3D$tempdir/stdout.fifo > fifo2=3D$tempdir/stderr.fifo > mkfifo "$fifo1" "$fifo2" || exit > logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_OUT" <"$fifo1" & > logger -p "$LOG_FACILITY" -t "$LOG_TOPIC_ERR" <"$fifo2" & > exec >"$fifo1" 2>"$fifo2" > rm -rf "$tempdir" > > Once the fifos have been opened for writing, the readers must have > already opened too, therefore it is safe to unlink them. We have switched today to this suitable solution (with minor change). So far, so good. Thank you very much. --=20 wbr, pluknet