From owner-freebsd-arch@FreeBSD.ORG Fri Feb 17 00:40:28 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBF6A106566C for ; Fri, 17 Feb 2012 00:40:28 +0000 (UTC) (envelope-from matthewstory@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id A798A8FC0C for ; Fri, 17 Feb 2012 00:40:28 +0000 (UTC) Received: by vcmm1 with SMTP id m1so2817117vcm.13 for ; Thu, 16 Feb 2012 16:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=m5AR1Rl0qWy8JdVbc2qkwHnuO4+94xcBaKD42RUHyTs=; b=CLZ/70PpZE9r1FUtplf5biNsHkvs/ijGt/aVE24z7/RWRzSnJfy3UbS6AvzovxJ9gx Yo/36O2Y9hiuFOCL2EFPO6mfIzRpMsLSF/bUpG+yZOTyIMRt3GNhkpPMNuuOzHS/Vtqh tiKxjqiBJq+YiiA73i5oPAFlgq0L+yh4lDJ/M= MIME-Version: 1.0 Received: by 10.220.228.73 with SMTP id jd9mr2756737vcb.58.1329437366314; Thu, 16 Feb 2012 16:09:26 -0800 (PST) Received: by 10.52.21.84 with HTTP; Thu, 16 Feb 2012 16:09:26 -0800 (PST) Date: Thu, 16 Feb 2012 19:09:26 -0500 Message-ID: From: Matthew Story To: freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Change to xargs to avoid orphaning of utility processes on signal|exit 255 from child X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Feb 2012 00:40:29 -0000 Apologies if this is the wrong list, I would like to submit a patch that changes the behavior of xargs(1) on signal to child utility process or child utility process exiting 255. The patch(es) is|are available here: http://axe0.blackskyresearch.net/patches/matt/xargs.no_orphan.patch.txt-- this version will apply to current xargs, and adds diagnostic information for exit 255|signal to utility, as required by POSIX (see PR165155). http://axe0.blackskyresearch.net/patches/matt/xargs.no_orphan.PR165155.patch.txt-- this version will apply on top of the patch in PR165155, as the errx calls in that patch need to be modified to warnx calls. I will happily provide a third option without diagnostics if that is useful independent of PR165155. Currently, if a child exits 255, or is terminated via signal, the xargs process exits immediately with exit status 1, with maxprocs > 1, this orphans all outstanding child processes: $ jot - 1 10 | time xargs -P2 -n1 sh -c 'echo "$1: $$"; sleep $1; [ $1 -eq 1 ] && kill $$; echo "$1: $$:done";' worker 1: 69752 2: 69753 1.00 real 0.00 user 0.00 sys $ 2: 69753:done $ # or ... $ jot - 1 10 | time xargs -P2 -n1 sh -c 'echo "$1: $$"; sleep $1; [ $1 -eq 1 ] && exit 255; echo "$1: $$:done";' worker 1: 70379 2: 70380 1.00 real 0.00 user 0.00 sys $ 2: 70380:done This behavior is expected per xargs(1): The xargs utility exits immediately (without processing any further input) if a command line cannot be assembled, utility cannot be invoked, an invocation of utility is terminated by a signal, or an invocation of utility exits with a value of 255. Per the POSIX specification, xargs is only required to stop processing input, and to exit 1-125, it is not exit immediately (utility here is the utility invoked by xargs, not xargs itself): If a command line meeting the specified requirements cannot be assembled, the utility cannot be invoked, an invocation of the utility is terminated by a signal, or an invocation of the utility exits with exit status 255, the *xargs* utility shall write a diagnostic message and exit without processing any remaining input. The patch preserves orphaning when the xargs process itself is terminated by signal, but augments the behavior when a child utility process is terminated by signal or exits 255 to wait for other existing child utilities until exiting 1. My reasoning for this (beyond orphaning nastiness) is that I always want to fail as soon as I know an operation is fatal, and then clean-up. By orphaning children, there is no reliable way to clean-up following use of the 255 exit code (or signal termination): $ # a contrived example forcing a race-condition, with a clean-up function. $ mkdir -p foo; jot - 1 10 | xargs -P5 -n1 sh -c 'sleep $1; touch foo/$1; exit 255;' worker || find foo -type f -delete $ # demonstration that cleanup is not possible $ sleep 5 && ls -l foo total 2 -rw-r--r-- 1 matt matt 0 Feb 16 19:01 2 -rw-r--r-- 1 matt matt 0 Feb 16 19:01 3 -rw-r--r-- 1 matt matt 0 Feb 16 19:01 4 -rw-r--r-- 1 matt matt 0 Feb 16 19:01 5 Following the patch, we get a nice and reliable cleanup, as we have no orphans: $ mkdir -p foo; jot - 1 10 | usr.bin/xargs/xargs -P5 -n1 sh -c 'sleep $1; touch foo/$1; exit 255;' worker || find foo -type f -delete xargs: sh: exited with status 255, aborting xargs: sh: exited with status 255, aborting xargs: sh: exited with status 255, aborting xargs: sh: exited with status 255, aborting xargs: sh: exited with status 255, aborting $ ls -l foo/ total 0 Please let me know what you think, I would very much like to see this patch make it's way into xargs as I find this short-circuit behavior nearly very usable, and it's only failing is that it orphans children unnecessarily, resulting in unpredictable behavior. -- regards, matt