From owner-freebsd-current@FreeBSD.ORG Fri May 22 18:05:58 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1F57C3B9; Fri, 22 May 2015 18:05:58 +0000 (UTC) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B921C1746; Fri, 22 May 2015 18:05:57 +0000 (UTC) Received: by wghq2 with SMTP id q2so24854451wgh.1; Fri, 22 May 2015 11:05:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=2T87+8ciq/y64VGhl7m1fCi7Fo4OM3rhquBT+SFsnKQ=; b=tHjsE5zK4xweu2fgXAg+JtTLgKj6dyk77z3BaieXdf9gzml3n/es5+IA6TgWg9aeRZ Z61VeARiA8VVGTYtPl55CD1MVIrosZsCT5Dldx3nhCi84dtUS8rQ2YXfv61oxLWyhGce gurtPWSP7HXSuzL29Is/nkR8gmt6xoGKvSLraLnEA0qy7Ab1e0rFxISj/LdOYPBOCaTT dTlQ/rB52/qpx69IOFwAam7lnS9WoEDpovyhySQpoSOTH2nTO9VWNSLTeM9XgWau5kxd AzBxFL+enXoG/c7rtuyZmmMttDdGWHynFZWZ3RMq6f3YdVr/vBi+TLMI2k0AvCf3lalP tiAA== X-Received: by 10.180.24.65 with SMTP id s1mr9651854wif.66.1432317956073; Fri, 22 May 2015 11:05:56 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id x3sm8782365wiy.20.2015.05.22.11.05.54 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Fri, 22 May 2015 11:05:55 -0700 (PDT) Date: Fri, 22 May 2015 20:05:52 +0200 From: Mateusz Guzik To: Nikolai Lifanov Cc: Allan Jude , freebsd-current@freebsd.org Subject: Re: xargs -P0 suport Message-ID: <20150522180552.GB15102@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Nikolai Lifanov , Allan Jude , freebsd-current@freebsd.org References: <555EA1C0.8010909@freebsd.org> <555F4BB9.1020001@mail.lifanov.com> <555F5A34.3090907@freebsd.org> <20150522172711.GA15102@dft-labs.eu> <555F6AB9.1040401@mail.lifanov.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <555F6AB9.1040401@mail.lifanov.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 18:05:58 -0000 On Fri, May 22, 2015 at 01:43:21PM -0400, Nikolai Lifanov wrote: > On 05/22/15 13:27, Mateusz Guzik wrote: > > On Fri, May 22, 2015 at 12:32:52PM -0400, Allan Jude wrote: > >> There is some question about if nargs is a sane value for maxprocs in > >> the negative case. 5000 does seem a bit high, and the behaviour can get > >> wonky depending on the order you specify -P and -n together on the > >> command line. > >> > >> Any suggestions? > >> > > > > GNU xargs imposes no limit whatsoever, but it also supports reallocating > > its process table, while our xargs allocates one upfront and does not > > change it. > > > > I would say reading hard proc resource limit and using that as the limit > > would do the job just fine. > > > > GNU xargs uses MAX_INT for this limit. Our xargs performs much worse > with it for a reason I haven't investigated. The 5000 number doesn't > seem high and I have workflows that do '.... | xargs -n1 -P0 ...' > spawning about this many jobs. > Strictly speaking MAX_INT is indeed the upper limit, but the number is so big it's not a limit in practice and it's not going to be in foreseeable future. As noted earlier our xargs allocates the table upfront, which with MAX_INT limit means several MBs allocated for no good reason. For all practical purposes grabbing hard limit for processes and capping it with pid_max will have the end result of xargs not limiting the amount of processes. -- Mateusz Guzik