From owner-freebsd-rc@freebsd.org Fri Oct 16 22:14:10 2015 Return-Path: Delivered-To: freebsd-rc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84DE1A16037; Fri, 16 Oct 2015 22:14:10 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EFA5ED5; Fri, 16 Oct 2015 22:14:09 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3nd1wY64LJzKm; Sat, 17 Oct 2015 00:14:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1445033642; x=1447625643; bh=zmt pQaUSTtZbXPjXmo0pGW/UaQGoOZdU2ncQOebgH+I=; b=hgnuVkSnLhPM/pd52qZ OeCwWdHT5SMLYkvrKw3+20NDI5n0wljRlnerVsHSGfIuQUXOw9HwgEWbGThle6Ae uUA1/ZS/dNEKObEHQF3phBKnMfAyOtre4BS/j1Xq1wIdKhcWvMO7lrCV50qJFaCt n7ZNGSaKWG6DCyUIN7yLukjc= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id uyPcUiMYkl6h; Sat, 17 Oct 2015 00:14:02 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 3nd1wT2pw2zKl; Sat, 17 Oct 2015 00:14:01 +0200 (CEST) Received: from nabiralnik.ijs.si (nabiralnik.ijs.si [IPv6:2001:1470:ff80::80:16]) by mildred.ijs.si (Postfix) with ESMTP id 3nd1wT1Jr7zwf; Sat, 17 Oct 2015 00:14:01 +0200 (CEST) Received: from sleepy.ijs.si (2001:1470:ff80:e001::1:1) by nabiralnik.ijs.si with HTTP (HTTP/1.1 POST); Sat, 17 Oct 2015 00:14:01 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 17 Oct 2015 00:14:01 +0200 From: Mark Martinec To: freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Cc: Cyril Vechera Subject: Re: rc(8) parallel tasks Organization: Jozef Stefan Institute In-Reply-To: <562143E7.3030104@jet9.net> References: <560EAC05.6050308@jet9.net> <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> <562143E7.3030104@jet9.net> Message-ID: <2c307d32ddd8ed582ac496d56a75c3e4@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.1.3 X-BeenThere: freebsd-rc@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion related to /etc/rc.d design and implementation." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 22:14:10 -0000 Tried a today's git checkout on one of our small virtual 10.2 boxes with not many services. The installation is almost trivial and a reboot went without any problems. Time for a reboot was shrunk by about 10 seconds (which is understandable with a small number of services). Very good, almost too good to be true for such fairly small and non-intrusive piece of shell/awk scripting - thanks! Mark 2015-10-16 20:37, Cyril Vechera wrote: > On 10/16/2015 07:06 PM, Mark Felder wrote: >> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >>> Hi there. >>> >>> We've got a small launcher script (~250 loc) for parallel services >>> start/stop etc. It is used on our embedded systems and our users >>> containers. And I've done a proof of concept for implanting it to the >>> FreeBSD's standard /etc/rc for execution starting scripts in >>> parallel. >>> It gave me a boot time reduction of rc part from 27 to 7 seconds, >>> mostly >>> on eliminating jams for network or other long-latency resources >>> waiting. >>> >>> The launcher is written in pure POSIX shell and uses FIFOs (named >>> pipes) >>> as a mutexes for synchronization. So it is embedded into /etc/rc and >>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >>> guarantees topological order (strict partial order) defined by >>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >>> mkfifo and a writeable file system. Due to last requirement, it can >>> be >>> run on the late stage or should be supplied by some kinf of writtable >>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to >>> change >>> rc.d scripts >>> >>> It's not a full init replacement or a kind of services supervision >>> tool. >>> It only starts or invokes a group of scripts in parallel with >>> resolving >>> and assuring execution in dependencies order. >>> >>> Please take a look at the script and patch set for FreeBSD: >>> >>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init >>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >>> >>> >> Your first link is a 404, but this looks really nice! > > In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to > 'jet9-multitask-flow' to avoid confusion with naming, because it's not > as it seems, from name, an init replacement, but just a parallel task > launcher. And now the actual repository URL is > https://github.com/cvss/jet9-multitask-flow > > In this commit I've complete the FreeBSD compatibility. Now a script > or dependency name can include minuses `-` and dots `.` (the first > stone I stumbled over was ftp-proxy). And I've cleaned up the main > script code > https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and > have split it to more functions that can be redefined. So it's now > easier to rewrite dependency extraction for FreeBSD rc-scripts - > current implementation is too rough and takes three 'awk' runs for > each rc-script. The last is not only time loss, but as DMarck > mentioned before, using awk restricts parallel rc to be run only after > FILESYSTEMS stage is done. Maybe it would be better to add to the > rcorder(8) some new option to dump the gathered dependencies in > tsort-compatible listing and insert them directly to flow execution > plan. > > I've also added rc.conf variable `rc_parallel` to turn on and off > parallel execution. There's a risk to discover an incomplete > dependency annotations in some rc-scripts that earlier were masked by > serial execution. I've done some checks by enabling as much rc.conf > variables as possible to start more rc-scripts, and didn't found any > error. But it looks too good to be true and I'm afraid that it's just > to poor testing. I think if some ordering conflict will be found, it > could be worked-around with introducing a white-list for script names > that must be run only in sequentially. > > So it remained first to check if it really works in different > conditions.