From owner-freebsd-rc@freebsd.org Sun Oct 18 15:06:39 2015 Return-Path: Delivered-To: freebsd-rc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53353A18F74 for ; Sun, 18 Oct 2015 15:06:39 +0000 (UTC) (envelope-from jmaloney@pcbsd.org) Received: from barracuda.ixsystems.com (mail.ixsystems.com [69.198.165.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 312EDE56 for ; Sun, 18 Oct 2015 15:06:38 +0000 (UTC) (envelope-from jmaloney@pcbsd.org) X-ASG-Debug-ID: 1445180796-08ca040e8500c30002-WHotAJ Received: from [10.0.1.52] (ip72-209-160-49.ks.ks.cox.net [72.209.160.49]) by barracuda.ixsystems.com with ESMTP id F2olhriWYHpLoehM (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 18 Oct 2015 08:06:36 -0700 (PDT) X-Barracuda-Envelope-From: jmaloney@pcbsd.org X-Barracuda-AUTH-User: jmaloney@pcbsd.org X-Barracuda-Apparent-Source-IP: 72.209.160.49 Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3094\)) Subject: Re: rc(8) parallel tasks From: Joe Maloney X-ASG-Orig-Subj: Re: rc(8) parallel tasks In-Reply-To: <562143E7.3030104@jet9.net> Date: Sun, 18 Oct 2015 10:06:35 -0500 Cc: Mark Felder , freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org, marck@rinet.ru Message-Id: <6D32B114-453C-4D25-8FF4-C8777D78C50C@pcbsd.org> References: <560EAC05.6050308@jet9.net> <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> <562143E7.3030104@jet9.net> To: Cyril Vechera X-Mailer: Apple Mail (2.3094) X-Barracuda-Connect: ip72-209-160-49.ks.ks.cox.net[72.209.160.49] X-Barracuda-Start-Time: 1445180796 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://10.2.0.41:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=8.0 tests=HTML_MESSAGE X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.23602 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 HTML_MESSAGE BODY: HTML included in message Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-rc@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion related to /etc/rc.d design and implementation." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2015 15:06:39 -0000 Thanks for sharing this. It worked great for me on FreeBSD as well with = minimal services, and does indeed make a difference. It blew up on = PCBSD which has many more services running out of box. I also noticed = /tmp/init has to be removed manually in some cases when boot hangs. = Very impressive otherwise. Joe Maloney > On Oct 16, 2015, at 1:37 PM, Cyril Vechera wrote: >=20 > On 10/16/2015 07:06 PM, Mark Felder wrote: >> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >>> Hi there. >>>=20 >>> We've got a small launcher script (~250 loc) for parallel services >>> start/stop etc. It is used on our embedded systems and our users >>> containers. And I've done a proof of concept for implanting it to = the >>> FreeBSD's standard /etc/rc for execution starting scripts in = parallel. >>> It gave me a boot time reduction of rc part from 27 to 7 seconds, = mostly >>> on eliminating jams for network or other long-latency resources = waiting. >>>=20 >>> The launcher is written in pure POSIX shell and uses FIFOs (named = pipes) >>> as a mutexes for synchronization. So it is embedded into /etc/rc and >>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, = it >>> guarantees topological order (strict partial order) defined by >>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >>> mkfifo and a writeable file system. Due to last requirement, it can = be >>> run on the late stage or should be supplied by some kinf of = writtable >>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to = change >>> rc.d scripts >>>=20 >>> It's not a full init replacement or a kind of services supervision = tool. >>> It only starts or invokes a group of scripts in parallel with = resolving >>> and assuring execution in dependencies order. >>>=20 >>> Please take a look at the script and patch set for FreeBSD: >>>=20 >>> = https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-ini= t >>> = https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >>>=20 >>>=20 >> Your first link is a 404, but this looks really nice! >=20 > In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to = 'jet9-multitask-flow' to avoid confusion with naming, because it's not = as it seems, from name, an init replacement, but just a parallel task = launcher. And now the actual repository URL is = https://github.com/cvss/jet9-multitask-flow = >=20 > In this commit I've complete the FreeBSD compatibility. Now a script = or dependency name can include minuses `-` and dots `.` (the first stone = I stumbled over was ftp-proxy). And I've cleaned up the main script = code https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow = and = have split it to more functions that can be redefined. So it's now = easier to rewrite dependency extraction for FreeBSD rc-scripts - current = implementation is too rough and takes three 'awk' runs for each = rc-script. The last is not only time loss, but as DMarck mentioned = before, using awk restricts parallel rc to be run only after FILESYSTEMS = stage is done. Maybe it would be better to add to the rcorder(8) some = new option to dump the gathered dependencies in tsort-compatible listing = and insert them directly to flow execution plan. >=20 > I've also added rc.conf variable `rc_parallel` to turn on and off = parallel execution. There's a risk to discover an incomplete dependency = annotations in some rc-scripts that earlier were masked by serial = execution. I've done some checks by enabling as much rc.conf variables = as possible to start more rc-scripts, and didn't found any error. But it = looks too good to be true and I'm afraid that it's just to poor testing. = I think if some ordering conflict will be found, it could be = worked-around with introducing a white-list for script names that must = be run only in sequentially. >=20 > So it remained first to check if it really works in different = conditions. >=20 >=20 >=20 >=20 > --=20 > Cyril Vechera >=20 > _______________________________________________ > freebsd-hackers@freebsd.org = mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers = > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org = "