From owner-freebsd-hackers@freebsd.org Sun Jan 10 10:46:01 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7126A6A7B5 for ; Sun, 10 Jan 2016 10:46:01 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from mail.rdsor.ro (mail.rdsor.ro [193.231.238.10]) by mx1.freebsd.org (Postfix) with ESMTP id D452D1440 for ; Sun, 10 Jan 2016 10:46:00 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from [192.168.1.101] (unknown [79.117.100.196]) by mail.rdsor.ro (Postfix) with ESMTP id BC11FDB1A; Sun, 10 Jan 2016 12:36:44 +0200 (EET) Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: relaunchd: a portable clone of launchd From: Dan Partelly In-Reply-To: Date: Sun, 10 Jan 2016 12:36:44 +0200 Cc: Dmitry Sivachenko , FreeBSD Hackers , Jonathan de Boyne Pollard , Mark Heily Message-Id: <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> To: Peter Beckman X-Mailer: Apple Mail (2.3112) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 10:46:01 -0000 Copying the linux way of doing things should not be a goal of the BSDs. = It is enough that unfortunately we are forced into Linuxisms and associated wrappers to support modern GPUs. = Understandable , given how few ppl work on BSDs, and how little code contributions do the BSDs receive from the massive = enterprises they power (with=20 some notable exceptions) Let me use this opportunity to thank Juniper = for the glorified printf system=20 they contributed to FreeBSD .=20 Can the BSDs go forward with rc systems alone ? Sure they can , at least = for the time beeing, and we will=20 continue to use them. But innovation is desirable.=20 Systemd might be a terrible implementation or not (I dont know, I dont = use it) but the ideas behind it are sane.=20 rc systems are indeed robust, but they should be ancient history. They = are nothing but glorified autoexec.bat systems. Modern OSes need sophisticated dynamic service management systems, fault = management, transactional OS configuration databases, centralised event systems supporting kernel = sources.=20 This is the problem domain things like sytemd and dbus are tring to = solve. They might doit the wrong way, I personally think the direction Solaris took to solve some of those problems is the way to = go, but at least they are trying to do something, and=20 they clearly explored the problem space. Meanwhile here, some engineers are trying to change the FreeBSD OS = configuration to a new DSL, but without any consideration for issues of centralising OS databases and add innovation like transactions = and full concurrency safety.=20 YOu gotta ask yourself, since it is only a language change, why even = doit ????? It adds no technical innovations, the new=20 systems are not well enough thought out to support technical = innovation added incrementally later. So why are they doing it ? To change the DSL only ? By now all BSDs user are familiar with all = adhoc databases the OS offer. We are familiar (experts, even) with=20 the language they use. Changing this language , when no technical = innovation is present, is , in my opinion, ill-advised.=20 It is change for the sake of change, it is change because =E2=80=9Csomeone= wrote the code=E2=80=9D, not because it solves any real problem , or is = a well thought out engineering solution. I really hope someone from the = developers wakes up and vetoes those changes for the sake of change, like Junipers libxo, and attemtps to change the DSLs for the sake of = changing the DSL. > On 08 Jan 2016, at 17:20, Peter Beckman wrote: >=20 > On Thu, 7 Jan 2016, Dmitry Sivachenko wrote: >=20 >>=20 >>> On 07 Jan 2016, at 05:12, Mark Heily wrote: >>>=20 >>> On Sat, Jan 2, 2016 at 8:42 AM, Jonathan de Boyne Pollard >>> wrote: >>>>=20 >>>> I recommend, to anyone going down this route, looking towards = finishing >>>> systembsd, especially instead of inventing a wholly new suite of = protocols. >>>>=20 >>>> * https://uglyman.kremlin.cc/gitweb/gitweb.cgi?p=3Dsystembsd.git >>>> * >>>> = http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/debian-systemd-p= ackaging-hoo-hah.html >>>> * https://news.ycombinator.com/item?id=3D10176275 >>>>=20 >>>> The reason is that finishing systemdbsd will make happy all of the = people >>>> who want the desktop environments whose design is driven largely by = Linux to >>>> work on FreeBSD/PC-BSD. The desktop environments that they'd like = to use >>>> have been or are being modified to work with these daemons, over = this D-Bus >>>> protocol. >>>>=20 >>>=20 >>> I strongly disagree with your recommendation to adopt DBus and = systemd >>> as core components of FreeBSD. >>>=20 >>> =46rom a practical perspective, the proposal has a low probability = of >>> success. Systemd is written for Linux and is largely driven by a >>> commercial Linux vendor. It is a rapidly moving target, with no = sense >>> of scope or boundaries. It eagerly consumes the latest and greatest >>> innovations in the Linux kernel, with open disdain for portability. >>>=20 >>> =46rom a philosophical perspective, I don't agree with the direction >>> that systemd is taking Linux. It's one of the reasons I switched to >>> BSD after many years in the Linux camp. To quote Spock, "Logic = clearly >>> dictates that the needs of the many outweigh the needs of the few". = In >>> case of FreeBSD, this means that the needs of the desktop users = should >>> not outweigh the needs of the server/jail/embedded/appliance users. = My >>> concern with systemd and DBus is that these tools are highly >>> desktop-centric, and introduce a large degree of unwanted change, >>> complexity, and risk to everyone else. >>=20 >>=20 >> I totally agree. >>=20 >> systemd is an ugly beast, solving simple problem in complex way. >>=20 >> After using FreeBSD's rc system for years, I think that switching to = something systemd-related would be huge mistake. >> No reason to clone everything that happens in Linux world. >=20 > Utterly and strongly agreed. >=20 > = --------------------------------------------------------------------------= - > Peter Beckman = Internet Guy > beckman@angryox.com = http://www.angryox.com/ > = --------------------------------------------------------------------------= - > _______________________________________________ > freebsd-hackers@freebsd.org = mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers = > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org = " From owner-freebsd-hackers@freebsd.org Sun Jan 10 12:49:15 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72B1BA5F39E for ; Sun, 10 Jan 2016 12:49:15 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from know-smtprelay-omc-4.server.virginmedia.net (know-smtprelay-omc-4.server.virginmedia.net [80.0.253.68]) by mx1.freebsd.org (Postfix) with ESMTP id CE1F214EA for ; Sun, 10 Jan 2016 12:49:14 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from [192.168.1.100] ([86.10.211.13]) by know-smtprelay-4-imp with bizsmtp id 4Co31s00Q0HtmFq01Co3MA; Sun, 10 Jan 2016 12:48:03 +0000 X-Originating-IP: [86.10.211.13] X-Spam: 0 X-Authority: v=2.1 cv=aIwN0uJm c=1 sm=1 tr=0 a=SB7hr1IvJSWWr45F2gQiKw==:117 a=SB7hr1IvJSWWr45F2gQiKw==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=NLZqzBF-AAAA:8 a=IkcTkHD0fZMA:10 a=VfMSWC024tg77r_jCnUA:9 a=QEXdDO2ut3YA:10 a=vncYq6r2dI0A:10 To: FreeBSD Hackers References: From: Jonathan de Boyne Pollard Subject: Re: relaunchd: a portable clone of launchd Message-ID: <569252F2.5050205@NTLWorld.com> Date: Sun, 10 Jan 2016 12:47:46 +0000 User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 12:49:15 -0000 Benjamin Kaduk: > A submission on this topic would be welcomed even if it came in on Saturday (the 9th). The problem is, of course, that you and I are working to the same calendar. You're taking the weekend to do stuff. I can tell by the mail that came in for my PC-BSD bug reports that some of the PC-BSD people are. So too am I. (-: I'm finishing up and testing version 1.24. From owner-freebsd-hackers@freebsd.org Sun Jan 10 16:55:49 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 021F4A6AE02 for ; Sun, 10 Jan 2016 16:55:49 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C0D9D17FD for ; Sun, 10 Jan 2016 16:55:48 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 849361534E8 for ; Sun, 10 Jan 2016 17:55:43 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id N8KNouS5-B_X; Sun, 10 Jan 2016 17:55:33 +0100 (CET) Received: from [192.168.10.10] (asus [192.168.10.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 27A671534E6 for ; Sun, 10 Jan 2016 17:55:33 +0100 (CET) To: freebsd-hackers@freebsd.org From: Willem Jan Withagen Subject: Attribute order in storing and retreiving extended attributes X-Enigmail-Draft-Status: N1110 Message-ID: <56928D06.4050500@digiware.nl> Date: Sun, 10 Jan 2016 17:55:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 16:55:49 -0000 Hi, I'm looking into why a snippet of Ceph testcode sometimes retuns error, and other times does complete oke. It seems that under Linux the way order of attributes is always consistent in the same order. Where as on FreeBSD it can happen that even with the same sequence of extattr_set_* will give a different order in the extattr_list_fd output. And because the outputbuffer is compared with memcmp an error is detected. Easy to verify with lsextattr, which also gives a different order. When things go oke the order is: # lsextattr user chain_xattr_listxatrr chain_xattr_listxatrr, size: 512 user.1111111111111111111111111111111111111111111111111111111111111111111 11111111111111111111111111111111111111111111111111111111@1 user.1111111111111111111111111111111111111111111111111111111111111111111 11111111111111111111111111111111111111111111111111111111 user.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ When thing go wrong, the order is: # lsextattr user chain_xattr_listxatrr chain_xattr_listxatrr, size: 512 user.1111111111111111111111111111111111111111111111111111111111111111111 11111111111111111111111111111111111111111111111111111111 user.1111111111111111111111111111111111111111111111111111111111111111111 11111111111111111111111111111111111111111111111111111111@1 user.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ This is on a ZFS filesystem. Is there any particular reason why these is not deterministic? I do not have a testbox with UFS available, so it is hard to verify that it occurs there too. Would this be easy to make more deterministic? --WjW From owner-freebsd-hackers@freebsd.org Sun Jan 10 17:04:17 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B652A6B24B for ; Sun, 10 Jan 2016 17:04:17 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 387EF1EEC for ; Sun, 10 Jan 2016 17:04:16 +0000 (UTC) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.55.3]) by phk.freebsd.dk (Postfix) with ESMTP id 676634F865; Sun, 10 Jan 2016 17:04:10 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.15.2/8.15.2) with ESMTP id u0AH48CY014172; Sun, 10 Jan 2016 17:04:09 GMT (envelope-from phk@phk.freebsd.dk) To: Willem Jan Withagen cc: freebsd-hackers@freebsd.org Subject: Re: Attribute order in storing and retreiving extended attributes In-reply-to: <56928D06.4050500@digiware.nl> From: "Poul-Henning Kamp" References: <56928D06.4050500@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <14170.1452445448.1@critter.freebsd.dk> Date: Sun, 10 Jan 2016 17:04:08 +0000 Message-ID: <14171.1452445448@critter.freebsd.dk> X-Mailman-Approved-At: Sun, 10 Jan 2016 17:18:08 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 17:04:17 -0000 -------- In message <56928D06.4050500@digiware.nl>, Willem Jan Withagen writes: >This is on a ZFS filesystem. >Is there any particular reason why these is not deterministic? You should not expect them to be in any particular order, no such guarantee is given by the API. Filesystem operations such as backups and restore can (also) rearrange order of extended attribute. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-hackers@freebsd.org Sun Jan 10 17:41:40 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9165A6BF8F for ; Sun, 10 Jan 2016 17:41:40 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 73D361571 for ; Sun, 10 Jan 2016 17:41:40 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 2954C15340A; Sun, 10 Jan 2016 18:41:38 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zeoIeKt8aN02; Sun, 10 Jan 2016 18:41:28 +0100 (CET) Received: from [192.168.10.10] (asus [192.168.10.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 4A95F153413; Sun, 10 Jan 2016 18:28:51 +0100 (CET) Subject: Re: Attribute order in storing and retreiving extended attributes To: Poul-Henning Kamp References: <56928D06.4050500@digiware.nl> <14171.1452445448@critter.freebsd.dk> Cc: freebsd-hackers@freebsd.org From: Willem Jan Withagen X-Enigmail-Draft-Status: N1110 Message-ID: <569294D5.6090800@digiware.nl> Date: Sun, 10 Jan 2016 18:28:53 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <14171.1452445448@critter.freebsd.dk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 17:41:40 -0000 On 10-1-2016 18:04, Poul-Henning Kamp wrote: > -------- > In message <56928D06.4050500@digiware.nl>, Willem Jan Withagen writes: > >> This is on a ZFS filesystem. >> Is there any particular reason why these is not deterministic? > > You should not expect them to be in any particular order, no such > guarantee is given by the API. > > Filesystem operations such as backups and restore can (also) rearrange > order of extended attribute. > Hi Poul-Henning, I went thru the man pages and there is indeed no such guarantee. Neither do the Linux man pages. So I'm contemplating what to do with the test. It is sort of hard to determine if the Ceph tools somewhere expect this ordering. But for the time being I think I'll just disable the test. --WjW From owner-freebsd-hackers@freebsd.org Sun Jan 10 17:46:26 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F236FA691BE for ; Sun, 10 Jan 2016 17:46:26 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from erouter6.ore.mailhop.org (erouter6.ore.mailhop.org [54.187.213.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B965D1952 for ; Sun, 10 Jan 2016 17:46:26 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from ilsoft.org (unknown [73.34.117.227]) by outbound3.ore.mailhop.org (Halon Mail Gateway) with ESMTPSA; Sun, 10 Jan 2016 17:45:07 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id u0AHkJxp008075; Sun, 10 Jan 2016 10:46:19 -0700 (MST) (envelope-from ian@freebsd.org) Message-ID: <1452447979.1523.11.camel@freebsd.org> Subject: Re: Attribute order in storing and retreiving extended attributes From: Ian Lepore To: Willem Jan Withagen , freebsd-hackers@freebsd.org Date: Sun, 10 Jan 2016 10:46:19 -0700 In-Reply-To: <56928D06.4050500@digiware.nl> References: <56928D06.4050500@digiware.nl> Content-Type: text/plain; charset="us-ascii" X-Mailer: Evolution 3.16.5 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 17:46:27 -0000 On Sun, 2016-01-10 at 17:55 +0100, Willem Jan Withagen wrote: > Hi, > > I'm looking into why a snippet of Ceph testcode sometimes retuns > error, > and other times does complete oke. > > It seems that under Linux the way order of attributes is always > consistent in the same order. > > Where as on FreeBSD it can happen that even with the same sequence of > extattr_set_* will give a different order in the extattr_list_fd > output. > And because the outputbuffer is compared with memcmp an error is > detected. > > Easy to verify with lsextattr, which also gives a different order. > > When things go oke the order is: > # lsextattr user chain_xattr_listxatrr > chain_xattr_listxatrr, size: 512 > user.1111111111111111111111111111111111111111111111111111111111111111 > 111 > 11111111111111111111111111111111111111111111111111111111@1 > user.1111111111111111111111111111111111111111111111111111111111111111 > 111 > 11111111111111111111111111111111111111111111111111111111 > user.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @@@ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @@@ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @@@ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > When thing go wrong, the order is: > # lsextattr user chain_xattr_listxatrr > chain_xattr_listxatrr, size: 512 > user.1111111111111111111111111111111111111111111111111111111111111111 > 111 > 11111111111111111111111111111111111111111111111111111111 > user.1111111111111111111111111111111111111111111111111111111111111111 > 111 > 11111111111111111111111111111111111111111111111111111111@1 > user.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @@@ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @@@ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @@@ > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > > This is on a ZFS filesystem. > Is there any particular reason why these is not deterministic? > I do not have a testbox with UFS available, so it is hard to verify > that > it occurs there too. > > Would this be easy to make more deterministic? > Could you just pipe the lsextattr output through sort to make it deterministic? -- Ian From owner-freebsd-hackers@freebsd.org Sun Jan 10 18:08:02 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4707AA69A8B for ; Sun, 10 Jan 2016 18:08:02 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (unknown [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0929B1A83; Sun, 10 Jan 2016 18:08:01 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id D95A91534E8; Sun, 10 Jan 2016 19:07:58 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nbGrJ4bUZgiZ; Sun, 10 Jan 2016 19:07:32 +0100 (CET) Received: from [192.168.10.10] (asus [192.168.10.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 206AC153416; Sun, 10 Jan 2016 19:07:32 +0100 (CET) Subject: Re: Attribute order in storing and retreiving extended attributes To: Ian Lepore , freebsd-hackers@freebsd.org References: <56928D06.4050500@digiware.nl> <1452447979.1523.11.camel@freebsd.org> From: Willem Jan Withagen Message-ID: <56929DE6.4040308@digiware.nl> Date: Sun, 10 Jan 2016 19:07:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <1452447979.1523.11.camel@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2016 18:08:02 -0000 On 10-1-2016 18:46, Ian Lepore wrote: > On Sun, 2016-01-10 at 17:55 +0100, Willem Jan Withagen wrote: >> Hi, >> >> I'm looking into why a snippet of Ceph testcode sometimes retuns >> error, >> and other times does complete oke. >> >> It seems that under Linux the way order of attributes is always >> consistent in the same order. >> >> Where as on FreeBSD it can happen that even with the same sequence of >> extattr_set_* will give a different order in the extattr_list_fd >> output. >> And because the outputbuffer is compared with memcmp an error is >> detected. >> >> Easy to verify with lsextattr, which also gives a different order. >> >> When things go oke the order is: >> # lsextattr user chain_xattr_listxatrr >> chain_xattr_listxatrr, size: 512 >> user.1111111111111111111111111111111111111111111111111111111111111111 >> 111 >> 11111111111111111111111111111111111111111111111111111111@1 >> user.1111111111111111111111111111111111111111111111111111111111111111 >> 111 >> 11111111111111111111111111111111111111111111111111111111 >> user.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> @@@ >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> @@@ >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> @@@ >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> >> When thing go wrong, the order is: >> # lsextattr user chain_xattr_listxatrr >> chain_xattr_listxatrr, size: 512 >> user.1111111111111111111111111111111111111111111111111111111111111111 >> 111 >> 11111111111111111111111111111111111111111111111111111111 >> user.1111111111111111111111111111111111111111111111111111111111111111 >> 111 >> 11111111111111111111111111111111111111111111111111111111@1 >> user.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> @@@ >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> @@@ >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> @@@ >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> >> This is on a ZFS filesystem. >> Is there any particular reason why these is not deterministic? >> I do not have a testbox with UFS available, so it is hard to verify >> that >> it occurs there too. >> >> Would this be easy to make more deterministic? >> > > Could you just pipe the lsextattr output through sort to make it > deterministic? Hi Ian, I'm just using lsextattr as a means of showing what is going on. In the Ceph code there is the expectancy that the attributes are returned in the same order the attributes are inserted. So for the time being I disabled the test, and perhaps I'm going to rewrite it into explicit test for the inserted attributes. --WjW From owner-freebsd-hackers@freebsd.org Mon Jan 11 12:25:58 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78C0EA6B6DC for ; Mon, 11 Jan 2016 12:25:58 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from mail.rdsor.ro (mail.rdsor.ro [193.231.238.10]) by mx1.freebsd.org (Postfix) with ESMTP id D166F10A4 for ; Mon, 11 Jan 2016 12:25:57 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from [192.168.1.155] (unknown [86.125.33.32]) by mail.rdsor.ro (Postfix) with ESMTP id 2FBB71F89E; Mon, 11 Jan 2016 14:25:50 +0200 (EET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: relaunchd: a portable clone of launchd From: Dan Partelly In-Reply-To: <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> Date: Mon, 11 Jan 2016 14:25:52 +0200 Cc: Peter Beckman , FreeBSD Hackers , Jonathan de Boyne Pollard , Dmitry Sivachenko , Mark Heily Content-Transfer-Encoding: quoted-printable Message-Id: References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> To: Jordan Hubbard X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2016 12:25:58 -0000 > It=E2=80=99s this latter point that makes me roll my eyes a bit = whenever folks use phrases like =E2=80=9Cthe linux way=E2=80=9D or = =E2=80=9Cthe BSD way=E2=80=9D since I=E2=80=99m not entirely sure that = those =E2=80=9Cways=E2=80=9D, at least not as I first heard them = articulated back in the 1990=E2=80=99s, actually exist anymore. I personally use those terms a bit tongue in check. BSDs have very = limited manpower , and so they are forced to use source code from = foreign OSes in the most straightforward way possible, which means = wrappers . (else see the fiasco with the DRM drivers in FreeBSD so far, = there is plainly too much effort to port Linux DRM code to BSD without = wrappers and adapters. Dragonfly did it the right way ) . So , you need = to wrappers and adapters to Linux specific API=E2=80=99s and data = structures to support DRM , OFED and god knows what else.=20 But this is a general problem not only limited to Linux.=20 FreeBSD has NDIS emulation to support some Wifi devices.=20 FreeBSD has Solaris API/ datastruct wrappers and adapters to support = ZFS/ NextBSD has now dragged =E2=80=9Chalf" of mach kernel inside to : 1. = Implement a better and more flexible IPC mechanism than Unix domain = sockets / posix message queues (Truth is,=20 IPC sucks in Linux and BSDs. It is my opinion that Windows LPC/ALPC are = way better designed. and 2: To easily port launchd/notifyd/whateverd=20 Now this is all good and dandy, but who will audit all this code for = bugs and security ? Some bugs will become manifest fast, but others may = lurk in those layers for years. > What problems are we here to solve, and what are the specific details = of those problems?=E2=80=9D It is my oppinion that FreeBSD needs to solve the following techical = problems in the future: =E2=80=94 problem : binary code reuse --- 1. Many utilites from BSD base are monolithic and closed, yet they = provide higher level interfaces to OS configuration. It is my opinion = that=20 all this functionality should be expose as libraries & demon services. = This is the very base upon which enterprise grade tools are built. = Libxo is not a proper solution to this problem. It should not exist in = FreeBSD base, yet somehow it slipped inside, and I seen on some papers = from BSD conferences that some even discussed to put it inside the OS = kernel . I frankly cant imagine what those ppl are/wehre thinking.=20 2. Of secondary importance is to build some demons to allow access to = geom, network , wifi management and so on and implement a proper auto = mounter . Let go to the 80s=20 and not use scaffolding of shell scripts to implement any of those.=20 Those two points also help a great deal towards having softtware = appliances which has only what you need and nothing else on them.=20 =E2=80=94 problem OS configuration and updating in a higly concurrent = world --- 3. Transactional OS wide configuration databases.=20 4. In a world where hundreds of machines, bee-it physical or virtual are = interconnected and talking to each other , safe concurrent access to OS = configuration will become soon very important. =E2=80=94 problem OS wide IPC mechanism to build a high performance = pub/sub system , to cope with the very dynamic world we move forward.=20 4. A new high performance IPC system should be introduced IMO. It should = allow both fast synchronous operation ( like solaris doors) and = asynchronous operations, working with kevent/kqueue. It should allow = kernel endpoints, and should cater to security concerns. It should be a = minimal API , not a full buss 5. a user mode pub / sub message bus. use the IPC api to implement a = high performance , OS wide, message bus. It should present clear = abstractions to clients, to insulate them=20 from using syscalls directly. It should sum some already existing = mechanisms, such as devd. =E2=80=94 problem of service management and fault management: 6. Much needed components of enterprise.=20 =20 Frankly, I like the direction Solaris took with Service management and = faults management.=20 > What I DO know won=E2=80=99t move the ball forward (on any field, in = any game) is arguing about this like it was a religious debate of some = sort, where sweeping generalizations are preferred over far more = pragmatic questions of =E2=80=9CWhat problems are we here to solve, and = what are the specific details of those problems?=E2=80=9D I raise some of those problems in past. Especially the problem of binary = code reuse . My perception was that nobody really gives a damn. The = answers where interesting: libxo is in base because someone coded it , = libxofication of utils proceeds in BSD , no matter that is a half assed = solution which offers a fast hack to appliance vendors like juniper = mostly, Others complained that people talk and nobody contributes code = , yet when you contribute code they dont look at it with months, and so = on. > In my last job, a lot of the questions revolved around =E2=80=9Chow do = we stick this OS into mobile devices with small batteries?=E2=80=9D and = those sorts of pragmatic concerns informed a lot of our engineering = work. =20 Yes. See, One thing I hear for decades is the Unix desktop. And = everybody and their mother started to write Windows managers, desktops , = whatever. We have OSes like PCBSD, which likes to masquerade itself like = a desktop. Yet it seems that nobody realised that a modern desktop , one = which is not a Rupe Goldberg contraption, or even worse, the software = frankenstein=E2=80=99s monster depends heavily on almost all the probem = domains I outlined above. Ironically, the some of the same base = technologies which can be used=20 to driver enteripise features like service management and fault = management, would enable the polar opposite, the user desktop. Yet what we see instead of someone asking those questions is people = jumping into coding layer after layer of half assed solutions, which = only make the situation worse. If people would at least agree on what has to be done, some would surely = study those problems and find some cool solutions to implement.=20 I like your NextBSD effort, save the fact you guys dragged half of Mach = kernel in it. I can understand why you did it, but I dont have to like = it. I think a new slim IPC mechanism=20 should be written , and that the major enterpises depending on BSD = should fork money to cover the development costs. Im happy that they use = BSD techs, but with very few exceptions they contribute almost nothing back.=20 > On 10 Jan 2016, at 20:32, Jordan Hubbard = wrote: >=20 >=20 >> On Jan 10, 2016, at 2:36 AM, Dan Partelly = wrote: >>=20 >> Copying the linux way of doing things should not be a goal of the = BSDs. >=20 > I=E2=80=99ve been trying to stay out of this since the discussion has, = at least in points, seemed a little on the =E2=80=9Cfnarr! fnarr!=E2=80=9D= polemic side (rather than focusing, as one would hope to see in a group = called =E2=80=9CHackers=E2=80=9D, on the engineering details and = specific technical pros-and-cons of each approach), but I guess I can no = longer resist. As one of the =E2=80=9Claunchd stake-holders=E2=80=9D in = the discussion, and one I hope to have at a conference in the near = future since email is a terrible communications medium for really = getting ones point=E2=80=99s across (and avoiding lots of points that = exist in the mind of the reader but not your own), let me just say that = our goals with NeXTBSD continue to be the following, though not = necessarily always in the following order: >=20 > 1. Stay in sync with FreeBSD-current, just to keep divergence to a = minimum >=20 > 2. Get all of the base technologies we are targeting (launchd et al) = fully working and as faithful in implementation to their upstream = origins (for the exact same reasons as #1) as possible. We=E2=80=99re = not slavishly following Apple, we=E2=80=99ll take any suitably licensed = technology that achieves our goals (see point #4). >=20 > 3. Start migrating all of the older facilities, like /etc/rc and = preferences files, to the new models - this one is really the =E2=80=9Clon= g pole=E2=80=9D and why we=E2=80=99re just staying quiet until this work = is largely completed and ready to show, since otherwise you=E2=80=99re = largely debating the pros and cons of vaporware vs vaporware. >=20 > 4. Use the specific and pragmatic world of Enterprise and Software = Appliance requirements to drive the feature set of the technologies we = choose and the urgency with which we pick them. >=20 > It=E2=80=99s this latter point that makes me roll my eyes a bit = whenever folks use phrases like =E2=80=9Cthe linux way=E2=80=9D or = =E2=80=9Cthe BSD way=E2=80=9D since I=E2=80=99m not entirely sure that = those =E2=80=9Cways=E2=80=9D, at least not as I first heard them = articulated back in the 1990=E2=80=99s, actually exist anymore. >=20 > Yeah, BSD has /usr/src and Linux has collections of packages and a = distro-centric model of looking at the universe, but those are = increasingly superficial distinctions when compared to the far more = pertinent distinctions today around *what* is being packaged and to what = purpose. What do application stacks look like in the year 2016 and = forward? How does mass-deployment work on your OS? Where are the = privilege domains around disparate collections of software drawn, and = how do you manage them? Do you have effective security models for = allowing entirely untrusted software to run on your OS? How are you = managing storage in clustered / hybrid (local + cloud) environments, and = how hard is it to bend the OS to your will as a DevOps person tasked = with any or all of the above? >=20 > I=E2=80=99m not pretending to have answers to even half of those = questions myself, don=E2=80=99t get me wrong, but I think the questions = are important just to staying alive. In my last job, a lot of the = questions revolved around =E2=80=9Chow do we stick this OS into mobile = devices with small batteries?=E2=80=9D and those sorts of pragmatic = concerns informed a lot of our engineering work. My current job asks = questions about Enterprise deployment and how to create Software = Appliances that have everything you need and nothing you don=E2=80=99t = in them, the Linux folks also complicating the picture with = =E2=80=9CContainers=E2=80=9D and =E2=80=9CDocker Apps=E2=80=9D (kudos to = them though - they managed to take a handful of things that had existed = for years and make them suddenly new and sexy again) and now I need = answers for those, too. >=20 > What I DO know won=E2=80=99t move the ball forward (on any field, in = any game) is arguing about this like it was a religious debate of some = sort, where sweeping generalizations are preferred over far more = pragmatic questions of =E2=80=9CWhat problems are we here to solve, and = what are the specific details of those problems?=E2=80=9D >=20 > This is also why I=E2=80=99m doing all of that stuff in a branch = called NextBSD. Who has time for religious debate when you=E2=80=99re = trying to just get code to work and solve a very specific set of = problems? :-) >=20 > - Jordan >=20 From owner-freebsd-hackers@freebsd.org Mon Jan 11 17:03:36 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0AFF8A6505D for ; Mon, 11 Jan 2016 17:03:36 +0000 (UTC) (envelope-from amesbury@oitsec.umn.edu) Received: from mail.oitsec.umn.edu (mail.oitsec.umn.edu [128.101.238.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.oitsec.umn.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1BD71CA8; Mon, 11 Jan 2016 17:03:35 +0000 (UTC) (envelope-from amesbury@oitsec.umn.edu) Received: from mail.oitsec.umn.edu (localhost [127.0.0.1]) by mail.oitsec.umn.edu (Postfix) with ESMTP id 982665C817; Mon, 11 Jan 2016 11:03:26 -0600 (CST) X-Virus-Scanned: amavisd-new at oitsec.umn.edu Received: from mail.oitsec.umn.edu ([127.0.0.1]) by mail.oitsec.umn.edu (mail.oitsec.umn.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0PXvrz80Uqny; Mon, 11 Jan 2016 11:03:25 -0600 (CST) Received: from optimator.oitsec.umn.edu (optimator.oitsec.umn.edu [134.84.23.1]) (Authenticated sender: amesbury) by mail.oitsec.umn.edu (Postfix) with ESMTPSA id E31F15C811; Mon, 11 Jan 2016 11:03:25 -0600 (CST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Subject: Re: master.passwd UID/GID number From: Alan Amesbury In-Reply-To: Date: Mon, 11 Jan 2016 11:03:27 -0600 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: To: araujo@freebsd.org X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2016 17:03:36 -0000 On Jan 6, 2016, at 23:11 , Eitan Adler wrote: > this was accidentally sent to a private list. Forwarding to the = correct list. >=20 > On 6 January 2016 at 19:48, Marcelo Araujo = wrote: >> Hello all, >>=20 >> I need to add a new special user on master.passwd for ypldap. >> I'm wondering what are the rules to choose the number for UID/GID if = there >> is any. >=20 > Pick an arbitrary but low number. Anything below 1000 is typically > okay - look to see what other users use. Have a look at ${PORTS}/UIDs and ${PORTS}/GIDs or http://svnweb.freebsd.org/ports/head/UIDs?view=3Dco http://svnweb.freebsd.org/ports/head/GIDs?view=3Dco to get an idea of what's in use by other software. --=20 Alan Amesbury University Information Security http://umn.edu/lookup/amesbury From owner-freebsd-hackers@freebsd.org Tue Jan 12 07:54:02 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92469A6D007 for ; Tue, 12 Jan 2016 07:54:02 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (mail.ixsystems.com [69.198.165.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D9D31C57 for ; Tue, 12 Jan 2016 07:54:02 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1452585241-08ca042abd51c770001-P5m3U7 Received: from mail.iXsystems.com ([10.2.55.1]) by barracuda.ixsystems.com with ESMTP id n9PEwfZkRUSsmY00 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 11 Jan 2016 23:54:01 -0800 (PST) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.2.55.1 X-ASG-Whitelist: Client Received: from [10.8.0.14] (unknown [10.8.0.14]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id E846081291; Mon, 11 Jan 2016 23:54:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ixsystems.com; s=newknight0; t=1452585241; bh=+XvQ0x16VturUTLAX9jgRfoEPXt37GnW1fZLwjoMJIo=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=EISKB50We/J8kvzEcfmXMbXcW9Pi9Tg7yLrZeBUuhO7+0JcbJDDn8LmTsjZIVWsBT oMN+LkQFzaOI2QMdnGRpm9YSYEstcIQOkwlt1tcewe+4y7ErWVpQ4DWOhlhwRN6oja T9u5m3+MVgGIXO7zz13ddT8p2AK9IeRw075D91Jk= Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: relaunchd: a portable clone of launchd From: Hubbard Jordan X-ASG-Orig-Subj: Re: relaunchd: a portable clone of launchd In-Reply-To: Date: Mon, 11 Jan 2016 23:53:59 -0800 Cc: FreeBSD Hackers , Jonathan de Boyne Pollard , Dmitry Sivachenko , Mark Heily , Peter Beckman Content-Transfer-Encoding: quoted-printable Message-Id: <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> To: Dan Partelly X-Mailer: Apple Mail (2.3112) X-Barracuda-Connect: UNKNOWN[10.2.55.1] X-Barracuda-Start-Time: 1452585241 X-Barracuda-Encrypted: DHE-RSA-CAMELLIA256-SHA X-Barracuda-URL: https://10.2.0.41:443/cgi-mod/mark.cgi X-Barracuda-BRTS-Status: 1 X-Virus-Scanned: by bsmtpd at ixsystems.com X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 07:54:02 -0000 > I personally use those terms a bit tongue in check. BSDs have very = limited manpower , and so they are forced to use source code from = foreign OSes in the most straightforward way possible, which means = wrappers . (else see the fiasco with the DRM drivers in FreeBSD so far, = there is plainly too much effort to port Linux DRM code to BSD without = wrappers and adapters. Dragonfly did it the right way ) . So , you need = to wrappers and adapters to Linux specific API=E2=80=99s and data = structures to support DRM , OFED and god knows what else.=20 I don=E2=80=99t disagree with the =E2=80=9Climited manpower=E2=80=9D = statement (it=E2=80=99s manifestly obvious), but perhaps a less negative = way of looking at this is to say that since OSes have become largely = commoditized (a natural consequence of changing educational trends in = computer science), we=E2=80=99re better off not reinventing any wheels = and simply adopting whatever off-the-shelf technologies and solutions = allow the mission goals to be achieved in the shortest period of time. > But this is a general problem not only limited to Linux.=20 Indeed. There=E2=80=99s also arguably a sweet spot between =E2=80=9Cemula= te all the things=E2=80=9D (and convince some segment of your own fan = base that you=E2=80=99re dying because nobody uses the =E2=80=9Cnative = offerings=E2=80=9D) and =E2=80=9Cemulate the most strategic things=E2=80=9D= vs spending so much time re-inventing wheels for the sake of it that = you miss the window of relevance entirely. > FreeBSD has NDIS emulation to support some Wifi devices.=20 Yes, the infamous =E2=80=9CProject Evil=E2=80=9D. :) I don=E2=80=99t = know how much that has actually been used in production scenario, but it = was an interesting project. > FreeBSD has Solaris API/ datastruct wrappers and adapters to support = ZFS/ Which has been a pretty huge success, really. ZFS is very stable on = FreeBSD (comparatively speaking to ZFS on Linux) and has opened some = entirely new markets for it. Had FreeBSD insisted on =E2=80=9Cdoing it = natively=E2=80=9D it=E2=80=99s pretty arguable that OpenZFS would have = gone its own way and ZFS would be an unstable science experiment that = nobody would trust their bits to. Sure, there is still some weirdness = with ARC behavior and the memory consumption of features like Dedup = (which is more OpenZFS=E2=80=99s fault than FreeBSD=E2=80=99s) but the = shims work well and make it possible for the same code base to build on = Illumos, which is still the reference code base, and FreeBSD. I=E2=80=99d= say this has been a Big Win. DTrace has also been a similar win, and is more =E2=80=9Cforeign DNA=E2=80= =9D in FreeBSD. You=E2=80=99d like to see more of Solaris=E2=80=99 SMF = in FreeBSD, and I agree. I wouldn=E2=80=99t mind Solaris Doors, either. = It would have made NDMP support far more of a drop-in than it has been = so far. I think we=E2=80=99re in violent agreement so far on the = subject of Solaris DNA! > NextBSD has now dragged =E2=80=9Chalf" of mach kernel inside to : 1. = Implement a better and more flexible IPC mechanism than Unix domain = sockets / posix message queues (Truth is, IPC sucks in Linux and BSDs. = It is my opinion that Windows LPC/ALPC are way better designed. and 2: = To easily port launchd/notifyd/whateverd That=E2=80=99s mostly all true, though I think Mach IPC=E2=80=99s = feature set deserves just a wee bit more respect than it traditionally = gets (or is fully described above). =20 The fact is, IPC on Unix has traditionally sucked and nobody has yet to = provide an alternative to Mach IPC which ticks all the same boxes. A = hierarchical namespace that can be used to create privilege domains = (providing a very key ability to control who can talk to who) - Check. = Supports service discovery as an intrinsic property without the presence = of a filesystem (e.g. pre-mountroot) - Check. Provides audit trailers = in every message that make it possible to validate who you=E2=80=99re = talking to and create =E2=80=9Ctrusted delegates=E2=80=9D for certain = services that would otherwise be forced to be in the kernel - Check. = The ability to pass shared memory descriptors out-of-band for large = messages - Check. That=E2=80=99s not even a full list, but it covers = some features that OS X and iOS use in many different ways to good = effect. Believe me, if there was something better, with the right high-level = semantics (which alternatives like L4 IPC do not provide) combined with = low-level capabilities like allowing the kernel to talk to userspace = using the very same IPC mechanisms, we=E2=80=99d use it. Since there = isn=E2=80=99t, we stick things like XPC (which NextBSD is doing a = clean-room implementation of) on top of Mach IPC to avoid the horror of = using MIG and doing RPC programming from the 1980s, and we spend the = time we=E2=80=99d otherwise have to spend creating an entirely new and = unproven IPC mechanism (which might very well fail to impress anyone = after we finished) and focus instead on creating the services that use = it. Seems like a reasonable example of =E2=80=9Ccode re-use=E2=80=9D to = me! > Now this is all good and dandy, but who will audit all this code for = bugs and security ? Some bugs will become manifest fast, but others may = lurk in those layers for years. I agree, which is actually all the more reason to adopt existing code. = It=E2=80=99s been vetted. It=E2=80=99s been tested. It=E2=80=99s been = used for many years across many different releases of software on = literally millions of devices. Seriously, how much better than that can = anyone reasonably expect or hope for? If there=E2=80=99s a way to set = the bar even higher for auditing / proving the efficacy of a body of = code, I=E2=80=99d be more than willing to learn about what that is. You = can probably imagine that a lot of people, both paid and unpaid, have = attacked the hell out of every byte of code in iOS and OS X. The = jailbreaking community alone has certainly had the incentive, and with = incentive comes inventiveness. > =E2=80=94 problem : binary code reuse --- > 1. Many utilites from BSD base are monolithic and closed, yet they = provide higher level interfaces to OS configuration. It is my opinion = that=20 > all this functionality should be expose as libraries & demon services. Violent agreement, for the most part. I don=E2=80=99t know how we=E2=80=99= d define =E2=80=9Call the functionality=E2=80=9D of course, and the idea = of wrapping everything from df(1) to yacc(1) would obviously cause = someone to question the sanity of any person proposing such a blanket = approach to the utilities in question, but you=E2=80=99d hardly be the = first person to suggest wrapping or re-writing all of the more useful = building-block commands in terms of libraries and services. Taking an = aggressively SOA-oriented approach is obviously what has allowed = companies like Amazon to encompass the almost unbelievable amount of = territory they now do without collapsing under their own weight, so if = anyone needs an existence proof, there it is (along with countless = others). > This is the very base upon which enterprise grade tools are built. = Libxo is not a proper solution to this problem. It should not exist in = FreeBSD base, yet somehow it slipped inside, and I seen on some papers = from BSD conferences that some even discussed to put it inside the OS = kernel . I frankly cant imagine what those ppl are/wehre thinking.=20 Oh, I dunno, I think you=E2=80=99re being a little harsh. I=E2=80=99ve = looked at libxo and I think it=E2=80=99s reasonable enough as a generic = text output library. But that=E2=80=99s all it is. It=E2=80=99s not a = policy, it=E2=80=99s certainly nothing like what we=E2=80=99re = describing above, it=E2=80=99s just a building block. Perhaps the = bigger criticisms should be reserved for what it=E2=80=99s not, instead: 1. It=E2=80=99s not a plan to actually take a set of commands like find, = xargs, du, ls and so on and impose some structure on them. It=E2=80=99s = certainly not a proposal which even attempts to identify =E2=80=9Cthe = right commands=E2=80=9D and postulate a set of use case scenarios that = will allow the user to create some truly interesting shell scripts which = use these new structured input/output capabilities to do things not = formerly possible. I speak from a little experience here as we did that science experiment = back around 2007 or so, just to see what it might look like to create a = subset of Unix command which all spoke XML natively, and the results = were surprisingly elegant when able to decouple =E2=80=9Cmechanism=E2=80=9D= from =E2=80=9Cpresentation=E2=80=9D (a command which simply stats a = file and presents the information from it as an XML blob doesn=E2=80=99t = have to worry about multi-column output or human readability concerns - = that=E2=80=99s for another tool further down the pipeline to do). I = kind of have no interest in this topic, however, because it just sets = off the =E2=80=9Cyou can have my 80 column, ISO-Latin1 text commands = when you pry them (and my VT100 terminal emulator) from my cold, dead = fingers!!=E2=80=9D crowd. :-) 2. It doesn=E2=80=99t provide one of the APIs MOST lacking from *BSD = right now, which is how to set and get structured preference information = for a utility of any complexity. Yeah, I=E2=80=99m talking about the = equivalent of CFPreferences here (and the services equivalent of = cfprefsd) - since everyone already expects me to quote Apple APIs, why = disappoint them? :) The biggest consumers of structured data are = actually services trying desperately to find a KVS somewhere, not people = writing shell scripts. Oh yeah, you touch on that later. More in a = second. > =E2=80=94 problem OS configuration and updating in a higly concurrent = world --- > 3. Transactional OS wide configuration databases.=20 Which, of course, we use every day with FreeNAS. When you=E2=80=99re an = appliance, you=E2=80=99re not allowed to look through half of /etc (or = export half of /etc as part of a =E2=80=9Csave configuration=E2=80=9D = operation) and yeah, we do it through a database abstraction layer that = lets anything from postgres to mongodb provide the actual storage. I = suppose it=E2=80=99s arguable whether the =E2=80=9Cconfiguration = database library + service=E2=80=9D actually needs to live in base, but = there are so many utilities that will continue to roll their own = otherwise, it seems silly not to. > 4. In a world where hundreds of machines, bee-it physical or virtual = are interconnected and talking to each other , safe concurrent access to = OS configuration will become soon very important. This is already the case. Most chef / puppet / vagrant setups bend over = backwards to try and impose order on what are otherwise rather = uncooperative OS instances deployed in the hundreds or thousands. > 4. A new high performance IPC system should be introduced IMO. It = should allow both fast synchronous operation ( like solaris doors) and = asynchronous operations, working with kevent/kqueue. It should allow = kernel endpoints, and should cater to security concerns. It should be a = minimal API , not a full buss Right. Mach IPC. :-) > 5. a user mode pub / sub message bus. use the IPC api to implement a = high performance , OS wide, message bus. It should present clear = abstractions to clients, to insulate them from using syscalls directly. = It should sum some already existing mechanisms, such as devd. XPC + system notifications do a pretty good job of this. It depends on = what you=E2=80=99re =E2=80=9Csubscribing=E2=80=9D to. A distributed = notification? Some peer-to-peer communication from another system = service? A configuration knob that can change dynamically? It=E2=80=99s = tempting to try and fold them into a single pub/sub namespace, but = it=E2=80=99s not clear to me that this is a good thing or not, or if = it=E2=80=99s simple better to allow each of the above services to = provide a nice call-back mechanism that works with a universal glue = library (like, *cough cough* XPC and/or libdispatch). > =E2=80=94 problem of service management and fault management: >=20 > 6. Much needed components of enterprise.=20 Do you know how much of Solaris SMF escaped into Illumos before Oracle = slammed the door there? I=E2=80=99ve honestly never looked. > I raise some of those problems in past. Especially the problem of = binary code reuse . My perception was that nobody really gives a damn. I think the problem with the FreeBSD community in general is that there = ARE people who give a damn, but they are essentially blocked into = ineffectiveness by another caucus who doesn=E2=80=99t want any of these = things to change. It=E2=80=99s like the Senate - you have Republicans = and Democrats who see eye-to-eye on almost nothing and their principle = power is in being able to block whatever initiatives =E2=80=9Cthe other = side=E2=80=9D proposes and, well, Democracy Inaction. :-) > Others complained that people talk and nobody contributes code , yet = when you contribute code they dont look at it with months, and so on. Hence us trying to just prove the =E2=80=9Cworking code=E2=80=9D = requirement first in NextBSD and if nobody looks, well, that=E2=80=99s = OK too - we can still use it for our own purposes. Maybe some other = Appliance vendors will jump into the mix and use it for their purposes = too. We have more progress to make before I=E2=80=99d even try to = =E2=80=9Csell=E2=80=9D it for any purpose whatsoever yet, however. One = step at a time. > Yes. See, One thing I hear for decades is the Unix desktop. Which I still think will probably never succeed for as long as the X = Window System is in the mix because its APIs, and the mechanisms it = forces into place, are just a horror show, even with all the KDE / Gnome = libraries and services that folks tried to add on top to paper over the = differences. You can=E2=80=99t polish a turd, you can only paint it. = :-) And yeah, I wrote Window Managers (like awm) and widget libraries = and such Back In The Day for X. I was a believer. Then I tried to port = Lotus 1-2-3 to it. :( - Jordan From owner-freebsd-hackers@freebsd.org Tue Jan 12 12:59:59 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D1ECA6BF56 for ; Tue, 12 Jan 2016 12:59:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D9DE7144F for ; Tue, 12 Jan 2016 12:59:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0CCxnmc015171 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 12 Jan 2016 14:59:49 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0CCxnmc015171 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0CCxmnf015170; Tue, 12 Jan 2016 14:59:48 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 12 Jan 2016 14:59:48 +0200 From: Konstantin Belousov To: Hubbard Jordan Cc: FreeBSD Hackers Subject: Re: relaunchd: a portable clone of launchd Message-ID: <20160112125948.GH3625@kib.kiev.ua> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 12:59:59 -0000 On Mon, Jan 11, 2016 at 11:53:59PM -0800, Hubbard Jordan wrote: > The fact is, IPC on Unix has traditionally sucked and nobody has yet to provide an alternative to Mach IPC which ticks all the same boxes. A hierarchical namespace that can be used to create privilege domains (providing a very key ability to control who can talk to who) - Check. Supports service discovery as an intrinsic property without the presence of a filesystem (e.g. pre-mountroot) - Check. Provides audit trailers in every message that make it possible to validate who you???re talking to and create ???trusted delegates??? for certain services that would otherwise be forced to be in the kernel - Check. The ability to pass shared memory descriptors out-of-band for large messages - Check. That???s not even a full list, but it covers some features that OS X and iOS use in many different ways to good effect. > > Believe me, if there was something better, with the right high-level semantics (which alternatives like L4 IPC do not provide) combined with low-level capabilities like allowing the kernel to talk to userspace using the very same IPC mechanisms, we???d use it. Since there isn???t, we stick things like XPC (which NextBSD is doing a clean-room implementation of) on top of Mach IPC to avoid the horror of using MIG and doing RPC programming from the 1980s, and we spend the time we???d otherwise have to spend creating an entirely new and unproven IPC mechanism (which might very well fail to impress anyone after we finished) and focus instead on creating the services that use it. Seems like a reasonable example of ???code re-use??? to me! > Anybody who used Win32 and even whispered a word about bloated and verbose interfaces that require two screens of boilerplate to do trivial things, would be much more amazed by Mach IPC. I highly recommend to Google for "Mach IPC sucks" if reader is really interested. Although, the main complaint you would find there is about extreme overhead, which is mostly due to unconditional support for the all the hundreds features partially listed above. Microkernels failed for many reasons, and uglyness of the basic underlying mechanics in the most advanced and popular specimen a was non-trivial factor. The theme of great softwares and codes from past which did not survived due to (conspiracy|underrate|worse is better) is quite popular and not debunked since proponents always point out perceived goodies and are silent about suck points. Because nobody used (and can try the stuff for real), the claims stay. See the memories of things like SunView, Display Postscript, etc. Fortunately people do not praise Corba yet, at least not with the straight face, but I am waiting. Even mentioned Doors are better then Mach IPC, but still dead, I think app space is completely consumed now by dbus or http-based hand-rolled protocols. From owner-freebsd-hackers@freebsd.org Tue Jan 12 13:13:50 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF9EEA6C598 for ; Tue, 12 Jan 2016 13:13:50 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: from mail.soaustin.net (mail.soaustin.net [66.135.54.68]) by mx1.freebsd.org (Postfix) with ESMTP id B5A661164 for ; Tue, 12 Jan 2016 13:13:50 +0000 (UTC) (envelope-from linimon@lonesome.com) Received: by mail.soaustin.net (Postfix, from userid 502) id 97DBE56163; Tue, 12 Jan 2016 07:13:44 -0600 (CST) Date: Tue, 12 Jan 2016 07:13:44 -0600 From: Mark Linimon To: Konstantin Belousov Cc: Hubbard Jordan , FreeBSD Hackers Subject: Re: relaunchd: a portable clone of launchd Message-ID: <20160112131344.GB26263@lonesome.com> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160112125948.GH3625@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-Mailman-Approved-At: Tue, 12 Jan 2016 13:43:24 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 13:13:50 -0000 On Tue, Jan 12, 2016 at 02:59:48PM +0200, Konstantin Belousov wrote: > Fortunately people do not praise Corba yet, at least not with the > straight face, but I am waiting. If they start, I'll be waiting off to one side with my cluebat. I had to suffer through it once. "Oh, you want error handling? That's nice." Bah. mcl From owner-freebsd-hackers@freebsd.org Tue Jan 12 13:49:56 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DAE4A6D951 for ; Tue, 12 Jan 2016 13:49:56 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from mail.rdsor.ro (mail.rdsor.ro [193.231.238.10]) by mx1.freebsd.org (Postfix) with ESMTP id C4AC814B3 for ; Tue, 12 Jan 2016 13:49:55 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from [192.168.1.155] (unknown [86.125.33.32]) by mail.rdsor.ro (Postfix) with ESMTP id 3A174DBFD3; Tue, 12 Jan 2016 15:49:48 +0200 (EET) Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: relaunchd: a portable clone of launchd From: Dan Partelly In-Reply-To: <20160112125948.GH3625@kib.kiev.ua> Date: Tue, 12 Jan 2016 15:49:48 +0200 Cc: Hubbard Jordan , FreeBSD Hackers Message-Id: References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.3112) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 13:49:56 -0000 Verbose is not bad , Konstantin . I actually prefer verbose source. It = is easy to read and understand. It is preferable all day long to clever hacks and obfuscated ways of = writing code Ill tell you one thing. Years ago, when Ms Leaked on their site = ntoskrnl.exe symbols with private debug=20 information , I dint had too much trouble understanding a lot of the = implementation details from the kernel ,=20 exactly because MS is verbose. And that .. in WinDbg assembly. The = source would have been much much=20 more easier to read.=20 Yeah, MS=E2=80=99s APIs are not the best I give you that. But regarding = *readability* , Windows is leaps and bounds ahead of any Unix I seen. It is extremely easy to figure out what a = certain function does, how they use data=20 structures, and what the members of said data structures represent. I like verbosity. Makes figuring things easier, makes maintenance = easier. I frankly hope to spend as little of possible=20 of my life reading terse source code. This is also why I like Ada = language. Its a joy to read Ada source.=20 > On 12 Jan 2016, at 14:59, Konstantin Belousov = wrote: >=20 > nybody who used Win32 and even whispered a word about bloated and = verbose > interfaces that require two screens of boilerplate to do trivial = things, > would be much more amazed by Mach IPC. From owner-freebsd-hackers@freebsd.org Tue Jan 12 15:59:14 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21AF5A6C2F6 for ; Tue, 12 Jan 2016 15:59:14 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (mail.ixsystems.com [69.198.165.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 06AFA1564 for ; Tue, 12 Jan 2016 15:59:13 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1452614352-08ca042abc527a10001-P5m3U7 Received: from mail.iXsystems.com ([10.2.55.1]) by barracuda.ixsystems.com with ESMTP id 1eRR5TIoVChcZrrA (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 12 Jan 2016 07:59:13 -0800 (PST) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.2.55.1 X-ASG-Whitelist: Client Received: from [10.8.0.14] (unknown [10.8.0.14]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id AAA31A37F9; Tue, 12 Jan 2016 07:59:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ixsystems.com; s=newknight0; t=1452614352; bh=aIkb6kYwmE/QX9Nq2DDHPwzcl3FJi/x3/UgzEhr1Zjc=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=QeG+uit0NnnRyVILUuGk9gGphPRU8eNYAvIx56pez9O5rM4C2kiuZVZVTibzMk7gU RQvniFeNZZKIqcuB3yF+eV4KfPM8WlLFcA7H+T+eZALH78N53UvupUjxCQ9/kEqnze K1mhAnHSnpFLFAirXUKUaWr5og96GEJPcmhz5ADs= Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: relaunchd: a portable clone of launchd From: Hubbard Jordan X-ASG-Orig-Subj: Re: relaunchd: a portable clone of launchd In-Reply-To: <20160112125948.GH3625@kib.kiev.ua> Date: Tue, 12 Jan 2016 07:59:11 -0800 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.3112) X-Barracuda-Connect: UNKNOWN[10.2.55.1] X-Barracuda-Start-Time: 1452614353 X-Barracuda-Encrypted: DHE-RSA-CAMELLIA256-SHA X-Barracuda-URL: https://10.2.0.41:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 15:59:14 -0000 > On Jan 12, 2016, at 4:59 AM, Konstantin Belousov = wrote: >=20 > I highly recommend to Google for "Mach IPC sucks" if reader is really = interested. And here we return to the usual trap=E2=80=A6 =E2=80=9CMach IPC sucks!=E2=80=9D =E2=80=9COK. What do you propose that will address all of the same = concerns?=E2=80=9D =E2=80=9Cdbus!=E2=80=9D =E2=80=9C*Sigh*. You haven=E2=80=99t even looked at the two = technologies in any depth, have you? Go read the dbus wikipedia page, = at least! Unix domain sockets underneath, no kernel<->userland = communication path, no trusted IPC mechanism, no support for large = messages=E2=80=A6=E2=80=9D =E2=80=9COK, so something new!! We should totally create an IPC for the = New Millennium!=E2=80=9D =E2=80=9CThat would be you then? Where=E2=80=99s your white paper? = Where=E2=80=99s your reference implementation?=E2=80=9D Sorry. Been there, had this debate, and while it=E2=80=99s always = extremely easy to fling poop at an existing mechanism, it turns out = it=E2=80=99s so much harder to actually *create an alternative* that = this kind of discussion only serves to throw cold water on evolution = (=E2=80=9Cthe perfect being the enemy of the good enough=E2=80=9D) and = the end-result is that evolution does not occur. I also already covered how it=E2=80=99s very easy to layer something = like XPC *on top* of Mach IPC such that you, the programmer, need never = be exposed to the Mach IPC APIs (but still get to leverage the internal = capabilities I=E2=80=99ve already covered). Sorry, Konstantin, but yours is a non-argument. - Jordan From owner-freebsd-hackers@freebsd.org Tue Jan 12 17:24:39 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8745A6D9F1 for ; Tue, 12 Jan 2016 17:24:39 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4A8871DA7 for ; Tue, 12 Jan 2016 17:24:38 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id u0CHK2cU010453; Tue, 12 Jan 2016 12:20:02 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Tue, 12 Jan 2016 12:20:02 -0500 (EST) Date: Tue, 12 Jan 2016 12:20:02 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Dan Partelly cc: Konstantin Belousov , FreeBSD Hackers , Hubbard Jordan Subject: Re: relaunchd: a portable clone of launchd In-Reply-To: Message-ID: References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=utf-8; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 17:24:39 -0000 On Tue, 12 Jan 2016, Dan Partelly wrote: > Verbose is not bad , Konstantin . I actually prefer verbose source.=20 > It is easy to read and understand. It is preferable all day long to=20 > clever hacks and obfuscated ways of writing code > > Ill tell you one thing. Years ago, when Ms Leaked on their site=20 > ntoskrnl.exe symbols with private debug information , I dint had too=20 > much trouble understanding a lot of the implementation details from=20 > the kernel , exactly because MS is verbose. And that .. in WinDbg=20 > assembly. The source would have been much much more easier to read. > > Yeah, MS=E2=80=99s APIs are not the best I give you that. But regarding= =20 > *readability* , Windows is leaps and bounds ahead of any Unix I seen.=20 > It is extremely easy to figure out what a certain function does, how=20 > they use data structures, and what the members of said data structures=20 > represent. > > I like verbosity. Makes figuring things easier, makes maintenance=20 > easier. I frankly hope to spend as little of possible of my life=20 > reading terse source code. This is also why I like Ada language. Its a=20 > joy to read Ada source. Ok, speaking as a software engineer for 31 years and has been developing and maintaining Ada programs for the last 25 of those years... Ada is just a language, I've seen both bad design and good design in it, as well as in C and Java. I've seen massive over verboseness in Ada and it makes maintaining it a nightmare. When your coding guidelines have a maximum line length of 132 or greater characters, and lines _routinely_ go _well_ over 80 characters, because of package hierarchy, and package and subprogram (methods) names being very long, you know something is wrong. It depends on how you define what is a good level of verboseness, and you can have that in pretty much any language. You just need to herd cats to follow the guidelines. I echo the sentiments about CORBA. Being forced to use bloated middleware to do something simple... I have to stop thinking about it because it makes me angry ;-) --=20 DE From owner-freebsd-hackers@freebsd.org Tue Jan 12 17:48:25 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FB67A6C17A for ; Tue, 12 Jan 2016 17:48:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47EE21361 for ; Tue, 12 Jan 2016 17:48:25 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0CHmKno084606 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 12 Jan 2016 19:48:20 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0CHmKno084606 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0CHmJKm084605; Tue, 12 Jan 2016 19:48:19 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 12 Jan 2016 19:48:19 +0200 From: Konstantin Belousov To: Hubbard Jordan Cc: FreeBSD Hackers Subject: Re: relaunchd: a portable clone of launchd Message-ID: <20160112174819.GL3625@kib.kiev.ua> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 17:48:25 -0000 On Tue, Jan 12, 2016 at 07:59:11AM -0800, Hubbard Jordan wrote: > > > On Jan 12, 2016, at 4:59 AM, Konstantin Belousov wrote: > > > > I highly recommend to Google for "Mach IPC sucks" if reader is really interested. > > And here we return to the usual trap??? > > ???Mach IPC sucks!??? > > ???OK. What do you propose that will address all of the same concerns???? > > ???dbus!??? I did not said this. > > ???*Sigh*. You haven???t even looked at the two technologies in any depth, have you? Go read the dbus wikipedia page, at least! Unix domain sockets underneath, no kernel<->userland communication path, no trusted IPC mechanism, no support for large messages?????? Is this directed at me, or just presents an imaginary dialog between you and some third party ? > > ???OK, so something new!! We should totally create an IPC for the New Millennium!??? > > ???That would be you then? Where???s your white paper? Where???s your reference implementation???? > > > > Sorry. Been there, had this debate, and while it???s always extremely easy to fling poop at an existing mechanism, it turns out it???s so much harder to actually *create an alternative* that this kind of discussion only serves to throw cold water on evolution (???the perfect being the enemy of the good enough???) and the end-result is that evolution does not occur. > > I also already covered how it???s very easy to layer something like XPC *on top* of Mach IPC such that you, the programmer, need never be exposed to the Mach IPC APIs (but still get to leverage the internal capabilities I???ve already covered). > > Sorry, Konstantin, but yours is a non-argument. What is a non-argument in my previous message ? I made two points: 1. story of Mach IPC being convenient or nice is not quite right (as most other stories of the great older tech which did not survived). 2. most people do not care anyway, and already use less ambitious but more practical alternatives. I did not suggested any substitution for Mach IPC, and I do not see much point on spending energy on discussing its alternatives or even trying to design new uber-IPC solution, mostly due to item 2. Item 1 is what caused my reaction to your text. From owner-freebsd-hackers@freebsd.org Tue Jan 12 17:52:13 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 694E0A6C377 for ; Tue, 12 Jan 2016 17:52:13 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from mail.rdsor.ro (mail.rdsor.ro [193.231.238.10]) by mx1.freebsd.org (Postfix) with ESMTP id 024A617C2; Tue, 12 Jan 2016 17:52:12 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from email.rdsor.ro (ftp.rdsor.ro [193.231.238.4]) by mail.rdsor.ro (Postfix) with ESMTP id 223AA232B4; Tue, 12 Jan 2016 19:52:11 +0200 (EET) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Date: Tue, 12 Jan 2016 19:52:24 +0200 From: dan_partelly To: Daniel Eischen Cc: Konstantin Belousov , FreeBSD Hackers , Hubbard Jordan Subject: Re: relaunchd: a portable clone of launchd In-Reply-To: References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> Message-ID: <6af37e1171c480dd1984409bbd029d0d@rdsor.ro> X-Sender: dan_partelly@rdsor.ro User-Agent: RoundCube Webmail/0.4-beta X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 17:52:13 -0000 >>It depends on how you define what is a good level of verboseness, >> and you can have that in pretty much any language. Sure, I get your point, and you are perfectly right. Winodws ntoskrnl is clearly C , but I am extremely sure that their verbosity (or perceived verbosity) is an asset , not a liability, for their source code maintenance. Some argue that MS systems team botched with the hungarian notation, and the app team used it better, but ... even so I found it useful. And yeah, Cutler didn't let any smarty-pants say "oh, "XX_CREATE_XX" is too verbose, let's contract it to "XX_CREAT_XX". On Tue, 12 Jan 2016 12:20:02 -0500 (EST), Daniel Eischen wrote: > On Tue, 12 Jan 2016, Dan Partelly wrote: > >> Verbose is not bad , Konstantin . I actually prefer verbose source. >> It is easy to read and understand. It is preferable all day long to >> clever hacks and obfuscated ways of writing code >> >> Ill tell you one thing. Years ago, when Ms Leaked on their site >> ntoskrnl.exe symbols with private debug information , I dint had too >> much trouble understanding a lot of the implementation details from >> the kernel , exactly because MS is verbose. And that .. in WinDbg >> assembly. The source would have been much much more easier to read. >> >> Yeah, MS’s APIs are not the best I give you that. But regarding >> *readability* , Windows is leaps and bounds ahead of any Unix I seen. >> It is extremely easy to figure out what a certain function does, how >> they use data structures, and what the members of said data structures >> represent. >> >> I like verbosity. Makes figuring things easier, makes maintenance >> easier. I frankly hope to spend as little of possible of my life >> reading terse source code. This is also why I like Ada language. Its a >> joy to read Ada source. > > Ok, speaking as a software engineer for 31 years and has been > developing and maintaining Ada programs for the last 25 of those > years... Ada is just a language, I've seen both bad design and > good design in it, as well as in C and Java. I've seen massive > over verboseness in Ada and it makes maintaining it a nightmare. > When your coding guidelines have a maximum line length of 132 > or greater characters, and lines _routinely_ go _well_ over 80 > characters, because of package hierarchy, and package and > subprogram (methods) names being very long, you know something > is wrong. > > It depends on how you define what is a good level of verboseness, > and you can have that in pretty much any language. You just > need to herd cats to follow the guidelines. > > I echo the sentiments about CORBA. Being forced to use > bloated middleware to do something simple... I have to > stop thinking about it because it makes me angry ;-) From owner-freebsd-hackers@freebsd.org Tue Jan 12 18:11:02 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7CC96A6CDF3 for ; Tue, 12 Jan 2016 18:11:02 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from mail.rdsor.ro (mail.rdsor.ro [193.231.238.10]) by mx1.freebsd.org (Postfix) with ESMTP id 17335173E for ; Tue, 12 Jan 2016 18:11:01 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from email.rdsor.ro (ftp.rdsor.ro [193.231.238.4]) by mail.rdsor.ro (Postfix) with ESMTP id ADE14120FD; Tue, 12 Jan 2016 20:11:00 +0200 (EET) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Date: Tue, 12 Jan 2016 20:11:14 +0200 From: dan_partelly To: Konstantin Belousov Cc: Hubbard Jordan , FreeBSD Hackers Subject: Re: relaunchd: a portable clone of launchd In-Reply-To: <20160112174819.GL3625@kib.kiev.ua> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> <20160112174819.GL3625@kib.kiev.ua> Message-ID: X-Sender: dan_partelly@rdsor.ro User-Agent: RoundCube Webmail/0.4-beta X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 18:11:02 -0000 On Tue, 12 Jan 2016 19:48:19 +0200, Konstantin Belousov wrote: > On Tue, Jan 12, 2016 at 07:59:11AM -0800, Hubbard Jordan wrote: >> >> > On Jan 12, 2016, at 4:59 AM, Konstantin Belousov >> > wrote: >> > >> > I highly recommend to Google for "Mach IPC sucks" if reader is really >> > interested. >> >> And here we return to the usual trap??? >> >> ???Mach IPC sucks!??? >> >> ???OK. What do you propose that will address all of the same >> concerns???? >> >> ???dbus!??? > I did not said this. On Tue, 12 Jan 2016 19:48:19 +0200, Konstantin Belousov wrote: >>2. most people do not care anyway, and already use less ambitious >> but more practical alternatives. Oh, ppl do care. Perhaps someone which enjoys building Rupe Goldberg contraptions from shell scripts will not care, but the rest will do. Why else do you think kd-bus exists, and that those developers continue to poor significant effort into it, even full redesigns , after the failed attempts to integrate it in the kernel ? >> story of Mach IPC being convenient or nice is not quite right but you cannot deny that it so far it did the job well for Apple so far. Even if Apple will roll a new IPC in the next years, so far Mach ports where useful for them. So it is not like NextBSD guys just pulled something unused from 80s from naphthalene. > >> >> ???*Sigh*. You haven???t even looked at the two technologies in any >> depth, have you? Go read the dbus wikipedia page, at least! Unix domain >> sockets underneath, no kernel<->userland communication path, no trusted >> IPC mechanism, no support for large messages?????? > > Is this directed at me, or just presents an imaginary dialog between you > and some third party ? > >> >> ???OK, so something new!! We should totally create an IPC for the New >> Millennium!??? >> >> ???That would be you then? Where???s your white paper? Where???s your >> reference implementation???? >> >> >> >> Sorry. Been there, had this debate, and while it???s always extremely >> easy to fling poop at an existing mechanism, it turns out it???s so much >> harder to actually *create an alternative* that this kind of discussion >> only serves to throw cold water on evolution (???the perfect being the >> enemy of the good enough???) and the end-result is that evolution does >> not occur. >> >> I also already covered how it???s very easy to layer something like XPC >> *on top* of Mach IPC such that you, the programmer, need never be exposed >> to the Mach IPC APIs (but still get to leverage the internal capabilities >> I???ve already covered). >> >> Sorry, Konstantin, but yours is a non-argument. > > What is a non-argument in my previous message ? I made two points: > 1. story of Mach IPC being convenient or nice is not quite right > (as most other stories of the great older tech which did not survived). > 2. most people do not care anyway, and already use less ambitious > but more practical alternatives. > > I did not suggested any substitution for Mach IPC, and I do not see much > point on spending energy on discussing its alternatives or even trying > to design new uber-IPC solution, mostly due to item 2. > > Item 1 is what caused my reaction to your text. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Tue Jan 12 18:15:07 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45D16A80066 for ; Tue, 12 Jan 2016 18:15:07 +0000 (UTC) (envelope-from jonathan@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 38BD31AD0; Tue, 12 Jan 2016 18:15:07 +0000 (UTC) (envelope-from jonathan@FreeBSD.org) Received: from [134.153.27.124] (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id BCF9D100C; Tue, 12 Jan 2016 18:15:06 +0000 (UTC) (envelope-from jonathan@FreeBSD.org) From: "Jonathan Anderson" To: "Konstantin Belousov" Cc: "Hubbard Jordan" , "FreeBSD Hackers" Subject: Re: relaunchd: a portable clone of launchd Date: Tue, 12 Jan 2016 14:45:06 -0330 Message-ID: <616D464E-0A67-4EF0-AB32-ADB7087D325B@FreeBSD.org> In-Reply-To: <20160112125948.GH3625@kib.kiev.ua> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> MIME-Version: 1.0 X-Mailer: MailMate (1.9.3r5187) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 18:15:07 -0000 On 12 Jan 2016, at 9:29, Konstantin Belousov wrote: > I highly recommend to Google for "Mach IPC sucks" if reader is really > interested. I did as you suggested and came up with: http://www.googlefight.com/mach+ipc+sucks-vs-unix+ipc+sucks.php So says the Internet. So say we all. Jon -- Jonathan Anderson jonathan@FreeBSD.org From owner-freebsd-hackers@freebsd.org Wed Jan 13 03:08:02 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8FDFA80C7B for ; Wed, 13 Jan 2016 03:08:02 +0000 (UTC) (envelope-from mark@heily.com) Received: from mail-io0-x233.google.com (mail-io0-x233.google.com [IPv6:2607:f8b0:4001:c06::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78239102B for ; Wed, 13 Jan 2016 03:08:02 +0000 (UTC) (envelope-from mark@heily.com) Received: by mail-io0-x233.google.com with SMTP id 77so373912123ioc.2 for ; Tue, 12 Jan 2016 19:08:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heily-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=cjccxjajb3Z0HvXgq4jV7wdffMJUmalW2vTf5enRgtk=; b=rciumxJcOFE8bIfdXFJiZovb8V2alSDLG4nSZjSaAVhYLyTOFTgtpBJm/e4Q2i3Wi5 4Cn9NZPI13HIaHTkEVRPls+/o9T6JEVWhoMa58j69MGg/XqbIXc+sfCtibn7q78I2JSD 2+VO+7ggOULDIynkMuLs9qLuHQiNAg/P3JUxywKfd0FgcXSdQBrbXLxPNxBYcaVu/c6/ KMqLhJZs8RolIaGfLxyaTz3OLOfjSPVeI3NhbWgw/zJyM13nbXUxUsJGiW98j2EcBplg Ll1DAFbck7oBuvNDLHHzsLp1L6pQ5wH5gMqc00JaRagEwUTBnBjlfvwSNWbAUK60+UH5 GUBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=cjccxjajb3Z0HvXgq4jV7wdffMJUmalW2vTf5enRgtk=; b=V2smJj7CEBYQkKzCJ4EAk7BgKNSozbyzupycsGk05ljVwRmt/jbhsVQiEnwfiyrASP k6NmEdzEWfRPAUy1Pq9jvVYRbzB6SJujQeSv6SW/myB+C0y4NTB49nPUHw3kiCYiR9Xe s+vQ0fDwgFE/DW5en70748f+oPFsbBO31NARSMJ+u4oYkFdL4RipmSCAAlcLSe+YyB9q fK+ehCtNBqfhrmIxtrGp9nzn/5U1Tto2w409cqG14UtBAVlK0Q46a7jY0in5wsSRCxFr qN7ah2FEygHyTEIourJsONzbPMZ0pJlWPvWudsarmXdfblMEPDVzxFzmEPVknIvWTiAl 8cnw== X-Gm-Message-State: ALoCoQmJNe6R2sTzNrD8fFA+njH2KeYhgmmAnVjIeUDeOZ2WYzEzM3up8fov8WWxDRe8eOpQQw++nxckGtDtsUdXB3e44m3tUA== MIME-Version: 1.0 X-Received: by 10.107.43.138 with SMTP id r132mr136416429ior.7.1452654481565; Tue, 12 Jan 2016 19:08:01 -0800 (PST) Received: by 10.79.34.196 with HTTP; Tue, 12 Jan 2016 19:08:01 -0800 (PST) X-Originating-IP: [71.70.175.250] In-Reply-To: <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> Date: Tue, 12 Jan 2016 22:08:01 -0500 Message-ID: Subject: Re: relaunchd: a portable clone of launchd From: Mark Heily To: Hubbard Jordan Cc: Konstantin Belousov , FreeBSD Hackers Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2016 03:08:02 -0000 On Tue, Jan 12, 2016 at 10:59 AM, Hubbard Jordan wrote: > >> On Jan 12, 2016, at 4:59 AM, Konstantin Belousov w= rote: >> >> I highly recommend to Google for "Mach IPC sucks" if reader is really in= terested. > > And here we return to the usual trap=E2=80=A6 > > =E2=80=9CMach IPC sucks!=E2=80=9D > > =E2=80=9COK. What do you propose that will address all of the same conce= rns?=E2=80=9D > > =E2=80=9Cdbus!=E2=80=9D > > =E2=80=9C*Sigh*. You haven=E2=80=99t even looked at the two technologies= in any depth, have you? Go read the dbus wikipedia page, at least! Unix = domain sockets underneath, no kernel<->userland communication path, no trus= ted IPC mechanism, no support for large messages=E2=80=A6=E2=80=9D > > =E2=80=9COK, so something new!! We should totally create an IPC for the = New Millennium!=E2=80=9D > > =E2=80=9CThat would be you then? Where=E2=80=99s your white paper? Wher= e=E2=80=99s your reference implementation?=E2=80=9D > > > > Sorry. Been there, had this debate, and while it=E2=80=99s always extrem= ely easy to fling poop at an existing mechanism, it turns out it=E2=80=99s = so much harder to actually *create an alternative* that this kind of discus= sion only serves to throw cold water on evolution (=E2=80=9Cthe perfect bei= ng the enemy of the good enough=E2=80=9D) and the end-result is that evolut= ion does not occur. > > You're right in that DBus is not a great solution, and I think we can do better. I've gone ahead and created a general-purpose IPC library that is a plausible alternative to Mach IPC. Here's the code: https://github.com/mheily/libipc It is not a bus, and it is not object-oriented. Instead, it tries to make calling a remote function as easy and seamless as calling a local function. I was planning on hacking on it for a few more weeks before announcing it, but it seems very relevant to the discussion we are having today. It's built on top of Unix-domain sockets, and includes an IDL compiler that takes a reasonably simple YAML file and generates the boring code that allows programs to do structured IPC without worrying about serialization and bootstrapping and all of the underlying protocol issues. You've raised some objections to Unix-domain sockets, which I'd like to respond to. I'll quote from the slide in your presentation about Unix-domain sockets: ------------------------------------------------------------ > > > What can Mach ports do that Unix domain sockets can't? > > Separate namespace for services (doesn't rely on file system naming or > permissions) > This is generally a bad thing, IMHO. Filesystem permissions are a good way to apply security rules to an object, and they are a highly standard and well understood concept. If you need more than the traditional user/group/other support, you can supplement this with POSIX ACLs. Using the filesystem as a way to coordinate activity is a perfectly natural thing to do, and only has a disadvantage in a small corner case (where the system is in the process of booting up and the filesystem is not mounted). Right now, there is no need to perform IPC in the early part of the boot process. A machine where the root filesystem is not mounted read/write is basically unusable, and the solution is to mount the filesystem ASAP before starting any services that rely on IPC. > > Message boundaries > Not true. You have to provide your own semantics for finding the message boundaries, but it is totally possible to send messages across Unix-domain sockets. > > Kernel as peer > This is true, but not there are other ways for userspace to communicate with the kernel, such as syscalls and ioctl. Monolithic kernels don't use message passing, so we aren't missing out on much by not having this feature. > > Pre-existing well defined RPC interface > If Mach has such a great RPC interface, why do we need to hide it behind another abstraction layer like XPC? The existence of a thing is not necessarily an argument for it being the right tool for the job. > > Receive messages directly in call to kevent() > This is a "nice to have" performance optimization. We can live without it. > > OOL (out of line) messages (arbitrarily sized with zero copy for large me= ssages) > You can do zero-copy over Unix-domain sockets by sending a file descriptor across the socket that is backed by an anonymous region of memory created with mmap(2). To get the out-of-line behavior, all you need are two Unix-domain sockets; one acting as the control channel, and the other as the data channel. > > Port send rights - can only send to a port for which the process has expl= icitly > received the right to send > Unix-domain sockets have file permissions that control access. This is functionally equivalent to "port send rights", right? > > Provenance - Yes, PROVENANCE, receiver can have the kernel append an audi= t > trailer containing full set of credentials of sender > Unix-domain sockets allow you retrieve UID, GID, and PID of the client process. Unless you mean something different by "PROVENANCE", this is all the information you really need to make security decisions in the current Unix security model. ------------------------------------------------------------ So the TL;DR is that you can use Unix-domain sockets to do most of what Mach IPC provides. There are some major benefits to using sockets, such as the fact that are portable and relatively standardized across different Unix implementations (and even Windows). I think if you do a cost-benefit analysis between Mach and sockets, sockets win. Anyway, I'm planning to implement IPC in relaunchd using libipc, and relaunchd will be able to create the IPC service and launch the job "on demand" whenever a client attempts to connect to it. We've taken up a lot of airtime on this mailing list, so I invite anyone who is interested in IPC to join me on the libipc mailing list to discuss this further: https://groups.google.com/d/forum/libipc-devel From owner-freebsd-hackers@freebsd.org Wed Jan 13 06:17:50 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B365BA6DBE5 for ; Wed, 13 Jan 2016 06:17:50 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from mail.rdsor.ro (mail.rdsor.ro [193.231.238.10]) by mx1.freebsd.org (Postfix) with ESMTP id 4C0A21121 for ; Wed, 13 Jan 2016 06:17:48 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from [192.168.1.100] (unknown [79.117.100.196]) by mail.rdsor.ro (Postfix) with ESMTP id 22C3C23286; Wed, 13 Jan 2016 08:17:47 +0200 (EET) Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: relaunchd: a portable clone of launchd From: Dan Partelly In-Reply-To: Date: Wed, 13 Jan 2016 08:17:46 +0200 Cc: Hubbard Jordan , Konstantin Belousov , FreeBSD Hackers Message-Id: References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> To: Mark Heily X-Mailer: Apple Mail (2.3112) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2016 06:17:50 -0000 >=20 > This is generally a bad thing, IMHO. Filesystem permissions are a > good way to apply security rules to an object,=20 It is a good thing to have a hierarchical namespace which do not depend = on the filesystem. Port right are also a good way to apply security rules to objects.=20 > Using the filesystem as a way to coordinate activity is a perfectly > natural thing to do Why would be so natural ?=20 > A machine where the root filesystem is not mounted read/write is > basically unusable, and the solution is to mount the filesystem ASAP > before starting any services that rely on IPC. Yet kernel events of interest may happen before a filesystem is = mounted.=20 > If Mach has such a great RPC interface, why do we need to hide it = behind > another abstraction layer like XPC? The usual reason we build abstractions over other abstractions. Same = reason you created a RPC library. Ask yourself why you created an rpc abstraction = on top of unix=20 domain sockets, and you will get the answer. > Monolithic kernels don't use > message passing, so we aren't missing out on much by not having this > feature. This is dogmatic and it misses the point of why kernel endpoints are = needed.=20 The situation has nothing to do with monolithic and microkernel = designs.=20 It allows any kernel component to pass messages to arbitrary = destinations. It is a generalised mechanism like devd. Good luck generalising that = with unix sockets. > There are some major benefits to > using sockets, such as the fact that are portable and relatively > standardized across different Unix implementations (and even Windows). And why is portability of such a subsystem relevant ? Its not like = anybody will jump to port launchd Windows :P Or even Linux. Nobody will care. >>=20 >> Receive messages directly in call to kevent() >>=20 >=20 > This is a "nice to have" performance optimization. We can live without = it. Tautology. We can live without sendfile(2). We can also live without = kqueue/kevent(2) themselves. We could have lived with DOS. > On 13 Jan 2016, at 05:08, Mark Heily wrote: >=20 > On Tue, Jan 12, 2016 at 10:59 AM, Hubbard Jordan > wrote: >>=20 >>> On Jan 12, 2016, at 4:59 AM, Konstantin Belousov = wrote: >>>=20 >>> I highly recommend to Google for "Mach IPC sucks" if reader is = really interested. >>=20 >> And here we return to the usual trap=E2=80=A6 >>=20 >> =E2=80=9CMach IPC sucks!=E2=80=9D >>=20 >> =E2=80=9COK. What do you propose that will address all of the same = concerns?=E2=80=9D >>=20 >> =E2=80=9Cdbus!=E2=80=9D >>=20 >> =E2=80=9C*Sigh*. You haven=E2=80=99t even looked at the two = technologies in any depth, have you? Go read the dbus wikipedia page, = at least! Unix domain sockets underneath, no kernel<->userland = communication path, no trusted IPC mechanism, no support for large = messages=E2=80=A6=E2=80=9D >>=20 >> =E2=80=9COK, so something new!! We should totally create an IPC for = the New Millennium!=E2=80=9D >>=20 >> =E2=80=9CThat would be you then? Where=E2=80=99s your white paper? = Where=E2=80=99s your reference implementation?=E2=80=9D >>=20 >> >>=20 >> Sorry. Been there, had this debate, and while it=E2=80=99s always = extremely easy to fling poop at an existing mechanism, it turns out = it=E2=80=99s so much harder to actually *create an alternative* that = this kind of discussion only serves to throw cold water on evolution = (=E2=80=9Cthe perfect being the enemy of the good enough=E2=80=9D) and = the end-result is that evolution does not occur. >>=20 >>=20 >=20 > You're right in that DBus is not a great solution, and I think we can = do > better. >=20 > I've gone ahead and created a general-purpose IPC library that is a > plausible alternative to Mach IPC. Here's the code: >=20 > https://github.com/mheily/libipc >=20 > It is not a bus, and it is not object-oriented. Instead, it tries to > make calling a remote function as easy and seamless as calling a > local function. I was planning on hacking on it for a few more weeks > before announcing it, but it seems very relevant to the discussion > we are having today. >=20 > It's built on top of Unix-domain sockets, and includes an IDL compiler > that takes a reasonably simple YAML file and generates the boring > code that allows programs to do structured IPC without worrying about > serialization and bootstrapping and all of the underlying protocol > issues. >=20 > You've raised some objections to Unix-domain sockets, which I'd like > to respond to. > I'll quote from the slide in your presentation about Unix-domain = sockets: >=20 > ------------------------------------------------------------ >=20 >>=20 >>=20 >> What can Mach ports do that Unix domain sockets can't? >>=20 >> Separate namespace for services (doesn't rely on file system naming = or >> permissions) >>=20 >=20 > This is generally a bad thing, IMHO. Filesystem permissions are a > good way to apply security rules to an object, and they are a highly > standard and well understood concept. If you need more than the > traditional user/group/other support, you can supplement this > with POSIX ACLs. >=20 > Using the filesystem as a way to coordinate activity is a perfectly > natural thing to do, and only has a disadvantage in a small corner = case > (where the system is in the process of booting up and the filesystem > is not mounted). >=20 > Right now, there is no need to perform IPC in the early part of the = boot > process. A machine where the root filesystem is not mounted read/write = is > basically unusable, and the solution is to mount the filesystem ASAP > before starting any services that rely on IPC. >=20 >>=20 >> Message boundaries >>=20 >=20 > Not true. You have to provide your own semantics for finding > the message boundaries, but it is totally possible to send messages > across Unix-domain sockets. >=20 >>=20 >> Kernel as peer >>=20 >=20 > This is true, but not there are other ways for userspace to = communicate > with the kernel, such as syscalls and ioctl. Monolithic kernels don't = use > message passing, so we aren't missing out on much by not having this > feature. >=20 >>=20 >> Pre-existing well defined RPC interface >>=20 >=20 > If Mach has such a great RPC interface, why do we need to hide it = behind > another abstraction layer like XPC? The existence of a thing is not > necessarily an argument for it being the right tool for the job. >=20 >>=20 >> Receive messages directly in call to kevent() >>=20 >=20 > This is a "nice to have" performance optimization. We can live without = it. >=20 >>=20 >> OOL (out of line) messages (arbitrarily sized with zero copy for = large messages) >>=20 >=20 > You can do zero-copy over Unix-domain sockets by sending a file = descriptor > across the socket that is backed by an anonymous region of memory > created with mmap(2). >=20 > To get the out-of-line behavior, all you need are two Unix-domain > sockets; one acting > as the control channel, and the other as the data channel. >=20 >>=20 >> Port send rights - can only send to a port for which the process has = explicitly >> received the right to send >>=20 >=20 > Unix-domain sockets have file permissions that control access. This is > functionally > equivalent to "port send rights", right? >=20 >>=20 >> Provenance - Yes, PROVENANCE, receiver can have the kernel append an = audit >> trailer containing full set of credentials of sender >>=20 >=20 > Unix-domain sockets allow you retrieve UID, GID, and PID of the client > process. Unless you mean something different by "PROVENANCE", this > is all the information you really need to make security decisions in > the current Unix security model. >=20 > ------------------------------------------------------------ >=20 > So the TL;DR is that you can use Unix-domain sockets to do > most of what Mach IPC provides. There are some major benefits to > using sockets, such as the fact that are portable and relatively > standardized across different Unix implementations (and even Windows). > I think if you do a cost-benefit analysis between Mach and sockets, > sockets win. >=20 > Anyway, I'm planning to implement IPC in relaunchd using libipc, > and relaunchd will be able to create the IPC service and launch > the job "on demand" whenever a client attempts to connect to it. >=20 > We've taken up a lot of airtime on this mailing list, so I invite > anyone who is interested in IPC to join me on the libipc mailing > list to discuss this further: >=20 > https://groups.google.com/d/forum/libipc-devel = > _______________________________________________ > freebsd-hackers@freebsd.org = mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers = > To unsubscribe, send any mail to = "freebsd-hackers-unsubscribe@freebsd.org = " From owner-freebsd-hackers@freebsd.org Wed Jan 13 07:05:21 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B803A80D66 for ; Wed, 13 Jan 2016 07:05:21 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (mail.ixsystems.com [69.198.165.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0F42D1189 for ; Wed, 13 Jan 2016 07:05:20 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1452668718-08ca042abd100d0002-P5m3U7 Received: from [10.11.111.236] (50-250-239-90-static.hfc.comcastbusiness.net [50.250.239.90]) by barracuda.ixsystems.com with ESMTP id wfaVbccPDfdRDjSa (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 12 Jan 2016 23:05:19 -0800 (PST) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-AUTH-User: jkh@ixsystems.com X-Barracuda-Apparent-Source-IP: 50.250.239.90 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3117\)) Subject: Re: relaunchd: a portable clone of launchd From: Hubbard Jordan X-ASG-Orig-Subj: Re: relaunchd: a portable clone of launchd In-Reply-To: Date: Tue, 12 Jan 2016 23:05:18 -0800 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <66E766F4-66D5-41E1-B6E7-18E218B3711F@ixsystems.com> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> To: Mark Heily X-Mailer: Apple Mail (2.3117) X-Barracuda-Connect: 50-250-239-90-static.hfc.comcastbusiness.net[50.250.239.90] X-Barracuda-Start-Time: 1452668719 X-Barracuda-Encrypted: ECDHE-RSA-AES256-SHA X-Barracuda-URL: https://10.2.0.41:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=8.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.26080 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2016 07:05:21 -0000 > On Jan 12, 2016, at 7:08 PM, Mark Heily wrote: >=20 > I've gone ahead and created a general-purpose IPC library that is a > plausible alternative to Mach IPC. Here's the code: >=20 > https://github.com/mheily/libipc Kudos for creating something for review, at least. It beats the usual = discussion-without-code and I won=E2=80=99t argue with the notion of = creating something new, though I would like to push back on at least a = few of the points you make about Mach IPC because they=E2=80=99re = important in understanding the full picture. >> Separate namespace for services (doesn't rely on file system naming = or >> permissions) >=20 > This is generally a bad thing, IMHO. Filesystem permissions are a > good way to apply security rules to an object, and they are a highly > standard and well understood concept. I think this is pretty subjective. Speaking as someone who has used a = services namespace for a long time, I would argue the exact opposite = with probably equal amounts of conviction and justification. I come = down firmly on the side of the argument that says it's a good thing and = Filesystem permissions are a terrible way of applying security rules to = an object, particularly as they only provide very limited semantics. = All of the ACLs in the world won=E2=80=99t help you to create an = extensible security trailer that identifies multiple clients of a single = server in a unique way (pid/uid/gid is practically useless for this). = However, I can=E2=80=99t force you to walk in the shoes of an OS X = internals developer (or even offer you that opportunity) so we=E2=80=99re = back to subjective and dueling opinions on this again. > Using the filesystem as a way to coordinate activity is a perfectly > natural thing to do, and only has a disadvantage in a small corner = case > (where the system is in the process of booting up and the filesystem > is not mounted). That=E2=80=99s actually not a corner case on the software appliances = we=E2=80=99ve been creating lately. That=E2=80=99s the usual case. :) = You start out with a bootstrap MFS and then selectively mount ZFS = datasets and such on top of various locations that would like to have = more storage (like /var/tmp or /var/db). This makes locating your Unix = domain sockets a bit tricky. Again, this is not just hypothetical - = we=E2=80=99ve run into multiple problems where we wound up covering up = our own mongodb / syslog sockets and hilarity ensued until we realized = the problem and sorted it out.=20 Again, I clearly can=E2=80=99t convince you otherwise, but a service = discovery mechanism built-in to the kernel is awesome and to say =E2=80=9C= you don=E2=80=99t need it=E2=80=9D is a bit like someone telling you = that you don=E2=80=99t need to make whatever amount of money you=E2=80=99r= e making but should be happy working as a night clerk at 7-11. Who is = anyone to make that kind of assertion without background? :) > Not true. You have to provide your own semantics for finding > the message boundaries, but it is totally possible to send messages > across Unix-domain sockets. Of course you can send messages over Unix-domain sockets. You can send = messages using a morse-code key and a Marconi spark-gap transmitter, for = that matter, but that doesn=E2=80=99t mean you don=E2=80=99t want = something else to take care of that for you. Mach IPC does, and = that=E2=80=99s all I meant on that slide. >> Kernel as peer >>=20 >=20 > This is true, but not there are other ways for userspace to = communicate > with the kernel, such as syscalls and ioctl. Yes, and they all suck and/or are implemented completely ad-hoc on a = case-by-case basis (the routing socket comes to mind) rather than having = an architecture make them conform to some common design patterns, both = for existing and for new mechanisms. I dunno, but it sure seems to me = like you=E2=80=99re going to such lengths to hate on Mach IPC that = you=E2=80=99re willing to essentially argue against architecture, clean = abstraction boundaries, and pretty much anything anyone finds valuable = in order to hold such a strong position. It=E2=80=99s sort of = reminiscent of various arguments I=E2=80=99ve had with assembly bigots = that high level languages are just a waste of time and CPU cycles and = there=E2=80=99s absolutely nothing worth doing that can=E2=80=99t be = done in assembly. I mean sure, they=E2=80=99re not out-and-out WRONG, = but the entire argument seems silly in the extreme. - Jordan From owner-freebsd-hackers@freebsd.org Wed Jan 13 07:55:09 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6292A6CBB6 for ; Wed, 13 Jan 2016 07:55:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C3B51877; Wed, 13 Jan 2016 07:55:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0D7sxhl017763 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 13 Jan 2016 09:54:59 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0D7sxhl017763 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0D7sxPs017751; Wed, 13 Jan 2016 09:54:59 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 13 Jan 2016 09:54:59 +0200 From: Konstantin Belousov To: Jonathan Anderson Cc: Hubbard Jordan , FreeBSD Hackers Subject: Re: relaunchd: a portable clone of launchd Message-ID: <20160113075459.GB72455@kib.kiev.ua> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <616D464E-0A67-4EF0-AB32-ADB7087D325B@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <616D464E-0A67-4EF0-AB32-ADB7087D325B@FreeBSD.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2016 07:55:10 -0000 On Tue, Jan 12, 2016 at 02:45:06PM -0330, Jonathan Anderson wrote: > On 12 Jan 2016, at 9:29, Konstantin Belousov wrote: > > I highly recommend to Google for "Mach IPC sucks" if reader is really > > interested. > > I did as you suggested and came up with: > http://www.googlefight.com/mach+ipc+sucks-vs-unix+ipc+sucks.php > > So says the Internet. So say we all. Very useful, I also did http://www.googlefight.com/apple-vs-orange.php to complete the case. From owner-freebsd-hackers@freebsd.org Wed Jan 13 08:08:46 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A48BA804C7 for ; Wed, 13 Jan 2016 08:08:46 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from udns.ultimatedns.net (static-24-113-41-81.wavecable.com [24.113.41.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 10CA012B4 for ; Wed, 13 Jan 2016 08:08:45 +0000 (UTC) (envelope-from bsd-lists@bsdforge.com) Received: from ultimatedns.net (localhost [127.0.0.1]) by udns.ultimatedns.net (8.14.9/8.14.9) with ESMTP id u0D7exlL045324 for ; Tue, 12 Jan 2016 23:41:05 -0800 (PST) (envelope-from bsd-lists@bsdforge.com) To: In-Reply-To: References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com>, From: "Chris H" Subject: Re: relaunchd: a portable clone of launchd Date: Tue, 12 Jan 2016 23:41:05 -0800 Content-Type: text/plain; charset=UTF-8; format=fixed MIME-Version: 1.0 Message-id: <0bd51315419aa2179b4f8319560d5340@ultimatedns.net> Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2016 08:08:46 -0000 Hello, Mark. Apologies for top posting, but I just wanted to say thanks for sharing this. A quick look at it. It appears promising. In the very least -- interesting. ;) Thanks again! --Chris On Tue, 12 Jan 2016 22:08:01 -0500 Mark Heily wrote > On Tue, Jan 12, 2016 at 10:59 AM, Hubbard Jordan wrote: > > > >> On Jan 12, 2016, at 4:59 AM, Konstantin Belousov > >> wrote: >> > >> I highly recommend to Google for "Mach IPC sucks" if reader is really > >> interested. > > > And here we return to the usual trap… > > > > “Mach IPC sucks!” > > > > “OK. What do you propose that will address all of the same concerns?” > > > > “dbus!” > > > > “*Sigh*. You haven’t even looked at the two technologies in any depth, > > have you? Go read the dbus wikipedia page, at least! Unix domain sockets > > underneath, no kernel<->userland communication path, no trusted IPC > > mechanism, no support for large messages…” > > > “OK, so something new!! We should totally create an IPC for the New > > Millennium!” > > > “That would be you then? Where’s your white paper? Where’s your > > reference implementation?” > > > > > > > Sorry. Been there, had this debate, and while it’s always extremely easy > > to fling poop at an existing mechanism, it turns out it’s so much harder to > > actually *create an alternative* that this kind of discussion only serves to > > throw cold water on evolution (“the perfect being the enemy of the good > > enough”) and the end-result is that evolution does not occur. > > > > > You're right in that DBus is not a great solution, and I think we can do > better. > > I've gone ahead and created a general-purpose IPC library that is a > plausible alternative to Mach IPC. Here's the code: > > https://github.com/mheily/libipc > > It is not a bus, and it is not object-oriented. Instead, it tries to > make calling a remote function as easy and seamless as calling a > local function. I was planning on hacking on it for a few more weeks > before announcing it, but it seems very relevant to the discussion > we are having today. > > It's built on top of Unix-domain sockets, and includes an IDL compiler > that takes a reasonably simple YAML file and generates the boring > code that allows programs to do structured IPC without worrying about > serialization and bootstrapping and all of the underlying protocol > issues. > > You've raised some objections to Unix-domain sockets, which I'd like > to respond to. > I'll quote from the slide in your presentation about Unix-domain sockets: > > ------------------------------------------------------------ > > > > > > > What can Mach ports do that Unix domain sockets can't? > > > > Separate namespace for services (doesn't rely on file system naming or > > permissions) > > > > This is generally a bad thing, IMHO. Filesystem permissions are a > good way to apply security rules to an object, and they are a highly > standard and well understood concept. If you need more than the > traditional user/group/other support, you can supplement this > with POSIX ACLs. > > Using the filesystem as a way to coordinate activity is a perfectly > natural thing to do, and only has a disadvantage in a small corner case > (where the system is in the process of booting up and the filesystem > is not mounted). > > Right now, there is no need to perform IPC in the early part of the boot > process. A machine where the root filesystem is not mounted read/write is > basically unusable, and the solution is to mount the filesystem ASAP > before starting any services that rely on IPC. > > > > > Message boundaries > > > > Not true. You have to provide your own semantics for finding > the message boundaries, but it is totally possible to send messages > across Unix-domain sockets. > > > > > Kernel as peer > > > > This is true, but not there are other ways for userspace to communicate > with the kernel, such as syscalls and ioctl. Monolithic kernels don't use > message passing, so we aren't missing out on much by not having this > feature. > > > > > Pre-existing well defined RPC interface > > > > If Mach has such a great RPC interface, why do we need to hide it behind > another abstraction layer like XPC? The existence of a thing is not > necessarily an argument for it being the right tool for the job. > > > > > Receive messages directly in call to kevent() > > > > This is a "nice to have" performance optimization. We can live without it. > > > > > OOL (out of line) messages (arbitrarily sized with zero copy for large > > messages) > > > You can do zero-copy over Unix-domain sockets by sending a file descriptor > across the socket that is backed by an anonymous region of memory > created with mmap(2). > > To get the out-of-line behavior, all you need are two Unix-domain > sockets; one acting > as the control channel, and the other as the data channel. > > > > > Port send rights - can only send to a port for which the process has > > explicitly received the right to send > > > > Unix-domain sockets have file permissions that control access. This is > functionally > equivalent to "port send rights", right? > > > > > Provenance - Yes, PROVENANCE, receiver can have the kernel append an audit > > trailer containing full set of credentials of sender > > > > Unix-domain sockets allow you retrieve UID, GID, and PID of the client > process. Unless you mean something different by "PROVENANCE", this > is all the information you really need to make security decisions in > the current Unix security model. > > ------------------------------------------------------------ > > So the TL;DR is that you can use Unix-domain sockets to do > most of what Mach IPC provides. There are some major benefits to > using sockets, such as the fact that are portable and relatively > standardized across different Unix implementations (and even Windows). > I think if you do a cost-benefit analysis between Mach and sockets, > sockets win. > > Anyway, I'm planning to implement IPC in relaunchd using libipc, > and relaunchd will be able to create the IPC service and launch > the job "on demand" whenever a client attempts to connect to it. > > We've taken up a lot of airtime on this mailing list, so I invite > anyone who is interested in IPC to join me on the libipc mailing > list to discuss this further: > > https://groups.google.com/d/forum/libipc-devel > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Wed Jan 13 09:26:38 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EFC1A80C0C for ; Wed, 13 Jan 2016 09:26:38 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from know-smtprelay-omc-6.server.virginmedia.net (know-smtprelay-omc-6.server.virginmedia.net [80.0.253.70]) by mx1.freebsd.org (Postfix) with ESMTP id E489B1A5E for ; Wed, 13 Jan 2016 09:26:37 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from [192.168.1.100] ([86.10.211.13]) by know-smtprelay-6-imp with bizsmtp id 5MRT1s0080HtmFq01MRTwc; Wed, 13 Jan 2016 09:25:27 +0000 X-Originating-IP: [86.10.211.13] X-Spam: 0 X-Authority: v=2.1 cv=P+nH/X0u c=1 sm=1 tr=0 a=SB7hr1IvJSWWr45F2gQiKw==:117 a=SB7hr1IvJSWWr45F2gQiKw==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=NLZqzBF-AAAA:8 a=IkcTkHD0fZMA:10 a=6I5d2MoRAAAA:8 a=5tJJrmGd2nTZD89FewQA:9 a=7Zwj6sZBwVKJAoWSPKxL6X1jA+E=:19 a=QEXdDO2ut3YA:10 a=XdyKOaxJwVsA:10 a=ZUGwP7LCt9cA:10 a=FSu5OgGmP5kA:10 Subject: nosh version 1.24 To: "supervision@list.skarnet.org" , FreeBSD Hackers , debian-user@lists.debian.org References: <54430B41.3010301@NTLWorld.com> <54B86FD5.3090203@NTLWorld.com> <554E53EF.4080600@NTLWorld.com> <554E93AF.3070709@NTLWorld.com> <556BA130.50708@NTLWorld.com> <55902328.8080602@NTLWorld.com> <55D5CFA2.5010402@NTLWorld.com> <55D8B9AC.6010209@NTLWorld.com> <56089268.6080007@NTLWorld.com> <56120D11.4080506@NTLWorld.com> <5636C75B.70000@NTLWorld.com> <5672BD8C.50303@NTLWorld.com> From: Jonathan de Boyne Pollard Message-ID: <569617F3.8000101@NTLWorld.com> Date: Wed, 13 Jan 2016 09:25:07 +0000 User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <5672BD8C.50303@NTLWorld.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2016 09:26:38 -0000 The nosh package is now up to version 1.24 . * http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html * https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project Minor items in this release include: * A fix for BSD keyboard layout import, that makes both "duml" and "ddia" be U+0308 for now. Technically, diaeresis and umlaut are distinguishable in Unicode decomposed forms (using U+034F). But for now, everything is simple unadorned combining diaeresis. * A few more service bundles, for DBMail and for sudo (which in its vanilla form puts its timestamp files in /var/lib instead of /var/run and needs a cleanup service -- see Debian Bug #786555). * Use of rtprio and idprio when converting system service units on FreeBSD/PC-BSD. * Improvements to the framebuffer video mode selection in user-space virtual terminals for FreeBSD/PC-BSD. It now comes up in the same display size as on Debian Linux on my test machines. * Doco and other fixes from user feedback on version 1.23. (I've already begun some further VirtualBox host adjustments, as we discussed, for 1.25.) There is one major item in this release. PC-BSD 10.2 =========== Until now, I'd been testing on a PC-BSD system that had been upgraded, with various contortions, from version 9. This was still using UFS filesystems, listed in /etc/fstab; which the external configuration import subsystem had been happily importing to native service bundles. Over Christmastide I installed a PC-BSD 10.2 system from scratch, discovering some interesting oddities. These included installation failing if you tell it that you are in the United Kingdom using a U.K. keyboard (PC-BSD Bug #12986); and the GRUB menu editor, as configured by the installer, operating in a truly eye-stretching 46 column by 28 row mode (by my count), and not displaying the underscore character correctly. The important thing to know is that PC-BSD has for some time (at least since 2013) been ZFS-only, as far as installation goes. (One can of course mount other filesystem types after installation.) As Henry Ford might have said "Any customer can install to any filesystem type that xe likes, as long as it is ZFS.". The result is that if installing from scratch one gets a whole load of ZFS datasets, and an empty (save for /proc and swap) /etc/fstab file. So the major push for version 1.24 has been to get the configuration import system to deal with this, which it now does. It will create mount services for all ZFS mounts, enable the ones that are "on", give them an inter-service ordering, and deal with the special-casing for the root (which the installer, oddly, marks as not automatically mounted, even though it of course is). Alongside this, a large chunk of the remaining NetBSD rc.d services, from the on-going project to entirely replace them, have been crossed off the list. These include mfs for /tmp, static networking and static ARP, pefs, serial port BPS and framing setup, ppp, rfcomm_pppd, persistent "entropy" for the randomness subsystem, and ipfw. The progress of this work has been open from the start, and you can follow along on the roadmap WWW page. Indeed, you can even join in, if you can convert any of the remaining few items. There's more work to be done. But I now have ZFS-only PC-BSD 10.2 running nosh system-managed and service-managed. Some notes for those eager to follow: * Yes; I'm working on a pcdm service. No; it doesn't help that it's undocumented. Yes; that hoopla and palaver with forked subshells and multiple while loops calling sleep is exactly the sort of thing that proper service management is intended to obviate. * If you have problems with devd, stale nologin from previous boots, and other things that use /var/run, it's because the convert_varrun service isn't enabled and your system has not been thus or otherwise migrated to /run. This will be properly enabled by a preset in the next version. Enable it and reboot. Or just start it and reboot. Or just boot into rescue mode and turn /var/run into a symbolic link to /run yourself. * No; the nosh-run-system-manager package doesn't work properly on PC-BSD, as it does on vanilla FreeBSD. PC-BSD 10.2 doesn't use the FreeBSD boot loader, like my old upgraded installation of PC-BSD 9 did. It uses GRUB. The PC-BSD people apparently plan to get rid of GRUB in the future, and use the FreeBSD loader once more. So this problem goes away when GRUB does. (-: In the meantime, use 'set kFreeBSD.init_path="/sbin/service-manager"' in the GRUB configuration. * The root-resizing subsystem that was new to FreeBSD version 10 still needs conversion. But ironically it doesn't work on PC-BSD 10.2 in the first place. It can only grow UFS volumes, and PC-BSD's root is not a UFS volume. From owner-freebsd-hackers@freebsd.org Thu Jan 14 13:40:49 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41C4AA813A3 for ; Thu, 14 Jan 2016 13:40:49 +0000 (UTC) (envelope-from mark@heily.com) Received: from mail-io0-x230.google.com (mail-io0-x230.google.com [IPv6:2607:f8b0:4001:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1205F1724 for ; Thu, 14 Jan 2016 13:40:48 +0000 (UTC) (envelope-from mark@heily.com) Received: by mail-io0-x230.google.com with SMTP id q21so452853840iod.0 for ; Thu, 14 Jan 2016 05:40:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heily-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=JfckE2xv5EUekELgvnTkSLi/gdF0bagHXsDLhGWzYSM=; b=fjpwUzQVtHx345AC4JN9zCCSakO+x/Yb4ITbJqgnywqaj6bOjI5VKuiuo7b/UpkJ8o Ze1jxgGbVxNPAJc0wI3OyiWZu0f/Plk4YFefUU1md396fQjpVq8duMnd87jEg9IDXxhK 9zn7lvGLHUR3KZpcN6uvdQBYMM9cFjiGiMz8nYz+SxbuaBu9DS4d+z2pyt9a2x7KwPns E0wVEv7kbbkGGKr2dgMLm7tJjg0JMYcsbMzczdqukUD28W4JMDVzKlY0Tk7GSu8co/f/ a4+Ls8oJ8s3jfpI0VVTCGFN0YQpdytL9XdaiPddVpG3bcReGSuIPi8/mDQZExGWU/rpw O4NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=JfckE2xv5EUekELgvnTkSLi/gdF0bagHXsDLhGWzYSM=; b=bhhMWlbUW1IHkURy34fbKwtbCUA1nFk+yN+MmQ6kYXB1xOpPe/XRt+vpxVO+BoVi3M UWV8goE7dYcUL4XJ3yh2MT0WZolwrDzPbgECYK/kwGGHavnWA7o4IUXl9GeAwKDSfVmn WnzxgVWymKN93yLWdDI5a++e4g70qZWjJmS+oqsLgSZh8vRrhCX+n7DrzLkZGsI5yVDg HAOGCd8dHLs+qkOeREKxp1aUvWYCqzwgUQMkfEMCEHl1gVtKBx7MQVNd/2srqR7fpQ3d AfSxKSt4/PsP+wXFMK8DXZT9wc0qdNmDY2NXrcI0UKuu//qPJY8gWp/ntIjpiLS90g7B 6JUw== X-Gm-Message-State: ALoCoQn+PGEffGRq5IFbj6X7p4nniQf3vD1M3HjsS/bJ4uYAAHFDRdDRNd22MZ8an0Plv6jvSS0OuIcJrfdmqanXctdI8Vwt8g== MIME-Version: 1.0 X-Received: by 10.107.43.138 with SMTP id r132mr5397826ior.7.1452778848035; Thu, 14 Jan 2016 05:40:48 -0800 (PST) Received: by 10.79.34.196 with HTTP; Thu, 14 Jan 2016 05:40:47 -0800 (PST) X-Originating-IP: [71.70.175.250] In-Reply-To: <66E766F4-66D5-41E1-B6E7-18E218B3711F@ixsystems.com> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> <66E766F4-66D5-41E1-B6E7-18E218B3711F@ixsystems.com> Date: Thu, 14 Jan 2016 08:40:47 -0500 Message-ID: Subject: Re: relaunchd: a portable clone of launchd From: Mark Heily To: Hubbard Jordan Cc: FreeBSD Hackers Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2016 13:40:49 -0000 On Wed, Jan 13, 2016 at 2:05 AM, Hubbard Jordan wrote: > > All of the ACLs in the world won=E2=80=99t help you to create an extensi= ble security trailer that identifies multiple clients of a single server in= a unique way (pid/uid/gid is practically useless for this). Do you have any specific examples of how an "extensible security trailer" would be used? Even better, can you demonstrate that Mach is the only way to implement this concept? Generally speaking, libipc is extensible because it is an abstraction layer and the wire protocol is not exposed directly to clients or servers. I could offer the example of extending the system to support MAC labels: if there was a requirement for the server to be able to check the MAC label of a client, one could extend the API to add a function named ipc_client_get_mac_label() that would do the needful under the covers. >> Using the filesystem as a way to coordinate activity is a perfectly >> natural thing to do, and only has a disadvantage in a small corner case >> (where the system is in the process of booting up and the filesystem >> is not mounted). > > That=E2=80=99s actually not a corner case on the software appliances we= =E2=80=99ve been creating lately. That=E2=80=99s the usual case. :) You = start out with a bootstrap MFS and then selectively mount ZFS datasets and = such on top of various locations that would like to have more storage (like= /var/tmp or /var/db). This makes locating your Unix domain sockets a bit = tricky. Again, this is not just hypothetical - we=E2=80=99ve run into mult= iple problems where we wound up covering up our own mongodb / syslog socket= s and hilarity ensued until we realized the problem and sorted it out. > Well, unless you have modified MongoDB to use Mach instead of Unix-domain sockets, I think we are talking about two different problems. Mach is not going to solve the problem you are talking about. My original comment was in the context of comparing libipc (which is socket-based) to Mach IPC. There is a legitimate advantage to Mach IPC in that it is available very early in the boot process, before the root filesystem is mounted. Once the root filesystem is mounted, the libipc service discovery directory will live under /var/run/ipc, which probably will never be covered up by another filesystem as you describe. OTOH, if everything were rewritten to take advantage of libipc for service discovery, rather than each program using their own ad-hoc sockets in /var/wherever, the kind of problem you are talking about goes away. That's why I'm making libipc portable, in the hope that it becomes ubiquitous. > Again, I clearly can=E2=80=99t convince you otherwise, but a service disc= overy mechanism built-in to the kernel is awesome and to say =E2=80=9Cyou d= on=E2=80=99t need it=E2=80=9D is a bit like someone telling you that you do= n=E2=80=99t need to make whatever amount of money you=E2=80=99re making but= should be happy working as a night clerk at 7-11. Who is anyone to make t= hat kind of assertion without background? :) > I never said "you don't need it". In fact, I've written libipc to provide a service discovery mechanism built-in to the kernel. It uses the standard open(2) syscall instead of Mach, but the intent is similar. >> Not true. You have to provide your own semantics for finding >> the message boundaries, but it is totally possible to send messages >> across Unix-domain sockets. > > Of course you can send messages over Unix-domain sockets. You can send m= essages using a morse-code key and a Marconi spark-gap transmitter, for tha= t matter, but that doesn=E2=80=99t mean you don=E2=80=99t want something el= se to take care of that for you. Mach IPC does, and that=E2=80=99s all I m= eant on that slide. If that's what you want, datagram sockets will preserve message boundaries for you. I selected stream sockets for libipc because I wanted to the ability to detect when the client exits, and be able to clean up resources on the server side. Honestly Jordan, this is such a poor argument you should probably take it off your slide and focus on more compelling advantages of Mach. > I dunno, but it sure seems to me like you=E2=80=99re going to such length= s to hate on Mach IPC that you=E2=80=99re willing to essentially argue agai= nst architecture, clean abstraction boundaries, and pretty much anything an= yone finds valuable in order to hold such a strong position. > It=E2=80=99s sort of reminiscent of various arguments I=E2=80=99ve had w= ith assembly bigots that high level languages are just a waste of time and = CPU cycles and there=E2=80=99s absolutely nothing worth doing that can=E2= =80=99t be done in assembly. I mean sure, they=E2=80=99re not out-and-out = WRONG, but the entire argument seems silly in the extreme. > I'm disappointed that you would resort to this level of ad-hominem attack. I've tried to keep this a purely technical discussion, and I'm not going to question your personal motives or integrity. Please offer me the same courtesy. - Mark From owner-freebsd-hackers@freebsd.org Thu Jan 14 17:00:42 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57FE8A81B43 for ; Thu, 14 Jan 2016 17:00:42 +0000 (UTC) (envelope-from jkh@ixsystems.com) Received: from barracuda.ixsystems.com (mail.ixsystems.com [69.198.165.135]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3876F1B11 for ; Thu, 14 Jan 2016 17:00:41 +0000 (UTC) (envelope-from jkh@ixsystems.com) X-ASG-Debug-ID: 1452790840-08ca042abd561880001-P5m3U7 Received: from mail.iXsystems.com ([10.2.55.1]) by barracuda.ixsystems.com with ESMTP id JiwehFykdnRAueoy (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 14 Jan 2016 09:00:40 -0800 (PST) X-Barracuda-Envelope-From: jkh@ixsystems.com X-Barracuda-RBL-Trusted-Forwarder: 10.2.55.1 X-ASG-Whitelist: Client Received: from [10.8.0.50] (unknown [10.8.0.50]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 8508DA2095; Thu, 14 Jan 2016 09:00:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ixsystems.com; s=newknight0; t=1452790840; bh=WZyFU1n9ejIjfOKUZ9zwjDrLmIbqZtRIUMsSLUmOo74=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=IufKSlPHC+F3RQA/Gwo+g3fg+BmIrmgLUI/TI06qyDokaA37WX1Hebmz167GzphjT xKhYn7w49ApF9jc+hs/V6pvvHez65OLnUJ2l/UAGGt/4Pg+qf6sSILgKNTmyjpziCf oSm0IdaarJbzCeWTA7L694AoBdlcLV4RppT50s8Q= Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3117\)) Subject: Re: relaunchd: a portable clone of launchd From: Hubbard Jordan X-ASG-Orig-Subj: Re: relaunchd: a portable clone of launchd In-Reply-To: Date: Thu, 14 Jan 2016 09:00:39 -0800 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <627C5AFF-6757-404D-AF6B-A27ECF19B555@ixsystems.com> References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> <66E766F4-66D5-41E1-B6E7-18E218B3711F@ixsystems.com> To: Mark Heily X-Mailer: Apple Mail (2.3117) X-Barracuda-Connect: UNKNOWN[10.2.55.1] X-Barracuda-Start-Time: 1452790840 X-Barracuda-Encrypted: DHE-RSA-CAMELLIA256-SHA X-Barracuda-URL: https://10.2.0.41:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at ixsystems.com X-Barracuda-BRTS-Status: 1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2016 17:00:42 -0000 > On Jan 14, 2016, at 5:40 AM, Mark Heily wrote: >=20 >=20 > Do you have any specific examples of how an "extensible security > trailer" would be used? securityd in OS X and how it=E2=80=99s part of the cryptographically = signed binary authentication mechanism (where only executables with = specific signatures can talk to other trusted services). You have to = have an un-spoofable and controllable startup process without race = conditions in the filesystem to do that kind of trusted IPC in a way = that=E2=80=99s =E2=80=9Cunbreakable enough=E2=80=9D to base the rest of = your security architecture on it. Again, I cannot give you direct experience with one of the oldest and = most widely deployed Mach IPC-based technologies in the world today, = that=E2=80=99s something you have to get for yourself. > Even better, can you demonstrate that Mach is > the only way to implement this concept? Of course it=E2=80=99s not the *only* way (one could arguably just = redesign something very similar to Mach but not Mach) but again, Mach = IPC already exists. Today. It=E2=80=99s been tested and vetted for = years. Any new solution would have to go through the same process, and = I certainly don=E2=80=99t see the win (or wisdom) of doing something = like that. > I'm disappointed that you would resort to this level of ad-hominem > attack. If you think that was an ad-hominem attack, you clearly have never = actually experienced one. :) I made no comments whatsoever about your = character, as an ad-hominem attack would require, but specifically said = that your arguments went to such lengths to dismiss Mach IPC as a = technology that it was like arguing with someone with such a strong bias = for some other technology (my analogy being programming languages) that = arguing was pointless, and I stand by that assertion since it so very = clearly is that, pointless. You are absolutely *determined* to rewrite things that already exist, = and that=E2=80=99s not =E2=80=9Can ad-hominem attack=E2=80=9D but a = simple observation of the facts, Mark! I=E2=80=99ve been telling you = that for some time, and your answers have always consistently added up = to =E2=80=9Cbut I don=E2=80=99t like those technologies, so I=E2=80=99m = going to do my own!=E2=80=9D and that=E2=80=99s FINE, it=E2=80=99s = absolutely something you are totally free to do, but when you go further = and try to paint your highly subjective preferences as somehow = objectively =E2=80=9Cbetter=E2=80=9D, I get annoyed because unlike you, = I can objectively point to a multi-year track record for the = technologies I=E2=80=99m championing and also make the rather = unassailable observation they already exist and have had their security = attack surfaces vetted by literally a cast of thousands, if not = millions. Those are objective truths, not subjective opinion. You=E2=80=99re not changing my mind and I=E2=80=99m obviously not = changing yours, however, so I think there would be nothing = =E2=80=9Cad-hominem=E2=80=9D about stating that this discussion in = Hackers has probably ceased to be interesting or enlightening to anyone, = though perhaps we=E2=80=99ve sold some popcorn. - Jordan From owner-freebsd-hackers@freebsd.org Thu Jan 14 20:55:37 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E3E8A823B6 for ; Thu, 14 Jan 2016 20:55:37 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from know-smtprelay-omc-10.server.virginmedia.net (know-smtprelay-omc-10.server.virginmedia.net [80.0.253.74]) by mx1.freebsd.org (Postfix) with ESMTP id E40E510AA for ; Thu, 14 Jan 2016 20:55:36 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from [192.168.1.100] ([86.10.211.13]) by know-smtprelay-10-imp with bizsmtp id 5wuS1s00T0HtmFq01wuS0f; Thu, 14 Jan 2016 20:54:26 +0000 X-Originating-IP: [86.10.211.13] X-Spam: 0 X-Authority: v=2.1 cv=MJad45tl c=1 sm=1 tr=0 a=SB7hr1IvJSWWr45F2gQiKw==:117 a=SB7hr1IvJSWWr45F2gQiKw==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=NLZqzBF-AAAA:8 a=IkcTkHD0fZMA:10 a=Z_7HjWGDEwx0gqjCmQsA:9 a=QEXdDO2ut3YA:10 Subject: Re: relaunchd: a portable clone of launchd References: <5687D3A9.5050400@NTLWorld.com> To: FreeBSD Hackers From: Jonathan de Boyne Pollard Message-ID: <56980AED.8090602@NTLWorld.com> Date: Thu, 14 Jan 2016 20:54:05 +0000 User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2016 20:55:37 -0000 Mark Heily: > [...] I'm thinking of using D-Bus as the RPC mechanism for relaunchd, > since a lot of open source programs are already using D-Bus. Jonathan de Boyne Pollard: > I recommend, to anyone going down this route, looking towards > finishing systembsd, [...] Mark Heily: > I strongly disagree with your recommendation to adopt DBus and systemd > as core components of FreeBSD. Ahem! From owner-freebsd-hackers@freebsd.org Thu Jan 14 21:25:26 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3991DA82D2D for ; Thu, 14 Jan 2016 21:25:26 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from mail.rdsor.ro (mail.rdsor.ro [193.231.238.10]) by mx1.freebsd.org (Postfix) with ESMTP id C69E311A2 for ; Thu, 14 Jan 2016 21:25:25 +0000 (UTC) (envelope-from dan_partelly@rdsor.ro) Received: from [192.168.1.100] (unknown [79.117.100.196]) by mail.rdsor.ro (Postfix) with ESMTP id A36321FACB; Thu, 14 Jan 2016 23:25:17 +0200 (EET) Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: relaunchd: a portable clone of launchd From: Dan Partelly In-Reply-To: Date: Thu, 14 Jan 2016 23:25:17 +0200 Cc: Hubbard Jordan , FreeBSD Hackers Message-Id: References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> <70975696-3E07-48B9-BFD1-3C2F51E715BB@icloud.com> <76E6AF2A-917B-41EB-883A-C27AB2BB9F71@ixsystems.com> <20160112125948.GH3625@kib.kiev.ua> <1D6BDF3C-28E7-40C4-A8A2-3A914A3CC76B@ixsystems.com> <66E766F4-66D5-41E1-B6E7-18E218B3711F@ixsystems.com> To: Mark Heily X-Mailer: Apple Mail (2.3112) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2016 21:25:26 -0000 > On 14 Jan 2016, at 15:40, Mark Heily wrote: >=20 > My original comment was in the context of comparing libipc (which is > socket-based) to Mach IPC. You can compare your library to dbus, as it sits at the same abstraction = level. You can=E2=80=99t really compare it to Mach IPC. API. What you = can do is compare Unix sockets to Mach Ports if you want. You basically = try to rewrite dbus.=20 > That's why I'm making libipc portable, in the hope that it > becomes ubiquitous. It wont.=20 First of all, because dbus is already ubiquitous , and second, because = it doesn't really solves any problems which ain=E2=80=99t already = solved, and doesn't bring enough innovation to be adopted widely . It = is not available early in boot, it doesn't support kernel endpoints, = it=E2=80=99s simply yet another abstraction over unix sockets :P There = are dozens of those.=20 Besides, right now, when I need to do IPC over Unix Domain sockets I = can do it very simply with the help of libnv(3) from FreeBSD 11. It is = part of the OS, in base, it is extremely simply to use, it doesn't = require me to install any 3rd party service like your library. From owner-freebsd-hackers@freebsd.org Fri Jan 15 02:41:30 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B7A6A82FD5 for ; Fri, 15 Jan 2016 02:41:30 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6EFC1763 for ; Fri, 15 Jan 2016 02:41:29 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22a.google.com with SMTP id u188so6433299wmu.1 for ; Thu, 14 Jan 2016 18:41:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=jO8qJX4dt9lE7MtLvsB5ynrY91e8CpMC9i321SneD88=; b=idnFe7vSOryI8n224Qp68MCEbh/IRZYTrPp5FEWZwUEywOlmyxfnaJzGH9ZDWHc5ms pDD4ahQ1dmAN0l9nEIJIrQIf9juUTpA9LecF0R6JBzFLbA4lmJq6kwD84TeVse7J9gRQ SQf39MDP4xmDqngMO3a+2PdRJK3B1okRODCHE72g5o3LncB8OGHiE+X1gYxM56uqXuQP zTEQJEkrypU5ftBIykc6xDxRIzOpUi3MglwWTr00LFVRE9A0sEUrBhA7LvPW9dSrFLsk LMeo1+EbxYk8hgF+NlnT6Qm3QLjylJxKwH9IP+Lmzt9SQh9AL/hH4xRYZkbF9j6RvFe8 vMhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=jO8qJX4dt9lE7MtLvsB5ynrY91e8CpMC9i321SneD88=; b=EpD/Sh7FNf1qJ1chFq2GGM+2QfLdYOmJdO3SdZgmj6ih0ObY49YB1L8/boVM2ajZ4v 4ROLgffKcST8BFEpKLinmNvhabsM7o+6KBGoPPdwmmQII/cSHw5K/U+zE9E9GoLWyn6Z juTS0fXWWd8vSmkxAfJ2AAbyFoOeia5Qi4BAnHLY7uw0PT3zjmzThMblH/2OQeXVgxBD SRkIu5PSBOxlck03tUXtQPMT5K+3aMSXoFeKVxyZYN+t6J9CrnOTCreCeoQ83WkJV9TN T9YAOoIxs7MuM2hZrKPuLWaHz7e8crm2+/nY7D+/XzbVzqZAg+aNcRkhggfom5vBOU1s 4PDw== X-Gm-Message-State: ALoCoQnwhNtklBn+y/XBRRQeC8JklORIx3drRWdS+jPowFrboOM7JMAsyUhyI+F05zQUupGb1zjhH5W7lFVM+xN8mFB3rdu1WA== X-Received: by 10.194.111.232 with SMTP id il8mr8530847wjb.150.1452825688326; Thu, 14 Jan 2016 18:41:28 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id f4sm4987762wjw.0.2016.01.14.18.41.27 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 14 Jan 2016 18:41:27 -0800 (PST) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: Eric McCorkle , Gabor Radnai , freebsd-hackers@freebsd.org References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> From: Steven Hartland Message-ID: <56985C6A.6040209@multiplay.co.uk> Date: Fri, 15 Jan 2016 02:41:46 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 02:41:30 -0000 Just wanted to let everyone know that I just finished committing these changes to the tree. Huge thanks to Eric's for his work on this, as well as everyone else who contributed. I've set the target for MFC of 2 weeks, so I hope to be able to get this into stable/10 before the 10.3 slush, so if you're interested in this change please test a head build > r294068 ASAP. Regards Steve From owner-freebsd-hackers@freebsd.org Fri Jan 15 09:42:04 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A41EA8451F for ; Fri, 15 Jan 2016 09:42:04 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from know-smtprelay-omc-10.server.virginmedia.net (know-smtprelay-omc-10.server.virginmedia.net [80.0.253.74]) by mx1.freebsd.org (Postfix) with ESMTP id AFC151CA1 for ; Fri, 15 Jan 2016 09:42:03 +0000 (UTC) (envelope-from j.deboynepollard-newsgroups@ntlworld.com) Received: from [192.168.1.100] ([86.10.211.13]) by know-smtprelay-10-imp with bizsmtp id 69i11s00V0HtmFq019i12x; Fri, 15 Jan 2016 09:42:01 +0000 X-Originating-IP: [86.10.211.13] X-Spam: 0 X-Authority: v=2.1 cv=MJad45tl c=1 sm=1 tr=0 a=SB7hr1IvJSWWr45F2gQiKw==:117 a=SB7hr1IvJSWWr45F2gQiKw==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=NLZqzBF-AAAA:8 a=IkcTkHD0fZMA:10 a=6I5d2MoRAAAA:8 a=_jgbcvs7w5xGjIQfmyQA:9 a=w7bNJm8QP8DlYTSS:21 a=PvWC23NGTYLYPMNX:21 a=QEXdDO2ut3YA:10 Subject: Re: nosh version 1.9 References: <54430B41.3010301@NTLWorld.com> <5443191E.5050208@mu.org> From: Jonathan de Boyne Pollard To: FreeBSD Hackers Message-ID: <5698BED3.9010102@NTLWorld.com> Date: Fri, 15 Jan 2016 09:41:39 +0000 User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 09:42:04 -0000 Jonathan de Boyne Pollard: > nosh version 1.9 is out. [...] > It's worth noting that OpenBSD 5.6 now specifies that > /etc/rc.conf{,.local} doesn't have shell expansions and isn't > necessarily sourced by a shell, [...] Alfred Perlstein: > Very cool. > > Wondering about the idea of /etc/rc.conf *not* being a shell script... > this is sort of bad imo as I can't see any other way to provide the > settings dynamically for the startup scripts at a glance. > > I'll give you an example... FreeNAS (and by extension the appliance we > are building at Norse) has /etc/rc.conf.local as a shell script that > pulls data from an sqlite database, this allows us to set various > services on/off based on the contents of that sqlite database file. > This in turn allows us to leverage most of the existing /etc/rc.d and > by extension the /usr/local/etc/rc.d files provided by ports. > > I'm wondering how one could still do that if /etc/rc.conf and > /etc/rc.conf.local were no longer scripts? Adrian Chadd: > The same way /etc/rc.conf and /etc/rc.conf.local is pulled in - via > the little snippet of stuff at the end of /etc/defaults/rc.conf , and > this bit of config in that file: > > local_startup="/usr/local/etc/rc.d" # startup script dirs. > script_name_sep=" " # Change if your startup scripts' names contain > spaces > rc_conf_files="/etc/rc.conf /etc/rc.conf.local" > > So, we just need some method of pulling in environment variables in > whatever order we need from whatever place we need. (God, why do I > know this stuff? Then I remembered - > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=17595 . God damnit.) > The tricky bit is trying to make it so we don't call sqlite like a > thousand times to pull out all of the environment variables for each > invocation of an rc script. The Introduction to the nosh Guide (q.v.) states the maxim that bootstrap time is not build again and again time, and this is a good example of where that maxim should be put into action. The time to run pwd_mkdb is not again and again whenever the password database is consulted, or at system bootstrap; it is after vipw has run the editor. And if one is generating /etc/rc.conf{,.local} from some source, such as a database, then the same principle applies. /etc/rc.conf{,.local} should not contain shell code that is parsed and executed again and again. The code to pull things out of an sqlite database and write out /etc/rc.conf{,.local} belongs at the end of the content update process, whatever that is (be it some TUI administrator settings tool or a daemon that provides an RPC interface for managing system settings to flashy GUI desktop widgets), for the sqlite database. There are many good things that fall out of this approach. Here are just some. /etc/rc.conf{,.local} don't need to be on a writable filesystem until the sqlite database itself is modifiable. The approach of making /etc/rc.freenas or /etc/rc.conf{,.local} or some other part of the bootstrap auto-update /etc/rc.conf{,.local} whenever it detects a database that is newer: does, however. (And as M. Chadd points out, the alternative approach of not writing out /etc/rc.conf{,.local} and simply pulling shell variables straight from the database has the disadvantages of running the database client again and again, and of needing it in rescue and emergency modes.) /etc/rc.conf{,.local} don't have to contain anything that requires a shell, just plain name=value assignments (with a small number of lexing rules for quoting) that can be parsed in perl, Python, C++, or a language of one's choice. One can see this design used in nosh system management itself from nosh 1.17 onwards. /etc/system-control/convert/ contains a suite of redo scripts that do a similar form of settings import. As you can see, it doesn't run over and over at bootstrap. It's designed to be run at various points *when the system administrator is changing settings*, using redo to check whether source files have changed and update the relevant target files, and service bundle settings, when they have. So running > rcctl get mdmfs@-tmp which is another way of saying > system-control print-service-env mdmfs@-tmp one obtains (on one of my systems) > size="20m" > flags="-S" These settings for the mdmfs@-tmp service, contained in the service bundle for that service, come directly from tmpsize and tmpmfs_flags in /etc/defaults/rc.conf (on that particular system). They are not regenerated at bootstrap. They aren't stored in shell script that is parsed and executed repeatedly. They aren't surrounded by a large number of additional variables unrelated to that service bundle, that have to be parsed again and again only to be then thrown away. They are in a daemontools-style envdir directory that is specific to the service bundle; and they are the only settings that that service bundle needs, and thus the only ones that get imported to that service bundle. Interestingly, the nosh configuration import system could be extended to replacing that FreeNAS mechanism. Notice that it already builds a hosts.conf from nsswitch.conf. This is a similar sort of exercise. I expect that the configuration import system could easily be extended, or copied, to build an rc.conf{,.local} from an sqlite database. One would need little more than an rc.conf.do and an rc.conf.local.do that executed some sqlite SELECTs and massaged the results appropriately; code to do which, you already have from FreeNAS. One wouldn't even need the current FreeNAS's hairy shell logic in rc.conf.local for repeatedly taking checksums of the database file(s) to see whether it has been modified since last time (even though the containing filesystems might be mounted read-only and there is no possibility of such a change). That comes free with the redo system. Put another way: All of that stuff comes out of FreeNAS rc.conf.local, which goes back to being a plain data file containing nothing but variable=value settings, and (minus the parts that come for free with redo) goes into (say) /etc/system-control/convert/rc.conf.local.do . You lose the need for /var/tmp/rc.conf.freenas entirely. You thus lose the need for a writable /var . And the data import process doesn't run at bootstrap, again and again. It runs from the system installer, from the flashy GUI tool that an administrator uses for adjusting the stuff on the sqlite database, from the humble TUI tools that do the same, or from (even) whatever DBus-speaking freenas-settings-d daemons you might care to dream up for speaking to the KDE and GNOME desktops and their ilk. From owner-freebsd-hackers@freebsd.org Fri Jan 15 10:36:33 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DC8AA8377C for ; Fri, 15 Jan 2016 10:36:33 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 40C1F138D for ; Fri, 15 Jan 2016 10:36:32 +0000 (UTC) (envelope-from bright@mu.org) Received: from Alfreds-MacBook-Pro-2.local (unknown [IPv6:2601:645:8001:cee1:e439:6c98:56dd:ba72]) by elvis.mu.org (Postfix) with ESMTPSA id D35D7345A971; Fri, 15 Jan 2016 02:36:25 -0800 (PST) Subject: Re: relaunchd: a portable clone of launchd To: Dan Partelly , Peter Beckman References: <5687D3A9.5050400@NTLWorld.com> <817860B6-5D67-41A3-ADD7-9757C7E67C35@gmail.com> <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> Cc: FreeBSD Hackers , Jonathan de Boyne Pollard , Dmitry Sivachenko , Mark Heily From: Alfred Perlstein Message-ID: <5698CBA8.8070501@mu.org> Date: Fri, 15 Jan 2016 02:36:24 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <07D83705-D89F-4125-B57B-920EDEBC8A85@rdsor.ro> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 10:36:33 -0000 +1 :) On 1/10/16 2:36 AM, Dan Partelly wrote: > Copying the linux way of doing things should not be a goal of the BSDs. It is enough that unfortunately we are forced > into Linuxisms and associated wrappers to support modern GPUs. Understandable , given how few ppl work on BSDs, > and how little code contributions do the BSDs receive from the massive enterprises they power (with > some notable exceptions) Let me use this opportunity to thank Juniper for the glorified printf system > they contributed to FreeBSD . > > Can the BSDs go forward with rc systems alone ? Sure they can , at least for the time beeing, and we will > continue to use them. But innovation is desirable. > > Systemd might be a terrible implementation or not (I dont know, I dont use it) but the ideas behind it are sane. > > rc systems are indeed robust, but they should be ancient history. They are nothing but glorified autoexec.bat systems. > Modern OSes need sophisticated dynamic service management systems, fault management, transactional OS > configuration databases, centralised event systems supporting kernel sources. > > > This is the problem domain things like sytemd and dbus are tring to solve. They might doit the wrong way, I personally think > the direction Solaris took to solve some of those problems is the way to go, but at least they are trying to do something, and > they clearly explored the problem space. > > > Meanwhile here, some engineers are trying to change the FreeBSD OS configuration to a new DSL, but without any consideration for > issues of centralising OS databases and add innovation like transactions and full concurrency safety. > > YOu gotta ask yourself, since it is only a language change, why even doit ????? It adds no technical innovations, the new > systems are not well enough thought out to support technical innovation added incrementally later. So why are they doing it ? > To change the DSL only ? By now all BSDs user are familiar with all adhoc databases the OS offer. We are familiar (experts, even) with > the language they use. Changing this language , when no technical innovation is present, is , in my opinion, ill-advised. > > It is change for the sake of change, it is change because “someone wrote the code”, not because it solves any real problem , or is a well > thought out engineering solution. I really hope someone from the developers wakes up and vetoes those changes for the sake of change, > like Junipers libxo, and attemtps to change the DSLs for the sake of changing the DSL. > > > >> On 08 Jan 2016, at 17:20, Peter Beckman wrote: >> >> On Thu, 7 Jan 2016, Dmitry Sivachenko wrote: >> >>>> On 07 Jan 2016, at 05:12, Mark Heily wrote: >>>> >>>> On Sat, Jan 2, 2016 at 8:42 AM, Jonathan de Boyne Pollard >>>> wrote: >>>>> I recommend, to anyone going down this route, looking towards finishing >>>>> systembsd, especially instead of inventing a wholly new suite of protocols. >>>>> >>>>> * https://uglyman.kremlin.cc/gitweb/gitweb.cgi?p=systembsd.git >>>>> * >>>>> http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/debian-systemd-packaging-hoo-hah.html >>>>> * https://news.ycombinator.com/item?id=10176275 >>>>> >>>>> The reason is that finishing systemdbsd will make happy all of the people >>>>> who want the desktop environments whose design is driven largely by Linux to >>>>> work on FreeBSD/PC-BSD. The desktop environments that they'd like to use >>>>> have been or are being modified to work with these daemons, over this D-Bus >>>>> protocol. >>>>> >>>> I strongly disagree with your recommendation to adopt DBus and systemd >>>> as core components of FreeBSD. >>>> >>>> From a practical perspective, the proposal has a low probability of >>>> success. Systemd is written for Linux and is largely driven by a >>>> commercial Linux vendor. It is a rapidly moving target, with no sense >>>> of scope or boundaries. It eagerly consumes the latest and greatest >>>> innovations in the Linux kernel, with open disdain for portability. >>>> >>>> From a philosophical perspective, I don't agree with the direction >>>> that systemd is taking Linux. It's one of the reasons I switched to >>>> BSD after many years in the Linux camp. To quote Spock, "Logic clearly >>>> dictates that the needs of the many outweigh the needs of the few". In >>>> case of FreeBSD, this means that the needs of the desktop users should >>>> not outweigh the needs of the server/jail/embedded/appliance users. My >>>> concern with systemd and DBus is that these tools are highly >>>> desktop-centric, and introduce a large degree of unwanted change, >>>> complexity, and risk to everyone else. >>> >>> I totally agree. >>> >>> systemd is an ugly beast, solving simple problem in complex way. >>> >>> After using FreeBSD's rc system for years, I think that switching to something systemd-related would be huge mistake. >>> No reason to clone everything that happens in Linux world. >> Utterly and strongly agreed. >> >> --------------------------------------------------------------------------- >> Peter Beckman Internet Guy >> beckman@angryox.com http://www.angryox.com/ >> --------------------------------------------------------------------------- >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org " > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Fri Jan 15 11:51:12 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB7CAA8323D for ; Fri, 15 Jan 2016 11:51:12 +0000 (UTC) (envelope-from garga.bsd@gmail.com) Received: from mail-qg0-x234.google.com (mail-qg0-x234.google.com [IPv6:2607:f8b0:400d:c04::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 946931A7D for ; Fri, 15 Jan 2016 11:51:12 +0000 (UTC) (envelope-from garga.bsd@gmail.com) Received: by mail-qg0-x234.google.com with SMTP id b35so375272324qge.0 for ; Fri, 15 Jan 2016 03:51:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=AgRomUw9Z+BEe5/zmMPubdbtLxlaLay8eonaX4VY7yE=; b=WK9qgb0sfIDsftBeL5TTSsEm4+wKVCr/7h/B99/1sWZXcHw7jMs4pzE2pXPi1izDRF sRRLWBXv/7dnfU/u0V4691YRFSZcANZRpXLz2rzLeZYNXewmkH+D22kIkhDRFjt11o8w zwwOgoT1fgK6JzFPdKwSOL5cb+Qs9UPjARJeKqHUAVCsfqiZKneOh1xGAsU1K5pK+jg7 BXdkkk5Xat4U0TObcjAsmPHG10zq8NWKFe9rcw4/liKQPYMNbPtM5OKxuUI/9F4BRInM 3/qX4cBuKCJuZOUb5SG6clvMaBnzNk4VuWKC3O8e7DEsV5OMAXXXISsEUJsPt716qQ4z 5MLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=AgRomUw9Z+BEe5/zmMPubdbtLxlaLay8eonaX4VY7yE=; b=J6ZH6gXRIZBqHK602qC/uYd/pyft+xYjoSYrMlUp0tzQWqZkMKukYc/iEJrfklY86F ZBGc73FH/FlEjdigGrQqW79Htr8q/yRzTkVdnAqbHeaFsj/VSIOhhBWb/sBhJET3eTuY qp8hmF71MBTWz9sYw94qGGd/3N0I4/wqTDy0UI0xB8S96cnAo39XjKzIK6noa6wy+wZh yO32Z8/7FWbi34QUFBejoSioAt4aEU53SzsWX/Vn4jU1+KtZGA3pqkBya11OS/aaQc6i pomkLmQvFDwwZpOakIn25Uo/HESxLRMZA++vWycWZDHPVjxRPz/pUWW7H9pyoLggP9Tn nHRg== X-Gm-Message-State: AG10YOQ02YNIfrLQGJS4AWvPgkHRnZWUDQPm/8+J4kVAwYe56rA2sK7MxZNZe5RVcYMfPQ== X-Received: by 10.140.248.213 with SMTP id t204mr724101qhc.97.1452858671819; Fri, 15 Jan 2016 03:51:11 -0800 (PST) Received: from mbp.home ([186.249.147.62]) by smtp.gmail.com with ESMTPSA id u65sm4337913qhc.9.2016.01.15.03.51.09 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 15 Jan 2016 03:51:11 -0800 (PST) Sender: Renato Botelho Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs From: Renato Botelho In-Reply-To: <56985C6A.6040209@multiplay.co.uk> Date: Fri, 15 Jan 2016 09:51:08 -0200 Cc: Eric McCorkle , Gabor Radnai , freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 11:51:12 -0000 > On Jan 15, 2016, at 00:41, Steven Hartland = wrote: >=20 > Just wanted to let everyone know that I just finished committing these = changes to the tree. >=20 > Huge thanks to Eric's for his work on this, as well as everyone else = who contributed. >=20 > I've set the target for MFC of 2 weeks, so I hope to be able to get = this into stable/10 before the 10.3 slush, so if you're interested in = this change please test a head build > r294068 ASAP. Great work, thanks! Is there a way to move a installed ZFS system to EFI? -- Renato Botelho From owner-freebsd-hackers@freebsd.org Fri Jan 15 12:22:12 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0AD80A847C6 for ; Fri, 15 Jan 2016 12:22:12 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6941D1F99 for ; Fri, 15 Jan 2016 12:22:11 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22e.google.com with SMTP id u188so20963356wmu.1 for ; Fri, 15 Jan 2016 04:22:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=s37uv10JoSMCRk9/Zb0fAqjm5lVx7L44sziIxx5HtnQ=; b=ZQlZ779ejkZImcUha9ljkvki2fG2xHFwi/TnFdar5ORWS1TA6EqH+J1qHvwFFKIOx1 wsu7GFsoHZvBx0mLsI1GNzfPb24wGx6CcGHUYqs1larXgN3MJbpxJdhYgB4OQ67v0MlT sjt6DXiTnKQoiYkd7CQvu6FfDhKNngAER52Cu5jnUvu7lmwa4MV1Az3i1t7LY5mrhSsX 22wOkY7iULCwIdukvl9bhllIzr9yk+ZmNw7yZjCsgZAuEmqhxgXkfI6SnHnZ6M9JSjaa 4ZKbwmJMPRNpmm7SKYmRzvX2/V6pZejzleVJq5LuGbepC0Sj9v7ik1eClooZ3uH39tMi rNdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=s37uv10JoSMCRk9/Zb0fAqjm5lVx7L44sziIxx5HtnQ=; b=S83PFbCu0OinSBXmtKX8dTJqDMwOWl/O4Q9/BhxhVnoxQFzQ8OsmnAQxawLwt1J7vc eXcDTpNdKX3aIxX8LjS3nFuMOFRn79qQri4iO03MXTxObQuV9rX7/vI1uemcf0ZICIs9 2Yz3QCq0Pj2253x0mxRpYUlREwvtGaXN8gZdQ1sQLMbqnSDgM8HMLuQ8AYkpS0q0j8X7 zLSU44qSkNVypXsVBii3nysreQTjmYLCAdzsnH/7YPJSgHuQ0byoILKP29EyoWr3TdDK kipMAFdwhRhjYq7xLSwx5NSyfMHpK0kbDs5EfguNwi/7V0v3lj6r+K3drBMGA3BC1YKO qBgg== X-Gm-Message-State: ALoCoQkFArllAxQuSc/k3QIjayFeu7JA98fm8aVZlBZttUDUBws8N1dWgvUqCJoahxQZtPZGO82UqSnWqU0bQfrZVNm9JVTowQ== X-Received: by 10.194.238.162 with SMTP id vl2mr9269840wjc.91.1452860529156; Fri, 15 Jan 2016 04:22:09 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id q75sm2324714wmd.6.2016.01.15.04.22.07 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 15 Jan 2016 04:22:07 -0800 (PST) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: Renato Botelho References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> Cc: Eric McCorkle , Gabor Radnai , freebsd-hackers@freebsd.org From: Steven Hartland Message-ID: <5698E483.4000808@multiplay.co.uk> Date: Fri, 15 Jan 2016 12:22:27 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 12:22:12 -0000 On 15/01/2016 11:51, Renato Botelho wrote: >> On Jan 15, 2016, at 00:41, Steven Hartland wrote: >> >> Just wanted to let everyone know that I just finished committing these changes to the tree. >> >> Huge thanks to Eric's for his work on this, as well as everyone else who contributed. >> >> I've set the target for MFC of 2 weeks, so I hope to be able to get this into stable/10 before the 10.3 slush, so if you're interested in this change please test a head build > r294068 ASAP. > Great work, thanks! > > Is there a way to move a installed ZFS system to EFI? All EFI needs is an valid EFI partition on a GPT disk and the updated loader.efi so if your devices is GPT based and you have space for an extra 800k partition then yes. The following should be the basic steps you need (untested so backup your data) 1. gpart add -t efi -s 800k 2. gpart bootcode -p /boot/boot1.efifat -i 3. Ensure your root filesystem has EFI ZFS boot compatible world e.g. cd /usr/src && make buildworld -jXX && make installworld If you have an active MBR then you will likely need to disable this as EFI seems to ignore disks with active MBR even if they have a valid EFI partition. If you don't have space then you'll need to migrate to a difference device to change the disk partition layout. Regards Steve From owner-freebsd-hackers@freebsd.org Fri Jan 15 12:43:17 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3DFF6A84D62 for ; Fri, 15 Jan 2016 12:43:17 +0000 (UTC) (envelope-from garga.bsd@gmail.com) Received: from mail-qg0-x235.google.com (mail-qg0-x235.google.com [IPv6:2607:f8b0:400d:c04::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA0F019DC for ; Fri, 15 Jan 2016 12:43:16 +0000 (UTC) (envelope-from garga.bsd@gmail.com) Received: by mail-qg0-x235.google.com with SMTP id o11so506191190qge.2 for ; Fri, 15 Jan 2016 04:43:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=p+nBDp0uXOqL59wMMPgrApFUjX28wpWibODgYmuvRLo=; b=muy2h1htCTInstV7CoYXMJsN5AEjJW18KFn3tay60T5kXHURph/Uu7ISY/Vv/242Pe qg+1QEpTuWGk8xIFEqIqoyaDiaR3y2zwBexk/OR5kr2A80kJFzlc2SFxVKW1Yw5gBaMU edfIrP2aLpDAhLz5odA/TWWrlGIovooSst9Ixo93iD3qZhpOzOuxL+FQVCjaPheFjKZa FkVyxnIk5hflPCN/KB5FMbBJv1GRzAqfLZD4Kt42Umf/1J4Ie2KTgqbN8wA+loDU3yPc m5SsyTHevvppfdmtAJFAiBO87k7il2AZdjeEGLViOxesdHMVvMFgE/0+l1knn2HX8k3/ qFlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=p+nBDp0uXOqL59wMMPgrApFUjX28wpWibODgYmuvRLo=; b=muVz/qgx5H4P8CggFzz1LnhjHS2BHoMr1c0szgB9KdiPNRzt1pO6br5155BXob3bDf 87LoaRVQJUy8OQpCqSAWBfIi6CDDRs6pxbGzyYLbeYnAU1h6cOZj6qkeEX1i7qiNPtR3 z21lqyCHvO86kh67Irq8rJuwNpiIKVTK7aPKg5xX+ATEuWsjmSSMmqfQNr3wcKQwH9PT xseGESoKA9k/6TyqwDRny1mrCkqi3evavRHYSjxt1pz8b+IBuhVzUV/p/GYeJiu3xC5f BncbCrXETL7xlu1McwcGiIQc0rHRgT1K6t16QdG+dr1x9ept6k5WK25RuKTGjPpVUWik QI4g== X-Gm-Message-State: ALoCoQkIuSCddOluA1d6dwtxr4S5vJNhbKecMMHQWUYQFDJem1gvRq8qV6cnDgF4j5yRpBKJ+RNW3zYF2idIaBRY6Dc4BqRuxg== X-Received: by 10.140.28.66 with SMTP id 60mr12673135qgy.74.1452861796106; Fri, 15 Jan 2016 04:43:16 -0800 (PST) Received: from mbp.home ([186.249.147.62]) by smtp.gmail.com with ESMTPSA id e187sm4463547qka.1.2016.01.15.04.43.14 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 15 Jan 2016 04:43:15 -0800 (PST) Sender: Renato Botelho Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs From: Renato Botelho In-Reply-To: <5698E483.4000808@multiplay.co.uk> Date: Fri, 15 Jan 2016 10:41:47 -0200 Cc: Eric McCorkle , Gabor Radnai , freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <181DDF14-32C0-47D3-BE50-C9FBF4CA39CB@FreeBSD.org> References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> <5698E483.4000808@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 12:43:17 -0000 > On Jan 15, 2016, at 10:22, Steven Hartland = wrote: >=20 > On 15/01/2016 11:51, Renato Botelho wrote: >>> On Jan 15, 2016, at 00:41, Steven Hartland = wrote: >>>=20 >>> Just wanted to let everyone know that I just finished committing = these changes to the tree. >>>=20 >>> Huge thanks to Eric's for his work on this, as well as everyone else = who contributed. >>>=20 >>> I've set the target for MFC of 2 weeks, so I hope to be able to get = this into stable/10 before the 10.3 slush, so if you're interested in = this change please test a head build > r294068 ASAP. >> Great work, thanks! >>=20 >> Is there a way to move a installed ZFS system to EFI? >=20 > All EFI needs is an valid EFI partition on a GPT disk and the updated = loader.efi so if your devices is GPT based and you have space for an = extra 800k partition then yes. >=20 > The following should be the basic steps you need (untested so backup = your data) > 1. gpart add -t efi -s 800k > 2. gpart bootcode -p /boot/boot1.efifat -i > 3. Ensure your root filesystem has EFI ZFS boot compatible world e.g. > cd /usr/src && make buildworld -jXX && make installworld >=20 > If you have an active MBR then you will likely need to disable this as = EFI seems to ignore disks with active MBR even if they have a valid EFI = partition. >=20 > If you don't have space then you'll need to migrate to a difference = device to change the disk partition layout. Thanks. Worked fine! -- Renato Botelho From owner-freebsd-hackers@freebsd.org Fri Jan 15 12:47:32 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 91E78A84EB1 for ; Fri, 15 Jan 2016 12:47:32 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FC0A1C95; Fri, 15 Jan 2016 12:47:32 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aK3mo-000GhZ-VQ; Fri, 15 Jan 2016 15:47:26 +0300 Date: Fri, 15 Jan 2016 15:47:26 +0300 From: Slawa Olhovchenkov To: Steven Hartland Cc: Renato Botelho , Eric McCorkle , freebsd-hackers@freebsd.org, Gabor Radnai Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs Message-ID: <20160115124726.GR4535@zxy.spb.ru> References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> <5698E483.4000808@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5698E483.4000808@multiplay.co.uk> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 12:47:32 -0000 On Fri, Jan 15, 2016 at 12:22:27PM +0000, Steven Hartland wrote: > On 15/01/2016 11:51, Renato Botelho wrote: > >> On Jan 15, 2016, at 00:41, Steven Hartland wrote: > >> > >> Just wanted to let everyone know that I just finished committing these changes to the tree. > >> > >> Huge thanks to Eric's for his work on this, as well as everyone else who contributed. > >> > >> I've set the target for MFC of 2 weeks, so I hope to be able to get this into stable/10 before the 10.3 slush, so if you're interested in this change please test a head build > r294068 ASAP. > > Great work, thanks! > > > > Is there a way to move a installed ZFS system to EFI? > > All EFI needs is an valid EFI partition on a GPT disk and the updated > loader.efi so if your devices is GPT based and you have space for an > extra 800k partition then yes. > > The following should be the basic steps you need (untested so backup > your data) > 1. gpart add -t efi -s 800k > 2. gpart bootcode -p /boot/boot1.efifat -i > 3. Ensure your root filesystem has EFI ZFS boot compatible world e.g. > cd /usr/src && make buildworld -jXX && make installworld > > If you have an active MBR then you will likely need to disable this as > EFI seems to ignore disks with active MBR even if they have a valid EFI > partition. What you mean "active MBR" and "ignore disks with active MBR"? > If you don't have space then you'll need to migrate to a difference > device to change the disk partition layout. > > Regards > Steve > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Fri Jan 15 13:44:00 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 549C0A832EE for ; Fri, 15 Jan 2016 13:44:00 +0000 (UTC) (envelope-from oliver.pinter@hardenedbsd.org) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0469B1513 for ; Fri, 15 Jan 2016 13:43:59 +0000 (UTC) (envelope-from oliver.pinter@hardenedbsd.org) Received: by mail-wm0-x22f.google.com with SMTP id f206so25283821wmf.0 for ; Fri, 15 Jan 2016 05:43:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=4Pl5xxHErclvFiZvKvDElz+nEodtyl2hdhf9LoHWjnY=; b=AI29FF7Ci4bDZiY+9CGCZSUHFIEM2AttlCjGLwuD42yaNnSUS2NCAZUWuMWh/i6OM/ YSxT7AEQf5Y0luO3lA11TzO3iHfNffNCTV8mJ0Zh+xH3lVEAzJsniWt2CYF3JggSY9Pw F0UIPvMv8xN4It2N+d7rlt6epD0eS2+TN7seBKMzaYb7CzWlLcbrbyuX3mUp5aMfBMiH UP9PciZfxMz0Uwft3kGkRv0cuEebXmpzYrOHYkDHVpExmNZzdrwhDhxJ0SQPsOrqesyV HDPF/L2waTLDrklaDSS9RC7ZeR9qBxF6egWHjVnn1hPD6Qx5CJ7RZPiHykOAsPzLt/SL PbRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=4Pl5xxHErclvFiZvKvDElz+nEodtyl2hdhf9LoHWjnY=; b=eTZFqFDRZOSr5eHoUw/lNIUhWc/Yc9uYkc3+hiuSJx8s+xrlhKSfOlrYPlTMus+xFN SdO131XsVpgseyZpgXYUUTgJtJ0k8L8sgiyOqIZqrxNDhjvKZjlyEg1PNKsL6dTDBS4U ActMRPh4s4yo+J152I9f1ZWA9DhjEmZOhzSJXsiTsLdYjtgVW44t8tlKwMIDFkdxk6JE XfK85ycmmStX+sbY6TZZ+6Fl83duz0oIbnyN/pHjhtZCyHI277ZP0pHvqpiiamfHw1Dc R1OB1/36iF0rl1oBGLIZ8Q6Bp0a0UicZ36A7kaBj1ymq+1PVEu3d9qQJ6G5ZhhKaorVc m7Iw== X-Gm-Message-State: ALoCoQk+BbjNLgQItBhpLalcytnK8Oww4BSQIfW2GYoLC5b7fwqmh4973zybe3wnd+9WKDfP5yG2YQ+UWeUR8LaxG9zzs5c1AB5baSkDdDoxE4DimRC1etQ= MIME-Version: 1.0 X-Received: by 10.194.171.66 with SMTP id as2mr10026858wjc.73.1452865438461; Fri, 15 Jan 2016 05:43:58 -0800 (PST) Received: by 10.194.85.167 with HTTP; Fri, 15 Jan 2016 05:43:58 -0800 (PST) In-Reply-To: <56985C6A.6040209@multiplay.co.uk> References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> Date: Fri, 15 Jan 2016 14:43:58 +0100 Message-ID: Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs From: Oliver Pinter To: Steven Hartland Cc: Eric McCorkle , Gabor Radnai , freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 13:44:00 -0000 On 1/15/16, Steven Hartland wrote: > Just wanted to let everyone know that I just finished committing these > changes to the tree. > > Huge thanks to Eric's for his work on this, as well as everyone else who > contributed. > > I've set the target for MFC of 2 weeks, so I hope to be able to get this > into stable/10 before the 10.3 slush, so if you're interested in this > change please test a head build > r294068 ASAP. If someone want to play with them, then here could access an installer: http://jenkins.hardenedbsd.org/builds/HardenedBSD-CURRENT-i915kms-amd64-LATEST/ISO-IMAGES/ This contains out HardenedBSD patches, dumbbell's i915kms update with Haswell support, and the latest EFI + ZFS updates. And you could follow the future build here: http://jenkins.hardenedbsd.org:8180/jenkins/job/HardenedBSD-CURRENT-i915kms-amd64/ > > Regards > Steve > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Fri Jan 15 13:59:14 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C323A838CC for ; Fri, 15 Jan 2016 13:59:14 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (mail.metricspace.net [IPv6:2001:470:1f11:617::107]) by mx1.freebsd.org (Postfix) with ESMTP id 114431E86 for ; Fri, 15 Jan 2016 13:59:13 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [IPv6:2001:470:1f11:617:ea2a:eaff:fe21:e067] (unknown [IPv6:2001:470:1f11:617:ea2a:eaff:fe21:e067]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id 37FA51D96; Fri, 15 Jan 2016 13:59:07 +0000 (UTC) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: Steven Hartland , Gabor Radnai , freebsd-hackers@freebsd.org References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> From: Eric McCorkle Message-ID: <5698FB2A.8030503@metricspace.net> Date: Fri, 15 Jan 2016 08:59:06 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <56985C6A.6040209@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 13:59:14 -0000 Thanks to Steven for picking this up, and thanks to everyone who helped with the integration effort (particularly for working out how to integrate libstand directly). I plan to look into booting from a GELI-encrypted partition next, and in the longer-term, I'm interested in some kind of cryptographic signature checking ability. On 01/14/16 21:41, Steven Hartland wrote: > Just wanted to let everyone know that I just finished committing these > changes to the tree. > > Huge thanks to Eric's for his work on this, as well as everyone else who > contributed. > > I've set the target for MFC of 2 weeks, so I hope to be able to get this > into stable/10 before the 10.3 slush, so if you're interested in this > change please test a head build > r294068 ASAP. > > Regards > Steve From owner-freebsd-hackers@freebsd.org Fri Jan 15 14:02:30 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CB72A83BC2 for ; Fri, 15 Jan 2016 14:02:30 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com [IPv6:2a00:1450:400c:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D968913C1 for ; Fri, 15 Jan 2016 14:02:29 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x233.google.com with SMTP id f206so26087285wmf.0 for ; Fri, 15 Jan 2016 06:02:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=ggMy8s3+J8+4uO2inqaE3bCBN0uSAn6A9Db1i/hV9ZQ=; b=a2ATlppe39t8PUcabeV84xk68n8RYVEMliQ1fg1its1Sd87DybZR+Xb6QnUBgINxTS /SSWFV8HjKWVbDb+LgmaxQSz92vDJogxvG+KaAscCqjysguhHqEEQvfc1JTsVrpby2Ev vH9mhTltraSTkM9a8Vu3dPM5WcHfcGYemFRAKSaA78RuqPnQpnV94dWFuaciLfZsjuPq /isYsui9u20OkI5qVLKEmUu49ORl42J1+nHgrIt5v73ckJS0llGYaZQy4X67eZNizcwI yBxM1HThYEI9VLgu/OuGs/6oinKkbv57ppP44akZU7OuOOG4VHceGz1EhTDb/wl2EsbA UcCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=ggMy8s3+J8+4uO2inqaE3bCBN0uSAn6A9Db1i/hV9ZQ=; b=JDBUqVUQ1TOywJ/98ULC+EPe9S2QR6Ws7xmV/qBYHoa5y9kK2OZ1ucOhYRTbqP21WM dEMpDMKJQVlgNjeZg4gzkgpEkZdmEkubot7MNbu67p/k5MulWyDXi01OU1HrWPGFGqud ac6NrEW/7WNS8l17H/IXkYj0UhZT/kjgEcXsF9sfU2udwnf0KU9Zh80IKOcJu3p9vQCa +HWBrTzww3HxYIpAlosAPxrx1sa6qOI+r6PF/ZFCrVP0GTROfcnE48rdod5by/5TNzua s1P+VXT0oxnjJzNUshYtQDO6yqh/cwinKmNIlhDV4CU6jSdoPW/gG+UnowaNTPnP2/bG /AeA== X-Gm-Message-State: ALoCoQlz5hKbXzup/Rj1HmIBUyzpVar0s7FfQ8xJX3rJltUXxdoPqUxTZTmXh/GEO1UUDHJFCgrgp0BeC9baSbIGUXkBOSTObg== X-Received: by 10.194.203.137 with SMTP id kq9mr11594452wjc.129.1452866548011; Fri, 15 Jan 2016 06:02:28 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id t195sm2683280wme.13.2016.01.15.06.02.26 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 15 Jan 2016 06:02:27 -0800 (PST) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: Eric McCorkle , Gabor Radnai , freebsd-hackers@freebsd.org References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> <5698FB2A.8030503@metricspace.net> From: Steven Hartland Message-ID: <5698FC07.50901@multiplay.co.uk> Date: Fri, 15 Jan 2016 14:02:47 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <5698FB2A.8030503@metricspace.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 14:02:30 -0000 On 15/01/2016 13:59, Eric McCorkle wrote: > Thanks to Steven for picking this up, and thanks to everyone who > helped with the integration effort (particularly for working out how > to integrate libstand directly). > > > I plan to look into booting from a GELI-encrypted partition next, and > in the longer-term, I'm interested in some kind of cryptographic > signature checking ability. https://reviews.freebsd.org/D4593 might be of interest to you then. Regards Steve From owner-freebsd-hackers@freebsd.org Fri Jan 15 14:23:11 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC455A84199 for ; Fri, 15 Jan 2016 14:23:10 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from mail.metricspace.net (mail.metricspace.net [IPv6:2001:470:1f11:617::107]) by mx1.freebsd.org (Postfix) with ESMTP id A5BBB1F5D; Fri, 15 Jan 2016 14:23:10 +0000 (UTC) (envelope-from eric@metricspace.net) Received: from [IPv6:2001:470:1f11:617:ea2a:eaff:fe21:e067] (unknown [IPv6:2001:470:1f11:617:ea2a:eaff:fe21:e067]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id 2004E1DB9; Fri, 15 Jan 2016 14:23:10 +0000 (UTC) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: Renato Botelho , Steven Hartland References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> Cc: Gabor Radnai , freebsd-hackers@freebsd.org From: Eric McCorkle Message-ID: <569900CD.2040003@metricspace.net> Date: Fri, 15 Jan 2016 09:23:09 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 14:23:11 -0000 On 01/15/16 06:51, Renato Botelho wrote: >> On Jan 15, 2016, at 00:41, Steven Hartland wrote: >> >> Just wanted to let everyone know that I just finished committing these changes to the tree. >> >> Huge thanks to Eric's for his work on this, as well as everyone else who contributed. >> >> I've set the target for MFC of 2 weeks, so I hope to be able to get this into stable/10 before the 10.3 slush, so if you're interested in this change please test a head build > r294068 ASAP. > > Great work, thanks! > > Is there a way to move a installed ZFS system to EFI? > (Refer to Steven's guide for the simple case where you can create an EFI partition) == Using GRUB == GRUB can be used with loader.efi on a ZFS system (I use this myself, as I have a Gentoo install in the same ZFS volume) Make sure you install GRUB with EFI support (the ports collection will handle this). The grub port comes with a script that auto-detects filesystems and sets up a grub.cfg in /boot/grub/. However, that script won't properly detect ZFS partitions, so you'll need to add it manually. The entry is simple. I have a zfs volume called "data", which has the freebsd system root on the filesystem data/freebsd. The GRUB entry then is: menuentry "FreeBSD" { search.fs_label data ZFS_PART chainloader ($ZFS_PART)/freebsd@/boot/loader.efi } The first line searches for the volume "data" and binds its device to the variable ZFS_PART. The second specifies that the chained bootloader is under the filesystem "freebsd" (the @ at the end of the name denotes a filesystem, not a path), with the path /boot/loader.efi == Disks without enough space to make the GPT or EFI partition == If you have a ZFS filesystem on an MBR disk, or on a GPT disk without enough space to create a workable EFI partition, you can use one of ZFS's lesser-known features: zfs send/recv. ZFS send and recv allow an entire filesystem to be serialized out to a stream, and then read back in. You can use this to dump the entire filesystem out to a removable storage or an NFS mount. Then, use an install disk or memstick and repartition your drive, using zfs recv to recreate the filesystem. == On a Mac == (Note: this is based on research that is several years old at this point. Also, I never actually field tested this myself.) Macs are wierd, due to their non-standard EFI. The Mac EFI implementation looks for an HFS+ partition, and loads the "blessed" file as the EFI bootloader (this is a special filesystem-level metadata unique to HFS+ filesystems). In order to do an EFI boot on a mac, you'd need some way of manufacturing an HFS+ partition containing only your bootloader, with that file being "blessed". The easiest way to do this would be to use a Mac OS install to create an empty HFS+ filesystem, add your boot loader, then use a shell command to "bless" it (this command exists, but I don't remember what it is offhand). It also wouldn't be too hard to write a tool to create an HFS+ image from a file (I have a half-written implementation of that lying around somewhere). Also note that Macs expect a 200MB EFI partition (which isn't actually used for anything), and I've heard reports of the firmware flaking out if it's not there. I believe there are also GRUB and rEFIt options for Mac, if you don't want to go to these lengths. From owner-freebsd-hackers@freebsd.org Fri Jan 15 14:47:45 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FB60A84965 for ; Fri, 15 Jan 2016 14:47:45 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 911491EAF for ; Fri, 15 Jan 2016 14:47:44 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x230.google.com with SMTP id f206so23218510wmf.0 for ; Fri, 15 Jan 2016 06:47:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type; bh=6lV+cFCGpwwt2Ty36YgzpJTIsJPCnHuFRvMPneW+vQQ=; b=KSjaERUZNU0iRUgkdP6QDJFHwRgEGBS2bMxXSN+5S/rV2yG08wEQ64z5d+cooxmqPz /bCULkMpWbwE96ghjaxNX9OLlBv0fLdu9DWmkrXZ8ov8muTlgzBiGS5cGxR10KaI6A5P FDvPDy7hCaaVjelRUaHDMZRWdzRsPlB1x+L2faVlmSwvxUGevUG8bKg0egnesKN5aBkR jjcnEpAjGJQEPQC9Qg2f/1XNCulVWN15+pHY4f5RTq+b+ofVz74YJSzf/QhljFjeIppp g/mHCvPtKgtpXshEbomprYG+oE88R8VJcZENz0fcyOt0vXRZRYwCUfK6w/CnUQgVYniI DXCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type; bh=6lV+cFCGpwwt2Ty36YgzpJTIsJPCnHuFRvMPneW+vQQ=; b=cQY+ZjlwMQEH8BpkWGDmnbWpxU1NZP09NzkaJgFcOYGWI5A3lmaNDzfsu4ktomoSwF nVQfjBaKFO/oTQIoO6cBjud0Mri/+f4JdMcCU3w3vDba6T/vpFVSJ5px/XiR1cnwKl/U RM6g4LR5EWtty0p+JRTuKm1A8XvYFDdFr922AmTP3YWYezYmePHJrsI3mTNw3VCyf14g dXosnX2DSuZrSXLwkfhm9LZ8mDs0csSJm7HggsW/FA8KPFVxZ+vdQn/uy9T6maRSNxuv daga9gIzx3NPOGNl36n91Lzi9+9TpEEd1aOjv+qOYMeNWMhpSwXPW1KPvYfT3Dk8H2rm z2ww== X-Gm-Message-State: AG10YOQYMz7/RYLWySkwaFXN7RCZ+xIHeU0FJyU4wIk6gmMW9t9Rarmnm7dLHkEJXEUNzhSs X-Received: by 10.28.51.17 with SMTP id z17mr4067254wmz.26.1452869262871; Fri, 15 Jan 2016 06:47:42 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id 17sm2861459wmy.15.2016.01.15.06.47.41 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 15 Jan 2016 06:47:41 -0800 (PST) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: krad , Eric McCorkle References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> <569900CD.2040003@metricspace.net> Cc: Renato Botelho , freebsd-hackers@freebsd.org, Gabor Radnai From: Steven Hartland Message-ID: <569906A1.8040700@multiplay.co.uk> Date: Fri, 15 Jan 2016 14:48:01 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 14:47:45 -0000 ZFS Boot Environments (BE) support was also wired up to Beastie last night by Allan for those interested in that :) On 15/01/2016 14:42, krad wrote: > Thanks this is useful information. I did have a look at the way pc bsd > was using grub to boot rootonzfs systems, and they used the Kfreebsd > options. This looked kind of handy as you might have been able to > specify the zfs file system to boot off. This would be a real boost > the boot environments as you could easily choose which one to boot > into, making upgrade recovery far easier. > > Presumably the freebsd@ part in your setup refers to the zfs file > system? In which case you could have multiple menu entries with > different file systems specified, this is assuming the grub config is > on the efi disk though? > > I'm also curious how this would his work when the are multiple pools > on the same disk for some reason. > > > > On 15 January 2016 at 14:23, Eric McCorkle > wrote: > > On 01/15/16 06:51, Renato Botelho wrote: > > On Jan 15, 2016, at 00:41, Steven Hartland > > > wrote: > > Just wanted to let everyone know that I just finished > committing these changes to the tree. > > Huge thanks to Eric's for his work on this, as well as > everyone else who contributed. > > I've set the target for MFC of 2 weeks, so I hope to be > able to get this into stable/10 before the 10.3 slush, so > if you're interested in this change please test a head > build > r294068 ASAP. > > > Great work, thanks! > > Is there a way to move a installed ZFS system to EFI? > > > (Refer to Steven's guide for the simple case where you can create > an EFI partition) > > > == Using GRUB == > > GRUB can be used with loader.efi on a ZFS system (I use this > myself, as I have a Gentoo install in the same ZFS volume) > > Make sure you install GRUB with EFI support (the ports collection > will handle this). The grub port comes with a script that > auto-detects filesystems and sets up a grub.cfg in /boot/grub/. > However, that script won't properly detect ZFS partitions, so > you'll need to add it manually. The entry is simple. > > I have a zfs volume called "data", which has the freebsd system > root on the filesystem data/freebsd. The GRUB entry then is: > > menuentry "FreeBSD" { > search.fs_label data ZFS_PART > chainloader ($ZFS_PART)/freebsd@/boot/loader.efi > } > > The first line searches for the volume "data" and binds its device > to the variable ZFS_PART. The second specifies that the chained > bootloader is under the filesystem "freebsd" (the @ at the end of > the name denotes a filesystem, not a path), with the path > /boot/loader.efi > > > == Disks without enough space to make the GPT or EFI partition == > > If you have a ZFS filesystem on an MBR disk, or on a GPT disk > without enough space to create a workable EFI partition, you can > use one of ZFS's lesser-known features: zfs send/recv. > > ZFS send and recv allow an entire filesystem to be serialized out > to a stream, and then read back in. You can use this to dump the > entire filesystem out to a removable storage or an NFS mount. > Then, use an install disk or memstick and repartition your drive, > using zfs recv to recreate the filesystem. > > > == On a Mac == > > (Note: this is based on research that is several years old at this > point. Also, I never actually field tested this myself.) > > Macs are wierd, due to their non-standard EFI. The Mac EFI > implementation looks for an HFS+ partition, and loads the > "blessed" file as the EFI bootloader (this is a special > filesystem-level metadata unique to HFS+ filesystems). > > In order to do an EFI boot on a mac, you'd need some way of > manufacturing an HFS+ partition containing only your bootloader, > with that file being "blessed". The easiest way to do this would > be to use a Mac OS install to create an empty HFS+ filesystem, add > your boot loader, then use a shell command to "bless" it (this > command exists, but I don't remember what it is offhand). It also > wouldn't be too hard to write a tool to create an HFS+ image from > a file (I have a half-written implementation of that lying around > somewhere). > > Also note that Macs expect a 200MB EFI partition (which isn't > actually used for anything), and I've heard reports of the > firmware flaking out if it's not there. > > I believe there are also GRUB and rEFIt options for Mac, if you > don't want to go to these lengths. > > _______________________________________________ > freebsd-hackers@freebsd.org > mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org > " > > From owner-freebsd-hackers@freebsd.org Fri Jan 15 15:02:27 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEC04A84F3E for ; Fri, 15 Jan 2016 15:02:27 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B46F1B0B for ; Fri, 15 Jan 2016 15:02:27 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id AEDBD22BB99 for ; Fri, 15 Jan 2016 08:52:56 -0600 (CST) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: freebsd-hackers@freebsd.org References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> <569900CD.2040003@metricspace.net> <569906A1.8040700@multiplay.co.uk> From: Karl Denninger Message-ID: <569907B2.4070700@denninger.net> Date: Fri, 15 Jan 2016 08:52:34 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <569906A1.8040700@multiplay.co.uk> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms040907010807050003010503" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 15:02:27 -0000 This is a cryptographically signed message in MIME format. --------------ms040907010807050003010503 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable THAT's useful around here :-) On 1/15/2016 08:48, Steven Hartland wrote: > ZFS Boot Environments (BE) support was also wired up to Beastie last > night by Allan for those interested in that :) > > On 15/01/2016 14:42, krad wrote: >> Thanks this is useful information. I did have a look at the way pc >> bsd was using grub to boot rootonzfs systems, and they used the >> Kfreebsd options. This looked kind of handy as you might have been >> able to specify the zfs file system to boot off. This would be a real >> boost the boot environments as you could easily choose which one to >> boot into, making upgrade recovery far easier. >> >> Presumably the freebsd@ part in your setup refers to the zfs file >> system? In which case you could have multiple menu entries with >> different file systems specified, this is assuming the grub config is >> on the efi disk though? >> >> I'm also curious how this would his work when the are multiple pools >> on the same disk for some reason. >> >> >> >> On 15 January 2016 at 14:23, Eric McCorkle > > wrote: >> >> On 01/15/16 06:51, Renato Botelho wrote: >> >> On Jan 15, 2016, at 00:41, Steven Hartland >> >= >> wrote: >> >> Just wanted to let everyone know that I just finished >> committing these changes to the tree. >> >> Huge thanks to Eric's for his work on this, as well as >> everyone else who contributed. >> >> I've set the target for MFC of 2 weeks, so I hope to be >> able to get this into stable/10 before the 10.3 slush, so >> if you're interested in this change please test a head >> build > r294068 ASAP. >> >> >> Great work, thanks! >> >> Is there a way to move a installed ZFS system to EFI? >> >> >> (Refer to Steven's guide for the simple case where you can create >> an EFI partition) >> >> >> =3D=3D Using GRUB =3D=3D >> >> GRUB can be used with loader.efi on a ZFS system (I use this >> myself, as I have a Gentoo install in the same ZFS volume) >> >> Make sure you install GRUB with EFI support (the ports collection >> will handle this). The grub port comes with a script that >> auto-detects filesystems and sets up a grub.cfg in /boot/grub/. >> However, that script won't properly detect ZFS partitions, so >> you'll need to add it manually. The entry is simple. >> >> I have a zfs volume called "data", which has the freebsd system >> root on the filesystem data/freebsd. The GRUB entry then is: >> >> menuentry "FreeBSD" { >> search.fs_label data ZFS_PART >> chainloader ($ZFS_PART)/freebsd@/boot/loader.efi >> } >> >> The first line searches for the volume "data" and binds its device= >> to the variable ZFS_PART. The second specifies that the chained >> bootloader is under the filesystem "freebsd" (the @ at the end of >> the name denotes a filesystem, not a path), with the path >> /boot/loader.efi >> >> >> =3D=3D Disks without enough space to make the GPT or EFI partition= =3D=3D >> >> If you have a ZFS filesystem on an MBR disk, or on a GPT disk >> without enough space to create a workable EFI partition, you can >> use one of ZFS's lesser-known features: zfs send/recv. >> >> ZFS send and recv allow an entire filesystem to be serialized out >> to a stream, and then read back in. You can use this to dump the >> entire filesystem out to a removable storage or an NFS mount. =20 >> Then, use an install disk or memstick and repartition your drive, >> using zfs recv to recreate the filesystem. >> >> >> =3D=3D On a Mac =3D=3D >> >> (Note: this is based on research that is several years old at this= >> point. Also, I never actually field tested this myself.) >> >> Macs are wierd, due to their non-standard EFI. The Mac EFI >> implementation looks for an HFS+ partition, and loads the >> "blessed" file as the EFI bootloader (this is a special >> filesystem-level metadata unique to HFS+ filesystems). >> >> In order to do an EFI boot on a mac, you'd need some way of >> manufacturing an HFS+ partition containing only your bootloader, >> with that file being "blessed". The easiest way to do this would >> be to use a Mac OS install to create an empty HFS+ filesystem, add= >> your boot loader, then use a shell command to "bless" it (this >> command exists, but I don't remember what it is offhand). It also= >> wouldn't be too hard to write a tool to create an HFS+ image from >> a file (I have a half-written implementation of that lying around >> somewhere). >> >> Also note that Macs expect a 200MB EFI partition (which isn't >> actually used for anything), and I've heard reports of the >> firmware flaking out if it's not there. >> >> I believe there are also GRUB and rEFIt options for Mac, if you >> don't want to go to these lengths. >> >> _______________________________________________ >> freebsd-hackers@freebsd.org >> mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to >> "freebsd-hackers-unsubscribe@freebsd.org >> " >> >> > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms040907010807050003010503 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjAxMTUxNDUyMzRaME8GCSqGSIb3DQEJBDFCBEC7 gvrswqNT3HiTqL9EVcLHgUcUzC3D378dO39kPTYXxmfD72Mf6YCZFY7QC1GsuqXE8Nl/mnR7 QHkrVU1SJ58DMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIALli38eQ+ Bur+gj2CyE2AkkhnBraYhzrz7Vd8I3uvjfnFEC3Xd8YONCXMRpFoFib6YzI4iPLaI4AQ7T/W jz6X3vfFgSHlknnbPAMAPyc2zTYXK1zDyO/1reBc4j65SejwCOEwh7UPrC/uf59ErYpN5Tbm 33jKvsRl6sa8DmvdqHx0xymy0jida/CyLvli87Jp1e0R9+UDhlMbPq8uRP63Mup/e2leKNBX WWWiIdjkcXUhCzgw2AZnaoNFrQNn5ZE5QUh7ZJGl6b/IS0bDDSyrXDFvDaEXjn/7+p5zVJza JkO1zE1oiRLqhDt+KLkUogRciL2kkfRbcpEpSbHmIXqTLRolqc6mRTvqmV6LS0TTlyT2yNCK wt7ACEmQPqWlO/tULuN28M8vNqbcL6sHSRqfo3yxozfRx/yuQEVR1utNnhrYyNayd4q0SGDg fJolQd/E9lXrtYCK0KNVAZBvTpzCb7x4t9aHUtdfiNF9p3Aexa33KnrHVDoKu/LpYi2EiYfV GWn0N9PhEosDMxMKgmXSv7WMNo5DULLkY4TmEYYm1q+j5skInsFCQ4URNspx3svqv89MkCAb 0xKTvlk5UhSd3Q7qSVXtEee2Ymhir0WcIkgW41g5Aspef9ckUCOxYkyTnlqc6x87ZSdL+tT0 zcxrkQ1wLJrtpjoiGeG2cbYfgFIAAAAAAAA= --------------ms040907010807050003010503-- From owner-freebsd-hackers@freebsd.org Fri Jan 15 14:18:34 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AE73A83FF1 for ; Fri, 15 Jan 2016 14:18:34 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D7C6F1B3C for ; Fri, 15 Jan 2016 14:18:33 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id u188so24946166wmu.1 for ; Fri, 15 Jan 2016 06:18:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=E3W+8uQTL5jfqmzAobhSF6hALXYaqH9+JHKYZ7BHXiA=; b=LSBvhFKiBespMYi8dPnvgxNfuwCOcR6sNxMnDi3y6kPPnYTN5nfGAQ4fsgyiEiwqLw Qgv8/8XQZeKFW5Ml2IT/UT49PZJVaL2Jnr8q1wHq/djTs48vvhWNtQWhAhUvKqN1cF62 EhFznHBBJM0k7h6N5x+HI0KUnTVQSssTG68S5dri6PcYuJxbnqtA89dy4LJ0Pi2Nqwtd FDLUFfK1BLaIy/2v0bQLCh4axIC80luxW847wKRHcbiMPuiYPZb80+To6lMSBGHOd5JB vbRcoJG12xK2naaUUI0JRypJ5WVY3KGlNrCUpiNDzMe51fMjtSPJendB+p8XtR1Kt98L t4jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=E3W+8uQTL5jfqmzAobhSF6hALXYaqH9+JHKYZ7BHXiA=; b=TxuxPyIVZTVOlki4WbBy4XGsNsKZ2XmIPDvwafPCryFmpWsRlqCDpY9eTKcJtAlhm5 VL2wbco+dJhsABgEWbzH3l4WxpbJkrZTQ8a3sjFcVSFO+MP9kgYPx/7JfmiP5f6i97L8 WPn0tpLRkzO6urhbjKFLbbhZUBa9dbMcbcFhgGiMPZLEyHqgLlvSvcGm/zXSYtMRq4hJ ddfK4VTFgUEHjaJ+Jk9D3dzTk5YJErUrTl4/QEUsCqY8hJ2fmXKPvMj+Rjope2aRqp4R v2oQkHGJ0yE3q2WmmFTeyX9WduQFwrbjc9kMLJ3hEO2dl4OPSzwidsFWztZQr9OXDywx KNvw== X-Gm-Message-State: AG10YOQvfR6bgr8S++UId6cTrnyuLMXWz164MFtNG4pWLlZh/rdLOP4g7rD+Ah9ItkZ4N/YCRwzBV7QGj+Vg1A== MIME-Version: 1.0 X-Received: by 10.28.127.85 with SMTP id a82mr3916446wmd.48.1452867512215; Fri, 15 Jan 2016 06:18:32 -0800 (PST) Received: by 10.28.55.132 with HTTP; Fri, 15 Jan 2016 06:18:32 -0800 (PST) In-Reply-To: <56985C6A.6040209@multiplay.co.uk> References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> Date: Fri, 15 Jan 2016 14:18:32 +0000 Message-ID: Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs From: krad To: Steven Hartland Cc: Eric McCorkle , Gabor Radnai , freebsd-hackers@freebsd.org X-Mailman-Approved-At: Fri, 15 Jan 2016 15:06:24 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 14:18:34 -0000 this is totally awesome. I might finally be able to upgrade the firmware on a bunch of intel nucs i run now. There was some bug and the never recognized the gptzfs boot code on anything but ancient firmware. Booting via efi should make this go away and i can apply the latest firmwares which will remove a bit more buggy behaviour. Will the current boot strap work ok with stable, or should I wait for the MFC? Its the 2820 nuc boards btw On 15 January 2016 at 02:41, Steven Hartland wrote: > Just wanted to let everyone know that I just finished committing these > changes to the tree. > > Huge thanks to Eric's for his work on this, as well as everyone else who > contributed. > > I've set the target for MFC of 2 weeks, so I hope to be able to get this > into stable/10 before the 10.3 slush, so if you're interested in this > change please test a head build > r294068 ASAP. > > Regards > Steve > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Fri Jan 15 14:42:07 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C20D5A84765 for ; Fri, 15 Jan 2016 14:42:07 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 79AE11B13; Fri, 15 Jan 2016 14:42:07 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id l65so22942380wmf.1; Fri, 15 Jan 2016 06:42:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=a504fexz1WRk4q+DbXhICyvptFY0eFRNv06Wyjij/DY=; b=Um7RafdKolblrk5EhOSB7ofY0vc9YPYXGpLn9IuScYuLtlRkawEocW6ETfe0moEQgu r1JVf8cHWQNGWUy+JBSZhb3+3o43XNHulfqdpWwFoVc6+fPMh5aMFHwEmYWdAGBmmjTl 1V/TPEEgVZuUVBmjBW/w8c9e8uzUPzIbYMB7+0a8+/raEJJcAd69OacxcXqgYXzi6nBf EEK5RW6vz/3qWXc0Ab7t5i2AXUwwz4Q7hioAENRhlMghDdRe/kjPbO6SCQNd8UMc8xrK wAHRNuFQPAlrJGNlh5WtiHaMU02ReqCAsJX7Kl+5WTzIu86iCrHw/s6oeyqvbqUNNHHe 9WkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=a504fexz1WRk4q+DbXhICyvptFY0eFRNv06Wyjij/DY=; b=PMBdLKh0YownzuZiLOUmsu49WPlljZwBAuM/pYBIv4OqEKfCaQEPly3TcB6UwLD1e/ QvYGPy+9MSMozUWKkBg2IARdM64gkdY46n1PaZveODB4f1JuAtnjZZ5soAKFGyxktRfT SFRubocIPZGEMUNBvMm02quaOrxNfkCf3e3nJ8urt4szuHcBuRMQBFDqY+aAemd5adMM pu5ONfSs40hf2DoburNhAfP+fAA7IAkOOsDpqMGPvXNYgmlj7qoIGRHttGvgZ/e2DNKs lTZdmdFx0UhY0/DPPNvaiz3Fd/9Etixu61YIlAuuUQNRmyTKuzTXAoCWFgoHLUynbgUq zP2A== X-Gm-Message-State: ALoCoQl1IxyAKwwLcOXsGYNO1hqbprOnup1TEa4dhIoZH7RwrFPoOE7nr4AJ+KkGnwZAkO8ULRq+skWGzcQyBpgg9aN30Hmj6w== MIME-Version: 1.0 X-Received: by 10.194.117.5 with SMTP id ka5mr10578274wjb.20.1452868925816; Fri, 15 Jan 2016 06:42:05 -0800 (PST) Received: by 10.28.55.132 with HTTP; Fri, 15 Jan 2016 06:42:05 -0800 (PST) In-Reply-To: <569900CD.2040003@metricspace.net> References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> <569900CD.2040003@metricspace.net> Date: Fri, 15 Jan 2016 14:42:05 +0000 Message-ID: Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs From: krad To: Eric McCorkle Cc: Renato Botelho , Steven Hartland , freebsd-hackers@freebsd.org, Gabor Radnai X-Mailman-Approved-At: Fri, 15 Jan 2016 15:14:31 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 14:42:07 -0000 Thanks this is useful information. I did have a look at the way pc bsd was using grub to boot rootonzfs systems, and they used the Kfreebsd options. This looked kind of handy as you might have been able to specify the zfs file system to boot off. This would be a real boost the boot environments as you could easily choose which one to boot into, making upgrade recovery far easier. Presumably the freebsd@ part in your setup refers to the zfs file system? In which case you could have multiple menu entries with different file systems specified, this is assuming the grub config is on the efi disk though? I'm also curious how this would his work when the are multiple pools on the same disk for some reason. On 15 January 2016 at 14:23, Eric McCorkle wrote: > On 01/15/16 06:51, Renato Botelho wrote: > >> On Jan 15, 2016, at 00:41, Steven Hartland >>> wrote: >>> >>> Just wanted to let everyone know that I just finished committing these >>> changes to the tree. >>> >>> Huge thanks to Eric's for his work on this, as well as everyone else who >>> contributed. >>> >>> I've set the target for MFC of 2 weeks, so I hope to be able to get this >>> into stable/10 before the 10.3 slush, so if you're interested in this >>> change please test a head build > r294068 ASAP. >>> >> >> Great work, thanks! >> >> Is there a way to move a installed ZFS system to EFI? >> >> > (Refer to Steven's guide for the simple case where you can create an EFI > partition) > > > == Using GRUB == > > GRUB can be used with loader.efi on a ZFS system (I use this myself, as I > have a Gentoo install in the same ZFS volume) > > Make sure you install GRUB with EFI support (the ports collection will > handle this). The grub port comes with a script that auto-detects > filesystems and sets up a grub.cfg in /boot/grub/. However, that script > won't properly detect ZFS partitions, so you'll need to add it manually. > The entry is simple. > > I have a zfs volume called "data", which has the freebsd system root on > the filesystem data/freebsd. The GRUB entry then is: > > menuentry "FreeBSD" { > search.fs_label data ZFS_PART > chainloader ($ZFS_PART)/freebsd@/boot/loader.efi > } > > The first line searches for the volume "data" and binds its device to the > variable ZFS_PART. The second specifies that the chained bootloader is > under the filesystem "freebsd" (the @ at the end of the name denotes a > filesystem, not a path), with the path /boot/loader.efi > > > == Disks without enough space to make the GPT or EFI partition == > > If you have a ZFS filesystem on an MBR disk, or on a GPT disk without > enough space to create a workable EFI partition, you can use one of ZFS's > lesser-known features: zfs send/recv. > > ZFS send and recv allow an entire filesystem to be serialized out to a > stream, and then read back in. You can use this to dump the entire > filesystem out to a removable storage or an NFS mount. Then, use an > install disk or memstick and repartition your drive, using zfs recv to > recreate the filesystem. > > > == On a Mac == > > (Note: this is based on research that is several years old at this point. > Also, I never actually field tested this myself.) > > Macs are wierd, due to their non-standard EFI. The Mac EFI implementation > looks for an HFS+ partition, and loads the "blessed" file as the EFI > bootloader (this is a special filesystem-level metadata unique to HFS+ > filesystems). > > In order to do an EFI boot on a mac, you'd need some way of manufacturing > an HFS+ partition containing only your bootloader, with that file being > "blessed". The easiest way to do this would be to use a Mac OS install to > create an empty HFS+ filesystem, add your boot loader, then use a shell > command to "bless" it (this command exists, but I don't remember what it is > offhand). It also wouldn't be too hard to write a tool to create an HFS+ > image from a file (I have a half-written implementation of that lying > around somewhere). > > Also note that Macs expect a 200MB EFI partition (which isn't actually > used for anything), and I've heard reports of the firmware flaking out if > it's not there. > > I believe there are also GRUB and rEFIt options for Mac, if you don't want > to go to these lengths. > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Fri Jan 15 15:00:53 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65970A84E07 for ; Fri, 15 Jan 2016 15:00:53 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 014791945; Fri, 15 Jan 2016 15:00:53 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id u188so26505896wmu.1; Fri, 15 Jan 2016 07:00:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=q3R/PK4YT2acz5gXZ51SUBms9EbbELLjUv7dCgTR6tA=; b=hECUTZ2mv/oDPCz7ZzgFA50r/v+MUQScPala2kDU/tM/ILKLA2JDLIWdpHYI7MA0Pw l9xv7MPHMkEW8KhcsDNOP+WQp4eKAIYiJmPXLl227d5q50i1I+0nK3pwrkH37KwFnrQP WQaM9EVRC70L5hXg2viyW00ScY+ueSAE8s555Yq/e7lijV5pW/rWxPmaaG0RikejJNxR Z3RziiVj6KANZhVxSOOdUJLx0PQ+APp5akIFy+YUOTqtzLNx1KnknFNTA94ED6GC/bxf 9xkIu08Rw4xxoLgXlL8LzDbgGxcp7l0rupl8cBvP9KWLFI4jj+c9QqIbNjQkIhY6VCId enSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=q3R/PK4YT2acz5gXZ51SUBms9EbbELLjUv7dCgTR6tA=; b=iN09gu0t1klgp2Ktgcr8qfBlH6ToLam1BqfdMwRS3ltvwP1Ex8jGHHjIBPGlZy5OiS pZmbrnO4+trSny5MH6ZgMW9K4RPlJdh+RiysypQ6dUrb1WBGf3oIZsYxEVCkmg1m2/bD D73balKcNLy+H2wondQsBx2AWjqnQFNXkyA+WIfmSpCE/6aTpYbc6Mfy7sna9rQidFct kYGFZnfoABY5RNdqTg1zJV+zfm49Wn9t4cHskXzOsHk8cp2xOfZ4ALORePWE6eLxxiBm T8BXMZaOkjhgKZneOlg83JOQBvJedHNuzMb+Cq2WbjFeJRtnCTZvoUAMykEjsG8una2z yj7A== X-Gm-Message-State: AG10YORwVgxw2ExfwT4xrqRVZiHmv0gzc150nKg7U/dISQXereU7UpQjJ4GQGY5LRQGxUJM2gTuVHkhKVk6W4g== MIME-Version: 1.0 X-Received: by 10.28.125.20 with SMTP id y20mr3853614wmc.19.1452870051483; Fri, 15 Jan 2016 07:00:51 -0800 (PST) Received: by 10.28.55.132 with HTTP; Fri, 15 Jan 2016 07:00:51 -0800 (PST) In-Reply-To: <569906A1.8040700@multiplay.co.uk> References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> <569900CD.2040003@metricspace.net> <569906A1.8040700@multiplay.co.uk> Date: Fri, 15 Jan 2016 15:00:51 +0000 Message-ID: Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs From: krad To: Steven Hartland Cc: Eric McCorkle , Renato Botelho , freebsd-hackers@freebsd.org, Gabor Radnai X-Mailman-Approved-At: Fri, 15 Jan 2016 15:15:18 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 15:00:53 -0000 It get even more awesome 8) On 15 January 2016 at 14:48, Steven Hartland wrote: > ZFS Boot Environments (BE) support was also wired up to Beastie last night > by Allan for those interested in that :) > > > On 15/01/2016 14:42, krad wrote: > > Thanks this is useful information. I did have a look at the way pc bsd was > using grub to boot rootonzfs systems, and they used the Kfreebsd options. > This looked kind of handy as you might have been able to specify the zfs > file system to boot off. This would be a real boost the boot environments > as you could easily choose which one to boot into, making upgrade recovery > far easier. > > Presumably the freebsd@ part in your setup refers to the zfs file system? > In which case you could have multiple menu entries with different file > systems specified, this is assuming the grub config is on the efi disk > though? > > I'm also curious how this would his work when the are multiple pools on > the same disk for some reason. > > > > On 15 January 2016 at 14:23, Eric McCorkle wrote: > >> On 01/15/16 06:51, Renato Botelho wrote: >> >>> On Jan 15, 2016, at 00:41, Steven Hartland < >>>> killing@multiplay.co.uk> wrote: >>>> >>>> Just wanted to let everyone know that I just finished committing these >>>> changes to the tree. >>>> >>>> Huge thanks to Eric's for his work on this, as well as everyone else >>>> who contributed. >>>> >>>> I've set the target for MFC of 2 weeks, so I hope to be able to get >>>> this into stable/10 before the 10.3 slush, so if you're interested in this >>>> change please test a head build > r294068 ASAP. >>>> >>> >>> Great work, thanks! >>> >>> Is there a way to move a installed ZFS system to EFI? >>> >>> >> (Refer to Steven's guide for the simple case where you can create an EFI >> partition) >> >> >> == Using GRUB == >> >> GRUB can be used with loader.efi on a ZFS system (I use this myself, as I >> have a Gentoo install in the same ZFS volume) >> >> Make sure you install GRUB with EFI support (the ports collection will >> handle this). The grub port comes with a script that auto-detects >> filesystems and sets up a grub.cfg in /boot/grub/. However, that script >> won't properly detect ZFS partitions, so you'll need to add it manually. >> The entry is simple. >> >> I have a zfs volume called "data", which has the freebsd system root on >> the filesystem data/freebsd. The GRUB entry then is: >> >> menuentry "FreeBSD" { >> search.fs_label data ZFS_PART >> chainloader ($ZFS_PART)/freebsd@/boot/loader.efi >> } >> >> The first line searches for the volume "data" and binds its device to the >> variable ZFS_PART. The second specifies that the chained bootloader is >> under the filesystem "freebsd" (the @ at the end of the name denotes a >> filesystem, not a path), with the path /boot/loader.efi >> >> >> == Disks without enough space to make the GPT or EFI partition == >> >> If you have a ZFS filesystem on an MBR disk, or on a GPT disk without >> enough space to create a workable EFI partition, you can use one of ZFS's >> lesser-known features: zfs send/recv. >> >> ZFS send and recv allow an entire filesystem to be serialized out to a >> stream, and then read back in. You can use this to dump the entire >> filesystem out to a removable storage or an NFS mount. Then, use an >> install disk or memstick and repartition your drive, using zfs recv to >> recreate the filesystem. >> >> >> == On a Mac == >> >> (Note: this is based on research that is several years old at this >> point. Also, I never actually field tested this myself.) >> >> Macs are wierd, due to their non-standard EFI. The Mac EFI >> implementation looks for an HFS+ partition, and loads the "blessed" file as >> the EFI bootloader (this is a special filesystem-level metadata unique to >> HFS+ filesystems). >> >> In order to do an EFI boot on a mac, you'd need some way of manufacturing >> an HFS+ partition containing only your bootloader, with that file being >> "blessed". The easiest way to do this would be to use a Mac OS install to >> create an empty HFS+ filesystem, add your boot loader, then use a shell >> command to "bless" it (this command exists, but I don't remember what it is >> offhand). It also wouldn't be too hard to write a tool to create an HFS+ >> image from a file (I have a half-written implementation of that lying >> around somewhere). >> >> Also note that Macs expect a 200MB EFI partition (which isn't actually >> used for anything), and I've heard reports of the firmware flaking out if >> it's not there. >> >> I believe there are also GRUB and rEFIt options for Mac, if you don't >> want to go to these lengths. >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to " >> >> freebsd-hackers-unsubscribe@freebsd.org" >> > > > From owner-freebsd-hackers@freebsd.org Fri Jan 15 15:53:56 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51244A84538 for ; Fri, 15 Jan 2016 15:53:56 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 329071329 for ; Fri, 15 Jan 2016 15:53:55 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id DEDA2D339 for ; Fri, 15 Jan 2016 15:53:54 +0000 (UTC) Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: freebsd-hackers@freebsd.org References: <9418E44F-114E-4ABA-A32D-416297BCDA9F@metricspace.net> <56985C6A.6040209@multiplay.co.uk> From: Allan Jude Message-ID: <5699161D.60503@freebsd.org> Date: Fri, 15 Jan 2016 10:54:05 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8Ws8tQwxN3shev8h9QRC1tnS1Hbfqevw1" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jan 2016 15:53:56 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --8Ws8tQwxN3shev8h9QRC1tnS1Hbfqevw1 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2016-01-15 09:18, krad wrote: > this is totally awesome. I might finally be able to upgrade the firmwar= e on > a bunch of intel nucs i run now. There was some bug and the > never recognized the gptzfs boot code on anything but ancient firmware.= > Booting via efi should make this go away and i can apply the latest > firmwares which will remove a bit more buggy behaviour. >=20 > Will the current boot strap work ok with stable, or should I wait for t= he > MFC? >=20 > Its the 2820 nuc boards btw >=20 >=20 Hey, before you change the firmware or reinstall the NUC with UEFI Can you try installing using the 'zfs auto' menu item in bsdinstall, and set the partition type to 'gpt + active', and see if that works on the NUC? If it does, I'll look at making the installer detect that hardware and work around that problem. --=20 Allan Jude --8Ws8tQwxN3shev8h9QRC1tnS1Hbfqevw1 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJWmRYgAAoJEBmVNT4SmAt+uMsQAMKfNrIyxDf9snV285VYGFbL NOpYo/TxcCaaNDpoUOTdzki/2JdG0NJPxiz/8xkUZljGE9kwW6HFAMfk+2GD9Zly 1cc4bl+VJjv5NevpqJ2+hDgC8y3avbFvnj+9r6JeNfhUydJfWJulLjJS+gfFFgnD u1xUtMRhtXxNtXQaMWnsvgHTaH+S8uwcx9s2lXxnxlMMkFVlHTutSMktxva4kE40 VmgXWNCJQJVYRli4DN7awA2gMnZy4EadbmxN1Uf42fSA5V3nUdn1yiUT00pu6PFO B/r8DMtgfYkkAsRCFfPP2VOuF/eBGu2LS5X5+ANRiWn6T0kGB7uAaHDJ3hp8ffQ9 yVsThPlcgM9mFmfFjsYK3OCCUGi9UGgMjg76Ov828ryGKcvMhQ/4scXgFSfwyhWH gULBk+zfhbI4uKHsQX544k4G8h01zxVsZ2e1CPZeynT+CmHsaJFKWyThmmJIPl7u l+y20SApxSaBZoUsQVEci+AIFHGe+txpndS7GJBHtMlxhcGy9ViiSVa2gL1aFCCL HMNwHev01zL3lYwTi75OiJcy8cSK8jv+yQZyUCOgHoaPt6TDUqC2AYhqWtxaEIEg mfKIHxtwKMEFJVFrQDXEOmAi4B2B110KX3nvfHS13SZDkVbZecDF7d4DHK67sR/r tNfibydd0nNe1Ym+VnEi =CwNM -----END PGP SIGNATURE----- --8Ws8tQwxN3shev8h9QRC1tnS1Hbfqevw1-- From owner-freebsd-hackers@freebsd.org Sat Jan 16 03:44:51 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77FA1A846D1 for ; Sat, 16 Jan 2016 03:44:51 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from shell1.rawbw.com (shell1.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 4D42413CA for ; Sat, 16 Jan 2016 03:44:51 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from yuri.doctorlan.com (c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128]) (authenticated bits=0) by shell1.rawbw.com (8.15.1/8.15.1) with ESMTPSA id u0G3aefS038133 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 15 Jan 2016 19:36:41 -0800 (PST) (envelope-from yuri@rawbw.com) X-Authentication-Warning: shell1.rawbw.com: Host c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128] claimed to be yuri.doctorlan.com To: Freebsd hackers list From: Yuri Subject: How to send EOF to the popen(3) pipe? Message-ID: <5699BAC9.3060407@rawbw.com> Date: Fri, 15 Jan 2016 19:36:41 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 03:44:51 -0000 Is there a way to send the EOF to popen(3) pipe? Imagine the situation when the child process works in a stream fashion, processes objects one by one, and stops on EOF from stdin. One has to be able to send EOF to get to the end of the last processed object. Otherwise reading from the descriptor will generally block. Linux man page says that popen is unidirectional on Linux. But FreeBSD supports bi-directional popen. Yuri From owner-freebsd-hackers@freebsd.org Sat Jan 16 04:37:11 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA239A6D794 for ; Sat, 16 Jan 2016 04:37:11 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from um-nip4-missouri-out.um.umsystem.edu (um-nip4-missouri-out.um.umsystem.edu [198.209.49.177]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "um-tip1.um.umsystem.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 481131C13 for ; Sat, 16 Jan 2016 04:37:10 +0000 (UTC) (envelope-from stephen@missouri.edu) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2CRCAB+yJlW/8aeoM9egzpSLEIFiFC1HCSFawKBLzwQAQEBAQEBAX8LhDQBAQEDAXgGCwIBCBgJFg8JAwIBAgEgJQIEDQgBAYgPCA69MgGDAAwBIItUhFWEaAWXGQGFRoVdhBgWNIclhTSKbINxOSuECoZhAYEHAQEB X-IPAS-Result: A2CRCAB+yJlW/8aeoM9egzpSLEIFiFC1HCSFawKBLzwQAQEBAQEBAX8LhDQBAQEDAXgGCwIBCBgJFg8JAwIBAgEgJQIEDQgBAYgPCA69MgGDAAwBIItUhFWEaAWXGQGFRoVdhBgWNIclhTSKbINxOSuECoZhAYEHAQEB Received: from um-tcas2.um.umsystem.edu ([207.160.158.198]) by um-nip4-exch-relay.um.umsystem.edu with ESMTP; 15 Jan 2016 22:36:00 -0600 Received: from UM-MBX-N02.um.umsystem.edu ([169.254.5.100]) by UM-TCAS2.um.umsystem.edu ([207.160.158.198]) with mapi id 14.03.0266.001; Fri, 15 Jan 2016 22:35:59 -0600 From: "Montgomery-Smith, Stephen" To: "freebsd-hackers@freebsd.org" Subject: Re: How to send EOF to the popen(3) pipe? Thread-Topic: How to send EOF to the popen(3) pipe? Thread-Index: AQHRUBBE+GmWWEt160m/VZ+mWUx+kJ798zuA Date: Sat, 16 Jan 2016 04:35:58 +0000 Message-ID: <5699C8AB.7070006@missouri.edu> References: <5699BAC9.3060407@rawbw.com> In-Reply-To: <5699BAC9.3060407@rawbw.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 x-originating-ip: [207.160.158.194] Content-Type: text/plain; charset="Windows-1252" Content-ID: <3DFE87A541E68B4CB727932DF854D0F6@missouri.edu> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 04:37:11 -0000 On 01/15/2016 09:36 PM, Yuri wrote: > Is there a way to send the EOF to popen(3) pipe? >=20 > Imagine the situation when the child process works in a stream fashion, > processes objects one by one, and stops on EOF from stdin. One has to be > able to send EOF to get to the end of the last processed object. > Otherwise reading from the descriptor will generally block. >=20 Maybe I am displaying my ignorance. But wouldn't you do this by invoking the function pclose? > Linux man page says that popen is unidirectional on Linux. But FreeBSD > supports bi-directional popen. My memory of using this was that this could gridlock because of buffering. Suppose process A popens a process B. A sends a message to B. But the end of the message often never arrives unless A also does a fflush. So then B just sits there waiting. If A does do a fflush, then B knows to reply back. But B also has to do a fflush after it has replied. If B happens to be a piece of code you haven't written yourself, there is nothing you can do to force it to do a fflush. If you run B in a terminal, you won't notice this behavior, because the stdio package automatically does a fflush at the end of every new line if it is writing out to a terminal. You can get around this by using pseudo-terminals to connect process A to process B, but I don't think popen is this sophisticated. But you could probably do this using a script called unbuffer: http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe This is included in the lang/expect port, but for some reason this script is not in /usr/local/bin but in /usr/local/share/expect But now I see what your original problem was. When you open a bidirectional popen, you really need to have two separate pclose command - one for each direction. You want to close the input stream to the program before you close the output stream to the program. Yes, I think un unsophisticated bidirectional popen is fairly useless. I think that is why Linux never bothered to implement it. Look at the first answer to this question: http://stackoverflow.com/questions/6171552/popen-simultaneous-read-and-writ= e From owner-freebsd-hackers@freebsd.org Sat Jan 16 09:00:24 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E51EA84225 for ; Sat, 16 Jan 2016 09:00:24 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-ob0-x22a.google.com (mail-ob0-x22a.google.com [IPv6:2607:f8b0:4003:c01::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C774119C9 for ; Sat, 16 Jan 2016 09:00:23 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: by mail-ob0-x22a.google.com with SMTP id is5so128757326obc.0 for ; Sat, 16 Jan 2016 01:00:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=fT/Jf3wbsapiDdVS8Jk3+2Jb2c53h2wQRFEc46KxLO8=; b=hrEtMmtsRijeXH4dSe6ATn0C9OLj1K3zEMYZO7jwf6fVJsblc6dc8g80k6s8/Gw1Zh 1LK6/qyEVXh0mQONxx/9ch8VSRxG5jEL70pdhAnfTJKPSgVP0Zmrcqjgl5e7dlt6OBuq C//FHhU9tRHP+sHSeZEdBf14HKvhcXclFEvwbzpaWY9w4cTcTbYaSbuqvN6nDXusf/rl N0Sg5D6BHTKTuKGherbi0k+mly7q5fypiXdkh/NjWQW4Ltn3Xv+9mWs5UvKxdxPF9SZZ Y7ygUkywavHg/ibINQKV5G8IVZiv6xFtDTPcTtIYAwkbruh0fpCpwfNAvUEwj2lNrv2y bBsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=fT/Jf3wbsapiDdVS8Jk3+2Jb2c53h2wQRFEc46KxLO8=; b=LtxOO7ga9vkB5jVhfR2jUSB0l4/z4OG6msaB5FOiyvyfsVEUDoJEKB80v5Ac92OiWk DS307EZFwj1Qe1qcCh4WuuyhBEePVxK0Nx+ROwrqOmH4wr8Eeb6034VGyUuhFa3/tr1n SCWBeuzyda9g5eesBkKzcgXwqCjE6VxrGjxXwnac3tnDs1MWGg8q69xYmkc4mvJ/sB7l 4q7dzaoEGSDJutscaXi9osfxYU/Mw0eP3SYkg9inqaTzJSL4Eh1DAZHnOkbz/k6ILqPc AmJ/U4rpdIwFipu1AdP9hHGH5aVxL2le/Bz4oZvFJ3i2YrSC8AtYod2ZoDPwsD4Q7qUt JJ8A== X-Gm-Message-State: ALoCoQnD1Gf+GmWK9wHWUPZdPVVLDb94IE0NZxpYFBbTJ3cemThQmda+z491bG9lnIEy9lOWcPDql85FD8bYhZtNH1mWiMT3xQ== X-Received: by 10.60.159.230 with SMTP id xf6mr11821255oeb.43.1452934823012; Sat, 16 Jan 2016 01:00:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.202.72.197 with HTTP; Sat, 16 Jan 2016 00:59:43 -0800 (PST) In-Reply-To: <5669C3BD.4010402@multiplay.co.uk> References: <5669C3BD.4010402@multiplay.co.uk> From: Outback Dingo Date: Sat, 16 Jan 2016 09:59:43 +0100 Message-ID: Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: Steven Hartland Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 09:00:24 -0000 On Thu, Dec 10, 2015 at 7:26 PM, Steven Hartland wrote: > We've used Eric's hard work which is currently under review here: > https://reviews.freebsd.org/D4104 > > I'm pleased to report we can now successfully EFI boot root ZFS from a > raidz2 pool on Intel P3700 NVMe drives :) > > Here's a guide for those interested: > http://blog.multiplay.co.uk/2015/12/freebsd-10-2-release-efi-zfs-root-boot/ > What exactly did you diff this from, i pull a clean release/10.2.0 and applied the patches from the site which resulted in 68 rejects. find . -name "*.rej" -print | wc -l 68 find . -name "*.rej" -print ./sys/boot/usb/Makefile.rej ./sys/boot/forth/loader.rc.rej ./sys/boot/forth/menu.rc.rej ./sys/boot/ficl/ia64/sysdep.c.rej ./sys/boot/ficl/ia64/sysdep.h.rej ./sys/boot/ia64/common/exec.c.rej ./sys/boot/ia64/common/libia64.h.rej ./sys/boot/ia64/common/autoload.c.rej ./sys/boot/ia64/common/copy.c.rej ./sys/boot/ia64/common/bootinfo.c.rej ./sys/boot/ia64/common/devicename.c.rej ./sys/boot/ia64/common/icache.c.rej ./sys/boot/ia64/ski/delay.c.rej ./sys/boot/ia64/ski/libski.h.rej ./sys/boot/ia64/ski/efi_stub.c.rej ./sys/boot/ia64/ski/acpi_stub.c.rej ./sys/boot/ia64/ski/sal_stub.c.rej ./sys/boot/ia64/ski/exit.c.rej ./sys/boot/ia64/ski/pal_stub.S.rej ./sys/boot/ia64/ski/ssc.c.rej ./sys/boot/ia64/ski/skiconsole.c.rej ./sys/boot/ia64/ski/main.c.rej ./sys/boot/ia64/ski/start.S.rej ./sys/boot/ia64/ski/time.c.rej ./sys/boot/ia64/ski/skimd.c.rej ./sys/boot/ia64/ski/skifs.c.rej ./sys/boot/ia64/ski/conf.c.rej ./sys/boot/ia64/efi/efimd.c.rej ./sys/boot/ia64/efi/main.c.rej ./sys/boot/ia64/efi/start.S.rej ./sys/boot/ia64/efi/conf.c.rej ./sys/boot/i386/boot0/boot0ext.S.rej ./sys/boot/i386/efi/elf32_freebsd.c.rej ./sys/boot/i386/efi/reloc.c.rej ./sys/boot/i386/efi/start.S.rej ./sys/boot/i386/efi/efimd.c.rej ./sys/boot/i386/efi/i386_copy.c.rej ./sys/boot/i386/efi/bootinfo.c.rej ./sys/boot/i386/efi/exec.c.rej ./sys/boot/fdt/dts/arm/p2041rdb.dts.rej ./sys/boot/fdt/dts/arm/p5020ds.dts.rej ./sys/boot/fdt/dts/arm/pandaboard.dts.rej ./sys/boot/fdt/dts/arm/beaglebone-black.dts.rej ./sys/boot/fdt/dts/arm/exynos5250-chromebook.dts.rej ./sys/boot/fdt/dts/arm/p3041ds.dts.rej ./sys/boot/fdt/dts/arm/am335x.dtsi.rej ./sys/boot/fdt/dts/arm/am335x-evm.dts.rej ./sys/boot/fdt/dts/arm/beaglebone.dts.rej ./sys/boot/amd64/Makefile.inc.rej ./sys/boot/amd64/efi/framebuffer.c.rej ./sys/boot/amd64/efi/autoload.c.rej ./sys/boot/amd64/efi/devicename.c.rej ./sys/boot/amd64/efi/conf.c.rej ./sys/boot/amd64/efi/amd64_tramp.S.rej ./sys/boot/amd64/efi/start.S.rej ./sys/boot/amd64/efi/reloc.c.rej ./sys/boot/amd64/efi/framebuffer.h.rej ./sys/boot/amd64/efi/x86_efi.h.rej ./sys/boot/amd64/efi/elf64_freebsd.c.rej ./sys/boot/amd64/efi/main.c.rej ./sys/boot/amd64/efi/copy.c.rej ./sys/boot/amd64/efi/bootinfo.c.rej ./sys/boot/amd64/boot1.efi/Makefile.fat.rej ./sys/boot/amd64/boot1.efi/boot1.c.rej ./sys/boot/amd64/boot1.efi/generate-fat.sh.rej ./sys/boot/amd64/boot1.efi/fat.tmpl.bz2.uu.rej ./sys/boot/common/loader.8.rej ./sys/boot/common/md.c.rej > > On 04/11/2015 12:35, krad wrote: > >> is there not anyway freebsd could provide patched signed binaries outside >> the main distros for testing purposes, as it should be fairly straight >> forward to drop them in? I think you might be a much bigger audience for >> testing then? >> >> On 2 November 2015 at 19:16, Gabor Radnai wrote: >> >> Appreciate, thank you very much. I think my confusion is because the >>> latest >>> patch you provided on the list >>> on *Fri Oct 23 11:19:07 UTC 2015* >>> >>> >>> http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20151023/db1ac571/attachment.bin >>> >>> is not a full patch but a diff to your original patch. Anyhow, I'm ok now >>> applying original patch and this latest diff everything seems fine >>> (apart that for some reason my server dislikes booting automatically from >>> the EFI partition, manually loading bootx64.efi works like charm. >>> but that's definitely nothing to do with your great work). >>> >>> Thanks. >>> _______________________________________________ >>> freebsd-hackers@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >>> To unsubscribe, send any mail to " >>> freebsd-hackers-unsubscribe@freebsd.org" >>> >>> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Sat Jan 16 13:00:05 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3391BA84AF8 for ; Sat, 16 Jan 2016 13:00:05 +0000 (UTC) (envelope-from gabor.radnai@gmail.com) Received: from mail-oi0-x22d.google.com (mail-oi0-x22d.google.com [IPv6:2607:f8b0:4003:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F132A118D for ; Sat, 16 Jan 2016 13:00:04 +0000 (UTC) (envelope-from gabor.radnai@gmail.com) Received: by mail-oi0-x22d.google.com with SMTP id o124so142147201oia.3 for ; Sat, 16 Jan 2016 05:00:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=I1MwSaAVxCrBOq8B96UZJzdH1KolReCbXMQJbZgtc4o=; b=eiRvpeCi8u/19cMegzavVwGHRjliFBfQzKjtLxdcbcJ3+UNiJhljEj+Ia2dWfjbNIa v3sOgqXdM7BqszVhUddoS2km1eEcLFcea3EnALpqW5FlN8nsLj2yy6tO3B9t1KFKD+cc ol4bpxe46+02LUlXdM9zcXm/yWwhIrfeXvM6ha3p5LYpSrspurkseL1lcVy8NZlAX+HR ALMOdcsunIN1f9WFDG455JwZ4k/QEKtGzf+Z7g4v4M96ZY9gAnUWAsnEbWQ0834jukFu AdZ1GG91IRhiimSLOJtfyj+RbZFlb8GiM53YPBnuPxHMwfragFDBUl1ge/sHy7XXt/Wj 2jIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=I1MwSaAVxCrBOq8B96UZJzdH1KolReCbXMQJbZgtc4o=; b=ZLq//oJ6g28tMk3WuQwlas/naAuu92GlHyAU9hHjg2oquBZKZbzgw5JEEXjVBPTBk2 gh7YR0aQ8vzF7zp5X38ZyDWw9P14C0fp3t1vnWGOinfNH+FAHfgEjc04oNfyWnkP/j3V mO4k0IOtd2HDJGpl11sNilvntpQ+VtkvS9B66YVjq7a/aEXtv0Cx44gjtPT7NXTNHuVw 70orA8EDS+QqvTBxmR6moxGg+xRuRaJlqvcV+2aBAoW53+qS3QTMRSHq/WjC52mwJNkB IQIxML3A8IBa617GaJ9dFVR7GRytB2kwlSbqaBh4U/xmyYNkKzeTB8iMQRe62nHgGVb3 ihTQ== X-Gm-Message-State: ALoCoQkj+U4G1Lh+fB/XlPsxeCVepwlHfmiz7udzMRZfxYC6o1a5a09++Thz4Kl2Aq8q1eSWOiSYRnFPF94h5NacWzlTL/0H3g== X-Received: by 10.202.64.8 with SMTP id n8mr11796787oia.112.1452949204272; Sat, 16 Jan 2016 05:00:04 -0800 (PST) MIME-Version: 1.0 From: Gabor Radnai Date: Sat, 16 Jan 2016 12:59:54 +0000 Message-ID: Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 13:00:05 -0000 Hi, I tried to check HardenedBSD for EFI + ZFS on root but the installer says this combo is not supported. What am I missing? Thanks From owner-freebsd-hackers@freebsd.org Sat Jan 16 13:15:33 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DE10A8421A for ; Sat, 16 Jan 2016 13:15:33 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk0-x230.google.com (mail-qk0-x230.google.com [IPv6:2607:f8b0:400d:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18E941CCC for ; Sat, 16 Jan 2016 13:15:33 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk0-x230.google.com with SMTP id b66so3570450qkf.3 for ; Sat, 16 Jan 2016 05:15:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=rTfnpTD1M17n76H18DmN6+WnNeurWEbwGCcp21IkBQ8=; b=CwNQjLRyL5bccmopFu4gVZwX1fmzxCX0uu0MkzyDnwNqGHnsmFCo4U5LQI4GSh3jzf xGWCnDDtCw/IfRsadt0YIfMEXvD2z+LVDnHmmgAtrUk0u7hbWLS/3zbkycVDZMSZhB6t qPxgNkhGAOUh93PENDv913KbuXVLV4STUKt8uhxzvhQBos3RsF+Cu5EfAYqcexdHp83Q g2nLsF8W9aDnbZ6ulfDJIkeuQiQXQCOhiFAohvj/JA8nd4e17Af3noAMLFynmo29KKWx JKa4mcD6suHTW9Pak852KpRz7xrpNF9il45vEbpkN1u/FahIcCJQn/tMEGASwXg2XmlZ OcxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=rTfnpTD1M17n76H18DmN6+WnNeurWEbwGCcp21IkBQ8=; b=TFRqt0Xd6Suln03DZzUDzrdvBTEIsjIt6PhWAjGgMNUkggEL9syyrg304U0KRReC8S JrkmwhjreAktiUsd3CC6MQ23K6g+52A4X4VPh+nHR3qg1txUzyjWOQkc1Sp6iSBdbWLL 3urucqsJEHpmjTP8F0CDQBuJvEPpE7LWIO68h9TTQZnwTq0TCJwVtqK8NdH+bckJfQU3 a7VOwAhtqACtUEzXuQUFukcdcBHrQXzVEgV6RbyWhQapjlcy/2OGPiwnBdqIdg9+RqP5 84BZ7EuC1gA/wSgFpQ9Ble9IKOwkXyubeBxObdF9Ewum4RZukSrgcJv1ZfKc4fRY1IGf eCLA== X-Gm-Message-State: ALoCoQnJQfMhPPpWC9sYoJ9lypOGUvy7ZTl8PvSmfZiEfUzd6hqtSO+A7ngiYmdkhzPMEHwgJBlJKZ0xW7z2Y/HjBW7KsIWM1Q== X-Received: by 10.55.71.76 with SMTP id u73mr20112050qka.6.1452950132199; Sat, 16 Jan 2016 05:15:32 -0800 (PST) Received: from mutt-hardenedbsd ([12.239.54.101]) by smtp.gmail.com with ESMTPSA id 67sm6386336qht.14.2016.01.16.05.15.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 16 Jan 2016 05:15:30 -0800 (PST) Date: Sat, 16 Jan 2016 08:15:29 -0500 From: Shawn Webb To: Gabor Radnai Cc: freebsd-hackers@freebsd.org Subject: Re: EFI/ZFS Update: successful tests, need more complex vdevs Message-ID: <20160116131529.GA59919@mutt-hardenedbsd> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="PEIAKu/WMn1b1Hv9" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 11.0-CURRENT-HBSD FreeBSD 11.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 13:15:33 -0000 --PEIAKu/WMn1b1Hv9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jan 16, 2016 at 12:59:54PM +0000, Gabor Radnai wrote: > Hi, >=20 > I tried to check HardenedBSD for EFI + ZFS on root but the installer says > this combo is not supported. What am I missing? The "zfs auto install" part of bsdinstall still needs to be updated to know about the new ZFS UEFI support. The manual partition selection (which is called SADE) supports it, but you won't get ZFS BE support by using that. I talked with Allan Jude a little bit about it and he inferred that he's working on zfs auto install support (though "inferred" doesn't mean he actually is). Thanks, --=20 Shawn Webb HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --PEIAKu/WMn1b1Hv9 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWmkJvAAoJEGqEZY9SRW7uvewP/2izzjgLH6z646z4Xp4kFMYl OLWwWrnxBL02ioeX9kgNIPmc4G2UpinmOf4C5cPQdZSAM9lgeGsS+a612JSI3J+i IcfUQnzaaYtVZjXAx5HeNS9nSsytN+8XVBaxs7TVEB7tcvI5eIT8CuxyTrgDv69J 11LlYzWcRrfmSEc1DAHfAw8QrVZqwTAF7FWbQR+svFA7XyGy9t+YMk72L/ll0OGw TM3cicWZtdIkOZj3M08L+ekz3f+HK6Agt8sg1/afc5ClSlVwP5LxzfqMdaPHUFYh IifbJNCDxSCHctZ3cUJlrP+tNsjctfZpOU7aC9yzQZA58cGzwJC+GAX3p3VNSeOH gKHGHz7gBDB7oLGukoWl6pSWUs/CcnK5/aexLAy2pUg8WTy/bCl3S00k5Jh3yv+P kBBEgLkvfJdE0wC7OX2Nz6CmfXJMhwjCuzVMM+DenqxEgGmIELv5rGJl5vIBqcwn XWvtSx8/PqW7ZwWTjGqPKKv/edxdfGVIO3klzq5yYkZ1Kh9WVnSehJPEN/mmJOHF vKfwpCEWopsCQDq4X6ATGDQ6GnS7BVeHutCkcfe7dUfg0OCXtZMo4KJZ/O9ZUDKL qs4rgYq0ZI6T/+VPN2G6i8y8jsg13p++EeDScfDGuoYlpmV71IrQNQ8mKV50+hUa tgsN0+u7dhFn1FUpIyUJ =IcuG -----END PGP SIGNATURE----- --PEIAKu/WMn1b1Hv9-- From owner-freebsd-hackers@freebsd.org Sat Jan 16 17:59:37 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F545A85B67 for ; Sat, 16 Jan 2016 17:59:37 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from shell1.rawbw.com (shell1.rawbw.com [198.144.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0CCEC19CD for ; Sat, 16 Jan 2016 17:59:36 +0000 (UTC) (envelope-from yuri@rawbw.com) Received: from yuri.doctorlan.com (c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128]) (authenticated bits=0) by shell1.rawbw.com (8.15.1/8.15.1) with ESMTPSA id u0GHxZqD012892 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sat, 16 Jan 2016 09:59:35 -0800 (PST) (envelope-from yuri@rawbw.com) X-Authentication-Warning: shell1.rawbw.com: Host c-50-184-63-128.hsd1.ca.comcast.net [50.184.63.128] claimed to be yuri.doctorlan.com Subject: Re: How to send EOF to the popen(3) pipe? To: "Montgomery-Smith, Stephen" , "freebsd-hackers@freebsd.org" References: <5699BAC9.3060407@rawbw.com> <5699C8AB.7070006@missouri.edu> From: Yuri Message-ID: <569A8508.80908@rawbw.com> Date: Sat, 16 Jan 2016 09:59:36 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <5699C8AB.7070006@missouri.edu> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 17:59:37 -0000 On 01/15/2016 20:35, Montgomery-Smith, Stephen wrote: > Maybe I am displaying my ignorance. But wouldn't you do this by > invoking the function pclose? No, pclose kills the process and returns the exit code. Half-closed connection though can be alive for a while, until the other side finishes and closes the pipe. > My memory of using this was that this could gridlock because of > buffering. Suppose process A popens a process B. A sends a message to Gridlocks are possible if reads/writes are performed in the wrong order. But this is besides the point of the original question. I think the answer to my question is "no". popen(3) can't send EOF. Protocol needs to support EOF signal on the application-level. Yuri From owner-freebsd-hackers@freebsd.org Sat Jan 16 18:08:58 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D52B9A84069 for ; Sat, 16 Jan 2016 18:08:58 +0000 (UTC) (envelope-from stephen@missouri.edu) Received: from mst-rip6-missouri-out.um.umsystem.edu (mst-rip6-missouri-out.um.umsystem.edu [198.209.50.149]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "um-tip1.um.umsystem.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 770F51A66 for ; Sat, 16 Jan 2016 18:08:58 +0000 (UTC) (envelope-from stephen@missouri.edu) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2AmBQDWhppW/9KeoM9egzpSc4hQtR4ihW0CgSY8EAEBAQEBAQGBCoQ1AQEEeBECAQgYCRYPCQMCAQIBICUCBAEMCAEBiBcOvFYBgwABAQEHAQEBAQEaBItUiT0FlxoBhUeCdJQkjl05K4QLhh4HPAGBBwEBAQ X-IPAS-Result: A2AmBQDWhppW/9KeoM9egzpSc4hQtR4ihW0CgSY8EAEBAQEBAQGBCoQ1AQEEeBECAQgYCRYPCQMCAQIBICUCBAEMCAEBiBcOvFYBgwABAQEHAQEBAQEaBItUiT0FlxoBhUeCdJQkjl05K4QLhh4HPAGBBwEBAQ Received: from um-ncas5.um.umsystem.edu ([207.160.158.210]) by mst-rip6-exch-relay.um.umsystem.edu with ESMTP; 16 Jan 2016 12:08:45 -0600 Received: from UM-MBX-N02.um.umsystem.edu ([169.254.5.100]) by UM-NCAS5.um.umsystem.edu ([207.160.158.210]) with mapi id 14.03.0266.001; Sat, 16 Jan 2016 12:08:45 -0600 From: "Montgomery-Smith, Stephen" To: Yuri , "freebsd-hackers@freebsd.org" Subject: Re: How to send EOF to the popen(3) pipe? Thread-Topic: How to send EOF to the popen(3) pipe? Thread-Index: AQHRUBBE+GmWWEt160m/VZ+mWUx+kJ798zuAgADgjACAAAKNAA== Date: Sat, 16 Jan 2016 18:08:44 +0000 Message-ID: <569A872C.2010806@missouri.edu> References: <5699BAC9.3060407@rawbw.com> <5699C8AB.7070006@missouri.edu> <569A8508.80908@rawbw.com> In-Reply-To: <569A8508.80908@rawbw.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 x-originating-ip: [207.160.158.194] Content-Type: text/plain; charset="Windows-1252" Content-ID: <11C04C1810695B4E9DACE3D33305AF3A@missouri.edu> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 18:08:58 -0000 On 01/16/2016 11:59 AM, Yuri wrote: > On 01/15/2016 20:35, Montgomery-Smith, Stephen wrote: >> Maybe I am displaying my ignorance. But wouldn't you do this by >> invoking the function pclose? >=20 > No, pclose kills the process and returns the exit code. Half-closed > connection though can be alive for a while, until the other side > finishes and closes the pipe. >=20 >> My memory of using this was that this could gridlock because of >> buffering. Suppose process A popens a process B. A sends a message to >=20 > Gridlocks are possible if reads/writes are performed in the wrong order. > But this is besides the point of the original question. >=20 > I think the answer to my question is "no". popen(3) can't send EOF. > Protocol needs to support EOF signal on the application-level. Yes. I think a "proper" implementation would be via a function like popen2 https://emergent.unpythonic.net/01108826729, which creates 2 file handles, not just one. Then to send the EOF, you would close one of the file handles (the "to_child"), and then only close the other file handle (the "from_child") when it receives an EOF. For the popen2 code to be compatible with stdio, you should run fdopen on each file descriptor to create FILE*s. From owner-freebsd-hackers@freebsd.org Sat Jan 16 19:03:57 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE483A859A0 for ; Sat, 16 Jan 2016 19:03:57 +0000 (UTC) (envelope-from n7w@delta.emu.st) Received: from f5.bushwire.net (f5.bushwire.net [IPv6:2607:fc50:1000:5b00::2]) by mx1.freebsd.org (Postfix) with ESMTP id D44A91AB6 for ; Sat, 16 Jan 2016 19:03:57 +0000 (UTC) (envelope-from n7w@delta.emu.st) Received: by f5.bushwire.net (Postfix, from userid 1001) id DAEC2AC909; Sat, 16 Jan 2016 11:03:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/simple; d=emu.st; s=2015; t=1452971030; bh=vuhqTHZ9e3sIIDNnMLdrYPcB11Y=; h=Comments:Received:Date:Message-ID:From:To:Subject:References: MIME-Version:Content-Type:Content-Disposition:In-Reply-To; b=YMp7IzDAdAvNEiSRDh1pUft54woS8DXSdCDxorJd99BZpTWYzdeyVcLQ8j/RUeHHf gXk9CXWvY1T+VXoyt0tWx/IYH/LM4dX0XpeOKZbA6GDEE+GDyH1fW9s0Ft8rkRfejt WKyO45/6oVw2VhsfBwfB9vyy13YXSOQepHYz4ueg=z4ueg= Comments: QMDA 0.3 Received: (qmail 4825 invoked by uid 1001); 16 Jan 2016 19:03:50 -0000 Date: 16 Jan 2016 19:03:50 +0000 Message-ID: <20160116190350.4824.qmail@f5-external.bushwire.net> From: "Mark Delany" To: freebsd-hackers@freebsd.org Subject: Re: How to send EOF to the popen(3) pipe? References: <5699BAC9.3060407@rawbw.com> <5699C8AB.7070006@missouri.edu> <569A8508.80908@rawbw.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <569A8508.80908@rawbw.com> X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 19:03:58 -0000 On 16Jan16, Yuri allegedly wrote: > No, pclose kills the process and returns the exit code. Half-closed > connection though can be alive for a while, until the other side > finishes and closes the pipe. > I think the answer to my question is "no". popen(3) can't send EOF. > Protocol needs to support EOF signal on the application-level. Right. Sounds like you need an fshutdown(FILE*, int how) that is analogous to shutdown(2). Unfortunately you can't hack it with: fflush(FILE*); shutdown(fileno(FILE*), SHUT_WR); as shutdown() only works on sockets, not pipes. I guess your best choice is to implement your own popen() with socketpair() then you can shutdown() on it. Mark. From owner-freebsd-hackers@freebsd.org Sat Jan 16 19:58:23 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5433A84EEE for ; Sat, 16 Jan 2016 19:58:23 +0000 (UTC) (envelope-from dchagin@chd.heemeyer.club) Received: from heemeyer.club (heemeyer.club [108.61.204.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "heemeyer.club", Issuer "heemeyer.club" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A19861830 for ; Sat, 16 Jan 2016 19:58:23 +0000 (UTC) (envelope-from dchagin@chd.heemeyer.club) Received: from chd.heemeyer.club (dchagin.static.corbina.ru [78.107.232.239]) by heemeyer.club (8.15.2/8.15.1) with ESMTPS id u0GJwKGx010345 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Sat, 16 Jan 2016 19:58:21 GMT (envelope-from dchagin@chd.heemeyer.club) X-Authentication-Warning: heemeyer.club: Host dchagin.static.corbina.ru [78.107.232.239] claimed to be chd.heemeyer.club Received: from chd.heemeyer.club (localhost [127.0.0.1]) by chd.heemeyer.club (8.15.2/8.15.1) with ESMTPS id u0GJwKIl041618 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 16 Jan 2016 22:58:20 +0300 (MSK) (envelope-from dchagin@chd.heemeyer.club) Received: (from dchagin@localhost) by chd.heemeyer.club (8.15.2/8.15.1/Submit) id u0GJwK38041617 for freebsd-hackers@freebsd.org; Sat, 16 Jan 2016 22:58:20 +0300 (MSK) (envelope-from dchagin) Date: Sat, 16 Jan 2016 22:58:19 +0300 From: Chagin Dmitry To: freebsd-hackers@freebsd.org Subject: irrelevant locking Message-ID: <20160116195819.GA41610@chd.heemeyer.club> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 19:58:23 -0000 hi, please, can someone explain the reason to take the process lock here: int sys_issetugid(register struct thread *td, struct issetugid_args *uap) { struct proc *p = td->td_proc; /* * Note: OpenBSD sets a P_SUGIDEXEC flag set at execve() time, * we use P_SUGID because we consider changing the owners as * "tainting" as well. * This is significant for procs that start as root and "become" * a user without an exec - programs cannot know *everything* * that libc *might* have put in their data segment. */ PROC_LOCK(p); td->td_retval[0] = (p->p_flag & P_SUGID) ? 1 : 0; PROC_UNLOCK(p); return (0); } From owner-freebsd-hackers@freebsd.org Sat Jan 16 20:26:49 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1D9E6A858D8 for ; Sat, 16 Jan 2016 20:26:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AD16E1A34; Sat, 16 Jan 2016 20:26:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u0GKQhmw095662 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 16 Jan 2016 22:26:44 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u0GKQhmw095662 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u0GKQhuv095661; Sat, 16 Jan 2016 22:26:43 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 16 Jan 2016 22:26:43 +0200 From: Konstantin Belousov To: Chagin Dmitry Cc: freebsd-hackers@freebsd.org Subject: Re: irrelevant locking Message-ID: <20160116202643.GL3942@kib.kiev.ua> References: <20160116195819.GA41610@chd.heemeyer.club> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160116195819.GA41610@chd.heemeyer.club> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 20:26:49 -0000 On Sat, Jan 16, 2016 at 10:58:19PM +0300, Chagin Dmitry wrote: > hi, please, can someone explain the reason to take the process lock here: There is no reason, I think that the PROC_LOCK() can be removed. > > int > sys_issetugid(register struct thread *td, struct issetugid_args *uap) > { > struct proc *p = td->td_proc; > > /* > * Note: OpenBSD sets a P_SUGIDEXEC flag set at execve() time, > * we use P_SUGID because we consider changing the owners as > * "tainting" as well. > * This is significant for procs that start as root and "become" > * a user without an exec - programs cannot know *everything* > * that libc *might* have put in their data segment. > */ > PROC_LOCK(p); > td->td_retval[0] = (p->p_flag & P_SUGID) ? 1 : 0; > PROC_UNLOCK(p); > return (0); > } From owner-freebsd-hackers@freebsd.org Sat Jan 16 20:44:47 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43435A85E80 for ; Sat, 16 Jan 2016 20:44:47 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D158010C8 for ; Sat, 16 Jan 2016 20:44:45 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u0GKiYwp097018 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 16 Jan 2016 21:44:35 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: yuri@rawbw.com Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u0GKiLHb069305 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 17 Jan 2016 03:44:22 +0700 (KRAT) (envelope-from eugen@grosbein.net) Subject: Re: How to send EOF to the popen(3) pipe? To: Yuri , Freebsd hackers list References: <5699BAC9.3060407@rawbw.com> From: Eugene Grosbein Message-ID: <569AABA2.8010505@grosbein.net> Date: Sun, 17 Jan 2016 03:44:18 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <5699BAC9.3060407@rawbw.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 20:44:47 -0000 16.01.2016 10:36, Yuri wrote: > Is there a way to send the EOF to popen(3) pipe? > > Imagine the situation when the child process works in a stream fashion, >processes objects one by one, and stops on EOF from stdin. > One has to be able to send EOF to get to the end of the last processed object. > Otherwise reading from the descriptor will generally block. You can use pipe2(), then vfork(), then close reading descriptor in child process, then just execv() for shell. In parent process, close writing descriptor before beginning of reading cycle. So, the subprocess can just fclose() its output and parent gets zero bytes read next time, that is EOF. Eugene Grosbein From owner-freebsd-hackers@freebsd.org Sat Jan 16 22:09:00 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 518C1A859A8 for ; Sat, 16 Jan 2016 22:09:00 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: from mail-vk0-x22c.google.com (mail-vk0-x22c.google.com [IPv6:2607:f8b0:400c:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12DEE138D; Sat, 16 Jan 2016 22:09:00 +0000 (UTC) (envelope-from vijju.singh@gmail.com) Received: by mail-vk0-x22c.google.com with SMTP id k1so315364292vkb.2; Sat, 16 Jan 2016 14:09:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=wfkq8mxjqUHoB+o6frkw+sVxAd4WtsJgeO5Ky/VKURg=; b=Ho9haDj0qFFz+wJNXkEZSFL4awuMT/NXWpSkB1ZcALVEAwi6V8nIuOTXLsNmIIEmJw Un0bPccoRO5S3UWHXYMyeZXe2x9Nd0AmECz/YQAGyv+c99cR491RZzZFOOjFL8ti3+ic YCE7UtL/AoKFQWCwJwOsVJnNsfV6RR2DYc4Cnsb1xK22aleoGZ9JQfr1FF8MHMJ8q86v xpBOvR796HJ+x4CgazCuSPuJgqjAFUrtxfYVg06nUJsDzBuyXS+Jgx3SnmY9673S01k3 8gs+ovw9Fch1517OUqEEAlkGZPApJBEe26YHK4NAxyb52hjDdgkjJ1bVmaBleM1YC2I+ hmAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=wfkq8mxjqUHoB+o6frkw+sVxAd4WtsJgeO5Ky/VKURg=; b=dHNermYV8FjdwP4B45v9Tjy5ktd0q41LxEGS/DKyWFpmCAPFRofgpPJrRTRFcnNUJW p8tHXw90HPiIFr9HVPVSpJf/w8mO7/+f91jNfSY4YU2K/MiTgR/geCZnq3Ia8dGRG3pm eAW3G3ubUf28lj8OPk+jg0A+KFbWBms/psPEB5gdnlNFEkUtbpUgVg/u8FNT1UvCMhkh 1XV7FNeblC/doav12HeRSYlCOLTdU1flqPRwEbpV54qgjrCbsrb3MX85YJZZgQNd174X PZcntgvmpTQD9xrUaTK0kV98oFSc+M7tpLfFJb7hcvoH1+btvPUbSp8GRfZU1EqbW9u3 oxKQ== X-Gm-Message-State: ALoCoQlCiS0NDj0JUbWwU/44wftVaF1F0fVRebyukWxshTKVl3j4I7InAf8kUCxoLi534jgdfO8VV5tNFWaZ/Yb/847mX6tnaA== MIME-Version: 1.0 X-Received: by 10.31.131.203 with SMTP id f194mr12837889vkd.90.1452982138735; Sat, 16 Jan 2016 14:08:58 -0800 (PST) Received: by 10.31.66.136 with HTTP; Sat, 16 Jan 2016 14:08:58 -0800 (PST) Received: by 10.31.66.136 with HTTP; Sat, 16 Jan 2016 14:08:58 -0800 (PST) In-Reply-To: <20160116202643.GL3942@kib.kiev.ua> References: <20160116195819.GA41610@chd.heemeyer.club> <20160116202643.GL3942@kib.kiev.ua> Date: Sat, 16 Jan 2016 14:08:58 -0800 Message-ID: Subject: Re: irrelevant locking From: Vijay Singh To: Konstantin Belousov Cc: Chagin Dmitry , freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 22:09:00 -0000 Couldn't the get & set race otherwise? On Jan 16, 2016 12:27 PM, "Konstantin Belousov" wrote: > On Sat, Jan 16, 2016 at 10:58:19PM +0300, Chagin Dmitry wrote: > > hi, please, can someone explain the reason to take the process lock here: > There is no reason, I think that the PROC_LOCK() can be removed. > > > > > int > > sys_issetugid(register struct thread *td, struct issetugid_args *uap) > > { > > struct proc *p = td->td_proc; > > > > /* > > * Note: OpenBSD sets a P_SUGIDEXEC flag set at execve() time, > > * we use P_SUGID because we consider changing the owners as > > * "tainting" as well. > > * This is significant for procs that start as root and "become" > > * a user without an exec - programs cannot know *everything* > > * that libc *might* have put in their data segment. > > */ > > PROC_LOCK(p); > > td->td_retval[0] = (p->p_flag & P_SUGID) ? 1 : 0; > > PROC_UNLOCK(p); > > return (0); > > } > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@freebsd.org Sat Jan 16 22:43:18 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8C47AA848C4 for ; Sat, 16 Jan 2016 22:43:18 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1D5D218F8; Sat, 16 Jan 2016 22:43:18 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-wm0-x22a.google.com with SMTP id b14so72943901wmb.1; Sat, 16 Jan 2016 14:43:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=szM6ppiBrti+zk2gWjRhH/UEfArv5uqnw4Kl5nXH9AE=; b=mOy7xCOAOlfkDgND7TQBfqf5ABe3wEUy27IK17Qn/gTVCFWVQLEIhHR7+ppeouhC8i FV/5MrvLSfaxyz1vlLMs1n8AxyGFLYeJ2ftGjq/Mxh6OmnmAtZIaP6VmSjcmYQvc12al BRTfFjjLTCP5PfUB3ajpthY0vskOZiTMK9B8NdKvbUo8CL6vbCAtD4tmOiOYegPpMY6R eC11LEjARcE6jnTLEm5t7qXcsiIRSDN4JTxiuDJCUExNrspQgdF0HpVCYf1jTdf5prRW Y19lCXkPqRUS6isEMzO7+VNmRoNluFh7SeuQslqQBLLk/dx0NOHy3pymoM9txV1SMh9O CHyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=szM6ppiBrti+zk2gWjRhH/UEfArv5uqnw4Kl5nXH9AE=; b=TO7B20deQagfcjTOpyw2YgStgw97Jgt0tbMhWwjkh5gDJrjAo+x6lhafsNmaXvtD6W oCf3b5mgKZsrugQcwRk+/7WPCMt7pX7aCvAtrEQnb+kGwI5eF8pC8bIX9XutUC6l/JMC EkWZLnxHCEOJ8yLW/R9NDElsrrvpmia8zvaPiBQVRq2E1kYojtLTJwbV3rqaX1vfwReC 0K0Y15Ybu53GjqkNqs7KPan8WyL1OPEyifU/PVqd2oofVI/mtx/ZmQytvEXlr1humIZG YD3BW9nD9tSBY2YULUrsOHvzqWVuKtt98knGFQ8z04NQzUXxmvz34ikzUnOGKsGjQL4U 8nEg== X-Gm-Message-State: AG10YOQWj2tIh96tI30vxaDzdrQdZfqHlF96OsynHoHPks4LNNOudoVYZemmKM8Ja4I8rQ== X-Received: by 10.28.127.85 with SMTP id a82mr5896826wmd.48.1452984195574; Sat, 16 Jan 2016 14:43:15 -0800 (PST) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by smtp.gmail.com with ESMTPSA id 75sm8663307wmo.22.2016.01.16.14.43.14 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Sat, 16 Jan 2016 14:43:15 -0800 (PST) Date: Sat, 16 Jan 2016 23:43:13 +0100 From: Mateusz Guzik To: Vijay Singh Cc: Konstantin Belousov , freebsd-hackers@freebsd.org, Chagin Dmitry Subject: Re: irrelevant locking Message-ID: <20160116224312.GA1963@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Vijay Singh , Konstantin Belousov , freebsd-hackers@freebsd.org, Chagin Dmitry References: <20160116195819.GA41610@chd.heemeyer.club> <20160116202643.GL3942@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2016 22:43:18 -0000 On Sat, Jan 16, 2016 at 02:08:58PM -0800, Vijay Singh wrote: > Couldn't the get & set race otherwise? Current locking plays no role in correctness here. Say that locking is left in place. A concurrent setgroups (or whatever) call resulting in setting P_SUGID is also being executed. Regardless of whether PROC_LOCK/PROC_UNLOCK pair is in place, it can set the bit before or after it is being tested by sys_issetugid. In principle, the very moment you drop a lock, your informatoin is stale. This does not matter here. It's only the process itself which can set the bit, so it would have to race with itself. Finally, the bit can be only unset during execve, which cannot be executed here - if it is being executed, there is only one thread doing work and, well, it is doing execve. The real question is if it would make sense to add the bit to elf aux vector to save the call as done by the loader. > On Jan 16, 2016 12:27 PM, "Konstantin Belousov" wrote: > > > On Sat, Jan 16, 2016 at 10:58:19PM +0300, Chagin Dmitry wrote: > > > hi, please, can someone explain the reason to take the process lock here: > > There is no reason, I think that the PROC_LOCK() can be removed. > > > > > > > > int > > > sys_issetugid(register struct thread *td, struct issetugid_args *uap) > > > { > > > struct proc *p = td->td_proc; > > > > > > /* > > > * Note: OpenBSD sets a P_SUGIDEXEC flag set at execve() time, > > > * we use P_SUGID because we consider changing the owners as > > > * "tainting" as well. > > > * This is significant for procs that start as root and "become" > > > * a user without an exec - programs cannot know *everything* > > > * that libc *might* have put in their data segment. > > > */ > > > PROC_LOCK(p); > > > td->td_retval[0] = (p->p_flag & P_SUGID) ? 1 : 0; > > > PROC_UNLOCK(p); > > > return (0); > > > } > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -- Mateusz Guzik