From owner-freebsd-arch@freebsd.org Sun Nov 29 18:58:50 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51C58A22EF1 for ; Sun, 29 Nov 2015 18:58:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qg0-x22c.google.com (mail-qg0-x22c.google.com [IPv6:2607:f8b0:400d:c04::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0271C1C1F for ; Sun, 29 Nov 2015 18:58:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by qgeb1 with SMTP id b1so104753885qge.1 for ; Sun, 29 Nov 2015 10:58:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=vAc4IdjrxBOEGT48D7sMUiLSwoq1q0xU/woPUOc2wTQ=; b=K98xmwhrJe3Pb7C3HHAk/YCq94XSbwWvQw7TqawQ9CjWS4FmAoXiPRFYPhrfpEz0a3 JbEMN8PkyT/T3dIcYPnI29z6xT/cub3yg+uG3BzoCz4SA5x95wKiCdyBrcVppkhGC0r2 a5bTOPRmCqTQhl1ho0eDnzvMidCFviAVwDtgT9i+uU3SeEq4Bd6j4jr2mAT9KLVz2fDp CKdVMVAj5FQbkpHg/56IPAcGI21EQSjPZOGOv/JqFNCvq9DvfLcrK1ybGNjdaxApfRdL r6QfPll5SczkYMGtn6gKPOQB0a8mE0+ZV6Igx4dMU3KIw2DYJWBEoJKdbpJ1DZNq4Kom +eaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vAc4IdjrxBOEGT48D7sMUiLSwoq1q0xU/woPUOc2wTQ=; b=Dwy2k7iacJkVrOGEB2fJTSjsbWfig0uYHMuIkdpcsSnnbYWI3i5R0e/TeDrDk57HlY B5GpwMZxX16qR+2C7nwaXIRB3eBLWAR0jecU4/XxKe9Z48y72nYH98sgS03b4ktXcTd5 JJZsLjChaqW17tihCPPX31ZGJoVhub0lPPJS3d41s1ZpaJPnsIkAgT1vVdZ/x+a3R8fr sKGQO4k5n8BjDkRkoRZcU9T0sHiJtFmSt8pezXVT+7LaQzGdWvADYBx4rJq6nEQuaiiC ok+0X1dIIkkhvwrAUleUwR7FIM6XMHnhXrq6sOnVr0ZVkf0Q1qbIlrrQedKt2EbNh6vd S3wQ== X-Gm-Message-State: ALoCoQmrxzVz6DAhJW8poj5Zw6sjSYesvoyhGmyx1oC9dy5L5KI+z6S55qC9dWIIyEXmgINSETYb MIME-Version: 1.0 X-Received: by 10.140.250.70 with SMTP id v67mr72858582qhc.43.1448823528479; Sun, 29 Nov 2015 10:58:48 -0800 (PST) Sender: wlosh@bsdimp.com Received: by 10.140.27.181 with HTTP; Sun, 29 Nov 2015 10:58:48 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: <44936.1448820967@critter.freebsd.dk> References: <44936.1448820967@critter.freebsd.dk> Date: Sun, 29 Nov 2015 11:58:48 -0700 X-Google-Sender-Auth: -kl1I4-yMiPzw2MsU5BGsWBxY5I Message-ID: Subject: Re: mtree "language" enhancements From: Warner Losh To: Poul-Henning Kamp Cc: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Nov 2015 18:58:50 -0000 On Sun, Nov 29, 2015 at 11:16 AM, Poul-Henning Kamp wrote: > -------- > In message < > CANCZdfrDtfkwKxMV3o9tcQNzBQDKZdTx1JErkTKtC7UZORT5aA@mail.gmail.com> > , Warner Losh writes: > > >As part of making NanoBSD buildable by non-root, I've found a need to have > >a richer mtree language than we currently have. > > >I'd like a new type called 'action' (so type=action in the records). This > >type is defined loosely to manipulate and earlier entry (or maybe entries, > >still unsure) in the file. > > I suggest you define this so that all records have an action, and that > the default action is "create" >From a practical point of view, I didn't consider this, but that is what would be a logical consequence of these extensions. > > >2. "move" which relocates a previous entry. An additional targetpath > >keyword specifies the ultimate destination for this entry. > >3. "copy" which duplicates a previous entry. It too takes targetpath. > > Is targetpath absolute or relative ? > relative to top of tree. > Can it reach out of the mtree root ? Nope. Those cases need entirely new entries. > > >4. "meta" which changes the meta data of the previous entry. All keywords > >on this are merged with the previous entry. > > System-III called this "chmog" if I recall correctly :-) I love that term. I'll steal it :) > > >The one other thing that my merging tool does is to remove all size > >keywords. > > That sounds wrong to me. Shouldn't you just emit "meta" records updating > the size as appropriate ? > Emitting records that change the size is possible, but would add an extra step. It's easy to catch mv, rm, etc, but hard to catch >>. I took the easy way out of just ignoring size changes, though one could add a nano_resize command that you need to call after changing the size of a file in the post-processing phase. > What about digest fields ? > In my use case, they are irrelevant. They aren't generated by buildworld's metalog, and aren't generally useful. They might add some protection against tampering between when the tree is created and when it is put into a partition, but that's racy. For an attacker, if they can replace the file after it is created but before the checksum is run, they win. So there's little value here for me. However, having said that, digest fields either should be discarded (for the same reason as size), or they should be correct before the dedup tool / enhanced mtree gets to them. This gets into the nuts and bolts of NanoBSD: we copy files around all the time, but have no spec for them. The usual answer is to have a bunch of chmod / chown calls that 'fix' them up and generate a mtree for the image so you can protect against corruption in the field (or at least know what changed). In a nopriv-build, you need to somehow record these changes. Do I continue the traditional behavior, or do I require a new mtree spec for all the files you wish top copy and use that to modify the metalog, or hack the permissions directly for the priv-build case. The decision between discard and check likely is an input to the dedup tool. For NanoBSD the decision is likely to default to discard. But other tools might want to check, and some NanoBSD users may wish to climb the hill to being correct by adding calls to correct the size everywhere. My first goal is to create a tool that produces correct images with the right permissions. A secondary goal would be to safe-guard the process from unintended changes that would be caught by size and/or digest changes. It isn't a current feature of NanoBSD, but that doesn't make it undesirable. Especially if your NanoBSD build process puts precious files onto the media that you want to make sure the rest of the build process doesn't tamper with accidentally to guard against bugs... Warner