From owner-freebsd-arch@freebsd.org Sun Nov 29 18:59:45 2015 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 336EFA22F4F for ; Sun, 29 Nov 2015 18:59:45 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (kientzle.com [142.254.26.11]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F15E41CD1; Sun, 29 Nov 2015 18:59:44 +0000 (UTC) (envelope-from tim@kientzle.com) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id tATJ0dUt078799; Sun, 29 Nov 2015 19:00:39 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.108] (192.168.1.101 [192.168.1.101]) by kientzle.com with SMTP id ad4h8n64z9wwwb5fgxrj5zs5vn; Sun, 29 Nov 2015 19:00:39 +0000 (UTC) (envelope-from tim@kientzle.com) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Subject: Re: mtree "language" enhancements From: Tim Kientzle In-Reply-To: Date: Sun, 29 Nov 2015 10:59:36 -0800 Cc: "freebsd-arch@freebsd.org" , Michal Ratajsky , Brooks Davis Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Warner Losh X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Nov 2015 18:59:45 -0000 Sounds interesting. Have you talked with Michal (CCed) who is working on a libmtree library? The capabilities you're describing here really need to be bundled into a = library, I think. In particular, the ability to "unlink", "copy", etc, = is much more useful if you can directly query the mtree file contents to = perform conditional changes. (For example, it may be important to = remove an empty directory which requires you to be able to query whether = a directory has files in it.) I would also be interested in a description of the processing model. It = sounds like you're assuming the same model used by the current mtree = program -- mtree files are processed sequentially line-by-line as they = are read. For instance, libarchive's mtree processor works differently; it reads = the entire input, merging redundant lines for the same file, and then = processes the list. This is more explicitly declarative, and simplifies = things like modifying the ownership or permissions of already-listed = files. > Each action entry would have an 'action' keyword. In terms of the language per se, this seems unnecessary. I've = proposed alternate language below that omits the unnecessary = "type=3Daction" by just adding new keywords. > The keywords I've defined > so far are as follows: > 1. "unlink" which throws away the previous entry. That entry has been > removed. It may apply to files or directories, but it is an error not = to > remove all entries in a directory when removing the directory. # When set on an entry, a matching file on disk will be removed. # This would also be useful for things like ObsoleteFiles unlink=3Dtrue > 2. "move" which relocates a previous entry. An additional targetpath > keyword specifies the ultimate destination for this entry. # When set on an entry, moves the existing file to the new name rename=3D # Example foo/bar type=3Dfile owner=3Droot mode=3D0755 rename=3Dfoo/baz > 3. "copy" which duplicates a previous entry. It too takes target path. # As with rename, except it copies the contents. copy_from=3D # properties that are not specified will be copied as well # Create foo/bar by copying foo/baz, preserving all attributes foo/bar type=3Dfile copy_from=3Dfoo/baz # Create foo/bar as above, but modify the owner foo/bar owner=3Ddialer type=3Dfile copy_from=3Dfoo/baz > 4. "meta" which changes the meta data of the previous entry. All = keywords > on this are merged with the previous entry. As above, libarchive's mtree processor already does this by default; no = language change is needed. > The one other thing that my merging tool does is to remove all size > keywords. ... [comments about modifying existing files] One common case here is appending new contents to an existing file. = That could similarly be handled with the same pattern: # Append from source foo/bar append_from=3D In particular, that removes the need to find the source file to modify = it in-place. I've run into various headaches with Crochet when the = /usr/obj layout changes between releases and Crochet cannot find the new = location of a file. This would remove the need to always modify the = file in-place. (But not all.) Cheers, Tim > On Nov 29, 2015, at 10:04 AM, Warner Losh wrote: >=20 > Greetings, >=20 > As part of making NanoBSD buildable by non-root, I've found a need to = have > a richer mtree language than we currently have. >=20 > mtree started out as a language to express hierarchies of files. It = does a > decent job at that, even if some of the tools that we have in the tree > aren't so great about manipulating them. One could easily wish for = better > tools, but that's not the topic of this thread. >=20 > So, I've started to move the language into one that can also journal > changes to a tree, and have been moving NanoBSD to using wrappers that = do > the changes to the tree and record the journal events at the end of = the > metalog produced from buildworld. I have a second tool that reads the = meta > log, and applies the actions to the earlier entries and then produces = a > final metalog that's used for makefs. These tools are still evolving, = but > before I got too close to the point of committing, I thought I'd post = a > proposed extension to mtree for comments so I don't have to change too = much. >=20 > I'd like a new type called 'action' (so type=3Daction in the records). = This > type is defined loosely to manipulate and earlier entry (or maybe = entries, > still unsure) in the file. >=20 > Each action entry would have an 'action' keyword. The keywords I've = defined > so far are as follows: > 1. "unlink" which throws away the previous entry. That entry has been > removed. It may apply to files or directories, but it is an error not = to > remove all entries in a directory when removing the directory. > 2. "move" which relocates a previous entry. An additional targetpath > keyword specifies the ultimate destination for this entry. > 3. "copy" which duplicates a previous entry. It too takes targetpath. > 4. "meta" which changes the meta data of the previous entry. All = keywords > on this are merged with the previous entry. >=20 > The one other thing that my merging tool does is to remove all size > keywords. In the NanoBSD environment, size is irrelevant. Files are > replaced and appended to all the time in the build process, and it = doesn't > make sense to track the size. makefs fails if the size is different, = so > post-processing of the tree, say to add a new default to > /etc/defaults/rc.conf or to tweak /etc/ttys to turn on/off a tty (or = append > a new entry) will cause it to fail. I would be nice of mtree could do = this, > but is simply can't (but see above for whining about better tools = being > beyond the scope of this). >=20 > If things go well, we could eventually move these extensions into = mtree so > that the post-processing stage is no longer necessary. I'm content to > maintain the hundred or two lines of awk I've written to implement it. = I > chose awk because it does the job well enough, though python might do = it > better. But I don't want to talk about that choice since right now it = is > purely internal to NanoBSD (though I hope that other build = orchestration > systems like src/release and crochet look to adopt). >=20 > Comments? >=20 > Warner > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to = "freebsd-arch-unsubscribe@freebsd.org" >=20