From owner-freebsd-questions@freebsd.org Sun Jan 21 21:16:04 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85749EBBC40 for ; Sun, 21 Jan 2018 21:16:04 +0000 (UTC) (envelope-from rfg@tristatelogic.com) Received: from outgoing.tristatelogic.com (segfault.tristatelogic.com [69.62.255.118]) by mx1.freebsd.org (Postfix) with ESMTP id 6AB457CC0D for ; Sun, 21 Jan 2018 21:16:03 +0000 (UTC) (envelope-from rfg@tristatelogic.com) Received: from segfault-nmh-helo.tristatelogic.com (localhost [127.0.0.1]) by segfault.tristatelogic.com (Postfix) with ESMTP id C99B53AEF2; Sun, 21 Jan 2018 13:15:57 -0800 (PST) From: "Ronald F. Guilmette" To: Paul Mather cc: freebsd-questions@freebsd.org Subject: Re: Splitting up sets of files for archiving (e.g. to tape, optical media) In-Reply-To: <2456E55A-D14F-41ED-B8DD-9633BD73ACF1@gromit.dlib.vt.edu> Date: Sun, 21 Jan 2018 13:15:57 -0800 Message-ID: <81742.1516569357@segfault.tristatelogic.com> X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 21:16:04 -0000 In message <2456E55A-D14F-41ED-B8DD-9633BD73ACF1@gromit.dlib.vt.edu>, Paul Mather wrote: >Have you looked at fpart (https://github.com/martymac/fpart)? It looks >to me like it is applicable to your problem (which sounds to me like a >variant of the bin packing problem). The fpart README even lists >packing music files onto fixed-size DVD media as one of its examples, >which sounds close to the archiving scenario you give above. Plus, it >claims to be developed on FreeBSD. I had never heard of that tool (fpart) till now, but I have just now skimmed the README file for that and yes, this tool appears to be quite close to what I had in mind. I will have to see if I can make contact with the developer of this tool, and discuss with him what I have been doing, whch is similar but different. (Maybe he will be motivated to include some suggestions from me.) The problem, as I personally have approached it, is indeed a bin packing problem, but it is not clear (from the README) that this fpart tool even attempts to do bin packing, i.e. in order to minimize the number of output "partitions". (That's an important feature if you are trying to minimize the number of archive volumes.) Also, whereas this (fpart) tool is clearly oriented towards creating output "partitions" which will then be the basis for subsequent rsync operations, in my case each group of files comprising one single output blob gets hard-linked into a newly created directory representing that specific output blob (where these new directoties have rather unimaginative names like "00", "01", "02"...) Since these directores are on the same partition as the input files to be archived, and since that whole partition is exported to my local network (courtesy of Samba), I can then just pop over to my Windoze system and use ImgBurn to write each output blob in turn to a blank BD-R disk. (I am actually doing this as we speak.) Where things really get tricky is where one tries to really and truly maximize the usage, down to the very last sector, of the output (archive) volumes. To do this perfectly, one would have to have deep knowledge of -exactly- how ImgBurn formats a set of input files and turns them into either a UDF or an ISO 9660 image. I don't have such knowlegde, at present, so I am forced to add small fudge factors in my calculations so that each of my output blobs will in fact fit onto a single blank BD-R, at least as far as ImgBurn is concerned. Even so, and even using a simple-minded bin packing algorithm, I am generally able to fill up output volumes to 99.9% or better. Anyway, thank you Paul, for pointing me at fpart. I quite definitely knew nothing about it till now... and that's why I posted my queation here in the first place, because there's such a wealth and breadth of knowledge here. Regards, rfg