From owner-freebsd-questions@freebsd.org Fri Jan 19 14:44:16 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13526EB5F99 for ; Fri, 19 Jan 2018 14:44:16 +0000 (UTC) (envelope-from freebsd-lists@gromit.dlib.vt.edu) Received: from gromit.dlib.vt.edu (gromit.dlib.vt.edu [128.173.126.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gromit.dlib.vt.edu", Issuer "Chumby Certificate Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C70598470A for ; Fri, 19 Jan 2018 14:44:15 +0000 (UTC) (envelope-from freebsd-lists@gromit.dlib.vt.edu) Received: from mather.chumby.lan (c-98-244-101-97.hsd1.va.comcast.net [98.244.101.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gromit.dlib.vt.edu (Postfix) with ESMTPSA id E4A46C5D; Fri, 19 Jan 2018 09:37:56 -0500 (EST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: Splitting up sets of files for archiving (e.g. to tape, optical media) From: Paul Mather In-Reply-To: Date: Fri, 19 Jan 2018 09:37:55 -0500 Cc: rfg@tristatelogic.com Content-Transfer-Encoding: quoted-printable Message-Id: <2456E55A-D14F-41ED-B8DD-9633BD73ACF1@gromit.dlib.vt.edu> References: To: freebsd-questions@freebsd.org X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Jan 2018 14:44:16 -0000 Ronald, > On Jan 19, 2018, at 2:59 AM, freebsd-questions-request@freebsd.org = wrote: >=20 > Date: Thu, 18 Jan 2018 18:06:58 -0800 > From: "Ronald F. Guilmette" > To: freebsd-questions@freebsd.org > Subject: Splitting up sets of files for archiving (e.g. to tape, > optical media) > Message-ID: <68377.1516327618@segfault.tristatelogic.com> >=20 >=20 > This isn't really FreeBSD specific, but in my experience the folks on > this list have a lot of knowledge about a lot nice, useful free = software > tools, so I hope nobody will begrudgd me for asking this question = here. >=20 > I'm looking for a pre-existing software tool, which may or may not = already > exist, and which will do the following job... >=20 > Problem statement: >=20 > Imagine that you have a big set of files that you would like to = archive > to some sort of archiving media, such as tapes, or optical media, = where > each unit of said archiving media has a capacity considerably less = than > the total aggregate size of all of the files you want to archive. >=20 > Imagine further that you would like your set of input files to be = spread > across the units of the output (archive) media such that no single = input > file is ever split across more than one unit of the output media, in = order > to simplify recovery/restore of individual files. >=20 > Lastly, assume that it is desired to minimize, as much as reasonably > possible, the total number of output (archive) media units used to > archive the entire set of input files. (And to further this goal, > it is acceptable for files from any single input subdirectory to be > scattered among the various output media units. >=20 > +_+_+_+_+_+_+_+_+_+_ >=20 > In my case, I want to archive several hundred gigabytes onto a set of > blank BD-R disks. >=20 > I plan to use ImgBurn to actually write the BD-R disks. >=20 > So basically, I just need a tool to analyze the input file set, = applying > some sort of bin packing algorithm, and then spit out a list of which > specific files should go into each specific archive volume, e.g. #01, = #02, > #03... etc. Each such set of files will then, in turn, be hard-linked > into a temporary directory, and then, one by one, ImghBurn will be = told > to write each of these temp directories to a single output BD-R disk. >=20 > I have written a small software tool to do the above "splitting" job, > and I am currently improving upon it, but it occured to me that I > should at least ask if someone else has perhaps already perfected this > exact wheel that I am busy re-inventing. >=20 >=20 > Regards, > rfg >=20 >=20 > P.S. It seems unlikely that I'm the first and only person to have = ever > written a tool to do this specific job, but on the off chance that I = am, > I am more than willing to contribute my little tool to the = ever-expanding > ports tree. Have you looked at fpart (https://github.com/martymac/fpart)? It looks = to me like it is applicable to your problem (which sounds to me like a = variant of the bin packing problem). The fpart README even lists = packing music files onto fixed-size DVD media as one of its examples, = which sounds close to the archiving scenario you give above. Plus, it = claims to be developed on FreeBSD. Cheers, Paul. Disclaimer: I have not used fpart myself.