Date: Fri, 15 Mar 2002 22:27:53 -0300 From: "Daniel C. Sobral" <dcs@newsguy.com> To: Maxim Sobolev <sobomax@FreeBSD.ORG> Cc: Michael Smith <msmith@FreeBSD.ORG>, hackers@FreeBSD.ORG, audit@FreeBSD.ORG, re@FreeBSD.ORG Subject: Re: Extending loader(8) for loading kerels/modules split across several disks Message-ID: <3C929F99.18C9EC86@newsguy.com> References: <200203070940.g279eTa04750@mass.dis.org> <3C92478F.2B7C11DB@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
For whatever it is worth, I like it.
Maxim Sobolev wrote:
>
> Michael Smith wrote:
> >
> > > > > Please review attached patch, which adds long overdue feature to our
> > > > > loader(8), allowing it to load sequence of files as a single object.
> > > >
> > > > I don't like this. I would much rather see support for 'split' files
> > > > implemented as a stacking filesystem layer like the gzip support, with
> > > > the simple recognition of 'foo.gz.aa' as the first part of a split
> > > > version of 'foo.gz', which in turn is recognised as a compressed version
> > > > of 'foo'.
> > >
> > > I am curious how in this case the layer is going to know how many
> > > parts the file contains?
> >
> > The simple way to do it is to keep asking for more parts until there are
> > no more.
> >
> > You can take the NetBSD approach of wrapping the file in a multipart tar
> > archive.
> >
> > Or a more elegant method involves the use of a control file.
> >
> > eg. the splitfs code, when asked to open "foo" looks for "foo.split"
> > which is a text file containing a list of filenames and media names, eg.
> >
> > foo.aa "Kernel floppy 1"
> > foo.ab "Kernel floppy 2"
> > foo.ac "Kernel and modules floppy"
> >
> > For each file segment, the process is:
> >
> > - try to open the file
> > - prompt "please insert the disk labelled <whatever>'"
> > - try to open the file
> > - return error
> >
> > At any rate, my key point is that the splitting should be invisible, and
> > *definitely* not pushed up into the loader.
>
> Ok, attached is the path, which does exactly what described. Please
> review and if there are no objections I would like to commit it
> shortly, so that our re@ team would be able to consider it for the
> forthcoming 5.0-DP1 release.
>
> Thanks!
>
> -Maxim
>
> ------------------------------------------------------------------------
> Index: src/lib/libstand/Makefile
> ===================================================================
> RCS file: /home/ncvs/src/lib/libstand/Makefile,v
> retrieving revision 1.27
> diff -d -u -r1.27 Makefile
> --- src/lib/libstand/Makefile 27 Feb 2002 17:15:37 -0000 1.27
> +++ src/lib/libstand/Makefile 15 Mar 2002 08:40:31 -0000
> @@ -153,6 +153,7 @@
> SRCS+= ufs.c nfs.c cd9660.c tftp.c zipfs.c bzipfs.c
> SRCS+= netif.c nfs.c
> SRCS+= dosfs.c ext2fs.c
> +SRCS+= splitfs.c
>
> beforeinstall:
> ${INSTALL} -C -o ${BINOWN} -g ${BINGRP} -m 444 ${.CURDIR}/stand.h \
> Index: src/lib/libstand/bzipfs.c
> ===================================================================
> RCS file: /home/ncvs/src/lib/libstand/bzipfs.c,v
> retrieving revision 1.3
> diff -d -u -r1.3 bzipfs.c
> --- src/lib/libstand/bzipfs.c 1 Feb 2002 16:33:40 -0000 1.3
> +++ src/lib/libstand/bzipfs.c 15 Mar 2002 08:40:31 -0000
> @@ -150,7 +150,7 @@
>
> /* If the name already ends in .gz or .bz2, ignore it */
> if ((cp = strrchr(fname, '.')) && (!strcmp(cp, ".gz")
> - || !strcmp(cp, ".bz2")))
> + || !strcmp(cp, ".bz2") || !strcmp(cp, ".split")))
> return(ENOENT);
>
> /* Construct new name */
> Index: src/lib/libstand/splitfs.c
> ===================================================================
> RCS file: src/lib/libstand/splitfs.c
> diff -N src/lib/libstand/splitfs.c
> --- /dev/null 1 Jan 1970 00:00:00 -0000
> +++ src/lib/libstand/splitfs.c 15 Mar 2002 08:40:31 -0000
> @@ -0,0 +1,287 @@
> +/*
> + * Copyright (c) 2002 Maxim Sobolev
> + * All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <sys/cdefs.h>
> +__FBSDID("$FreeBSD$");
> +
> +#include "stand.h"
> +
> +#define NTRIES (3)
> +#define CONF_BUF (512)
> +#define SEEK_BUF (512)
> +
> +struct split_file
> +{
> + char **filesv; /* Filenames */
> + char **descsv; /* Descriptions */
> + int filesc; /* Number of parts */
> + int curfile; /* Current file number */
> + int curfd; /* Current file descriptor */
> + off_t tot_pos; /* Offset from the beginning of the sequence */
> + off_t file_pos; /* Offset from the beginning of the slice */
> +};
> +
> +static int splitfs_open(const char *path, struct open_file *f);
> +static int splitfs_close(struct open_file *f);
> +static int splitfs_read(struct open_file *f, void *buf, size_t size, size_t *resid);
> +static off_t splitfs_seek(struct open_file *f, off_t offset, int where);
> +static int splitfs_stat(struct open_file *f, struct stat *sb);
> +
> +struct fs_ops splitfs_fsops = {
> + "split",
> + splitfs_open,
> + splitfs_close,
> + splitfs_read,
> + null_write,
> + splitfs_seek,
> + splitfs_stat,
> + null_readdir
> +};
> +
> +static void
> +split_file_destroy(struct split_file *sf)
> +{
> + int i;
> +
> + if (sf->filesc > 0) {
> + for (i = 0; i < sf->filesc; i++) {
> + free(sf->filesv[i]);
> + free(sf->descsv[i]);
> + }
> + free(sf->filesv);
> + free(sf->descsv);
> + }
> + free(sf);
> +}
> +
> +static int
> +splitfs_open(const char *fname, struct open_file *f)
> +{
> + char *buf, *confname, *cp;
> + int conffd;
> + struct split_file *sf;
> + struct stat sb;
> +
> + printf("%s\n", fname);
> + /* Have to be in "just read it" mode */
> + if (f->f_flags != F_READ)
> + return(EPERM);
> +
> + /* If the name already ends in `.split', ignore it */
> + if ((cp = strrchr(fname, '.')) && (!strcmp(cp, ".split")))
> + return(ENOENT);
> +
> + /* Construct new name */
> + confname = malloc(strlen(fname) + 7);
> + sprintf(confname, "%s.split", fname);
> +
> + /* Try to open the configuration file */
> + conffd = open(confname, O_RDONLY);
> + free(confname);
> + if (conffd == -1)
> + return(ENOENT);
> +
> + if (fstat(conffd, &sb) < 0) {
> + printf("splitfs_open: stat failed\n");
> + close(conffd);
> + return(ENOENT);
> + }
> + if (!S_ISREG(sb.st_mode)) {
> + printf("splitfs_open: not a file\n");
> + close(conffd);
> + return(EISDIR); /* best guess */
> + }
> +
> + /* Allocate a split_file structure, populate it from the config file */
> + sf = malloc(sizeof(struct split_file));
> + bzero(sf, sizeof(struct split_file));
> + buf = malloc(CONF_BUF);
> + while (fgetstr(buf, CONF_BUF, conffd) > 0) {
> + cp = buf;
> + while ((*cp != '\0') && (isspace(*cp) == 0))
> + cp++;
> + if (*cp != '\0') {
> + *cp = '\0';
> + cp++;
> + }
> + while ((*cp != '\0') && (isspace(*cp) != 0))
> + cp++;
> + if (*cp == '\0')
> + cp = buf;
> + sf->filesc++;
> + sf->filesv = realloc(sf->filesv, sizeof(*(sf->filesv)) * sf->filesc);
> + sf->descsv = realloc(sf->descsv, sizeof(*(sf->descsv)) * sf->filesc);
> + sf->filesv[sf->filesc - 1] = strdup(buf);
> + sf->descsv[sf->filesc - 1] = strdup(cp);
> + }
> + free(buf);
> + close(conffd);
> +
> + if ((sf->filesc == 0) || ((sf->curfd = open(sf->filesv[0], O_RDONLY)) == -1)) {
> + split_file_destroy(sf);
> + return(ENOENT);
> + }
> +
> + /* Looks OK, we'll take it */
> + f->f_fsdata = sf;
> + return (0);
> +}
> +
> +static int
> +splitfs_close(struct open_file *f)
> +{
> + int fd;
> + struct split_file *sf;
> +
> + sf = (struct split_file *)f->f_fsdata;
> + fd = sf->curfd;
> + split_file_destroy(sf);
> + return(close(fd));
> +}
> +
> +static int
> +splitfs_read(struct open_file *f, void *buf, size_t size, size_t *resid)
> +{
> + int i, nread, totread;
> + struct split_file *sf;
> +
> + sf = (struct split_file *)f->f_fsdata;
> + totread = 0;
> + do {
> + nread = read(sf->curfd, buf, size - totread);
> +
> + /* Error? */
> + if (nread == -1)
> + return (errno);
> +
> + sf->tot_pos += nread;
> + sf->file_pos += nread;
> + totread += nread;
> + buf += nread;
> +
> + if (totread < size) { /* EOF */
> + if (sf->curfile == (sf->filesc - 1)) /* Last slice */
> + break;
> +
> + /* Close previous slice */
> + if (close(sf->curfd) != 0)
> + return (errno);
> +
> + sf->curfile++;
> + for (i = 0;; i++) {
> + sf->curfd = open(sf->filesv[sf->curfile], O_RDONLY);
> + if (sf->curfd >= 0)
> + break;
> + if ((sf->curfd == -1) && (errno != ENOENT))
> + return (errno);
> + if (i == NTRIES)
> + return (EIO);
> + printf("\nInsert disk labelled %s and press any key...", sf->descsv[sf->curfile]);
> + getchar();putchar('\n');
> + }
> + sf->file_pos = 0;
> + }
> + } while (totread < size);
> +
> + if (resid != NULL)
> + *resid = size - totread;
> +
> + return (0);
> +}
> +
> +static off_t
> +splitfs_seek(struct open_file *f, off_t offset, int where)
> +{
> + int nread;
> + size_t resid;
> + off_t new_pos, seek_by;
> + struct split_file *sf;
> +
> + sf = (struct split_file *)f->f_fsdata;
> +
> + seek_by = offset;
> + switch (where) {
> + case SEEK_SET:
> + seek_by -= sf->tot_pos;
> + break;
> + case SEEK_CUR:
> + break;
> + case SEEK_END:
> + panic("splitfs_seek: SEEK_END not supported");
> + break;
> + }
> +
> + if (seek_by > 0) {
> + /*
> + * Seek forward - implemented using splitfs_read(), because otherwise we'll be
> + * unable to detect that we have crossed slice boundary and hence
> + * unable to do a long seek crossing that boundary.
> + */
> + void *tmp;
> +
> + tmp = malloc(SEEK_BUF);
> + if (tmp == NULL)
> + return (-1);
> +
> + nread = 0;
> + for (; seek_by > 0; seek_by -= nread) {
> + resid = 0;
> + errno = splitfs_read(f, tmp, min(seek_by, SEEK_BUF), &resid);
> + nread = min(seek_by, SEEK_BUF) - resid;
> + if ((errno != 0) || (nread == 0))
> + /* Error or EOF */
> + break;
> + }
> + free(tmp);
> + if (errno != 0)
> + return (-1);
> + }
> +
> + if (seek_by != 0) {
> + /* Seek backward or seek past the boundary of the last slice */
> + if (sf->file_pos + seek_by < 0)
> + panic("splitfs_seek: can't seek past the beginning of the slice");
> + new_pos = lseek(sf->curfd, seek_by, SEEK_CUR);
> + if (new_pos < 0)
> + return (-1);
> + sf->tot_pos += new_pos - sf->file_pos;
> + sf->file_pos = new_pos;
> + }
> +
> + return (sf->tot_pos);
> +}
> +
> +static int
> +splitfs_stat(struct open_file *f, struct stat *sb)
> +{
> + int result;
> + struct split_file *sf = (struct split_file *)f->f_fsdata;
> +
> + /* stat as normal, but indicate that size is unknown */
> + if ((result = fstat(sf->curfd, sb)) == 0)
> + sb->st_size = -1;
> + return (result);
> +}
> Index: src/lib/libstand/stand.h
> ===================================================================
> RCS file: /home/ncvs/src/lib/libstand/stand.h,v
> retrieving revision 1.29
> diff -d -u -r1.29 stand.h
> --- src/lib/libstand/stand.h 9 Mar 2002 21:02:11 -0000 1.29
> +++ src/lib/libstand/stand.h 15 Mar 2002 08:40:31 -0000
> @@ -125,6 +125,7 @@
> extern struct fs_ops bzipfs_fsops;
> extern struct fs_ops dosfs_fsops;
> extern struct fs_ops ext2fs_fsops;
> +extern struct fs_ops splitfs_fsops;
>
> /* where values for lseek(2) */
> #define SEEK_SET 0 /* set file offset to offset */
> Index: src/lib/libstand/zipfs.c
> ===================================================================
> RCS file: /home/ncvs/src/lib/libstand/zipfs.c,v
> retrieving revision 1.8
> diff -d -u -r1.8 zipfs.c
> --- src/lib/libstand/zipfs.c 30 Sep 2001 22:28:01 -0000 1.8
> +++ src/lib/libstand/zipfs.c 15 Mar 2002 08:40:31 -0000
> @@ -175,7 +175,7 @@
>
> /* If the name already ends in .gz or .bz2, ignore it */
> if ((cp = strrchr(fname, '.')) && (!strcmp(cp, ".gz")
> - || !strcmp(cp, ".bz2")))
> + || !strcmp(cp, ".bz2") || !strcmp(cp, ".split")))
> return(ENOENT);
>
> /* Construct new name */
> Index: src/sys/boot/i386/loader/conf.c
> ===================================================================
> RCS file: /home/ncvs/src/sys/boot/i386/loader/conf.c,v
> retrieving revision 1.18
> diff -d -u -r1.18 conf.c
> --- src/sys/boot/i386/loader/conf.c 5 Nov 2001 18:59:13 -0000 1.18
> +++ src/sys/boot/i386/loader/conf.c 15 Mar 2002 08:40:31 -0000
> @@ -60,6 +60,7 @@
> &ext2fs_fsops,
> &dosfs_fsops,
> &cd9660_fsops,
> + &splitfs_fsops,
> #ifdef LOADER_GZIP_SUPPORT
> &zipfs_fsops,
> #endif
--
Daniel C. Sobral (8-DCS)
dcs@newsguy.com
dcs@freebsd.org
capo@international.bsdconspiracy.net
"They did what they could to help her, using human skills -- and then,
when that failed, left it in the hands of the gods. In this case," he
bowed slightly, "myself. Like it or not," the demon continued, "that is
my status in this region. Take it up with my priests if it bothers you."
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-audit" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C929F99.18C9EC86>