From owner-freebsd-current@FreeBSD.ORG Wed Jun 11 06:57:25 2003 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5EB8A37B401 for ; Wed, 11 Jun 2003 06:57:25 -0700 (PDT) Received: from mail00.svc.cra.dublin.eircom.net (mail00.svc.cra.dublin.eircom.net [159.134.118.16]) by mx1.FreeBSD.org (Postfix) with SMTP id B369E43FBD for ; Wed, 11 Jun 2003 06:57:21 -0700 (PDT) (envelope-from pmedwards@eircom.net) Received: (qmail 28259 messnum 2020775 invoked from network[159.134.237.90/webmail05.eircom.net]); 11 Jun 2003 13:57:20 -0000 Received: from webmail05.eircom.net (HELO webmail.eircom.net) (159.134.237.90) by mail00.svc.cra.dublin.eircom.net (qp 28259) with SMTP; 11 Jun 2003 13:57:20 -0000 From: "Peter Edwards" To: phk@phk.freebsd.dk Date: Wed, 11 Jun 2003 14:52:51 +0100 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="-------VS3G0JS0PZ73H2Q5VNBV4UY7" Content-Transfer-Encoding: 8bit X-Originating-IP: 62.17.151.61 X-Mailer: Eircom Net CRC Webmail (http://www.eircom.net/) Organization: Eircom Net (http://www.eircom.net/) Message-Id: <20030611135721.B369E43FBD@mx1.FreeBSD.org> cc: dyeske@yahoo.com cc: current@freebsd.org Subject: Re: imgact_gzip.c X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2003 13:57:25 -0000 This is a multi-part message in MIME format. ---------VS3G0JS0PZ73H2Q5VNBV4UY7 Content-Type: text/plain Content-Transfer-Encoding: 7bit On Sat, 2003-06-07 at 10:13, Poul-Henning Kamp wrote: > In message <20030606221137.44110.qmail@web13504.mail.yahoo.com>, David Yeske writes: >> imgact_gzip.c seems to be pretty stale.  Has anyone considered fixing this?  If this were fixed >> then kldload() / linker_load_module() could deal with a gzipped .ko file, and gzipped elf >> executables would work also? > > At least originally imgact_gzip.c was heavily a.out aware. Interesting. Making imgact_gzip elf-aware would not make the kernel capable of loading gzipped modules, only executables. There's a separate link_elf.c that the kernel uses for linking ELF images into itself (rather than activating ELF executables at exec() time, with imgact_elf) I've been fiddling a little with compressed data in the kernel already, and was able to hack together a patch for link_elf pretty quickly. The "quickly" means that there's no boot loader support, and the gzip handling is quite braindamaged, extracting the entire zipped file into allocated memory before parsing the ELF structure. This is mainly because the ELF parsing bits of link_elf assume they can make random access to the file. It'd take a bit of rework to make it work with a serial data stream. The whole thing's very rough around the edges, but I can gzip most of /boot/kernel/*.ko, and load the gzipped versions. I can polish this up, and/or add gzipped executable support, if there's any interest in reviewing or committing it. The patch adds "GZLOADER" and "INFLATE" options for the kernel, removing "GZIP" (which was busted anyway, and considered "inflate.c" to be part of the ELF support, while it's pretty much a standalone decompressor.) There's a "COMPAT_GZAOUT" option added, but it's just as bust as GZIP was before. E&OE. Patch may crash your kernel, delete your data, make your cat unwell, etc. Cheers, Peter. ---------VS3G0JS0PZ73H2Q5VNBV4UY7 Content-Type: text/plain Content-Disposition: attachment; filename="kogz.txt" Index: conf/files =================================================================== RCS file: /pub/FreeBSD/development/FreeBSD-CVS/src/sys/conf/files,v retrieving revision 1.791 diff -u -r1.791 files --- conf/files 9 Jun 2003 19:25:06 -0000 1.791 +++ conf/files 11 Jun 2003 12:48:40 -0000 @@ -1011,7 +1011,7 @@ isofs/cd9660/cd9660_vnops.c optional cd9660 kern/imgact_elf.c standard kern/imgact_shell.c standard -kern/inflate.c optional gzip +kern/inflate.c optional inflate kern/init_main.c standard kern/init_sysent.c standard kern/kern_acct.c standard Index: conf/files.i386 =================================================================== RCS file: /pub/FreeBSD/development/FreeBSD-CVS/src/sys/conf/files.i386,v retrieving revision 1.445 diff -u -r1.445 files.i386 --- conf/files.i386 31 May 2003 17:06:19 -0000 1.445 +++ conf/files.i386 11 Jun 2003 12:51:27 -0000 @@ -407,7 +407,7 @@ isa/syscons_isa.c optional sc isa/vga_isa.c optional vga kern/imgact_aout.c optional compat_aout -kern/imgact_gzip.c optional gzip +kern/imgact_gzip.c optional compat_gzaout libkern/divdi3.c standard libkern/moddi3.c standard libkern/qdivrem.c standard Index: conf/options =================================================================== RCS file: /pub/FreeBSD/development/FreeBSD-CVS/src/sys/conf/options,v retrieving revision 1.393 diff -u -r1.393 options --- conf/options 18 May 2003 03:46:30 -0000 1.393 +++ conf/options 11 Jun 2003 12:54:32 -0000 @@ -603,3 +603,8 @@ # options for hifn driver HIFN_DEBUG opt_hifn.h HIFN_RNDTEST opt_hifn.h + +# options for gzip/"inflate" related functionality +INFLATE opt_inflate.h +COMPAT_GZAOUT opt_gzaout.h +GZLOADER opt_gzloader.h Index: kern/link_elf.c =================================================================== RCS file: /pub/FreeBSD/development/FreeBSD-CVS/src/sys/kern/link_elf.c,v retrieving revision 1.73 diff -u -r1.73 link_elf.c --- kern/link_elf.c 12 May 2003 15:08:10 -0000 1.73 +++ kern/link_elf.c 11 Jun 2003 13:24:50 -0000 @@ -28,6 +28,7 @@ #include "opt_ddb.h" #include "opt_mac.h" +#include "opt_gzloader.h" #include #include @@ -42,6 +43,10 @@ #include #include +#ifdef GZLOADER +#include +#endif + #include #ifdef GPROF #include @@ -98,9 +103,40 @@ #endif } *elf_file_t; +struct vnreader { + struct vnode *vnodep; + struct thread *thread; +}; + +#ifdef GZLOADER +#define MAXGZPAGES (1024 * 1024 / PAGE_SIZE) // Allow modules up to 1MB (uncompressed) + +struct gzreader { + /* reading from gzipped file. */ + int error; + struct vnode *vn; + unsigned char *inPage; + struct thread *td; + int inPageSize; + int inPageOffset; + off_t inFileOffset; + int inPageCount; + + /* gzip context */ + struct inflate inflator; + + /* Writing to inflated output */ + int outPageRes; + int outPageCount; + int outPageOffset; // Size of last page. + unsigned char *pages[MAXGZPAGES]; +}; +#endif + static int link_elf_link_common_finish(linker_file_t); static int link_elf_link_preload(linker_class_t cls, const char*, linker_file_t*); + static int link_elf_link_preload_finish(linker_file_t); static int link_elf_load_file(linker_class_t, const char*, linker_file_t*); static int link_elf_lookup_symbol(linker_file_t, const char*, @@ -118,6 +154,22 @@ void *); static void link_elf_reloc_local(linker_file_t); +#ifdef GZLOADER +static int link_gz_link_preload_finish(linker_file_t); +static int link_gz_load_file(linker_class_t, const char*, linker_file_t*); +static int link_gz_link_preload(linker_class_t cls, + const char*, linker_file_t*); +static void release_gzreader(struct gzreader *zr); +static int gzreadfunc(void *, unsigned char *, int, off_t, int *); +static int gzin(void *vp); +static int gzout(void *vp, unsigned char *data, unsigned long size); +#endif +static int link_elf_load_object(void *, int (*)(void *, unsigned char *, int, off_t, int *), + const char *filename, linker_file_t* result); + +static int vnreadfunc(void *, unsigned char *, int, off_t, int *); + + static kobj_method_t link_elf_methods[] = { KOBJMETHOD(linker_lookup_symbol, link_elf_lookup_symbol), KOBJMETHOD(linker_symbol_values, link_elf_symbol_values), @@ -140,6 +192,37 @@ link_elf_methods, sizeof(struct elf_file) }; +#ifdef GZLOADER +/* + * The gzip loader is almost the same as the ELF loader, only when it comes to + * reading the file from disk. Symbol lookups, unloading, etc, are all thesame + * as for ELF. For the moment, preloaded files aren't supported: some work in + * the boot loader is required. + */ + +static kobj_method_t link_gz_methods[] = { + KOBJMETHOD(linker_lookup_symbol, link_elf_lookup_symbol), + KOBJMETHOD(linker_symbol_values, link_elf_symbol_values), + KOBJMETHOD(linker_search_symbol, link_elf_search_symbol), + KOBJMETHOD(linker_unload, link_elf_unload_file), + KOBJMETHOD(linker_load_file, link_gz_load_file), + KOBJMETHOD(linker_link_preload, link_gz_link_preload), + KOBJMETHOD(linker_link_preload_finish, link_gz_link_preload_finish), + KOBJMETHOD(linker_lookup_set, link_elf_lookup_set), + KOBJMETHOD(linker_each_function_name, link_elf_each_function_name), + { 0, 0 } +}; + +static struct linker_class link_gz_class = { +#if ELF_TARG_CLASS == ELFCLASS32 + "gzelf32", +#else + "gzelf64", +#endif + link_gz_methods, sizeof(struct elf_file) +}; +#endif + static int parse_dynamic(elf_file_t ef); static int relocate_file(elf_file_t ef); static int link_elf_preload_parse_symbols(elf_file_t ef); @@ -255,6 +338,9 @@ char *modname; linker_add_class(&link_elf_class); +#ifdef GZLOADER + linker_add_class(&link_gz_class); +#endif dp = (Elf_Dyn*) &_DYNAMIC; modname = NULL; @@ -524,11 +610,10 @@ } static int -link_elf_load_file(linker_class_t cls, const char* filename, - linker_file_t* result) +link_elf_load_object(void *readCookie, + int (*readfunc)(void *, unsigned char *data, int, off_t, int *), + const char *filename, linker_file_t* result) { - struct nameidata nd; - struct thread* td = curthread; /* XXX */ Elf_Ehdr *hdr; caddr_t firstpage; int nbytes, i; @@ -544,7 +629,6 @@ Elf_Addr base_vaddr; Elf_Addr base_vlimit; int error = 0; - int resid, flags; elf_file_t ef; linker_file_t lf; Elf_Shdr *shdr; @@ -552,26 +636,13 @@ int symstrindex; int symcnt; int strcnt; + int resid; GIANT_REQUIRED; shdr = NULL; lf = NULL; - NDINIT(&nd, LOOKUP, FOLLOW, UIO_SYSSPACE, filename, td); - flags = FREAD; - error = vn_open(&nd, &flags, 0); - if (error) - return error; - NDFREE(&nd, NDF_ONLY_PNBUF); -#ifdef MAC - error = mac_check_kld_load(curthread->td_ucred, nd.ni_vp); - if (error) { - firstpage = NULL; - goto out; - } -#endif - /* * Read the elf header from the file. */ @@ -581,9 +652,7 @@ goto out; } hdr = (Elf_Ehdr *)firstpage; - error = vn_rdwr(UIO_READ, nd.ni_vp, firstpage, PAGE_SIZE, 0, - UIO_SYSSPACE, IO_NODELOCKED, td->td_ucred, NOCRED, - &resid, td); + error = readfunc(readCookie, firstpage, PAGE_SIZE, 0, &resid); nbytes = PAGE_SIZE - resid; if (error) goto out; @@ -727,10 +796,8 @@ */ for (i = 0; i < 2; i++) { caddr_t segbase = mapbase + segs[i]->p_vaddr - base_vaddr; - error = vn_rdwr(UIO_READ, nd.ni_vp, - segbase, segs[i]->p_filesz, segs[i]->p_offset, - UIO_SYSSPACE, IO_NODELOCKED, td->td_ucred, NOCRED, - &resid, td); + error = readfunc(readCookie, + segbase, segs[i]->p_filesz, segs[i]->p_offset, &resid); if (error) { goto out; } @@ -790,10 +857,7 @@ error = ENOMEM; goto out; } - error = vn_rdwr(UIO_READ, nd.ni_vp, - (caddr_t)shdr, nbytes, hdr->e_shoff, - UIO_SYSSPACE, IO_NODELOCKED, td->td_ucred, NOCRED, - &resid, td); + error = readfunc(readCookie, (caddr_t)shdr, nbytes, hdr->e_shoff, &resid); if (error) goto out; symtabindex = -1; @@ -816,16 +880,12 @@ error = ENOMEM; goto out; } - error = vn_rdwr(UIO_READ, nd.ni_vp, - ef->symbase, symcnt, shdr[symtabindex].sh_offset, - UIO_SYSSPACE, IO_NODELOCKED, td->td_ucred, NOCRED, - &resid, td); + error = readfunc(readCookie, ef->symbase, symcnt, + shdr[symtabindex].sh_offset, &resid); if (error) goto out; - error = vn_rdwr(UIO_READ, nd.ni_vp, - ef->strbase, strcnt, shdr[symstrindex].sh_offset, - UIO_SYSSPACE, IO_NODELOCKED, td->td_ucred, NOCRED, - &resid, td); + error = readfunc(readCookie, ef->strbase, strcnt, + shdr[symstrindex].sh_offset, &resid); if (error) goto out; @@ -849,12 +909,54 @@ free(shdr, M_LINKER); if (firstpage) free(firstpage, M_LINKER); - VOP_UNLOCK(nd.ni_vp, 0, td); - vn_close(nd.ni_vp, FREAD, td->td_ucred, td); return error; } +static int +vnreadfunc(void *readCookie, unsigned char *data, int len, off_t offset, int *residp) +{ + struct vnreader *vnr = (struct vnreader *)readCookie; + + return vn_rdwr(UIO_READ, vnr->vnodep, data, len, offset, + UIO_SYSSPACE, IO_NODELOCKED, vnr->thread->td_ucred, NOCRED, + residp, vnr->thread); +} + +static int +link_elf_load_file(linker_class_t cls, const char* filename, + linker_file_t* result) +{ + struct nameidata nd; + struct vnreader vr; + int error, flags; + struct thread *td = curthread; + + NDINIT(&nd, LOOKUP, FOLLOW, UIO_SYSSPACE, filename, td); + flags = FREAD; + error = vn_open(&nd, &flags, 0); + if (error) + return error; + NDFREE(&nd, NDF_ONLY_PNBUF); +#ifdef MAC + error = mac_check_kld_load(curthread->td_ucred, nd.ni_vp); + if (error) { + firstpage = NULL; + goto out; + } +#endif + vr.vnodep = nd.ni_vp; + vr.thread = curthread; + error = link_elf_load_object(&vr, vnreadfunc, filename, result); +#ifdef MAC +out: +#endif + vn_close(nd.ni_vp, FREAD, td->td_ucred, td); + VOP_UNLOCK(nd.ni_vp, 0, td); + return error; +} + + static void link_elf_unload_file(linker_file_t file) { @@ -1310,3 +1412,206 @@ } } } + +#ifdef GZLOADER +static int +gzreadfunc(void *readCookie, unsigned char *data, int len, off_t offset, int *residp) +{ + struct gzreader *zr = (struct gzreader *)readCookie; + int pageId, pageOff, pageSize, copySize; + + while (len) { + pageId = offset / PAGE_SIZE; + /* Cannot read beyond last page. */ + if (pageId < 0 || pageId >= zr->outPageCount) + break; + pageOff = offset % PAGE_SIZE; + + if (pageId == zr->outPageCount - 1) { + pageSize = zr->outPageOffset; + /* Last page may not be a full page size */ + if (pageOff >= zr->outPageOffset) + break; + } else { + pageSize = PAGE_SIZE; + } + copySize = MIN(pageSize - pageOff, len); + bcopy(zr->pages[pageId] + pageOff, data, copySize); + len -= copySize; + offset += copySize; + data += copySize; + } + if (residp) + *residp = len; + return 0; +} + +static int +link_gz_load_file(linker_class_t cls, const char* filename, + linker_file_t* result) +{ + int resid; + const unsigned char *p; + struct nameidata nd; + struct gzreader *zr = 0; + int error, flags; + struct thread *td = curthread; + + NDINIT(&nd, LOOKUP, FOLLOW, UIO_SYSSPACE, filename, td); + flags = FREAD; + error = vn_open(&nd, &flags, 0); + if (error) + return error; + NDFREE(&nd, NDF_ONLY_PNBUF); +#ifdef MAC + error = mac_check_kld_load(td->td_ucred, nd.ni_vp); + if (error) + goto out; +#endif + + zr = malloc(sizeof *zr, M_LINKER, M_WAITOK); + if (zr == 0) + goto out; + + bzero(zr, sizeof *zr); + zr->td = td; + zr->vn = nd.ni_vp; + zr->inflator.gz_private = zr; + zr->inflator.gz_input = gzin; + zr->inflator.gz_output = gzout; + + /* + * XXX: Would it be better to map the VM pages of the vnode, rather than + * using malloc/vn_rdwr()? + */ + zr->inPage = malloc(PAGE_SIZE, M_LINKER, M_WAITOK); + if (zr->inPage == 0) + goto out; + + error = vn_rdwr(UIO_READ, nd.ni_vp, zr->inPage, PAGE_SIZE, 0, + UIO_SYSSPACE, IO_NODELOCKED, td->td_ucred, NOCRED, &resid, td); + if (error) + goto out; + zr->inFileOffset = zr->inPageSize = PAGE_SIZE - resid; + + p = zr->inPage; + + /* Magic from kern/imgact_gzip.c */ + if (p[0] != 0x1f || p[1] != 0x8b || p[2] != 0x08 /* gzip magic */ + || p[9] != 0x03 /* Compression type */ + || p[3] & ~0x18 /* Extra fields: just support filename and comment */ + ) { + error = ENOEXEC; + goto out; + } + + zr->inPageOffset = 10; + + /* Skip filename in gzip file if present */ + if (p[3] & 0x8) { + while (p[zr->inPageOffset++]) { + if (zr->inPageOffset == zr->inPageSize) { + error = ENOEXEC; + goto out; + } + } + } + + /* Skip comment in gzip file if present */ + if (p[3] & 0x10) { + while (p[zr->inPageOffset++]) { + if (zr->inPageOffset == zr->inPageSize) { + error = ENOEXEC; + goto out; + } + } + } + error = inflate(&zr->inflator); /* inflate the entire file */ + if (error) + goto out; + if (zr->error) { + error = zr->error; + goto out; + } + error = link_elf_load_object(zr, gzreadfunc, filename, result); +out: + vn_close(nd.ni_vp, FREAD, td->td_ucred, td); + VOP_UNLOCK(nd.ni_vp, 0, td); + if (zr) + release_gzreader(zr); + return error; +} + +static void +release_gzreader(struct gzreader *zr) +{ + if (zr->inPage) + free(zr->inPage, M_LINKER); + while (zr->outPageCount--) + free(zr->pages[zr->outPageCount], M_LINKER); + free(zr, M_LINKER); +} + +static int +link_gz_link_preload(linker_class_t cls, + const char* filename, linker_file_t *result) +{ + return ENOENT; +} + +static int +link_gz_link_preload_finish(linker_file_t l) +{ + return ENOENT; +} + +static int +gzin(void *vp) +{ + struct gzreader *zr = (struct gzreader *) vp; + if (zr->inPageSize == zr->inPageOffset) { + int resid; + /* We have consumed the entire page. */ + zr->error = vn_rdwr(UIO_READ, zr->vn, zr->inPage, PAGE_SIZE, + zr->inFileOffset, UIO_SYSSPACE, IO_NODELOCKED, + zr->td->td_ucred, NOCRED, &resid, zr->td); + if (zr->error) + return GZ_EOF; + if (resid == PAGE_SIZE) + return GZ_EOF; + zr->inFileOffset += PAGE_SIZE - resid; + zr->inPageOffset = 0; + } + return zr->inPage[zr->inPageOffset++]; +} + +static int +gzout(void *vp, unsigned char *data, unsigned long size) +{ + unsigned char *page; + int space; + + struct gzreader *zr = (struct gzreader *) vp; + while (size) { + if (zr->outPageCount == 0 || zr->outPageOffset == PAGE_SIZE) { + /* We need a new page to generate output into. */ + if (zr->outPageCount == MAXGZPAGES) + return ENOEXEC; + zr->pages[zr->outPageCount] = malloc(PAGE_SIZE, M_LINKER, M_WAITOK); + if (zr->pages[zr->outPageCount] == 0) + return ENOEXEC; + zr->outPageOffset = 0; + zr->outPageCount++; /* That's one more to free. */ + } + + /* Copy inflated data into our image */ + page = zr->pages[zr->outPageCount - 1]; + space = MIN(PAGE_SIZE - zr->outPageOffset, size); + bcopy(data, page + zr->outPageOffset, space); + data += space; + zr->outPageOffset += space; + size -= space; + } + return 0; +} +#endif Index: nfsclient/nfs_vnops.c =================================================================== RCS file: /pub/FreeBSD/development/FreeBSD-CVS/src/sys/nfsclient/nfs_vnops.c,v retrieving revision 1.205 diff -u -r1.205 nfs_vnops.c --- nfsclient/nfs_vnops.c 15 May 2003 21:12:08 -0000 1.205 +++ nfsclient/nfs_vnops.c 16 May 2003 21:29:55 -0000 @@ -670,6 +670,13 @@ (error = nfs_vinvalbuf(vp, V_SAVE, ap->a_cred, ap->a_td, 1)) == EINTR) return (error); + + /* + * It's likely that changing the file's mode will affect it's + * accessibilty: invalidate access cache + */ + if (vap->va_mode != (mode_t)VNOVAL) + np->n_modestamp = 0; error = nfs_setattrrpc(vp, vap, ap->a_cred, ap->a_td); if (error && vap->va_size != VNOVAL) { np->n_size = np->n_vattr.va_size = tsize; ---------VS3G0JS0PZ73H2Q5VNBV4UY7--