Date: Mon, 12 Sep 2016 22:07:35 +0000 (UTC) From: Martin Matuska <mm@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r305755 - in stable/10: contrib/libarchive contrib/libarchive/libarchive contrib/libarchive/libarchive/test lib/libarchive Message-ID: <201609122207.u8CM7ZaV029748@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: mm Date: Mon Sep 12 22:07:35 2016 New Revision: 305755 URL: https://svnweb.freebsd.org/changeset/base/305755 Log: MFC r305422: Sync libarchive with vendor Vendor issues fixed: PR #777: Multiple bugfixes for setup_acls() This includes a bugfix for a bug that caused ACLs not to be read properly for files and directories inside subdirectories and as a result not being stored or being incorrectly stored in tar archives. Added: stable/10/contrib/libarchive/README.md - copied unchanged from r305422, head/contrib/libarchive/README.md Deleted: stable/10/contrib/libarchive/README Modified: stable/10/contrib/libarchive/libarchive/archive_acl.c stable/10/contrib/libarchive/libarchive/archive_read_disk_entry_from_file.c stable/10/contrib/libarchive/libarchive/archive_read_disk_posix.c stable/10/contrib/libarchive/libarchive/test/test_acl_freebsd_posix1e.c stable/10/lib/libarchive/config_freebsd.h Directory Properties: stable/10/ (props changed) Copied: stable/10/contrib/libarchive/README.md (from r305422, head/contrib/libarchive/README.md) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ stable/10/contrib/libarchive/README.md Mon Sep 12 22:07:35 2016 (r305755, copy of r305422, head/contrib/libarchive/README.md) @@ -0,0 +1,222 @@ +# Welcome to libarchive! + +The libarchive project develops a portable, efficient C library that +can read and write streaming archives in a variety of formats. It +also includes implementations of the common `tar`, `cpio`, and `zcat` +command-line tools that use the libarchive library. + +## Questions? Issues? + +* http://www.libarchive.org is the home for ongoing + libarchive development, including documentation, + and links to the libarchive mailing lists. +* To report an issue, use the issue tracker at + https://github.com/libarchive/libarchive/issues +* To submit an enhancement to libarchive, please + submit a pull request via GitHub: https://github.com/libarchive/libarchive/pulls + +## Contents of the Distribution + +This distribution bundle includes the following major components: + +* **libarchive**: a library for reading and writing streaming archives +* **tar**: the 'bsdtar' program is a full-featured 'tar' implementation built on libarchive +* **cpio**: the 'bsdcpio' program is a different interface to essentially the same functionality +* **cat**: the 'bsdcat' program is a simple replacement tool for zcat, bzcat, xzcat, and such +* **examples**: Some small example programs that you may find useful. +* **examples/minitar**: a compact sample demonstrating use of libarchive. +* **contrib**: Various items sent to me by third parties; please contact the authors with any questions. + +The top-level directory contains the following information files: + +* **NEWS** - highlights of recent changes +* **COPYING** - what you can do with this +* **INSTALL** - installation instructions +* **README** - this file +* **CMakeLists.txt** - input for "cmake" build tool, see INSTALL +* **configure** - configuration script, see INSTALL for details. If your copy of the source lacks a `configure` script, you can try to construct it by running the script in `build/autogen.sh` (or use `cmake`). + +The following files in the top-level directory are used by the 'configure' script: +* `Makefile.am`, `aclocal.m4`, `configure.ac` - used to build this distribution, only needed by maintainers +* `Makefile.in`, `config.h.in` - templates used by configure script + +## Documentation + +In addition to the informational articles and documentation +in the online [libarchive Wiki](https://github.com/libarchive/libarchive/wiki), +the distribution also includes a number of manual pages: + + * bsdtar.1 explains the use of the bsdtar program + * bsdcpio.1 explains the use of the bsdcpio program + * bsdcat.1 explains the use of the bsdcat program + * libarchive.3 gives an overview of the library as a whole + * archive_read.3, archive_write.3, archive_write_disk.3, and + archive_read_disk.3 provide detailed calling sequences for the read + and write APIs + * archive_entry.3 details the "struct archive_entry" utility class + * archive_internals.3 provides some insight into libarchive's + internal structure and operation. + * libarchive-formats.5 documents the file formats supported by the library + * cpio.5, mtree.5, and tar.5 provide detailed information about these + popular archive formats, including hard-to-find details about + modern cpio and tar variants. + +The manual pages above are provided in the 'doc' directory in +a number of different formats. + +You should also read the copious comments in `archive.h` and the +source code for the sample programs for more details. Please let us +know about any errors or omissions you find. + +## Supported Formats + +Currently, the library automatically detects and reads the following fomats: + * Old V7 tar archives + * POSIX ustar + * GNU tar format (including GNU long filenames, long link names, and sparse files) + * Solaris 9 extended tar format (including ACLs) + * POSIX pax interchange format + * POSIX octet-oriented cpio + * SVR4 ASCII cpio + * POSIX octet-oriented cpio + * Binary cpio (big-endian or little-endian) + * ISO9660 CD-ROM images (with optional Rockridge or Joliet extensions) + * ZIP archives (with uncompressed or "deflate" compressed entries, including support for encrypted Zip archives) + * GNU and BSD 'ar' archives + * 'mtree' format + * 7-Zip archives + * Microsoft CAB format + * LHA and LZH archives + * RAR archives (with some limitations due to RAR's proprietary status) + * XAR archives + +The library also detects and handles any of the following before evaluating the archive: + * uuencoded files + * files with RPM wrapper + * gzip compression + * bzip2 compression + * compress/LZW compression + * lzma, lzip, and xz compression + * lz4 compression + * lzop compression + +The library can create archives in any of the following formats: + * POSIX ustar + * POSIX pax interchange format + * "restricted" pax format, which will create ustar archives except for + entries that require pax extensions (for long filenames, ACLs, etc). + * Old GNU tar format + * Old V7 tar format + * POSIX octet-oriented cpio + * SVR4 "newc" cpio + * shar archives + * ZIP archives (with uncompressed or "deflate" compressed entries) + * GNU and BSD 'ar' archives + * 'mtree' format + * ISO9660 format + * 7-Zip archives + * XAR archives + +When creating archives, the result can be filtered with any of the following: + * uuencode + * gzip compression + * bzip2 compression + * compress/LZW compression + * lzma, lzip, and xz compression + * lz4 compression + * lzop compression + +## Notes about the Library Design + +The following notes address many of the most common +questions we are asked about libarchive: + +* This is a heavily stream-oriented system. That means that + it is optimized to read or write the archive in a single + pass from beginning to end. For example, this allows + libarchive to process archives too large to store on disk + by processing them on-the-fly as they are read from or + written to a network or tape drive. This also makes + libarchive useful for tools that need to produce + archives on-the-fly (such as webservers that provide + archived contents of a users account). + +* In-place modification and random access to the contents + of an archive are not directly supported. For some formats, + this is not an issue: For example, tar.gz archives are not + designed for random access. In some other cases, libarchive + can re-open an archive and scan it from the beginning quickly + enough to provide the needed abilities even without true + random access. Of course, some applications do require true + random access; those applications should consider alternatives + to libarchive. + +* The library is designed to be extended with new compression and + archive formats. The only requirement is that the format be + readable or writable as a stream and that each archive entry be + independent. There are articles on the libarchive Wiki explaining + how to extend libarchive. + +* On read, compression and format are always detected automatically. + +* The same API is used for all formats; in particular, it's very + easy for software using libarchive to transparently handle + any of libarchive's archiving formats. + +* Libarchive's automatic support for decompression can be used + without archiving by explicitly selecting the "raw" and "empty" + formats. + +* I've attempted to minimize static link pollution. If you don't + explicitly invoke a particular feature (such as support for a + particular compression or format), it won't get pulled in to + statically-linked programs. In particular, if you don't explicitly + enable a particular compression or decompression support, you won't + need to link against the corresponding compression or decompression + libraries. This also reduces the size of statically-linked + binaries in environments where that matters. + +* The library is generally _thread safe_ depending on the platform: + it does not define any global variables of its own. However, some + platforms do not provide fully thread-safe versions of key C library + functions. On those platforms, libarchive will use the non-thread-safe + functions. Patches to improve this are of great interest to us. + +* In particular, libarchive's modules to read or write a directory + tree do use `chdir()` to optimize the directory traversals. This + can cause problems for programs that expect to do disk access from + multiple threads. Of course, those modules are completely + optional and you can use the rest of libarchive without them. + +* The library is _not_ thread aware, however. It does no locking + or thread management of any kind. If you create a libarchive + object and need to access it from multiple threads, you will + need to provide your own locking. + +* On read, the library accepts whatever blocks you hand it. + Your read callback is free to pass the library a byte at a time + or mmap the entire archive and give it to the library at once. + On write, the library always produces correctly-blocked output. + +* The object-style approach allows you to have multiple archive streams + open at once. bsdtar uses this in its "@archive" extension. + +* The archive itself is read/written using callback functions. + You can read an archive directly from an in-memory buffer or + write it to a socket, if you wish. There are some utility + functions to provide easy-to-use "open file," etc, capabilities. + +* The read/write APIs are designed to allow individual entries + to be read or written to any data source: You can create + a block of data in memory and add it to a tar archive without + first writing a temporary file. You can also read an entry from + an archive and write the data directly to a socket. If you want + to read/write entries to disk, there are convenience functions to + make this especially easy. + +* Note: The "pax interchange format" is a POSIX standard extended tar + format that should be used when the older _ustar_ format is not + appropriate. It has many advantages over other tar formats + (including the legacy GNU tar format) and is widely supported by + current tar implementations. + Modified: stable/10/contrib/libarchive/libarchive/archive_acl.c ============================================================================== --- stable/10/contrib/libarchive/libarchive/archive_acl.c Mon Sep 12 22:07:00 2016 (r305754) +++ stable/10/contrib/libarchive/libarchive/archive_acl.c Mon Sep 12 22:07:35 2016 (r305755) @@ -707,10 +707,11 @@ archive_acl_text_l(struct archive_acl *a if (r != 0) return (-1); *p++ = separator; - if (flags & ARCHIVE_ENTRY_ACL_STYLE_EXTRA_ID) + if (name == NULL || (flags & ARCHIVE_ENTRY_ACL_STYLE_EXTRA_ID)) { id = ap->id; - else + } else { id = -1; + } append_entry(&p, NULL, ap->tag, name, ap->permset, id); count++; Modified: stable/10/contrib/libarchive/libarchive/archive_read_disk_entry_from_file.c ============================================================================== --- stable/10/contrib/libarchive/libarchive/archive_read_disk_entry_from_file.c Mon Sep 12 22:07:00 2016 (r305754) +++ stable/10/contrib/libarchive/libarchive/archive_read_disk_entry_from_file.c Mon Sep 12 22:07:35 2016 (r305755) @@ -419,12 +419,32 @@ setup_acls(struct archive_read_disk *a, if (accpath == NULL) accpath = archive_entry_pathname(entry); + if (*fd < 0 && a->tree != NULL) { + if (a->follow_symlinks || + archive_entry_filetype(entry) != AE_IFLNK) + *fd = a->open_on_current_dir(a->tree, + accpath, O_RDONLY | O_NONBLOCK); + if (*fd < 0) { + if (a->tree_enter_working_dir(a->tree) != 0) { + archive_set_error(&a->archive, errno, + "Couldn't access %s", accpath); + return (ARCHIVE_FAILED); + } + } + } + archive_entry_acl_clear(entry); + acl = NULL; + #ifdef ACL_TYPE_NFS4 /* Try NFS4 ACL first. */ if (*fd >= 0) +#if HAVE_ACL_GET_FD_NP + acl = acl_get_fd_np(*fd, ACL_TYPE_NFS4); +#else acl = acl_get_fd(*fd); +#endif #if HAVE_ACL_GET_LINK_NP else if (!a->follow_symlinks) acl = acl_get_link_np(accpath, ACL_TYPE_NFS4); @@ -437,12 +457,19 @@ setup_acls(struct archive_read_disk *a, #endif else acl = acl_get_file(accpath, ACL_TYPE_NFS4); + #if HAVE_ACL_IS_TRIVIAL_NP - /* Ignore "trivial" ACLs that just mirror the file mode. */ - acl_is_trivial_np(acl, &r); - if (r) { - acl_free(acl); - acl = NULL; + if (acl != NULL && acl_is_trivial_np(acl, &r) == 0) { + /* Ignore "trivial" ACLs that just mirror the file mode. */ + if (r) { + acl_free(acl); + acl = NULL; + /* + * Simultaneous NFSv4 and POSIX.1e ACLs for the same + * entry are not allowed, so we should return here + */ + return (ARCHIVE_OK); + } } #endif if (acl != NULL) { @@ -450,7 +477,7 @@ setup_acls(struct archive_read_disk *a, acl_free(acl); return (ARCHIVE_OK); } -#endif +#endif /* ACL_TYPE_NFS4 */ /* Retrieve access ACL from file. */ if (*fd >= 0) @@ -467,10 +494,22 @@ setup_acls(struct archive_read_disk *a, #endif else acl = acl_get_file(accpath, ACL_TYPE_ACCESS); + +#if HAVE_ACL_IS_TRIVIAL_NP + /* Ignore "trivial" ACLs that just mirror the file mode. */ + if (acl != NULL && acl_is_trivial_np(acl, &r) == 0) { + if (r) { + acl_free(acl); + acl = NULL; + } + } +#endif + if (acl != NULL) { translate_acl(a, entry, acl, ARCHIVE_ENTRY_ACL_TYPE_ACCESS); acl_free(acl); + acl = NULL; } /* Only directories can have default ACLs. */ Modified: stable/10/contrib/libarchive/libarchive/archive_read_disk_posix.c ============================================================================== --- stable/10/contrib/libarchive/libarchive/archive_read_disk_posix.c Mon Sep 12 22:07:00 2016 (r305754) +++ stable/10/contrib/libarchive/libarchive/archive_read_disk_posix.c Mon Sep 12 22:07:35 2016 (r305755) @@ -1504,10 +1504,19 @@ setup_current_filesystem(struct archive_ struct tree *t = a->tree; struct statfs sfs; #if defined(HAVE_GETVFSBYNAME) && defined(VFCF_SYNTHETIC) -# if defined(HAVE_STRUCT_VFSCONF) - struct vfsconf vfc; -# else +/* TODO: configure should set GETVFSBYNAME_ARG_TYPE to make + * this accurate; some platforms have both and we need the one that's + * used by getvfsbyname() + * + * Then the following would become: + * #if defined(GETVFSBYNAME_ARG_TYPE) + * GETVFSBYNAME_ARG_TYPE vfc; + * #endif + */ +# if defined(HAVE_STRUCT_XVFSCONF) struct xvfsconf vfc; +# else + struct vfsconf vfc; # endif #endif int r, xr = 0; Modified: stable/10/contrib/libarchive/libarchive/test/test_acl_freebsd_posix1e.c ============================================================================== --- stable/10/contrib/libarchive/libarchive/test/test_acl_freebsd_posix1e.c Mon Sep 12 22:07:00 2016 (r305754) +++ stable/10/contrib/libarchive/libarchive/test/test_acl_freebsd_posix1e.c Mon Sep 12 22:07:35 2016 (r305755) @@ -70,15 +70,9 @@ set_acls(struct archive_entry *ae, struc } static int -acl_match(acl_entry_t aclent, struct myacl_t *myacl) -{ - gid_t g, *gp; - uid_t u, *up; - acl_tag_t tag_type; - acl_permset_t opaque_ps; +acl_entry_get_perm(acl_entry_t aclent) { int permset = 0; - - acl_get_tag_type(aclent, &tag_type); + acl_permset_t opaque_ps; /* translate the silly opaque permset to a bitmap */ acl_get_permset(aclent, &opaque_ps); @@ -88,10 +82,61 @@ acl_match(acl_entry_t aclent, struct mya permset |= ARCHIVE_ENTRY_ACL_WRITE; if (acl_get_perm_np(opaque_ps, ACL_READ)) permset |= ARCHIVE_ENTRY_ACL_READ; + return permset; +} + +#if 0 +static int +acl_get_specific_entry(acl_t acl, acl_tag_t requested_tag_type, int requested_tag) { + int entry_id = ACL_FIRST_ENTRY; + acl_entry_t acl_entry; + acl_tag_t acl_tag_type; + + while (1 == acl_get_entry(acl, entry_id, &acl_entry)) { + /* After the first time... */ + entry_id = ACL_NEXT_ENTRY; + + /* If this matches, return perm mask */ + acl_get_tag_type(acl_entry, &acl_tag_type); + if (acl_tag_type == requested_tag_type) { + switch (acl_tag_type) { + case ACL_USER_OBJ: + if ((uid_t)requested_tag == *(uid_t *)(acl_get_qualifier(acl_entry))) { + return acl_entry_get_perm(acl_entry); + } + break; + case ACL_GROUP_OBJ: + if ((gid_t)requested_tag == *(gid_t *)(acl_get_qualifier(acl_entry))) { + return acl_entry_get_perm(acl_entry); + } + break; + case ACL_USER: + case ACL_GROUP: + case ACL_OTHER: + return acl_entry_get_perm(acl_entry); + default: + failure("Unexpected ACL tag type"); + assert(0); + } + } + - if (permset != myacl->permset) + } + return -1; +} +#endif + +static int +acl_match(acl_entry_t aclent, struct myacl_t *myacl) +{ + gid_t g, *gp; + uid_t u, *up; + acl_tag_t tag_type; + + if (myacl->permset != acl_entry_get_perm(aclent)) return (0); + acl_get_tag_type(aclent, &tag_type); switch (tag_type) { case ACL_USER_OBJ: if (myacl->tag != ARCHIVE_ENTRY_ACL_USER_OBJ) return (0); @@ -190,7 +235,7 @@ compare_acls(acl_t acl, struct myacl_t * * Verify ACL restore-to-disk. This test is FreeBSD-specific. */ -DEFINE_TEST(test_acl_freebsd_posix1e) +DEFINE_TEST(test_acl_freebsd_posix1e_restore) { #if !defined(__FreeBSD__) skipping("FreeBSD-specific ACL restore test"); @@ -263,3 +308,111 @@ DEFINE_TEST(test_acl_freebsd_posix1e) acl_free(acl); #endif } + +/* + * Verify ACL reaed-from-disk. This test is FreeBSD-specific. + */ +DEFINE_TEST(test_acl_freebsd_posix1e_read) +{ +#if !defined(__FreeBSD__) + skipping("FreeBSD-specific ACL read test"); +#elif __FreeBSD__ < 5 + skipping("ACL read supported only on FreeBSD 5.0 and later"); +#else + struct archive *a; + struct archive_entry *ae; + int n, fd; + const char *acl1_text, *acl2_text; + acl_t acl1, acl2; + + /* + * Manually construct a directory and two files with + * different ACLs. This also serves to verify that ACLs + * are supported on the local filesystem. + */ + + /* Create a test file f1 with acl1 */ + acl1_text = "user::rwx,group::rwx,other::rwx,user:1:rw-,group:15:r-x,mask::rwx"; + acl1 = acl_from_text(acl1_text); + assert((void *)acl1 != NULL); + fd = open("f1", O_WRONLY | O_CREAT | O_EXCL, 0777); + failure("Could not create test file?!"); + if (!assert(fd >= 0)) { + acl_free(acl1); + return; + } + n = acl_set_fd(fd, acl1); + acl_free(acl1); + if (n != 0 && errno == EOPNOTSUPP) { + close(fd); + skipping("ACL tests require that ACL support be enabled on the filesystem"); + return; + } + if (n != 0 && errno == EINVAL) { + close(fd); + skipping("This filesystem does not support POSIX.1e ACLs"); + return; + } + failure("acl_set_fd(): errno = %d (%s)", + errno, strerror(errno)); + assertEqualInt(0, n); + close(fd); + + assertMakeDir("d", 0700); + + /* + * Create file d/f1 with acl2 + * + * This differs from acl1 in the u:1: and g:15: permissions. + * + * This file deliberately has the same name but a different ACL. + * Github Issue #777 explains how libarchive's directory traversal + * did not always correctly enter directories before attempting + * to read ACLs, resulting in reading the ACL from a like-named + * file in the wrong directory. + */ + acl2_text = "user::rwx,group::rwx,other::---,user:1:r--,group:15:r--,mask::rwx"; + acl2 = acl_from_text(acl2_text); + assert((void *)acl2 != NULL); + fd = open("d/f1", O_WRONLY | O_CREAT | O_EXCL, 0777); + failure("Could not create test file?!"); + if (!assert(fd >= 0)) { + acl_free(acl2); + return; + } + n = acl_set_fd(fd, acl2); + acl_free(acl2); + if (n != 0 && errno == EOPNOTSUPP) { + close(fd); + skipping("ACL tests require that ACL support be enabled on the filesystem"); + return; + } + if (n != 0 && errno == EINVAL) { + close(fd); + skipping("This filesystem does not support POSIX.1e ACLs"); + return; + } + failure("acl_set_fd(): errno = %d (%s)", + errno, strerror(errno)); + assertEqualInt(0, n); + close(fd); + + /* Create a read-from-disk object. */ + assert(NULL != (a = archive_read_disk_new())); + assertEqualIntA(a, ARCHIVE_OK, archive_read_disk_open(a, ".")); + assert(NULL != (ae = archive_entry_new())); + + /* Walk the dir until we see both of the files */ + while (ARCHIVE_OK == archive_read_next_header2(a, ae)) { + archive_read_disk_descend(a); + if (strcmp(archive_entry_pathname(ae), "./f1") == 0) { + assertEqualString(archive_entry_acl_text(ae, ARCHIVE_ENTRY_ACL_TYPE_ACCESS), acl1_text); + + } else if (strcmp(archive_entry_pathname(ae), "./d/f1") == 0) { + assertEqualString(archive_entry_acl_text(ae, ARCHIVE_ENTRY_ACL_TYPE_ACCESS), acl2_text); + } + } + + archive_free(a); +#endif +} Modified: stable/10/lib/libarchive/config_freebsd.h ============================================================================== --- stable/10/lib/libarchive/config_freebsd.h Mon Sep 12 22:07:00 2016 (r305754) +++ stable/10/lib/libarchive/config_freebsd.h Mon Sep 12 22:07:35 2016 (r305755) @@ -28,6 +28,7 @@ /* FreeBSD 5.0 and later have ACL and extattr support. */ #if __FreeBSD__ > 4 #define HAVE_ACL_CREATE_ENTRY 1 +#define HAVE_ACL_GET_FD_NP 1 #define HAVE_ACL_GET_LINK_NP 1 #define HAVE_ACL_GET_PERM_NP 1 #define HAVE_ACL_INIT 1 @@ -43,6 +44,7 @@ #define HAVE_EXTATTR_LIST_FILE 1 #define HAVE_EXTATTR_SET_FD 1 #define HAVE_EXTATTR_SET_FILE 1 +#define HAVE_STRUCT_XVFSCONF 1 #define HAVE_SYS_ACL_H 1 #define HAVE_SYS_EXTATTR_H 1 #endif
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201609122207.u8CM7ZaV029748>