From owner-svn-src-all@freebsd.org Thu Dec 1 15:39:35 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7BE5C61A3C; Thu, 1 Dec 2016 15:39:35 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B62621EF7; Thu, 1 Dec 2016 15:39:35 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id uB1FdZdh091126; Thu, 1 Dec 2016 15:39:35 GMT (envelope-from mm@FreeBSD.org) Received: (from mm@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id uB1FdXRF091114; Thu, 1 Dec 2016 15:39:33 GMT (envelope-from mm@FreeBSD.org) Message-Id: <201612011539.uB1FdXRF091114@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mm set sender to mm@FreeBSD.org using -f From: Martin Matuska Date: Thu, 1 Dec 2016 15:39:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r309362 - in vendor/libarchive/dist: . libarchive libarchive/test tar tar/test X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Dec 2016 15:39:36 -0000 Author: mm Date: Thu Dec 1 15:39:33 2016 New Revision: 309362 URL: https://svnweb.freebsd.org/changeset/base/309362 Log: Update vendor/libarchive to git 2d2b3e928605f795515b03f060fd638c265b0778 Small improvements, style fixes, bugfixes. Important vendor bugfixes: Restore compatibility with Perl Archive::Tar that was broken with #825 Added: vendor/libarchive/dist/libarchive/test/test_compat_perl_archive_tar.c (contents, props changed) vendor/libarchive/dist/libarchive/test/test_compat_perl_archive_tar.tar.uu Modified: vendor/libarchive/dist/Makefile.am vendor/libarchive/dist/libarchive/archive.h vendor/libarchive/dist/libarchive/archive_read_support_format_mtree.c vendor/libarchive/dist/libarchive/archive_read_support_format_tar.c vendor/libarchive/dist/libarchive/test/CMakeLists.txt vendor/libarchive/dist/tar/creation_set.c vendor/libarchive/dist/tar/test/main.c vendor/libarchive/dist/tar/test/test_copy.c vendor/libarchive/dist/tar/test/test_windows.c vendor/libarchive/dist/tar/write.c Modified: vendor/libarchive/dist/Makefile.am ============================================================================== --- vendor/libarchive/dist/Makefile.am Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/Makefile.am Thu Dec 1 15:39:33 2016 (r309362) @@ -370,6 +370,7 @@ libarchive_test_SOURCES= \ libarchive/test/test_compat_lzop.c \ libarchive/test/test_compat_mac.c \ libarchive/test/test_compat_pax_libarchive_2x.c \ + libarchive/test/test_compat_perl_archive_tar.c \ libarchive/test/test_compat_solaris_tar_acl.c \ libarchive/test/test_compat_solaris_pax_sparse.c \ libarchive/test/test_compat_star_acl_posix1e.c \ @@ -599,6 +600,7 @@ libarchive_test_EXTRA_DIST=\ libarchive/test/test_compat_bzip2_2.tbz.uu \ libarchive/test/test_compat_cpio_1.cpio.uu \ libarchive/test/test_compat_gtar_1.tar.uu \ + libarchive/test/test_compat_gtar_2.tar.uu \ libarchive/test/test_compat_gzip_1.tgz.uu \ libarchive/test/test_compat_gzip_2.tgz.uu \ libarchive/test/test_compat_lz4_1.tar.lz4.uu \ @@ -624,6 +626,7 @@ libarchive_test_EXTRA_DIST=\ libarchive/test/test_compat_mac-1.tar.Z.uu \ libarchive/test/test_compat_mac-2.tar.Z.uu \ libarchive/test/test_compat_pax_libarchive_2x.tar.Z.uu \ + libarchive/test/test_compat_perl_archive_tar.tar.uu \ libarchive/test/test_compat_solaris_pax_sparse_1.pax.Z.uu \ libarchive/test/test_compat_solaris_pax_sparse_2.pax.Z.uu \ libarchive/test/test_compat_solaris_tar_acl.tar.uu \ @@ -769,6 +772,7 @@ libarchive_test_EXTRA_DIST=\ libarchive/test/test_read_format_rar_subblock.rar.uu \ libarchive/test/test_read_format_rar_unicode.rar.uu \ libarchive/test/test_read_format_rar_windows.rar.uu \ + libarchive/test/test_read_format_raw.bufr.uu \ libarchive/test/test_read_format_raw.data.Z.uu \ libarchive/test/test_read_format_raw.data.uu \ libarchive/test/test_read_format_tar_concatenated.tar.uu \ Modified: vendor/libarchive/dist/libarchive/archive.h ============================================================================== --- vendor/libarchive/dist/libarchive/archive.h Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/libarchive/archive.h Thu Dec 1 15:39:33 2016 (r309362) @@ -562,7 +562,7 @@ __LA_DECL la_int64_t archive_read_head * we cannot say whether there are encrypted entries, then * ARCHIVE_READ_FORMAT_ENCRYPTION_DONT_KNOW is returned. * In general, this function will return values below zero when the - * reader is uncertain or totally uncapable of encryption support. + * reader is uncertain or totally incapable of encryption support. * When this function returns 0 you can be sure that the reader * supports encryption detection but no encrypted entries have * been found yet. @@ -984,12 +984,12 @@ __LA_DECL int archive_read_disk_can_desc __LA_DECL int archive_read_disk_current_filesystem(struct archive *); __LA_DECL int archive_read_disk_current_filesystem_is_synthetic(struct archive *); __LA_DECL int archive_read_disk_current_filesystem_is_remote(struct archive *); -/* Request that the access time of the entry visited by travesal be restored. */ +/* Request that the access time of the entry visited by traversal be restored. */ __LA_DECL int archive_read_disk_set_atime_restored(struct archive *); /* * Set behavior. The "flags" argument selects optional behavior. */ -/* Request that the access time of the entry visited by travesal be restored. +/* Request that the access time of the entry visited by traversal be restored. * This is the same as archive_read_disk_set_atime_restored. */ #define ARCHIVE_READDISK_RESTORE_ATIME (0x0001) /* Default: Do not skip an entry which has nodump flags. */ @@ -1124,7 +1124,7 @@ __LA_DECL int archive_match_time_exclude /* * Flags to tell a matching type of time stamps. These are used for - * following functinos. + * following functions. */ /* Time flag: mtime to be tested. */ #define ARCHIVE_MATCH_MTIME (0x0100) @@ -1144,7 +1144,7 @@ __LA_DECL int archive_match_include_date const char *_datestr); __LA_DECL int archive_match_include_date_w(struct archive *, int _flag, const wchar_t *_datestr); -/* Set inclusion time by a particluar file. */ +/* Set inclusion time by a particular file. */ __LA_DECL int archive_match_include_file_time(struct archive *, int _flag, const char *_pathname); __LA_DECL int archive_match_include_file_time_w(struct archive *, Modified: vendor/libarchive/dist/libarchive/archive_read_support_format_mtree.c ============================================================================== --- vendor/libarchive/dist/libarchive/archive_read_support_format_mtree.c Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/libarchive/archive_read_support_format_mtree.c Thu Dec 1 15:39:33 2016 (r309362) @@ -75,6 +75,8 @@ __FBSDID("$FreeBSD: head/lib/libarchive/ #define MTREE_HAS_OPTIONAL 0x0800 #define MTREE_HAS_NOCHANGE 0x1000 /* FreeBSD specific */ +#define MTREE_HASHTABLE_SIZE 1024 + struct mtree_option { struct mtree_option *next; char *value; @@ -86,6 +88,8 @@ struct mtree_entry { char *name; char full; char used; + unsigned int name_hash; + struct mtree_entry *hashtable_next; }; struct mtree { @@ -98,6 +102,7 @@ struct mtree { const char *archive_format_name; struct mtree_entry *entries; struct mtree_entry *this_entry; + struct mtree_entry *entry_hashtable[MTREE_HASHTABLE_SIZE]; struct archive_string current_dir; struct archive_string contents_name; @@ -110,6 +115,7 @@ struct mtree { static int bid_keycmp(const char *, const char *, ssize_t); static int cleanup(struct archive_read *); static int detect_form(struct archive_read *, int *); +static unsigned int hash(const char *); static int mtree_bid(struct archive_read *, int); static int parse_file(struct archive_read *, struct archive_entry *, struct mtree *, struct mtree_entry *, int *); @@ -862,11 +868,12 @@ process_add_entry(struct archive_read *a struct mtree_option **global, const char *line, ssize_t line_len, struct mtree_entry **last_entry, int is_form_d) { - struct mtree_entry *entry; + struct mtree_entry *entry, *ht_iter; struct mtree_option *iter; const char *next, *eq, *name, *end; size_t name_len, len; int r, i; + unsigned int ht_idx; if ((entry = malloc(sizeof(*entry))) == NULL) { archive_set_error(&a->archive, errno, "Can't allocate memory"); @@ -877,6 +884,8 @@ process_add_entry(struct archive_read *a entry->name = NULL; entry->used = 0; entry->full = 0; + entry->name_hash = 0; + entry->hashtable_next = NULL; /* Add this entry to list. */ if (*last_entry == NULL) @@ -929,6 +938,16 @@ process_add_entry(struct archive_read *a memcpy(entry->name, name, name_len); entry->name[name_len] = '\0'; parse_escapes(entry->name, entry); + entry->name_hash = hash(entry->name); + + ht_idx = entry->name_hash % MTREE_HASHTABLE_SIZE; + if ((ht_iter = mtree->entry_hashtable[ht_idx]) != NULL) { + while (ht_iter->hashtable_next) + ht_iter = ht_iter->hashtable_next; + ht_iter->hashtable_next = entry; + } else { + mtree->entry_hashtable[ht_idx] = entry; + } for (iter = *global; iter != NULL; iter = iter->next) { r = add_option(a, &entry->options, iter->value, @@ -1122,9 +1141,10 @@ parse_file(struct archive_read *a, struc * with pathname canonicalization, which is a very * tricky subject.) */ - for (mp = mentry->next; mp != NULL; mp = mp->next) { + for (mp = mentry->hashtable_next; mp != NULL; mp = mp->hashtable_next) { if (mp->full && !mp->used - && strcmp(mentry->name, mp->name) == 0) { + && mentry->name_hash == mp->name_hash + && strcmp(mentry->name, mp->name) == 0) { /* Later lines override earlier ones. */ mp->used = 1; r1 = parse_line(a, entry, mtree, mp, @@ -2000,3 +2020,19 @@ readline(struct archive_read *a, struct find_off = u - mtree->line.s; } } + +static unsigned int +hash(const char *p) +{ + /* A 32-bit version of Peter Weinberger's (PJW) hash algorithm, + as used by ELF for hashing function names. */ + unsigned g, h = 0; + while (*p != '\0') { + h = (h << 4) + *p++; + if ((g = h & 0xF0000000) != 0) { + h ^= g >> 24; + h &= 0x0FFFFFFF; + } + } + return h; +} Modified: vendor/libarchive/dist/libarchive/archive_read_support_format_tar.c ============================================================================== --- vendor/libarchive/dist/libarchive/archive_read_support_format_tar.c Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/libarchive/archive_read_support_format_tar.c Thu Dec 1 15:39:33 2016 (r309362) @@ -294,8 +294,14 @@ archive_read_format_tar_cleanup(struct a return (ARCHIVE_OK); } +/* + * Validate number field + * + * Flags: + * 1 - allow double \0 at field end + */ static int -validate_number_field(const char* p_field, size_t i_size) +validate_number_field(const char* p_field, size_t i_size, int flags) { unsigned char marker = (unsigned char)p_field[0]; /* octal? */ @@ -305,14 +311,24 @@ validate_number_field(const char* p_fiel for (i = 0; i < i_size; ++i) { switch (p_field[i]) { - case ' ': /* skip any leading spaces and trailing space*/ + case ' ': + /* skip any leading spaces and trailing space */ if (octal_found == 0 || i == i_size - 1) { continue; } break; - case '\0': /* null is allowed only at the end */ + case '\0': + /* + * null should be allowed only at the end + * + * Perl Archive::Tar terminates some fields + * with two nulls. We must allow this to stay + * compatible. + */ if (i != i_size - 1) { - return 0; + if (((flags & 1) == 0) + || i != i_size - 2) + return 0; } break; /* rest must be octal digits */ @@ -390,18 +406,25 @@ archive_read_format_tar_bid(struct archi * Check format of mode/uid/gid/mtime/size/rdevmajor/rdevminor fields. * These are usually octal numbers but GNU tar encodes "big" values as * base256 and leading zeroes are sometimes replaced by spaces. - * Even the null terminator is sometimes omitted. Anyway, must be checked - * to avoid false positives. + * Even the null terminator is sometimes omitted. Anyway, must be + * checked to avoid false positives. + * + * Perl Archive::Tar does not follow the spec and terminates mode, uid, + * gid, rdevmajor and rdevminor with a double \0. For compatibility + * reasons we allow this deviation. */ - if (bid > 0 && - (validate_number_field(header->mode, sizeof(header->mode)) == 0 || - validate_number_field(header->uid, sizeof(header->uid)) == 0 || - validate_number_field(header->gid, sizeof(header->gid)) == 0 || - validate_number_field(header->mtime, sizeof(header->mtime)) == 0 || - validate_number_field(header->size, sizeof(header->size)) == 0 || - validate_number_field(header->rdevmajor, sizeof(header->rdevmajor)) == 0 || - validate_number_field(header->rdevminor, sizeof(header->rdevminor)) == 0)) { - bid = 0; + if (bid > 0 && ( + validate_number_field(header->mode, sizeof(header->mode), 1) == 0 + || validate_number_field(header->uid, sizeof(header->uid), 1) == 0 + || validate_number_field(header->gid, sizeof(header->gid), 1) == 0 + || validate_number_field(header->mtime, sizeof(header->mtime), + 0) == 0 + || validate_number_field(header->size, sizeof(header->size), 0) == 0 + || validate_number_field(header->rdevmajor, + sizeof(header->rdevmajor), 1) == 0 + || validate_number_field(header->rdevminor, + sizeof(header->rdevminor), 1) == 0)) { + bid = 0; } return (bid); Modified: vendor/libarchive/dist/libarchive/test/CMakeLists.txt ============================================================================== --- vendor/libarchive/dist/libarchive/test/CMakeLists.txt Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/libarchive/test/CMakeLists.txt Thu Dec 1 15:39:33 2016 (r309362) @@ -58,6 +58,7 @@ IF(ENABLE_TEST) test_compat_lzop.c test_compat_mac.c test_compat_pax_libarchive_2x.c + test_compat_perl_archive_tar.c test_compat_solaris_pax_sparse.c test_compat_solaris_tar_acl.c test_compat_star_acl_posix1e.c Added: vendor/libarchive/dist/libarchive/test/test_compat_perl_archive_tar.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libarchive/dist/libarchive/test/test_compat_perl_archive_tar.c Thu Dec 1 15:39:33 2016 (r309362) @@ -0,0 +1,66 @@ +/*- + * Copyright (c) 2016 Martin Matuska + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + * IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ +#include "test.h" +__FBSDID("$FreeBSD"); + +/* + * Verify our ability to read sample files created by Perl module Archive::Tar + */ + +DEFINE_TEST(test_compat_perl_archive_tar) +{ + char name[] = "test_compat_perl_archive_tar.tar"; + struct archive_entry *ae; + struct archive *a; + int r; + + assert((a = archive_read_new()) != NULL); + assertEqualIntA(a, ARCHIVE_OK, archive_read_support_filter_all(a)); + assertEqualIntA(a, ARCHIVE_OK, archive_read_support_format_all(a)); + extract_reference_file(name); + assertEqualIntA(a, ARCHIVE_OK, archive_read_open_filename(a, name, + 10240)); + + /* Read first entry. */ + assertEqualIntA(a, ARCHIVE_OK, r = archive_read_next_header(a, &ae)); + if (r != ARCHIVE_OK) { + archive_read_free(a); + return; + } + assertEqualString("file1", archive_entry_pathname(ae)); + assertEqualInt(1480603099, archive_entry_mtime(ae)); + assertEqualInt(1000, archive_entry_uid(ae)); + assertEqualString("john", archive_entry_uname(ae)); + assertEqualInt(1000, archive_entry_gid(ae)); + assertEqualString("john", archive_entry_gname(ae)); + assertEqualInt(0100644, archive_entry_mode(ae)); + + /* Verify that the format detection worked. */ + assertEqualInt(archive_filter_code(a, 0), ARCHIVE_FILTER_NONE); + assertEqualInt(archive_format(a), ARCHIVE_FORMAT_TAR_USTAR); + + assertEqualInt(ARCHIVE_OK, archive_read_close(a)); + assertEqualInt(ARCHIVE_OK, archive_read_free(a)); +} Added: vendor/libarchive/dist/libarchive/test/test_compat_perl_archive_tar.tar.uu ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libarchive/dist/libarchive/test/test_compat_perl_archive_tar.tar.uu Thu Dec 1 15:39:33 2016 (r309362) @@ -0,0 +1,49 @@ +begin 644 test_compat_perl_archive_tar.tar +M9FEL93$````````````````````````````````````````````````````` +M```````````````````````````````````````````````````````````` +M`````````````#`P,#8T-```,#`Q-S4P```P,#$W-3```"`@("`@("`@("`U +M`#$S,#(P,#,R-S,S`"`Q,3$R,P`@,``````````````````````````````` +M```````````````````````````````````````````````````````````` +M``````````````````````````````````````````!Ufilter_count); @@ -308,7 +308,7 @@ cset_auto_compress(struct creation_set * cset->filters = v; return (1); } else { - /* Put previos filters back. */ + /* Put previous filters back. */ cset->filters = old_filters; cset->filter_count = old_filter_count; return (0); Modified: vendor/libarchive/dist/tar/test/main.c ============================================================================== --- vendor/libarchive/dist/tar/test/main.c Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/tar/test/main.c Thu Dec 1 15:39:33 2016 (r309362) @@ -521,7 +521,7 @@ _utf8_to_unicode(uint32_t *pwc, const ch return (0); /* Standard: return 0 for end-of-string. */ cnt = utf8_count[ch]; - /* Invalide sequence or there are not plenty bytes. */ + /* Invalid sequence or there are not plenty bytes. */ if (n < (size_t)cnt) return (-1); @@ -560,7 +560,7 @@ _utf8_to_unicode(uint32_t *pwc, const ch return (-1); } - /* The code point larger than 0x10FFFF is not leagal + /* The code point larger than 0x10FFFF is not legal * Unicode values. */ if (wc > 0x10FFFF) return (-1); Modified: vendor/libarchive/dist/tar/test/test_copy.c ============================================================================== --- vendor/libarchive/dist/tar/test/test_copy.c Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/tar/test/test_copy.c Thu Dec 1 15:39:33 2016 (r309362) @@ -285,7 +285,7 @@ copy_basic(void) /* NOTE: for proper operation on cygwin-1.5 and windows, the * length of the name of the directory below, "plain", must be - * less than or equal to the lengthe of the name of the original + * less than or equal to the length of the name of the original * directory, "original" This restriction derives from the * extremely limited pathname lengths on those platforms. */ @@ -327,7 +327,7 @@ copy_ustar(void) /* NOTE: for proper operation on cygwin-1.5 and windows, the * length of the name of the directory below, "ustar", must be - * less than or equal to the lengthe of the name of the original + * less than or equal to the length of the name of the original * directory, "original" This restriction derives from the * extremely limited pathname lengths on those platforms. */ Modified: vendor/libarchive/dist/tar/test/test_windows.c ============================================================================== --- vendor/libarchive/dist/tar/test/test_windows.c Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/tar/test/test_windows.c Thu Dec 1 15:39:33 2016 (r309362) @@ -133,7 +133,7 @@ DEFINE_TEST(test_windows) char *fp1, *fp2; /* - * Preparre tests. + * Prepare tests. * Create directories and files. */ assertMakeDir("tmp", 0775); Modified: vendor/libarchive/dist/tar/write.c ============================================================================== --- vendor/libarchive/dist/tar/write.c Thu Dec 1 14:31:05 2016 (r309361) +++ vendor/libarchive/dist/tar/write.c Thu Dec 1 15:39:33 2016 (r309362) @@ -527,7 +527,7 @@ write_archive(struct archive *a, struct struct archive *disk = bsdtar->diskreader; /* - * This tricky code here is to correctly read the cotents + * This tricky code here is to correctly read the contents * of the entry because the disk reader bsdtar->diskreader * is pointing at does not have any information about the * entry by this time and using archive_read_data_block()