From owner-svn-src-stable@freebsd.org Wed Aug 15 02:14:50 2018 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6BABE106E6D9; Wed, 15 Aug 2018 02:14:50 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C7AA72E2D; Wed, 15 Aug 2018 02:14:50 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E62F123F7A; Wed, 15 Aug 2018 02:14:49 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w7F2Enr6089946; Wed, 15 Aug 2018 02:14:49 GMT (envelope-from eadler@FreeBSD.org) Received: (from eadler@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w7F2ElCX089932; Wed, 15 Aug 2018 02:14:47 GMT (envelope-from eadler@FreeBSD.org) Message-Id: <201808150214.w7F2ElCX089932@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: eadler set sender to eadler@FreeBSD.org using -f From: Eitan Adler Date: Wed, 15 Aug 2018 02:14:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r337827 - in stable/11: contrib/file contrib/file/doc contrib/file/magic contrib/file/magic/Magdir contrib/file/python contrib/file/src contrib/file/tests lib/libmagic X-SVN-Group: stable-11 X-SVN-Commit-Author: eadler X-SVN-Commit-Paths: in stable/11: contrib/file contrib/file/doc contrib/file/magic contrib/file/magic/Magdir contrib/file/python contrib/file/src contrib/file/tests lib/libmagic X-SVN-Commit-Revision: 337827 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Aug 2018 02:14:50 -0000 Author: eadler Date: Wed Aug 15 02:14:46 2018 New Revision: 337827 URL: https://svnweb.freebsd.org/changeset/base/337827 Log: MFC r333919, r333922, r333944, r337442: Update file to file 5.34 (through 5.33) Added: stable/11/contrib/file/magic/Magdir/beetle - copied unchanged from r333919, head/contrib/file/magic/Magdir/beetle stable/11/contrib/file/magic/Magdir/dataone - copied unchanged from r337442, head/contrib/file/magic/Magdir/dataone stable/11/contrib/file/magic/Magdir/dbpf - copied, changed from r333919, head/contrib/file/magic/Magdir/dbpf stable/11/contrib/file/magic/Magdir/measure - copied, changed from r333919, head/contrib/file/magic/Magdir/measure stable/11/contrib/file/magic/Magdir/rpi - copied unchanged from r333919, head/contrib/file/magic/Magdir/rpi stable/11/contrib/file/magic/Magdir/tplink - copied unchanged from r333919, head/contrib/file/magic/Magdir/tplink stable/11/contrib/file/magic/Magdir/zip - copied unchanged from r333919, head/contrib/file/magic/Magdir/zip stable/11/contrib/file/python/CHANGELOG.md - copied unchanged from r337442, head/contrib/file/python/CHANGELOG.md stable/11/contrib/file/python/LICENSE - copied unchanged from r337442, head/contrib/file/python/LICENSE stable/11/contrib/file/python/README.md - copied unchanged from r337442, head/contrib/file/python/README.md stable/11/contrib/file/python/tests.py - copied unchanged from r337442, head/contrib/file/python/tests.py stable/11/contrib/file/src/buffer.c - copied unchanged from r333919, head/contrib/file/src/buffer.c stable/11/contrib/file/src/seccomp.c - copied, changed from r333919, head/contrib/file/src/seccomp.c stable/11/contrib/file/tests/JW07022A.mp3.result - copied unchanged from r337442, head/contrib/file/tests/JW07022A.mp3.result stable/11/contrib/file/tests/JW07022A.mp3.testfile - copied unchanged from r337442, head/contrib/file/tests/JW07022A.mp3.testfile Deleted: stable/11/contrib/file/python/README Modified: stable/11/contrib/file/ChangeLog stable/11/contrib/file/README stable/11/contrib/file/TODO stable/11/contrib/file/config.h.in stable/11/contrib/file/configure stable/11/contrib/file/configure.ac stable/11/contrib/file/doc/file.man stable/11/contrib/file/doc/magic.man stable/11/contrib/file/magic/Magdir/acorn stable/11/contrib/file/magic/Magdir/animation stable/11/contrib/file/magic/Magdir/apple stable/11/contrib/file/magic/Magdir/archive stable/11/contrib/file/magic/Magdir/audio stable/11/contrib/file/magic/Magdir/c64 stable/11/contrib/file/magic/Magdir/cad stable/11/contrib/file/magic/Magdir/compress stable/11/contrib/file/magic/Magdir/console stable/11/contrib/file/magic/Magdir/database stable/11/contrib/file/magic/Magdir/dump stable/11/contrib/file/magic/Magdir/elf stable/11/contrib/file/magic/Magdir/filesystems stable/11/contrib/file/magic/Magdir/fonts stable/11/contrib/file/magic/Magdir/fsav stable/11/contrib/file/magic/Magdir/games stable/11/contrib/file/magic/Magdir/geo stable/11/contrib/file/magic/Magdir/gnu stable/11/contrib/file/magic/Magdir/images stable/11/contrib/file/magic/Magdir/intel stable/11/contrib/file/magic/Magdir/linux stable/11/contrib/file/magic/Magdir/macintosh stable/11/contrib/file/magic/Magdir/make stable/11/contrib/file/magic/Magdir/mozilla stable/11/contrib/file/magic/Magdir/msdos stable/11/contrib/file/magic/Magdir/msooxml stable/11/contrib/file/magic/Magdir/netbsd stable/11/contrib/file/magic/Magdir/ole2compounddocs stable/11/contrib/file/magic/Magdir/pdf stable/11/contrib/file/magic/Magdir/pgp stable/11/contrib/file/magic/Magdir/revision stable/11/contrib/file/magic/Magdir/riff stable/11/contrib/file/magic/Magdir/sgi stable/11/contrib/file/magic/Magdir/sgml stable/11/contrib/file/magic/Magdir/spectrum stable/11/contrib/file/magic/Magdir/ssl stable/11/contrib/file/magic/Magdir/terminfo stable/11/contrib/file/magic/Magdir/vorbis stable/11/contrib/file/magic/Magdir/windows stable/11/contrib/file/magic/Makefile.am stable/11/contrib/file/magic/Makefile.in stable/11/contrib/file/python/Makefile.am stable/11/contrib/file/python/Makefile.in stable/11/contrib/file/python/magic.py stable/11/contrib/file/python/setup.py stable/11/contrib/file/src/Makefile.am stable/11/contrib/file/src/Makefile.in stable/11/contrib/file/src/apprentice.c stable/11/contrib/file/src/ascmagic.c stable/11/contrib/file/src/cdf.c stable/11/contrib/file/src/compress.c stable/11/contrib/file/src/der.c stable/11/contrib/file/src/encoding.c stable/11/contrib/file/src/file.c stable/11/contrib/file/src/file.h stable/11/contrib/file/src/file_opts.h stable/11/contrib/file/src/funcs.c stable/11/contrib/file/src/is_tar.c stable/11/contrib/file/src/readcdf.c stable/11/contrib/file/src/readelf.c stable/11/contrib/file/src/readelf.h stable/11/contrib/file/src/softmagic.c stable/11/contrib/file/tests/Makefile.am stable/11/contrib/file/tests/Makefile.in stable/11/lib/libmagic/Makefile Directory Properties: stable/11/ (props changed) Modified: stable/11/contrib/file/ChangeLog ============================================================================== --- stable/11/contrib/file/ChangeLog Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/ChangeLog Wed Aug 15 02:14:46 2018 (r337827) @@ -1,3 +1,40 @@ +2018-07-25 8:50 Christos Zoulas + + * release 5.34 + +2018-06-22 16:38 Christos Zoulas + + * Add Quad indirect offsets + +2018-05-24 14:10 Christos Zoulas + + * Enable parsing of ELF dynamic sections to handle PIE better + +2018-04-15 14:52 Christos Zoulas + + * release 5.33 + +2018-02-24 14:50 Christos Zoulas + + * extend the support for ${x?:} expansions for magic descriptions + +2018-02-21 16:25 Christos Zoulas + + * add support for ${x?:} in mime types to handle + pie binaries. + +2017-11-03 9:23 Christos Zoulas + + * add support for negative offsets (offsets from the end of file) + +2017-09-26 8:22 Christos Zoulas + + * close the file on error when writing magic (Steve Grubb) + +2017-09-24 12:02 Christos Zoulas + + * seccomp support (Paul Moore) + 2017-09-02 11:53 Christos Zoulas * release 5.32 Modified: stable/11/contrib/file/README ============================================================================== --- stable/11/contrib/file/README Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/README Wed Aug 15 02:14:46 2018 (r337827) @@ -1,14 +1,13 @@ ## README for file(1) Command ## - @(#) $File: README,v 1.50 2016/04/16 22:40:54 christos Exp $ + @(#) $File: README,v 1.54 2018/05/30 03:06:56 christos Exp $ -Mailing List: file@mx.gw.com -Mailing List archives: http://mx.gw.com/pipermail/file/ -Bug tracker: http://bugs.gw.com/ +Mailing List: file@astron.com +Mailing List archives: http://mailman.astron.com/pipermail/file/ +Bug tracker: http://bugs.astron.com/ E-mail: christos@astron.com +Build Status: https://travis-ci.org/file/file -[![Build Status](https://travis-ci.org/file/file.png?branch=master)](https://travis-ci.org/file/file) - Phone: Do not even think of telephoning me about this program. Send cash first! This is Release 5.x of Ian Darwin's (copyright but distributable) @@ -67,28 +66,6 @@ in magic(5) format please, to the maintainer, Christos COPYING - read this first. README - read this second (you are currently reading this file). INSTALL - read on how to install -src/localtime_r.c -src/magic.c -src/magic.h -src/mygetopt.h -src/newtest2.c -src/newtest3.c -src/pread.c -src/print.c -src/readcdf.c -src/readelf.c -src/readelf.h -src/regex.c -src/regex2.c -src/softmagic.c -src/strcasestr.c -src/strlcat.c -src/strlcpy.c -src/strndup.c -src/tar.h -src/teststrchr.c -src/vasprintf.c -src/x.c src/apprentice.c - parses /etc/magic to learn magic src/apptype.c - used for OS/2 specific application type magic src/ascmagic.c - third & last set of tests, based on hardwired assumptions. @@ -96,6 +73,7 @@ src/asctime_r.c - replacement for OS's that don't have src/asprintf.c - replacement for OS's that don't have it. src/asctime_r.c - replacement for OS's that don't have it. src/asprintf.c - replacement for OS's that don't have it. +src/buffer.c - buffer handling functions. src/cdf.[ch] - parser for Microsoft Compound Document Files src/cdf_time.c - time converter for CDF. src/compress.c - handles decompressing files to look inside. @@ -128,6 +106,7 @@ src/mygetopt.h - replacement for OS's that don't have src/strcasestr.c - replacement for OS's that don't have it. src/strlcat.c - replacement for OS's that don't have it. src/strlcpy.c - replacement for OS's that don't have it. +src/strndup.c - replacement for OS's that don't have it. src/tar.h - tar file definitions src/vasprintf.c - for systems that don't have it. doc/file.man - man page for the command @@ -155,6 +134,19 @@ guidelines: * Further reference, such as documentation of format ------------------------------------------------------------------------------ + +gpg for dummies: + +$ gpg --verify file-X.YY.tar.gz.asc file-X.YY.tar.gz +gpg: assuming signed data in `file-X.YY.tar.gz' +gpg: Signature made WWW MMM DD HH:MM:SS YYYY ZZZ using DSA key ID KKKKKKKK + +To download the key: + +$ gpg --keyserver hkp://keys.gnupg.net --recv-keys KKKKKKKK + +------------------------------------------------------------------------------ + Parts of this software were developed at SoftQuad Inc., developers of SGML/HTML/XML publishing software, in Toronto, Canada. Modified: stable/11/contrib/file/TODO ============================================================================== --- stable/11/contrib/file/TODO Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/TODO Wed Aug 15 02:14:46 2018 (r337827) @@ -31,6 +31,19 @@ could require structural changes to the matching code # rule 2 >0 .... ... +--- +- Merge the stat code dance in one place and keep it in one place + (perhaps struct buffer). +- Enable seeking around if offset > nbytes if possible (the fd + is seekable). +- We could use file_pipe2file more (for EOF offsets, CDF documents), + but that is expensive; perhaps we should provide a way to disable it +- The implementation of struct buffer needs re-thinking and more work. + For example we don't always pass the fd in the child. This is not + important yet as we don't have yet cases where use/indirect magic + needs negative offsets. +- Really the whole thing just needs here's an (offset, buffer, size) + you have (filebuffer, filebuffersize &&|| fd), fill the buffer with + data from offset. The buffer API should be changed to just do that. christos - Modified: stable/11/contrib/file/config.h.in ============================================================================== --- stable/11/contrib/file/config.h.in Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/config.h.in Wed Aug 15 02:14:46 2018 (r337827) @@ -77,6 +77,9 @@ /* Define to 1 if you have the `gnurx' library (-lgnurx). */ #undef HAVE_LIBGNURX +/* Define to 1 if you have the `seccomp' library (-lseccomp). */ +#undef HAVE_LIBSECCOMP + /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ Modified: stable/11/contrib/file/configure ============================================================================== --- stable/11/contrib/file/configure Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/configure Wed Aug 15 02:14:46 2018 (r337827) @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for file 5.32. +# Generated by GNU Autoconf 2.69 for file 5.34. # # Report bugs to . # @@ -590,8 +590,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='file' PACKAGE_TARNAME='file' -PACKAGE_VERSION='5.32' -PACKAGE_STRING='file 5.32' +PACKAGE_VERSION='5.34' +PACKAGE_STRING='file 5.34' PACKAGE_BUGREPORT='christos@astron.com' PACKAGE_URL='' @@ -767,6 +767,7 @@ enable_silent_rules enable_elf enable_elf_core enable_zlib +enable_libseccomp enable_fsect_man5 enable_dependency_tracking enable_static @@ -1328,7 +1329,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures file 5.32 to adapt to many kinds of systems. +\`configure' configures file 5.34 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1398,7 +1399,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of file 5.32:";; + short | recursive ) echo "Configuration of file 5.34:";; esac cat <<\_ACEOF @@ -1411,6 +1412,7 @@ Optional Features: --disable-elf disable builtin ELF support --disable-elf-core disable ELF core file support --disable-zlib disable zlib compression support [default=auto] + --disable-libseccomp disable libseccomp sandboxing [default=auto] --enable-fsect-man5 enable file formats in man section 5 --enable-dependency-tracking do not reject slow dependency extractors @@ -1509,7 +1511,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -file configure 5.32 +file configure 5.34 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -2165,7 +2167,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by file $as_me 5.32, which was +It was created by file $as_me 5.34, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -3031,7 +3033,7 @@ fi # Define the identity of the package. PACKAGE='file' - VERSION='5.32' + VERSION='5.34' cat >>confdefs.h <<_ACEOF @@ -3230,6 +3232,16 @@ fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_zlib" >&5 $as_echo "$enable_zlib" >&6; } +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for libseccomp support" >&5 +$as_echo_n "checking for libseccomp support... " >&6; } +# Check whether --enable-libseccomp was given. +if test "${enable_libseccomp+set}" = set; then : + enableval=$enable_libseccomp; +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_libseccomp" >&5 +$as_echo "$enable_libseccomp" >&6; } + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for file formats in man section 5" >&5 $as_echo_n "checking for file formats in man section 5... " >&6; } # Check whether --enable-fsect-man5 was given. @@ -14461,6 +14473,53 @@ _ACEOF fi fi +if test "$enable_libseccomp" != "no"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for seccomp_init in -lseccomp" >&5 +$as_echo_n "checking for seccomp_init in -lseccomp... " >&6; } +if ${ac_cv_lib_seccomp_seccomp_init+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lseccomp $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char seccomp_init (); +int +main () +{ +return seccomp_init (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_seccomp_seccomp_init=yes +else + ac_cv_lib_seccomp_seccomp_init=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_seccomp_seccomp_init" >&5 +$as_echo "$ac_cv_lib_seccomp_seccomp_init" >&6; } +if test "x$ac_cv_lib_seccomp_seccomp_init" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSECCOMP 1 +_ACEOF + + LIBS="-lseccomp $LIBS" + +fi + +fi if test "$MINGW" = 1; then { $as_echo "$as_me:${as_lineno-$LINENO}: checking for regexec in -lgnurx" >&5 $as_echo_n "checking for regexec in -lgnurx... " >&6; } @@ -15075,7 +15134,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by file $as_me 5.32, which was +This file was extended by file $as_me 5.34, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -15141,7 +15200,7 @@ _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -file config.status 5.32 +file config.status 5.34 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" Modified: stable/11/contrib/file/configure.ac ============================================================================== --- stable/11/contrib/file/configure.ac Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/configure.ac Wed Aug 15 02:14:46 2018 (r337827) @@ -1,5 +1,5 @@ dnl Process this file with autoconf to produce a configure script. -AC_INIT([file],[5.32],[christos@astron.com]) +AC_INIT([file],[5.34],[christos@astron.com]) AM_INIT_AUTOMAKE([subdir-objects foreign]) m4_ifdef([AM_SILENT_RULES], [AM_SILENT_RULES([yes])]) @@ -39,6 +39,11 @@ AC_ARG_ENABLE(zlib, [AS_HELP_STRING([--disable-zlib], [disable zlib compression support @<:@default=auto@:>@])]) AC_MSG_RESULT($enable_zlib) +AC_MSG_CHECKING(for libseccomp support) +AC_ARG_ENABLE(libseccomp, +[AS_HELP_STRING([--disable-libseccomp], [disable libseccomp sandboxing @<:@default=auto@:>@])]) +AC_MSG_RESULT($enable_libseccomp) + AC_MSG_CHECKING(for file formats in man section 5) AC_ARG_ENABLE(fsect-man5, [ --enable-fsect-man5 enable file formats in man section 5], @@ -157,6 +162,9 @@ AC_REPLACE_FUNCS(getopt_long asprintf vasprintf strlcp dnl Checks for libraries if test "$enable_zlib" != "no"; then AC_CHECK_LIB(z, gzopen) +fi +if test "$enable_libseccomp" != "no"; then + AC_CHECK_LIB(seccomp, seccomp_init) fi if test "$MINGW" = 1; then AC_CHECK_LIB(gnurx,regexec,,AC_MSG_ERROR([libgnurx is required to build file(1) with MinGW])) Modified: stable/11/contrib/file/doc/file.man ============================================================================== --- stable/11/contrib/file/doc/file.man Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/doc/file.man Wed Aug 15 02:14:46 2018 (r337827) @@ -1,5 +1,5 @@ -.\" $File: file.man,v 1.125 2017/01/03 11:24:46 christos Exp $ -.Dd October 19, 2016 +.\" $File: file.man,v 1.131 2018/07/24 21:33:56 christos Exp $ +.Dd July 25, 2018 .Dt FILE __CSECTION__ .Os .Sh NAME @@ -8,7 +8,7 @@ .Sh SYNOPSIS .Nm .Bk -words -.Op Fl bcdEhiklLNnprsvzZ0 +.Op Fl bcdEhiklLNnprsSvzZ0 .Op Fl Fl apple .Op Fl Fl extension .Op Fl Fl mime-encoding @@ -168,7 +168,8 @@ in any of the character sets listed above is simply sa .Bl -tag -width indent .It Fl Fl apple Causes the file command to output the file type and creator code as -used by older MacOS versions. The code consists of eight letters, +used by older MacOS versions. +The code consists of eight letters, the first describing the file type, the latter the creator. .It Fl b , Fl Fl brief Do not prepend filenames to output lines (brief mode). @@ -215,12 +216,14 @@ elf magic is found. .It soft Consults magic files. .It tar -Examines tar files. +Examines tar files by verifying the checksum of the 512 byte tar header. +Excluding this test can provide more detailed content description by using +the soft magic method. .It text A synonym for .Sq ascii . .El -.It Fl Fl extension +.It Fl Fl extension Print a slash-separated list of valid extensions for the file type found. .It Fl F , Fl Fl separator Ar separator Use the specified string as the separator between the filename and the @@ -350,6 +353,16 @@ This option also causes to disregard the file size as reported by .Xr stat 2 since on some systems it reports a zero size for raw disk partitions. +.It Fl S , Fl Fl no-sandbox +On systems where libseccomp +.Pa ( https://github.com/seccomp/libseccomp ) +is available, the +.Fl S +flag disables sandboxing which is enabled by default. +This option is needed for file to execute external descompressing programs, +i.e. when the +.Fl z +flag is specified and the built-in decompressors are not available. .It Fl v , Fl Fl version Print the version of the program and exit. .It Fl z , Fl Fl uncompress @@ -366,20 +379,13 @@ Nice to the output. This does not affect the separator, which is still printed. .Pp -If this option is repeated more than once, then +If this option is repeated more than once, then .Nm prints just the filename followed by a NUL followed by the description (or ERROR: text) followed by a second NUL for each entry. .It Fl -help Print a help message and exit. .El -.Sh FILES -.Bl -tag -width __MAGIC__.mgc -compact -.It Pa __MAGIC__.mgc -Default compiled list of magic. -.It Pa __MAGIC__ -Directory containing default magic files. -.El .Sh ENVIRONMENT The environment variable .Ev MAGIC @@ -392,11 +398,6 @@ will not attempt to open adds .Dq Pa .mgc to the value of this variable as appropriate. -However, -.Pa file -has to exist in order for -.Pa file.mime -to be considered. The environment variable .Ev POSIXLY_CORRECT controls (on systems that support symbolic links), whether @@ -410,6 +411,65 @@ This is also controlled by the and .Fl h options. +.Sh FILES +.Bl -tag -width __MAGIC__.mgc -compact +.It Pa __MAGIC__.mgc +Default compiled list of magic. +.It Pa __MAGIC__ +Directory containing default magic files. +.El +.Sh EXIT STATUS +.Nm +will exit with +.Dv 0 +if the operation was successful or +.Dv >0 +if an error was encountered. +The following errors cause diagnostic messages, but don't affect the program +exit code (as POSIX requires), unless +.Fl E +is specified: +.Bl -bullet -compact -offset indent +.It +A file cannot be found +.It +There is no permission to read a file +.It +The file type cannot be determined +.El +.Sh EXAMPLES +.Bd -literal -offset indent +$ file file.c file /dev/{wd0a,hda} +file.c: C program text +file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), + dynamically linked (uses shared libs), stripped +/dev/wd0a: block special (0/0) +/dev/hda: block special (3/0) + +$ file -s /dev/wd0{b,d} +/dev/wd0b: data +/dev/wd0d: x86 boot sector + +$ file -s /dev/hda{,1,2,3,4,5,6,7,8,9,10} +/dev/hda: x86 boot sector +/dev/hda1: Linux/i386 ext2 filesystem +/dev/hda2: x86 boot sector +/dev/hda3: x86 boot sector, extended partition table +/dev/hda4: Linux/i386 ext2 filesystem +/dev/hda5: Linux/i386 swap file +/dev/hda6: Linux/i386 swap file +/dev/hda7: Linux/i386 swap file +/dev/hda8: Linux/i386 swap file +/dev/hda9: empty +/dev/hda10: empty + +$ file -i file.c file /dev/{wd0a,hda} +file.c: text/x-c +file: application/x-executable +/dev/hda: application/x-not-regular-file +/dev/wd0a: application/x-not-regular-file + +.Ed .Sh SEE ALSO .Xr hexdump 1 , .Xr od 1 , @@ -462,6 +522,23 @@ for example, .Bd -literal -offset indent \*[Gt]16 long\*[Am]0x7fffffff \*[Gt]0 not stripped .Ed +.Sh SECURITY +On systems where libseccomp +.Pa ( https://github.com/seccomp/libseccomp ) +is available, +.Nm +is enforces limiting system calls to only the ones necessary for the +operation of the program. +This enforcement does not provide any security benefit when +.Nm +is asked to decompress input files running external programs with +the +.Fl z +option. +To enable execution of external decompressors, one needs to disable +sandboxing using the +.Fl S +flag. .Sh MAGIC DIRECTORY The magic file entries have been collected from various sources, mainly USENET, and contributed by various authors. @@ -479,39 +556,6 @@ command uses a magic file, keep the old magic file around for comparison purposes (rename it to .Pa __MAGIC__.orig ) . -.Sh EXAMPLES -.Bd -literal -offset indent -$ file file.c file /dev/{wd0a,hda} -file.c: C program text -file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), - dynamically linked (uses shared libs), stripped -/dev/wd0a: block special (0/0) -/dev/hda: block special (3/0) - -$ file -s /dev/wd0{b,d} -/dev/wd0b: data -/dev/wd0d: x86 boot sector - -$ file -s /dev/hda{,1,2,3,4,5,6,7,8,9,10} -/dev/hda: x86 boot sector -/dev/hda1: Linux/i386 ext2 filesystem -/dev/hda2: x86 boot sector -/dev/hda3: x86 boot sector, extended partition table -/dev/hda4: Linux/i386 ext2 filesystem -/dev/hda5: Linux/i386 swap file -/dev/hda6: Linux/i386 swap file -/dev/hda7: Linux/i386 swap file -/dev/hda8: Linux/i386 swap file -/dev/hda9: empty -/dev/hda10: empty - -$ file -i file.c file /dev/{wd0a,hda} -file.c: text/x-c -file: application/x-executable -/dev/hda: application/x-not-regular-file -/dev/wd0a: application/x-not-regular-file - -.Ed .Sh HISTORY There has been a .Nm @@ -533,7 +577,7 @@ Geoff Collyer found several inadequacies and provided some magic file entries. Contributions of the .Sq \*[Am] -operator by Rob McMahon, +operator by Rob McMahon, .Aq cudcv@warwick.ac.uk , 1989. .Pp @@ -583,16 +627,13 @@ and were written by John Gilmore from his public-domain .Xr tar 1 program, and are not covered by the above license. -.Sh RETURN CODE -.Nm -returns 0 on success, and non-zero on error. .Sh BUGS Please report bugs and send patches to the bug tracker at -.Pa http://bugs.gw.com/ +.Pa http://bugs.astron.com/ or the mailing list at -.Aq file@mx.gw.com +.Aq file@astron.com (visit -.Pa http://mx.gw.com/mailman/listinfo/file +.Pa http://mailman.astron.com/mailman/listinfo/file first to subscribe). .Sh TODO Fix output so that tests for MIME and APPLE flags are not needed all Modified: stable/11/contrib/file/doc/magic.man ============================================================================== --- stable/11/contrib/file/doc/magic.man Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/doc/magic.man Wed Aug 15 02:14:46 2018 (r337827) @@ -1,5 +1,6 @@ -.\" $File: magic.man,v 1.91 2017/02/12 15:30:08 christos Exp $ -.Dd February 12, 2017 +.It S2 +.\" $File: magic.man,v 1.93 2018/06/22 20:39:49 christos Exp $ +.Dd June 22, 2018 .Dt MAGIC __FSECTION__ .Os .\" install as magic.4 on USG, magic.5 on V7, Berkeley and Linux systems. @@ -36,8 +37,20 @@ If the test succeeds, a message is printed. The line consists of the following fields: .Bl -tag -width ".Dv message" .It Dv offset -A number specifying the offset, in bytes, into the file of the data +A number specifying the offset (in bytes) into the file of the data which is to be tested. +This offset can be a negative number if it is: +.Bl -bullet -compact +.It +The first direct offset of the magic entry (at continuation level 0), +in which case it is interpreted an offset from end end of the file +going backwards. +This works only when a file descriptor to the file is a available and it +is a regular file. +.It +A continuation offset relative to the end of the last up-level field +.Dv ( \*[Am] ) . +.El .It Dv type The type of the data to be tested. The possible values are: @@ -541,12 +554,12 @@ the file. The value at that offset is read, and is used again as an offset in the file. Indirect offsets are of the form: -.Em (( x [[.,][bislBISL]][+\-][ y ]) . +.Em (( x [[.,][bBcCeEfFgGhHiIlmsSqQ]][+\-][ y ]) . The value of .Em x is used as an offset in the file. A byte, id3 length, short or long is read at that offset depending on the -.Em [bislBISLm] +.Em [bBcCeEfFgGhHiIlmsSqQ] type specifier. The value is treated as signed if .Dq , @@ -563,6 +576,20 @@ To that number the value of .Em y is added and the result is used as an offset in the file. The default type if one is not specified is long. +The following types are recognized: +.Bl -column -offset indent "Type" "Half/Short" "Little" "Size" +.It Sy Type Sy Mnemonic Sy Endian Sy Size +.It bcBc Byte/Char N/A 1 +.It efg Double Little 8 +.It EFG Double Big 8 +.It hs Half/Short Little 2 +.It HS Half/Short Big 2 +.It i ID3 Little 4 +.It I ID3 Big 4 +.It m Middle Middle 4 +.It q Quad Little 8 +.It Q Quad Big 8 +.El .Pp That way variable length structures can be examined: .Bd -literal -offset indent Modified: stable/11/contrib/file/magic/Magdir/acorn ============================================================================== --- stable/11/contrib/file/magic/Magdir/acorn Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/magic/Magdir/acorn Wed Aug 15 02:14:46 2018 (r337827) @@ -1,6 +1,6 @@ #------------------------------------------------------------------------------ -# $File: acorn,v 1.5 2009/09/19 16:28:07 christos Exp $ +# $File: acorn,v 1.6 2017/10/19 16:40:37 christos Exp $ # acorn: file(1) magic for files found on Acorn systems # @@ -67,3 +67,36 @@ >>8 byte x version %d, >>10 leshort =1 1 pattern >>10 leshort !1 %d patterns + +# From: Joerg Jenderek +# URL: https://www.kyzer.me.uk/pack/xad/#PackDir +# reference: https://www.kyzer.me.uk/pack/xad/xad_PackDir.lha/PackDir.c +# GRR: line below is too general as it matches also "Git pack" in ./revision +0 string PACK\0 +# check for valid compression method 0-4 +>5 ulelong <5 +# https://www.riscosopen.org/wiki/documentation/show/Introduction%20To%20Filing%20Systems +# To skip "Git pack" version 0 test for root directory object like +# ADFS::RPC.$.websitezip.FONTFIX +>>9 string >ADFS\ PackDir archive (RISC OS) +# TrID labels above as "Acorn PackDir compressed Archive" +# compression mode y (0 - 4) for GIF LZW with a maximum n bits +# (y~n,0~12,1~13,2~14,3~15,4~16) +>>>5 ulelong+12 x \b, LZW %u-bits compression +# http://www.filebase.org.uk/filetypes +# !Packdir compressed archive has three hexadecimal digits code 68E +!:mime application/x-acorn-68E +!:ext pkd/bin +# null terminated root directory object like IDEFS::IDE-4.$.Apps.GRAPHICS.!XFMPdemo +>>>9 string x \b, root "%s" +# load address 0xFFFtttdd, ttt is the object filetype and dddddddddd is time +>>>>&1 ulelong x \b, load address 0x%x +# execution address 0xdddddddd dddddddddd is 40 bit unsigned centiseconds since 1.1.1900 UTC +>>>>&5 ulelong x \b, exec address 0x%x +# attributes (bits: 0~owner read,1~owner write,3~no delete,4~public read,5~public write) +>>>>&9 ulelong x \b, attributes 0x%x +# number of entries in this directory. for root dir 0 +#>>>&13 ulelong x \b, entries 0x%x +# the entries start here with object name +>>>>&17 string x \b, 1st object "%s" + Modified: stable/11/contrib/file/magic/Magdir/animation ============================================================================== --- stable/11/contrib/file/magic/Magdir/animation Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/magic/Magdir/animation Wed Aug 15 02:14:46 2018 (r337827) @@ -1,6 +1,6 @@ #------------------------------------------------------------------------------ -# $File: animation,v 1.63 2017/05/26 14:33:07 christos Exp $ +# $File: animation,v 1.68 2018/05/06 16:08:07 christos Exp $ # animation: file(1) magic for animation/movie formats # # animation formats @@ -56,20 +56,20 @@ >>11 byte 6 \b, Release 6 MBMS Extended Presentations >>11 byte 7 \b, Release 7 MBMS Extended Presentations >8 string 3gg \b, MPEG v4 system, 3GPP ->11 byte 6 \b, Release 6 General Profile !:mime video/3gpp +>>11 byte 6 \b, Release 6 General Profile >8 string 3gp \b, MPEG v4 system, 3GPP ->11 byte 1 \b, Release %d (non existent) ->11 byte 2 \b, Release %d (non existent) ->11 byte 3 \b, Release %d (non existent) ->11 byte 4 \b, Release %d ->11 byte 5 \b, Release %d ->11 byte 6 \b, Release %d ->11 byte 7 \b, Release %d Streaming Servers !:mime video/3gpp +>>11 byte 1 \b, Release %d (non existent) +>>11 byte 2 \b, Release %d (non existent) +>>11 byte 3 \b, Release %d (non existent) +>>11 byte 4 \b, Release %d +>>11 byte 5 \b, Release %d +>>11 byte 6 \b, Release %d +>>11 byte 7 \b, Release %d Streaming Servers >8 string 3gs \b, MPEG v4 system, 3GPP ->11 byte 7 \b, Release %d Streaming Servers !:mime video/3gpp +>>11 byte 7 \b, Release %d Streaming Servers >8 string avc1 \b, MPEG v4 system, 3GPP JVT AVC [ISO 14496-12:2005] !:mime video/mp4 >8 string/W qt \b, Apple QuickTime movie @@ -188,6 +188,36 @@ >8 string pana \b, Panasonic Digital Camera >8 string qt \b, Apple QuickTime (.MOV/QT) !:mime video/quicktime +# HEIF image format +# see https://nokiatech.github.io/heif/technical.html +>8 string mif1 \b, HEIF Image +!:mime image/heif +>8 string msf1 \b, HEIF Image Sequence +!:mime image/heif-sequence +>8 string heic \b, HEIF Image HEVC Main or Main Still Picture Profile +!:mime image/heic +>8 string heix \b, HEIF Image HEVC Main 10 Profile +!:mime image/heic +>8 string hevc \b, HEIF Image Sequenz HEVC Main or Main Still Picture Profile +!:mime image/heic-sequence +>8 string hevx \b, HEIF Image Sequence HEVC Main 10 Profile +!:mime image/heic-sequence +# following HEIF brands are not mentioned in the heif technical info currently (Oct 2017) +# but used in the reference implementation: +# https://github.com/nokiatech/heif/blob/d5e9a21c8ba8df712bdf643021dd9f6518134776/Srcs/reader/hevcimagefilereader.cpp +>8 string heim \b, HEIF Image L-HEVC +!:mime image/heif +>8 string heis \b, HEIF Image L-HEVC +!:mime image/heif +>8 string avic \b, HEIF Image AVC +!:mime image/heif +>8 string hevm \b, HEIF Image Sequence L-HEVC +!:mime image/heif-sequence +>8 string hevs \b, HEIF Image Sequence L-HEVC +!:mime image/heif-sequence +>8 string avcs \b, HEIF Image Sequence AVC +!:mime image/heif-sequence + >8 string ROSS \b, Ross Video >8 string sdv \b, SD Memory Card Video >8 string ssc1 \b, Samsung stereo, single stream (patent pending) @@ -1027,3 +1057,10 @@ 0 string \x06\x0e\x2b\x34\x02\x05\x01\x01\x0d\x01\x02\x01\x01\x02 Material exchange container format !:ext mxf !:mime application/mxf + +# Recognize LucasArts Smush video files (cf. +# https://wiki.multimedia.cx/index.php/Smush) +0 string ANIM +>8 string AHDR LucasArts Smush Animation Format (SAN) video +0 string SANM +>8 string SHDR LucasArts Smush v2 (SANM) video Modified: stable/11/contrib/file/magic/Magdir/apple ============================================================================== --- stable/11/contrib/file/magic/Magdir/apple Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/magic/Magdir/apple Wed Aug 15 02:14:46 2018 (r337827) @@ -1,6 +1,6 @@ #------------------------------------------------------------------------------ -# $File: apple,v 1.36 2017/03/17 21:35:28 christos Exp $ +# $File: apple,v 1.39 2018/03/02 15:26:39 christos Exp $ # apple: file(1) magic for Apple file formats # 0 search/1/t FiLeStArTfIlEsTaRt binscii (apple ][) text @@ -431,3 +431,19 @@ >>>>0 use appleworks >0 belong 0x0481ad00 >>0 use appleworks + +# magic for Apple File System (APFS) +# from Alex Myczko +32 string NXSB Apple File System (APFS) +>36 ulelong x \b, blocksize %u + +# iTunes cover art (versions 1 and 2) +4 string itch +>24 string artw +>>0x1e8 string data iTunes cover art +>>>0x1ed string PNG (PNG) +>>>0x1ec beshort 0xffd8 (JPEG) + +# MacPaint image +65 string PNTGMPNT MacPaint image data +#0 belong 2 MacPaint image data Modified: stable/11/contrib/file/magic/Magdir/archive ============================================================================== --- stable/11/contrib/file/magic/Magdir/archive Wed Aug 15 01:29:02 2018 (r337826) +++ stable/11/contrib/file/magic/Magdir/archive Wed Aug 15 02:14:46 2018 (r337827) @@ -1,16 +1,151 @@ #------------------------------------------------------------------------------ -# $File: archive,v 1.108 2017/08/30 13:45:10 christos Exp $ +# $File: archive,v 1.119 2018/04/24 23:19:45 christos Exp $ # archive: file(1) magic for archive formats (see also "msdos" for self- # extracting compressed archives) # # cpio, ar, arc, arj, hpack, lha/lharc, rar, squish, uc2, zip, zoo, etc. -# pre-POSIX "tar" archives are handled in the C code. +# pre-POSIX "tar" archives are also handled in the C code ../../src/is_tar.c. # POSIX tar archives -257 string ustar\0 POSIX tar archive -!:mime application/x-tar # encoding: posix -257 string ustar\040\040\0 GNU tar archive -!:mime application/x-tar # encoding: gnu +# URL: https://en.wikipedia.org/wiki/Tar_(computing) +# Reference: https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5&manpath=FreeBSD+8-current +# header mainly padded with nul bytes +500 quad 0 +!:strength /2 +# filename or extended attribute printable strings in range space null til umlaut ue +>0 ubeshort >0x1F00 +>>0 ubeshort <0xFCFD +# last 4 header bytes often null but tar\0 in gtarfail2.tar gtarfail.tar-bad +# at https://sourceforge.net/projects/s-tar/files/testscripts/ +>>>508 ubelong&0x8B9E8DFF 0 +# nul, space or ascii digit 0-7 at start of mode +>>>>100 ubyte&0xC8 =0 +>>>>>101 ubyte&0xC8 =0 +# nul, space at end of check sum +>>>>>>155 ubyte&0xDF =0 +# space or ascii digit 0 at start of check sum +>>>>>>>148 ubyte&0xEF =0x20 +>>>>>>>>0 use tar-file +# minimal check and then display tar archive information which can also be +# embedded inside others like Android Backup, Clam AntiVirus database +0 name tar-file +>257 string !ustar +# header padded with nuls +>>257 ulong =0 +# GNU tar version 1.29 with non pax format option without refusing +# creates misleading V7 header for Long path, Multi-volume, Volume type +>>>156 ubyte 0x4c GNU tar archive +!:mime application/x-gtar +!:ext tar/gtar +>>>156 ubyte 0x4d GNU tar archive +!:mime application/x-gtar +!:ext tar/gtar +>>>156 ubyte 0x56 GNU tar archive +!:mime application/x-gtar +!:ext tar/gtar +>>>156 default x tar archive (V7) +!:mime application/x-tar +!:ext tar +# other stuff in padding +# some implementations add new fields to the blank area at the end of the header record +# created for example by DOS TAR 3.20g 1994 Tim V.Shapore with -j option +>>257 ulong !0 tar archive (old) +!:mime application/x-tar +!:ext tar +# magic in newer, GNU, posix variants +>257 string =ustar +# 2 last char of magic and UStar version because string expression does not work +# 2 space characters followed by a null for GNU variant +>>261 ubelong =0x72202000 POSIX tar archive (GNU) +!:mime application/x-gtar +!:ext tar/gtar +# UStar version with ASCII "00" +>>261 ubelong 0x72003030 POSIX +# gLOBAL and ExTENSION type only found in POSIX.1-2001 format +>>>156 ubyte 0x67 \b.1-2001 +>>>156 ubyte 0x78 \b.1-2001 +>>>156 ubyte x tar archive +!:mime application/x-ustar +!:ext tar/ustar +# version with 2 binary nuls embedded in Android Backup like com.android.settings.ab +>>261 ubelong 0x72000000 tar archive (ustar) +!:mime application/x-ustar +!:ext tar/ustar +# not seen ustar variant with garbish version +>>261 default x tar archive (unknown ustar) +!:mime application/x-ustar +!:ext tar/ustar +# type flag of 1st tar archive member +#>156 ubyte x \b, %c-type +>156 ubyte x +>>156 ubyte 0 \b, file +>>156 ubyte 0x30 \b, file +>>156 ubyte 0x31 \b, hard link +>>156 ubyte 0x32 \b, symlink +>>156 ubyte 0x33 \b, char device +>>156 ubyte 0x34 \b, block device +>>156 ubyte 0x35 \b, directory +>>156 ubyte 0x36 \b, fifo +>>156 ubyte 0x37 \b, reserved +>>156 ubyte 0x4c \b, long path +>>156 ubyte 0x4d \b, multi volume +>>156 ubyte 0x56 \b, volume +>>156 ubyte 0x67 \b, global +>>156 ubyte 0x78 \b, extension +>>156 default x \b, type +>>>156 ubyte x '%c' +# name[100] +>0 string >\0 %-.60s +# mode mainly stored as an octal number in ASCII null or space terminated +>100 string >\0 \b, mode %-.7s +# user id mainly as octal numbers in ASCII null or space terminated +>108 string >\0 \b, uid %-.7s +# group id mainly as octal numbers in ASCII null or space terminated +>116 string >\0 \b, gid %-.7s +# size mainly as octal number in ASCII +>124 ubyte <0x38 +>>124 string >\0 \b, size %-.12s +# coding indicated by setting the high-order bit of the leftmost byte +>124 ubyte >0xEF \b, size 0x +>>124 ubyte !0xff \b%2.2x +>>125 ubyte !0xff \b%2.2x +>>126 ubyte !0xff \b%2.2x +>>127 ubyte !0xff \b%2.2x +>>128 ubyte !0xff \b%2.2x +>>129 ubyte !0xff \b%2.2x +>>130 ubyte !0xff \b%2.2x +>>131 ubyte !0xff \b%2.2x +>>132 ubyte !0xff \b%2.2x +>>133 ubyte !0xff \b%2.2x +>>134 ubyte !0xff \b%2.2x +>>135 ubyte !0xff \b%2.2x +# seconds since 0:0:0 1 jan 1970 UTC as octal number mainly in ASCII null or space terminated +>136 string >\0 \b, seconds %-.11s +# header checksum stored as an octal number in ASCII null or space terminated +#>148 string x \b, cksum %.7s +# linkname[100] +>157 string >\0 \b, linkname %-.40s +# additional fields for ustar +>257 string =ustar +# owner user name null terminated +>>265 string >\0 \b, user %-.32s +# group name null terminated +>>297 string >\0 \b, group %-.32s +# device major minor if not zero *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***