From owner-svn-src-vendor@freebsd.org Sun Jul 29 05:49:50 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE7A8104C154; Sun, 29 Jul 2018 05:49:49 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5D0B2727E2; Sun, 29 Jul 2018 05:49:49 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3244A11ABF; Sun, 29 Jul 2018 05:49:49 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6T5nmSv019202; Sun, 29 Jul 2018 05:49:48 GMT (envelope-from eadler@FreeBSD.org) Received: (from eadler@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6T5nmwZ019197; Sun, 29 Jul 2018 05:49:48 GMT (envelope-from eadler@FreeBSD.org) Message-Id: <201807290549.w6T5nmwZ019197@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: eadler set sender to eadler@FreeBSD.org using -f From: Eitan Adler Date: Sun, 29 Jul 2018 05:49:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336860 - in vendor/file/5.34: . doc m4 magic magic/Magdir python src tests X-SVN-Group: vendor X-SVN-Commit-Author: eadler X-SVN-Commit-Paths: in vendor/file/5.34: . doc m4 magic magic/Magdir python src tests X-SVN-Commit-Revision: 336860 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jul 2018 05:49:50 -0000 Author: eadler Date: Sun Jul 29 05:49:48 2018 New Revision: 336860 URL: https://svnweb.freebsd.org/changeset/base/336860 Log: vendor/file: add 5.34 gpg: Good signature from "Christos Zoulas (personal key) " [unknown] gpg: aka "Christos Zoulas (NetBSD key) " [unknown] Obtained from: ftp://ftp.astron.com/pub/file/ Added: vendor/file/5.34/ vendor/file/5.34/AUTHORS vendor/file/5.34/COPYING vendor/file/5.34/ChangeLog vendor/file/5.34/INSTALL vendor/file/5.34/MAINT vendor/file/5.34/Makefile.am (contents, props changed) vendor/file/5.34/Makefile.in (contents, props changed) vendor/file/5.34/NEWS vendor/file/5.34/README vendor/file/5.34/TODO vendor/file/5.34/acinclude.m4 vendor/file/5.34/aclocal.m4 vendor/file/5.34/compile (contents, props changed) vendor/file/5.34/config.guess (contents, props changed) vendor/file/5.34/config.h.in (contents, props changed) vendor/file/5.34/config.sub (contents, props changed) vendor/file/5.34/configure (contents, props changed) vendor/file/5.34/configure.ac vendor/file/5.34/depcomp (contents, props changed) vendor/file/5.34/doc/ vendor/file/5.34/doc/Makefile.am (contents, props changed) vendor/file/5.34/doc/Makefile.in (contents, props changed) vendor/file/5.34/doc/file.man vendor/file/5.34/doc/libmagic.man vendor/file/5.34/doc/magic.man vendor/file/5.34/install-sh (contents, props changed) vendor/file/5.34/ltmain.sh (contents, props changed) vendor/file/5.34/m4/ vendor/file/5.34/m4/libtool.m4 vendor/file/5.34/m4/ltoptions.m4 vendor/file/5.34/m4/ltsugar.m4 vendor/file/5.34/m4/ltversion.m4 vendor/file/5.34/m4/lt~obsolete.m4 vendor/file/5.34/magic/ vendor/file/5.34/magic/Header vendor/file/5.34/magic/Localstuff vendor/file/5.34/magic/Magdir/ vendor/file/5.34/magic/Magdir/acorn vendor/file/5.34/magic/Magdir/adi vendor/file/5.34/magic/Magdir/adventure vendor/file/5.34/magic/Magdir/algol68 vendor/file/5.34/magic/Magdir/allegro vendor/file/5.34/magic/Magdir/alliant vendor/file/5.34/magic/Magdir/amanda vendor/file/5.34/magic/Magdir/amigaos vendor/file/5.34/magic/Magdir/android vendor/file/5.34/magic/Magdir/animation vendor/file/5.34/magic/Magdir/aout vendor/file/5.34/magic/Magdir/apache (contents, props changed) vendor/file/5.34/magic/Magdir/apl vendor/file/5.34/magic/Magdir/apple (contents, props changed) vendor/file/5.34/magic/Magdir/application vendor/file/5.34/magic/Magdir/applix vendor/file/5.34/magic/Magdir/apt vendor/file/5.34/magic/Magdir/archive vendor/file/5.34/magic/Magdir/assembler vendor/file/5.34/magic/Magdir/asterix vendor/file/5.34/magic/Magdir/att3b vendor/file/5.34/magic/Magdir/audio vendor/file/5.34/magic/Magdir/basis vendor/file/5.34/magic/Magdir/beetle vendor/file/5.34/magic/Magdir/ber vendor/file/5.34/magic/Magdir/bflt vendor/file/5.34/magic/Magdir/bhl vendor/file/5.34/magic/Magdir/bioinformatics vendor/file/5.34/magic/Magdir/blackberry vendor/file/5.34/magic/Magdir/blcr vendor/file/5.34/magic/Magdir/blender vendor/file/5.34/magic/Magdir/blit vendor/file/5.34/magic/Magdir/bout vendor/file/5.34/magic/Magdir/bsdi vendor/file/5.34/magic/Magdir/bsi vendor/file/5.34/magic/Magdir/btsnoop vendor/file/5.34/magic/Magdir/c-lang vendor/file/5.34/magic/Magdir/c64 vendor/file/5.34/magic/Magdir/cad vendor/file/5.34/magic/Magdir/cafebabe vendor/file/5.34/magic/Magdir/cbor vendor/file/5.34/magic/Magdir/cddb vendor/file/5.34/magic/Magdir/chord vendor/file/5.34/magic/Magdir/cisco vendor/file/5.34/magic/Magdir/citrus vendor/file/5.34/magic/Magdir/clarion vendor/file/5.34/magic/Magdir/claris vendor/file/5.34/magic/Magdir/clipper vendor/file/5.34/magic/Magdir/coff vendor/file/5.34/magic/Magdir/commands vendor/file/5.34/magic/Magdir/communications vendor/file/5.34/magic/Magdir/compress vendor/file/5.34/magic/Magdir/console vendor/file/5.34/magic/Magdir/convex vendor/file/5.34/magic/Magdir/coverage vendor/file/5.34/magic/Magdir/cracklib vendor/file/5.34/magic/Magdir/ctags vendor/file/5.34/magic/Magdir/ctf vendor/file/5.34/magic/Magdir/cubemap vendor/file/5.34/magic/Magdir/cups vendor/file/5.34/magic/Magdir/dact vendor/file/5.34/magic/Magdir/database vendor/file/5.34/magic/Magdir/dataone vendor/file/5.34/magic/Magdir/dbpf vendor/file/5.34/magic/Magdir/der vendor/file/5.34/magic/Magdir/diamond vendor/file/5.34/magic/Magdir/diff vendor/file/5.34/magic/Magdir/digital vendor/file/5.34/magic/Magdir/dolby vendor/file/5.34/magic/Magdir/dump vendor/file/5.34/magic/Magdir/dyadic vendor/file/5.34/magic/Magdir/ebml vendor/file/5.34/magic/Magdir/editors vendor/file/5.34/magic/Magdir/efi vendor/file/5.34/magic/Magdir/elf vendor/file/5.34/magic/Magdir/encore vendor/file/5.34/magic/Magdir/epoc vendor/file/5.34/magic/Magdir/erlang vendor/file/5.34/magic/Magdir/esri vendor/file/5.34/magic/Magdir/fcs vendor/file/5.34/magic/Magdir/filesystems vendor/file/5.34/magic/Magdir/finger vendor/file/5.34/magic/Magdir/flash vendor/file/5.34/magic/Magdir/flif vendor/file/5.34/magic/Magdir/fonts vendor/file/5.34/magic/Magdir/fortran vendor/file/5.34/magic/Magdir/frame vendor/file/5.34/magic/Magdir/freebsd vendor/file/5.34/magic/Magdir/fsav vendor/file/5.34/magic/Magdir/fusecompress vendor/file/5.34/magic/Magdir/games vendor/file/5.34/magic/Magdir/gcc vendor/file/5.34/magic/Magdir/gconv vendor/file/5.34/magic/Magdir/geo vendor/file/5.34/magic/Magdir/geos vendor/file/5.34/magic/Magdir/gimp vendor/file/5.34/magic/Magdir/gnome vendor/file/5.34/magic/Magdir/gnu vendor/file/5.34/magic/Magdir/gnumeric vendor/file/5.34/magic/Magdir/gpt vendor/file/5.34/magic/Magdir/gpu vendor/file/5.34/magic/Magdir/grace vendor/file/5.34/magic/Magdir/graphviz vendor/file/5.34/magic/Magdir/gringotts vendor/file/5.34/magic/Magdir/guile vendor/file/5.34/magic/Magdir/hitachi-sh vendor/file/5.34/magic/Magdir/hp vendor/file/5.34/magic/Magdir/human68k vendor/file/5.34/magic/Magdir/ibm370 vendor/file/5.34/magic/Magdir/ibm6000 vendor/file/5.34/magic/Magdir/icc vendor/file/5.34/magic/Magdir/iff vendor/file/5.34/magic/Magdir/images vendor/file/5.34/magic/Magdir/inform vendor/file/5.34/magic/Magdir/intel vendor/file/5.34/magic/Magdir/interleaf vendor/file/5.34/magic/Magdir/island vendor/file/5.34/magic/Magdir/ispell vendor/file/5.34/magic/Magdir/isz vendor/file/5.34/magic/Magdir/java vendor/file/5.34/magic/Magdir/javascript vendor/file/5.34/magic/Magdir/jpeg vendor/file/5.34/magic/Magdir/karma vendor/file/5.34/magic/Magdir/kde vendor/file/5.34/magic/Magdir/keepass vendor/file/5.34/magic/Magdir/kerberos vendor/file/5.34/magic/Magdir/kml vendor/file/5.34/magic/Magdir/lecter vendor/file/5.34/magic/Magdir/lex vendor/file/5.34/magic/Magdir/lif vendor/file/5.34/magic/Magdir/linux vendor/file/5.34/magic/Magdir/lisp vendor/file/5.34/magic/Magdir/llvm vendor/file/5.34/magic/Magdir/lua vendor/file/5.34/magic/Magdir/luks vendor/file/5.34/magic/Magdir/m4 vendor/file/5.34/magic/Magdir/mach vendor/file/5.34/magic/Magdir/macintosh vendor/file/5.34/magic/Magdir/macos vendor/file/5.34/magic/Magdir/magic vendor/file/5.34/magic/Magdir/mail.news vendor/file/5.34/magic/Magdir/make vendor/file/5.34/magic/Magdir/map vendor/file/5.34/magic/Magdir/maple vendor/file/5.34/magic/Magdir/marc21 vendor/file/5.34/magic/Magdir/mathcad vendor/file/5.34/magic/Magdir/mathematica vendor/file/5.34/magic/Magdir/matroska vendor/file/5.34/magic/Magdir/mcrypt vendor/file/5.34/magic/Magdir/measure vendor/file/5.34/magic/Magdir/mercurial vendor/file/5.34/magic/Magdir/metastore vendor/file/5.34/magic/Magdir/meteorological vendor/file/5.34/magic/Magdir/microfocus vendor/file/5.34/magic/Magdir/mime vendor/file/5.34/magic/Magdir/mips vendor/file/5.34/magic/Magdir/mirage vendor/file/5.34/magic/Magdir/misctools vendor/file/5.34/magic/Magdir/mkid vendor/file/5.34/magic/Magdir/mlssa vendor/file/5.34/magic/Magdir/mmdf vendor/file/5.34/magic/Magdir/modem vendor/file/5.34/magic/Magdir/motorola vendor/file/5.34/magic/Magdir/mozilla vendor/file/5.34/magic/Magdir/msdos vendor/file/5.34/magic/Magdir/msooxml vendor/file/5.34/magic/Magdir/msvc vendor/file/5.34/magic/Magdir/msx vendor/file/5.34/magic/Magdir/mup vendor/file/5.34/magic/Magdir/music vendor/file/5.34/magic/Magdir/nasa vendor/file/5.34/magic/Magdir/natinst vendor/file/5.34/magic/Magdir/ncr vendor/file/5.34/magic/Magdir/neko vendor/file/5.34/magic/Magdir/netbsd vendor/file/5.34/magic/Magdir/netscape vendor/file/5.34/magic/Magdir/netware vendor/file/5.34/magic/Magdir/news vendor/file/5.34/magic/Magdir/nitpicker vendor/file/5.34/magic/Magdir/oasis vendor/file/5.34/magic/Magdir/ocaml vendor/file/5.34/magic/Magdir/octave vendor/file/5.34/magic/Magdir/ole2compounddocs vendor/file/5.34/magic/Magdir/olf vendor/file/5.34/magic/Magdir/os2 vendor/file/5.34/magic/Magdir/os400 vendor/file/5.34/magic/Magdir/os9 vendor/file/5.34/magic/Magdir/osf1 vendor/file/5.34/magic/Magdir/palm vendor/file/5.34/magic/Magdir/parix vendor/file/5.34/magic/Magdir/parrot vendor/file/5.34/magic/Magdir/pascal vendor/file/5.34/magic/Magdir/pbf vendor/file/5.34/magic/Magdir/pbm vendor/file/5.34/magic/Magdir/pc88 vendor/file/5.34/magic/Magdir/pc98 vendor/file/5.34/magic/Magdir/pdf (contents, props changed) vendor/file/5.34/magic/Magdir/pdp vendor/file/5.34/magic/Magdir/perl vendor/file/5.34/magic/Magdir/pgf vendor/file/5.34/magic/Magdir/pgp vendor/file/5.34/magic/Magdir/pkgadd vendor/file/5.34/magic/Magdir/plan9 vendor/file/5.34/magic/Magdir/plus5 vendor/file/5.34/magic/Magdir/polyml vendor/file/5.34/magic/Magdir/printer vendor/file/5.34/magic/Magdir/project vendor/file/5.34/magic/Magdir/psdbms vendor/file/5.34/magic/Magdir/psl vendor/file/5.34/magic/Magdir/pulsar vendor/file/5.34/magic/Magdir/pwsafe vendor/file/5.34/magic/Magdir/pyramid vendor/file/5.34/magic/Magdir/python vendor/file/5.34/magic/Magdir/qt vendor/file/5.34/magic/Magdir/revision vendor/file/5.34/magic/Magdir/riff vendor/file/5.34/magic/Magdir/rpi vendor/file/5.34/magic/Magdir/rpm vendor/file/5.34/magic/Magdir/rtf vendor/file/5.34/magic/Magdir/ruby vendor/file/5.34/magic/Magdir/sc vendor/file/5.34/magic/Magdir/sccs vendor/file/5.34/magic/Magdir/scientific vendor/file/5.34/magic/Magdir/securitycerts vendor/file/5.34/magic/Magdir/selinux vendor/file/5.34/magic/Magdir/sendmail vendor/file/5.34/magic/Magdir/sequent vendor/file/5.34/magic/Magdir/sereal vendor/file/5.34/magic/Magdir/sgi vendor/file/5.34/magic/Magdir/sgml vendor/file/5.34/magic/Magdir/sharc vendor/file/5.34/magic/Magdir/sinclair vendor/file/5.34/magic/Magdir/sisu vendor/file/5.34/magic/Magdir/sketch vendor/file/5.34/magic/Magdir/smalltalk vendor/file/5.34/magic/Magdir/smile vendor/file/5.34/magic/Magdir/sniffer vendor/file/5.34/magic/Magdir/softquad vendor/file/5.34/magic/Magdir/spec vendor/file/5.34/magic/Magdir/spectrum vendor/file/5.34/magic/Magdir/sql vendor/file/5.34/magic/Magdir/ssh vendor/file/5.34/magic/Magdir/ssl vendor/file/5.34/magic/Magdir/sun vendor/file/5.34/magic/Magdir/symbos vendor/file/5.34/magic/Magdir/sysex vendor/file/5.34/magic/Magdir/tcl vendor/file/5.34/magic/Magdir/teapot vendor/file/5.34/magic/Magdir/terminfo vendor/file/5.34/magic/Magdir/tex vendor/file/5.34/magic/Magdir/tgif vendor/file/5.34/magic/Magdir/ti-8x vendor/file/5.34/magic/Magdir/timezone vendor/file/5.34/magic/Magdir/tplink vendor/file/5.34/magic/Magdir/troff vendor/file/5.34/magic/Magdir/tuxedo vendor/file/5.34/magic/Magdir/typeset vendor/file/5.34/magic/Magdir/unicode vendor/file/5.34/magic/Magdir/unknown vendor/file/5.34/magic/Magdir/uterus vendor/file/5.34/magic/Magdir/uuencode vendor/file/5.34/magic/Magdir/vacuum-cleaner vendor/file/5.34/magic/Magdir/varied.out vendor/file/5.34/magic/Magdir/varied.script vendor/file/5.34/magic/Magdir/vax vendor/file/5.34/magic/Magdir/vicar vendor/file/5.34/magic/Magdir/virtual vendor/file/5.34/magic/Magdir/virtutech vendor/file/5.34/magic/Magdir/visx vendor/file/5.34/magic/Magdir/vms vendor/file/5.34/magic/Magdir/vmware vendor/file/5.34/magic/Magdir/vorbis vendor/file/5.34/magic/Magdir/vxl vendor/file/5.34/magic/Magdir/warc vendor/file/5.34/magic/Magdir/weak vendor/file/5.34/magic/Magdir/webassembly vendor/file/5.34/magic/Magdir/windows vendor/file/5.34/magic/Magdir/wireless vendor/file/5.34/magic/Magdir/wordprocessors vendor/file/5.34/magic/Magdir/wsdl vendor/file/5.34/magic/Magdir/x68000 vendor/file/5.34/magic/Magdir/xdelta vendor/file/5.34/magic/Magdir/xenix vendor/file/5.34/magic/Magdir/xilinx vendor/file/5.34/magic/Magdir/xo65 vendor/file/5.34/magic/Magdir/xwindows vendor/file/5.34/magic/Magdir/yara vendor/file/5.34/magic/Magdir/zfs vendor/file/5.34/magic/Magdir/zilog vendor/file/5.34/magic/Magdir/zip vendor/file/5.34/magic/Magdir/zyxel vendor/file/5.34/magic/Makefile.am (contents, props changed) vendor/file/5.34/magic/Makefile.in (contents, props changed) vendor/file/5.34/missing (contents, props changed) vendor/file/5.34/python/ vendor/file/5.34/python/CHANGELOG.md vendor/file/5.34/python/LICENSE vendor/file/5.34/python/Makefile.am (contents, props changed) vendor/file/5.34/python/Makefile.in (contents, props changed) vendor/file/5.34/python/README.md vendor/file/5.34/python/example.py (contents, props changed) vendor/file/5.34/python/magic.py (contents, props changed) vendor/file/5.34/python/setup.py (contents, props changed) vendor/file/5.34/python/tests.py (contents, props changed) vendor/file/5.34/src/ vendor/file/5.34/src/Makefile.am (contents, props changed) vendor/file/5.34/src/Makefile.in (contents, props changed) vendor/file/5.34/src/apprentice.c (contents, props changed) vendor/file/5.34/src/apptype.c (contents, props changed) vendor/file/5.34/src/ascmagic.c (contents, props changed) vendor/file/5.34/src/asctime_r.c (contents, props changed) vendor/file/5.34/src/asprintf.c (contents, props changed) vendor/file/5.34/src/buffer.c (contents, props changed) vendor/file/5.34/src/cdf.c (contents, props changed) vendor/file/5.34/src/cdf.h (contents, props changed) vendor/file/5.34/src/cdf_time.c (contents, props changed) vendor/file/5.34/src/compress.c (contents, props changed) vendor/file/5.34/src/ctime_r.c (contents, props changed) vendor/file/5.34/src/der.c (contents, props changed) vendor/file/5.34/src/der.h (contents, props changed) vendor/file/5.34/src/dprintf.c (contents, props changed) vendor/file/5.34/src/elfclass.h (contents, props changed) vendor/file/5.34/src/encoding.c (contents, props changed) vendor/file/5.34/src/file.c (contents, props changed) vendor/file/5.34/src/file.h (contents, props changed) vendor/file/5.34/src/file_opts.h (contents, props changed) vendor/file/5.34/src/fmtcheck.c (contents, props changed) vendor/file/5.34/src/fsmagic.c (contents, props changed) vendor/file/5.34/src/funcs.c (contents, props changed) vendor/file/5.34/src/getline.c (contents, props changed) vendor/file/5.34/src/getopt_long.c (contents, props changed) vendor/file/5.34/src/gmtime_r.c (contents, props changed) vendor/file/5.34/src/is_tar.c (contents, props changed) vendor/file/5.34/src/localtime_r.c (contents, props changed) vendor/file/5.34/src/magic.c (contents, props changed) vendor/file/5.34/src/magic.h.in (contents, props changed) vendor/file/5.34/src/mygetopt.h (contents, props changed) vendor/file/5.34/src/pread.c (contents, props changed) vendor/file/5.34/src/print.c (contents, props changed) vendor/file/5.34/src/readcdf.c (contents, props changed) vendor/file/5.34/src/readelf.c (contents, props changed) vendor/file/5.34/src/readelf.h (contents, props changed) vendor/file/5.34/src/seccomp.c (contents, props changed) vendor/file/5.34/src/softmagic.c (contents, props changed) vendor/file/5.34/src/strcasestr.c (contents, props changed) vendor/file/5.34/src/strlcat.c (contents, props changed) vendor/file/5.34/src/strlcpy.c (contents, props changed) vendor/file/5.34/src/tar.h (contents, props changed) vendor/file/5.34/src/vasprintf.c (contents, props changed) vendor/file/5.34/tests/ vendor/file/5.34/tests/JW07022A.mp3.result vendor/file/5.34/tests/JW07022A.mp3.testfile (contents, props changed) vendor/file/5.34/tests/Makefile.am (contents, props changed) vendor/file/5.34/tests/Makefile.in (contents, props changed) vendor/file/5.34/tests/README vendor/file/5.34/tests/escapevel.result vendor/file/5.34/tests/escapevel.testfile (contents, props changed) vendor/file/5.34/tests/gedcom.result vendor/file/5.34/tests/gedcom.testfile vendor/file/5.34/tests/hddrawcopytool.result vendor/file/5.34/tests/hddrawcopytool.testfile (contents, props changed) vendor/file/5.34/tests/issue311docx.result vendor/file/5.34/tests/issue311docx.testfile (contents, props changed) vendor/file/5.34/tests/test.c (contents, props changed) Added: vendor/file/5.34/AUTHORS ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/file/5.34/AUTHORS Sun Jul 29 05:49:48 2018 (r336860) @@ -0,0 +1 @@ +See COPYING. \ No newline at end of file Added: vendor/file/5.34/COPYING ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/file/5.34/COPYING Sun Jul 29 05:49:48 2018 (r336860) @@ -0,0 +1,29 @@ +$File: COPYING,v 1.1 2008/02/05 19:08:11 christos Exp $ +Copyright (c) Ian F. Darwin 1986, 1987, 1989, 1990, 1991, 1992, 1994, 1995. +Software written by Ian F. Darwin and others; +maintained 1994- Christos Zoulas. + +This software is not subject to any export provision of the United States +Department of Commerce, and may be exported to any country or planet. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice immediately at the beginning of the file, without modification, + this list of conditions, and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR +ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +SUCH DAMAGE. Added: vendor/file/5.34/ChangeLog ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/file/5.34/ChangeLog Sun Jul 29 05:49:48 2018 (r336860) @@ -0,0 +1,1753 @@ +2018-07-25 8:50 Christos Zoulas + + * release 5.34 + +2018-06-22 16:38 Christos Zoulas + + * Add Quad indirect offsets + +2018-05-24 14:10 Christos Zoulas + + * Enable parsing of ELF dynamic sections to handle PIE better + +2018-04-15 14:52 Christos Zoulas + + * release 5.33 + +2018-02-24 14:50 Christos Zoulas + + * extend the support for ${x?:} expansions for magic descriptions + +2018-02-21 16:25 Christos Zoulas + + * add support for ${x?:} in mime types to handle + pie binaries. + +2017-11-03 9:23 Christos Zoulas + + * add support for negative offsets (offsets from the end of file) + +2017-09-26 8:22 Christos Zoulas + + * close the file on error when writing magic (Steve Grubb) + +2017-09-24 12:02 Christos Zoulas + + * seccomp support (Paul Moore) + +2017-09-02 11:53 Christos Zoulas + + * release 5.32 + +2017-08-28 16:37 Christos Zoulas + + * Always reset state in {file,buffer}_apprentice (Krzysztof Wilczynski) + +2017-08-27 03:55 Christos Zoulas + + * Fix always true condition (Thomas Jarosch) + +2017-05-24 17:30 Christos Zoulas + + * pickier parsing of numeric values in magic files. + +2017-05-23 17:55 Christos Zoulas + + * PR/615 add magic_getflags() + +2017-05-23 13:55 Christos Zoulas + + * release 5.31 + +2017-03-17 20:32 Christos Zoulas + + * remove trailing spaces from magic files + * refactor is_tar + * better bounds checks for cdf + +2017-02-10 12:24 Christos Zoulas + + * release 5.30 + +2017-02-07 23:27 Christos Zoulas + + * If we exceeded the offset in a search return no match + (Christoph Biedl) + * Be more lenient on corrupt CDF files (Christoph Biedl) + +2017-02-04 16:46 Christos Zoulas + + * pacify ubsan sign extension (oss-fuzz/524) + +2017-02-01 12:42 Christos Zoulas + + * off by one in cdf parsing (PR/593) + * report debugging sections in elf (PR/591) + +2016-11-06 10:52 Christos Zoulas + + * Allow @@@ in extensions + * Add missing overflow check in der magic (Jonas Wagner) + +2016-10-25 10:40 Christos Zoulas + + * release 5.29 + +2016-10-24 11:20 Christos Zoulas + + * der getlength overflow (Jonas Wagner) + * multiple magic file load failure (Christoph Biedl) + +2016-10-17 11:26 Christos Zoulas + + * CDF parsing improvements (Guy Helmer) + +2016-07-20 7:26 Christos Zoulas + + * Add support for signed indirect offsets + +2016-07-18 7:41 Christos Zoulas + + * cat /dev/null | file - should print empty (Christoph Biedl) + +2016-07-05 15:20 Christos Zoulas + + * Bump string size from 64 to 96. + +2016-06-13 20:20 Christos Zoulas + + * PR/556: Fix separators on annotations. + +2016-06-13 19:40 Christos Zoulas + + * release 5.28 + * fix leak on allocation failure + +2016-06-01 1:20 Christos Zoulas + + * PR/555: Avoid overflow for offset > nbytes + * PR/550: Segv on DER parsing: + - use the correct variable for length + - set offset to 0 on failure. + +2016-05-13 12:00 Christos Zoulas + + * release 5.27 + +2016-04-18 9:35 Christos Zoulas + + * Errors comparing DER entries or computing offsets + are just indications of malformed non-DER files. + Don't print them. + * Offset comparison was off-by-one. + * Fix compression code (Werner Fink) + * Put new bytes constant in the right file (not the generated one) + +2016-04-16 18:34 Christos Zoulas + + * release 5.26 + +2016-03-31 13:50 Christos Zoulas + + * make the number of bytes read from files configurable. + +2016-03-21 13:40 Christos Zoulas + + * Add bounds checks for DER code (discovered by Thomas Jarosch) + * Change indirect recursion limit to indirect use count and + bump from 15 to 50 to prevent abuse. + +2016-03-13 20:39 Christos Zoulas + + * Add -00 which prints filename\0description\0 + +2016-03-01 13:28 Christos Zoulas + + * Fix ID3 indirect parsing + +2016-01-19 10:18 Christos Zoulas + + * add DER parsing capability + +2015-11-13 10:35 Christos Zoulas + + * provide dprintf(3) for the OS's that don't have it. + +2015-11-11 16:25 Christos Zoulas + + * redo the compression code report decompression errors + +2015-11-10 23:25 Christos Zoulas + + * REG_STARTEND code is not working as expected, delete it. + +2015-11-09 16:05 Christos Zoulas + + * Add zlib support if we have it. + +2015-11-05 11:22 Christos Zoulas + + * PR/492: compression forking was broken with magic_buffer. + +2015-09-16 9:50 Christos Zoulas + + * release 5.25 + +2015-09-11 13:25 Christos Zoulas + + * add a limit to the length of regex searches + +2015-09-08 9:50 Christos Zoulas + + * fix problems with --parameter (Christoph Biedl) + +2015-07-11 10:35 Christos Zoulas + + * Windows fixes PR/466 (Jason Hood) + +2015-07-09 10:35 Christos Zoulas + + * release 5.24 + +2015-06-11 8:52 Christos Zoulas + + * redo long option encoding to fix off-by-one in 5.23 + +2015-06-10 13:50 Christos Zoulas + + * release 5.23 + +2015-06-09 16:10 Christos Zoulas + + * Fix issue with regex range for magic with offset + * Always return true from mget with USE (success to mget not match + indication). Fixes mime evaluation after USE magic + * PR/459: Don't insert magic entries to the list if there are parsing + errors for them. + +2015-06-03 16:00 Christos Zoulas + + * PR/455: Add utf-7 encoding + +2015-06-03 14:30 Christos Zoulas + + * PR/455: Implement -Z, look inside, but don't report on compression + * PR/454: Fix allocation error on bad magic. + +2015-05-29 10:30 Christos Zoulas + + * handle MAGIC_CONTINUE everywhere, not just in softmagic + +2015-05-21 14:30 Christos Zoulas + + * don't print descriptions for NAME types when mime. + +2015-04-09 15:59 Christos Zoulas + + * Add --extension to list the known extensions for this file type + Idea by Andrew J Roazen + +2015-02-14 12:23 Christos Zoulas + + * Bump file search buffer size to 1M. + +2015-01-09 14:35 Christos Zoulas + + * Fix multiple issues with date formats reported by Christoph Biedl: + - T_LOCAL meaning was reversed + - Arithmetic did not work + Also stop adjusting daylight savings for gmt printing. + +2015-01-05 13:00 Christos Zoulas + + * PR/411: Fix memory corruption from corrupt cdf file. + +2015-01-02 15:15 Christos Zoulas + + * release 5.22 + +2015-01-01 12:01 Christos Zoulas + + * add indirect relative for TIFF/Exif + +2014-12-16 18:10 Christos Zoulas + + * restructure elf note printing to avoid repeated messages + * add note limit, suggested by Alexander Cherepanov + +2014-12-16 16:53 Christos Zoulas + + * Bail out on partial pread()'s (Alexander Cherepanov) + * Fix incorrect bounds check in file_printable (Alexander Cherepanov) + +2014-12-11 20:01 Christos Zoulas + + * PR/405: ignore SIGPIPE from uncompress programs + * change printable -> file_printable and use it in + more places for safety + * in ELF, instead of "(uses dynamic libraries)" when PT_INTERP + is present print the interpreter name. + +2014-12-10 20:01 Christos Zoulas + + * release 5.21 + +2014-11-27 18:40 Christos Zoulas + + * Allow setting more parameters from the command line. + * Split name/use and indirect magic recursion limits. + +2014-11-27 11:12 Christos Zoulas + + * Adjust ELF parameters and the default recursion + level. + * Allow setting the recursion level dynamically. + +2014-11-24 8:55 Christos Zoulas + + * The following fixes resulted from Thomas Jarosch's fuzzing + tests that revealed severe performance issues on pathological + input: + - limit number of elf program and sections processing + - abort elf note processing quickly + - reduce the number of recursion levels from 20 to 10 + - preserve error messages in indirect magic handling + + This is tracked as CVE-2014-8116 and CVE-2014-8117 + +2014-11-12 10:30 Christos Zoulas + + * fix bogus free in the user buffer case. + +2014-11-11 12:35 Christos Zoulas + + * fix out of bounds read for pascal strings + * fix memory leak (not freeing the head of each mlist) + +2014-11-07 10:25 Christos Zoulas + + * When printing strings from a file, convert them to printable + on a byte by byte basis, so that we don't get issues with + locale's trying to interpret random byte streams as UTF-8 and + having printf error out with EILSEQ. + +2014-10-17 11:48 Christos Zoulas + + * fix bounds in note reading (Francisco Alonso / Red Hat) + +2014-10-11 15:02 Christos Zoulas + + * fix autoconf glue for setlocale and locale_t; some OS's + have locale_t in xlocale.h + +2014-10-10 15:01 Christos Zoulas + + * release 5.20 + +2014-08-17 10:01 Christos Zoulas + + * recognize encrypted CDF documents + +2014-08-04 9:18 Christos Zoulas + + * add magic_load_buffers from Brooks Davis + +2014-07-24 16:40 Christos Zoulas + + * add thumbs.db support + +2014-06-12 12:28 Christos Zoulas + + * release 5.19 + +2014-06-09 9:04 Christos Zoulas + + * Misc buffer overruns and missing buffer size tests in cdf parsing + (Francisco Alonso, Jan Kaluza) + +2014-06-02 14:50 Christos Zoulas + + * Enforce limit of 8K on regex searches that have no limits + * Allow the l modifier for regex to mean line count. Default + to byte count. If line count is specified, assume a max + of 80 characters per line to limit the byte count. + * Don't allow conversions to be used for dates, allowing + the mask field to be used as an offset. + +2014-05-30 12:51 Christos Zoulas + + * Make the range operator limit the length of the + regex search. + +2014-05-14 19:23 Christos Zoulas + + * PR/347: Windows fixes + * PR/352: Hangul word processor recognition + * PR/354: Encoding irregularities in text files + +2014-05-06 6:12 Christos Zoulas + + * Fix uninitialized title in CDF files (Jan Kaluza) + +2014-05-04 14:55 Christos Zoulas + + * PR/351: Fix compilation of empty files + +2014-04-30 17:39 Christos Zoulas + + * Fix integer formats: We don't specify 'l' or + 'h' and 'hh' specifiers anymore, only 'll' for + quads and nothing for the rest. This is so that + magic writing is simpler. + +2014-04-01 15:25 Christos Zoulas + + * PR/341: Jan Kaluza, fix memory leak + * PR/342: Jan Kaluza, fix out of bounds read + +2014-03-28 15:25 Christos Zoulas + + * Fix issue with long formats not matching fmtcheck + +2014-03-26 11:25 Christos Zoulas + + * release 5.18 + +2014-03-15 17:45 Christos Zoulas + + * add fmtcheck(3) for those who don't have it + +2014-03-14 15:12 Christos Zoulas + + * prevent mime entries from being attached to magic + entries with no descriptions + + * adjust magic strength for regex type + + * remove superfluous ascmagic with encoding test + +2014-03-06 12:01 Christos Zoulas + + * fix regression fix echo -ne "\012\013\014" | file -i - + which printed "binary" instead of "application/octet-stream" + + * add size_t overflow check for magic file size + +2014-02-27 16:01 Christos Zoulas + + * experimental support for matching with CFD CLSID + +2014-02-18 13:04 Kimmo Suominen (kimmo@suominen.com) + + * Cache old LC_CTYPE locale before setting it to "C", so + we can use it to restore LC_CTYPE instead of asking + setlocale() to scan the environment variables. + +2014-02-12 18:21 Christos Zoulas + + * Count recursion levels through indirect magic + +2014-02-11 10:40 Christos Zoulas + + * Prevent infinite recursion on files with indirect offsets of 0 + +2014-01-30 21:00 Christos Zoulas + + * Add -E flag that makes file print filesystem errors to stderr + and exit. + +2014-01-08 17:20 Christos Zoulas + + * mime printing could print results from multiple magic entries + if there were multiple matches. + * in some cases overflow was not detected when computing offsets + in softmagic. + +2013-12-05 12:00 Christos Zoulas + + * use strcasestr() to for cdf strings + * reset to the "C" locale while doing regex operations, or case + insensitive comparisons; this is provisional + +2013-11-19 20:10 Christos Zoulas + + * always leave magic file loaded, don't unload for magic_check, etc. + * fix default encoding to binary instead of unknown which broke recently + * handle empty and one byte files, less specially so that + --mime-encoding does not break completely. + ` +2013-11-06 14:40 Christos Zoulas + + * fix erroneous non-zero exit code from non-existent file and message + +2013-10-29 14:25 Christos Zoulas + + * add CDF MSI file detection (Guy Helmer) + +2013-09-03 11:56 Christos Zoulas + + * Don't mix errors and regular output if there was an error + * in magic_descriptor() don't close the file and try to restore + its position + +2013-05-30 17:25 Christos Zoulas + + * Don't treat magic as an error if offset was past EOF (Christoph Biedl) + +2013-05-28 17:25 Christos Zoulas + + * Fix spacing issues in softmagic and elf (Jan Kaluza) + +2013-05-02 18:00 Christos Zoulas + + * Fix segmentation fault with multiple magic_load commands. + +2013-04-22 11:20 Christos Zoulas + + * The way "default" was implemented was not very useful + because the "if something was printed at that level" + was not easily controlled by the user, and the format + was bound to a string which is too restrictive. Add + a "clear" for that level keyword and make "default" + void. This way one can do: + + >>13 clear x + >>13 lelong 1 foo + >>13 lelong 2 bar + >>13 default x + >>>13 lelong x unknown %x + +2013-03-25 13:20 Christos Zoulas + + * disallow strength setting in "name" entries + +2013-03-06 21:24 Christos Zoulas + + * fix recursive magic separator printing + +2013-02-26 19:28 Christos Zoulas + + * limit recursion level for mget + * fix pread() related breakage in cdf + * handle offsets properly in recursive "use" + +2013-02-18 10:39 Christos Zoulas + + * add elf reading of debug info to determine if file is stripped + (Jan Kaluza) + * use pread() + +2013-01-25 18:05 Christos Zoulas + + * change mime description size from 64 to 80 to accommodate OOXML. + +2013-01-11 14:50 Christos Zoulas + + * Warn about inconsistent continuation levels. + * Change fsmagic to add a space after it prints. + +2013-01-10 21:00 Christos Zoulas + + * Make getline public so that file can link against it. + Perhaps it is better to rename it, or hide it differently. + Fixes builds on platforms that do not provide it. + +2013-01-07 16:30 Christos Zoulas + + * Add SuS d{,1,2,4,8}, u{,1,2,4,8} and document + what long, int, short, etc is (Guy Harris) + +2013-01-06 11:20 Christos Zoulas + + * add magic_version function and constant + * Redo memory allocation and de-allocation. + (prevents double frees on non mmap platforms) + * Fix bug with name/use having to do with passing + found state from the parent to the child and back. + +2012-12-19 8:47 Christos Zoulas + + * Only print elf capabilities for archs we know (Jan Kaluza) + +2012-10-30 19:14 Christos Zoulas + + * Add "name" and "use" file types in order to look + inside mach-o files. + +2012-09-06 10:40 Christos Zoulas + + * make --version exit 0 (Matthew Schultz) + * add string/T (Jan Kaluza) + +2012-08-09 2:15 Christos Zoulas + + * add z and t modifiers for our own vasprintf + * search for $HOME/.magic.mgc if it is there first + * fix reads from a pipe, and preserve errno + +2012-05-15 13:12 Christos Zoulas + + * use ctime_r, asctime_r + +2012-04-06 17:18 Christos Zoulas + + * Fixes for indirect offsets to handle apple disk formats + +2012-04-03 18:26 Christos Zoulas + + * Add windows date field types + * More info for windows shortcuts (incomplete) + +2012-02-20 17:33 Christos Zoulas + + * Fix CDF parsing issues found by CERT's fuzzing tool (Will Dormann) + +2011-12-15 12:17 Chris Metcalf + + * Support Tilera architectures (tile64, tilepro, tilegx). + +2011-12-16 16:33 Reuben Thomas + + * Add magic for /usr/bin/env Perl scripts + * Weaken generic script magic to avoid clashing with + language-specific magic. + +2011-12-08 13:37 Reuben Thomas + + * Simplify if (p) free(p) to free(p). + +2011-12-08 13:07 Reuben Thomas + + * Remove hardwired token finding (names.h), turning it into soft + magic. Patterns are either anchored regexs or search/8192. English + language detection and PL/1 detection have been removed as they + were too fragile. -e tokens is still accepted for backwards + compatibility. + * Move 3ds patterns (which are commented out anyway) into autodesk + (they were, oddly, in c-lang). + +2011-12-06 00:16 Reuben Thomas + + * Tweak strength of generic hash-bang detectors to be less than + specific ones. + * Make an inconsistent description of Python scripts consistent. + +2011-12-05 23:58 Reuben Thomas + + * Fix minor error in file(1). + +2011-11-05 00:00 Reuben Thomas + + * Fix issue #150 (I hope). + +2011-09-22 12:57 Christos Zoulas + + * Python3 binding fixes from Kelly Anderson + +2011-09-20 11:32 Christos Zoulas + + * If a string type magic entry is marked as text or binary + only match text files against text entries and binary + files against binary entries. + +2011-09-01 12:12 Christos Zoulas + + * Don't wait for any subprocess, just the one we forked. + +2011-08-26 16:40 Christos Zoulas + + * If the application name is not set in a cdf file, try to see + if it has a directory with the application name on it. + +2011-08-17 14:32 Christos Zoulas + + * Fix ELF lseek(2) madness. Inspired by PR/134 by Jan Kaluza + +2011-08-14 09:03 Christos Zoulas + + * Don't use variable string formats. + +2011-07-12 12:32 Reuben Thomas + + * Fix detection of Zip files (Mantis #128). + * Make some minor improvements to file(1). + * Rename MIME types for filesystem objects for consistency with + xdg-utils. Typically this means that application/x-foo becomes + inode/foo, but some names also change slightly, e.g. + application/x-character-device becomes inode/chardevice. + +2011-05-10 20:57 Christos Zoulas + + * fix mingw compilation (Abradoks) + +2011-05-10 20:57 Christos Zoulas + + * remove patchlevel.h + * Fix read past allocated memory caused by double-incrementing + a pointer in a loop (reported by Roberto Maar) + +2011-03-30 15:45 Christos Zoulas + + * Fix cdf string buffer setting (Sven Anders) + +2011-03-20 16:35 Christos Zoulas + + * Eliminate MAXPATHLEN and use dynamic allocation for + path and file buffers. + +2011-03-15 18:15 Christos Zoulas + + * binary tests on magic entries with masks could spuriously + get converted to ascii. + +2011-03-12 18:06 Reuben Thomas + + * Improve file.man (remove BUGS, present email addresses consistently). + +2011-03-07 19:38 Christos Zoulas + + * add lrzip support (from Ville Skytta) + +2011-02-10 16:36 Christos Zoulas + + * fix CDF bounds checking (Guy Helmer) + +2011-02-10 12:03 Christos Zoulas + + * add cdf_ctime() that prints a meaningful error when time cannot + be converted. + +2011-02-02 20:40 Christos Zoulas + + * help and version output to stdout. + + * When matching softmagic for ascii files, don't just print + the softmagic classification, keep going and print the + text classification too. This fixes broken troff files when + we moved them from keyword recognition to softmagic + (they stopped printing "with CRLF" etc.) + Reported by Doug McIlroy. + +2011-01-16 19:31 Reuben Thomas + + * Fix two potential buffer overruns in apprentice_list. + +2011-01-14 22:33 Reuben Thomas + + * New Python binding in pure Python. + * Update libmagic(3). + +2011-01-06 21:40 Reuben Thomas + + * Fix Python bindings (including recent Python 3 compatibility + update). + +2011-01-04 18:43 Reuben Thomas + + * magic/Makefile.am: make it easier to recover from magic build failures. + * Fix pstring length specifier parsing to avoid generating invalid + magic files. + * Add pstring length "J" (for "JPEG") to specify that the length + include itself. + * Fix JPEG comment parsing at last using pstring/HJ! + * Ignore section 5 man pages in doc/.cvsignore. + +2010-12-22 13:12 Christos Zoulas + + * Add pstring/BHhLl to specify the type of the length of pascal + strings. + +2010-11-26 18:39 Reuben Thomas + + * Fix "-e soft": it was ignored when softmagic was called + during asciimagic. + * Improve comments and use "unsigned char" in tar.h/is_tar.c. + +2010-11-05 17:26 Reuben Thomas + + * Make bug reporting addresses more visible. + +2010-11-01 18:35 Reuben Thomas + + * Add tcl magic from Gustaf Neumann + +2010-10-24 10:42 Christos Zoulas + + * Fix the whitespace comparing code (Christopher Chittleborough) + +2010-10-06 21:05 Christos Zoulas + + * allow string/t to work (Jan Kaluza) + +2010-09-20 22:11 Reuben Thomas + + * Apply some patches from Ubuntu and Fedora. + +2010-09-20 21:16 Reuben Thomas + + * Apply all patches from Debian package 5.04-6 which have not + already been applied and are not Debian-specific. + +2010-09-20 15:24 Reuben Thomas + + * Minor security fix to softmagic.c (don't use untrusted + string as printf format). + +2010-07-21 12:20 Christos Zoulas + + * MINGW32 portability from LRN + + * Don't warn about escaping magic regex chars when we are in a regex. + +2010-07-19 10:55 Christos Zoulas + + * Only try to print prpsinfo for core files. (Jan Kaluza) + +2010-04-22 12:55 Christos Zoulas + + * Try more elf offsets for Debian core files. (Arnaud Giersch) + +2010-02-20 15:18 Reuben Thomas + + * Clarify which sort of CDF we mean. + +2010-02-14 22:58 Reuben Thomas + + * Re-jig Zip file type magic so that unsupported special + Zip types (those with "mimetype" at offset 30) can be + recognized. + +2010-02-02 21:50 Reuben Thomas + + * Add support for OCF (EPUB) files (application/epub+zip) + +2010-01-28 18:25 Christos Zoulas + + * Fix core-dump from unbound loop: + https://bugzilla.redhat.com/show_bug.cgi?id=533245 + +2010-01-22 15:45 Christos Zoulas + + * print proper mime for crystal reports file + + * print the last summary information of a cdf document, not the + first so that nested documents print the right info + +2010-01-16 18:42 Charles Longeau + + * bring back some fixes from OpenBSD: + - make gcc2 builds file + - fix typos in a magic file comment + +2009-11-17 18:35 Christos Zoulas + + * ctime/asctime can return NULL on some OS's although + they should not (Toshit Antani) + +2009-09-14 13:49 Christos Zoulas + + * Centralize magic path handling routines and remove the + special-casing from file.c so that the python module for + example comes up with the same magic path (Fixes ~/.magic + handling) (from Gab) + +2009-09-11 23:38 Reuben Thomas + + * When magic argument is a directory, read the files in + strcmp-sorted order (fixes Debian bug #488562 and our own FIXME). + +2009-09-11 13:11 Reuben Thomas + + * Combine overlapping epoc and psion magic files into one (epoc). + + * Add some more EPOC MIME types. + +2009-08-19 15:55 Christos Zoulas + + * Fix 3 bugs (From Ian Darwin): + - file_showstr could move one past the end of the array + - parse_apple did not nul terminate the string in the overflow case + - parse_mime truncated the wrong string in the overflow case + +2009-08-12 12:28 Robert Byrnes + + * Include Localstuff when compiling magic. + +2009-07-15 10:05 Christos Zoulas + + * Fix logic for including mygetopts.h + + * Make cdf.c compile again with debugging + + * Add the necessary field handling for crystal reports files to work + +2009-06-23 01:34 Reuben Thomas + + * Stop "(if" identifying Lisp files, that's plain dumb! + +2009-06-09 22:13 Reuben Thomas + + * Add a couple of missing MP3 MIME types. + +2009-05-27 23:00 Reuben Thomas + + * Add full range of hash-bang tests for Python and Ruby. + + * Add MIME types for Python and Ruby scripts. + +2009-05-13 10:44 Christos Zoulas + + * off by one in parsing hw capabilities in elf + (Cheng Renquan) + +2009-05-08 13:40 Christos Zoulas + + * lint fixes and more from NetBSD + +2009-05-06 10:25 Christos Zoulas + + * Avoid null dereference in cdf code (Drew Yao) + + * More cdf bounds checks and overflow checks + +2009-05-01 18:37 Christos Zoulas + + * Buffer overflow fixes from Drew Yao + +2009-04-30 17:10 Christos Zoulas + + * Fix more cdf lossage. All the documents I have + right now print the correct information. + +2009-03-27 18:43 Christos Zoulas + + * don't print \012- separators in the same magic entry + if it consists of multiple magic printing lines. + +2009-03-23 10:20 Christos Zoulas + + * Avoid file descriptor leak in compress code from + (Daniel Novotny) + +2009-03-18 16:50 Christos Zoulas + + * Allow escaping of relation characters, so that we can say \^[A-Z] + and the ^ is not eaten as a relation char. + + * Fix troff and fortran to their previous glory using + regex. This was broken since their removel from ascmagic. + +2009-03-10 16:50 Christos Zoulas + + * don't use strlen in strndup() (Toby Peterson) + +2009-03-10 7:45 Christos Zoulas + + * avoid c99 syntax. + +2009-02-23 15:45 Christos Zoulas + + * make the cdf code use the buffer first if available, + and then the fd code. + +2009-02-13 13:45 Christos Zoulas + + * look for struct option to determine if getopt.h is usable for IRIX. + + * sanitize cdf document strings + +2009-02-04 13:25 Christos Zoulas + *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sun Jul 29 05:51:15 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 549DE104C358; Sun, 29 Jul 2018 05:51:15 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A86B729A6; Sun, 29 Jul 2018 05:51:15 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C5C9311AF0; Sun, 29 Jul 2018 05:51:14 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6T5pEfl020106; Sun, 29 Jul 2018 05:51:14 GMT (envelope-from eadler@FreeBSD.org) Received: (from eadler@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6T5pE7K020105; Sun, 29 Jul 2018 05:51:14 GMT (envelope-from eadler@FreeBSD.org) Message-Id: <201807290551.w6T5pE7K020105@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: eadler set sender to eadler@FreeBSD.org using -f From: Eitan Adler Date: Sun, 29 Jul 2018 05:51:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336861 - vendor/file/5.34 X-SVN-Group: vendor X-SVN-Commit-Author: eadler X-SVN-Commit-Paths: vendor/file/5.34 X-SVN-Commit-Revision: 336861 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jul 2018 05:51:15 -0000 Author: eadler Date: Sun Jul 29 05:51:14 2018 New Revision: 336861 URL: https://svnweb.freebsd.org/changeset/base/336861 Log: Remove 5.34 I inverted the order of the import. Deleted: vendor/file/5.34/ From owner-svn-src-vendor@freebsd.org Sun Jul 29 05:53:51 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8DB5C104C4BF; Sun, 29 Jul 2018 05:53:51 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F11372D72; Sun, 29 Jul 2018 05:53:51 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 209B811C53; Sun, 29 Jul 2018 05:53:51 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6T5rpdu024291; Sun, 29 Jul 2018 05:53:51 GMT (envelope-from eadler@FreeBSD.org) Received: (from eadler@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6T5roAQ024285; Sun, 29 Jul 2018 05:53:50 GMT (envelope-from eadler@FreeBSD.org) Message-Id: <201807290553.w6T5roAQ024285@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: eadler set sender to eadler@FreeBSD.org using -f From: Eitan Adler Date: Sun, 29 Jul 2018 05:53:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336862 - in vendor/file/dist: . doc magic magic/Magdir python src tests X-SVN-Group: vendor X-SVN-Commit-Author: eadler X-SVN-Commit-Paths: in vendor/file/dist: . doc magic magic/Magdir python src tests X-SVN-Commit-Revision: 336862 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jul 2018 05:53:52 -0000 Author: eadler Date: Sun Jul 29 05:53:49 2018 New Revision: 336862 URL: https://svnweb.freebsd.org/changeset/base/336862 Log: vendor/file: add 5.34 gpg: Good signature from "Christos Zoulas (personal key) " [unknown] gpg: aka "Christos Zoulas (NetBSD key) " [unknown] Obtained from: ftp://ftp.astron.com/pub/file/ Added: vendor/file/dist/magic/Magdir/dataone vendor/file/dist/python/CHANGELOG.md vendor/file/dist/python/LICENSE vendor/file/dist/python/README.md vendor/file/dist/python/tests.py (contents, props changed) vendor/file/dist/tests/JW07022A.mp3.result vendor/file/dist/tests/JW07022A.mp3.testfile (contents, props changed) Deleted: vendor/file/dist/python/README Modified: vendor/file/dist/ChangeLog vendor/file/dist/Makefile.in vendor/file/dist/README vendor/file/dist/aclocal.m4 vendor/file/dist/compile vendor/file/dist/config.guess vendor/file/dist/config.h.in vendor/file/dist/config.sub vendor/file/dist/configure vendor/file/dist/configure.ac vendor/file/dist/depcomp vendor/file/dist/doc/Makefile.in vendor/file/dist/doc/file.man vendor/file/dist/doc/magic.man vendor/file/dist/install-sh vendor/file/dist/ltmain.sh vendor/file/dist/magic/Magdir/animation vendor/file/dist/magic/Magdir/archive vendor/file/dist/magic/Magdir/audio vendor/file/dist/magic/Magdir/cad vendor/file/dist/magic/Magdir/console vendor/file/dist/magic/Magdir/database vendor/file/dist/magic/Magdir/dbpf vendor/file/dist/magic/Magdir/dump vendor/file/dist/magic/Magdir/elf vendor/file/dist/magic/Magdir/fsav vendor/file/dist/magic/Magdir/images vendor/file/dist/magic/Magdir/linux vendor/file/dist/magic/Magdir/make vendor/file/dist/magic/Magdir/measure vendor/file/dist/magic/Magdir/msdos vendor/file/dist/magic/Magdir/msooxml vendor/file/dist/magic/Magdir/pdf vendor/file/dist/magic/Magdir/sgi vendor/file/dist/magic/Makefile.am vendor/file/dist/magic/Makefile.in vendor/file/dist/missing vendor/file/dist/python/Makefile.am vendor/file/dist/python/Makefile.in vendor/file/dist/python/magic.py vendor/file/dist/python/setup.py vendor/file/dist/src/Makefile.in vendor/file/dist/src/apprentice.c vendor/file/dist/src/compress.c vendor/file/dist/src/der.c vendor/file/dist/src/file.h vendor/file/dist/src/funcs.c vendor/file/dist/src/readcdf.c vendor/file/dist/src/readelf.c vendor/file/dist/src/readelf.h vendor/file/dist/src/seccomp.c vendor/file/dist/src/softmagic.c vendor/file/dist/tests/Makefile.am vendor/file/dist/tests/Makefile.in Modified: vendor/file/dist/ChangeLog ============================================================================== --- vendor/file/dist/ChangeLog Sun Jul 29 05:51:14 2018 (r336861) +++ vendor/file/dist/ChangeLog Sun Jul 29 05:53:49 2018 (r336862) @@ -1,3 +1,15 @@ +2018-07-25 8:50 Christos Zoulas + + * release 5.34 + +2018-06-22 16:38 Christos Zoulas + + * Add Quad indirect offsets + +2018-05-24 14:10 Christos Zoulas + + * Enable parsing of ELF dynamic sections to handle PIE better + 2018-04-15 14:52 Christos Zoulas * release 5.33 Modified: vendor/file/dist/Makefile.in ============================================================================== --- vendor/file/dist/Makefile.in Sun Jul 29 05:51:14 2018 (r336861) +++ vendor/file/dist/Makefile.in Sun Jul 29 05:53:49 2018 (r336862) @@ -1,7 +1,7 @@ -# Makefile.in generated by automake 1.13.1 from Makefile.am. +# Makefile.in generated by automake 1.15 from Makefile.am. # @configure_input@ -# Copyright (C) 1994-2012 Free Software Foundation, Inc. +# Copyright (C) 1994-2014 Free Software Foundation, Inc. # This Makefile.in is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -14,23 +14,61 @@ @SET_MAKE@ VPATH = @srcdir@ -am__make_dryrun = \ - { \ - am__dry=no; \ +am__is_gnu_make = { \ + if test -z '$(MAKELEVEL)'; then \ + false; \ + elif test -n '$(MAKE_HOST)'; then \ + true; \ + elif test -n '$(MAKE_VERSION)' && test -n '$(CURDIR)'; then \ + true; \ + else \ + false; \ + fi; \ +} +am__make_running_with_option = \ + case $${target_option-} in \ + ?) ;; \ + *) echo "am__make_running_with_option: internal error: invalid" \ + "target option '$${target_option-}' specified" >&2; \ + exit 1;; \ + esac; \ + has_opt=no; \ + sane_makeflags=$$MAKEFLAGS; \ + if $(am__is_gnu_make); then \ + sane_makeflags=$$MFLAGS; \ + else \ case $$MAKEFLAGS in \ *\\[\ \ ]*) \ - echo 'am--echo: ; @echo "AM" OK' | $(MAKE) -f - 2>/dev/null \ - | grep '^AM OK$$' >/dev/null || am__dry=yes;; \ - *) \ - for am__flg in $$MAKEFLAGS; do \ - case $$am__flg in \ - *=*|--*) ;; \ - *n*) am__dry=yes; break;; \ - esac; \ - done;; \ + bs=\\; \ + sane_makeflags=`printf '%s\n' "$$MAKEFLAGS" \ + | sed "s/$$bs$$bs[$$bs $$bs ]*//g"`;; \ esac; \ - test $$am__dry = yes; \ - } + fi; \ + skip_next=no; \ + strip_trailopt () \ + { \ + flg=`printf '%s\n' "$$flg" | sed "s/$$1.*$$//"`; \ + }; \ + for flg in $$sane_makeflags; do \ + test $$skip_next = yes && { skip_next=no; continue; }; \ + case $$flg in \ + *=*|--*) continue;; \ + -*I) strip_trailopt 'I'; skip_next=yes;; \ + -*I?*) strip_trailopt 'I';; \ + -*O) strip_trailopt 'O'; skip_next=yes;; \ + -*O?*) strip_trailopt 'O';; \ + -*l) strip_trailopt 'l'; skip_next=yes;; \ + -*l?*) strip_trailopt 'l';; \ + -[dEDm]) skip_next=yes;; \ + -[JT]) skip_next=yes;; \ + esac; \ + case $$flg in \ + *$$target_option*) has_opt=yes; break;; \ + esac; \ + done; \ + test $$has_opt = yes +am__make_dryrun = (target_option=n; $(am__make_running_with_option)) +am__make_keepgoing = (target_option=k; $(am__make_running_with_option)) pkgincludedir = $(includedir)/@PACKAGE@ pkglibdir = $(libdir)/@PACKAGE@ pkglibexecdir = $(libexecdir)/@PACKAGE@ @@ -49,11 +87,6 @@ POST_UNINSTALL = : build_triplet = @build@ host_triplet = @host@ subdir = . -DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/Makefile.am \ - $(top_srcdir)/configure $(am__configure_deps) \ - $(srcdir)/config.h.in AUTHORS COPYING ChangeLog INSTALL NEWS \ - README TODO compile config.guess config.sub depcomp install-sh \ - missing ltmain.sh ACLOCAL_M4 = $(top_srcdir)/aclocal.m4 am__aclocal_m4_deps = $(top_srcdir)/m4/libtool.m4 \ $(top_srcdir)/m4/ltoptions.m4 $(top_srcdir)/m4/ltsugar.m4 \ @@ -61,6 +94,8 @@ am__aclocal_m4_deps = $(top_srcdir)/m4/libtool.m4 \ $(top_srcdir)/acinclude.m4 $(top_srcdir)/configure.ac am__configure_deps = $(am__aclocal_m4_deps) $(CONFIGURE_DEPENDENCIES) \ $(ACLOCAL_M4) +DIST_COMMON = $(srcdir)/Makefile.am $(top_srcdir)/configure \ + $(am__configure_deps) $(am__DIST_COMMON) am__CONFIG_DISTCLEAN_FILES = config.status config.cache config.log \ configure.lineno config.status.lineno mkinstalldirs = $(install_sh) -d @@ -124,6 +159,9 @@ ETAGS = etags CTAGS = ctags CSCOPE = cscope DIST_SUBDIRS = $(SUBDIRS) +am__DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/config.h.in AUTHORS \ + COPYING ChangeLog INSTALL NEWS README TODO compile \ + config.guess config.sub depcomp install-sh ltmain.sh missing DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) distdir = $(PACKAGE)-$(VERSION) top_distdir = $(distdir) @@ -178,6 +216,7 @@ AWK = @AWK@ CC = @CC@ CCDEPMODE = @CCDEPMODE@ CFLAGS = @CFLAGS@ +CFLAG_VISIBILITY = @CFLAG_VISIBILITY@ CPP = @CPP@ CPPFLAGS = @CPPFLAGS@ CYGPATH_W = @CYGPATH_W@ @@ -193,6 +232,7 @@ EGREP = @EGREP@ EXEEXT = @EXEEXT@ FGREP = @FGREP@ GREP = @GREP@ +HAVE_VISIBILITY = @HAVE_VISIBILITY@ INSTALL = @INSTALL@ INSTALL_DATA = @INSTALL_DATA@ INSTALL_PROGRAM = @INSTALL_PROGRAM@ @@ -306,7 +346,6 @@ $(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__c echo ' cd $(top_srcdir) && $(AUTOMAKE) --foreign Makefile'; \ $(am__cd) $(top_srcdir) && \ $(AUTOMAKE) --foreign Makefile -.PRECIOUS: Makefile Makefile: $(srcdir)/Makefile.in $(top_builddir)/config.status @case '$?' in \ *config.status*) \ @@ -327,8 +366,8 @@ $(ACLOCAL_M4): $(am__aclocal_m4_deps) $(am__aclocal_m4_deps): config.h: stamp-h1 - @if test ! -f $@; then rm -f stamp-h1; else :; fi - @if test ! -f $@; then $(MAKE) $(AM_MAKEFLAGS) stamp-h1; else :; fi + @test -f $@ || rm -f stamp-h1 + @test -f $@ || $(MAKE) $(AM_MAKEFLAGS) stamp-h1 stamp-h1: $(srcdir)/config.h.in $(top_builddir)/config.status @rm -f stamp-h1 @@ -357,13 +396,12 @@ distclean-libtool: # (which will cause the Makefiles to be regenerated when you run 'make'); # (2) otherwise, pass the desired values on the 'make' command line. $(am__recursive_targets): - @fail= failcom='exit 1'; \ - for f in x $$MAKEFLAGS; do \ - case $$f in \ - *=* | --[!k]*);; \ - *k*) failcom='fail=yes';; \ - esac; \ - done; \ + @fail=; \ + if $(am__make_keepgoing); then \ + failcom='fail=yes'; \ + else \ + failcom='exit 1'; \ + fi; \ dot_seen=no; \ target=`echo $@ | sed s/-recursive//`; \ case "$@" in \ @@ -538,10 +576,16 @@ dist-xz: distdir $(am__post_remove_distdir) dist-tarZ: distdir + @echo WARNING: "Support for distribution archives compressed with" \ + "legacy program 'compress' is deprecated." >&2 + @echo WARNING: "It will be removed altogether in Automake 2.0" >&2 tardir=$(distdir) && $(am__tar) | compress -c >$(distdir).tar.Z $(am__post_remove_distdir) dist-shar: distdir + @echo WARNING: "Support for shar distribution archives is" \ + "deprecated." >&2 + @echo WARNING: "It will be removed altogether in Automake 2.0" >&2 shar $(distdir) | GZIP=$(GZIP_ENV) gzip -c >$(distdir).shar.gz $(am__post_remove_distdir) @@ -576,16 +620,17 @@ distcheck: dist esac chmod -R a-w $(distdir) chmod u+w $(distdir) - mkdir $(distdir)/_build $(distdir)/_inst + mkdir $(distdir)/_build $(distdir)/_build/sub $(distdir)/_inst chmod a-w $(distdir) test -d $(distdir)/_build || exit 0; \ dc_install_base=`$(am__cd) $(distdir)/_inst && pwd | sed -e 's,^[^:\\/]:[\\/],/,'` \ && dc_destdir="$${TMPDIR-/tmp}/am-dc-$$$$/" \ && am__cwd=`pwd` \ - && $(am__cd) $(distdir)/_build \ - && ../configure --srcdir=.. --prefix="$$dc_install_base" \ + && $(am__cd) $(distdir)/_build/sub \ + && ../../configure \ $(AM_DISTCHECK_CONFIGURE_FLAGS) \ $(DISTCHECK_CONFIGURE_FLAGS) \ + --srcdir=../.. --prefix="$$dc_install_base" \ && $(MAKE) $(AM_MAKEFLAGS) \ && $(MAKE) $(AM_MAKEFLAGS) dvi \ && $(MAKE) $(AM_MAKEFLAGS) check \ @@ -761,6 +806,8 @@ uninstall-am: maintainer-clean-generic mostlyclean mostlyclean-generic \ mostlyclean-libtool pdf pdf-am ps ps-am tags tags-am uninstall \ uninstall-am + +.PRECIOUS: Makefile # Tell versions [3.59,3.63) of GNU make to not export all variables. Modified: vendor/file/dist/README ============================================================================== --- vendor/file/dist/README Sun Jul 29 05:51:14 2018 (r336861) +++ vendor/file/dist/README Sun Jul 29 05:53:49 2018 (r336862) @@ -1,10 +1,10 @@ ## README for file(1) Command ## - @(#) $File: README,v 1.53 2018/03/11 13:06:47 glen Exp $ + @(#) $File: README,v 1.54 2018/05/30 03:06:56 christos Exp $ -Mailing List: file@mx.gw.com [currently down] -Mailing List archives: http://mx.gw.com/pipermail/file/ [currently down] -Bug tracker: http://bugs.gw.com/ [currently down] +Mailing List: file@astron.com +Mailing List archives: http://mailman.astron.com/pipermail/file/ +Bug tracker: http://bugs.astron.com/ E-mail: christos@astron.com Build Status: https://travis-ci.org/file/file Modified: vendor/file/dist/aclocal.m4 ============================================================================== --- vendor/file/dist/aclocal.m4 Sun Jul 29 05:51:14 2018 (r336861) +++ vendor/file/dist/aclocal.m4 Sun Jul 29 05:53:49 2018 (r336862) @@ -1,6 +1,6 @@ -# generated automatically by aclocal 1.13.1 -*- Autoconf -*- +# generated automatically by aclocal 1.15 -*- Autoconf -*- -# Copyright (C) 1996-2012 Free Software Foundation, Inc. +# Copyright (C) 1996-2014 Free Software Foundation, Inc. # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -20,7 +20,85 @@ You have another version of autoconf. It may work, bu If you have problems, you may need to regenerate the build system entirely. To do so, use the procedure documented by the package, typically 'autoreconf'.])]) -# Copyright (C) 2002-2013 Free Software Foundation, Inc. +# visibility.m4 serial 5 (gettext-0.18.2) +dnl Copyright (C) 2005, 2008, 2010-2016 Free Software Foundation, Inc. +dnl This file is free software; the Free Software Foundation +dnl gives unlimited permission to copy and/or distribute it, +dnl with or without modifications, as long as this notice is preserved. + +dnl From Bruno Haible. + +dnl Tests whether the compiler supports the command-line option +dnl -fvisibility=hidden and the function and variable attributes +dnl __attribute__((__visibility__("hidden"))) and +dnl __attribute__((__visibility__("default"))). +dnl Does *not* test for __visibility__("protected") - which has tricky +dnl semantics (see the 'vismain' test in glibc) and does not exist e.g. on +dnl Mac OS X. +dnl Does *not* test for __visibility__("internal") - which has processor +dnl dependent semantics. +dnl Does *not* test for #pragma GCC visibility push(hidden) - which is +dnl "really only recommended for legacy code". +dnl Set the variable CFLAG_VISIBILITY. +dnl Defines and sets the variable HAVE_VISIBILITY. + +AC_DEFUN([gl_VISIBILITY], +[ + AC_REQUIRE([AC_PROG_CC]) + CFLAG_VISIBILITY= + HAVE_VISIBILITY=0 + if test -n "$GCC"; then + dnl First, check whether -Werror can be added to the command line, or + dnl whether it leads to an error because of some other option that the + dnl user has put into $CC $CFLAGS $CPPFLAGS. + AC_MSG_CHECKING([whether the -Werror option is usable]) + AC_CACHE_VAL([gl_cv_cc_vis_werror], [ + gl_save_CFLAGS="$CFLAGS" + CFLAGS="$CFLAGS -Werror" + AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM([[]], [[]])], + [gl_cv_cc_vis_werror=yes], + [gl_cv_cc_vis_werror=no]) + CFLAGS="$gl_save_CFLAGS"]) + AC_MSG_RESULT([$gl_cv_cc_vis_werror]) + dnl Now check whether visibility declarations are supported. + AC_MSG_CHECKING([for simple visibility declarations]) + AC_CACHE_VAL([gl_cv_cc_visibility], [ + gl_save_CFLAGS="$CFLAGS" + CFLAGS="$CFLAGS -fvisibility=hidden" + dnl We use the option -Werror and a function dummyfunc, because on some + dnl platforms (Cygwin 1.7) the use of -fvisibility triggers a warning + dnl "visibility attribute not supported in this configuration; ignored" + dnl at the first function definition in every compilation unit, and we + dnl don't want to use the option in this case. + if test $gl_cv_cc_vis_werror = yes; then + CFLAGS="$CFLAGS -Werror" + fi + AC_COMPILE_IFELSE( + [AC_LANG_PROGRAM( + [[extern __attribute__((__visibility__("hidden"))) int hiddenvar; + extern __attribute__((__visibility__("default"))) int exportedvar; + extern __attribute__((__visibility__("hidden"))) int hiddenfunc (void); + extern __attribute__((__visibility__("default"))) int exportedfunc (void); + void dummyfunc (void) {} + ]], + [[]])], + [gl_cv_cc_visibility=yes], + [gl_cv_cc_visibility=no]) + CFLAGS="$gl_save_CFLAGS"]) + AC_MSG_RESULT([$gl_cv_cc_visibility]) + if test $gl_cv_cc_visibility = yes; then + CFLAG_VISIBILITY="-fvisibility=hidden" + HAVE_VISIBILITY=1 + fi + fi + AC_SUBST([CFLAG_VISIBILITY]) + AC_SUBST([HAVE_VISIBILITY]) + AC_DEFINE_UNQUOTED([HAVE_VISIBILITY], [$HAVE_VISIBILITY], + [Define to 1 or 0, depending whether the compiler supports simple visibility declarations.]) +]) + +# Copyright (C) 2002-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -32,10 +110,10 @@ To do so, use the procedure documented by the package, # generated from the m4 files accompanying Automake X.Y. # (This private macro should not be called outside this file.) AC_DEFUN([AM_AUTOMAKE_VERSION], -[am__api_version='1.13' +[am__api_version='1.15' dnl Some users find AM_AUTOMAKE_VERSION and mistake it for a way to dnl require some minimum version. Point them to the right macro. -m4_if([$1], [1.13.1], [], +m4_if([$1], [1.15], [], [AC_FATAL([Do not call $0, use AM_INIT_AUTOMAKE([$1]).])])dnl ]) @@ -51,14 +129,14 @@ m4_define([_AM_AUTOCONF_VERSION], []) # Call AM_AUTOMAKE_VERSION and AM_AUTOMAKE_VERSION so they can be traced. # This function is AC_REQUIREd by AM_INIT_AUTOMAKE. AC_DEFUN([AM_SET_CURRENT_AUTOMAKE_VERSION], -[AM_AUTOMAKE_VERSION([1.13.1])dnl +[AM_AUTOMAKE_VERSION([1.15])dnl m4_ifndef([AC_AUTOCONF_VERSION], [m4_copy([m4_PACKAGE_VERSION], [AC_AUTOCONF_VERSION])])dnl _AM_AUTOCONF_VERSION(m4_defn([AC_AUTOCONF_VERSION]))]) # AM_AUX_DIR_EXPAND -*- Autoconf -*- -# Copyright (C) 2001-2013 Free Software Foundation, Inc. +# Copyright (C) 2001-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -103,15 +181,14 @@ _AM_AUTOCONF_VERSION(m4_defn([AC_AUTOCONF_VERSION]))]) # configured tree to be moved without reconfiguration. AC_DEFUN([AM_AUX_DIR_EXPAND], -[dnl Rely on autoconf to set up CDPATH properly. -AC_PREREQ([2.50])dnl -# expand $ac_aux_dir to an absolute path -am_aux_dir=`cd $ac_aux_dir && pwd` +[AC_REQUIRE([AC_CONFIG_AUX_DIR_DEFAULT])dnl +# Expand $ac_aux_dir to an absolute path. +am_aux_dir=`cd "$ac_aux_dir" && pwd` ]) # AM_CONDITIONAL -*- Autoconf -*- -# Copyright (C) 1997-2013 Free Software Foundation, Inc. +# Copyright (C) 1997-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -142,7 +219,7 @@ AC_CONFIG_COMMANDS_PRE( Usually this means the macro was only invoked conditionally.]]) fi])]) -# Copyright (C) 1999-2013 Free Software Foundation, Inc. +# Copyright (C) 1999-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -333,7 +410,7 @@ _AM_SUBST_NOTMAKE([am__nodep])dnl # Generate code to set up dependency tracking. -*- Autoconf -*- -# Copyright (C) 1999-2013 Free Software Foundation, Inc. +# Copyright (C) 1999-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -373,7 +450,7 @@ AC_DEFUN([_AM_OUTPUT_DEPENDENCY_COMMANDS], DEPDIR=`sed -n 's/^DEPDIR = //p' < "$mf"` test -z "$DEPDIR" && continue am__include=`sed -n 's/^am__include = //p' < "$mf"` - test -z "am__include" && continue + test -z "$am__include" && continue am__quote=`sed -n 's/^am__quote = //p' < "$mf"` # Find all dependency output files, they are included files with # $(DEPDIR) in their names. We invoke sed twice because it is the @@ -409,7 +486,7 @@ AC_DEFUN([AM_OUTPUT_DEPENDENCY_COMMANDS], # Do all the work for Automake. -*- Autoconf -*- -# Copyright (C) 1996-2013 Free Software Foundation, Inc. +# Copyright (C) 1996-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -418,6 +495,12 @@ AC_DEFUN([AM_OUTPUT_DEPENDENCY_COMMANDS], # This macro actually does too much. Some checks are only needed if # your package does certain things. But this isn't really a big deal. +dnl Redefine AC_PROG_CC to automatically invoke _AM_PROG_CC_C_O. +m4_define([AC_PROG_CC], +m4_defn([AC_PROG_CC]) +[_AM_PROG_CC_C_O +]) + # AM_INIT_AUTOMAKE(PACKAGE, VERSION, [NO-DEFINE]) # AM_INIT_AUTOMAKE([OPTIONS]) # ----------------------------------------------- @@ -493,8 +576,8 @@ AC_REQUIRE([AC_PROG_MKDIR_P])dnl # # AC_SUBST([mkdir_p], ['$(MKDIR_P)']) -# We need awk for the "check" target. The system "awk" is bad on -# some platforms. +# We need awk for the "check" target (and possibly the TAP driver). The +# system "awk" is bad on some platforms. AC_REQUIRE([AC_PROG_AWK])dnl AC_REQUIRE([AC_PROG_MAKE_SET])dnl AC_REQUIRE([AM_SET_LEADING_DOT])dnl @@ -526,6 +609,51 @@ dnl macro is hooked onto _AC_COMPILER_EXEEXT early, se AC_CONFIG_COMMANDS_PRE(dnl [m4_provide_if([_AM_COMPILER_EXEEXT], [AM_CONDITIONAL([am__EXEEXT], [test -n "$EXEEXT"])])])dnl + +# POSIX will say in a future version that running "rm -f" with no argument +# is OK; and we want to be able to make that assumption in our Makefile +# recipes. So use an aggressive probe to check that the usage we want is +# actually supported "in the wild" to an acceptable degree. +# See automake bug#10828. +# To make any issue more visible, cause the running configure to be aborted +# by default if the 'rm' program in use doesn't match our expectations; the +# user can still override this though. +if rm -f && rm -fr && rm -rf; then : OK; else + cat >&2 <<'END' +Oops! + +Your 'rm' program seems unable to run without file operands specified +on the command line, even when the '-f' option is present. This is contrary +to the behaviour of most rm programs out there, and not conforming with +the upcoming POSIX standard: + +Please tell bug-automake@gnu.org about your system, including the value +of your $PATH and any error possibly output before this message. This +can help us improve future automake versions. + +END + if test x"$ACCEPT_INFERIOR_RM_PROGRAM" = x"yes"; then + echo 'Configuration will proceed anyway, since you have set the' >&2 + echo 'ACCEPT_INFERIOR_RM_PROGRAM variable to "yes"' >&2 + echo >&2 + else + cat >&2 <<'END' +Aborting the configuration process, to ensure you take notice of the issue. + +You can download and install GNU coreutils to get an 'rm' implementation +that behaves properly: . + +If you want to complete the configuration process using your problematic +'rm' anyway, export the environment variable ACCEPT_INFERIOR_RM_PROGRAM +to "yes", and re-run configure. + +END + AC_MSG_ERROR([Your 'rm' program is bad, sorry.]) + fi +fi +dnl The trailing newline in this macro's definition is deliberate, for +dnl backward compatibility and to allow trailing 'dnl'-style comments +dnl after the AM_INIT_AUTOMAKE invocation. See automake bug#16841. ]) dnl Hook into '_AC_COMPILER_EXEEXT' early to learn its expansion. Do not @@ -534,7 +662,6 @@ dnl mangled by Autoconf and run in a shell conditional m4_define([_AC_COMPILER_EXEEXT], m4_defn([_AC_COMPILER_EXEEXT])[m4_provide([_AM_COMPILER_EXEEXT])]) - # When config.status generates a header, we must update the stamp-h file. # This file resides in the same directory as the config header # that is generated. The stamp files are numbered to have different names. @@ -556,7 +683,7 @@ for _am_header in $config_headers :; do done echo "timestamp for $_am_arg" >`AS_DIRNAME(["$_am_arg"])`/stamp-h[]$_am_stamp_count]) -# Copyright (C) 2001-2013 Free Software Foundation, Inc. +# Copyright (C) 2001-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -567,7 +694,7 @@ echo "timestamp for $_am_arg" >`AS_DIRNAME(["$_am_arg" # Define $install_sh. AC_DEFUN([AM_PROG_INSTALL_SH], [AC_REQUIRE([AM_AUX_DIR_EXPAND])dnl -if test x"${install_sh}" != xset; then +if test x"${install_sh+set}" != xset; then case $am_aux_dir in *\ * | *\ *) install_sh="\${SHELL} '$am_aux_dir/install-sh'" ;; @@ -577,7 +704,7 @@ if test x"${install_sh}" != xset; then fi AC_SUBST([install_sh])]) -# Copyright (C) 2003-2013 Free Software Foundation, Inc. +# Copyright (C) 2003-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -598,7 +725,7 @@ AC_SUBST([am__leading_dot])]) # Check to see how 'make' treats includes. -*- Autoconf -*- -# Copyright (C) 2001-2013 Free Software Foundation, Inc. +# Copyright (C) 2001-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -646,41 +773,9 @@ AC_MSG_RESULT([$_am_result]) rm -f confinc confmf ]) -# Copyright (C) 1999-2013 Free Software Foundation, Inc. -# -# This file is free software; the Free Software Foundation -# gives unlimited permission to copy and/or distribute it, -# with or without modifications, as long as this notice is preserved. - -# AM_PROG_CC_C_O -# -------------- -# Like AC_PROG_CC_C_O, but changed for automake. -AC_DEFUN([AM_PROG_CC_C_O], -[AC_REQUIRE([AC_PROG_CC_C_O])dnl -AC_REQUIRE([AM_AUX_DIR_EXPAND])dnl -AC_REQUIRE_AUX_FILE([compile])dnl -# FIXME: we rely on the cache variable name because -# there is no other way. -set dummy $CC -am_cc=`echo $[2] | sed ['s/[^a-zA-Z0-9_]/_/g;s/^[0-9]/_/']` -eval am_t=\$ac_cv_prog_cc_${am_cc}_c_o -if test "$am_t" != yes; then - # Losing compiler, so override with the script. - # FIXME: It is wrong to rewrite CC. - # But if we don't then we get into trouble of one sort or another. - # A longer-term fix would be to have automake use am__CC in this case, - # and then we could set am__CC="\$(top_srcdir)/compile \$(CC)" - CC="$am_aux_dir/compile $CC" -fi -dnl Make sure AC_PROG_CC is never called again, or it will override our -dnl setting of CC. -m4_define([AC_PROG_CC], - [m4_fatal([AC_PROG_CC cannot be called after AM_PROG_CC_C_O])]) -]) - # Fake the existence of programs that GNU maintainers use. -*- Autoconf -*- -# Copyright (C) 1997-2013 Free Software Foundation, Inc. +# Copyright (C) 1997-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -719,7 +814,7 @@ fi # Helper functions for option handling. -*- Autoconf -*- -# Copyright (C) 2001-2013 Free Software Foundation, Inc. +# Copyright (C) 2001-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -748,9 +843,73 @@ AC_DEFUN([_AM_SET_OPTIONS], AC_DEFUN([_AM_IF_OPTION], [m4_ifset(_AM_MANGLE_OPTION([$1]), [$2], [$3])]) +# Copyright (C) 1999-2014 Free Software Foundation, Inc. +# +# This file is free software; the Free Software Foundation +# gives unlimited permission to copy and/or distribute it, +# with or without modifications, as long as this notice is preserved. + +# _AM_PROG_CC_C_O +# --------------- +# Like AC_PROG_CC_C_O, but changed for automake. We rewrite AC_PROG_CC +# to automatically call this. +AC_DEFUN([_AM_PROG_CC_C_O], +[AC_REQUIRE([AM_AUX_DIR_EXPAND])dnl +AC_REQUIRE_AUX_FILE([compile])dnl +AC_LANG_PUSH([C])dnl +AC_CACHE_CHECK( + [whether $CC understands -c and -o together], + [am_cv_prog_cc_c_o], + [AC_LANG_CONFTEST([AC_LANG_PROGRAM([])]) + # Make sure it works both with $CC and with simple cc. + # Following AC_PROG_CC_C_O, we do the test twice because some + # compilers refuse to overwrite an existing .o file with -o, + # though they will create one. + am_cv_prog_cc_c_o=yes + for am_i in 1 2; do + if AM_RUN_LOG([$CC -c conftest.$ac_ext -o conftest2.$ac_objext]) \ + && test -f conftest2.$ac_objext; then + : OK + else + am_cv_prog_cc_c_o=no + break + fi + done + rm -f core conftest* + unset am_i]) +if test "$am_cv_prog_cc_c_o" != yes; then + # Losing compiler, so override with the script. + # FIXME: It is wrong to rewrite CC. + # But if we don't then we get into trouble of one sort or another. + # A longer-term fix would be to have automake use am__CC in this case, + # and then we could set am__CC="\$(top_srcdir)/compile \$(CC)" + CC="$am_aux_dir/compile $CC" +fi +AC_LANG_POP([C])]) + +# For backward compatibility. +AC_DEFUN_ONCE([AM_PROG_CC_C_O], [AC_REQUIRE([AC_PROG_CC])]) + +# Copyright (C) 2001-2014 Free Software Foundation, Inc. +# +# This file is free software; the Free Software Foundation +# gives unlimited permission to copy and/or distribute it, +# with or without modifications, as long as this notice is preserved. + +# AM_RUN_LOG(COMMAND) +# ------------------- +# Run COMMAND, save the exit status in ac_status, and log it. +# (This has been adapted from Autoconf's _AC_RUN_LOG macro.) +AC_DEFUN([AM_RUN_LOG], +[{ echo "$as_me:$LINENO: $1" >&AS_MESSAGE_LOG_FD + ($1) >&AS_MESSAGE_LOG_FD 2>&AS_MESSAGE_LOG_FD + ac_status=$? + echo "$as_me:$LINENO: \$? = $ac_status" >&AS_MESSAGE_LOG_FD + (exit $ac_status); }]) + # Check to make sure that the build environment is sane. -*- Autoconf -*- -# Copyright (C) 1996-2013 Free Software Foundation, Inc. +# Copyright (C) 1996-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -831,7 +990,7 @@ AC_CONFIG_COMMANDS_PRE( rm -f conftest.file ]) -# Copyright (C) 2009-2013 Free Software Foundation, Inc. +# Copyright (C) 2009-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -891,7 +1050,7 @@ AC_SUBST([AM_BACKSLASH])dnl _AM_SUBST_NOTMAKE([AM_BACKSLASH])dnl ]) -# Copyright (C) 2001-2013 Free Software Foundation, Inc. +# Copyright (C) 2001-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -919,7 +1078,7 @@ fi INSTALL_STRIP_PROGRAM="\$(install_sh) -c -s" AC_SUBST([INSTALL_STRIP_PROGRAM])]) -# Copyright (C) 2006-2013 Free Software Foundation, Inc. +# Copyright (C) 2006-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -938,7 +1097,7 @@ AC_DEFUN([AM_SUBST_NOTMAKE], [_AM_SUBST_NOTMAKE($@)]) # Check how to create a tarball. -*- Autoconf -*- -# Copyright (C) 2004-2013 Free Software Foundation, Inc. +# Copyright (C) 2004-2014 Free Software Foundation, Inc. # # This file is free software; the Free Software Foundation # gives unlimited permission to copy and/or distribute it, @@ -957,76 +1116,114 @@ AC_DEFUN([AM_SUBST_NOTMAKE], [_AM_SUBST_NOTMAKE($@)]) # Substitute a variable $(am__untar) that extract such # a tarball read from stdin. # $(am__untar) < result.tar +# AC_DEFUN([_AM_PROG_TAR], [# Always define AMTAR for backward compatibility. Yes, it's still used # in the wild :-( We should find a proper way to deprecate it ... AC_SUBST([AMTAR], ['$${TAR-tar}']) -m4_if([$1], [v7], - [am__tar='$${TAR-tar} chof - "$$tardir"' am__untar='$${TAR-tar} xf -'], - [m4_case([$1], [ustar],, [pax],, - [m4_fatal([Unknown tar format])]) -AC_MSG_CHECKING([how to create a $1 tar archive]) -# Loop over all known methods to create a tar archive until one works. + +# We'll loop over all known methods to create a tar archive until one works. _am_tools='gnutar m4_if([$1], [ustar], [plaintar]) pax cpio none' -_am_tools=${am_cv_prog_tar_$1-$_am_tools} -# Do not fold the above two line into one, because Tru64 sh and -# Solaris sh will not grok spaces in the rhs of '-'. -for _am_tool in $_am_tools -do - case $_am_tool in - gnutar) - for _am_tar in tar gnutar gtar; - do - AM_RUN_LOG([$_am_tar --version]) && break - done - am__tar="$_am_tar --format=m4_if([$1], [pax], [posix], [$1]) -chf - "'"$$tardir"' - am__tar_="$_am_tar --format=m4_if([$1], [pax], [posix], [$1]) -chf - "'"$tardir"' - am__untar="$_am_tar -xf -" - ;; - plaintar) - # Must skip GNU tar: if it does not support --format= it doesn't create - # ustar tarball either. - (tar --version) >/dev/null 2>&1 && continue - am__tar='tar chf - "$$tardir"' - am__tar_='tar chf - "$tardir"' - am__untar='tar xf -' - ;; - pax) - am__tar='pax -L -x $1 -w "$$tardir"' - am__tar_='pax -L -x $1 -w "$tardir"' - am__untar='pax -r' - ;; - cpio) - am__tar='find "$$tardir" -print | cpio -o -H $1 -L' - am__tar_='find "$tardir" -print | cpio -o -H $1 -L' - am__untar='cpio -i -H $1 -d' - ;; - none) - am__tar=false - am__tar_=false - am__untar=false - ;; - esac - # If the value was cached, stop now. We just wanted to have am__tar - # and am__untar set. - test -n "${am_cv_prog_tar_$1}" && break +m4_if([$1], [v7], + [am__tar='$${TAR-tar} chof - "$$tardir"' am__untar='$${TAR-tar} xf -'], - # tar/untar a dummy directory, and stop if the command works + [m4_case([$1], + [ustar], + [# The POSIX 1988 'ustar' format is defined with fixed-size fields. + # There is notably a 21 bits limit for the UID and the GID. In fact, + # the 'pax' utility can hang on bigger UID/GID (see automake bug#8343 + # and bug#13588). + am_max_uid=2097151 # 2^21 - 1 + am_max_gid=$am_max_uid + # The $UID and $GID variables are not portable, so we need to resort + # to the POSIX-mandated id(1) utility. Errors in the 'id' calls + # below are definitely unexpected, so allow the users to see them + # (that is, avoid stderr redirection). + am_uid=`id -u || echo unknown` + am_gid=`id -g || echo unknown` + AC_MSG_CHECKING([whether UID '$am_uid' is supported by ustar format]) + if test $am_uid -le $am_max_uid; then + AC_MSG_RESULT([yes]) + else + AC_MSG_RESULT([no]) + _am_tools=none + fi + AC_MSG_CHECKING([whether GID '$am_gid' is supported by ustar format]) + if test $am_gid -le $am_max_gid; then + AC_MSG_RESULT([yes]) + else + AC_MSG_RESULT([no]) + _am_tools=none + fi], + + [pax], + [], + + [m4_fatal([Unknown tar format])]) + + AC_MSG_CHECKING([how to create a $1 tar archive]) + + # Go ahead even if we have the value already cached. We do so because we + # need to set the values for the 'am__tar' and 'am__untar' variables. + _am_tools=${am_cv_prog_tar_$1-$_am_tools} + + for _am_tool in $_am_tools; do + case $_am_tool in + gnutar) + for _am_tar in tar gnutar gtar; do + AM_RUN_LOG([$_am_tar --version]) && break + done + am__tar="$_am_tar --format=m4_if([$1], [pax], [posix], [$1]) -chf - "'"$$tardir"' + am__tar_="$_am_tar --format=m4_if([$1], [pax], [posix], [$1]) -chf - "'"$tardir"' + am__untar="$_am_tar -xf -" + ;; + plaintar) + # Must skip GNU tar: if it does not support --format= it doesn't create + # ustar tarball either. + (tar --version) >/dev/null 2>&1 && continue + am__tar='tar chf - "$$tardir"' + am__tar_='tar chf - "$tardir"' + am__untar='tar xf -' + ;; + pax) + am__tar='pax -L -x $1 -w "$$tardir"' + am__tar_='pax -L -x $1 -w "$tardir"' + am__untar='pax -r' + ;; + cpio) + am__tar='find "$$tardir" -print | cpio -o -H $1 -L' + am__tar_='find "$tardir" -print | cpio -o -H $1 -L' + am__untar='cpio -i -H $1 -d' + ;; + none) + am__tar=false + am__tar_=false + am__untar=false + ;; + esac + + # If the value was cached, stop now. We just wanted to have am__tar + # and am__untar set. + test -n "${am_cv_prog_tar_$1}" && break + + # tar/untar a dummy directory, and stop if the command works. + rm -rf conftest.dir + mkdir conftest.dir + echo GrepMe > conftest.dir/file + AM_RUN_LOG([tardir=conftest.dir && eval $am__tar_ >conftest.tar]) + rm -rf conftest.dir + if test -s conftest.tar; then + AM_RUN_LOG([$am__untar /dev/null 2>&1 && break + fi + done rm -rf conftest.dir - mkdir conftest.dir - echo GrepMe > conftest.dir/file - AM_RUN_LOG([tardir=conftest.dir && eval $am__tar_ >conftest.tar]) - rm -rf conftest.dir - if test -s conftest.tar; then - AM_RUN_LOG([$am__untar /dev/null 2>&1 && break - fi -done -rm -rf conftest.dir -AC_CACHE_VAL([am_cv_prog_tar_$1], [am_cv_prog_tar_$1=$_am_tool]) -AC_MSG_RESULT([$am_cv_prog_tar_$1])]) + AC_CACHE_VAL([am_cv_prog_tar_$1], [am_cv_prog_tar_$1=$_am_tool]) + AC_MSG_RESULT([$am_cv_prog_tar_$1])]) + AC_SUBST([am__tar]) AC_SUBST([am__untar]) ]) # _AM_PROG_TAR Modified: vendor/file/dist/compile ============================================================================== --- vendor/file/dist/compile Sun Jul 29 05:51:14 2018 (r336861) +++ vendor/file/dist/compile Sun Jul 29 05:53:49 2018 (r336862) @@ -3,7 +3,7 @@ scriptversion=2012-10-14.11; # UTC -# Copyright (C) 1999-2013 Free Software Foundation, Inc. +# Copyright (C) 1999-2014 Free Software Foundation, Inc. # Written by Tom Tromey . # # This program is free software; you can redistribute it and/or modify Modified: vendor/file/dist/config.guess ============================================================================== --- vendor/file/dist/config.guess Sun Jul 29 05:51:14 2018 (r336861) +++ vendor/file/dist/config.guess Sun Jul 29 05:53:49 2018 (r336862) @@ -1,10 +1,8 @@ #! /bin/sh # Attempt to guess a canonical system name. -# Copyright (C) 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, -# 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, -# 2011, 2012, 2013 Free Software Foundation, Inc. +# Copyright 1992-2017 Free Software Foundation, Inc. -timestamp='2012-12-29' +timestamp='2017-01-01' # This file is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by @@ -26,12 +24,12 @@ timestamp='2012-12-29' # program. This Exception is an additional permission under section 7 # of the GNU General Public License, version 3 ("GPLv3"). # -# Originally written by Per Bothner. +# Originally written by Per Bothner; maintained since 2000 by Ben Elliston. # # You can get the latest version of this script from: -# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD +# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess # -# Please send patches with a ChangeLog entry to config-patches@gnu.org. +# Please send patches to . me=`echo "$0" | sed -e 's,.*/,,'` @@ -52,9 +50,7 @@ version="\ GNU config.guess ($timestamp) Originally written by Per Bothner. -Copyright (C) 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, -2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -2012, 2013 Free Software Foundation, Inc. +Copyright 1992-2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE." @@ -136,6 +132,27 @@ UNAME_RELEASE=`(uname -r) 2>/dev/null` || UNAME_RELEAS UNAME_SYSTEM=`(uname -s) 2>/dev/null` || UNAME_SYSTEM=unknown UNAME_VERSION=`(uname -v) 2>/dev/null` || UNAME_VERSION=unknown +case "${UNAME_SYSTEM}" in +Linux|GNU|GNU/*) + # If the system lacks a compiler, then just pick glibc. + # We could probably try harder. + LIBC=gnu + + eval $set_cc_for_build + cat <<-EOF > $dummy.c + #include + #if defined(__UCLIBC__) + LIBC=uclibc + #elif defined(__dietlibc__) + LIBC=dietlibc + #else + LIBC=gnu + #endif + EOF + eval `$CC_FOR_BUILD -E $dummy.c 2>/dev/null | grep '^LIBC' | sed 's, ,,g'` + ;; +esac + # Note: order is significant - the case branches are not exclusive. case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE}:${UNAME_VERSION}" in @@ -151,19 +168,29 @@ case "${UNAME_MACHINE}:${UNAME_SYSTEM}:${UNAME_RELEASE # Note: NetBSD doesn't particularly care about the vendor # portion of the name. We always set it to "unknown". sysctl="sysctl -n hw.machine_arch" - UNAME_MACHINE_ARCH=`(/sbin/$sysctl 2>/dev/null || \ - /usr/sbin/$sysctl 2>/dev/null || echo unknown)` + UNAME_MACHINE_ARCH=`(uname -p 2>/dev/null || \ + /sbin/$sysctl 2>/dev/null || \ + /usr/sbin/$sysctl 2>/dev/null || \ + echo unknown)` case "${UNAME_MACHINE_ARCH}" in armeb) machine=armeb-unknown ;; arm*) machine=arm-unknown ;; sh3el) machine=shl-unknown ;; sh3eb) machine=sh-unknown ;; sh5el) machine=sh5le-unknown ;; + earmv*) + arch=`echo ${UNAME_MACHINE_ARCH} | sed -e 's,^e\(armv[0-9]\).*$,\1,'` + endian=`echo ${UNAME_MACHINE_ARCH} | sed -ne 's,^.*\(eb\)$,\1,p'` + machine=${arch}${endian}-unknown *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sun Jul 29 05:54:56 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4C46104C574; Sun, 29 Jul 2018 05:54:55 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 598E072ECF; Sun, 29 Jul 2018 05:54:55 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3A70711C55; Sun, 29 Jul 2018 05:54:55 +0000 (UTC) (envelope-from eadler@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6T5stN7024374; Sun, 29 Jul 2018 05:54:55 GMT (envelope-from eadler@FreeBSD.org) Received: (from eadler@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6T5stTq024372; Sun, 29 Jul 2018 05:54:55 GMT (envelope-from eadler@FreeBSD.org) Message-Id: <201807290554.w6T5stTq024372@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: eadler set sender to eadler@FreeBSD.org using -f From: Eitan Adler Date: Sun, 29 Jul 2018 05:54:55 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336863 - in vendor/file/5.34: . doc magic magic/Magdir python src tests X-SVN-Group: vendor X-SVN-Commit-Author: eadler X-SVN-Commit-Paths: in vendor/file/5.34: . doc magic magic/Magdir python src tests X-SVN-Commit-Revision: 336863 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jul 2018 05:54:56 -0000 Author: eadler Date: Sun Jul 29 05:54:54 2018 New Revision: 336863 URL: https://svnweb.freebsd.org/changeset/base/336863 Log: vendor/file: tag 5.34 Added: vendor/file/5.34/ - copied from r336859, vendor/file/dist/ vendor/file/5.34/magic/Magdir/dataone - copied unchanged from r336862, vendor/file/dist/magic/Magdir/dataone vendor/file/5.34/python/CHANGELOG.md - copied unchanged from r336862, vendor/file/dist/python/CHANGELOG.md vendor/file/5.34/python/LICENSE - copied unchanged from r336862, vendor/file/dist/python/LICENSE vendor/file/5.34/python/README.md - copied unchanged from r336862, vendor/file/dist/python/README.md vendor/file/5.34/python/tests.py - copied unchanged from r336862, vendor/file/dist/python/tests.py vendor/file/5.34/tests/JW07022A.mp3.result - copied unchanged from r336862, vendor/file/dist/tests/JW07022A.mp3.result vendor/file/5.34/tests/JW07022A.mp3.testfile - copied unchanged from r336862, vendor/file/dist/tests/JW07022A.mp3.testfile Replaced: vendor/file/5.34/ChangeLog - copied unchanged from r336862, vendor/file/dist/ChangeLog vendor/file/5.34/Makefile.in - copied unchanged from r336862, vendor/file/dist/Makefile.in vendor/file/5.34/README - copied unchanged from r336862, vendor/file/dist/README vendor/file/5.34/aclocal.m4 - copied unchanged from r336862, vendor/file/dist/aclocal.m4 vendor/file/5.34/compile - copied unchanged from r336862, vendor/file/dist/compile vendor/file/5.34/config.guess - copied unchanged from r336862, vendor/file/dist/config.guess vendor/file/5.34/config.h.in - copied unchanged from r336862, vendor/file/dist/config.h.in vendor/file/5.34/config.sub - copied unchanged from r336862, vendor/file/dist/config.sub vendor/file/5.34/configure - copied unchanged from r336862, vendor/file/dist/configure vendor/file/5.34/configure.ac - copied unchanged from r336862, vendor/file/dist/configure.ac vendor/file/5.34/depcomp - copied unchanged from r336862, vendor/file/dist/depcomp vendor/file/5.34/doc/Makefile.in - copied unchanged from r336862, vendor/file/dist/doc/Makefile.in vendor/file/5.34/doc/file.man - copied unchanged from r336862, vendor/file/dist/doc/file.man vendor/file/5.34/doc/magic.man - copied unchanged from r336862, vendor/file/dist/doc/magic.man vendor/file/5.34/install-sh - copied unchanged from r336862, vendor/file/dist/install-sh vendor/file/5.34/ltmain.sh - copied unchanged from r336862, vendor/file/dist/ltmain.sh vendor/file/5.34/magic/Magdir/animation - copied unchanged from r336862, vendor/file/dist/magic/Magdir/animation vendor/file/5.34/magic/Magdir/archive - copied unchanged from r336862, vendor/file/dist/magic/Magdir/archive vendor/file/5.34/magic/Magdir/audio - copied unchanged from r336862, vendor/file/dist/magic/Magdir/audio vendor/file/5.34/magic/Magdir/cad - copied unchanged from r336862, vendor/file/dist/magic/Magdir/cad vendor/file/5.34/magic/Magdir/console - copied unchanged from r336862, vendor/file/dist/magic/Magdir/console vendor/file/5.34/magic/Magdir/database - copied unchanged from r336862, vendor/file/dist/magic/Magdir/database vendor/file/5.34/magic/Magdir/dbpf - copied unchanged from r336862, vendor/file/dist/magic/Magdir/dbpf vendor/file/5.34/magic/Magdir/dump - copied unchanged from r336862, vendor/file/dist/magic/Magdir/dump vendor/file/5.34/magic/Magdir/elf - copied unchanged from r336862, vendor/file/dist/magic/Magdir/elf vendor/file/5.34/magic/Magdir/fsav - copied unchanged from r336862, vendor/file/dist/magic/Magdir/fsav vendor/file/5.34/magic/Magdir/images - copied unchanged from r336862, vendor/file/dist/magic/Magdir/images vendor/file/5.34/magic/Magdir/linux - copied unchanged from r336862, vendor/file/dist/magic/Magdir/linux vendor/file/5.34/magic/Magdir/make - copied unchanged from r336862, vendor/file/dist/magic/Magdir/make vendor/file/5.34/magic/Magdir/measure - copied unchanged from r336862, vendor/file/dist/magic/Magdir/measure vendor/file/5.34/magic/Magdir/msdos - copied unchanged from r336862, vendor/file/dist/magic/Magdir/msdos vendor/file/5.34/magic/Magdir/msooxml - copied unchanged from r336862, vendor/file/dist/magic/Magdir/msooxml vendor/file/5.34/magic/Magdir/pdf - copied unchanged from r336862, vendor/file/dist/magic/Magdir/pdf vendor/file/5.34/magic/Magdir/sgi - copied unchanged from r336862, vendor/file/dist/magic/Magdir/sgi vendor/file/5.34/magic/Makefile.am - copied unchanged from r336862, vendor/file/dist/magic/Makefile.am vendor/file/5.34/magic/Makefile.in - copied unchanged from r336862, vendor/file/dist/magic/Makefile.in vendor/file/5.34/missing - copied unchanged from r336862, vendor/file/dist/missing vendor/file/5.34/python/Makefile.am - copied unchanged from r336862, vendor/file/dist/python/Makefile.am vendor/file/5.34/python/Makefile.in - copied unchanged from r336862, vendor/file/dist/python/Makefile.in vendor/file/5.34/python/magic.py - copied unchanged from r336862, vendor/file/dist/python/magic.py vendor/file/5.34/python/setup.py - copied unchanged from r336862, vendor/file/dist/python/setup.py vendor/file/5.34/src/Makefile.in - copied unchanged from r336862, vendor/file/dist/src/Makefile.in vendor/file/5.34/src/apprentice.c - copied unchanged from r336862, vendor/file/dist/src/apprentice.c vendor/file/5.34/src/compress.c - copied unchanged from r336862, vendor/file/dist/src/compress.c vendor/file/5.34/src/der.c - copied unchanged from r336862, vendor/file/dist/src/der.c vendor/file/5.34/src/file.h - copied unchanged from r336862, vendor/file/dist/src/file.h vendor/file/5.34/src/funcs.c - copied unchanged from r336862, vendor/file/dist/src/funcs.c vendor/file/5.34/src/readcdf.c - copied unchanged from r336862, vendor/file/dist/src/readcdf.c vendor/file/5.34/src/readelf.c - copied unchanged from r336862, vendor/file/dist/src/readelf.c vendor/file/5.34/src/readelf.h - copied unchanged from r336862, vendor/file/dist/src/readelf.h vendor/file/5.34/src/seccomp.c - copied unchanged from r336862, vendor/file/dist/src/seccomp.c vendor/file/5.34/src/softmagic.c - copied unchanged from r336862, vendor/file/dist/src/softmagic.c vendor/file/5.34/tests/Makefile.am - copied unchanged from r336862, vendor/file/dist/tests/Makefile.am vendor/file/5.34/tests/Makefile.in - copied unchanged from r336862, vendor/file/dist/tests/Makefile.in Deleted: vendor/file/5.34/python/README Copied: vendor/file/5.34/ChangeLog (from r336862, vendor/file/dist/ChangeLog) ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/file/5.34/ChangeLog Sun Jul 29 05:54:54 2018 (r336863, copy of r336862, vendor/file/dist/ChangeLog) @@ -0,0 +1,1753 @@ +2018-07-25 8:50 Christos Zoulas + + * release 5.34 + +2018-06-22 16:38 Christos Zoulas + + * Add Quad indirect offsets + +2018-05-24 14:10 Christos Zoulas + + * Enable parsing of ELF dynamic sections to handle PIE better + +2018-04-15 14:52 Christos Zoulas + + * release 5.33 + +2018-02-24 14:50 Christos Zoulas + + * extend the support for ${x?:} expansions for magic descriptions + +2018-02-21 16:25 Christos Zoulas + + * add support for ${x?:} in mime types to handle + pie binaries. + +2017-11-03 9:23 Christos Zoulas + + * add support for negative offsets (offsets from the end of file) + +2017-09-26 8:22 Christos Zoulas + + * close the file on error when writing magic (Steve Grubb) + +2017-09-24 12:02 Christos Zoulas + + * seccomp support (Paul Moore) + +2017-09-02 11:53 Christos Zoulas + + * release 5.32 + +2017-08-28 16:37 Christos Zoulas + + * Always reset state in {file,buffer}_apprentice (Krzysztof Wilczynski) + +2017-08-27 03:55 Christos Zoulas + + * Fix always true condition (Thomas Jarosch) + +2017-05-24 17:30 Christos Zoulas + + * pickier parsing of numeric values in magic files. + +2017-05-23 17:55 Christos Zoulas + + * PR/615 add magic_getflags() + +2017-05-23 13:55 Christos Zoulas + + * release 5.31 + +2017-03-17 20:32 Christos Zoulas + + * remove trailing spaces from magic files + * refactor is_tar + * better bounds checks for cdf + +2017-02-10 12:24 Christos Zoulas + + * release 5.30 + +2017-02-07 23:27 Christos Zoulas + + * If we exceeded the offset in a search return no match + (Christoph Biedl) + * Be more lenient on corrupt CDF files (Christoph Biedl) + +2017-02-04 16:46 Christos Zoulas + + * pacify ubsan sign extension (oss-fuzz/524) + +2017-02-01 12:42 Christos Zoulas + + * off by one in cdf parsing (PR/593) + * report debugging sections in elf (PR/591) + +2016-11-06 10:52 Christos Zoulas + + * Allow @@@ in extensions + * Add missing overflow check in der magic (Jonas Wagner) + +2016-10-25 10:40 Christos Zoulas + + * release 5.29 + +2016-10-24 11:20 Christos Zoulas + + * der getlength overflow (Jonas Wagner) + * multiple magic file load failure (Christoph Biedl) + +2016-10-17 11:26 Christos Zoulas + + * CDF parsing improvements (Guy Helmer) + +2016-07-20 7:26 Christos Zoulas + + * Add support for signed indirect offsets + +2016-07-18 7:41 Christos Zoulas + + * cat /dev/null | file - should print empty (Christoph Biedl) + +2016-07-05 15:20 Christos Zoulas + + * Bump string size from 64 to 96. + +2016-06-13 20:20 Christos Zoulas + + * PR/556: Fix separators on annotations. + +2016-06-13 19:40 Christos Zoulas + + * release 5.28 + * fix leak on allocation failure + +2016-06-01 1:20 Christos Zoulas + + * PR/555: Avoid overflow for offset > nbytes + * PR/550: Segv on DER parsing: + - use the correct variable for length + - set offset to 0 on failure. + +2016-05-13 12:00 Christos Zoulas + + * release 5.27 + +2016-04-18 9:35 Christos Zoulas + + * Errors comparing DER entries or computing offsets + are just indications of malformed non-DER files. + Don't print them. + * Offset comparison was off-by-one. + * Fix compression code (Werner Fink) + * Put new bytes constant in the right file (not the generated one) + +2016-04-16 18:34 Christos Zoulas + + * release 5.26 + +2016-03-31 13:50 Christos Zoulas + + * make the number of bytes read from files configurable. + +2016-03-21 13:40 Christos Zoulas + + * Add bounds checks for DER code (discovered by Thomas Jarosch) + * Change indirect recursion limit to indirect use count and + bump from 15 to 50 to prevent abuse. + +2016-03-13 20:39 Christos Zoulas + + * Add -00 which prints filename\0description\0 + +2016-03-01 13:28 Christos Zoulas + + * Fix ID3 indirect parsing + +2016-01-19 10:18 Christos Zoulas + + * add DER parsing capability + +2015-11-13 10:35 Christos Zoulas + + * provide dprintf(3) for the OS's that don't have it. + +2015-11-11 16:25 Christos Zoulas + + * redo the compression code report decompression errors + +2015-11-10 23:25 Christos Zoulas + + * REG_STARTEND code is not working as expected, delete it. + +2015-11-09 16:05 Christos Zoulas + + * Add zlib support if we have it. + +2015-11-05 11:22 Christos Zoulas + + * PR/492: compression forking was broken with magic_buffer. + +2015-09-16 9:50 Christos Zoulas + + * release 5.25 + +2015-09-11 13:25 Christos Zoulas + + * add a limit to the length of regex searches + +2015-09-08 9:50 Christos Zoulas + + * fix problems with --parameter (Christoph Biedl) + +2015-07-11 10:35 Christos Zoulas + + * Windows fixes PR/466 (Jason Hood) + +2015-07-09 10:35 Christos Zoulas + + * release 5.24 + +2015-06-11 8:52 Christos Zoulas + + * redo long option encoding to fix off-by-one in 5.23 + +2015-06-10 13:50 Christos Zoulas + + * release 5.23 + +2015-06-09 16:10 Christos Zoulas + + * Fix issue with regex range for magic with offset + * Always return true from mget with USE (success to mget not match + indication). Fixes mime evaluation after USE magic + * PR/459: Don't insert magic entries to the list if there are parsing + errors for them. + +2015-06-03 16:00 Christos Zoulas + + * PR/455: Add utf-7 encoding + +2015-06-03 14:30 Christos Zoulas + + * PR/455: Implement -Z, look inside, but don't report on compression + * PR/454: Fix allocation error on bad magic. + +2015-05-29 10:30 Christos Zoulas + + * handle MAGIC_CONTINUE everywhere, not just in softmagic + +2015-05-21 14:30 Christos Zoulas + + * don't print descriptions for NAME types when mime. + +2015-04-09 15:59 Christos Zoulas + + * Add --extension to list the known extensions for this file type + Idea by Andrew J Roazen + +2015-02-14 12:23 Christos Zoulas + + * Bump file search buffer size to 1M. + +2015-01-09 14:35 Christos Zoulas + + * Fix multiple issues with date formats reported by Christoph Biedl: + - T_LOCAL meaning was reversed + - Arithmetic did not work + Also stop adjusting daylight savings for gmt printing. + +2015-01-05 13:00 Christos Zoulas + + * PR/411: Fix memory corruption from corrupt cdf file. + +2015-01-02 15:15 Christos Zoulas + + * release 5.22 + +2015-01-01 12:01 Christos Zoulas + + * add indirect relative for TIFF/Exif + +2014-12-16 18:10 Christos Zoulas + + * restructure elf note printing to avoid repeated messages + * add note limit, suggested by Alexander Cherepanov + +2014-12-16 16:53 Christos Zoulas + + * Bail out on partial pread()'s (Alexander Cherepanov) + * Fix incorrect bounds check in file_printable (Alexander Cherepanov) + +2014-12-11 20:01 Christos Zoulas + + * PR/405: ignore SIGPIPE from uncompress programs + * change printable -> file_printable and use it in + more places for safety + * in ELF, instead of "(uses dynamic libraries)" when PT_INTERP + is present print the interpreter name. + +2014-12-10 20:01 Christos Zoulas + + * release 5.21 + +2014-11-27 18:40 Christos Zoulas + + * Allow setting more parameters from the command line. + * Split name/use and indirect magic recursion limits. + +2014-11-27 11:12 Christos Zoulas + + * Adjust ELF parameters and the default recursion + level. + * Allow setting the recursion level dynamically. + +2014-11-24 8:55 Christos Zoulas + + * The following fixes resulted from Thomas Jarosch's fuzzing + tests that revealed severe performance issues on pathological + input: + - limit number of elf program and sections processing + - abort elf note processing quickly + - reduce the number of recursion levels from 20 to 10 + - preserve error messages in indirect magic handling + + This is tracked as CVE-2014-8116 and CVE-2014-8117 + +2014-11-12 10:30 Christos Zoulas + + * fix bogus free in the user buffer case. + +2014-11-11 12:35 Christos Zoulas + + * fix out of bounds read for pascal strings + * fix memory leak (not freeing the head of each mlist) + +2014-11-07 10:25 Christos Zoulas + + * When printing strings from a file, convert them to printable + on a byte by byte basis, so that we don't get issues with + locale's trying to interpret random byte streams as UTF-8 and + having printf error out with EILSEQ. + +2014-10-17 11:48 Christos Zoulas + + * fix bounds in note reading (Francisco Alonso / Red Hat) + +2014-10-11 15:02 Christos Zoulas + + * fix autoconf glue for setlocale and locale_t; some OS's + have locale_t in xlocale.h + +2014-10-10 15:01 Christos Zoulas + + * release 5.20 + +2014-08-17 10:01 Christos Zoulas + + * recognize encrypted CDF documents + +2014-08-04 9:18 Christos Zoulas + + * add magic_load_buffers from Brooks Davis + +2014-07-24 16:40 Christos Zoulas + + * add thumbs.db support + +2014-06-12 12:28 Christos Zoulas + + * release 5.19 + +2014-06-09 9:04 Christos Zoulas + + * Misc buffer overruns and missing buffer size tests in cdf parsing + (Francisco Alonso, Jan Kaluza) + +2014-06-02 14:50 Christos Zoulas + + * Enforce limit of 8K on regex searches that have no limits + * Allow the l modifier for regex to mean line count. Default + to byte count. If line count is specified, assume a max + of 80 characters per line to limit the byte count. + * Don't allow conversions to be used for dates, allowing + the mask field to be used as an offset. + +2014-05-30 12:51 Christos Zoulas + + * Make the range operator limit the length of the + regex search. + +2014-05-14 19:23 Christos Zoulas + + * PR/347: Windows fixes + * PR/352: Hangul word processor recognition + * PR/354: Encoding irregularities in text files + +2014-05-06 6:12 Christos Zoulas + + * Fix uninitialized title in CDF files (Jan Kaluza) + +2014-05-04 14:55 Christos Zoulas + + * PR/351: Fix compilation of empty files + +2014-04-30 17:39 Christos Zoulas + + * Fix integer formats: We don't specify 'l' or + 'h' and 'hh' specifiers anymore, only 'll' for + quads and nothing for the rest. This is so that + magic writing is simpler. + +2014-04-01 15:25 Christos Zoulas + + * PR/341: Jan Kaluza, fix memory leak + * PR/342: Jan Kaluza, fix out of bounds read + +2014-03-28 15:25 Christos Zoulas + + * Fix issue with long formats not matching fmtcheck + +2014-03-26 11:25 Christos Zoulas + + * release 5.18 + +2014-03-15 17:45 Christos Zoulas + + * add fmtcheck(3) for those who don't have it + +2014-03-14 15:12 Christos Zoulas + + * prevent mime entries from being attached to magic + entries with no descriptions + + * adjust magic strength for regex type + + * remove superfluous ascmagic with encoding test + +2014-03-06 12:01 Christos Zoulas + + * fix regression fix echo -ne "\012\013\014" | file -i - + which printed "binary" instead of "application/octet-stream" + + * add size_t overflow check for magic file size + +2014-02-27 16:01 Christos Zoulas + + * experimental support for matching with CFD CLSID + +2014-02-18 13:04 Kimmo Suominen (kimmo@suominen.com) + + * Cache old LC_CTYPE locale before setting it to "C", so + we can use it to restore LC_CTYPE instead of asking + setlocale() to scan the environment variables. + +2014-02-12 18:21 Christos Zoulas + + * Count recursion levels through indirect magic + +2014-02-11 10:40 Christos Zoulas + + * Prevent infinite recursion on files with indirect offsets of 0 + +2014-01-30 21:00 Christos Zoulas + + * Add -E flag that makes file print filesystem errors to stderr + and exit. + +2014-01-08 17:20 Christos Zoulas + + * mime printing could print results from multiple magic entries + if there were multiple matches. + * in some cases overflow was not detected when computing offsets + in softmagic. + +2013-12-05 12:00 Christos Zoulas + + * use strcasestr() to for cdf strings + * reset to the "C" locale while doing regex operations, or case + insensitive comparisons; this is provisional + +2013-11-19 20:10 Christos Zoulas + + * always leave magic file loaded, don't unload for magic_check, etc. + * fix default encoding to binary instead of unknown which broke recently + * handle empty and one byte files, less specially so that + --mime-encoding does not break completely. + ` +2013-11-06 14:40 Christos Zoulas + + * fix erroneous non-zero exit code from non-existent file and message + +2013-10-29 14:25 Christos Zoulas + + * add CDF MSI file detection (Guy Helmer) + +2013-09-03 11:56 Christos Zoulas + + * Don't mix errors and regular output if there was an error + * in magic_descriptor() don't close the file and try to restore + its position + +2013-05-30 17:25 Christos Zoulas + + * Don't treat magic as an error if offset was past EOF (Christoph Biedl) + +2013-05-28 17:25 Christos Zoulas + + * Fix spacing issues in softmagic and elf (Jan Kaluza) + +2013-05-02 18:00 Christos Zoulas + + * Fix segmentation fault with multiple magic_load commands. + +2013-04-22 11:20 Christos Zoulas + + * The way "default" was implemented was not very useful + because the "if something was printed at that level" + was not easily controlled by the user, and the format + was bound to a string which is too restrictive. Add + a "clear" for that level keyword and make "default" + void. This way one can do: + + >>13 clear x + >>13 lelong 1 foo + >>13 lelong 2 bar + >>13 default x + >>>13 lelong x unknown %x + +2013-03-25 13:20 Christos Zoulas + + * disallow strength setting in "name" entries + +2013-03-06 21:24 Christos Zoulas + + * fix recursive magic separator printing + +2013-02-26 19:28 Christos Zoulas + + * limit recursion level for mget + * fix pread() related breakage in cdf + * handle offsets properly in recursive "use" + +2013-02-18 10:39 Christos Zoulas + + * add elf reading of debug info to determine if file is stripped + (Jan Kaluza) + * use pread() + +2013-01-25 18:05 Christos Zoulas + + * change mime description size from 64 to 80 to accommodate OOXML. + +2013-01-11 14:50 Christos Zoulas + + * Warn about inconsistent continuation levels. + * Change fsmagic to add a space after it prints. + +2013-01-10 21:00 Christos Zoulas + + * Make getline public so that file can link against it. + Perhaps it is better to rename it, or hide it differently. + Fixes builds on platforms that do not provide it. + +2013-01-07 16:30 Christos Zoulas + + * Add SuS d{,1,2,4,8}, u{,1,2,4,8} and document + what long, int, short, etc is (Guy Harris) + +2013-01-06 11:20 Christos Zoulas + + * add magic_version function and constant + * Redo memory allocation and de-allocation. + (prevents double frees on non mmap platforms) + * Fix bug with name/use having to do with passing + found state from the parent to the child and back. + +2012-12-19 8:47 Christos Zoulas + + * Only print elf capabilities for archs we know (Jan Kaluza) + +2012-10-30 19:14 Christos Zoulas + + * Add "name" and "use" file types in order to look + inside mach-o files. + +2012-09-06 10:40 Christos Zoulas + + * make --version exit 0 (Matthew Schultz) + * add string/T (Jan Kaluza) + +2012-08-09 2:15 Christos Zoulas + + * add z and t modifiers for our own vasprintf + * search for $HOME/.magic.mgc if it is there first + * fix reads from a pipe, and preserve errno + +2012-05-15 13:12 Christos Zoulas + + * use ctime_r, asctime_r + +2012-04-06 17:18 Christos Zoulas + + * Fixes for indirect offsets to handle apple disk formats + +2012-04-03 18:26 Christos Zoulas + + * Add windows date field types + * More info for windows shortcuts (incomplete) + +2012-02-20 17:33 Christos Zoulas + + * Fix CDF parsing issues found by CERT's fuzzing tool (Will Dormann) + +2011-12-15 12:17 Chris Metcalf + + * Support Tilera architectures (tile64, tilepro, tilegx). + +2011-12-16 16:33 Reuben Thomas + + * Add magic for /usr/bin/env Perl scripts + * Weaken generic script magic to avoid clashing with + language-specific magic. + +2011-12-08 13:37 Reuben Thomas + + * Simplify if (p) free(p) to free(p). + +2011-12-08 13:07 Reuben Thomas + + * Remove hardwired token finding (names.h), turning it into soft + magic. Patterns are either anchored regexs or search/8192. English + language detection and PL/1 detection have been removed as they + were too fragile. -e tokens is still accepted for backwards + compatibility. + * Move 3ds patterns (which are commented out anyway) into autodesk + (they were, oddly, in c-lang). + +2011-12-06 00:16 Reuben Thomas + + * Tweak strength of generic hash-bang detectors to be less than + specific ones. + * Make an inconsistent description of Python scripts consistent. + +2011-12-05 23:58 Reuben Thomas + + * Fix minor error in file(1). + +2011-11-05 00:00 Reuben Thomas + + * Fix issue #150 (I hope). + +2011-09-22 12:57 Christos Zoulas + + * Python3 binding fixes from Kelly Anderson + +2011-09-20 11:32 Christos Zoulas + + * If a string type magic entry is marked as text or binary + only match text files against text entries and binary + files against binary entries. + +2011-09-01 12:12 Christos Zoulas + + * Don't wait for any subprocess, just the one we forked. + +2011-08-26 16:40 Christos Zoulas + + * If the application name is not set in a cdf file, try to see + if it has a directory with the application name on it. + +2011-08-17 14:32 Christos Zoulas + + * Fix ELF lseek(2) madness. Inspired by PR/134 by Jan Kaluza + +2011-08-14 09:03 Christos Zoulas + + * Don't use variable string formats. + +2011-07-12 12:32 Reuben Thomas + + * Fix detection of Zip files (Mantis #128). + * Make some minor improvements to file(1). + * Rename MIME types for filesystem objects for consistency with + xdg-utils. Typically this means that application/x-foo becomes + inode/foo, but some names also change slightly, e.g. + application/x-character-device becomes inode/chardevice. + +2011-05-10 20:57 Christos Zoulas + + * fix mingw compilation (Abradoks) + +2011-05-10 20:57 Christos Zoulas + + * remove patchlevel.h + * Fix read past allocated memory caused by double-incrementing + a pointer in a loop (reported by Roberto Maar) + +2011-03-30 15:45 Christos Zoulas + + * Fix cdf string buffer setting (Sven Anders) + +2011-03-20 16:35 Christos Zoulas + + * Eliminate MAXPATHLEN and use dynamic allocation for + path and file buffers. + +2011-03-15 18:15 Christos Zoulas + + * binary tests on magic entries with masks could spuriously + get converted to ascii. + +2011-03-12 18:06 Reuben Thomas + + * Improve file.man (remove BUGS, present email addresses consistently). + +2011-03-07 19:38 Christos Zoulas + + * add lrzip support (from Ville Skytta) + +2011-02-10 16:36 Christos Zoulas + + * fix CDF bounds checking (Guy Helmer) + +2011-02-10 12:03 Christos Zoulas + + * add cdf_ctime() that prints a meaningful error when time cannot + be converted. + +2011-02-02 20:40 Christos Zoulas + + * help and version output to stdout. + + * When matching softmagic for ascii files, don't just print + the softmagic classification, keep going and print the + text classification too. This fixes broken troff files when + we moved them from keyword recognition to softmagic + (they stopped printing "with CRLF" etc.) + Reported by Doug McIlroy. + +2011-01-16 19:31 Reuben Thomas + + * Fix two potential buffer overruns in apprentice_list. + +2011-01-14 22:33 Reuben Thomas + + * New Python binding in pure Python. + * Update libmagic(3). + +2011-01-06 21:40 Reuben Thomas + + * Fix Python bindings (including recent Python 3 compatibility + update). + +2011-01-04 18:43 Reuben Thomas + + * magic/Makefile.am: make it easier to recover from magic build failures. + * Fix pstring length specifier parsing to avoid generating invalid + magic files. + * Add pstring length "J" (for "JPEG") to specify that the length + include itself. + * Fix JPEG comment parsing at last using pstring/HJ! + * Ignore section 5 man pages in doc/.cvsignore. + +2010-12-22 13:12 Christos Zoulas + + * Add pstring/BHhLl to specify the type of the length of pascal + strings. + +2010-11-26 18:39 Reuben Thomas + + * Fix "-e soft": it was ignored when softmagic was called + during asciimagic. + * Improve comments and use "unsigned char" in tar.h/is_tar.c. + +2010-11-05 17:26 Reuben Thomas + + * Make bug reporting addresses more visible. + +2010-11-01 18:35 Reuben Thomas + + * Add tcl magic from Gustaf Neumann + +2010-10-24 10:42 Christos Zoulas + + * Fix the whitespace comparing code (Christopher Chittleborough) + +2010-10-06 21:05 Christos Zoulas + + * allow string/t to work (Jan Kaluza) + +2010-09-20 22:11 Reuben Thomas + + * Apply some patches from Ubuntu and Fedora. + +2010-09-20 21:16 Reuben Thomas + + * Apply all patches from Debian package 5.04-6 which have not + already been applied and are not Debian-specific. + +2010-09-20 15:24 Reuben Thomas + + * Minor security fix to softmagic.c (don't use untrusted + string as printf format). + +2010-07-21 12:20 Christos Zoulas + + * MINGW32 portability from LRN + + * Don't warn about escaping magic regex chars when we are in a regex. + +2010-07-19 10:55 Christos Zoulas + + * Only try to print prpsinfo for core files. (Jan Kaluza) + +2010-04-22 12:55 Christos Zoulas + + * Try more elf offsets for Debian core files. (Arnaud Giersch) + +2010-02-20 15:18 Reuben Thomas + + * Clarify which sort of CDF we mean. + +2010-02-14 22:58 Reuben Thomas + + * Re-jig Zip file type magic so that unsupported special + Zip types (those with "mimetype" at offset 30) can be + recognized. + +2010-02-02 21:50 Reuben Thomas + + * Add support for OCF (EPUB) files (application/epub+zip) + +2010-01-28 18:25 Christos Zoulas + + * Fix core-dump from unbound loop: + https://bugzilla.redhat.com/show_bug.cgi?id=533245 + +2010-01-22 15:45 Christos Zoulas + + * print proper mime for crystal reports file + + * print the last summary information of a cdf document, not the + first so that nested documents print the right info + +2010-01-16 18:42 Charles Longeau + + * bring back some fixes from OpenBSD: + - make gcc2 builds file + - fix typos in a magic file comment + +2009-11-17 18:35 Christos Zoulas + + * ctime/asctime can return NULL on some OS's although + they should not (Toshit Antani) + +2009-09-14 13:49 Christos Zoulas + + * Centralize magic path handling routines and remove the + special-casing from file.c so that the python module for + example comes up with the same magic path (Fixes ~/.magic + handling) (from Gab) + +2009-09-11 23:38 Reuben Thomas + + * When magic argument is a directory, read the files in + strcmp-sorted order (fixes Debian bug #488562 and our own FIXME). + +2009-09-11 13:11 Reuben Thomas + + * Combine overlapping epoc and psion magic files into one (epoc). + + * Add some more EPOC MIME types. + +2009-08-19 15:55 Christos Zoulas + + * Fix 3 bugs (From Ian Darwin): + - file_showstr could move one past the end of the array + - parse_apple did not nul terminate the string in the overflow case + - parse_mime truncated the wrong string in the overflow case + +2009-08-12 12:28 Robert Byrnes + + * Include Localstuff when compiling magic. + +2009-07-15 10:05 Christos Zoulas + + * Fix logic for including mygetopts.h + + * Make cdf.c compile again with debugging + + * Add the necessary field handling for crystal reports files to work + +2009-06-23 01:34 Reuben Thomas + + * Stop "(if" identifying Lisp files, that's plain dumb! + +2009-06-09 22:13 Reuben Thomas + + * Add a couple of missing MP3 MIME types. + +2009-05-27 23:00 Reuben Thomas + + * Add full range of hash-bang tests for Python and Ruby. + + * Add MIME types for Python and Ruby scripts. + +2009-05-13 10:44 Christos Zoulas + + * off by one in parsing hw capabilities in elf + (Cheng Renquan) + +2009-05-08 13:40 Christos Zoulas + + * lint fixes and more from NetBSD + +2009-05-06 10:25 Christos Zoulas + + * Avoid null dereference in cdf code (Drew Yao) + + * More cdf bounds checks and overflow checks + +2009-05-01 18:37 Christos Zoulas + + * Buffer overflow fixes from Drew Yao + +2009-04-30 17:10 Christos Zoulas + + * Fix more cdf lossage. All the documents I have + right now print the correct information. + +2009-03-27 18:43 Christos Zoulas + + * don't print \012- separators in the same magic entry + if it consists of multiple magic printing lines. + +2009-03-23 10:20 Christos Zoulas + + * Avoid file descriptor leak in compress code from + (Daniel Novotny) + +2009-03-18 16:50 Christos Zoulas + + * Allow escaping of relation characters, so that we can say \^[A-Z] + and the ^ is not eaten as a relation char. + + * Fix troff and fortran to their previous glory using + regex. This was broken since their removel from ascmagic. + +2009-03-10 16:50 Christos Zoulas + + * don't use strlen in strndup() (Toby Peterson) + +2009-03-10 7:45 Christos Zoulas + + * avoid c99 syntax. + +2009-02-23 15:45 Christos Zoulas + + * make the cdf code use the buffer first if available, + and then the fd code. + +2009-02-13 13:45 Christos Zoulas + + * look for struct option to determine if getopt.h is usable for IRIX. + + * sanitize cdf document strings + +2009-02-04 13:25 Christos Zoulas + + * fix OS/2 warnings. + +2008-12-12 15:50 Christos Zoulas + + * fix initial offset calculation for non 4K sector files + + * add loop limits to avoid DoS attacks by constructing + looping sector references. + +2008-12-03 13:05 Christos Zoulas + + * fix memory botches on cdf file parsing. + + * exit with non-zero value for any error, not just for the last + file processed. + +2008-11-09 20:42 Charles Longeau + + * Replace all str{cpy,cat} functions with strl{cpy,cat} + * Ensure that strl{cpy,cat} are included in libmagic, + as needed. + +2008-11-06 18:18 Christos Zoulas + + * Handle ID3 format files. + +2008-11-06 23:00 Reuben Thomas + + * Fix --mime, --mime-type and --mime-encoding under new scheme. + + * Rename "ascii" to "text" and add "encoding" test. + + * Return a precise ("utf-16le" or "utf-16be") MIME charset for + UTF-16. + + * Fix error in comment caused by automatic indentation adding + words! *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Mon Jul 30 19:44:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42D31105F4EB; Mon, 30 Jul 2018 19:44:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB79181519; Mon, 30 Jul 2018 19:44:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B70F7F3F; Mon, 30 Jul 2018 19:44:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6UJiFAG043396; Mon, 30 Jul 2018 19:44:15 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6UJiFdh043393; Mon, 30 Jul 2018 19:44:15 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807301944.w6UJiFdh043393@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 30 Jul 2018 19:44:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336930 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 336930 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 19:44:16 -0000 Author: mav Date: Mon Jul 30 19:44:14 2018 New Revision: 336930 URL: https://svnweb.freebsd.org/changeset/base/336930 Log: 9284 arc_reclaim_thread has 2 jobs `arc_reclaim_thread()` calls `arc_adjust()` after calling `arc_kmem_reap_now()`; `arc_adjust()` signals `arc_get_data_buf()` to indicate that we may no longer be `arc_is_overflowing()`. The problem is, `arc_kmem_reap_now()` can take several seconds to complete, has no impact on `arc_is_overflowing()`, but due to how the code is structured, can impact how long the ARC will remain in the `arc_is_overflowing()` state. The fix is to use seperate threads to: 1. keep `arc_size` under `arc_c`, by calling `arc_adjust()`, which improves `arc_is_overflowing()` 2. keep enough free memory in the system, by calling `arc_kmem_reap_now()` plus `arc_shrink()`, which improves `arc_available_memory()`. illumos/illumos-gate@de753e34f9c399037936e8bc547d823bba9d4b0d Reviewed by: Matt Ahrens Reviewed by: Serapheim Dimitropoulos Reviewed by: Pavel Zakharov Reviewed by: Dan Kimmel Reviewed by: Paul Dagnelie Reviewed by: Dan McDonald Reviewed by: Tim Kordas Approved by: Garrett D'Amore Author: Brad Lewis Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zthr.h vendor-sys/illumos/dist/uts/common/fs/zfs/zthr.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Mon Jul 30 19:41:54 2018 (r336929) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Mon Jul 30 19:44:14 2018 (r336930) @@ -274,6 +274,7 @@ #endif #include #include +#include #include #include #include @@ -284,11 +285,23 @@ boolean_t arc_watch = B_FALSE; int arc_procfd; #endif -static kmutex_t arc_reclaim_lock; -static kcondvar_t arc_reclaim_thread_cv; -static boolean_t arc_reclaim_thread_exit; -static kcondvar_t arc_reclaim_waiters_cv; +/* + * This thread's job is to keep enough free memory in the system, by + * calling arc_kmem_reap_now() plus arc_shrink(), which improves + * arc_available_memory(). + */ +static zthr_t *arc_reap_zthr; +/* + * This thread's job is to keep arc_size under arc_c, by calling + * arc_adjust(), which improves arc_is_overflowing(). + */ +static zthr_t *arc_adjust_zthr; + +static kmutex_t arc_adjust_lock; +static kcondvar_t arc_adjust_waiters_cv; +static boolean_t arc_adjust_needed = B_FALSE; + uint_t arc_reduce_dnlc_percent = 3; /* @@ -301,19 +314,23 @@ uint_t arc_reduce_dnlc_percent = 3; int zfs_arc_evict_batch_limit = 10; /* number of seconds before growing cache again */ -static int arc_grow_retry = 60; +int arc_grow_retry = 60; -/* number of milliseconds before attempting a kmem-cache-reap */ -static int arc_kmem_cache_reap_retry_ms = 1000; +/* + * Minimum time between calls to arc_kmem_reap_soon(). Note that this will + * be converted to ticks, so with the default hz=100, a setting of 15 ms + * will actually wait 2 ticks, or 20ms. + */ +int arc_kmem_cache_reap_retry_ms = 1000; /* shift of arc_c for calculating overflow limit in arc_get_data_impl */ -int zfs_arc_overflow_shift = 8; +int zfs_arc_overflow_shift = 8; /* shift of arc_c for calculating both min and max arc_p */ -static int arc_p_min_shift = 4; +int arc_p_min_shift = 4; /* log2(fraction of arc to reclaim) */ -static int arc_shrink_shift = 7; +int arc_shrink_shift = 7; /* * log2(fraction of ARC which must be free to allow growing). @@ -338,7 +355,7 @@ static int arc_min_prefetch_lifespan; */ int arc_lotsfree_percent = 10; -static int arc_dead; +static boolean_t arc_initialized; /* * The arc has filled available memory and has now warmed up. @@ -840,6 +857,7 @@ aggsum_t astat_other_size; aggsum_t astat_l2_hdr_size; static int arc_no_grow; /* Don't try to grow cache size */ +static hrtime_t arc_growtime; static uint64_t arc_tempreserve; static uint64_t arc_loaned_bytes; @@ -1399,8 +1417,8 @@ hdr_recl(void *unused) * umem calls the reclaim func when we destroy the buf cache, * which is after we do arc_fini(). */ - if (!arc_dead) - cv_signal(&arc_reclaim_thread_cv); + if (arc_initialized) + zthr_wakeup(arc_reap_zthr); } static void @@ -3413,13 +3431,14 @@ arc_evict_state_impl(multilist_t *ml, int idx, arc_buf * function should proceed in this case). * * If threads are left sleeping, due to not - * using cv_broadcast, they will be woken up - * just before arc_reclaim_thread() sleeps. + * using cv_broadcast here, they will be woken + * up via cv_broadcast in arc_adjust_cb() just + * before arc_adjust_zthr sleeps. */ - mutex_enter(&arc_reclaim_lock); + mutex_enter(&arc_adjust_lock); if (!arc_is_overflowing()) - cv_signal(&arc_reclaim_waiters_cv); - mutex_exit(&arc_reclaim_lock); + cv_signal(&arc_adjust_waiters_cv); + mutex_exit(&arc_adjust_lock); } else { ARCSTAT_BUMP(arcstat_mutex_miss); } @@ -3892,8 +3911,8 @@ arc_flush(spa_t *spa, boolean_t retry) (void) arc_flush_state(arc_mfu_ghost, guid, ARC_BUFC_METADATA, retry); } -void -arc_shrink(int64_t to_free) +static void +arc_reduce_target_size(int64_t to_free) { uint64_t asize = aggsum_value(&arc_size); if (arc_c > arc_c_min) { @@ -3912,8 +3931,13 @@ arc_shrink(int64_t to_free) ASSERT((int64_t)arc_p >= 0); } - if (asize > arc_c) - (void) arc_adjust(); + if (asize > arc_c) { + /* See comment in arc_adjust_cb_check() on why lock+flag */ + mutex_enter(&arc_adjust_lock); + arc_adjust_needed = B_TRUE; + mutex_exit(&arc_adjust_lock); + zthr_wakeup(arc_adjust_zthr); + } } typedef enum free_memory_reason_t { @@ -4065,7 +4089,7 @@ arc_reclaim_needed(void) } static void -arc_kmem_reap_now(void) +arc_kmem_reap_soon(void) { size_t i; kmem_cache_t *prev_cache = NULL; @@ -4091,16 +4115,6 @@ arc_kmem_reap_now(void) #endif #endif - /* - * If a kmem reap is already active, don't schedule more. We must - * check for this because kmem_cache_reap_soon() won't actually - * block on the cache being reaped (this is to prevent callers from - * becoming implicitly blocked by a system-wide kmem reap -- which, - * on a system with many, many full magazines, can take minutes). - */ - if (kmem_cache_reap_active()) - return; - for (i = 0; i < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; i++) { if (zio_buf_cache[i] != prev_cache) { prev_cache = zio_buf_cache[i]; @@ -4126,139 +4140,162 @@ arc_kmem_reap_now(void) } } +/* ARGSUSED */ +static boolean_t +arc_adjust_cb_check(void *arg, zthr_t *zthr) +{ + /* + * This is necessary in order for the mdb ::arc dcmd to + * show up to date information. Since the ::arc command + * does not call the kstat's update function, without + * this call, the command may show stale stats for the + * anon, mru, mru_ghost, mfu, and mfu_ghost lists. Even + * with this change, the data might be up to 1 second + * out of date(the arc_adjust_zthr has a maximum sleep + * time of 1 second); but that should suffice. The + * arc_state_t structures can be queried directly if more + * accurate information is needed. + */ + if (arc_ksp != NULL) + arc_ksp->ks_update(arc_ksp, KSTAT_READ); + + /* + * We have to rely on arc_get_data_impl() to tell us when to adjust, + * rather than checking if we are overflowing here, so that we are + * sure to not leave arc_get_data_impl() waiting on + * arc_adjust_waiters_cv. If we have become "not overflowing" since + * arc_get_data_impl() checked, we need to wake it up. We could + * broadcast the CV here, but arc_get_data_impl() may have not yet + * gone to sleep. We would need to use a mutex to ensure that this + * function doesn't broadcast until arc_get_data_impl() has gone to + * sleep (e.g. the arc_adjust_lock). However, the lock ordering of + * such a lock would necessarily be incorrect with respect to the + * zthr_lock, which is held before this function is called, and is + * held by arc_get_data_impl() when it calls zthr_wakeup(). + */ + return (arc_adjust_needed); +} + /* - * Threads can block in arc_get_data_impl() waiting for this thread to evict - * enough data and signal them to proceed. When this happens, the threads in - * arc_get_data_impl() are sleeping while holding the hash lock for their - * particular arc header. Thus, we must be careful to never sleep on a - * hash lock in this thread. This is to prevent the following deadlock: - * - * - Thread A sleeps on CV in arc_get_data_impl() holding hash lock "L", - * waiting for the reclaim thread to signal it. - * - * - arc_reclaim_thread() tries to acquire hash lock "L" using mutex_enter, - * fails, and goes to sleep forever. - * - * This possible deadlock is avoided by always acquiring a hash lock - * using mutex_tryenter() from arc_reclaim_thread(). + * Keep arc_size under arc_c by running arc_adjust which evicts data + * from the ARC. */ /* ARGSUSED */ -static void -arc_reclaim_thread(void *unused) +static int +arc_adjust_cb(void *arg, zthr_t *zthr) { - hrtime_t growtime = 0; - hrtime_t kmem_reap_time = 0; - callb_cpr_t cpr; + uint64_t evicted = 0; - CALLB_CPR_INIT(&cpr, &arc_reclaim_lock, callb_generic_cpr, FTAG); + /* Evict from cache */ + evicted = arc_adjust(); - mutex_enter(&arc_reclaim_lock); - while (!arc_reclaim_thread_exit) { - uint64_t evicted = 0; - + /* + * If evicted is zero, we couldn't evict anything + * via arc_adjust(). This could be due to hash lock + * collisions, but more likely due to the majority of + * arc buffers being unevictable. Therefore, even if + * arc_size is above arc_c, another pass is unlikely to + * be helpful and could potentially cause us to enter an + * infinite loop. Additionally, zthr_iscancelled() is + * checked here so that if the arc is shutting down, the + * broadcast will wake any remaining arc adjust waiters. + */ + mutex_enter(&arc_adjust_lock); + arc_adjust_needed = !zthr_iscancelled(arc_adjust_zthr) && + evicted > 0 && aggsum_compare(&arc_size, arc_c) > 0; + if (!arc_adjust_needed) { /* - * This is necessary in order for the mdb ::arc dcmd to - * show up to date information. Since the ::arc command - * does not call the kstat's update function, without - * this call, the command may show stale stats for the - * anon, mru, mru_ghost, mfu, and mfu_ghost lists. Even - * with this change, the data might be up to 1 second - * out of date; but that should suffice. The arc_state_t - * structures can be queried directly if more accurate - * information is needed. + * We're either no longer overflowing, or we + * can't evict anything more, so we should wake + * up any waiters. */ - if (arc_ksp != NULL) - arc_ksp->ks_update(arc_ksp, KSTAT_READ); + cv_broadcast(&arc_adjust_waiters_cv); + } + mutex_exit(&arc_adjust_lock); - mutex_exit(&arc_reclaim_lock); + return (0); +} +/* ARGSUSED */ +static boolean_t +arc_reap_cb_check(void *arg, zthr_t *zthr) +{ + int64_t free_memory = arc_available_memory(); + + /* + * If a kmem reap is already active, don't schedule more. We must + * check for this because kmem_cache_reap_soon() won't actually + * block on the cache being reaped (this is to prevent callers from + * becoming implicitly blocked by a system-wide kmem reap -- which, + * on a system with many, many full magazines, can take minutes). + */ + if (!kmem_cache_reap_active() && + free_memory < 0) { + arc_no_grow = B_TRUE; + arc_warm = B_TRUE; /* - * We call arc_adjust() before (possibly) calling - * arc_kmem_reap_now(), so that we can wake up - * arc_get_data_impl() sooner. + * Wait at least zfs_grow_retry (default 60) seconds + * before considering growing. */ - evicted = arc_adjust(); + arc_growtime = gethrtime() + SEC2NSEC(arc_grow_retry); + return (B_TRUE); + } else if (free_memory < arc_c >> arc_no_grow_shift) { + arc_no_grow = B_TRUE; + } else if (gethrtime() >= arc_growtime) { + arc_no_grow = B_FALSE; + } - int64_t free_memory = arc_available_memory(); - if (free_memory < 0) { - hrtime_t curtime = gethrtime(); - arc_no_grow = B_TRUE; - arc_warm = B_TRUE; + return (B_FALSE); +} - /* - * Wait at least zfs_grow_retry (default 60) seconds - * before considering growing. - */ - growtime = curtime + SEC2NSEC(arc_grow_retry); +/* + * Keep enough free memory in the system by reaping the ARC's kmem + * caches. To cause more slabs to be reapable, we may reduce the + * target size of the cache (arc_c), causing the arc_adjust_cb() + * to free more buffers. + */ +/* ARGSUSED */ +static int +arc_reap_cb(void *arg, zthr_t *zthr) +{ + int64_t free_memory; - /* - * Wait at least arc_kmem_cache_reap_retry_ms - * between arc_kmem_reap_now() calls. Without - * this check it is possible to end up in a - * situation where we spend lots of time - * reaping caches, while we're near arc_c_min. - */ - if (curtime >= kmem_reap_time) { - arc_kmem_reap_now(); - kmem_reap_time = gethrtime() + - MSEC2NSEC(arc_kmem_cache_reap_retry_ms); - } + /* + * Kick off asynchronous kmem_reap()'s of all our caches. + */ + arc_kmem_reap_soon(); - /* - * If we are still low on memory, shrink the ARC - * so that we have arc_shrink_min free space. - */ - free_memory = arc_available_memory(); + /* + * Wait at least arc_kmem_cache_reap_retry_ms between + * arc_kmem_reap_soon() calls. Without this check it is possible to + * end up in a situation where we spend lots of time reaping + * caches, while we're near arc_c_min. Waiting here also gives the + * subsequent free memory check a chance of finding that the + * asynchronous reap has already freed enough memory, and we don't + * need to call arc_reduce_target_size(). + */ + delay((hz * arc_kmem_cache_reap_retry_ms + 999) / 1000); - int64_t to_free = - (arc_c >> arc_shrink_shift) - free_memory; - if (to_free > 0) { + /* + * Reduce the target size as needed to maintain the amount of free + * memory in the system at a fraction of the arc_size (1/128th by + * default). If oversubscribed (free_memory < 0) then reduce the + * target arc_size by the deficit amount plus the fractional + * amount. If free memory is positive but less then the fractional + * amount, reduce by what is needed to hit the fractional amount. + */ + free_memory = arc_available_memory(); + + int64_t to_free = + (arc_c >> arc_shrink_shift) - free_memory; + if (to_free > 0) { #ifdef _KERNEL - to_free = MAX(to_free, ptob(needfree)); + to_free = MAX(to_free, ptob(needfree)); #endif - arc_shrink(to_free); - } - } else if (free_memory < arc_c >> arc_no_grow_shift) { - arc_no_grow = B_TRUE; - } else if (gethrtime() >= growtime) { - arc_no_grow = B_FALSE; - } - - mutex_enter(&arc_reclaim_lock); - - /* - * If evicted is zero, we couldn't evict anything via - * arc_adjust(). This could be due to hash lock - * collisions, but more likely due to the majority of - * arc buffers being unevictable. Therefore, even if - * arc_size is above arc_c, another pass is unlikely to - * be helpful and could potentially cause us to enter an - * infinite loop. - */ - if (aggsum_compare(&arc_size, arc_c) <= 0|| evicted == 0) { - /* - * We're either no longer overflowing, or we - * can't evict anything more, so we should wake - * up any threads before we go to sleep. - */ - cv_broadcast(&arc_reclaim_waiters_cv); - - /* - * Block until signaled, or after one second (we - * might need to perform arc_kmem_reap_now() - * even if we aren't being signalled) - */ - CALLB_CPR_SAFE_BEGIN(&cpr); - (void) cv_timedwait_hires(&arc_reclaim_thread_cv, - &arc_reclaim_lock, SEC2NSEC(1), MSEC2NSEC(1), 0); - CALLB_CPR_SAFE_END(&cpr, &arc_reclaim_lock); - } + arc_reduce_target_size(to_free); } - arc_reclaim_thread_exit = B_FALSE; - cv_broadcast(&arc_reclaim_thread_cv); - CALLB_CPR_EXIT(&cpr); /* drops arc_reclaim_lock */ - thread_exit(); + return (0); } /* @@ -4302,11 +4339,15 @@ arc_adapt(int bytes, arc_state_t *state) } ASSERT((int64_t)arc_p >= 0); + /* + * Wake reap thread if we do not have any available memory + */ if (arc_reclaim_needed()) { - cv_signal(&arc_reclaim_thread_cv); + zthr_wakeup(arc_reap_zthr); return; } + if (arc_no_grow) return; @@ -4410,7 +4451,7 @@ arc_get_data_impl(arc_buf_hdr_t *hdr, uint64_t size, v * overflowing; thus we don't use a while loop here. */ if (arc_is_overflowing()) { - mutex_enter(&arc_reclaim_lock); + mutex_enter(&arc_adjust_lock); /* * Now that we've acquired the lock, we may no longer be @@ -4424,11 +4465,12 @@ arc_get_data_impl(arc_buf_hdr_t *hdr, uint64_t size, v * shouldn't cause any harm. */ if (arc_is_overflowing()) { - cv_signal(&arc_reclaim_thread_cv); - cv_wait(&arc_reclaim_waiters_cv, &arc_reclaim_lock); + arc_adjust_needed = B_TRUE; + zthr_wakeup(arc_adjust_zthr); + (void) cv_wait(&arc_adjust_waiters_cv, + &arc_adjust_lock); } - - mutex_exit(&arc_reclaim_lock); + mutex_exit(&arc_adjust_lock); } VERIFY3U(hdr->b_type, ==, type); @@ -6081,11 +6123,9 @@ arc_init(void) #else uint64_t allmem = (physmem * PAGESIZE) / 2; #endif + mutex_init(&arc_adjust_lock, NULL, MUTEX_DEFAULT, NULL); + cv_init(&arc_adjust_waiters_cv, NULL, CV_DEFAULT, NULL); - mutex_init(&arc_reclaim_lock, NULL, MUTEX_DEFAULT, NULL); - cv_init(&arc_reclaim_thread_cv, NULL, CV_DEFAULT, NULL); - cv_init(&arc_reclaim_waiters_cv, NULL, CV_DEFAULT, NULL); - /* Convert seconds to clock ticks */ arc_min_prefetch_lifespan = 1 * hz; @@ -6169,10 +6209,15 @@ arc_init(void) arc_c = arc_c_min; arc_state_init(); + + /* + * The arc must be "uninitialized", so that hdr_recl() (which is + * registered by buf_init()) will not access arc_reap_zthr before + * it is created. + */ + ASSERT(!arc_initialized); buf_init(); - arc_reclaim_thread_exit = B_FALSE; - arc_ksp = kstat_create("zfs", 0, "arcstats", "misc", KSTAT_TYPE_NAMED, sizeof (arc_stats) / sizeof (kstat_named_t), KSTAT_FLAG_VIRTUAL); @@ -6182,10 +6227,12 @@ arc_init(void) kstat_install(arc_ksp); } - (void) thread_create(NULL, 0, arc_reclaim_thread, NULL, 0, &p0, - TS_RUN, minclsyspri); + arc_adjust_zthr = zthr_create(arc_adjust_cb_check, + arc_adjust_cb, NULL); + arc_reap_zthr = zthr_create_timer(arc_reap_cb_check, + arc_reap_cb, NULL, SEC2NSEC(1)); - arc_dead = B_FALSE; + arc_initialized = B_TRUE; arc_warm = B_FALSE; /* @@ -6207,31 +6254,24 @@ arc_init(void) void arc_fini(void) { - mutex_enter(&arc_reclaim_lock); - arc_reclaim_thread_exit = B_TRUE; - /* - * The reclaim thread will set arc_reclaim_thread_exit back to - * B_FALSE when it is finished exiting; we're waiting for that. - */ - while (arc_reclaim_thread_exit) { - cv_signal(&arc_reclaim_thread_cv); - cv_wait(&arc_reclaim_thread_cv, &arc_reclaim_lock); - } - mutex_exit(&arc_reclaim_lock); - /* Use B_TRUE to ensure *all* buffers are evicted */ arc_flush(NULL, B_TRUE); - arc_dead = B_TRUE; + arc_initialized = B_FALSE; if (arc_ksp != NULL) { kstat_delete(arc_ksp); arc_ksp = NULL; } - mutex_destroy(&arc_reclaim_lock); - cv_destroy(&arc_reclaim_thread_cv); - cv_destroy(&arc_reclaim_waiters_cv); + (void) zthr_cancel(arc_adjust_zthr); + zthr_destroy(arc_adjust_zthr); + + (void) zthr_cancel(arc_reap_zthr); + zthr_destroy(arc_reap_zthr); + + mutex_destroy(&arc_adjust_lock); + cv_destroy(&arc_adjust_waiters_cv); arc_state_fini(); buf_fini(); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zthr.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zthr.h Mon Jul 30 19:41:54 2018 (r336929) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zthr.h Mon Jul 30 19:44:14 2018 (r336930) @@ -29,6 +29,7 @@ struct zthr { kmutex_t zthr_lock; kcondvar_t zthr_cv; boolean_t zthr_cancel; + hrtime_t zthr_wait_time; zthr_checkfunc_t *zthr_checkfunc; zthr_func_t *zthr_func; @@ -38,6 +39,9 @@ struct zthr { extern zthr_t *zthr_create(zthr_checkfunc_t checkfunc, zthr_func_t *func, void *arg); +extern zthr_t *zthr_create_timer(zthr_checkfunc_t *checkfunc, + zthr_func_t *func, void *arg, hrtime_t nano_wait); + extern void zthr_exit(zthr_t *t, int rc); extern void zthr_destroy(zthr_t *t); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zthr.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zthr.c Mon Jul 30 19:41:54 2018 (r336929) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zthr.c Mon Jul 30 19:44:14 2018 (r336930) @@ -47,6 +47,10 @@ * 3] When the zthr is done, it changes the indicator to stopped, allowing * a new cycle to start. * + * Besides being awakened by other threads, a zthr can be configured + * during creation to wakeup on it's own after a specified interval + * [see zthr_create_timer()]. + * * == ZTHR creation * * Every zthr needs three inputs to start running: @@ -74,6 +78,9 @@ * * To start a zthr: * zthr_t *zthr_pointer = zthr_create(checkfunc, func, args); + * or + * zthr_t *zthr_pointer = zthr_create_timer(checkfunc, func, + * args, max_sleep); * * After that you should be able to wakeup, cancel, and resume the * zthr from another thread using zthr_pointer. @@ -189,7 +196,13 @@ zthr_procedure(void *arg) mutex_enter(&t->zthr_lock); } else { /* go to sleep */ - cv_wait(&t->zthr_cv, &t->zthr_lock); + if (t->zthr_wait_time == 0) { + cv_wait(&t->zthr_cv, &t->zthr_lock); + } else { + (void) cv_timedwait_hires(&t->zthr_cv, + &t->zthr_lock, t->zthr_wait_time, + MSEC2NSEC(1), 0); + } } } mutex_exit(&t->zthr_lock); @@ -200,6 +213,18 @@ zthr_procedure(void *arg) zthr_t * zthr_create(zthr_checkfunc_t *checkfunc, zthr_func_t *func, void *arg) { + return (zthr_create_timer(checkfunc, func, arg, (hrtime_t)0)); +} + +/* + * Create a zthr with specified maximum sleep time. If the time + * in sleeping state exceeds max_sleep, a wakeup(do the check and + * start working if required) will be triggered. + */ +zthr_t * +zthr_create_timer(zthr_checkfunc_t *checkfunc, zthr_func_t *func, + void *arg, hrtime_t max_sleep) +{ zthr_t *t = kmem_zalloc(sizeof (*t), KM_SLEEP); mutex_init(&t->zthr_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&t->zthr_cv, NULL, CV_DEFAULT, NULL); @@ -208,6 +233,7 @@ zthr_create(zthr_checkfunc_t *checkfunc, zthr_func_t * t->zthr_checkfunc = checkfunc; t->zthr_func = func; t->zthr_arg = arg; + t->zthr_wait_time = max_sleep; t->zthr_thread = thread_create(NULL, 0, zthr_procedure, t, 0, &p0, TS_RUN, minclsyspri); From owner-svn-src-vendor@freebsd.org Mon Jul 30 21:59:45 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CA9410621FF; Mon, 30 Jul 2018 21:59:45 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 05CF8867C4; Mon, 30 Jul 2018 21:59:45 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D6D6F250E; Mon, 30 Jul 2018 21:59:44 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6ULxieu012088; Mon, 30 Jul 2018 21:59:44 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6ULxi1A012086; Mon, 30 Jul 2018 21:59:44 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807302159.w6ULxi1A012086@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 30 Jul 2018 21:59:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336942 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 336942 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 21:59:45 -0000 Author: mav Date: Mon Jul 30 21:59:44 2018 New Revision: 336942 URL: https://svnweb.freebsd.org/changeset/base/336942 Log: 9189 Add debug to vdev_label_read_config when txg check fails illumos/illumos-gate@b6bf6e1540f30bd97b8d6e2c21d95e17841e0f23 Reviewed by: George Wilson Reviewed by: Prashanth Sreenivasa Approved by: Matt Ahrens Author: Pavel Zakharov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Mon Jul 30 21:29:26 2018 (r336941) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Mon Jul 30 21:59:44 2018 (r336942) @@ -1585,7 +1585,8 @@ vdev_validate(vdev_t *vd) if ((label = vdev_label_read_config(vd, txg)) == NULL) { vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN, VDEV_AUX_BAD_LABEL); - vdev_dbgmsg(vd, "vdev_validate: failed reading config"); + vdev_dbgmsg(vd, "vdev_validate: failed reading config for " + "txg %llu", (u_longlong_t)txg); return (0); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Mon Jul 30 21:29:26 2018 (r336941) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Mon Jul 30 21:59:44 2018 (r336942) @@ -540,6 +540,7 @@ vdev_label_read_config(vdev_t *vd, uint64_t txg) abd_t *vp_abd; zio_t *zio; uint64_t best_txg = 0; + uint64_t label_txg = 0; int error = 0; int flags = ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE; @@ -565,8 +566,6 @@ retry: if (zio_wait(zio) == 0 && nvlist_unpack(vp->vp_nvlist, sizeof (vp->vp_nvlist), &label, 0) == 0) { - uint64_t label_txg = 0; - /* * Auxiliary vdevs won't have txg values in their * labels and newly added vdevs may not have been @@ -595,6 +594,15 @@ retry: if (config == NULL && !(flags & ZIO_FLAG_TRYHARD)) { flags |= ZIO_FLAG_TRYHARD; goto retry; + } + + /* + * We found a valid label but it didn't pass txg restrictions. + */ + if (config == NULL && label_txg != 0) { + vdev_dbgmsg(vd, "label discarded as txg is too large " + "(%llu > %llu)", (u_longlong_t)label_txg, + (u_longlong_t)txg); } abd_free(vp_abd); From owner-svn-src-vendor@freebsd.org Mon Jul 30 22:10:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1D5210625D2; Mon, 30 Jul 2018 22:10:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5491D86DFD; Mon, 30 Jul 2018 22:10:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 30C5826A3; Mon, 30 Jul 2018 22:10:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6UMAFrc017085; Mon, 30 Jul 2018 22:10:15 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6UMAFkO017082; Mon, 30 Jul 2018 22:10:15 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807302210.w6UMAFkO017082@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 30 Jul 2018 22:10:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336944 - in vendor/illumos/dist: lib/libzfs/common man/man1m X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor/illumos/dist: lib/libzfs/common man/man1m X-SVN-Commit-Revision: 336944 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 22:10:16 -0000 Author: mav Date: Mon Jul 30 22:10:15 2018 New Revision: 336944 URL: https://svnweb.freebsd.org/changeset/base/336944 Log: 9286 want refreservation=auto When a ZFS volume is created with zfs create -V (but without -s), the refreservation property is set to a value that is volsize plus the maximum size of metadata. If refreservation is ever set to another value, it is impossible to set it back to the automatically determined value. There are other cases where refreservation may be wrong. These include receiving a volume that was sent without properties and zfs clone. We need: zfs set refreservation=auto zfs clone -o refreservation=auto Each one would use the same function used by zfs create -V to determine the proper value for refreservation. illumos/illumos-gate@1c10ae76c0cb31326c320e7cef1d3f24a1f47125 Reviewed by: Allan Jude Reviewed by: Matthew Ahrens Reviewed by: John Kennedy Reviewed by: Andy Stormont Approved by: Richard Lowe Author: Mike Gerdts Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/man/man1m/zfs.1m Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Mon Jul 30 22:03:29 2018 (r336943) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Mon Jul 30 22:10:15 2018 (r336944) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright (c) 2018, Joyent, Inc. All rights reserved. * Copyright (c) 2011, 2016 by Delphix. All rights reserved. * Copyright (c) 2012 DEY Storage Systems, Inc. All rights reserved. * Copyright (c) 2011-2012 Pawel Jakub Dawidek. All rights reserved. @@ -1387,7 +1387,6 @@ badlabel: switch (prop) { case ZFS_PROP_RESERVATION: - case ZFS_PROP_REFRESERVATION: if (intval > volsize) { zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "'%s' is greater than current " @@ -1398,6 +1397,17 @@ badlabel: } break; + case ZFS_PROP_REFRESERVATION: + if (intval > volsize && intval != UINT64_MAX) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "'%s' is greater than current " + "volume size"), propname); + (void) zfs_error(hdl, EZFS_BADPROP, + errbuf); + goto error; + } + break; + case ZFS_PROP_VOLSIZE: if (intval % blocksize != 0) { zfs_nicenum(blocksize, buf, @@ -1499,6 +1509,61 @@ zfs_add_synthetic_resv(zfs_handle_t *zhp, nvlist_t *nv return (1); } +/* + * Helper for 'zfs {set|clone} refreservation=auto'. Must be called after + * zfs_valid_proplist(), as it is what sets the UINT64_MAX sentinal value. + * Return codes must match zfs_add_synthetic_resv(). + */ +static int +zfs_fix_auto_resv(zfs_handle_t *zhp, nvlist_t *nvl) +{ + uint64_t volsize; + uint64_t resvsize; + zfs_prop_t prop; + nvlist_t *props; + + if (!ZFS_IS_VOLUME(zhp)) { + return (0); + } + + if (zfs_which_resv_prop(zhp, &prop) != 0) { + return (-1); + } + + if (prop != ZFS_PROP_REFRESERVATION) { + return (0); + } + + if (nvlist_lookup_uint64(nvl, zfs_prop_to_name(prop), &resvsize) != 0) { + /* No value being set, so it can't be "auto" */ + return (0); + } + if (resvsize != UINT64_MAX) { + /* Being set to a value other than "auto" */ + return (0); + } + + props = fnvlist_alloc(); + + fnvlist_add_uint64(props, zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE), + zfs_prop_get_int(zhp, ZFS_PROP_VOLBLOCKSIZE)); + + if (nvlist_lookup_uint64(nvl, zfs_prop_to_name(ZFS_PROP_VOLSIZE), + &volsize) != 0) { + volsize = zfs_prop_get_int(zhp, ZFS_PROP_VOLSIZE); + } + + resvsize = zvol_volsize_to_reservation(volsize, props); + fnvlist_free(props); + + (void) nvlist_remove_all(nvl, zfs_prop_to_name(prop)); + if (nvlist_add_uint64(nvl, zfs_prop_to_name(prop), resvsize) != 0) { + (void) no_memory(zhp->zfs_hdl); + return (-1); + } + return (1); +} + void zfs_setprop_error(libzfs_handle_t *hdl, zfs_prop_t prop, int err, char *errbuf) @@ -1664,6 +1729,12 @@ zfs_prop_set_list(zfs_handle_t *zhp, nvlist_t *props) goto error; } } + + if (added_resv != 1 && + (added_resv = zfs_fix_auto_resv(zhp, nvl)) == -1) { + goto error; + } + /* * Check how many properties we're setting and allocate an array to * store changelist pointers for postfix(). @@ -3666,6 +3737,7 @@ zfs_clone(zfs_handle_t *zhp, const char *target, nvlis if (props) { zfs_type_t type; + if (ZFS_IS_VOLUME(zhp)) { type = ZFS_TYPE_VOLUME; } else { @@ -3674,6 +3746,10 @@ zfs_clone(zfs_handle_t *zhp, const char *target, nvlis if ((props = zfs_valid_proplist(hdl, type, props, zoned, zhp, zhp->zpool_hdl, errbuf)) == NULL) return (-1); + if (zfs_fix_auto_resv(zhp, props) == -1) { + nvlist_free(props); + return (-1); + } } ret = lzc_clone(target, zhp->zfs_name, props); Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_util.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Mon Jul 30 22:03:29 2018 (r336943) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Mon Jul 30 22:10:15 2018 (r336944) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright (c) 2018 Joyent, Inc. * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright 2016 Igor Kozhukhov * Copyright (c) 2017 Datto Inc. @@ -1184,6 +1184,7 @@ zprop_parse_value(libzfs_handle_t *hdl, nvpair_t *elem const char *propname; char *value; boolean_t isnone = B_FALSE; + boolean_t isauto = B_FALSE; if (type == ZFS_TYPE_POOL) { proptype = zpool_prop_get_type(prop); @@ -1219,8 +1220,9 @@ zprop_parse_value(libzfs_handle_t *hdl, nvpair_t *elem (void) nvpair_value_string(elem, &value); if (strcmp(value, "none") == 0) { isnone = B_TRUE; - } else if (zfs_nicestrtonum(hdl, value, ivalp) - != 0) { + } else if (strcmp(value, "auto") == 0) { + isauto = B_TRUE; + } else if (zfs_nicestrtonum(hdl, value, ivalp) != 0) { goto error; } } else if (datatype == DATA_TYPE_UINT64) { @@ -1250,6 +1252,31 @@ zprop_parse_value(libzfs_handle_t *hdl, nvpair_t *elem prop == ZFS_PROP_SNAPSHOT_LIMIT)) { *ivalp = UINT64_MAX; } + + /* + * Special handling for setting 'refreservation' to 'auto'. Use + * UINT64_MAX to tell the caller to use zfs_fix_auto_resv(). + * 'auto' is only allowed on volumes. + */ + if (isauto) { + switch (prop) { + case ZFS_PROP_REFRESERVATION: + if ((type & ZFS_TYPE_VOLUME) == 0) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "'%s=auto' only allowed on " + "volumes"), nvpair_name(elem)); + goto error; + } + *ivalp = UINT64_MAX; + break; + default: + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "'auto' is invalid value for '%s'"), + nvpair_name(elem)); + goto error; + } + } + break; case PROP_TYPE_INDEX: Modified: vendor/illumos/dist/man/man1m/zfs.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zfs.1m Mon Jul 30 22:03:29 2018 (r336943) +++ vendor/illumos/dist/man/man1m/zfs.1m Mon Jul 30 22:10:15 2018 (r336944) @@ -27,6 +27,7 @@ .\" Copyright (c) 2014 by Adam Stevko. All rights reserved. .\" Copyright (c) 2014 Integros [integros.com] .\" Copyright 2017 Nexenta Systems, Inc. +.\" Copyright 2018 Joyent, Inc. .\" .Dd December 6, 2017 .Dt ZFS 1M @@ -1345,7 +1346,7 @@ Limits the amount of space a dataset can consume. This property enforces a hard limit on the amount of space used. This hard limit does not include space used by descendents, including file systems and snapshots. -.It Sy refreservation Ns = Ns Em size Ns | Ns Sy none +.It Sy refreservation Ns = Ns Em size Ns | Ns Sy none Ns | Ns Sy auto The minimum amount of space guaranteed to a dataset, not including its descendents. When the amount of space used is below this value, the dataset is treated as if @@ -1363,6 +1364,22 @@ this reservation to accommodate the current number of .Qq referenced bytes in the dataset. .Pp +If +.Sy refreservation +is set to +.Sy auto , +a volume is thick provisioned +.Po or +.Qq not sparse +.Pc . +.Sy refreservation Ns = Ns Sy auto +is only supported on volumes. +See +.Sy volsize +in the +.Sx Native Properties +section for more information about sparse volumes. +.Pp This property can also be referred to by its shortened column name, .Sy refreserv . .It Sy reservation Ns = Ns Em size Ns | Ns Sy none @@ -1577,22 +1594,39 @@ Extreme care should be used when adjusting the volume Though not recommended, a .Qq sparse volume .Po also known as -.Qq thin provisioning +.Qq thin provisioned .Pc can be created by specifying the .Fl s option to the .Nm zfs Cm create Fl V -command, or by changing the reservation after the volume has been created. +command, or by changing the value of the +.Sy refreservation +property +.Po or +.Sy reservation +property on pool version 8 or earlier +.Pc +after the volume has been created. A .Qq sparse volume -is a volume where the reservation is less then the volume size. +is a volume where the value of +.Sy refreservation +is less than the size of the volume plus the space required to store its +metadata. Consequently, writes to a sparse volume can fail with .Er ENOSPC when the pool is low on space. For a sparse volume, changes to .Sy volsize -are not reflected in the reservation. +are not reflected in the +.Sy refreservation. +A volume that is not sparse is said to be +.Qq thick provisioned . +A sparse volume can become thick provisioned by setting +.Sy refreservation +to +.Sy auto . .It Sy vscan Ns = Ns Sy on Ns | Ns Sy off Controls whether regular files should be scanned for viruses when a file is opened and closed. From owner-svn-src-vendor@freebsd.org Mon Jul 30 22:56:27 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79C01106365A; Mon, 30 Jul 2018 22:56:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CABE89729; Mon, 30 Jul 2018 22:56:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0DA282F0F; Mon, 30 Jul 2018 22:56:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6UMuQDX043718; Mon, 30 Jul 2018 22:56:26 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6UMuQMD043714; Mon, 30 Jul 2018 22:56:26 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807302256.w6UMuQMD043714@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 30 Jul 2018 22:56:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336946 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/... X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist... X-SVN-Commit-Revision: 336946 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 22:56:27 -0000 Author: mav Date: Mon Jul 30 22:56:24 2018 New Revision: 336946 URL: https://svnweb.freebsd.org/changeset/base/336946 Log: 9238 ZFS Spacemap Encoding V2 The current space map encoding has the following disadvantages: [1] Assuming 512 sector size each entry can represent at most 16MB for a segment. This makes the encoding very inefficient for large regions of space. [2] As vdev-wide space maps have started to be used by new features (i.e. device removal, zpool checkpoint) we've started imposing limits in the vdevs that can be used with them based on the maximum addressable offset (currently 64PB for a top-level vdev). The new remains backwards compatible with the old one. The introduced two-word entry format, besides extending the limits imposed by the single-entry layout, also includes a vdev field and some extra padding after its prefix. The extra padding after the prefix should is reserved for future usage (e.g. new prefixes for future encodings or new fields for flags). The new vdev field not only makes the space maps more self-descriptive, but also opens the doors for pool-wide space maps. One final important note is that the number of bits used for vdevs is reduced to 24 bits for blkptrs. That was decided as we don't know of any setups that use more than 16M vdevs for the time being and we wanted to fit the vdev field in the space map. In addition that gives us some extra bits in dva_t. illumos/illumos-gate@17f11284b49b98353b5119463254074fd9bc0a28 Reviewed by: Matt Ahrens Reviewed by: George Wilson Approved by: Gordon Ross Author: Serapheim Dimitropoulos Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/man/man5/zpool-features.5 Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/space_map.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Mon Jul 30 22:39:30 2018 (r336945) +++ vendor/illumos/dist/cmd/zdb/zdb.c Mon Jul 30 22:56:24 2018 (r336946) @@ -774,7 +774,6 @@ verify_spacemap_refcounts(spa_t *spa) static void dump_spacemap(objset_t *os, space_map_t *sm) { - uint64_t alloc, offset, entry; char *ddata[] = { "ALLOC", "FREE", "CONDENSE", "INVALID", "INVALID", "INVALID", "INVALID", "INVALID" }; @@ -791,41 +790,73 @@ dump_spacemap(objset_t *os, space_map_t *sm) /* * Print out the freelist entries in both encoded and decoded form. */ - alloc = 0; - for (offset = 0; offset < space_map_length(sm); - offset += sizeof (entry)) { - uint8_t mapshift = sm->sm_shift; + uint8_t mapshift = sm->sm_shift; + int64_t alloc = 0; + uint64_t word; + for (uint64_t offset = 0; offset < space_map_length(sm); + offset += sizeof (word)) { VERIFY0(dmu_read(os, space_map_object(sm), offset, - sizeof (entry), &entry, DMU_READ_PREFETCH)); - if (SM_DEBUG_DECODE(entry)) { + sizeof (word), &word, DMU_READ_PREFETCH)); + if (sm_entry_is_debug(word)) { (void) printf("\t [%6llu] %s: txg %llu, pass %llu\n", - (u_longlong_t)(offset / sizeof (entry)), - ddata[SM_DEBUG_ACTION_DECODE(entry)], - (u_longlong_t)SM_DEBUG_TXG_DECODE(entry), - (u_longlong_t)SM_DEBUG_SYNCPASS_DECODE(entry)); + (u_longlong_t)(offset / sizeof (word)), + ddata[SM_DEBUG_ACTION_DECODE(word)], + (u_longlong_t)SM_DEBUG_TXG_DECODE(word), + (u_longlong_t)SM_DEBUG_SYNCPASS_DECODE(word)); + continue; + } + + uint8_t words; + char entry_type; + uint64_t entry_off, entry_run, entry_vdev = SM_NO_VDEVID; + + if (sm_entry_is_single_word(word)) { + entry_type = (SM_TYPE_DECODE(word) == SM_ALLOC) ? + 'A' : 'F'; + entry_off = (SM_OFFSET_DECODE(word) << mapshift) + + sm->sm_start; + entry_run = SM_RUN_DECODE(word) << mapshift; + words = 1; } else { - (void) printf("\t [%6llu] %c range:" - " %010llx-%010llx size: %06llx\n", - (u_longlong_t)(offset / sizeof (entry)), - SM_TYPE_DECODE(entry) == SM_ALLOC ? 'A' : 'F', - (u_longlong_t)((SM_OFFSET_DECODE(entry) << - mapshift) + sm->sm_start), - (u_longlong_t)((SM_OFFSET_DECODE(entry) << - mapshift) + sm->sm_start + - (SM_RUN_DECODE(entry) << mapshift)), - (u_longlong_t)(SM_RUN_DECODE(entry) << mapshift)); - if (SM_TYPE_DECODE(entry) == SM_ALLOC) - alloc += SM_RUN_DECODE(entry) << mapshift; - else - alloc -= SM_RUN_DECODE(entry) << mapshift; + /* it is a two-word entry so we read another word */ + ASSERT(sm_entry_is_double_word(word)); + + uint64_t extra_word; + offset += sizeof (extra_word); + VERIFY0(dmu_read(os, space_map_object(sm), offset, + sizeof (extra_word), &extra_word, + DMU_READ_PREFETCH)); + + ASSERT3U(offset, <=, space_map_length(sm)); + + entry_run = SM2_RUN_DECODE(word) << mapshift; + entry_vdev = SM2_VDEV_DECODE(word); + entry_type = (SM2_TYPE_DECODE(extra_word) == SM_ALLOC) ? + 'A' : 'F'; + entry_off = (SM2_OFFSET_DECODE(extra_word) << + mapshift) + sm->sm_start; + words = 2; } + + (void) printf("\t [%6llu] %c range:" + " %010llx-%010llx size: %06llx vdev: %06llu words: %u\n", + (u_longlong_t)(offset / sizeof (word)), + entry_type, (u_longlong_t)entry_off, + (u_longlong_t)(entry_off + entry_run), + (u_longlong_t)entry_run, + (u_longlong_t)entry_vdev, words); + + if (entry_type == 'A') + alloc += entry_run; + else + alloc -= entry_run; } - if (alloc != space_map_allocated(sm)) { - (void) printf("space_map_object alloc (%llu) INCONSISTENT " - "with space map summary (%llu)\n", - (u_longlong_t)space_map_allocated(sm), (u_longlong_t)alloc); + if ((uint64_t)alloc != space_map_allocated(sm)) { + (void) printf("space_map_object alloc (%lld) INCONSISTENT " + "with space map summary (%lld)\n", + (longlong_t)space_map_allocated(sm), (longlong_t)alloc); } } @@ -1153,7 +1184,7 @@ dump_ddt(ddt_t *ddt, enum ddt_type type, enum ddt_clas while ((error = ddt_object_walk(ddt, type, class, &walk, &dde)) == 0) dump_dde(ddt, &dde, walk); - ASSERT(error == ENOENT); + ASSERT3U(error, ==, ENOENT); (void) printf("\n"); } @@ -3068,15 +3099,14 @@ typedef struct checkpoint_sm_exclude_entry_arg { } checkpoint_sm_exclude_entry_arg_t; static int -checkpoint_sm_exclude_entry_cb(maptype_t type, uint64_t offset, uint64_t size, - void *arg) +checkpoint_sm_exclude_entry_cb(space_map_entry_t *sme, void *arg) { checkpoint_sm_exclude_entry_arg_t *cseea = arg; vdev_t *vd = cseea->cseea_vd; - metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift]; - uint64_t end = offset + size; + metaslab_t *ms = vd->vdev_ms[sme->sme_offset >> vd->vdev_ms_shift]; + uint64_t end = sme->sme_offset + sme->sme_run; - ASSERT(type == SM_FREE); + ASSERT(sme->sme_type == SM_FREE); /* * Since the vdev_checkpoint_sm exists in the vdev level @@ -3094,7 +3124,7 @@ checkpoint_sm_exclude_entry_cb(maptype_t type, uint64_ * metaslab boundaries. So if needed we could add code * that handles metaslab-crossing segments in the future. */ - VERIFY3U(offset, >=, ms->ms_start); + VERIFY3U(sme->sme_offset, >=, ms->ms_start); VERIFY3U(end, <=, ms->ms_start + ms->ms_size); /* @@ -3102,10 +3132,10 @@ checkpoint_sm_exclude_entry_cb(maptype_t type, uint64_ * also verify that the entry is there to begin with. */ mutex_enter(&ms->ms_lock); - range_tree_remove(ms->ms_allocatable, offset, size); + range_tree_remove(ms->ms_allocatable, sme->sme_offset, sme->sme_run); mutex_exit(&ms->ms_lock); - cseea->cseea_checkpoint_size += size; + cseea->cseea_checkpoint_size += sme->sme_run; return (0); } @@ -4080,15 +4110,14 @@ typedef struct verify_checkpoint_sm_entry_cb_arg { #define ENTRIES_PER_PROGRESS_UPDATE 10000 static int -verify_checkpoint_sm_entry_cb(maptype_t type, uint64_t offset, uint64_t size, - void *arg) +verify_checkpoint_sm_entry_cb(space_map_entry_t *sme, void *arg) { verify_checkpoint_sm_entry_cb_arg_t *vcsec = arg; vdev_t *vd = vcsec->vcsec_vd; - metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift]; - uint64_t end = offset + size; + metaslab_t *ms = vd->vdev_ms[sme->sme_offset >> vd->vdev_ms_shift]; + uint64_t end = sme->sme_offset + sme->sme_run; - ASSERT(type == SM_FREE); + ASSERT(sme->sme_type == SM_FREE); if ((vcsec->vcsec_entryid % ENTRIES_PER_PROGRESS_UPDATE) == 0) { (void) fprintf(stderr, @@ -4102,7 +4131,7 @@ verify_checkpoint_sm_entry_cb(maptype_t type, uint64_t /* * See comment in checkpoint_sm_exclude_entry_cb() */ - VERIFY3U(offset, >=, ms->ms_start); + VERIFY3U(sme->sme_offset, >=, ms->ms_start); VERIFY3U(end, <=, ms->ms_start + ms->ms_size); /* @@ -4111,7 +4140,7 @@ verify_checkpoint_sm_entry_cb(maptype_t type, uint64_t * their respective ms_allocateable trees should not contain them. */ mutex_enter(&ms->ms_lock); - range_tree_verify(ms->ms_allocatable, offset, size); + range_tree_verify(ms->ms_allocatable, sme->sme_offset, sme->sme_run); mutex_exit(&ms->ms_lock); return (0); @@ -4357,7 +4386,7 @@ verify_checkpoint(spa_t *spa) DMU_POOL_ZPOOL_CHECKPOINT, sizeof (uint64_t), sizeof (uberblock_t) / sizeof (uint64_t), &checkpoint); - if (error == ENOENT) { + if (error == ENOENT && !dump_opt['L']) { /* * If the feature is active but the uberblock is missing * then we must be in the middle of discarding the @@ -4380,7 +4409,7 @@ verify_checkpoint(spa_t *spa) error = 3; } - if (error == 0) + if (error == 0 && !dump_opt['L']) verify_checkpoint_blocks(spa); return (error); @@ -4485,7 +4514,7 @@ dump_zpool(spa_t *spa) if (dump_opt['h']) dump_history(spa); - if (rc == 0 && !dump_opt['L']) + if (rc == 0) rc = verify_checkpoint(spa); if (rc != 0) { Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Mon Jul 30 22:39:30 2018 (r336945) +++ vendor/illumos/dist/cmd/ztest/ztest.c Mon Jul 30 22:56:24 2018 (r336946) @@ -193,6 +193,7 @@ extern uint64_t zfs_deadman_synctime_ms; extern int metaslab_preload_limit; extern boolean_t zfs_compressed_arc_enabled; extern boolean_t zfs_abd_scatter_enabled; +extern boolean_t zfs_force_some_double_word_sm_entries; static ztest_shared_opts_t *ztest_shared_opts; static ztest_shared_opts_t ztest_opts; @@ -6390,6 +6391,12 @@ main(int argc, char **argv) dprintf_setup(&argc, argv); zfs_deadman_synctime_ms = 300000; + /* + * As two-word space map entries may not come up often (especially + * if pool and vdev sizes are small) we want to force at least some + * of them so the feature get tested. + */ + zfs_force_some_double_word_sm_entries = B_TRUE; ztest_fd_rand = open("/dev/urandom", O_RDONLY); ASSERT3S(ztest_fd_rand, >=, 0); Modified: vendor/illumos/dist/man/man5/zpool-features.5 ============================================================================== --- vendor/illumos/dist/man/man5/zpool-features.5 Mon Jul 30 22:39:30 2018 (r336945) +++ vendor/illumos/dist/man/man5/zpool-features.5 Mon Jul 30 22:56:24 2018 (r336946) @@ -423,7 +423,6 @@ This feature becomes \fBactive\fR as soon as it is ena never return to being \fBenabled\fR. .RE - .sp .ne 2 .na @@ -488,6 +487,34 @@ This feature becomes \fBactive\fR when the "zpool chec is used to checkpoint the pool. The feature will only return back to being \fBenabled\fR when the pool is rewound or the checkpoint has been discarded. + +.RE +.sp +.ne 2 +.na +\fB\fBspacemap_v2\fR\fR +.ad +.RS 4n +.TS +l l . +GUID com.delphix:spacemap_v2 +READ\-ONLY COMPATIBLE yes +DEPENDENCIES none +.TE + +This feature enables the use of the new space map encoding which +consists of two words (instead of one) whenever it is advantageous. +The new encoding allows space maps to represent large regions of +space more efficiently on-disk while also increasing their maximum +addressable offset. + +This feature becomes \fBactive\fR once it is \fBenabled\fR, and never +returns back to being \fBenabled\fR. + +.RE +.sp +.ne 2 +.na \fB\fBlarge_blocks\fR\fR .ad .RS 4n From owner-svn-src-vendor@freebsd.org Mon Jul 30 22:56:27 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B93801063655; Mon, 30 Jul 2018 22:56:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A66C89728; Mon, 30 Jul 2018 22:56:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 470C02F0E; Mon, 30 Jul 2018 22:56:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6UMuQId043708; Mon, 30 Jul 2018 22:56:26 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6UMuPvv043700; Mon, 30 Jul 2018 22:56:25 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807302256.w6UMuPvv043700@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 30 Jul 2018 22:56:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336946 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/... X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist... X-SVN-Commit-Revision: 336946 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 22:56:27 -0000 Author: mav Date: Mon Jul 30 22:56:24 2018 New Revision: 336946 URL: https://svnweb.freebsd.org/changeset/base/336946 Log: 9238 ZFS Spacemap Encoding V2 The current space map encoding has the following disadvantages: [1] Assuming 512 sector size each entry can represent at most 16MB for a segment. This makes the encoding very inefficient for large regions of space. [2] As vdev-wide space maps have started to be used by new features (i.e. device removal, zpool checkpoint) we've started imposing limits in the vdevs that can be used with them based on the maximum addressable offset (currently 64PB for a top-level vdev). The new remains backwards compatible with the old one. The introduced two-word entry format, besides extending the limits imposed by the single-entry layout, also includes a vdev field and some extra padding after its prefix. The extra padding after the prefix should is reserved for future usage (e.g. new prefixes for future encodings or new fields for flags). The new vdev field not only makes the space maps more self-descriptive, but also opens the doors for pool-wide space maps. One final important note is that the number of bits used for vdevs is reduced to 24 bits for blkptrs. That was decided as we don't know of any setups that use more than 16M vdevs for the time being and we wanted to fit the vdev field in the space map. In addition that gives us some extra bits in dva_t. illumos/illumos-gate@17f11284b49b98353b5119463254074fd9bc0a28 Reviewed by: Matt Ahrens Reviewed by: George Wilson Approved by: Gordon Ross Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c vendor-sys/illumos/dist/common/zfs/zfeature_common.h vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/space_map.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect_mapping.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/man/man5/zpool-features.5 Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.c Mon Jul 30 22:39:30 2018 (r336945) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.c Mon Jul 30 22:56:24 2018 (r336946) @@ -230,6 +230,12 @@ zpool_feature_init(void) "Pool state can be checkpointed, allowing rewind later.", ZFEATURE_FLAG_READONLY_COMPAT, NULL); + zfeature_register(SPA_FEATURE_SPACEMAP_V2, + "com.delphix:spacemap_v2", "spacemap_v2", + "Space maps representing large segments are more efficient.", + ZFEATURE_FLAG_READONLY_COMPAT | ZFEATURE_FLAG_ACTIVATE_ON_ENABLE, + NULL); + static const spa_feature_t large_blocks_deps[] = { SPA_FEATURE_EXTENSIBLE_DATASET, SPA_FEATURE_NONE Modified: vendor-sys/illumos/dist/common/zfs/zfeature_common.h ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfeature_common.h Mon Jul 30 22:39:30 2018 (r336945) +++ vendor-sys/illumos/dist/common/zfs/zfeature_common.h Mon Jul 30 22:56:24 2018 (r336946) @@ -59,6 +59,7 @@ typedef enum spa_feature { SPA_FEATURE_DEVICE_REMOVAL, SPA_FEATURE_OBSOLETE_COUNTS, SPA_FEATURE_POOL_CHECKPOINT, + SPA_FEATURE_SPACEMAP_V2, SPA_FEATURES } spa_feature_t; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Mon Jul 30 22:39:30 2018 (r336945) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Mon Jul 30 22:56:24 2018 (r336946) @@ -2091,17 +2091,6 @@ metaslab_group_preload(metaslab_group_t *mg) * * 3. The on-disk size of the space map should actually decrease. * - * Checking the first condition is tricky since we don't want to walk - * the entire AVL tree calculating the estimated on-disk size. Instead we - * use the size-ordered range tree in the metaslab and calculate the - * size required to write out the largest segment in our free tree. If the - * size required to represent that segment on disk is larger than the space - * map object then we avoid condensing this map. - * - * To determine the second criterion we use a best-case estimate and assume - * each segment can be represented on-disk as a single 64-bit entry. We refer - * to this best-case estimate as the space map's minimal form. - * * Unfortunately, we cannot compute the on-disk size of the space map in this * context because we cannot accurately compute the effects of compression, etc. * Instead, we apply the heuristic described in the block comment for @@ -2112,9 +2101,6 @@ static boolean_t metaslab_should_condense(metaslab_t *msp) { space_map_t *sm = msp->ms_sm; - range_seg_t *rs; - uint64_t size, entries, segsz, object_size, optimal_size, record_size; - dmu_object_info_t doi; vdev_t *vd = msp->ms_group->mg_vd; uint64_t vdev_blocksize = 1 << vd->vdev_ashift; uint64_t current_txg = spa_syncing_txg(vd->vdev_spa); @@ -2140,34 +2126,22 @@ metaslab_should_condense(metaslab_t *msp) msp->ms_condense_checked_txg = current_txg; /* - * Use the ms_allocatable_by_size range tree, which is ordered by - * size, to obtain the largest segment in the free tree. We always - * condense metaslabs that are empty and metaslabs for which a - * condense request has been made. + * We always condense metaslabs that are empty and metaslabs for + * which a condense request has been made. */ - rs = avl_last(&msp->ms_allocatable_by_size); - if (rs == NULL || msp->ms_condense_wanted) + if (avl_is_empty(&msp->ms_allocatable_by_size) || + msp->ms_condense_wanted) return (B_TRUE); - /* - * Calculate the number of 64-bit entries this segment would - * require when written to disk. If this single segment would be - * larger on-disk than the entire current on-disk structure, then - * clearly condensing will increase the on-disk structure size. - */ - size = (rs->rs_end - rs->rs_start) >> sm->sm_shift; - entries = size / (MIN(size, SM_RUN_MAX)); - segsz = entries * sizeof (uint64_t); + uint64_t object_size = space_map_length(msp->ms_sm); + uint64_t optimal_size = space_map_estimate_optimal_size(sm, + msp->ms_allocatable, SM_NO_VDEVID); - optimal_size = - sizeof (uint64_t) * avl_numnodes(&msp->ms_allocatable->rt_root); - object_size = space_map_length(msp->ms_sm); - + dmu_object_info_t doi; dmu_object_info_from_db(sm->sm_dbuf, &doi); - record_size = MAX(doi.doi_data_block_size, vdev_blocksize); + uint64_t record_size = MAX(doi.doi_data_block_size, vdev_blocksize); - return (segsz <= object_size && - object_size >= (optimal_size * zfs_condense_pct / 100) && + return (object_size >= (optimal_size * zfs_condense_pct / 100) && object_size > zfs_metaslab_condense_block_threshold * record_size); } @@ -2242,11 +2216,11 @@ metaslab_condense(metaslab_t *msp, uint64_t txg, dmu_t * optimal, this is typically close to optimal, and much cheaper to * compute. */ - space_map_write(sm, condense_tree, SM_ALLOC, tx); + space_map_write(sm, condense_tree, SM_ALLOC, SM_NO_VDEVID, tx); range_tree_vacate(condense_tree, NULL, NULL); range_tree_destroy(condense_tree); - space_map_write(sm, msp->ms_allocatable, SM_FREE, tx); + space_map_write(sm, msp->ms_allocatable, SM_FREE, SM_NO_VDEVID, tx); mutex_enter(&msp->ms_lock); msp->ms_condensing = B_FALSE; } @@ -2358,8 +2332,10 @@ metaslab_sync(metaslab_t *msp, uint64_t txg) metaslab_condense(msp, txg, tx); } else { mutex_exit(&msp->ms_lock); - space_map_write(msp->ms_sm, alloctree, SM_ALLOC, tx); - space_map_write(msp->ms_sm, msp->ms_freeing, SM_FREE, tx); + space_map_write(msp->ms_sm, alloctree, SM_ALLOC, + SM_NO_VDEVID, tx); + space_map_write(msp->ms_sm, msp->ms_freeing, SM_FREE, + SM_NO_VDEVID, tx); mutex_enter(&msp->ms_lock); } @@ -2374,7 +2350,7 @@ metaslab_sync(metaslab_t *msp, uint64_t txg) */ mutex_exit(&msp->ms_lock); space_map_write(vd->vdev_checkpoint_sm, - msp->ms_checkpointing, SM_FREE, tx); + msp->ms_checkpointing, SM_FREE, SM_NO_VDEVID, tx); mutex_enter(&msp->ms_lock); space_map_update(vd->vdev_checkpoint_sm); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c Mon Jul 30 22:39:30 2018 (r336945) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c Mon Jul 30 22:56:24 2018 (r336946) @@ -179,7 +179,7 @@ range_tree_add(void *arg, uint64_t start, uint64_t siz } /* Make sure we don't overlap with either of our neighbors */ - VERIFY(rs == NULL); + VERIFY3P(rs, ==, NULL); rs_before = avl_nearest(&rt->rt_root, where, AVL_BEFORE); rs_after = avl_nearest(&rt->rt_root, where, AVL_AFTER); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c Mon Jul 30 22:39:30 2018 (r336945) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_checkpoint.c Mon Jul 30 22:56:24 2018 (r336946) @@ -203,13 +203,12 @@ typedef struct spa_checkpoint_discard_sync_callback_ar } spa_checkpoint_discard_sync_callback_arg_t; static int -spa_checkpoint_discard_sync_callback(maptype_t type, uint64_t offset, - uint64_t size, void *arg) +spa_checkpoint_discard_sync_callback(space_map_entry_t *sme, void *arg) { spa_checkpoint_discard_sync_callback_arg_t *sdc = arg; vdev_t *vd = sdc->sdc_vd; - metaslab_t *ms = vd->vdev_ms[offset >> vd->vdev_ms_shift]; - uint64_t end = offset + size; + metaslab_t *ms = vd->vdev_ms[sme->sme_offset >> vd->vdev_ms_shift]; + uint64_t end = sme->sme_offset + sme->sme_run; if (sdc->sdc_entry_limit == 0) return (EINTR); @@ -224,8 +223,8 @@ spa_checkpoint_discard_sync_callback(maptype_t type, u * metaslab boundaries. So if needed we could add code * that handles metaslab-crossing segments in the future. */ - VERIFY3U(type, ==, SM_FREE); - VERIFY3U(offset, >=, ms->ms_start); + VERIFY3U(sme->sme_type, ==, SM_FREE); + VERIFY3U(sme->sme_offset, >=, ms->ms_start); VERIFY3U(end, <=, ms->ms_start + ms->ms_size); /* @@ -237,14 +236,15 @@ spa_checkpoint_discard_sync_callback(maptype_t type, u mutex_enter(&ms->ms_lock); if (range_tree_is_empty(ms->ms_freeing)) vdev_dirty(vd, VDD_METASLAB, ms, sdc->sdc_txg); - range_tree_add(ms->ms_freeing, offset, size); + range_tree_add(ms->ms_freeing, sme->sme_offset, sme->sme_run); mutex_exit(&ms->ms_lock); - ASSERT3U(vd->vdev_spa->spa_checkpoint_info.sci_dspace, >=, size); - ASSERT3U(vd->vdev_stat.vs_checkpoint_space, >=, size); + ASSERT3U(vd->vdev_spa->spa_checkpoint_info.sci_dspace, >=, + sme->sme_run); + ASSERT3U(vd->vdev_stat.vs_checkpoint_space, >=, sme->sme_run); - vd->vdev_spa->spa_checkpoint_info.sci_dspace -= size; - vd->vdev_stat.vs_checkpoint_space -= size; + vd->vdev_spa->spa_checkpoint_info.sci_dspace -= sme->sme_run; + vd->vdev_stat.vs_checkpoint_space -= sme->sme_run; sdc->sdc_entry_limit--; return (0); @@ -289,13 +289,14 @@ spa_checkpoint_discard_thread_sync(void *arg, dmu_tx_t * Thus, we set the maximum entries that the space map callback * will be applied to be half the entries that could fit in the * imposed memory limit. + * + * Note that since this is a conservative estimate we also + * assume the worst case scenario in our computation where each + * entry is two-word. */ uint64_t max_entry_limit = - (zfs_spa_discard_memory_limit / sizeof (uint64_t)) >> 1; + (zfs_spa_discard_memory_limit / (2 * sizeof (uint64_t))) >> 1; - uint64_t entries_in_sm = - space_map_length(vd->vdev_checkpoint_sm) / sizeof (uint64_t); - /* * Iterate from the end of the space map towards the beginning, * placing its entries on ms_freeing and removing them from the @@ -318,14 +319,15 @@ spa_checkpoint_discard_thread_sync(void *arg, dmu_tx_t spa_checkpoint_discard_sync_callback_arg_t sdc; sdc.sdc_vd = vd; sdc.sdc_txg = tx->tx_txg; - sdc.sdc_entry_limit = MIN(entries_in_sm, max_entry_limit); + sdc.sdc_entry_limit = max_entry_limit; - uint64_t entries_before = entries_in_sm; + uint64_t words_before = + space_map_length(vd->vdev_checkpoint_sm) / sizeof (uint64_t); error = space_map_incremental_destroy(vd->vdev_checkpoint_sm, spa_checkpoint_discard_sync_callback, &sdc, tx); - uint64_t entries_after = + uint64_t words_after = space_map_length(vd->vdev_checkpoint_sm) / sizeof (uint64_t); #ifdef DEBUG @@ -333,9 +335,9 @@ spa_checkpoint_discard_thread_sync(void *arg, dmu_tx_t #endif zfs_dbgmsg("discarding checkpoint: txg %llu, vdev id %d, " - "deleted %llu entries - %llu entries are left", - tx->tx_txg, vd->vdev_id, (entries_before - entries_after), - entries_after); + "deleted %llu words - %llu words are left", + tx->tx_txg, vd->vdev_id, (words_before - words_after), + words_after); if (error != EINTR) { if (error != 0) { @@ -344,15 +346,15 @@ spa_checkpoint_discard_thread_sync(void *arg, dmu_tx_t "space map of vdev %llu\n", error, vd->vdev_id); } - ASSERT0(entries_after); + ASSERT0(words_after); ASSERT0(vd->vdev_checkpoint_sm->sm_alloc); - ASSERT0(vd->vdev_checkpoint_sm->sm_length); + ASSERT0(space_map_length(vd->vdev_checkpoint_sm)); space_map_free(vd->vdev_checkpoint_sm, tx); space_map_close(vd->vdev_checkpoint_sm); vd->vdev_checkpoint_sm = NULL; - VERIFY0(zap_remove(vd->vdev_spa->spa_meta_objset, + VERIFY0(zap_remove(spa_meta_objset(vd->vdev_spa), vd->vdev_top_zap, VDEV_TOP_ZAP_POOL_CHECKPOINT_SM, tx)); } } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c Mon Jul 30 22:39:30 2018 (r336945) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c Mon Jul 30 22:56:24 2018 (r336946) @@ -41,68 +41,194 @@ * Note on space map block size: * * The data for a given space map can be kept on blocks of any size. - * Larger blocks entail fewer i/o operations, but they also cause the - * DMU to keep more data in-core, and also to waste more i/o bandwidth + * Larger blocks entail fewer I/O operations, but they also cause the + * DMU to keep more data in-core, and also to waste more I/O bandwidth * when only a few blocks have changed since the last transaction group. */ /* + * Enabled whenever we want to stress test the use of double-word + * space map entries. + */ +boolean_t zfs_force_some_double_word_sm_entries = B_FALSE; + +boolean_t +sm_entry_is_debug(uint64_t e) +{ + return (SM_PREFIX_DECODE(e) == SM_DEBUG_PREFIX); +} + +boolean_t +sm_entry_is_single_word(uint64_t e) +{ + uint8_t prefix = SM_PREFIX_DECODE(e); + return (prefix != SM_DEBUG_PREFIX && prefix != SM2_PREFIX); +} + +boolean_t +sm_entry_is_double_word(uint64_t e) +{ + return (SM_PREFIX_DECODE(e) == SM2_PREFIX); +} + +/* * Iterate through the space map, invoking the callback on each (non-debug) * space map entry. */ int space_map_iterate(space_map_t *sm, sm_cb_t callback, void *arg) { - uint64_t *entry, *entry_map, *entry_map_end; - uint64_t bufsize, size, offset, end; + uint64_t sm_len = space_map_length(sm); + ASSERT3U(sm->sm_blksz, !=, 0); + + dmu_prefetch(sm->sm_os, space_map_object(sm), 0, 0, sm_len, + ZIO_PRIORITY_SYNC_READ); + + uint64_t blksz = sm->sm_blksz; int error = 0; + for (uint64_t block_base = 0; block_base < sm_len && error == 0; + block_base += blksz) { + dmu_buf_t *db; + error = dmu_buf_hold(sm->sm_os, space_map_object(sm), + block_base, FTAG, &db, DMU_READ_PREFETCH); + if (error != 0) + return (error); - end = space_map_length(sm); + uint64_t *block_start = db->db_data; + uint64_t block_length = MIN(sm_len - block_base, blksz); + uint64_t *block_end = block_start + + (block_length / sizeof (uint64_t)); - bufsize = MAX(sm->sm_blksz, SPA_MINBLOCKSIZE); - entry_map = zio_buf_alloc(bufsize); + VERIFY0(P2PHASE(block_length, sizeof (uint64_t))); + VERIFY3U(block_length, !=, 0); + ASSERT3U(blksz, ==, db->db_size); - if (end > bufsize) { - dmu_prefetch(sm->sm_os, space_map_object(sm), 0, bufsize, - end - bufsize, ZIO_PRIORITY_SYNC_READ); + for (uint64_t *block_cursor = block_start; + block_cursor < block_end && error == 0; block_cursor++) { + uint64_t e = *block_cursor; + + if (sm_entry_is_debug(e)) /* Skip debug entries */ + continue; + + uint64_t raw_offset, raw_run, vdev_id; + maptype_t type; + if (sm_entry_is_single_word(e)) { + type = SM_TYPE_DECODE(e); + vdev_id = SM_NO_VDEVID; + raw_offset = SM_OFFSET_DECODE(e); + raw_run = SM_RUN_DECODE(e); + } else { + /* it is a two-word entry */ + ASSERT(sm_entry_is_double_word(e)); + raw_run = SM2_RUN_DECODE(e); + vdev_id = SM2_VDEV_DECODE(e); + + /* move on to the second word */ + block_cursor++; + e = *block_cursor; + VERIFY3P(block_cursor, <=, block_end); + + type = SM2_TYPE_DECODE(e); + raw_offset = SM2_OFFSET_DECODE(e); + } + + uint64_t entry_offset = (raw_offset << sm->sm_shift) + + sm->sm_start; + uint64_t entry_run = raw_run << sm->sm_shift; + + VERIFY0(P2PHASE(entry_offset, 1ULL << sm->sm_shift)); + VERIFY0(P2PHASE(entry_run, 1ULL << sm->sm_shift)); + ASSERT3U(entry_offset, >=, sm->sm_start); + ASSERT3U(entry_offset, <, sm->sm_start + sm->sm_size); + ASSERT3U(entry_run, <=, sm->sm_size); + ASSERT3U(entry_offset + entry_run, <=, + sm->sm_start + sm->sm_size); + + space_map_entry_t sme = { + .sme_type = type, + .sme_vdev = vdev_id, + .sme_offset = entry_offset, + .sme_run = entry_run + }; + error = callback(&sme, arg); + } + dmu_buf_rele(db, FTAG); } + return (error); +} - for (offset = 0; offset < end && error == 0; offset += bufsize) { - size = MIN(end - offset, bufsize); - VERIFY(P2PHASE(size, sizeof (uint64_t)) == 0); - VERIFY(size != 0); - ASSERT3U(sm->sm_blksz, !=, 0); +/* + * Reads the entries from the last block of the space map into + * buf in reverse order. Populates nwords with number of words + * in the last block. + * + * Refer to block comment within space_map_incremental_destroy() + * to understand why this function is needed. + */ +static int +space_map_reversed_last_block_entries(space_map_t *sm, uint64_t *buf, + uint64_t bufsz, uint64_t *nwords) +{ + int error = 0; + dmu_buf_t *db; - dprintf("object=%llu offset=%llx size=%llx\n", - space_map_object(sm), offset, size); + /* + * Find the offset of the last word in the space map and use + * that to read the last block of the space map with + * dmu_buf_hold(). + */ + uint64_t last_word_offset = + sm->sm_phys->smp_objsize - sizeof (uint64_t); + error = dmu_buf_hold(sm->sm_os, space_map_object(sm), last_word_offset, + FTAG, &db, DMU_READ_NO_PREFETCH); + if (error != 0) + return (error); - error = dmu_read(sm->sm_os, space_map_object(sm), offset, size, - entry_map, DMU_READ_PREFETCH); - if (error != 0) - break; + ASSERT3U(sm->sm_object, ==, db->db_object); + ASSERT3U(sm->sm_blksz, ==, db->db_size); + ASSERT3U(bufsz, >=, db->db_size); + ASSERT(nwords != NULL); - entry_map_end = entry_map + (size / sizeof (uint64_t)); - for (entry = entry_map; entry < entry_map_end && error == 0; - entry++) { - uint64_t e = *entry; - uint64_t offset, size; + uint64_t *words = db->db_data; + *nwords = + (sm->sm_phys->smp_objsize - db->db_offset) / sizeof (uint64_t); - if (SM_DEBUG_DECODE(e)) /* Skip debug entries */ - continue; + ASSERT3U(*nwords, <=, bufsz / sizeof (uint64_t)); - offset = (SM_OFFSET_DECODE(e) << sm->sm_shift) + - sm->sm_start; - size = SM_RUN_DECODE(e) << sm->sm_shift; + uint64_t n = *nwords; + uint64_t j = n - 1; + for (uint64_t i = 0; i < n; i++) { + uint64_t entry = words[i]; + if (sm_entry_is_double_word(entry)) { + /* + * Since we are populating the buffer backwards + * we have to be extra careful and add the two + * words of the double-word entry in the right + * order. + */ + ASSERT3U(j, >, 0); + buf[j - 1] = entry; - VERIFY0(P2PHASE(offset, 1ULL << sm->sm_shift)); - VERIFY0(P2PHASE(size, 1ULL << sm->sm_shift)); - VERIFY3U(offset, >=, sm->sm_start); - VERIFY3U(offset + size, <=, sm->sm_start + sm->sm_size); - error = callback(SM_TYPE_DECODE(e), offset, size, arg); + i++; + ASSERT3U(i, <, n); + entry = words[i]; + buf[j] = entry; + j -= 2; + } else { + ASSERT(sm_entry_is_debug(entry) || + sm_entry_is_single_word(entry)); + buf[j] = entry; + j--; } } - zio_buf_free(entry_map, bufsize); + /* + * Assert that we wrote backwards all the + * way to the beginning of the buffer. + */ + ASSERT3S(j, ==, -1); + + dmu_buf_rele(db, FTAG); return (error); } @@ -116,124 +242,122 @@ int space_map_incremental_destroy(space_map_t *sm, sm_cb_t callback, void *arg, dmu_tx_t *tx) { - uint64_t bufsize, len; - uint64_t *entry_map; - int error = 0; + uint64_t bufsz = MAX(sm->sm_blksz, SPA_MINBLOCKSIZE); + uint64_t *buf = zio_buf_alloc(bufsz); - len = space_map_length(sm); - bufsize = MAX(sm->sm_blksz, SPA_MINBLOCKSIZE); - entry_map = zio_buf_alloc(bufsize); - dmu_buf_will_dirty(sm->sm_dbuf, tx); /* - * Since we can't move the starting offset of the space map - * (e.g there are reference on-disk pointing to it), we destroy - * its entries incrementally starting from the end. + * Ideally we would want to iterate from the beginning of the + * space map to the end in incremental steps. The issue with this + * approach is that we don't have any field on-disk that points + * us where to start between each step. We could try zeroing out + * entries that we've destroyed, but this doesn't work either as + * an entry that is 0 is a valid one (ALLOC for range [0x0:0x200]). * - * The logic that follows is basically the same as the one used - * in space_map_iterate() but it traverses the space map - * backwards: + * As a result, we destroy its entries incrementally starting from + * the end after applying the callback to each of them. * - * 1] We figure out the size of the buffer that we want to use - * to read the on-disk space map entries. - * 2] We figure out the offset at the end of the space map where - * we will start reading entries into our buffer. - * 3] We read the on-disk entries into the buffer. - * 4] We iterate over the entries from end to beginning calling - * the callback function on each one. As we move from entry - * to entry we decrease the size of the space map, deleting - * effectively each entry. - * 5] If there are no more entries in the space map or the - * callback returns a value other than 0, we stop iterating - * over the space map. If there are entries remaining and - * the callback returned zero we go back to step [1]. + * The problem with this approach is that we cannot literally + * iterate through the words in the space map backwards as we + * can't distinguish two-word space map entries from their second + * word. Thus we do the following: + * + * 1] We get all the entries from the last block of the space map + * and put them into a buffer in reverse order. This way the + * last entry comes first in the buffer, the second to last is + * second, etc. + * 2] We iterate through the entries in the buffer and we apply + * the callback to each one. As we move from entry to entry we + * we decrease the size of the space map, deleting effectively + * each entry. + * 3] If there are no more entries in the space map or the callback + * returns a value other than 0, we stop iterating over the + * space map. If there are entries remaining and the callback + * returned 0, we go back to step [1]. */ - uint64_t offset = 0, size = 0; - while (len > 0 && error == 0) { - size = MIN(bufsize, len); - - VERIFY(P2PHASE(size, sizeof (uint64_t)) == 0); - VERIFY3U(size, >, 0); - ASSERT3U(sm->sm_blksz, !=, 0); - - offset = len - size; - - IMPLY(bufsize > len, offset == 0); - IMPLY(bufsize == len, offset == 0); - IMPLY(bufsize < len, offset > 0); - - - EQUIV(size == len, offset == 0); - IMPLY(size < len, bufsize < len); - - dprintf("object=%llu offset=%llx size=%llx\n", - space_map_object(sm), offset, size); - - error = dmu_read(sm->sm_os, space_map_object(sm), - offset, size, entry_map, DMU_READ_PREFETCH); + int error = 0; + while (space_map_length(sm) > 0 && error == 0) { + uint64_t nwords = 0; + error = space_map_reversed_last_block_entries(sm, buf, bufsz, + &nwords); if (error != 0) break; - uint64_t num_entries = size / sizeof (uint64_t); + ASSERT3U(nwords, <=, bufsz / sizeof (uint64_t)); - ASSERT3U(num_entries, >, 0); + for (uint64_t i = 0; i < nwords; i++) { + uint64_t e = buf[i]; - while (num_entries > 0) { - uint64_t e, entry_offset, entry_size; + if (sm_entry_is_debug(e)) { + sm->sm_phys->smp_objsize -= sizeof (uint64_t); + space_map_update(sm); + continue; + } + + int words = 1; + uint64_t raw_offset, raw_run, vdev_id; maptype_t type; + if (sm_entry_is_single_word(e)) { + type = SM_TYPE_DECODE(e); + vdev_id = SM_NO_VDEVID; + raw_offset = SM_OFFSET_DECODE(e); + raw_run = SM_RUN_DECODE(e); + } else { + ASSERT(sm_entry_is_double_word(e)); + words = 2; - e = entry_map[num_entries - 1]; + raw_run = SM2_RUN_DECODE(e); + vdev_id = SM2_VDEV_DECODE(e); - ASSERT3U(num_entries, >, 0); - ASSERT0(error); + /* move to the second word */ + i++; + e = buf[i]; - if (SM_DEBUG_DECODE(e)) { - sm->sm_phys->smp_objsize -= sizeof (uint64_t); - space_map_update(sm); - len -= sizeof (uint64_t); - num_entries--; - continue; + ASSERT3P(i, <=, nwords); + + type = SM2_TYPE_DECODE(e); + raw_offset = SM2_OFFSET_DECODE(e); } - type = SM_TYPE_DECODE(e); - entry_offset = (SM_OFFSET_DECODE(e) << sm->sm_shift) + - sm->sm_start; - entry_size = SM_RUN_DECODE(e) << sm->sm_shift; + uint64_t entry_offset = + (raw_offset << sm->sm_shift) + sm->sm_start; + uint64_t entry_run = raw_run << sm->sm_shift; VERIFY0(P2PHASE(entry_offset, 1ULL << sm->sm_shift)); - VERIFY0(P2PHASE(entry_size, 1ULL << sm->sm_shift)); + VERIFY0(P2PHASE(entry_run, 1ULL << sm->sm_shift)); VERIFY3U(entry_offset, >=, sm->sm_start); - VERIFY3U(entry_offset + entry_size, <=, + VERIFY3U(entry_offset, <, sm->sm_start + sm->sm_size); + VERIFY3U(entry_run, <=, sm->sm_size); + VERIFY3U(entry_offset + entry_run, <=, sm->sm_start + sm->sm_size); - error = callback(type, entry_offset, entry_size, arg); + space_map_entry_t sme = { + .sme_type = type, + .sme_vdev = vdev_id, + .sme_offset = entry_offset, + .sme_run = entry_run + }; + error = callback(&sme, arg); if (error != 0) break; if (type == SM_ALLOC) - sm->sm_phys->smp_alloc -= entry_size; + sm->sm_phys->smp_alloc -= entry_run; else - sm->sm_phys->smp_alloc += entry_size; - - sm->sm_phys->smp_objsize -= sizeof (uint64_t); + sm->sm_phys->smp_alloc += entry_run; + sm->sm_phys->smp_objsize -= words * sizeof (uint64_t); space_map_update(sm); - len -= sizeof (uint64_t); - num_entries--; } - IMPLY(error == 0, num_entries == 0); - EQUIV(offset == 0 && error == 0, len == 0 && num_entries == 0); } - if (len == 0) { + if (space_map_length(sm) == 0) { ASSERT0(error); - ASSERT0(offset); - ASSERT0(sm->sm_length); ASSERT0(sm->sm_phys->smp_objsize); ASSERT0(sm->sm_alloc); } - zio_buf_free(entry_map, bufsize); + zio_buf_free(buf, bufsz); return (error); } @@ -244,16 +368,15 @@ typedef struct space_map_load_arg { } space_map_load_arg_t; static int -space_map_load_callback(maptype_t type, uint64_t offset, uint64_t size, - void *arg) +space_map_load_callback(space_map_entry_t *sme, void *arg) { space_map_load_arg_t *smla = arg; - if (type == smla->smla_type) { - VERIFY3U(range_tree_space(smla->smla_rt) + size, <=, + if (sme->sme_type == smla->smla_type) { + VERIFY3U(range_tree_space(smla->smla_rt) + sme->sme_run, <=, smla->smla_sm->sm_size); - range_tree_add(smla->smla_rt, offset, size); + range_tree_add(smla->smla_rt, sme->sme_offset, sme->sme_run); } else { - range_tree_remove(smla->smla_rt, offset, size); + range_tree_remove(smla->smla_rt, sme->sme_offset, sme->sme_run); } return (0); @@ -365,43 +488,239 @@ space_map_histogram_add(space_map_t *sm, range_tree_t } } -uint64_t -space_map_entries(space_map_t *sm, range_tree_t *rt) +static void +space_map_write_intro_debug(space_map_t *sm, maptype_t maptype, dmu_tx_t *tx) { - avl_tree_t *t = &rt->rt_root; - range_seg_t *rs; - uint64_t size, entries; + dmu_buf_will_dirty(sm->sm_dbuf, tx); + uint64_t dentry = SM_PREFIX_ENCODE(SM_DEBUG_PREFIX) | + SM_DEBUG_ACTION_ENCODE(maptype) | + SM_DEBUG_SYNCPASS_ENCODE(spa_sync_pass(tx->tx_pool->dp_spa)) | + SM_DEBUG_TXG_ENCODE(dmu_tx_get_txg(tx)); + + dmu_write(sm->sm_os, space_map_object(sm), sm->sm_phys->smp_objsize, + sizeof (dentry), &dentry, tx); + + sm->sm_phys->smp_objsize += sizeof (dentry); +} + +/* + * Writes one or more entries given a segment. + * + * Note: The function may release the dbuf from the pointer initially + * passed to it, and return a different dbuf. Also, the space map's + * dbuf must be dirty for the changes in sm_phys to take effect. + */ +static void +space_map_write_seg(space_map_t *sm, range_seg_t *rs, maptype_t maptype, + uint64_t vdev_id, uint8_t words, dmu_buf_t **dbp, void *tag, dmu_tx_t *tx) +{ + ASSERT3U(words, !=, 0); + ASSERT3U(words, <=, 2); + + /* ensure the vdev_id can be represented by the space map */ + ASSERT3U(vdev_id, <=, SM_NO_VDEVID); + /* - * All space_maps always have a debug entry so account for it here. + * if this is a single word entry, ensure that no vdev was + * specified. */ - entries = 1; + IMPLY(words == 1, vdev_id == SM_NO_VDEVID); + dmu_buf_t *db = *dbp; + ASSERT3U(db->db_size, ==, sm->sm_blksz); + + uint64_t *block_base = db->db_data; + uint64_t *block_end = block_base + (sm->sm_blksz / sizeof (uint64_t)); + uint64_t *block_cursor = block_base + + (sm->sm_phys->smp_objsize - db->db_offset) / sizeof (uint64_t); + + ASSERT3P(block_cursor, <=, block_end); + + uint64_t size = (rs->rs_end - rs->rs_start) >> sm->sm_shift; + uint64_t start = (rs->rs_start - sm->sm_start) >> sm->sm_shift; + uint64_t run_max = (words == 2) ? SM2_RUN_MAX : SM_RUN_MAX; + + ASSERT3U(rs->rs_start, >=, sm->sm_start); + ASSERT3U(rs->rs_start, <, sm->sm_start + sm->sm_size); + ASSERT3U(rs->rs_end - rs->rs_start, <=, sm->sm_size); + ASSERT3U(rs->rs_end, <=, sm->sm_start + sm->sm_size); + + while (size != 0) { + ASSERT3P(block_cursor, <=, block_end); + + /* + * If we are at the end of this block, flush it and start + * writing again from the beginning. + */ + if (block_cursor == block_end) { + dmu_buf_rele(db, tag); + + uint64_t next_word_offset = sm->sm_phys->smp_objsize; + VERIFY0(dmu_buf_hold(sm->sm_os, + space_map_object(sm), next_word_offset, + tag, &db, DMU_READ_PREFETCH)); + dmu_buf_will_dirty(db, tx); + + /* update caller's dbuf */ + *dbp = db; + + ASSERT3U(db->db_size, ==, sm->sm_blksz); + + block_base = db->db_data; + block_cursor = block_base; + block_end = block_base + + (db->db_size / sizeof (uint64_t)); + } + + /* + * If we are writing a two-word entry and we only have one + * word left on this block, just pad it with an empty debug + * entry and write the two-word entry in the next block. + */ + uint64_t *next_entry = block_cursor + 1; + if (next_entry == block_end && words > 1) { + ASSERT3U(words, ==, 2); + *block_cursor = SM_PREFIX_ENCODE(SM_DEBUG_PREFIX) | + SM_DEBUG_ACTION_ENCODE(0) | + SM_DEBUG_SYNCPASS_ENCODE(0) | + SM_DEBUG_TXG_ENCODE(0); + block_cursor++; + sm->sm_phys->smp_objsize += sizeof (uint64_t); + ASSERT3P(block_cursor, ==, block_end); + continue; + } + + uint64_t run_len = MIN(size, run_max); + switch (words) { + case 1: + *block_cursor = SM_OFFSET_ENCODE(start) | + SM_TYPE_ENCODE(maptype) | + SM_RUN_ENCODE(run_len); + block_cursor++; + break; + case 2: + /* write the first word of the entry */ + *block_cursor = SM_PREFIX_ENCODE(SM2_PREFIX) | + SM2_RUN_ENCODE(run_len) | + SM2_VDEV_ENCODE(vdev_id); + block_cursor++; + + /* move on to the second word of the entry */ + ASSERT3P(block_cursor, <, block_end); + *block_cursor = SM2_TYPE_ENCODE(maptype) | + SM2_OFFSET_ENCODE(start); + block_cursor++; + break; + default: + panic("%d-word space map entries are not supported", + words); + break; + } + sm->sm_phys->smp_objsize += words * sizeof (uint64_t); + + start += run_len; + size -= run_len; + } + ASSERT0(size); + +} + +/* + * Note: The space map's dbuf must be dirty for the changes in sm_phys to + * take effect. + */ +static void +space_map_write_impl(space_map_t *sm, range_tree_t *rt, maptype_t maptype, + uint64_t vdev_id, dmu_tx_t *tx) +{ + spa_t *spa = tx->tx_pool->dp_spa; + dmu_buf_t *db; + + space_map_write_intro_debug(sm, maptype, tx); + +#ifdef DEBUG /* - * Traverse the range tree and calculate the number of space map - * entries that would be required to write out the range tree. + * We do this right after we write the intro debug entry + * because the estimate does not take it into account. */ - for (rs = avl_first(t); rs != NULL; rs = AVL_NEXT(t, rs)) { - size = (rs->rs_end - rs->rs_start) >> sm->sm_shift; - entries += howmany(size, SM_RUN_MAX); + uint64_t initial_objsize = sm->sm_phys->smp_objsize; + uint64_t estimated_growth = + space_map_estimate_optimal_size(sm, rt, SM_NO_VDEVID); + uint64_t estimated_final_objsize = initial_objsize + estimated_growth; +#endif + + /* + * Find the offset right after the last word in the space map + * and use that to get a hold of the last block, so we can + * start appending to it. + */ + uint64_t next_word_offset = sm->sm_phys->smp_objsize; + VERIFY0(dmu_buf_hold(sm->sm_os, space_map_object(sm), + next_word_offset, FTAG, &db, DMU_READ_PREFETCH)); + ASSERT3U(db->db_size, ==, sm->sm_blksz); + + dmu_buf_will_dirty(db, tx); + + avl_tree_t *t = &rt->rt_root; + for (range_seg_t *rs = avl_first(t); rs != NULL; rs = AVL_NEXT(t, rs)) { + uint64_t offset = (rs->rs_start - sm->sm_start) >> sm->sm_shift; + uint64_t length = (rs->rs_end - rs->rs_start) >> sm->sm_shift; + uint8_t words = 1; + + /* + * We only write two-word entries when both of the following + * are true: + * + * [1] The feature is enabled. + * [2] The offset or run is too big for a single-word entry, + * or the vdev_id is set (meaning not equal to + * SM_NO_VDEVID). + * + * Note that for purposes of testing we've added the case that + * we write two-word entries occasionally when the feature is + * enabled and zfs_force_some_double_word_sm_entries has been + * set. + */ + if (spa_feature_is_active(spa, SPA_FEATURE_SPACEMAP_V2) && + (offset >= (1ULL << SM_OFFSET_BITS) || + length > SM_RUN_MAX || + vdev_id != SM_NO_VDEVID || + (zfs_force_some_double_word_sm_entries && + spa_get_random(100) == 0))) + words = 2; + + space_map_write_seg(sm, rs, maptype, vdev_id, words, + &db, FTAG, tx); } - return (entries); + + dmu_buf_rele(db, FTAG); + +#ifdef DEBUG + /* + * We expect our estimation to be based on the worst case + * scenario [see comment in space_map_estimate_optimal_size()]. + * Therefore we expect the actual objsize to be equal or less + * than whatever we estimated it to be. + */ + ASSERT3U(estimated_final_objsize, >=, sm->sm_phys->smp_objsize); +#endif } +/* + * Note: This function manipulates the state of the given space map but + * does not hold any locks implicitly. Thus the caller is responsible + * for synchronizing writes to the space map. + */ void space_map_write(space_map_t *sm, range_tree_t *rt, maptype_t maptype, - dmu_tx_t *tx) + uint64_t vdev_id, dmu_tx_t *tx) { objset_t *os = sm->sm_os; - spa_t *spa = dmu_objset_spa(os); - avl_tree_t *t = &rt->rt_root; - range_seg_t *rs; - uint64_t size, total, rt_space, nodes; - uint64_t *entry, *entry_map, *entry_map_end; - uint64_t expected_entries, actual_entries = 1; ASSERT(dsl_pool_sync_context(dmu_objset_pool(os))); VERIFY3U(space_map_object(sm), !=, 0); + dmu_buf_will_dirty(sm->sm_dbuf, tx); /* @@ -421,68 +740,17 @@ space_map_write(space_map_t *sm, range_tree_t *rt, map else sm->sm_phys->smp_alloc -= range_tree_space(rt); - expected_entries = space_map_entries(sm, rt); + uint64_t nodes = avl_numnodes(&rt->rt_root); + uint64_t rt_space = range_tree_space(rt); - entry_map = zio_buf_alloc(sm->sm_blksz); - entry_map_end = entry_map + (sm->sm_blksz / sizeof (uint64_t)); - entry = entry_map; + space_map_write_impl(sm, rt, maptype, vdev_id, tx); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Mon Jul 30 23:53:26 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DD6B106471A; Mon, 30 Jul 2018 23:53:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32A4D8B688; Mon, 30 Jul 2018 23:53:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1394C3CB1; Mon, 30 Jul 2018 23:53:26 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6UNrPVT074421; Mon, 30 Jul 2018 23:53:25 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6UNrPFQ074418; Mon, 30 Jul 2018 23:53:25 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807302353.w6UNrPFQ074418@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Mon, 30 Jul 2018 23:53:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336948 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 336948 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jul 2018 23:53:26 -0000 Author: mav Date: Mon Jul 30 23:53:25 2018 New Revision: 336948 URL: https://svnweb.freebsd.org/changeset/base/336948 Log: 9112 Improve allocation performance on high-end systems On high-end systems running async sequential write workloads, especially NUMA systems with flash or NVMe storage, one significant performance bottleneck is selecting a metaslab to do allocations from. This process can be parallelized, providing significant performance increases for these workloads. illumos/illumos-gate@f78cdc34af236a6199dd9e21376f4a46348c0d56 Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: Serapheim Dimitropoulos Reviewed by: Alexander Motin Approved by: Gordon Ross Author: Paul Dagnelie Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Mon Jul 30 23:47:38 2018 (r336947) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Mon Jul 30 23:53:25 2018 (r336948) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2018 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2014 Integros [integros.com] */ @@ -212,6 +212,8 @@ static uint64_t metaslab_weight(metaslab_t *); static void metaslab_set_fragmentation(metaslab_t *); static void metaslab_free_impl(vdev_t *, uint64_t, uint64_t, boolean_t); static void metaslab_check_free_impl(vdev_t *, uint64_t, uint64_t); +static void metaslab_passivate(metaslab_t *msp, uint64_t weight); +static uint64_t metaslab_weight_from_range_tree(metaslab_t *msp); kmem_cache_t *metaslab_alloc_trace_cache; @@ -231,7 +233,12 @@ metaslab_class_create(spa_t *spa, metaslab_ops_t *ops) mc->mc_rotor = NULL; mc->mc_ops = ops; mutex_init(&mc->mc_lock, NULL, MUTEX_DEFAULT, NULL); - refcount_create_tracked(&mc->mc_alloc_slots); + mc->mc_alloc_slots = kmem_zalloc(spa->spa_alloc_count * + sizeof (refcount_t), KM_SLEEP); + mc->mc_alloc_max_slots = kmem_zalloc(spa->spa_alloc_count * + sizeof (uint64_t), KM_SLEEP); + for (int i = 0; i < spa->spa_alloc_count; i++) + refcount_create_tracked(&mc->mc_alloc_slots[i]); return (mc); } @@ -245,7 +252,12 @@ metaslab_class_destroy(metaslab_class_t *mc) ASSERT(mc->mc_space == 0); ASSERT(mc->mc_dspace == 0); - refcount_destroy(&mc->mc_alloc_slots); + for (int i = 0; i < mc->mc_spa->spa_alloc_count; i++) + refcount_destroy(&mc->mc_alloc_slots[i]); + kmem_free(mc->mc_alloc_slots, mc->mc_spa->spa_alloc_count * + sizeof (refcount_t)); + kmem_free(mc->mc_alloc_max_slots, mc->mc_spa->spa_alloc_count * + sizeof (uint64_t)); mutex_destroy(&mc->mc_lock); kmem_free(mc, sizeof (metaslab_class_t)); } @@ -442,6 +454,30 @@ metaslab_compare(const void *x1, const void *x2) const metaslab_t *m1 = x1; const metaslab_t *m2 = x2; + int sort1 = 0; + int sort2 = 0; + if (m1->ms_allocator != -1 && m1->ms_primary) + sort1 = 1; + else if (m1->ms_allocator != -1 && !m1->ms_primary) + sort1 = 2; + if (m2->ms_allocator != -1 && m2->ms_primary) + sort2 = 1; + else if (m2->ms_allocator != -1 && !m2->ms_primary) + sort2 = 2; + + /* + * Sort inactive metaslabs first, then primaries, then secondaries. When + * selecting a metaslab to allocate from, an allocator first tries its + * primary, then secondary active metaslab. If it doesn't have active + * metaslabs, or can't allocate from them, it searches for an inactive + * metaslab to activate. If it can't find a suitable one, it will steal + * a primary or secondary metaslab from another allocator. + */ + if (sort1 < sort2) + return (-1); + if (sort1 > sort2) + return (1); + if (m1->ms_weight < m2->ms_weight) return (1); if (m1->ms_weight > m2->ms_weight) @@ -593,12 +629,16 @@ metaslab_group_alloc_update(metaslab_group_t *mg) } metaslab_group_t * -metaslab_group_create(metaslab_class_t *mc, vdev_t *vd) +metaslab_group_create(metaslab_class_t *mc, vdev_t *vd, int allocators) { metaslab_group_t *mg; mg = kmem_zalloc(sizeof (metaslab_group_t), KM_SLEEP); mutex_init(&mg->mg_lock, NULL, MUTEX_DEFAULT, NULL); + mg->mg_primaries = kmem_zalloc(allocators * sizeof (metaslab_t *), + KM_SLEEP); + mg->mg_secondaries = kmem_zalloc(allocators * sizeof (metaslab_t *), + KM_SLEEP); avl_create(&mg->mg_metaslab_tree, metaslab_compare, sizeof (metaslab_t), offsetof(struct metaslab, ms_group_node)); mg->mg_vd = vd; @@ -606,8 +646,17 @@ metaslab_group_create(metaslab_class_t *mc, vdev_t *vd mg->mg_activation_count = 0; mg->mg_initialized = B_FALSE; mg->mg_no_free_space = B_TRUE; - refcount_create_tracked(&mg->mg_alloc_queue_depth); + mg->mg_allocators = allocators; + mg->mg_alloc_queue_depth = kmem_zalloc(allocators * sizeof (refcount_t), + KM_SLEEP); + mg->mg_cur_max_alloc_queue_depth = kmem_zalloc(allocators * + sizeof (uint64_t), KM_SLEEP); + for (int i = 0; i < allocators; i++) { + refcount_create_tracked(&mg->mg_alloc_queue_depth[i]); + mg->mg_cur_max_alloc_queue_depth[i] = 0; + } + mg->mg_taskq = taskq_create("metaslab_group_taskq", metaslab_load_pct, minclsyspri, 10, INT_MAX, TASKQ_THREADS_CPU_PCT); @@ -628,8 +677,20 @@ metaslab_group_destroy(metaslab_group_t *mg) taskq_destroy(mg->mg_taskq); avl_destroy(&mg->mg_metaslab_tree); + kmem_free(mg->mg_primaries, mg->mg_allocators * sizeof (metaslab_t *)); + kmem_free(mg->mg_secondaries, mg->mg_allocators * + sizeof (metaslab_t *)); mutex_destroy(&mg->mg_lock); - refcount_destroy(&mg->mg_alloc_queue_depth); + + for (int i = 0; i < mg->mg_allocators; i++) { + refcount_destroy(&mg->mg_alloc_queue_depth[i]); + mg->mg_cur_max_alloc_queue_depth[i] = 0; + } + kmem_free(mg->mg_alloc_queue_depth, mg->mg_allocators * + sizeof (refcount_t)); + kmem_free(mg->mg_cur_max_alloc_queue_depth, mg->mg_allocators * + sizeof (uint64_t)); + kmem_free(mg, sizeof (metaslab_group_t)); } @@ -708,6 +769,22 @@ metaslab_group_passivate(metaslab_group_t *mg) taskq_wait(mg->mg_taskq); spa_config_enter(spa, locks & ~(SCL_ZIO - 1), spa, RW_WRITER); metaslab_group_alloc_update(mg); + for (int i = 0; i < mg->mg_allocators; i++) { + metaslab_t *msp = mg->mg_primaries[i]; + if (msp != NULL) { + mutex_enter(&msp->ms_lock); + metaslab_passivate(msp, + metaslab_weight_from_range_tree(msp)); + mutex_exit(&msp->ms_lock); + } + msp = mg->mg_secondaries[i]; + if (msp != NULL) { + mutex_enter(&msp->ms_lock); + metaslab_passivate(msp, + metaslab_weight_from_range_tree(msp)); + mutex_exit(&msp->ms_lock); + } + } mgprev = mg->mg_prev; mgnext = mg->mg_next; @@ -848,6 +925,17 @@ metaslab_group_remove(metaslab_group_t *mg, metaslab_t } static void +metaslab_group_sort_impl(metaslab_group_t *mg, metaslab_t *msp, uint64_t weight) +{ + ASSERT(MUTEX_HELD(&mg->mg_lock)); + ASSERT(msp->ms_group == mg); + avl_remove(&mg->mg_metaslab_tree, msp); + msp->ms_weight = weight; + avl_add(&mg->mg_metaslab_tree, msp); + +} + +static void metaslab_group_sort(metaslab_group_t *mg, metaslab_t *msp, uint64_t weight) { /* @@ -858,10 +946,7 @@ metaslab_group_sort(metaslab_group_t *mg, metaslab_t * ASSERT(MUTEX_HELD(&msp->ms_lock)); mutex_enter(&mg->mg_lock); - ASSERT(msp->ms_group == mg); - avl_remove(&mg->mg_metaslab_tree, msp); - msp->ms_weight = weight; - avl_add(&mg->mg_metaslab_tree, msp); + metaslab_group_sort_impl(mg, msp, weight); mutex_exit(&mg->mg_lock); } @@ -909,7 +994,7 @@ metaslab_group_fragmentation(metaslab_group_t *mg) */ static boolean_t metaslab_group_allocatable(metaslab_group_t *mg, metaslab_group_t *rotor, - uint64_t psize) + uint64_t psize, int allocator) { spa_t *spa = mg->mg_vd->vdev_spa; metaslab_class_t *mc = mg->mg_class; @@ -938,7 +1023,7 @@ metaslab_group_allocatable(metaslab_group_t *mg, metas if (mg->mg_allocatable) { metaslab_group_t *mgp; int64_t qdepth; - uint64_t qmax = mg->mg_max_alloc_queue_depth; + uint64_t qmax = mg->mg_cur_max_alloc_queue_depth[allocator]; if (!mc->mc_alloc_throttle_enabled) return (B_TRUE); @@ -950,7 +1035,7 @@ metaslab_group_allocatable(metaslab_group_t *mg, metas if (mg->mg_no_free_space) return (B_FALSE); - qdepth = refcount_count(&mg->mg_alloc_queue_depth); + qdepth = refcount_count(&mg->mg_alloc_queue_depth[allocator]); /* * If this metaslab group is below its qmax or it's @@ -969,9 +1054,10 @@ metaslab_group_allocatable(metaslab_group_t *mg, metas * groups at the same time when we make this check. */ for (mgp = mg->mg_next; mgp != rotor; mgp = mgp->mg_next) { - qmax = mgp->mg_max_alloc_queue_depth; + qmax = mgp->mg_cur_max_alloc_queue_depth[allocator]; - qdepth = refcount_count(&mgp->mg_alloc_queue_depth); + qdepth = refcount_count( + &mgp->mg_alloc_queue_depth[allocator]); /* * If there is another metaslab group that @@ -1458,6 +1544,8 @@ metaslab_init(metaslab_group_t *mg, uint64_t id, uint6 ms->ms_id = id; ms->ms_start = id << vd->vdev_ms_shift; ms->ms_size = 1ULL << vd->vdev_ms_shift; + ms->ms_allocator = -1; + ms->ms_new = B_TRUE; /* * We only open space map objects that already exist. All others @@ -1553,6 +1641,7 @@ metaslab_fini(metaslab_t *msp) cv_destroy(&msp->ms_load_cv); mutex_destroy(&msp->ms_lock); mutex_destroy(&msp->ms_sync_lock); + ASSERT3U(msp->ms_allocator, ==, -1); kmem_free(msp, sizeof (metaslab_t)); } @@ -1949,19 +2038,59 @@ metaslab_weight(metaslab_t *msp) } static int -metaslab_activate(metaslab_t *msp, uint64_t activation_weight) +metaslab_activate_allocator(metaslab_group_t *mg, metaslab_t *msp, + int allocator, uint64_t activation_weight) { + /* + * If we're activating for the claim code, we don't want to actually + * set the metaslab up for a specific allocator. + */ + if (activation_weight == METASLAB_WEIGHT_CLAIM) + return (0); + metaslab_t **arr = (activation_weight == METASLAB_WEIGHT_PRIMARY ? + mg->mg_primaries : mg->mg_secondaries); + ASSERT(MUTEX_HELD(&msp->ms_lock)); + mutex_enter(&mg->mg_lock); + if (arr[allocator] != NULL) { + mutex_exit(&mg->mg_lock); + return (EEXIST); + } + arr[allocator] = msp; + ASSERT3S(msp->ms_allocator, ==, -1); + msp->ms_allocator = allocator; + msp->ms_primary = (activation_weight == METASLAB_WEIGHT_PRIMARY); + mutex_exit(&mg->mg_lock); + + return (0); +} + +static int +metaslab_activate(metaslab_t *msp, int allocator, uint64_t activation_weight) +{ + ASSERT(MUTEX_HELD(&msp->ms_lock)); + if ((msp->ms_weight & METASLAB_ACTIVE_MASK) == 0) { + int error = 0; metaslab_load_wait(msp); if (!msp->ms_loaded) { - int error = metaslab_load(msp); - if (error) { + if ((error = metaslab_load(msp)) != 0) { metaslab_group_sort(msp->ms_group, msp, 0); return (error); } } + if ((msp->ms_weight & METASLAB_ACTIVE_MASK) != 0) { + /* + * The metaslab was activated for another allocator + * while we were waiting, we should reselect. + */ + return (EBUSY); + } + if ((error = metaslab_activate_allocator(msp->ms_group, msp, + allocator, activation_weight)) != 0) { + return (error); + } msp->ms_activation_weight = msp->ms_weight; metaslab_group_sort(msp->ms_group, msp, @@ -1974,6 +2103,34 @@ metaslab_activate(metaslab_t *msp, uint64_t activation } static void +metaslab_passivate_allocator(metaslab_group_t *mg, metaslab_t *msp, + uint64_t weight) +{ + ASSERT(MUTEX_HELD(&msp->ms_lock)); + if (msp->ms_weight & METASLAB_WEIGHT_CLAIM) { + metaslab_group_sort(mg, msp, weight); + return; + } + + mutex_enter(&mg->mg_lock); + ASSERT3P(msp->ms_group, ==, mg); + if (msp->ms_primary) { + ASSERT3U(0, <=, msp->ms_allocator); + ASSERT3U(msp->ms_allocator, <, mg->mg_allocators); + ASSERT3P(mg->mg_primaries[msp->ms_allocator], ==, msp); + ASSERT(msp->ms_weight & METASLAB_WEIGHT_PRIMARY); + mg->mg_primaries[msp->ms_allocator] = NULL; + } else { + ASSERT(msp->ms_weight & METASLAB_WEIGHT_SECONDARY); + ASSERT3P(mg->mg_secondaries[msp->ms_allocator], ==, msp); + mg->mg_secondaries[msp->ms_allocator] = NULL; + } + msp->ms_allocator = -1; + metaslab_group_sort_impl(mg, msp, weight); + mutex_exit(&mg->mg_lock); +} + +static void metaslab_passivate(metaslab_t *msp, uint64_t weight) { uint64_t size = weight & ~METASLAB_WEIGHT_TYPE; @@ -1988,7 +2145,7 @@ metaslab_passivate(metaslab_t *msp, uint64_t weight) ASSERT0(weight & METASLAB_ACTIVE_MASK); msp->ms_activation_weight = 0; - metaslab_group_sort(msp->ms_group, msp, weight); + metaslab_passivate_allocator(msp->ms_group, msp, weight); ASSERT((msp->ms_weight & METASLAB_ACTIVE_MASK) == 0); } @@ -2542,11 +2699,18 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg) vdev_dirty(vd, VDD_METASLAB, msp, txg + 1); } + if (msp->ms_new) { + msp->ms_new = B_FALSE; + mutex_enter(&mg->mg_lock); + mg->mg_ms_ready++; + mutex_exit(&mg->mg_lock); + } /* * Calculate the new weights before unloading any metaslabs. * This will give us the most accurate weighting. */ - metaslab_group_sort(mg, msp, metaslab_weight(msp)); + metaslab_group_sort(mg, msp, metaslab_weight(msp) | + (msp->ms_weight & METASLAB_ACTIVE_MASK)); /* * If the metaslab is loaded and we've not tried to load or allocate @@ -2558,6 +2722,10 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg) VERIFY0(range_tree_space( msp->ms_allocating[(txg + t) & TXG_MASK])); } + if (msp->ms_allocator != -1) { + metaslab_passivate(msp, msp->ms_weight & + ~METASLAB_ACTIVE_MASK); + } if (!metaslab_debug_unload) metaslab_unload(msp); @@ -2651,7 +2819,8 @@ metaslab_alloc_trace_fini(void) */ static void metaslab_trace_add(zio_alloc_list_t *zal, metaslab_group_t *mg, - metaslab_t *msp, uint64_t psize, uint32_t dva_id, uint64_t offset) + metaslab_t *msp, uint64_t psize, uint32_t dva_id, uint64_t offset, + int allocator) { if (!metaslab_trace_enabled) return; @@ -2684,6 +2853,7 @@ metaslab_trace_add(zio_alloc_list_t *zal, metaslab_gro mat->mat_dva_id = dva_id; mat->mat_offset = offset; mat->mat_weight = 0; + mat->mat_allocator = allocator; if (msp != NULL) mat->mat_weight = msp->ms_weight; @@ -2724,35 +2894,56 @@ metaslab_trace_fini(zio_alloc_list_t *zal) */ static void -metaslab_group_alloc_increment(spa_t *spa, uint64_t vdev, void *tag, int flags) +metaslab_group_alloc_increment(spa_t *spa, uint64_t vdev, void *tag, int flags, + int allocator) { if (!(flags & METASLAB_ASYNC_ALLOC) || - flags & METASLAB_DONT_THROTTLE) + (flags & METASLAB_DONT_THROTTLE)) return; metaslab_group_t *mg = vdev_lookup_top(spa, vdev)->vdev_mg; if (!mg->mg_class->mc_alloc_throttle_enabled) return; - (void) refcount_add(&mg->mg_alloc_queue_depth, tag); + (void) refcount_add(&mg->mg_alloc_queue_depth[allocator], tag); } +static void +metaslab_group_increment_qdepth(metaslab_group_t *mg, int allocator) +{ + uint64_t max = mg->mg_max_alloc_queue_depth; + uint64_t cur = mg->mg_cur_max_alloc_queue_depth[allocator]; + while (cur < max) { + if (atomic_cas_64(&mg->mg_cur_max_alloc_queue_depth[allocator], + cur, cur + 1) == cur) { + atomic_inc_64( + &mg->mg_class->mc_alloc_max_slots[allocator]); + return; + } + cur = mg->mg_cur_max_alloc_queue_depth[allocator]; + } +} + void -metaslab_group_alloc_decrement(spa_t *spa, uint64_t vdev, void *tag, int flags) +metaslab_group_alloc_decrement(spa_t *spa, uint64_t vdev, void *tag, int flags, + int allocator, boolean_t io_complete) { if (!(flags & METASLAB_ASYNC_ALLOC) || - flags & METASLAB_DONT_THROTTLE) + (flags & METASLAB_DONT_THROTTLE)) return; metaslab_group_t *mg = vdev_lookup_top(spa, vdev)->vdev_mg; if (!mg->mg_class->mc_alloc_throttle_enabled) return; - (void) refcount_remove(&mg->mg_alloc_queue_depth, tag); + (void) refcount_remove(&mg->mg_alloc_queue_depth[allocator], tag); + if (io_complete) + metaslab_group_increment_qdepth(mg, allocator); } void -metaslab_group_alloc_verify(spa_t *spa, const blkptr_t *bp, void *tag) +metaslab_group_alloc_verify(spa_t *spa, const blkptr_t *bp, void *tag, + int allocator) { #ifdef ZFS_DEBUG const dva_t *dva = bp->blk_dva; @@ -2761,7 +2952,8 @@ metaslab_group_alloc_verify(spa_t *spa, const blkptr_t for (int d = 0; d < ndvas; d++) { uint64_t vdev = DVA_GET_VDEV(&dva[d]); metaslab_group_t *mg = vdev_lookup_top(spa, vdev)->vdev_mg; - VERIFY(refcount_not_held(&mg->mg_alloc_queue_depth, tag)); + VERIFY(refcount_not_held(&mg->mg_alloc_queue_depth[allocator], + tag)); } #endif } @@ -2803,91 +2995,146 @@ metaslab_block_alloc(metaslab_t *msp, uint64_t size, u return (start); } +/* + * Find the metaslab with the highest weight that is less than what we've + * already tried. In the common case, this means that we will examine each + * metaslab at most once. Note that concurrent callers could reorder metaslabs + * by activation/passivation once we have dropped the mg_lock. If a metaslab is + * activated by another thread, and we fail to allocate from the metaslab we + * have selected, we may not try the newly-activated metaslab, and instead + * activate another metaslab. This is not optimal, but generally does not cause + * any problems (a possible exception being if every metaslab is completely full + * except for the the newly-activated metaslab which we fail to examine). + */ +static metaslab_t * +find_valid_metaslab(metaslab_group_t *mg, uint64_t activation_weight, + dva_t *dva, int d, uint64_t min_distance, uint64_t asize, int allocator, + zio_alloc_list_t *zal, metaslab_t *search, boolean_t *was_active) +{ + avl_index_t idx; + avl_tree_t *t = &mg->mg_metaslab_tree; + metaslab_t *msp = avl_find(t, search, &idx); + if (msp == NULL) + msp = avl_nearest(t, idx, AVL_AFTER); + + for (; msp != NULL; msp = AVL_NEXT(t, msp)) { + int i; + if (!metaslab_should_allocate(msp, asize)) { + metaslab_trace_add(zal, mg, msp, asize, d, + TRACE_TOO_SMALL, allocator); + continue; + } + + /* + * If the selected metaslab is condensing, skip it. + */ + if (msp->ms_condensing) + continue; + + *was_active = msp->ms_allocator != -1; + /* + * If we're activating as primary, this is our first allocation + * from this disk, so we don't need to check how close we are. + * If the metaslab under consideration was already active, + * we're getting desperate enough to steal another allocator's + * metaslab, so we still don't care about distances. + */ + if (activation_weight == METASLAB_WEIGHT_PRIMARY || *was_active) + break; + + uint64_t target_distance = min_distance + + (space_map_allocated(msp->ms_sm) != 0 ? 0 : + min_distance >> 1); + + for (i = 0; i < d; i++) { + if (metaslab_distance(msp, &dva[i]) < target_distance) + break; + } + if (i == d) + break; + } + + if (msp != NULL) { + search->ms_weight = msp->ms_weight; + search->ms_start = msp->ms_start + 1; + search->ms_allocator = msp->ms_allocator; + search->ms_primary = msp->ms_primary; + } + return (msp); +} + +/* ARGSUSED */ static uint64_t metaslab_group_alloc_normal(metaslab_group_t *mg, zio_alloc_list_t *zal, - uint64_t asize, uint64_t txg, uint64_t min_distance, dva_t *dva, int d) + uint64_t asize, uint64_t txg, uint64_t min_distance, dva_t *dva, int d, + int allocator) { metaslab_t *msp = NULL; uint64_t offset = -1ULL; uint64_t activation_weight; - uint64_t target_distance; - int i; + boolean_t tertiary = B_FALSE; activation_weight = METASLAB_WEIGHT_PRIMARY; - for (i = 0; i < d; i++) { - if (DVA_GET_VDEV(&dva[i]) == mg->mg_vd->vdev_id) { + for (int i = 0; i < d; i++) { + if (activation_weight == METASLAB_WEIGHT_PRIMARY && + DVA_GET_VDEV(&dva[i]) == mg->mg_vd->vdev_id) { activation_weight = METASLAB_WEIGHT_SECONDARY; + } else if (activation_weight == METASLAB_WEIGHT_SECONDARY && + DVA_GET_VDEV(&dva[i]) == mg->mg_vd->vdev_id) { + tertiary = B_TRUE; break; } } + /* + * If we don't have enough metaslabs active to fill the entire array, we + * just use the 0th slot. + */ + if (mg->mg_ms_ready < mg->mg_allocators * 2) { + tertiary = B_FALSE; + allocator = 0; + } + + ASSERT3U(mg->mg_vd->vdev_ms_count, >=, 2); + metaslab_t *search = kmem_alloc(sizeof (*search), KM_SLEEP); search->ms_weight = UINT64_MAX; search->ms_start = 0; + /* + * At the end of the metaslab tree are the already-active metaslabs, + * first the primaries, then the secondaries. When we resume searching + * through the tree, we need to consider ms_allocator and ms_primary so + * we start in the location right after where we left off, and don't + * accidentally loop forever considering the same metaslabs. + */ + search->ms_allocator = -1; + search->ms_primary = B_TRUE; for (;;) { - boolean_t was_active; - avl_tree_t *t = &mg->mg_metaslab_tree; - avl_index_t idx; + boolean_t was_active = B_FALSE; mutex_enter(&mg->mg_lock); - /* - * Find the metaslab with the highest weight that is less - * than what we've already tried. In the common case, this - * means that we will examine each metaslab at most once. - * Note that concurrent callers could reorder metaslabs - * by activation/passivation once we have dropped the mg_lock. - * If a metaslab is activated by another thread, and we fail - * to allocate from the metaslab we have selected, we may - * not try the newly-activated metaslab, and instead activate - * another metaslab. This is not optimal, but generally - * does not cause any problems (a possible exception being - * if every metaslab is completely full except for the - * the newly-activated metaslab which we fail to examine). - */ - msp = avl_find(t, search, &idx); - if (msp == NULL) - msp = avl_nearest(t, idx, AVL_AFTER); - for (; msp != NULL; msp = AVL_NEXT(t, msp)) { - - if (!metaslab_should_allocate(msp, asize)) { - metaslab_trace_add(zal, mg, msp, asize, d, - TRACE_TOO_SMALL); - continue; - } - - /* - * If the selected metaslab is condensing, skip it. - */ - if (msp->ms_condensing) - continue; - - was_active = msp->ms_weight & METASLAB_ACTIVE_MASK; - if (activation_weight == METASLAB_WEIGHT_PRIMARY) - break; - - target_distance = min_distance + - (space_map_allocated(msp->ms_sm) != 0 ? 0 : - min_distance >> 1); - - for (i = 0; i < d; i++) { - if (metaslab_distance(msp, &dva[i]) < - target_distance) - break; - } - if (i == d) - break; + if (activation_weight == METASLAB_WEIGHT_PRIMARY && + mg->mg_primaries[allocator] != NULL) { + msp = mg->mg_primaries[allocator]; + was_active = B_TRUE; + } else if (activation_weight == METASLAB_WEIGHT_SECONDARY && + mg->mg_secondaries[allocator] != NULL && !tertiary) { + msp = mg->mg_secondaries[allocator]; + was_active = B_TRUE; + } else { + msp = find_valid_metaslab(mg, activation_weight, dva, d, + min_distance, asize, allocator, zal, search, + &was_active); } + mutex_exit(&mg->mg_lock); if (msp == NULL) { kmem_free(search, sizeof (*search)); return (-1ULL); } - search->ms_weight = msp->ms_weight; - search->ms_start = msp->ms_start + 1; mutex_enter(&msp->ms_lock); - /* * Ensure that the metaslab we have selected is still * capable of handling our request. It's possible that @@ -2901,18 +3148,32 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ continue; } - if ((msp->ms_weight & METASLAB_WEIGHT_SECONDARY) && - activation_weight == METASLAB_WEIGHT_PRIMARY) { - metaslab_passivate(msp, - msp->ms_weight & ~METASLAB_ACTIVE_MASK); + /* + * If the metaslab is freshly activated for an allocator that + * isn't the one we're allocating from, or if it's a primary and + * we're seeking a secondary (or vice versa), we go back and + * select a new metaslab. + */ + if (!was_active && (msp->ms_weight & METASLAB_ACTIVE_MASK) && + (msp->ms_allocator != -1) && + (msp->ms_allocator != allocator || ((activation_weight == + METASLAB_WEIGHT_PRIMARY) != msp->ms_primary))) { mutex_exit(&msp->ms_lock); continue; } - if (metaslab_activate(msp, activation_weight) != 0) { + if (msp->ms_weight & METASLAB_WEIGHT_CLAIM) { + metaslab_passivate(msp, msp->ms_weight & + ~METASLAB_WEIGHT_CLAIM); mutex_exit(&msp->ms_lock); continue; } + + if (metaslab_activate(msp, allocator, activation_weight) != 0) { + mutex_exit(&msp->ms_lock); + continue; + } + msp->ms_selected_txg = txg; /* @@ -2925,7 +3186,7 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ if (!metaslab_should_allocate(msp, asize)) { /* Passivate this metaslab and select a new one. */ metaslab_trace_add(zal, mg, msp, asize, d, - TRACE_TOO_SMALL); + TRACE_TOO_SMALL, allocator); goto next; } @@ -2936,13 +3197,15 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ */ if (msp->ms_condensing) { metaslab_trace_add(zal, mg, msp, asize, d, - TRACE_CONDENSING); + TRACE_CONDENSING, allocator); + metaslab_passivate(msp, msp->ms_weight & + ~METASLAB_ACTIVE_MASK); mutex_exit(&msp->ms_lock); continue; } offset = metaslab_block_alloc(msp, asize, txg); - metaslab_trace_add(zal, mg, msp, asize, d, offset); + metaslab_trace_add(zal, mg, msp, asize, d, offset, allocator); if (offset != -1ULL) { /* Proactively passivate the metaslab, if needed */ @@ -2998,19 +3261,20 @@ next: static uint64_t metaslab_group_alloc(metaslab_group_t *mg, zio_alloc_list_t *zal, - uint64_t asize, uint64_t txg, uint64_t min_distance, dva_t *dva, int d) + uint64_t asize, uint64_t txg, uint64_t min_distance, dva_t *dva, int d, + int allocator) { uint64_t offset; ASSERT(mg->mg_initialized); offset = metaslab_group_alloc_normal(mg, zal, asize, txg, - min_distance, dva, d); + min_distance, dva, d, allocator); mutex_enter(&mg->mg_lock); if (offset == -1ULL) { mg->mg_failed_allocations++; metaslab_trace_add(zal, mg, NULL, asize, d, - TRACE_GROUP_FAILURE); + TRACE_GROUP_FAILURE, allocator); if (asize == SPA_GANGBLOCKSIZE) { /* * This metaslab group was unable to allocate @@ -3045,7 +3309,7 @@ int ditto_same_vdev_distance_shift = 3; int metaslab_alloc_dva(spa_t *spa, metaslab_class_t *mc, uint64_t psize, dva_t *dva, int d, dva_t *hintdva, uint64_t txg, int flags, - zio_alloc_list_t *zal) + zio_alloc_list_t *zal, int allocator) { metaslab_group_t *mg, *rotor; vdev_t *vd; @@ -3057,7 +3321,8 @@ metaslab_alloc_dva(spa_t *spa, metaslab_class_t *mc, u * For testing, make some blocks above a certain size be gang blocks. */ if (psize >= metaslab_force_ganging && (ddi_get_lbolt() & 3) == 0) { - metaslab_trace_add(zal, NULL, NULL, psize, d, TRACE_FORCE_GANG); + metaslab_trace_add(zal, NULL, NULL, psize, d, TRACE_FORCE_GANG, + allocator); return (SET_ERROR(ENOSPC)); } @@ -3143,12 +3408,12 @@ top: */ if (allocatable && !GANG_ALLOCATION(flags) && !try_hard) { allocatable = metaslab_group_allocatable(mg, rotor, - psize); + psize, allocator); } if (!allocatable) { metaslab_trace_add(zal, mg, NULL, psize, d, - TRACE_NOT_ALLOCATABLE); + TRACE_NOT_ALLOCATABLE, allocator); goto next; } @@ -3163,7 +3428,7 @@ top: vd->vdev_state < VDEV_STATE_HEALTHY) && d == 0 && !try_hard && vd->vdev_children == 0) { metaslab_trace_add(zal, mg, NULL, psize, d, - TRACE_VDEV_ERROR); + TRACE_VDEV_ERROR, allocator); goto next; } @@ -3187,7 +3452,7 @@ top: ASSERT(P2PHASE(asize, 1ULL << vd->vdev_ashift) == 0); uint64_t offset = metaslab_group_alloc(mg, zal, asize, txg, - distance, dva, d); + distance, dva, d, allocator); if (offset != -1ULL) { /* @@ -3250,7 +3515,7 @@ next: bzero(&dva[d], sizeof (dva_t)); - metaslab_trace_add(zal, rotor, NULL, psize, d, TRACE_ENOSPC); + metaslab_trace_add(zal, rotor, NULL, psize, d, TRACE_ENOSPC, allocator); return (SET_ERROR(ENOSPC)); } @@ -3551,18 +3816,20 @@ metaslab_free_dva(spa_t *spa, const dva_t *dva, boolea * the reservation. */ boolean_t -metaslab_class_throttle_reserve(metaslab_class_t *mc, int slots, zio_t *zio, - int flags) +metaslab_class_throttle_reserve(metaslab_class_t *mc, int slots, int allocator, + zio_t *zio, int flags) { uint64_t available_slots = 0; boolean_t slot_reserved = B_FALSE; + uint64_t max = mc->mc_alloc_max_slots[allocator]; ASSERT(mc->mc_alloc_throttle_enabled); mutex_enter(&mc->mc_lock); - uint64_t reserved_slots = refcount_count(&mc->mc_alloc_slots); - if (reserved_slots < mc->mc_alloc_max_slots) - available_slots = mc->mc_alloc_max_slots - reserved_slots; + uint64_t reserved_slots = + refcount_count(&mc->mc_alloc_slots[allocator]); + if (reserved_slots < max) + available_slots = max - reserved_slots; if (slots <= available_slots || GANG_ALLOCATION(flags)) { /* @@ -3570,7 +3837,9 @@ metaslab_class_throttle_reserve(metaslab_class_t *mc, * them individually when an I/O completes. */ for (int d = 0; d < slots; d++) { - reserved_slots = refcount_add(&mc->mc_alloc_slots, zio); + reserved_slots = + refcount_add(&mc->mc_alloc_slots[allocator], + zio); } zio->io_flags |= ZIO_FLAG_IO_ALLOCATING; slot_reserved = B_TRUE; @@ -3581,12 +3850,14 @@ metaslab_class_throttle_reserve(metaslab_class_t *mc, } void -metaslab_class_throttle_unreserve(metaslab_class_t *mc, int slots, zio_t *zio) +metaslab_class_throttle_unreserve(metaslab_class_t *mc, int slots, + int allocator, zio_t *zio) { ASSERT(mc->mc_alloc_throttle_enabled); mutex_enter(&mc->mc_lock); for (int d = 0; d < slots; d++) { - (void) refcount_remove(&mc->mc_alloc_slots, zio); + (void) refcount_remove(&mc->mc_alloc_slots[allocator], + zio); } mutex_exit(&mc->mc_lock); } @@ -3608,7 +3879,13 @@ metaslab_claim_concrete(vdev_t *vd, uint64_t offset, u mutex_enter(&msp->ms_lock); if ((txg != 0 && spa_writeable(spa)) || !msp->ms_loaded) - error = metaslab_activate(msp, METASLAB_WEIGHT_SECONDARY); + error = metaslab_activate(msp, 0, METASLAB_WEIGHT_CLAIM); + /* + * No need to fail in that case; someone else has activated the + * metaslab, but that doesn't preclude us from using it. + */ + if (error == EBUSY) + error = 0; if (error == 0 && !range_tree_contains(msp->ms_allocatable, offset, size)) @@ -3713,7 +3990,7 @@ metaslab_claim_dva(spa_t *spa, const dva_t *dva, uint6 int metaslab_alloc(spa_t *spa, metaslab_class_t *mc, uint64_t psize, blkptr_t *bp, int ndvas, uint64_t txg, blkptr_t *hintbp, int flags, - zio_alloc_list_t *zal, zio_t *zio) + zio_alloc_list_t *zal, zio_t *zio, int allocator) { dva_t *dva = bp->blk_dva; dva_t *hintdva = hintbp->blk_dva; @@ -3736,12 +4013,13 @@ metaslab_alloc(spa_t *spa, metaslab_class_t *mc, uint6 for (int d = 0; d < ndvas; d++) { error = metaslab_alloc_dva(spa, mc, psize, dva, d, hintdva, - txg, flags, zal); + txg, flags, zal, allocator); if (error != 0) { for (d--; d >= 0; d--) { metaslab_unalloc_dva(spa, &dva[d], txg); metaslab_group_alloc_decrement(spa, - DVA_GET_VDEV(&dva[d]), zio, flags); + DVA_GET_VDEV(&dva[d]), zio, flags, + allocator, B_FALSE); bzero(&dva[d], sizeof (dva_t)); } spa_config_exit(spa, SCL_ALLOC, FTAG); @@ -3752,7 +4030,7 @@ metaslab_alloc(spa_t *spa, metaslab_class_t *mc, uint6 * based on the newly allocated dva. */ metaslab_group_alloc_increment(spa, - DVA_GET_VDEV(&dva[d]), zio, flags); + DVA_GET_VDEV(&dva[d]), zio, flags, allocator); } } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Mon Jul 30 23:47:38 2018 (r336947) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Mon Jul 30 23:53:25 2018 (r336948) @@ -7398,9 +7398,11 @@ spa_sync(spa_t *spa, uint64_t txg) spa->spa_syncing_txg = txg; spa->spa_sync_pass = 0; - mutex_enter(&spa->spa_alloc_lock); - VERIFY0(avl_numnodes(&spa->spa_alloc_tree)); - mutex_exit(&spa->spa_alloc_lock); + for (int i = 0; i < spa->spa_alloc_count; i++) { + mutex_enter(&spa->spa_alloc_locks[i]); + VERIFY0(avl_numnodes(&spa->spa_alloc_trees[i])); + mutex_exit(&spa->spa_alloc_locks[i]); + } /* * If there are any pending vdev state changes, convert them @@ -7459,7 +7461,7 @@ spa_sync(spa_t *spa, uint64_t txg) * The max queue depth will not change in the middle of syncing * out this txg. */ - uint64_t queue_depth_total = 0; + uint64_t slots_per_allocator = 0; for (int c = 0; c < rvd->vdev_children; c++) { vdev_t *tvd = rvd->vdev_child[c]; metaslab_group_t *mg = tvd->vdev_mg; @@ -7473,18 +7475,23 @@ spa_sync(spa_t *spa, uint64_t txg) * allocations look at mg_max_alloc_queue_depth, and async * allocations all happen from spa_sync(). */ - ASSERT0(refcount_count(&mg->mg_alloc_queue_depth)); + for (int i = 0; i < spa->spa_alloc_count; i++) + ASSERT0(refcount_count(&(mg->mg_alloc_queue_depth[i]))); mg->mg_max_alloc_queue_depth = max_queue_depth; - queue_depth_total += mg->mg_max_alloc_queue_depth; + + for (int i = 0; i < spa->spa_alloc_count; i++) { + mg->mg_cur_max_alloc_queue_depth[i] = + zfs_vdev_def_queue_depth; + } + slots_per_allocator += zfs_vdev_def_queue_depth; } metaslab_class_t *mc = spa_normal_class(spa); - ASSERT0(refcount_count(&mc->mc_alloc_slots)); - mc->mc_alloc_max_slots = queue_depth_total; + for (int i = 0; i < spa->spa_alloc_count; i++) { + ASSERT0(refcount_count(&mc->mc_alloc_slots[i])); + mc->mc_alloc_max_slots[i] = slots_per_allocator; + } mc->mc_alloc_throttle_enabled = zio_dva_throttle_enabled; - ASSERT3U(mc->mc_alloc_max_slots, <=, - max_queue_depth * rvd->vdev_children); - for (int c = 0; c < rvd->vdev_children; c++) { vdev_t *vd = rvd->vdev_child[c]; vdev_indirect_state_sync_verify(vd); @@ -7661,9 +7668,11 @@ spa_sync(spa_t *spa, uint64_t txg) dsl_pool_sync_done(dp, txg); - mutex_enter(&spa->spa_alloc_lock); - VERIFY0(avl_numnodes(&spa->spa_alloc_tree)); - mutex_exit(&spa->spa_alloc_lock); + for (int i = 0; i < spa->spa_alloc_count; i++) { *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Tue Jul 31 00:13:07 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D9471065B79; Tue, 31 Jul 2018 00:13:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 201E88E4AA; Tue, 31 Jul 2018 00:13:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F0B0A4058; Tue, 31 Jul 2018 00:13:06 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6V0D6fB088739; Tue, 31 Jul 2018 00:13:06 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6V0D48W088729; Tue, 31 Jul 2018 00:13:04 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807310013.w6V0D48W088729@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 00:13:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336950 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist/lib/libzf... X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 336950 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 00:13:07 -0000 Author: mav Date: Tue Jul 31 00:13:04 2018 New Revision: 336950 URL: https://svnweb.freebsd.org/changeset/base/336950 Log: 9290 device removal reduces redundancy of mirrors Mirrors are supposed to provide redundancy in the face of whole-disk failure and silent damage (e.g. some data on disk is not right, but ZFS hasn't detected the whole device as being broken). However, the current device removal implementation bypasses some of the mirror's redundancy. illumos/illumos-gate@3a4b1be953ee5601bab748afa07c26ed4996cde6 Reviewed by: George Wilson Reviewed by: Prashanth Sreenivasa Reviewed by: Sara Hartse Reviewed by: Serapheim Dimitropoulos Reviewed by: Brian Behlendorf Reviewed by: Tim Chase Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Tue Jul 31 00:13:04 2018 (r336950) @@ -1988,7 +1988,16 @@ dsl_scan_scrub_cb(dsl_pool_t *dp, /* if it's a resilver, this may not be in the target range */ if (!needs_io) { - if (DVA_GET_GANG(&bp->blk_dva[d])) { + if (vd->vdev_ops == &vdev_indirect_ops) { + /* + * The indirect vdev can point to multiple + * vdevs. For simplicity, always create + * the resilver zio_t. zio_vdev_io_start() + * will bypass the child resilver i/o's if + * they are on vdevs that don't have DTL's. + */ + needs_io = B_TRUE; + } else if (DVA_GET_GANG(&bp->blk_dva[d])) { /* * Gang members may be spread across multiple * vdevs, so the best estimate we have is the Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Tue Jul 31 00:13:04 2018 (r336950) @@ -3582,7 +3582,7 @@ metaslab_free_impl(vdev_t *vd, uint64_t offset, uint64 return; if (spa->spa_vdev_removal != NULL && - spa->spa_vdev_removal->svr_vdev == vd && + spa->spa_vdev_removal->svr_vdev_id == vd->vdev_id && vdev_is_concrete(vd)) { /* * Note: we check if the vdev is concrete because when Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Tue Jul 31 00:13:04 2018 (r336950) @@ -5509,8 +5509,7 @@ spa_vdev_add(spa_t *spa, nvlist_t *nvroot) for (int c = 0; c < vd->vdev_children; c++) { tvd = vd->vdev_child[c]; if (spa->spa_vdev_removal != NULL && - tvd->vdev_ashift != - spa->spa_vdev_removal->svr_vdev->vdev_ashift) { + tvd->vdev_ashift != spa->spa_max_ashift) { return (spa_vdev_exit(spa, vd, txg, EINVAL)); } /* Fail if top level vdev is raidz */ @@ -5626,10 +5625,8 @@ spa_vdev_attach(spa_t *spa, uint64_t guid, nvlist_t *n return (spa_vdev_exit(spa, NULL, txg, error)); } - if (spa->spa_vdev_removal != NULL || - spa->spa_removing_phys.sr_prev_indirect_vdev != -1) { + if (spa->spa_vdev_removal != NULL) return (spa_vdev_exit(spa, NULL, txg, EBUSY)); - } if (oldvd == NULL) return (spa_vdev_exit(spa, NULL, txg, ENODEV)); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Tue Jul 31 00:13:04 2018 (r336950) @@ -1761,9 +1761,12 @@ spa_update_dspace(spa_t *spa) * allocated twice (on the old device and the new * device). */ - vdev_t *vd = spa->spa_vdev_removal->svr_vdev; + spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); + vdev_t *vd = + vdev_lookup_top(spa, spa->spa_vdev_removal->svr_vdev_id); spa->spa_dspace -= spa_deflate(spa) ? vd->vdev_stat.vs_dspace : vd->vdev_stat.vs_space; + spa_config_exit(spa, SCL_VDEV, FTAG); } } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h Tue Jul 31 00:13:04 2018 (r336950) @@ -30,7 +30,7 @@ extern "C" { #endif typedef struct spa_vdev_removal { - vdev_t *svr_vdev; + uint64_t svr_vdev_id; uint64_t svr_max_offset_to_sync[TXG_SIZE]; /* Thread performing a vdev removal. */ kthread_t *svr_thread; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h Tue Jul 31 00:13:04 2018 (r336950) @@ -550,7 +550,7 @@ extern zio_t *zio_vdev_child_io(zio_t *zio, blkptr_t * zio_done_func_t *done, void *private); extern zio_t *zio_vdev_delegated_io(vdev_t *vd, uint64_t offset, - struct abd *data, uint64_t size, int type, zio_priority_t priority, + struct abd *data, uint64_t size, zio_type_t type, zio_priority_t priority, enum zio_flag flags, zio_done_func_t *done, void *private); extern void zio_vdev_io_bypass(zio_t *zio); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Tue Jul 31 00:13:04 2018 (r336950) @@ -867,6 +867,32 @@ vdev_top_transfer(vdev_t *svd, vdev_t *tvd) svd->vdev_stat.vs_space = 0; svd->vdev_stat.vs_dspace = 0; + /* + * State which may be set on a top-level vdev that's in the + * process of being removed. + */ + ASSERT0(tvd->vdev_indirect_config.vic_births_object); + ASSERT0(tvd->vdev_indirect_config.vic_mapping_object); + ASSERT3U(tvd->vdev_indirect_config.vic_prev_indirect_vdev, ==, -1ULL); + ASSERT3P(tvd->vdev_indirect_mapping, ==, NULL); + ASSERT3P(tvd->vdev_indirect_births, ==, NULL); + ASSERT3P(tvd->vdev_obsolete_sm, ==, NULL); + ASSERT0(tvd->vdev_removing); + tvd->vdev_removing = svd->vdev_removing; + tvd->vdev_indirect_config = svd->vdev_indirect_config; + tvd->vdev_indirect_mapping = svd->vdev_indirect_mapping; + tvd->vdev_indirect_births = svd->vdev_indirect_births; + range_tree_swap(&svd->vdev_obsolete_segments, + &tvd->vdev_obsolete_segments); + tvd->vdev_obsolete_sm = svd->vdev_obsolete_sm; + svd->vdev_indirect_config.vic_mapping_object = 0; + svd->vdev_indirect_config.vic_births_object = 0; + svd->vdev_indirect_config.vic_prev_indirect_vdev = -1ULL; + svd->vdev_indirect_mapping = NULL; + svd->vdev_indirect_births = NULL; + svd->vdev_obsolete_sm = NULL; + svd->vdev_removing = 0; + for (t = 0; t < TXG_SIZE; t++) { while ((msp = txg_list_remove(&svd->vdev_ms_list, t)) != NULL) (void) txg_list_add(&tvd->vdev_ms_list, msp, t); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Tue Jul 31 00:13:04 2018 (r336950) @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -46,10 +47,11 @@ * "vdev_remap" operation that executes a callback on each contiguous * segment of the new location. This function is used in multiple ways: * - * - reads and repair writes to this device use the callback to create - * a child io for each mapped segment. + * - i/os to this vdev use the callback to determine where the + * data is now located, and issue child i/os for each segment's new + * location. * - * - frees and claims to this device use the callback to free or claim + * - frees and claims to this vdev use the callback to free or claim * each mapped segment. (Note that we don't actually need to claim * log blocks on indirect vdevs, because we don't allocate to * removing vdevs. However, zdb uses zio_claim() for its leak @@ -204,6 +206,94 @@ uint64_t zfs_condense_min_mapping_bytes = 128 * 1024; int zfs_condense_indirect_commit_entry_delay_ticks = 0; /* + * If a split block contains more than this many segments, consider it too + * computationally expensive to check all (2^num_segments) possible + * combinations. Instead, try at most 2^_segments_max randomly-selected + * combinations. + * + * This is reasonable if only a few segment copies are damaged and the + * majority of segment copies are good. This allows all the segment copies to + * participate fairly in the reconstruction and prevents the repeated use of + * one bad copy. + */ +int zfs_reconstruct_indirect_segments_max = 10; + +/* + * The indirect_child_t represents the vdev that we will read from, when we + * need to read all copies of the data (e.g. for scrub or reconstruction). + * For plain (non-mirror) top-level vdevs (i.e. is_vdev is not a mirror), + * ic_vdev is the same as is_vdev. However, for mirror top-level vdevs, + * ic_vdev is a child of the mirror. + */ +typedef struct indirect_child { + abd_t *ic_data; + vdev_t *ic_vdev; +} indirect_child_t; + +/* + * The indirect_split_t represents one mapped segment of an i/o to the + * indirect vdev. For non-split (contiguously-mapped) blocks, there will be + * only one indirect_split_t, with is_split_offset==0 and is_size==io_size. + * For split blocks, there will be several of these. + */ +typedef struct indirect_split { + list_node_t is_node; /* link on iv_splits */ + + /* + * is_split_offset is the offset into the i/o. + * This is the sum of the previous splits' is_size's. + */ + uint64_t is_split_offset; + + vdev_t *is_vdev; /* top-level vdev */ + uint64_t is_target_offset; /* offset on is_vdev */ + uint64_t is_size; + int is_children; /* number of entries in is_child[] */ + + /* + * is_good_child is the child that we are currently using to + * attempt reconstruction. + */ + int is_good_child; + + indirect_child_t is_child[1]; /* variable-length */ +} indirect_split_t; + +/* + * The indirect_vsd_t is associated with each i/o to the indirect vdev. + * It is the "Vdev-Specific Data" in the zio_t's io_vsd. + */ +typedef struct indirect_vsd { + boolean_t iv_split_block; + boolean_t iv_reconstruct; + + list_t iv_splits; /* list of indirect_split_t's */ +} indirect_vsd_t; + +static void +vdev_indirect_map_free(zio_t *zio) +{ + indirect_vsd_t *iv = zio->io_vsd; + + indirect_split_t *is; + while ((is = list_head(&iv->iv_splits)) != NULL) { + for (int c = 0; c < is->is_children; c++) { + indirect_child_t *ic = &is->is_child[c]; + if (ic->ic_data != NULL) + abd_free(ic->ic_data); + } + list_remove(&iv->iv_splits, is); + kmem_free(is, + offsetof(indirect_split_t, is_child[is->is_children])); + } + kmem_free(iv, sizeof (*iv)); +} + +static const zio_vsd_ops_t vdev_indirect_vsd_ops = { + vdev_indirect_map_free, + zio_vsd_default_cksum_report +}; +/* * Mark the given offset and size as being obsolete. */ void @@ -817,12 +907,6 @@ vdev_indirect_close(vdev_t *vd) } /* ARGSUSED */ -static void -vdev_indirect_io_done(zio_t *zio) -{ -} - -/* ARGSUSED */ static int vdev_indirect_open(vdev_t *vd, uint64_t *psize, uint64_t *max_psize, uint64_t *ashift) @@ -1065,39 +1149,473 @@ vdev_indirect_child_io_done(zio_t *zio) abd_put(zio->io_abd); } +/* + * This is a callback for vdev_indirect_remap() which allocates an + * indirect_split_t for each split segment and adds it to iv_splits. + */ static void -vdev_indirect_io_start_cb(uint64_t split_offset, vdev_t *vd, uint64_t offset, +vdev_indirect_gather_splits(uint64_t split_offset, vdev_t *vd, uint64_t offset, uint64_t size, void *arg) { zio_t *zio = arg; + indirect_vsd_t *iv = zio->io_vsd; ASSERT3P(vd, !=, NULL); if (vd->vdev_ops == &vdev_indirect_ops) return; - zio_nowait(zio_vdev_child_io(zio, NULL, vd, offset, - abd_get_offset(zio->io_abd, split_offset), - size, zio->io_type, zio->io_priority, - 0, vdev_indirect_child_io_done, zio)); + int n = 1; + if (vd->vdev_ops == &vdev_mirror_ops) + n = vd->vdev_children; + + indirect_split_t *is = + kmem_zalloc(offsetof(indirect_split_t, is_child[n]), KM_SLEEP); + + is->is_children = n; + is->is_size = size; + is->is_split_offset = split_offset; + is->is_target_offset = offset; + is->is_vdev = vd; + + /* + * Note that we only consider multiple copies of the data for + * *mirror* vdevs. We don't for "replacing" or "spare" vdevs, even + * though they use the same ops as mirror, because there's only one + * "good" copy under the replacing/spare. + */ + if (vd->vdev_ops == &vdev_mirror_ops) { + for (int i = 0; i < n; i++) { + is->is_child[i].ic_vdev = vd->vdev_child[i]; + } + } else { + is->is_child[0].ic_vdev = vd; + } + + list_insert_tail(&iv->iv_splits, is); } static void +vdev_indirect_read_split_done(zio_t *zio) +{ + indirect_child_t *ic = zio->io_private; + + if (zio->io_error != 0) { + /* + * Clear ic_data to indicate that we do not have data for this + * child. + */ + abd_free(ic->ic_data); + ic->ic_data = NULL; + } +} + +/* + * Issue reads for all copies (mirror children) of all splits. + */ +static void +vdev_indirect_read_all(zio_t *zio) +{ + indirect_vsd_t *iv = zio->io_vsd; + + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) { + for (int i = 0; i < is->is_children; i++) { + indirect_child_t *ic = &is->is_child[i]; + + if (!vdev_readable(ic->ic_vdev)) + continue; + + /* + * Note, we may read from a child whose DTL + * indicates that the data may not be present here. + * While this might result in a few i/os that will + * likely return incorrect data, it simplifies the + * code since we can treat scrub and resilver + * identically. (The incorrect data will be + * detected and ignored when we verify the + * checksum.) + */ + + ic->ic_data = abd_alloc_sametype(zio->io_abd, + is->is_size); + + zio_nowait(zio_vdev_child_io(zio, NULL, + ic->ic_vdev, is->is_target_offset, ic->ic_data, + is->is_size, zio->io_type, zio->io_priority, 0, + vdev_indirect_read_split_done, ic)); + } + } + iv->iv_reconstruct = B_TRUE; +} + +static void vdev_indirect_io_start(zio_t *zio) { spa_t *spa = zio->io_spa; + indirect_vsd_t *iv = kmem_zalloc(sizeof (*iv), KM_SLEEP); + list_create(&iv->iv_splits, + sizeof (indirect_split_t), offsetof(indirect_split_t, is_node)); + zio->io_vsd = iv; + zio->io_vsd_ops = &vdev_indirect_vsd_ops; + ASSERT(spa_config_held(spa, SCL_ALL, RW_READER) != 0); if (zio->io_type != ZIO_TYPE_READ) { ASSERT3U(zio->io_type, ==, ZIO_TYPE_WRITE); - ASSERT((zio->io_flags & - (ZIO_FLAG_SELF_HEAL | ZIO_FLAG_INDUCE_DAMAGE)) != 0); + /* + * Note: this code can handle other kinds of writes, + * but we don't expect them. + */ + ASSERT((zio->io_flags & (ZIO_FLAG_SELF_HEAL | + ZIO_FLAG_RESILVER | ZIO_FLAG_INDUCE_DAMAGE)) != 0); } vdev_indirect_remap(zio->io_vd, zio->io_offset, zio->io_size, - vdev_indirect_io_start_cb, zio); + vdev_indirect_gather_splits, zio); + indirect_split_t *first = list_head(&iv->iv_splits); + if (first->is_size == zio->io_size) { + /* + * This is not a split block; we are pointing to the entire + * data, which will checksum the same as the original data. + * Pass the BP down so that the child i/o can verify the + * checksum, and try a different location if available + * (e.g. on a mirror). + * + * While this special case could be handled the same as the + * general (split block) case, doing it this way ensures + * that the vast majority of blocks on indirect vdevs + * (which are not split) are handled identically to blocks + * on non-indirect vdevs. This allows us to be less strict + * about performance in the general (but rare) case. + */ + ASSERT0(first->is_split_offset); + ASSERT3P(list_next(&iv->iv_splits, first), ==, NULL); + zio_nowait(zio_vdev_child_io(zio, zio->io_bp, + first->is_vdev, first->is_target_offset, + abd_get_offset(zio->io_abd, 0), + zio->io_size, zio->io_type, zio->io_priority, 0, + vdev_indirect_child_io_done, zio)); + } else { + iv->iv_split_block = B_TRUE; + if (zio->io_flags & (ZIO_FLAG_SCRUB | ZIO_FLAG_RESILVER)) { + /* + * Read all copies. Note that for simplicity, + * we don't bother consulting the DTL in the + * resilver case. + */ + vdev_indirect_read_all(zio); + } else { + /* + * Read one copy of each split segment, from the + * top-level vdev. Since we don't know the + * checksum of each split individually, the child + * zio can't ensure that we get the right data. + * E.g. if it's a mirror, it will just read from a + * random (healthy) leaf vdev. We have to verify + * the checksum in vdev_indirect_io_done(). + */ + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) { + zio_nowait(zio_vdev_child_io(zio, NULL, + is->is_vdev, is->is_target_offset, + abd_get_offset(zio->io_abd, + is->is_split_offset), + is->is_size, zio->io_type, + zio->io_priority, 0, + vdev_indirect_child_io_done, zio)); + } + } + } + zio_execute(zio); +} + +/* + * Report a checksum error for a child. + */ +static void +vdev_indirect_checksum_error(zio_t *zio, + indirect_split_t *is, indirect_child_t *ic) +{ + vdev_t *vd = ic->ic_vdev; + + if (zio->io_flags & ZIO_FLAG_SPECULATIVE) + return; + + mutex_enter(&vd->vdev_stat_lock); + vd->vdev_stat.vs_checksum_errors++; + mutex_exit(&vd->vdev_stat_lock); + + zio_bad_cksum_t zbc = { 0 }; + void *bad_buf = abd_borrow_buf_copy(ic->ic_data, is->is_size); + abd_t *good_abd = is->is_child[is->is_good_child].ic_data; + void *good_buf = abd_borrow_buf_copy(good_abd, is->is_size); + zfs_ereport_post_checksum(zio->io_spa, vd, zio, + is->is_target_offset, is->is_size, good_buf, bad_buf, &zbc); + abd_return_buf(ic->ic_data, bad_buf, is->is_size); + abd_return_buf(good_abd, good_buf, is->is_size); +} + +/* + * Issue repair i/os for any incorrect copies. We do this by comparing + * each split segment's correct data (is_good_child's ic_data) with each + * other copy of the data. If they differ, then we overwrite the bad data + * with the good copy. Note that we do this without regard for the DTL's, + * which simplifies this code and also issues the optimal number of writes + * (based on which copies actually read bad data, as opposed to which we + * think might be wrong). For the same reason, we always use + * ZIO_FLAG_SELF_HEAL, to bypass the DTL check in zio_vdev_io_start(). + */ +static void +vdev_indirect_repair(zio_t *zio) +{ + indirect_vsd_t *iv = zio->io_vsd; + + enum zio_flag flags = ZIO_FLAG_IO_REPAIR; + + if (!(zio->io_flags & (ZIO_FLAG_SCRUB | ZIO_FLAG_RESILVER))) + flags |= ZIO_FLAG_SELF_HEAL; + + if (!spa_writeable(zio->io_spa)) + return; + + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) { + indirect_child_t *good_child = &is->is_child[is->is_good_child]; + + for (int c = 0; c < is->is_children; c++) { + indirect_child_t *ic = &is->is_child[c]; + if (ic == good_child) + continue; + if (ic->ic_data == NULL) + continue; + if (abd_cmp(good_child->ic_data, ic->ic_data, + is->is_size) == 0) + continue; + + zio_nowait(zio_vdev_child_io(zio, NULL, + ic->ic_vdev, is->is_target_offset, + good_child->ic_data, is->is_size, + ZIO_TYPE_WRITE, ZIO_PRIORITY_ASYNC_WRITE, + ZIO_FLAG_IO_REPAIR | ZIO_FLAG_SELF_HEAL, + NULL, NULL)); + + vdev_indirect_checksum_error(zio, is, ic); + } + } +} + +/* + * Report checksum errors on all children that we read from. + */ +static void +vdev_indirect_all_checksum_errors(zio_t *zio) +{ + indirect_vsd_t *iv = zio->io_vsd; + + if (zio->io_flags & ZIO_FLAG_SPECULATIVE) + return; + + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) { + for (int c = 0; c < is->is_children; c++) { + indirect_child_t *ic = &is->is_child[c]; + + if (ic->ic_data == NULL) + continue; + + vdev_t *vd = ic->ic_vdev; + + mutex_enter(&vd->vdev_stat_lock); + vd->vdev_stat.vs_checksum_errors++; + mutex_exit(&vd->vdev_stat_lock); + + zfs_ereport_post_checksum(zio->io_spa, vd, zio, + is->is_target_offset, is->is_size, + NULL, NULL, NULL); + } + } +} + +/* + * This function is called when we have read all copies of the data and need + * to try to find a combination of copies that gives us the right checksum. + * + * If we pointed to any mirror vdevs, this effectively does the job of the + * mirror. The mirror vdev code can't do its own job because we don't know + * the checksum of each split segment individually. We have to try every + * combination of copies of split segments, until we find one that checksums + * correctly. (Or until we have tried all combinations, or have tried + * 2^zfs_reconstruct_indirect_segments_max combinations. In these cases we + * set io_error to ECKSUM to propagate the error up to the user.) + * + * For example, if we have 3 segments in the split, + * and each points to a 2-way mirror, we will have the following pieces of + * data: + * + * | mirror child + * split | [0] [1] + * ======|===================== + * A | data_A_0 data_A_1 + * B | data_B_0 data_B_1 + * C | data_C_0 data_C_1 + * + * We will try the following (mirror children)^(number of splits) (2^3=8) + * combinations, which is similar to bitwise-little-endian counting in + * binary. In general each "digit" corresponds to a split segment, and the + * base of each digit is is_children, which can be different for each + * digit. + * + * "low bit" "high bit" + * v v + * data_A_0 data_B_0 data_C_0 + * data_A_1 data_B_0 data_C_0 + * data_A_0 data_B_1 data_C_0 + * data_A_1 data_B_1 data_C_0 + * data_A_0 data_B_0 data_C_1 + * data_A_1 data_B_0 data_C_1 + * data_A_0 data_B_1 data_C_1 + * data_A_1 data_B_1 data_C_1 + * + * Note that the split segments may be on the same or different top-level + * vdevs. In either case, we try lots of combinations (see + * zfs_reconstruct_indirect_segments_max). This ensures that if a mirror has + * small silent errors on all of its children, we can still reconstruct the + * correct data, as long as those errors are at sufficiently-separated + * offsets (specifically, separated by the largest block size - default of + * 128KB, but up to 16MB). + */ +static void +vdev_indirect_reconstruct_io_done(zio_t *zio) +{ + indirect_vsd_t *iv = zio->io_vsd; + uint64_t attempts = 0; + uint64_t attempts_max = 1ULL << zfs_reconstruct_indirect_segments_max; + int segments = 0; + + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) + segments++; + + for (;;) { + /* copy data from splits to main zio */ + int ret; + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) { + + /* + * If this child failed, its ic_data will be NULL. + * Skip this combination. + */ + if (is->is_child[is->is_good_child].ic_data == NULL) { + ret = EIO; + goto next; + } + + abd_copy_off(zio->io_abd, + is->is_child[is->is_good_child].ic_data, + is->is_split_offset, 0, is->is_size); + } + + /* See if this checksum matches. */ + zio_bad_cksum_t zbc; + ret = zio_checksum_error(zio, &zbc); + if (ret == 0) { + /* Found a matching checksum. Issue repair i/os. */ + vdev_indirect_repair(zio); + zio_checksum_verified(zio); + return; + } + + /* + * Checksum failed; try a different combination of split + * children. + */ + boolean_t more; +next: + more = B_FALSE; + if (segments <= zfs_reconstruct_indirect_segments_max) { + /* + * There are relatively few segments, so + * deterministically check all combinations. We do + * this by by adding one to the first split's + * good_child. If it overflows, then "carry over" to + * the next split (like counting in base is_children, + * but each digit can have a different base). + */ + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) { + is->is_good_child++; + if (is->is_good_child < is->is_children) { + more = B_TRUE; + break; + } + is->is_good_child = 0; + } + } else if (++attempts < attempts_max) { + /* + * There are too many combinations to try all of them + * in a reasonable amount of time, so try a fixed + * number of random combinations, after which we'll + * consider the block unrecoverable. + */ + for (indirect_split_t *is = list_head(&iv->iv_splits); + is != NULL; is = list_next(&iv->iv_splits, is)) { + is->is_good_child = + spa_get_random(is->is_children); + } + more = B_TRUE; + } + if (!more) { + /* All combinations failed. */ + zio->io_error = ret; + vdev_indirect_all_checksum_errors(zio); + zio_checksum_verified(zio); + return; + } + } +} + +static void +vdev_indirect_io_done(zio_t *zio) +{ + indirect_vsd_t *iv = zio->io_vsd; + + if (iv->iv_reconstruct) { + /* + * We have read all copies of the data (e.g. from mirrors), + * either because this was a scrub/resilver, or because the + * one-copy read didn't checksum correctly. + */ + vdev_indirect_reconstruct_io_done(zio); + return; + } + + if (!iv->iv_split_block) { + /* + * This was not a split block, so we passed the BP down, + * and the checksum was handled by the (one) child zio. + */ + return; + } + + zio_bad_cksum_t zbc; + int ret = zio_checksum_error(zio, &zbc); + if (ret == 0) { + zio_checksum_verified(zio); + return; + } + + /* + * The checksum didn't match. Read all copies of all splits, and + * then we will try to reconstruct. The next time + * vdev_indirect_io_done() is called, iv_reconstruct will be set. + */ + vdev_indirect_read_all(zio); + + zio_vdev_io_redone(zio); } vdev_ops_t vdev_indirect_ops = { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c Tue Jul 31 00:13:04 2018 (r336950) @@ -24,7 +24,7 @@ */ /* - * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. */ #include @@ -345,12 +345,15 @@ vdev_mirror_io_start(zio_t *zio) } if (zio->io_type == ZIO_TYPE_READ) { - if ((zio->io_flags & ZIO_FLAG_SCRUB) && !mm->mm_resilvering) { + if (zio->io_bp != NULL && + (zio->io_flags & ZIO_FLAG_SCRUB) && !mm->mm_resilvering) { /* - * For scrubbing reads we need to allocate a read - * buffer for each child and issue reads to all - * children. If any child succeeds, it will copy its - * data into zio->io_data in vdev_mirror_scrub_done. + * For scrubbing reads (if we can verify the + * checksum here, as indicated by io_bp being + * non-NULL) we need to allocate a read buffer for + * each child and issue reads to all children. If + * any child succeeds, it will copy its data into + * zio->io_data in vdev_mirror_scrub_done. */ for (c = 0; c < mm->mm_children; c++) { mc = &mm->mm_child[c]; @@ -502,7 +505,21 @@ vdev_mirror_io_done(zio_t *zio) if (mc->mc_error == 0) { if (mc->mc_tried) continue; + /* + * We didn't try this child. We need to + * repair it if: + * 1. it's a scrub (in which case we have + * tried everything that was healthy) + * - or - + * 2. it's an indirect vdev (in which case + * it could point to any other vdev, which + * might have a bad DTL) + * - or - + * 3. the DTL indicates that this data is + * missing from this vdev + */ if (!(zio->io_flags & ZIO_FLAG_SCRUB) && + mc->mc_vd->vdev_ops != &vdev_indirect_ops && !vdev_dtl_contains(mc->mc_vd, DTL_PARTIAL, zio->io_txg, 1)) continue; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Tue Jul 31 00:13:04 2018 (r336950) @@ -83,18 +83,12 @@ typedef struct vdev_copy_arg { kmutex_t vca_lock; } vdev_copy_arg_t; -typedef struct vdev_copy_seg_arg { - vdev_copy_arg_t *vcsa_copy_arg; - uint64_t vcsa_txg; - dva_t *vcsa_dest_dva; - blkptr_t *vcsa_dest_bp; -} vdev_copy_seg_arg_t; - /* - * The maximum amount of allowed data we're allowed to copy from a device - * at a time when removing it. + * The maximum amount of memory we can use for outstanding i/o while + * doing a device removal. This determines how much i/o we can have + * in flight concurrently. */ -int zfs_remove_max_copy_bytes = 8 * 1024 * 1024; +int zfs_remove_max_copy_bytes = 64 * 1024 * 1024; /* * The largest contiguous segment that we will attempt to allocate when @@ -176,7 +170,7 @@ spa_vdev_removal_create(vdev_t *vd) mutex_init(&svr->svr_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&svr->svr_cv, NULL, CV_DEFAULT, NULL); svr->svr_allocd_segs = range_tree_create(NULL, NULL); - svr->svr_vdev = vd; + svr->svr_vdev_id = vd->vdev_id; for (int i = 0; i < TXG_SIZE; i++) { svr->svr_frees[i] = range_tree_create(NULL, NULL); @@ -218,9 +212,10 @@ spa_vdev_removal_destroy(spa_vdev_removal_t *svr) static void vdev_remove_initiate_sync(void *arg, dmu_tx_t *tx) { - vdev_t *vd = arg; + int vdev_id = (uintptr_t)arg; + spa_t *spa = dmu_tx_pool(tx)->dp_spa; + vdev_t *vd = vdev_lookup_top(spa, vdev_id); vdev_indirect_config_t *vic = &vd->vdev_indirect_config; - spa_t *spa = vd->vdev_spa; objset_t *mos = spa->spa_dsl_pool->dp_meta_objset; spa_vdev_removal_t *svr = NULL; uint64_t txg = dmu_tx_get_txg(tx); @@ -342,7 +337,7 @@ vdev_remove_initiate_sync(void *arg, dmu_tx_t *tx) ASSERT3P(spa->spa_vdev_removal, ==, NULL); spa->spa_vdev_removal = svr; svr->svr_thread = thread_create(NULL, 0, - spa_vdev_remove_thread, vd, 0, &p0, TS_RUN, minclsyspri); + spa_vdev_remove_thread, spa, 0, &p0, TS_RUN, minclsyspri); } /* @@ -383,21 +378,24 @@ spa_remove_init(spa_t *spa) spa_config_enter(spa, SCL_STATE, FTAG, RW_READER); vdev_t *vd = vdev_lookup_top(spa, spa->spa_removing_phys.sr_removing_vdev); - spa_config_exit(spa, SCL_STATE, FTAG); - if (vd == NULL) + if (vd == NULL) { + spa_config_exit(spa, SCL_STATE, FTAG); return (EINVAL); + } vdev_indirect_config_t *vic = &vd->vdev_indirect_config; ASSERT(vdev_is_concrete(vd)); spa_vdev_removal_t *svr = spa_vdev_removal_create(vd); - ASSERT(svr->svr_vdev->vdev_removing); + ASSERT3U(svr->svr_vdev_id, ==, vd->vdev_id); + ASSERT(vd->vdev_removing); vd->vdev_indirect_mapping = vdev_indirect_mapping_open( spa->spa_meta_objset, vic->vic_mapping_object); vd->vdev_indirect_births = vdev_indirect_births_open( spa->spa_meta_objset, vic->vic_births_object); + spa_config_exit(spa, SCL_STATE, FTAG); spa->spa_vdev_removal = svr; } @@ -450,15 +448,8 @@ spa_restart_removal(spa_t *spa) if (!spa_writeable(spa)) return; - vdev_t *vd = svr->svr_vdev; - vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; - - ASSERT3P(vd, !=, NULL); - ASSERT(vd->vdev_removing); - - zfs_dbgmsg("restarting removal of %llu at count=%llu", - vd->vdev_id, vdev_indirect_mapping_num_entries(vim)); - svr->svr_thread = thread_create(NULL, 0, spa_vdev_remove_thread, vd, + zfs_dbgmsg("restarting removal of %llu", svr->svr_vdev_id); + svr->svr_thread = thread_create(NULL, 0, spa_vdev_remove_thread, spa, 0, &p0, TS_RUN, minclsyspri); } @@ -479,7 +470,7 @@ free_from_removing_vdev(vdev_t *vd, uint64_t offset, u ASSERT(vd->vdev_indirect_config.vic_mapping_object != 0); ASSERT3U(vd->vdev_indirect_config.vic_mapping_object, ==, vdev_indirect_mapping_object(vim)); - ASSERT3P(vd, ==, svr->svr_vdev); + ASSERT3U(vd->vdev_id, ==, svr->svr_vdev_id); mutex_enter(&svr->svr_lock); @@ -662,7 +653,7 @@ spa_finish_removal(spa_t *spa, dsl_scan_state_t state, if (state == DSS_FINISHED) { spa_removing_phys_t *srp = &spa->spa_removing_phys; - vdev_t *vd = svr->svr_vdev; + vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id); vdev_indirect_config_t *vic = &vd->vdev_indirect_config; if (srp->sr_prev_indirect_vdev != UINT64_MAX) { @@ -705,7 +696,7 @@ vdev_mapping_sync(void *arg, dmu_tx_t *tx) { spa_vdev_removal_t *svr = arg; spa_t *spa = dmu_tx_pool(tx)->dp_spa; - vdev_t *vd = svr->svr_vdev; + vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id); vdev_indirect_config_t *vic = &vd->vdev_indirect_config; uint64_t txg = dmu_tx_get_txg(tx); vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; @@ -733,64 +724,128 @@ vdev_mapping_sync(void *arg, dmu_tx_t *tx) spa_sync_removing_state(spa, tx); } +/* + * All reads and writes associated with a call to spa_vdev_copy_segment() + * are done. + */ static void +spa_vdev_copy_nullzio_done(zio_t *zio) +{ + spa_config_exit(zio->io_spa, SCL_STATE, zio->io_spa); +} + +/* + * The write of the new location is done. + */ +static void spa_vdev_copy_segment_write_done(zio_t *zio) { - vdev_copy_seg_arg_t *vcsa = zio->io_private; - vdev_copy_arg_t *vca = vcsa->vcsa_copy_arg; - spa_config_exit(zio->io_spa, SCL_STATE, FTAG); + vdev_copy_arg_t *vca = zio->io_private; + abd_free(zio->io_abd); mutex_enter(&vca->vca_lock); vca->vca_outstanding_bytes -= zio->io_size; cv_signal(&vca->vca_cv); mutex_exit(&vca->vca_lock); - - ASSERT0(zio->io_error); - kmem_free(vcsa->vcsa_dest_bp, sizeof (blkptr_t)); - kmem_free(vcsa, sizeof (vdev_copy_seg_arg_t)); } +/* + * The read of the old location is done. The parent zio is the write to + * the new location. Allow it to start. + */ static void spa_vdev_copy_segment_read_done(zio_t *zio) { - vdev_copy_seg_arg_t *vcsa = zio->io_private; - dva_t *dest_dva = vcsa->vcsa_dest_dva; - uint64_t txg = vcsa->vcsa_txg; - spa_t *spa = zio->io_spa; - vdev_t *dest_vd = vdev_lookup_top(spa, DVA_GET_VDEV(dest_dva)); - blkptr_t *bp = NULL; - dva_t *dva = NULL; - uint64_t size = zio->io_size; + zio_nowait(zio_unique_parent(zio)); +} - ASSERT3P(dest_vd, !=, NULL); - ASSERT0(zio->io_error); +/* + * If the old and new vdevs are mirrors, we will read both sides of the old + * mirror, and write each copy to the corresponding side of the new mirror. + * If the old and new vdevs have a different number of children, we will do + * this as best as possible. Since we aren't verifying checksums, this + * ensures that as long as there's a good copy of the data, we'll have a + * good copy after the removal, even if there's silent damage to one side + * of the mirror. If we're removing a mirror that has some silent damage, + * we'll have exactly the same damage in the new location (assuming that + * the new location is also a mirror). + * + * We accomplish this by creating a tree of zio_t's, with as many writes as + * there are "children" of the new vdev (a non-redundant vdev counts as one + * child, a 2-way mirror has 2 children, etc). Each write has an associated + * read from a child of the old vdev. Typically there will be the same + * number of children of the old and new vdevs. However, if there are more + * children of the new vdev, some child(ren) of the old vdev will be issued *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Tue Jul 31 00:13:08 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D5511065B7D; Tue, 31 Jul 2018 00:13:08 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D53878E4AB; Tue, 31 Jul 2018 00:13:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B72AE4059; Tue, 31 Jul 2018 00:13:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6V0D7Uk088747; Tue, 31 Jul 2018 00:13:07 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6V0D7dV088745; Tue, 31 Jul 2018 00:13:07 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807310013.w6V0D7dV088745@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 00:13:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336950 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist/lib/libzf... X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zdb vendor/illumos/dist/cmd/ztest vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 336950 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 00:13:08 -0000 Author: mav Date: Tue Jul 31 00:13:04 2018 New Revision: 336950 URL: https://svnweb.freebsd.org/changeset/base/336950 Log: 9290 device removal reduces redundancy of mirrors Mirrors are supposed to provide redundancy in the face of whole-disk failure and silent damage (e.g. some data on disk is not right, but ZFS hasn't detected the whole device as being broken). However, the current device removal implementation bypasses some of the mirror's redundancy. illumos/illumos-gate@3a4b1be953ee5601bab748afa07c26ed4996cde6 Reviewed by: George Wilson Reviewed by: Prashanth Sreenivasa Reviewed by: Sara Hartse Reviewed by: Serapheim Dimitropoulos Reviewed by: Brian Behlendorf Reviewed by: Tim Chase Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor/illumos/dist/cmd/zdb/zdb.c Tue Jul 31 00:13:04 2018 (r336950) @@ -3004,7 +3004,7 @@ zdb_claim_removing(spa_t *spa, zdb_cb_t *zcb) spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); spa_vdev_removal_t *svr = spa->spa_vdev_removal; - vdev_t *vd = svr->svr_vdev; + vdev_t *vd = vdev_lookup_top(spa, svr->svr_vdev_id); vdev_indirect_mapping_t *vim = vd->vdev_indirect_mapping; for (uint64_t msi = 0; msi < vd->vdev_ms_count; msi++) { @@ -3020,13 +3020,17 @@ zdb_claim_removing(spa_t *spa, zdb_cb_t *zcb) svr->svr_allocd_segs, SM_ALLOC)); /* - * Clear everything past what has been synced, - * because we have not allocated mappings for it yet. + * Clear everything past what has been synced unless + * it's past the spacemap, because we have not allocated + * mappings for it yet. */ - range_tree_clear(svr->svr_allocd_segs, - vdev_indirect_mapping_max_offset(vim), - msp->ms_sm->sm_start + msp->ms_sm->sm_size - - vdev_indirect_mapping_max_offset(vim)); + uint64_t vim_max_offset = + vdev_indirect_mapping_max_offset(vim); + uint64_t sm_end = msp->ms_sm->sm_start + + msp->ms_sm->sm_size; + if (sm_end > vim_max_offset) + range_tree_clear(svr->svr_allocd_segs, + vim_max_offset, sm_end - vim_max_offset); } zcb->zcb_removing_size += Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor/illumos/dist/cmd/ztest/ztest.c Tue Jul 31 00:13:04 2018 (r336950) @@ -436,6 +436,7 @@ static ztest_ds_t *ztest_ds; static kmutex_t ztest_vdev_lock; static kmutex_t ztest_checkpoint_lock; +static boolean_t ztest_device_removal_active = B_FALSE; /* * The ztest_name_lock protects the pool and dataset namespace used by @@ -2880,7 +2881,7 @@ ztest_vdev_attach_detach(ztest_ds_t *zd, uint64_t id) * value. Don't bother trying to attach while we are in the middle * of removal. */ - if (spa->spa_vdev_removal != NULL) { + if (ztest_device_removal_active) { spa_config_exit(spa, SCL_ALL, FTAG); mutex_exit(&ztest_vdev_lock); return; @@ -3055,16 +3056,49 @@ ztest_device_removal(ztest_ds_t *zd, uint64_t id) spa_t *spa = ztest_spa; vdev_t *vd; uint64_t guid; + int error; mutex_enter(&ztest_vdev_lock); + if (ztest_device_removal_active) { + mutex_exit(&ztest_vdev_lock); + return; + } + + /* + * Remove a random top-level vdev and wait for removal to finish. + */ spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); vd = vdev_lookup_top(spa, ztest_random_vdev_top(spa, B_FALSE)); guid = vd->vdev_guid; spa_config_exit(spa, SCL_VDEV, FTAG); - (void) spa_vdev_remove(spa, guid, B_FALSE); + error = spa_vdev_remove(spa, guid, B_FALSE); + if (error == 0) { + ztest_device_removal_active = B_TRUE; + mutex_exit(&ztest_vdev_lock); + while (spa->spa_vdev_removal != NULL) + txg_wait_synced(spa_get_dsl(spa), 0); + } else { + mutex_exit(&ztest_vdev_lock); + return; + } + + /* + * The pool needs to be scrubbed after completing device removal. + * Failure to do so may result in checksum errors due to the + * strategy employed by ztest_fault_inject() when selecting which + * offset are redundant and can be damaged. + */ + error = spa_scan(spa, POOL_SCAN_SCRUB); + if (error == 0) { + while (dsl_scan_scrubbing(spa_get_dsl(spa))) + txg_wait_synced(spa_get_dsl(spa), 0); + } + + mutex_enter(&ztest_vdev_lock); + ztest_device_removal_active = B_FALSE; mutex_exit(&ztest_vdev_lock); } @@ -3203,7 +3237,7 @@ ztest_vdev_LUN_growth(ztest_ds_t *zd, uint64_t id) * that the metaslab_class space increased (because it decreases * when the device removal completes). */ - if (spa->spa_vdev_removal != NULL) { + if (ztest_device_removal_active) { spa_config_exit(spa, SCL_STATE, spa); mutex_exit(&ztest_vdev_lock); mutex_exit(&ztest_checkpoint_lock); @@ -4985,6 +5019,18 @@ ztest_fault_inject(ztest_ds_t *zd, uint64_t id) boolean_t islog = B_FALSE; mutex_enter(&ztest_vdev_lock); + + /* + * Device removal is in progress, fault injection must be disabled + * until it completes and the pool is scrubbed. The fault injection + * strategy for damaging blocks does not take in to account evacuated + * blocks which may have already been damaged. + */ + if (ztest_device_removal_active) { + mutex_exit(&ztest_vdev_lock); + return; + } + maxfaults = MAXFAULTS(); leaves = MAX(zs->zs_mirrors, 1) * ztest_opts.zo_raidz; mirror_save = zs->zs_mirrors; @@ -5329,6 +5375,12 @@ void ztest_scrub(ztest_ds_t *zd, uint64_t id) { spa_t *spa = ztest_spa; + + /* + * Scrub in progress by device removal. + */ + if (ztest_device_removal_active) + return; (void) spa_scan(spa, POOL_SCAN_SCRUB); (void) poll(NULL, 0, 100); /* wait a moment, then force a restart */ Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Tue Jul 31 00:02:42 2018 (r336949) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Tue Jul 31 00:13:04 2018 (r336950) @@ -2807,7 +2807,7 @@ zpool_vdev_attach(zpool_handle_t *zhp, case EBUSY: zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "%s is busy, " - "or pool has removing/removed vdevs"), + "or device removal is in progress"), new_disk); (void) zfs_error(hdl, EZFS_BADDEV, msg); break; From owner-svn-src-vendor@freebsd.org Tue Jul 31 00:34:40 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E82D310662BB; Tue, 31 Jul 2018 00:34:39 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B3218F08C; Tue, 31 Jul 2018 00:34:39 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 77C3043D9; Tue, 31 Jul 2018 00:34:39 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6V0YdTE099743; Tue, 31 Jul 2018 00:34:39 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6V0YdAp099742; Tue, 31 Jul 2018 00:34:39 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807310034.w6V0YdAp099742@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 00:34:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336952 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 336952 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 00:34:40 -0000 Author: mav Date: Tue Jul 31 00:34:39 2018 New Revision: 336952 URL: https://svnweb.freebsd.org/changeset/base/336952 Log: 9192 explicitly pass good_writes to vdev_uberblock/label_sync Currently vdev_label_sync and vdev_uberblock_sync take a zio_t and assume that its io_private is a pointer to the good_writes count. They should instead accept this argument explicitly. illumos/illumos-gate@a3b5583021b7b45676bf1f0cc68adf7a97900b56 Reviewed by: Pavel Zakharov Reviewed by: George Wilson Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Tue Jul 31 00:25:39 2018 (r336951) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Tue Jul 31 00:34:39 2018 (r336952) @@ -1117,10 +1117,13 @@ vdev_uberblock_sync_done(zio_t *zio) * Write the uberblock to all labels of all leaves of the specified vdev. */ static void -vdev_uberblock_sync(zio_t *zio, uberblock_t *ub, vdev_t *vd, int flags) +vdev_uberblock_sync(zio_t *zio, uint64_t *good_writes, + uberblock_t *ub, vdev_t *vd, int flags) { - for (uint64_t c = 0; c < vd->vdev_children; c++) - vdev_uberblock_sync(zio, ub, vd->vdev_child[c], flags); + for (uint64_t c = 0; c < vd->vdev_children; c++) { + vdev_uberblock_sync(zio, good_writes, + ub, vd->vdev_child[c], flags); + } if (!vd->vdev_ops->vdev_op_leaf) return; @@ -1138,7 +1141,7 @@ vdev_uberblock_sync(zio_t *zio, uberblock_t *ub, vdev_ for (int l = 0; l < VDEV_LABELS; l++) vdev_label_write(zio, vd, l, ub_abd, VDEV_UBERBLOCK_OFFSET(vd, n), VDEV_UBERBLOCK_SIZE(vd), - vdev_uberblock_sync_done, zio->io_private, + vdev_uberblock_sync_done, good_writes, flags | ZIO_FLAG_DONT_PROPAGATE); abd_free(ub_abd); @@ -1152,10 +1155,10 @@ vdev_uberblock_sync_list(vdev_t **svd, int svdcount, u zio_t *zio; uint64_t good_writes = 0; - zio = zio_root(spa, NULL, &good_writes, flags); + zio = zio_root(spa, NULL, NULL, flags); for (int v = 0; v < svdcount; v++) - vdev_uberblock_sync(zio, ub, svd[v], flags); + vdev_uberblock_sync(zio, &good_writes, ub, svd[v], flags); (void) zio_wait(zio); @@ -1216,7 +1219,8 @@ vdev_label_sync_ignore_done(zio_t *zio) * Write all even or odd labels to all leaves of the specified vdev. */ static void -vdev_label_sync(zio_t *zio, vdev_t *vd, int l, uint64_t txg, int flags) +vdev_label_sync(zio_t *zio, uint64_t *good_writes, + vdev_t *vd, int l, uint64_t txg, int flags) { nvlist_t *label; vdev_phys_t *vp; @@ -1224,8 +1228,10 @@ vdev_label_sync(zio_t *zio, vdev_t *vd, int l, uint64_ char *buf; size_t buflen; - for (int c = 0; c < vd->vdev_children; c++) - vdev_label_sync(zio, vd->vdev_child[c], l, txg, flags); + for (int c = 0; c < vd->vdev_children; c++) { + vdev_label_sync(zio, good_writes, + vd->vdev_child[c], l, txg, flags); + } if (!vd->vdev_ops->vdev_op_leaf) return; @@ -1250,7 +1256,7 @@ vdev_label_sync(zio_t *zio, vdev_t *vd, int l, uint64_ vdev_label_write(zio, vd, l, vp_abd, offsetof(vdev_label_t, vl_vdev_phys), sizeof (vdev_phys_t), - vdev_label_sync_done, zio->io_private, + vdev_label_sync_done, good_writes, flags | ZIO_FLAG_DONT_PROPAGATE); } } @@ -1282,7 +1288,7 @@ vdev_label_sync_list(spa_t *spa, int l, uint64_t txg, (vd->vdev_islog || vd->vdev_aux != NULL) ? vdev_label_sync_ignore_done : vdev_label_sync_top_done, good_writes, flags); - vdev_label_sync(vio, vd, l, txg, flags); + vdev_label_sync(vio, good_writes, vd, l, txg, flags); zio_nowait(vio); } From owner-svn-src-vendor@freebsd.org Tue Jul 31 00:42:34 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 595EB1066849; Tue, 31 Jul 2018 00:42:34 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0BE0E8FA7E; Tue, 31 Jul 2018 00:42:34 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C8C6E456E; Tue, 31 Jul 2018 00:42:33 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6V0gXaj004807; Tue, 31 Jul 2018 00:42:33 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6V0gXnf004806; Tue, 31 Jul 2018 00:42:33 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807310042.w6V0gXnf004806@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 00:42:33 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336955 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 336955 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 00:42:34 -0000 Author: mav Date: Tue Jul 31 00:42:31 2018 New Revision: 336955 URL: https://svnweb.freebsd.org/changeset/base/336955 Log: 9236 nuke spa_dbgmsg We should use zfs_dbgmsg instead of spa_dbgmsg. Or at least, metaslab_condense() should call zfs_dbgmsg because it's important and rare enough to always log. It's possible that the message in zio_dva_allocate() would be too high-frequency for zfs_dbgmsg. illumos/illumos-gate@21f7c81cc1156e9202ce3412d3ecaa697c3b2222 Reviewed by: Serapheim Dimitropoulos Reviewed by: Pavel Zakharov Reviewed by: George Wilson Reviewed by: Richard Elling Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor/illumos/dist/cmd/ztest/ztest.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_debug.h vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Tue Jul 31 00:37:45 2018 (r336954) +++ vendor/illumos/dist/cmd/ztest/ztest.c Tue Jul 31 00:42:31 2018 (r336955) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2016 by Delphix. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2013 Steven Hartland. All rights reserved. * Copyright (c) 2014 Integros [integros.com] @@ -5919,7 +5919,6 @@ ztest_run(ztest_shared_t *zs) */ kernel_init(FREAD | FWRITE); VERIFY0(spa_open(ztest_opts.zo_pool, &spa, FTAG)); - spa->spa_debug = B_TRUE; metaslab_preload_limit = ztest_random(20) + 1; ztest_spa = spa; @@ -6076,7 +6075,6 @@ ztest_freeze(void) kernel_init(FREAD | FWRITE); VERIFY3U(0, ==, spa_open(ztest_opts.zo_pool, &spa, FTAG)); VERIFY3U(0, ==, ztest_dataset_open(0)); - spa->spa_debug = B_TRUE; ztest_spa = spa; /* @@ -6147,7 +6145,6 @@ ztest_freeze(void) VERIFY3U(0, ==, ztest_dataset_open(0)); ztest_dataset_close(0); - spa->spa_debug = B_TRUE; ztest_spa = spa; txg_wait_synced(spa_get_dsl(spa), 0); ztest_reguid(NULL, 0); From owner-svn-src-vendor@freebsd.org Tue Jul 31 00:42:34 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DBCA31066846; Tue, 31 Jul 2018 00:42:33 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 912408FA7D; Tue, 31 Jul 2018 00:42:33 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 725FD456D; Tue, 31 Jul 2018 00:42:33 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6V0gX01004801; Tue, 31 Jul 2018 00:42:33 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6V0gW98004795; Tue, 31 Jul 2018 00:42:32 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807310042.w6V0gW98004795@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 00:42:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336955 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Commit-Revision: 336955 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 00:42:34 -0000 Author: mav Date: Tue Jul 31 00:42:31 2018 New Revision: 336955 URL: https://svnweb.freebsd.org/changeset/base/336955 Log: 9236 nuke spa_dbgmsg We should use zfs_dbgmsg instead of spa_dbgmsg. Or at least, metaslab_condense() should call zfs_dbgmsg because it's important and rare enough to always log. It's possible that the message in zio_dva_allocate() would be too high-frequency for zfs_dbgmsg. illumos/illumos-gate@21f7c81cc1156e9202ce3412d3ecaa697c3b2222 Reviewed by: Serapheim Dimitropoulos Reviewed by: Pavel Zakharov Reviewed by: George Wilson Reviewed by: Richard Elling Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_debug.h vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/ztest/ztest.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Tue Jul 31 00:37:45 2018 (r336954) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Tue Jul 31 00:42:31 2018 (r336955) @@ -1733,7 +1733,7 @@ metaslab_set_fragmentation(metaslab_t *msp) if (spa_writeable(spa) && txg < spa_final_dirty_txg(spa)) { msp->ms_condense_wanted = B_TRUE; vdev_dirty(vd, VDD_METASLAB, msp, txg + 1); - spa_dbgmsg(spa, "txg %llu, requesting force condense: " + zfs_dbgmsg("txg %llu, requesting force condense: " "ms_id %llu, vdev_id %llu", txg, msp->ms_id, vd->vdev_id); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Tue Jul 31 00:37:45 2018 (r336954) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Tue Jul 31 00:42:31 2018 (r336955) @@ -245,7 +245,7 @@ int spa_mode_global; * Everything except dprintf, spa, and indirect_remap is on by default * in debug builds. */ -int zfs_flags = ~(ZFS_DEBUG_DPRINTF | ZFS_DEBUG_SPA | ZFS_DEBUG_INDIRECT_REMAP); +int zfs_flags = ~(ZFS_DEBUG_DPRINTF | ZFS_DEBUG_INDIRECT_REMAP); #else int zfs_flags = 0; #endif @@ -707,8 +707,6 @@ spa_add(const char *name, nvlist_t *config, const char kstat_install(spa->spa_iokstat); } - spa->spa_debug = ((zfs_flags & ZFS_DEBUG_SPA) != 0); - spa->spa_min_ashift = INT_MAX; spa->spa_max_ashift = 0; @@ -2135,12 +2133,6 @@ spa_scan_get_stats(spa_t *spa, pool_scan_stat_t *ps) ps->pss_pass_scrub_spent_paused = spa->spa_scan_pass_scrub_spent_paused; return (0); -} - -boolean_t -spa_debug_enabled(spa_t *spa) -{ - return (spa->spa_debug); } int Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Tue Jul 31 00:37:45 2018 (r336954) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Tue Jul 31 00:42:31 2018 (r336955) @@ -927,13 +927,6 @@ _NOTE(CONSTCOND) } while (0) #define dprintf_bp(bp, fmt, ...) #endif -extern boolean_t spa_debug_enabled(spa_t *spa); -#define spa_dbgmsg(spa, ...) \ -{ \ - if (spa_debug_enabled(spa)) \ - zfs_dbgmsg(__VA_ARGS__); \ -} - extern int spa_mode_global; /* mode, e.g. FREAD | FWRITE */ #ifdef __cplusplus Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Tue Jul 31 00:37:45 2018 (r336954) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Tue Jul 31 00:42:31 2018 (r336955) @@ -328,7 +328,6 @@ struct spa { kcondvar_t spa_suspend_cv; /* notification of resume */ uint8_t spa_suspended; /* pool is suspended */ uint8_t spa_claiming; /* pool is doing zil_claim() */ - boolean_t spa_debug; /* debug enabled? */ boolean_t spa_is_root; /* pool is root */ int spa_minref; /* num refs when first opened */ int spa_mode; /* FREAD | FWRITE */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_debug.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_debug.h Tue Jul 31 00:37:45 2018 (r336954) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_debug.h Tue Jul 31 00:42:31 2018 (r336955) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2016 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. */ #ifndef _SYS_ZFS_DEBUG_H @@ -55,7 +55,7 @@ extern boolean_t zfs_free_leak_on_eio; #define ZFS_DEBUG_DNODE_VERIFY (1 << 2) #define ZFS_DEBUG_SNAPNAMES (1 << 3) #define ZFS_DEBUG_MODIFY (1 << 4) -#define ZFS_DEBUG_SPA (1 << 5) +/* 1<<5 was previously used, try not to reuse */ #define ZFS_DEBUG_ZIO_FREE (1 << 6) #define ZFS_DEBUG_HISTOGRAM_VERIFY (1 << 7) #define ZFS_DEBUG_METASLAB_VERIFY (1 << 8) Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Tue Jul 31 00:37:45 2018 (r336954) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Tue Jul 31 00:42:31 2018 (r336955) @@ -2932,7 +2932,7 @@ zio_dva_allocate(zio_t *zio) &zio->io_alloc_list, zio, zio->io_allocator); if (error != 0) { - spa_dbgmsg(spa, "%s: metaslab allocation failure: zio %p, " + zfs_dbgmsg("%s: metaslab allocation failure: zio %p, " "size %llu, error %d", spa_name(spa), zio, zio->io_size, error); if (error == ENOSPC && zio->io_size > SPA_MINBLOCKSIZE) From owner-svn-src-vendor@freebsd.org Tue Jul 31 00:56:43 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD9FD1066D58; Tue, 31 Jul 2018 00:56:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5CB787051D; Tue, 31 Jul 2018 00:56:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 373AF4704; Tue, 31 Jul 2018 00:56:43 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6V0uhKd010293; Tue, 31 Jul 2018 00:56:43 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6V0ufMF010286; Tue, 31 Jul 2018 00:56:41 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807310056.w6V0ufMF010286@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 00:56:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336958 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 336958 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 00:56:44 -0000 Author: mav Date: Tue Jul 31 00:56:41 2018 New Revision: 336958 URL: https://svnweb.freebsd.org/changeset/base/336958 Log: 9337 zfs get all is slow due to uncached metadata This project's goal is to make read-heavy channel programs and zfs(1m) administrative commands faster by caching all the metadata that they will need in the dbuf layer. This will prevent the data from being evicted, so that any future call to i.e. zfs get all won't have to go to disk (very much). illumos/illumos-gate@adb52d9262f45a04318fc6e188fe2b7f59d989a5 Reviewed by: Prakash Surya Reviewed by: George Wilson Reviewed by: Thomas Caputi Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vfsops.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Tue Jul 31 00:48:08 2018 (r336957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Tue Jul 31 00:56:41 2018 (r336958) @@ -49,6 +49,7 @@ #include #include #include +#include uint_t zfs_dbuf_evict_key; @@ -74,24 +75,58 @@ static kcondvar_t dbuf_evict_cv; static boolean_t dbuf_evict_thread_exit; /* - * LRU cache of dbufs. The dbuf cache maintains a list of dbufs that - * are not currently held but have been recently released. These dbufs - * are not eligible for arc eviction until they are aged out of the cache. - * Dbufs are added to the dbuf cache once the last hold is released. If a - * dbuf is later accessed and still exists in the dbuf cache, then it will - * be removed from the cache and later re-added to the head of the cache. - * Dbufs that are aged out of the cache will be immediately destroyed and - * become eligible for arc eviction. + * There are two dbuf caches; each dbuf can only be in one of them at a time. + * + * 1. Cache of metadata dbufs, to help make read-heavy administrative commands + * from /sbin/zfs run faster. The "metadata cache" specifically stores dbufs + * that represent the metadata that describes filesystems/snapshots/ + * bookmarks/properties/etc. We only evict from this cache when we export a + * pool, to short-circuit as much I/O as possible for all administrative + * commands that need the metadata. There is no eviction policy for this + * cache, because we try to only include types in it which would occupy a + * very small amount of space per object but create a large impact on the + * performance of these commands. Instead, after it reaches a maximum size + * (which should only happen on very small memory systems with a very large + * number of filesystem objects), we stop taking new dbufs into the + * metadata cache, instead putting them in the normal dbuf cache. + * + * 2. LRU cache of dbufs. The "dbuf cache" maintains a list of dbufs that + * are not currently held but have been recently released. These dbufs + * are not eligible for arc eviction until they are aged out of the cache. + * Dbufs that are aged out of the cache will be immediately destroyed and + * become eligible for arc eviction. + * + * Dbufs are added to these caches once the last hold is released. If a dbuf is + * later accessed and still exists in the dbuf cache, then it will be removed + * from the cache and later re-added to the head of the cache. + * + * If a given dbuf meets the requirements for the metadata cache, it will go + * there, otherwise it will be considered for the generic LRU dbuf cache. The + * caches and the refcounts tracking their sizes are stored in an array indexed + * by those caches' matching enum values (from dbuf_cached_state_t). */ -static multilist_t *dbuf_cache; -static refcount_t dbuf_cache_size; -uint64_t dbuf_cache_max_bytes = 0; +typedef struct dbuf_cache { + multilist_t *cache; + refcount_t size; +} dbuf_cache_t; +dbuf_cache_t dbuf_caches[DB_CACHE_MAX]; -/* Set the default size of the dbuf cache to log2 fraction of arc size. */ +/* Size limits for the caches */ +uint64_t dbuf_cache_max_bytes = 0; +uint64_t dbuf_metadata_cache_max_bytes = 0; +/* Set the default sizes of the caches to log2 fraction of arc size */ int dbuf_cache_shift = 5; +int dbuf_metadata_cache_shift = 6; /* - * The dbuf cache uses a three-stage eviction policy: + * For diagnostic purposes, this is incremented whenever we can't add + * something to the metadata cache because it's full, and instead put + * the data in the regular dbuf cache. + */ +uint64_t dbuf_metadata_cache_overflow; + +/* + * The LRU dbuf cache uses a three-stage eviction policy: * - A low water marker designates when the dbuf eviction thread * should stop evicting from the dbuf cache. * - When we reach the maximum size (aka mid water mark), we @@ -394,6 +429,41 @@ dbuf_is_metadata(dmu_buf_impl_t *db) } /* + * This returns whether this dbuf should be stored in the metadata cache, which + * is based on whether it's from one of the dnode types that store data related + * to traversing dataset hierarchies. + */ +static boolean_t +dbuf_include_in_metadata_cache(dmu_buf_impl_t *db) +{ + DB_DNODE_ENTER(db); + dmu_object_type_t type = DB_DNODE(db)->dn_type; + DB_DNODE_EXIT(db); + + /* Check if this dbuf is one of the types we care about */ + if (DMU_OT_IS_METADATA_CACHED(type)) { + /* If we hit this, then we set something up wrong in dmu_ot */ + ASSERT(DMU_OT_IS_METADATA(type)); + + /* + * Sanity check for small-memory systems: don't allocate too + * much memory for this purpose. + */ + if (refcount_count(&dbuf_caches[DB_DBUF_METADATA_CACHE].size) > + dbuf_metadata_cache_max_bytes) { + dbuf_metadata_cache_overflow++; + DTRACE_PROBE1(dbuf__metadata__cache__overflow, + dmu_buf_impl_t *, db); + return (B_FALSE); + } + + return (B_TRUE); + } + + return (B_FALSE); +} + +/* * This function *must* return indices evenly distributed between all * sublists of the multilist. This is needed due to how the dbuf eviction * code is laid out; dbuf_evict_thread() assumes dbufs are evenly @@ -428,7 +498,7 @@ dbuf_cache_above_hiwater(void) uint64_t dbuf_cache_hiwater_bytes = (dbuf_cache_max_bytes * dbuf_cache_hiwater_pct) / 100; - return (refcount_count(&dbuf_cache_size) > + return (refcount_count(&dbuf_caches[DB_DBUF_CACHE].size) > dbuf_cache_max_bytes + dbuf_cache_hiwater_bytes); } @@ -438,7 +508,7 @@ dbuf_cache_above_lowater(void) uint64_t dbuf_cache_lowater_bytes = (dbuf_cache_max_bytes * dbuf_cache_lowater_pct) / 100; - return (refcount_count(&dbuf_cache_size) > + return (refcount_count(&dbuf_caches[DB_DBUF_CACHE].size) > dbuf_cache_max_bytes - dbuf_cache_lowater_bytes); } @@ -448,8 +518,9 @@ dbuf_cache_above_lowater(void) static void dbuf_evict_one(void) { - int idx = multilist_get_random_index(dbuf_cache); - multilist_sublist_t *mls = multilist_sublist_lock(dbuf_cache, idx); + int idx = multilist_get_random_index(dbuf_caches[DB_DBUF_CACHE].cache); + multilist_sublist_t *mls = multilist_sublist_lock( + dbuf_caches[DB_DBUF_CACHE].cache, idx); ASSERT(!MUTEX_HELD(&dbuf_evict_lock)); @@ -472,8 +543,10 @@ dbuf_evict_one(void) if (db != NULL) { multilist_sublist_remove(mls, db); multilist_sublist_unlock(mls); - (void) refcount_remove_many(&dbuf_cache_size, + (void) refcount_remove_many(&dbuf_caches[DB_DBUF_CACHE].size, db->db.db_size, db); + ASSERT3U(db->db_caching_status, ==, DB_DBUF_CACHE); + db->db_caching_status = DB_NO_CACHE; dbuf_destroy(db); } else { multilist_sublist_unlock(mls); @@ -560,7 +633,8 @@ dbuf_evict_notify(void) * because it's OK to occasionally make the wrong decision here, * and grabbing the lock results in massive lock contention. */ - if (refcount_count(&dbuf_cache_size) > dbuf_cache_max_bytes) { + if (refcount_count(&dbuf_caches[DB_DBUF_CACHE].size) > + dbuf_cache_max_bytes) { if (dbuf_cache_above_hiwater()) dbuf_evict_one(); cv_signal(&dbuf_evict_cv); @@ -600,15 +674,21 @@ retry: mutex_init(&h->hash_mutexes[i], NULL, MUTEX_DEFAULT, NULL); /* - * Setup the parameters for the dbuf cache. We set the size of the - * dbuf cache to 1/32nd (default) of the size of the ARC. If the value - * has been set in /etc/system and it's not greater than the size of - * the ARC, then we honor that value. + * Setup the parameters for the dbuf caches. We set the sizes of the + * dbuf cache and the metadata cache to 1/32nd and 1/16th (default) + * of the size of the ARC, respectively. If the values are set in + * /etc/system and they're not greater than the size of the ARC, then + * we honor that value. */ if (dbuf_cache_max_bytes == 0 || dbuf_cache_max_bytes >= arc_max_bytes()) { dbuf_cache_max_bytes = arc_max_bytes() >> dbuf_cache_shift; } + if (dbuf_metadata_cache_max_bytes == 0 || + dbuf_metadata_cache_max_bytes >= arc_max_bytes()) { + dbuf_metadata_cache_max_bytes = + arc_max_bytes() >> dbuf_metadata_cache_shift; + } /* * All entries are queued via taskq_dispatch_ent(), so min/maxalloc @@ -616,10 +696,13 @@ retry: */ dbu_evict_taskq = taskq_create("dbu_evict", 1, minclsyspri, 0, 0, 0); - dbuf_cache = multilist_create(sizeof (dmu_buf_impl_t), - offsetof(dmu_buf_impl_t, db_cache_link), - dbuf_cache_multilist_index_func); - refcount_create(&dbuf_cache_size); + for (dbuf_cached_state_t dcs = 0; dcs < DB_CACHE_MAX; dcs++) { + dbuf_caches[dcs].cache = + multilist_create(sizeof (dmu_buf_impl_t), + offsetof(dmu_buf_impl_t, db_cache_link), + dbuf_cache_multilist_index_func); + refcount_create(&dbuf_caches[dcs].size); + } tsd_create(&zfs_dbuf_evict_key, NULL); dbuf_evict_thread_exit = B_FALSE; @@ -653,8 +736,10 @@ dbuf_fini(void) mutex_destroy(&dbuf_evict_lock); cv_destroy(&dbuf_evict_cv); - refcount_destroy(&dbuf_cache_size); - multilist_destroy(dbuf_cache); + for (dbuf_cached_state_t dcs = 0; dcs < DB_CACHE_MAX; dcs++) { + refcount_destroy(&dbuf_caches[dcs].size); + multilist_destroy(dbuf_caches[dcs].cache); + } } /* @@ -2037,9 +2122,15 @@ dbuf_destroy(dmu_buf_impl_t *db) dbuf_clear_data(db); if (multilist_link_active(&db->db_cache_link)) { - multilist_remove(dbuf_cache, db); - (void) refcount_remove_many(&dbuf_cache_size, + ASSERT(db->db_caching_status == DB_DBUF_CACHE || + db->db_caching_status == DB_DBUF_METADATA_CACHE); + + multilist_remove(dbuf_caches[db->db_caching_status].cache, db); + (void) refcount_remove_many( + &dbuf_caches[db->db_caching_status].size, db->db.db_size, db); + + db->db_caching_status = DB_NO_CACHE; } ASSERT(db->db_state == DB_UNCACHED || db->db_state == DB_NOFILL); @@ -2093,6 +2184,7 @@ dbuf_destroy(dmu_buf_impl_t *db) ASSERT(db->db_hash_next == NULL); ASSERT(db->db_blkptr == NULL); ASSERT(db->db_data_pending == NULL); + ASSERT3U(db->db_caching_status, ==, DB_NO_CACHE); ASSERT(!multilist_link_active(&db->db_cache_link)); kmem_cache_free(dbuf_kmem_cache, db); @@ -2231,6 +2323,7 @@ dbuf_create(dnode_t *dn, uint8_t level, uint64_t blkid ASSERT3U(db->db.db_size, >=, dn->dn_bonuslen); db->db.db_offset = DMU_BONUS_BLKID; db->db_state = DB_UNCACHED; + db->db_caching_status = DB_NO_CACHE; /* the bonus dbuf is not placed in the hash table */ arc_space_consume(sizeof (dmu_buf_impl_t), ARC_SPACE_OTHER); return (db); @@ -2263,6 +2356,7 @@ dbuf_create(dnode_t *dn, uint8_t level, uint64_t blkid avl_add(&dn->dn_dbufs, db); db->db_state = DB_UNCACHED; + db->db_caching_status = DB_NO_CACHE; mutex_exit(&dn->dn_dbufs_mtx); arc_space_consume(sizeof (dmu_buf_impl_t), ARC_SPACE_OTHER); @@ -2597,9 +2691,15 @@ top: if (multilist_link_active(&db->db_cache_link)) { ASSERT(refcount_is_zero(&db->db_holds)); - multilist_remove(dbuf_cache, db); - (void) refcount_remove_many(&dbuf_cache_size, + ASSERT(db->db_caching_status == DB_DBUF_CACHE || + db->db_caching_status == DB_DBUF_METADATA_CACHE); + + multilist_remove(dbuf_caches[db->db_caching_status].cache, db); + (void) refcount_remove_many( + &dbuf_caches[db->db_caching_status].size, db->db.db_size, db); + + db->db_caching_status = DB_NO_CACHE; } (void) refcount_add(&db->db_holds, tag); DBUF_VERIFY(db); @@ -2816,12 +2916,22 @@ dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag) db->db_pending_evict) { dbuf_destroy(db); } else if (!multilist_link_active(&db->db_cache_link)) { - multilist_insert(dbuf_cache, db); - (void) refcount_add_many(&dbuf_cache_size, + ASSERT3U(db->db_caching_status, ==, + DB_NO_CACHE); + + dbuf_cached_state_t dcs = + dbuf_include_in_metadata_cache(db) ? + DB_DBUF_METADATA_CACHE : DB_DBUF_CACHE; + db->db_caching_status = dcs; + + multilist_insert(dbuf_caches[dcs].cache, db); + (void) refcount_add_many(&dbuf_caches[dcs].size, db->db.db_size, db); mutex_exit(&db->db_mtx); - dbuf_evict_notify(); + if (db->db_caching_status == DB_DBUF_CACHE) { + dbuf_evict_notify(); + } } if (do_arc_evict) Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Tue Jul 31 00:48:08 2018 (r336957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Tue Jul 31 00:56:41 2018 (r336958) @@ -73,60 +73,60 @@ uint32_t zfs_per_txg_dirty_frees_percent = 30; int zfs_object_remap_one_indirect_delay_ticks = 0; const dmu_object_type_info_t dmu_ot[DMU_OT_NUMTYPES] = { - { DMU_BSWAP_UINT8, TRUE, "unallocated" }, - { DMU_BSWAP_ZAP, TRUE, "object directory" }, - { DMU_BSWAP_UINT64, TRUE, "object array" }, - { DMU_BSWAP_UINT8, TRUE, "packed nvlist" }, - { DMU_BSWAP_UINT64, TRUE, "packed nvlist size" }, - { DMU_BSWAP_UINT64, TRUE, "bpobj" }, - { DMU_BSWAP_UINT64, TRUE, "bpobj header" }, - { DMU_BSWAP_UINT64, TRUE, "SPA space map header" }, - { DMU_BSWAP_UINT64, TRUE, "SPA space map" }, - { DMU_BSWAP_UINT64, TRUE, "ZIL intent log" }, - { DMU_BSWAP_DNODE, TRUE, "DMU dnode" }, - { DMU_BSWAP_OBJSET, TRUE, "DMU objset" }, - { DMU_BSWAP_UINT64, TRUE, "DSL directory" }, - { DMU_BSWAP_ZAP, TRUE, "DSL directory child map"}, - { DMU_BSWAP_ZAP, TRUE, "DSL dataset snap map" }, - { DMU_BSWAP_ZAP, TRUE, "DSL props" }, - { DMU_BSWAP_UINT64, TRUE, "DSL dataset" }, - { DMU_BSWAP_ZNODE, TRUE, "ZFS znode" }, - { DMU_BSWAP_OLDACL, TRUE, "ZFS V0 ACL" }, - { DMU_BSWAP_UINT8, FALSE, "ZFS plain file" }, - { DMU_BSWAP_ZAP, TRUE, "ZFS directory" }, - { DMU_BSWAP_ZAP, TRUE, "ZFS master node" }, - { DMU_BSWAP_ZAP, TRUE, "ZFS delete queue" }, - { DMU_BSWAP_UINT8, FALSE, "zvol object" }, - { DMU_BSWAP_ZAP, TRUE, "zvol prop" }, - { DMU_BSWAP_UINT8, FALSE, "other uint8[]" }, - { DMU_BSWAP_UINT64, FALSE, "other uint64[]" }, - { DMU_BSWAP_ZAP, TRUE, "other ZAP" }, - { DMU_BSWAP_ZAP, TRUE, "persistent error log" }, - { DMU_BSWAP_UINT8, TRUE, "SPA history" }, - { DMU_BSWAP_UINT64, TRUE, "SPA history offsets" }, - { DMU_BSWAP_ZAP, TRUE, "Pool properties" }, - { DMU_BSWAP_ZAP, TRUE, "DSL permissions" }, - { DMU_BSWAP_ACL, TRUE, "ZFS ACL" }, - { DMU_BSWAP_UINT8, TRUE, "ZFS SYSACL" }, - { DMU_BSWAP_UINT8, TRUE, "FUID table" }, - { DMU_BSWAP_UINT64, TRUE, "FUID table size" }, - { DMU_BSWAP_ZAP, TRUE, "DSL dataset next clones"}, - { DMU_BSWAP_ZAP, TRUE, "scan work queue" }, - { DMU_BSWAP_ZAP, TRUE, "ZFS user/group used" }, - { DMU_BSWAP_ZAP, TRUE, "ZFS user/group quota" }, - { DMU_BSWAP_ZAP, TRUE, "snapshot refcount tags"}, - { DMU_BSWAP_ZAP, TRUE, "DDT ZAP algorithm" }, - { DMU_BSWAP_ZAP, TRUE, "DDT statistics" }, - { DMU_BSWAP_UINT8, TRUE, "System attributes" }, - { DMU_BSWAP_ZAP, TRUE, "SA master node" }, - { DMU_BSWAP_ZAP, TRUE, "SA attr registration" }, - { DMU_BSWAP_ZAP, TRUE, "SA attr layouts" }, - { DMU_BSWAP_ZAP, TRUE, "scan translations" }, - { DMU_BSWAP_UINT8, FALSE, "deduplicated block" }, - { DMU_BSWAP_ZAP, TRUE, "DSL deadlist map" }, - { DMU_BSWAP_UINT64, TRUE, "DSL deadlist map hdr" }, - { DMU_BSWAP_ZAP, TRUE, "DSL dir clones" }, - { DMU_BSWAP_UINT64, TRUE, "bpobj subobj" } + { DMU_BSWAP_UINT8, TRUE, FALSE, "unallocated" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "object directory" }, + { DMU_BSWAP_UINT64, TRUE, TRUE, "object array" }, + { DMU_BSWAP_UINT8, TRUE, FALSE, "packed nvlist" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "packed nvlist size" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "bpobj" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "bpobj header" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "SPA space map header" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "SPA space map" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "ZIL intent log" }, + { DMU_BSWAP_DNODE, TRUE, FALSE, "DMU dnode" }, + { DMU_BSWAP_OBJSET, TRUE, TRUE, "DMU objset" }, + { DMU_BSWAP_UINT64, TRUE, TRUE, "DSL directory" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "DSL directory child map" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "DSL dataset snap map" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "DSL props" }, + { DMU_BSWAP_UINT64, TRUE, TRUE, "DSL dataset" }, + { DMU_BSWAP_ZNODE, TRUE, FALSE, "ZFS znode" }, + { DMU_BSWAP_OLDACL, TRUE, FALSE, "ZFS V0 ACL" }, + { DMU_BSWAP_UINT8, FALSE, FALSE, "ZFS plain file" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "ZFS directory" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "ZFS master node" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "ZFS delete queue" }, + { DMU_BSWAP_UINT8, FALSE, FALSE, "zvol object" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "zvol prop" }, + { DMU_BSWAP_UINT8, FALSE, FALSE, "other uint8[]" }, + { DMU_BSWAP_UINT64, FALSE, FALSE, "other uint64[]" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "other ZAP" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "persistent error log" }, + { DMU_BSWAP_UINT8, TRUE, FALSE, "SPA history" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "SPA history offsets" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "Pool properties" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "DSL permissions" }, + { DMU_BSWAP_ACL, TRUE, FALSE, "ZFS ACL" }, + { DMU_BSWAP_UINT8, TRUE, FALSE, "ZFS SYSACL" }, + { DMU_BSWAP_UINT8, TRUE, FALSE, "FUID table" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "FUID table size" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "DSL dataset next clones" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "scan work queue" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "ZFS user/group used" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "ZFS user/group quota" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "snapshot refcount tags" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "DDT ZAP algorithm" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "DDT statistics" }, + { DMU_BSWAP_UINT8, TRUE, FALSE, "System attributes" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "SA master node" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "SA attr registration" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "SA attr layouts" }, + { DMU_BSWAP_ZAP, TRUE, FALSE, "scan translations" }, + { DMU_BSWAP_UINT8, FALSE, FALSE, "deduplicated block" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "DSL deadlist map" }, + { DMU_BSWAP_UINT64, TRUE, TRUE, "DSL deadlist map hdr" }, + { DMU_BSWAP_ZAP, TRUE, TRUE, "DSL dir clones" }, + { DMU_BSWAP_UINT64, TRUE, FALSE, "bpobj subobj" } }; const dmu_object_byteswap_info_t dmu_ot_byteswap[DMU_BSWAP_NUMFUNCS] = { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Tue Jul 31 00:48:08 2018 (r336957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Tue Jul 31 00:56:41 2018 (r336958) @@ -496,6 +496,14 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, bl os->os_primary_cache = ZFS_CACHE_ALL; os->os_secondary_cache = ZFS_CACHE_ALL; } + /* + * These properties will be filled in by the logic in zfs_get_zplprop() + * when they are queried for the first time. + */ + os->os_version = OBJSET_PROP_UNINITIALIZED; + os->os_normalization = OBJSET_PROP_UNINITIALIZED; + os->os_utf8only = OBJSET_PROP_UNINITIALIZED; + os->os_casesensitivity = OBJSET_PROP_UNINITIALIZED; if (ds == NULL || !ds->ds_is_snapshot) os->os_zil_header = os->os_phys->os_zil_header; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h Tue Jul 31 00:48:08 2018 (r336957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h Tue Jul 31 00:56:41 2018 (r336958) @@ -83,6 +83,13 @@ typedef enum dbuf_states { DB_EVICTING } dbuf_states_t; +typedef enum dbuf_cached_state { + DB_NO_CACHE = -1, + DB_DBUF_CACHE, + DB_DBUF_METADATA_CACHE, + DB_CACHE_MAX +} dbuf_cached_state_t; + struct dnode; struct dmu_tx; @@ -229,10 +236,11 @@ typedef struct dmu_buf_impl { */ avl_node_t db_link; - /* - * Link in dbuf_cache. - */ + /* Link in dbuf_cache or dbuf_metadata_cache */ multilist_node_t db_cache_link; + + /* Tells us which dbuf cache this dbuf is in, if any */ + dbuf_cached_state_t db_caching_status; /* Data which is unique to data (leaf) blocks: */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Tue Jul 31 00:48:08 2018 (r336957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Tue Jul 31 00:56:41 2018 (r336958) @@ -109,7 +109,8 @@ typedef enum dmu_object_byteswap { /* * Defines a uint8_t object type. Object types specify if the data * in the object is metadata (boolean) and how to byteswap the data - * (dmu_object_byteswap_t). + * (dmu_object_byteswap_t). All of the types created by this method + * are cached in the dbuf metadata cache. */ #define DMU_OT(byteswap, metadata) \ (DMU_OT_NEWTYPE | \ @@ -124,6 +125,9 @@ typedef enum dmu_object_byteswap { ((ot) & DMU_OT_METADATA) : \ dmu_ot[(ot)].ot_metadata) +#define DMU_OT_IS_METADATA_CACHED(ot) (((ot) & DMU_OT_NEWTYPE) ? \ + B_TRUE : dmu_ot[(ot)].ot_dbuf_metadata_cache) + /* * These object types use bp_fill != 1 for their L0 bp's. Therefore they can't * have their data embedded (i.e. use a BP_IS_EMBEDDED() bp), because bp_fill @@ -799,6 +803,7 @@ typedef void arc_byteswap_func_t(void *buf, size_t siz typedef struct dmu_object_type_info { dmu_object_byteswap_t ot_byteswap; boolean_t ot_metadata; + boolean_t ot_dbuf_metadata_cache; char *ot_name; } dmu_object_type_info_t; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h Tue Jul 31 00:48:08 2018 (r336957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_objset.h Tue Jul 31 00:56:41 2018 (r336958) @@ -39,6 +39,7 @@ #include #include #include +#include #ifdef __cplusplus extern "C" { @@ -69,6 +70,7 @@ typedef struct objset_phys { dnode_phys_t os_groupused_dnode; } objset_phys_t; +#define OBJSET_PROP_UNINITIALIZED ((uint64_t)-1) struct objset { /* Immutable: */ struct dsl_dataset *os_dsl_dataset; @@ -100,6 +102,16 @@ struct objset { zfs_sync_type_t os_sync; zfs_redundant_metadata_type_t os_redundant_metadata; int os_recordsize; + /* + * The next four values are used as a cache of whatever's on disk, and + * are initialized the first time these properties are queried. Before + * being initialized with their real values, their values are + * OBJSET_PROP_UNINITIALIZED. + */ + uint64_t os_version; + uint64_t os_normalization; + uint64_t os_utf8only; + uint64_t os_casesensitivity; /* * Pointer is constant; the blkptr it points to is protected by Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vfsops.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vfsops.c Tue Jul 31 00:48:08 2018 (r336957) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vfsops.c Tue Jul 31 00:56:41 2018 (r336958) @@ -2232,6 +2232,7 @@ zfs_set_version(zfsvfs_t *zfsvfs, uint64_t newvers) dmu_tx_commit(tx); zfsvfs->z_version = newvers; + os->os_version = newvers; zfs_set_fuid_feature(zfsvfs); @@ -2244,17 +2245,47 @@ zfs_set_version(zfsvfs_t *zfsvfs, uint64_t newvers) int zfs_get_zplprop(objset_t *os, zfs_prop_t prop, uint64_t *value) { - const char *pname; - int error = ENOENT; + uint64_t *cached_copy = NULL; /* - * Look up the file system's value for the property. For the - * version property, we look up a slightly different string. + * Figure out where in the objset_t the cached copy would live, if it + * is available for the requested property. */ - if (prop == ZFS_PROP_VERSION) + if (os != NULL) { + switch (prop) { + case ZFS_PROP_VERSION: + cached_copy = &os->os_version; + break; + case ZFS_PROP_NORMALIZE: + cached_copy = &os->os_normalization; + break; + case ZFS_PROP_UTF8ONLY: + cached_copy = &os->os_utf8only; + break; + case ZFS_PROP_CASE: + cached_copy = &os->os_casesensitivity; + break; + default: + break; + } + } + if (cached_copy != NULL && *cached_copy != OBJSET_PROP_UNINITIALIZED) { + *value = *cached_copy; + return (0); + } + + /* + * If the property wasn't cached, look up the file system's value for + * the property. For the version property, we look up a slightly + * different string. + */ + const char *pname; + int error = ENOENT; + if (prop == ZFS_PROP_VERSION) { pname = ZPL_VERSION_STR; - else + } else { pname = zfs_prop_to_name(prop); + } if (os != NULL) { ASSERT3U(os->os_phys->os_type, ==, DMU_OST_ZFS); @@ -2279,6 +2310,15 @@ zfs_get_zplprop(objset_t *os, zfs_prop_t prop, uint64_ } error = 0; } + + /* + * If one of the methods for getting the property value above worked, + * copy it into the objset_t's cache. + */ + if (error == 0 && cached_copy != NULL) { + *cached_copy = *value; + } + return (error); } From owner-svn-src-vendor@freebsd.org Tue Jul 31 01:00:49 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11A741066EBB; Tue, 31 Jul 2018 01:00:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A7DE9709F3; Tue, 31 Jul 2018 01:00:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 868F8471D; Tue, 31 Jul 2018 01:00:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6V10lKs010774; Tue, 31 Jul 2018 01:00:47 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6V10lH3010773; Tue, 31 Jul 2018 01:00:47 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807310100.w6V10lH3010773@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 01:00:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336960 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 336960 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 01:00:49 -0000 Author: mav Date: Tue Jul 31 01:00:46 2018 New Revision: 336960 URL: https://svnweb.freebsd.org/changeset/base/336960 Log: 9256 zfs send space estimation off by > 10% on some datasets illumos/illummos-gate@df477c0afa111b5205c872dab36dbfde391656de Reviewed by: Matt Ahrens Reviewed by: John Kennedy Approved by: Richard Lowe Author: Paul Dagnelie Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Tue Jul 31 00:58:21 2018 (r336959) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Tue Jul 31 01:00:46 2018 (r336960) @@ -66,6 +66,11 @@ int zfs_send_set_freerecords_bit = B_TRUE; static char *dmu_recv_tag = "dmu_recv_tag"; const char *recv_clone_name = "%recv"; +/* + * Use this to override the recordsize calculation for fast zfs send estimates. + */ +uint64_t zfs_override_estimate_recordsize = 0; + #define BP_SPAN(datablkszsec, indblkshift, level) \ (((uint64_t)datablkszsec) << (SPA_MINBLOCKSHIFT + \ (level) * (indblkshift - SPA_BLKPTRSHIFT))) @@ -1093,7 +1098,7 @@ static int dmu_adjust_send_estimate_for_indirects(dsl_dataset_t *ds, uint64_t uncompressed, uint64_t compressed, boolean_t stream_compressed, uint64_t *sizep) { - int err; + int err = 0; uint64_t size; /* * Assume that space (both on-disk and in-stream) is dominated by @@ -1106,7 +1111,9 @@ dmu_adjust_send_estimate_for_indirects(dsl_dataset_t * VERIFY0(dmu_objset_from_ds(ds, &os)); /* Assume all (uncompressed) blocks are recordsize. */ - if (os->os_phys->os_type == DMU_OST_ZVOL) { + if (zfs_override_estimate_recordsize != 0) { + recordsize = zfs_override_estimate_recordsize; + } else if (os->os_phys->os_type == DMU_OST_ZVOL) { err = dsl_prop_get_int_ds(ds, zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE), &recordsize); } else { From owner-svn-src-vendor@freebsd.org Tue Jul 31 18:49:14 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06FC31061743; Tue, 31 Jul 2018 18:49:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B030D76C1C; Tue, 31 Jul 2018 18:49:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 919A1177C0; Tue, 31 Jul 2018 18:49:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VInD0h065572; Tue, 31 Jul 2018 18:49:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VInBQN065564; Tue, 31 Jul 2018 18:49:11 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807311849.w6VInBQN065564@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 18:49:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336991 - vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendo... X-SVN-Commit-Revision: 336991 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 18:49:14 -0000 Author: mav Date: Tue Jul 31 18:49:07 2018 New Revision: 336991 URL: https://svnweb.freebsd.org/changeset/base/336991 Log: 9102 zfs should be able to initialize storage devices The first access to a disk block can incur a performance penalty on some platforms (e.g. AWS's EBS, VMware VMDKs). Therefore it is recommended that volumes be "thick provisioned", where supported by the platform (VMware). Thick provisioning is time consuming and often is ignored. If the thick provision step is omitted, customers will see suboptimal performance until we have written to all parts of the LUN. ZFS should be able to initialize any unused storage to remove any first-write penalty that exists. illumos/illumos-gate@094e47e980b0796b94b1b8f51f462a64d246e516 Reviewed by: John Wren Kennedy Reviewed by: Matthew Ahrens Reviewed by: Pavel Zakharov Reviewed by: Prakash Surya Approved by: Richard Lowe Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h vendor/illumos/dist/man/man1m/zpool.1m Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_priority.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Tue Jul 31 18:49:07 2018 (r336991) @@ -84,6 +84,7 @@ static int zpool_do_detach(int, char **); static int zpool_do_replace(int, char **); static int zpool_do_split(int, char **); +static int zpool_do_initialize(int, char **); static int zpool_do_scrub(int, char **); static int zpool_do_import(int, char **); @@ -133,6 +134,7 @@ typedef enum { HELP_ONLINE, HELP_REPLACE, HELP_REMOVE, + HELP_INITIALIZE, HELP_SCRUB, HELP_STATUS, HELP_UPGRADE, @@ -184,6 +186,7 @@ static zpool_command_t command_table[] = { { "replace", zpool_do_replace, HELP_REPLACE }, { "split", zpool_do_split, HELP_SPLIT }, { NULL }, + { "initialize", zpool_do_initialize, HELP_INITIALIZE }, { "scrub", zpool_do_scrub, HELP_SCRUB }, { NULL }, { "import", zpool_do_import, HELP_IMPORT }, @@ -257,6 +260,8 @@ get_usage(zpool_help_t idx) return (gettext("\tremove [-nps] ...\n")); case HELP_REOPEN: return (gettext("\treopen \n")); + case HELP_INITIALIZE: + return (gettext("\tinitialize [-cs] [ ...]\n")); case HELP_SCRUB: return (gettext("\tscrub [-s | -p] ...\n")); case HELP_STATUS: @@ -1589,6 +1594,43 @@ print_status_config(zpool_handle_t *zhp, const char *n "resilvering" : "repairing"); } + if ((vs->vs_initialize_state == VDEV_INITIALIZE_ACTIVE || + vs->vs_initialize_state == VDEV_INITIALIZE_SUSPENDED || + vs->vs_initialize_state == VDEV_INITIALIZE_COMPLETE) && + !vs->vs_scan_removing) { + char zbuf[1024]; + char tbuf[256]; + struct tm zaction_ts; + + time_t t = vs->vs_initialize_action_time; + int initialize_pct = 100; + if (vs->vs_initialize_state != VDEV_INITIALIZE_COMPLETE) { + initialize_pct = (vs->vs_initialize_bytes_done * 100 / + (vs->vs_initialize_bytes_est + 1)); + } + + (void) localtime_r(&t, &zaction_ts); + (void) strftime(tbuf, sizeof (tbuf), "%c", &zaction_ts); + + switch (vs->vs_initialize_state) { + case VDEV_INITIALIZE_SUSPENDED: + (void) snprintf(zbuf, sizeof (zbuf), + ", suspended, started at %s", tbuf); + break; + case VDEV_INITIALIZE_ACTIVE: + (void) snprintf(zbuf, sizeof (zbuf), + ", started at %s", tbuf); + break; + case VDEV_INITIALIZE_COMPLETE: + (void) snprintf(zbuf, sizeof (zbuf), + ", completed at %s", tbuf); + break; + } + + (void) printf(gettext(" (%d%% initialized%s)"), + initialize_pct, zbuf); + } + (void) printf("\n"); for (c = 0; c < children; c++) { @@ -4163,6 +4205,119 @@ zpool_do_scrub(int argc, char **argv) } return (for_each_pool(argc, argv, B_TRUE, NULL, scrub_callback, &cb)); +} + +static void +zpool_collect_leaves(zpool_handle_t *zhp, nvlist_t *nvroot, nvlist_t *res) +{ + uint_t children = 0; + nvlist_t **child; + uint_t i; + + (void) nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_CHILDREN, + &child, &children); + + if (children == 0) { + char *path = zpool_vdev_name(g_zfs, zhp, nvroot, B_FALSE); + fnvlist_add_boolean(res, path); + free(path); + return; + } + + for (i = 0; i < children; i++) { + zpool_collect_leaves(zhp, child[i], res); + } +} + +/* + * zpool initialize [-cs] [ ...] + * Initialize all unused blocks in the specified vdevs, or all vdevs in the pool + * if none specified. + * + * -c Cancel. Ends active initializing. + * -s Suspend. Initializing can then be restarted with no flags. + */ +int +zpool_do_initialize(int argc, char **argv) +{ + int c; + char *poolname; + zpool_handle_t *zhp; + nvlist_t *vdevs; + int err = 0; + + struct option long_options[] = { + {"cancel", no_argument, NULL, 'c'}, + {"suspend", no_argument, NULL, 's'}, + {0, 0, 0, 0} + }; + + pool_initialize_func_t cmd_type = POOL_INITIALIZE_DO; + while ((c = getopt_long(argc, argv, "cs", long_options, NULL)) != -1) { + switch (c) { + case 'c': + if (cmd_type != POOL_INITIALIZE_DO) { + (void) fprintf(stderr, gettext("-c cannot be " + "combined with other options\n")); + usage(B_FALSE); + } + cmd_type = POOL_INITIALIZE_CANCEL; + break; + case 's': + if (cmd_type != POOL_INITIALIZE_DO) { + (void) fprintf(stderr, gettext("-s cannot be " + "combined with other options\n")); + usage(B_FALSE); + } + cmd_type = POOL_INITIALIZE_SUSPEND; + break; + case '?': + if (optopt != 0) { + (void) fprintf(stderr, + gettext("invalid option '%c'\n"), optopt); + } else { + (void) fprintf(stderr, + gettext("invalid option '%s'\n"), + argv[optind - 1]); + } + usage(B_FALSE); + } + } + + argc -= optind; + argv += optind; + + if (argc < 1) { + (void) fprintf(stderr, gettext("missing pool name argument\n")); + usage(B_FALSE); + return (-1); + } + + poolname = argv[0]; + zhp = zpool_open(g_zfs, poolname); + if (zhp == NULL) + return (-1); + + vdevs = fnvlist_alloc(); + if (argc == 1) { + /* no individual leaf vdevs specified, so add them all */ + nvlist_t *config = zpool_get_config(zhp, NULL); + nvlist_t *nvroot = fnvlist_lookup_nvlist(config, + ZPOOL_CONFIG_VDEV_TREE); + zpool_collect_leaves(zhp, nvroot, vdevs); + } else { + int i; + for (i = 1; i < argc; i++) { + fnvlist_add_boolean(vdevs, argv[i]); + } + } + + err = zpool_initialize(zhp, cmd_type, vdevs); + + fnvlist_free(vdevs); + zpool_close(zhp); + + return (err); } typedef struct status_cbdata { Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/cmd/ztest/ztest.c Tue Jul 31 18:49:07 2018 (r336991) @@ -103,6 +103,7 @@ #include #include #include +#include #include #include #include @@ -346,6 +347,7 @@ ztest_func_t ztest_spa_upgrade; ztest_func_t ztest_device_removal; ztest_func_t ztest_remap_blocks; ztest_func_t ztest_spa_checkpoint_create_discard; +ztest_func_t ztest_initialize; uint64_t zopt_always = 0ULL * NANOSEC; /* all the time */ uint64_t zopt_incessant = 1ULL * NANOSEC / 10; /* every 1/10 second */ @@ -389,7 +391,8 @@ ztest_info_t ztest_info[] = { &ztest_opts.zo_vdevtime }, { ztest_device_removal, 1, &zopt_sometimes }, { ztest_remap_blocks, 1, &zopt_sometimes }, - { ztest_spa_checkpoint_create_discard, 1, &zopt_rarely } + { ztest_spa_checkpoint_create_discard, 1, &zopt_rarely }, + { ztest_initialize, 1, &zopt_sometimes } }; #define ZTEST_FUNCS (sizeof (ztest_info) / sizeof (ztest_info_t)) @@ -5468,6 +5471,97 @@ ztest_spa_rename(ztest_ds_t *zd, uint64_t id) umem_free(newname, strlen(newname) + 1); rw_exit(&ztest_name_lock); +} + +static vdev_t * +ztest_random_concrete_vdev_leaf(vdev_t *vd) +{ + if (vd == NULL) + return (NULL); + + if (vd->vdev_children == 0) + return (vd); + + vdev_t *eligible[vd->vdev_children]; + int eligible_idx = 0, i; + for (i = 0; i < vd->vdev_children; i++) { + vdev_t *cvd = vd->vdev_child[i]; + if (cvd->vdev_top->vdev_removing) + continue; + if (cvd->vdev_children > 0 || + (vdev_is_concrete(cvd) && !cvd->vdev_detached)) { + eligible[eligible_idx++] = cvd; + } + } + VERIFY(eligible_idx > 0); + + uint64_t child_no = ztest_random(eligible_idx); + return (ztest_random_concrete_vdev_leaf(eligible[child_no])); +} + +/* ARGSUSED */ +void +ztest_initialize(ztest_ds_t *zd, uint64_t id) +{ + spa_t *spa = ztest_spa; + int error = 0; + + mutex_enter(&ztest_vdev_lock); + + spa_config_enter(spa, SCL_VDEV, FTAG, RW_READER); + + /* Random leaf vdev */ + vdev_t *rand_vd = ztest_random_concrete_vdev_leaf(spa->spa_root_vdev); + if (rand_vd == NULL) { + spa_config_exit(spa, SCL_VDEV, FTAG); + mutex_exit(&ztest_vdev_lock); + return; + } + + /* + * The random vdev we've selected may change as soon as we + * drop the spa_config_lock. We create local copies of things + * we're interested in. + */ + uint64_t guid = rand_vd->vdev_guid; + char *path = strdup(rand_vd->vdev_path); + boolean_t active = rand_vd->vdev_initialize_thread != NULL; + + zfs_dbgmsg("vd %p, guid %llu", rand_vd, guid); + spa_config_exit(spa, SCL_VDEV, FTAG); + + uint64_t cmd = ztest_random(POOL_INITIALIZE_FUNCS); + error = spa_vdev_initialize(spa, guid, cmd); + switch (cmd) { + case POOL_INITIALIZE_CANCEL: + if (ztest_opts.zo_verbose >= 4) { + (void) printf("Cancel initialize %s", path); + if (!active) + (void) printf(" failed (no initialize active)"); + (void) printf("\n"); + } + break; + case POOL_INITIALIZE_DO: + if (ztest_opts.zo_verbose >= 4) { + (void) printf("Start initialize %s", path); + if (active && error == 0) + (void) printf(" failed (already active)"); + else if (error != 0) + (void) printf(" failed (error %d)", error); + (void) printf("\n"); + } + break; + case POOL_INITIALIZE_SUSPEND: + if (ztest_opts.zo_verbose >= 4) { + (void) printf("Suspend initialize %s", path); + if (!active) + (void) printf(" failed (no initialize active)"); + (void) printf("\n"); + } + break; + } + free(path); + mutex_exit(&ztest_vdev_lock); } /* Modified: vendor/illumos/dist/lib/libzfs/common/libzfs.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs.h Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/lib/libzfs/common/libzfs.h Tue Jul 31 18:49:07 2018 (r336991) @@ -136,6 +136,9 @@ typedef enum zfs_error { EZFS_NO_CHECKPOINT, /* pool has no checkpoint */ EZFS_DEVRM_IN_PROGRESS, /* a device is currently being removed */ EZFS_VDEV_TOO_BIG, /* a device is too big to be used */ + EZFS_TOOMANY, /* argument list too long */ + EZFS_INITIALIZING, /* currently initializing */ + EZFS_NO_INITIALIZE, /* no active initialize */ EZFS_UNKNOWN } zfs_error_t; @@ -260,6 +263,8 @@ typedef struct splitflags { * Functions to manipulate pool and vdev state */ extern int zpool_scan(zpool_handle_t *, pool_scan_func_t, pool_scrub_cmd_t); +extern int zpool_initialize(zpool_handle_t *, pool_initialize_func_t, + nvlist_t *); extern int zpool_clear(zpool_handle_t *, const char *, nvlist_t *); extern int zpool_reguid(zpool_handle_t *); extern int zpool_reopen(zpool_handle_t *); Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Tue Jul 31 18:49:07 2018 (r336991) @@ -1969,6 +1969,100 @@ zpool_scan(zpool_handle_t *zhp, pool_scan_func_t func, } } +static int +xlate_init_err(int err) +{ + switch (err) { + case ENODEV: + return (EZFS_NODEVICE); + case EINVAL: + case EROFS: + return (EZFS_BADDEV); + case EBUSY: + return (EZFS_INITIALIZING); + case ESRCH: + return (EZFS_NO_INITIALIZE); + } + return (err); +} + +/* + * Begin, suspend, or cancel the initialization (initializing of all free + * blocks) for the given vdevs in the given pool. + */ +int +zpool_initialize(zpool_handle_t *zhp, pool_initialize_func_t cmd_type, + nvlist_t *vds) +{ + char msg[1024]; + libzfs_handle_t *hdl = zhp->zpool_hdl; + + nvlist_t *errlist; + + /* translate vdev names to guids */ + nvlist_t *vdev_guids = fnvlist_alloc(); + nvlist_t *guids_to_paths = fnvlist_alloc(); + boolean_t spare, cache; + nvlist_t *tgt; + nvpair_t *elem; + + for (elem = nvlist_next_nvpair(vds, NULL); elem != NULL; + elem = nvlist_next_nvpair(vds, elem)) { + char *vd_path = nvpair_name(elem); + tgt = zpool_find_vdev(zhp, vd_path, &spare, &cache, NULL); + + if ((tgt == NULL) || cache || spare) { + (void) snprintf(msg, sizeof (msg), + dgettext(TEXT_DOMAIN, "cannot initialize '%s'"), + vd_path); + int err = (tgt == NULL) ? EZFS_NODEVICE : + (spare ? EZFS_ISSPARE : EZFS_ISL2CACHE); + fnvlist_free(vdev_guids); + fnvlist_free(guids_to_paths); + return (zfs_error(hdl, err, msg)); + } + + uint64_t guid = fnvlist_lookup_uint64(tgt, ZPOOL_CONFIG_GUID); + fnvlist_add_uint64(vdev_guids, vd_path, guid); + + (void) snprintf(msg, sizeof (msg), "%llu", guid); + fnvlist_add_string(guids_to_paths, msg, vd_path); + } + + int err = lzc_initialize(zhp->zpool_name, cmd_type, vdev_guids, + &errlist); + fnvlist_free(vdev_guids); + + if (err == 0) { + fnvlist_free(guids_to_paths); + return (0); + } + + nvlist_t *vd_errlist = NULL; + if (errlist != NULL) { + vd_errlist = fnvlist_lookup_nvlist(errlist, + ZPOOL_INITIALIZE_VDEVS); + } + + (void) snprintf(msg, sizeof (msg), + dgettext(TEXT_DOMAIN, "operation failed")); + + for (elem = nvlist_next_nvpair(vd_errlist, NULL); elem != NULL; + elem = nvlist_next_nvpair(vd_errlist, elem)) { + int64_t vd_error = xlate_init_err(fnvpair_value_int64(elem)); + char *path = fnvlist_lookup_string(guids_to_paths, + nvpair_name(elem)); + (void) zfs_error_fmt(hdl, vd_error, "cannot initialize '%s'", + path); + } + + fnvlist_free(guids_to_paths); + if (vd_errlist != NULL) + return (-1); + + return (zpool_standard_error(hdl, err, msg)); +} + /* * This provides a very minimal check whether a given string is likely a * c#t#d# style string. Users of this are expected to do their own Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_util.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_util.c Tue Jul 31 18:49:07 2018 (r336991) @@ -249,6 +249,13 @@ libzfs_error_description(libzfs_handle_t *hdl) return (dgettext(TEXT_DOMAIN, "device removal in progress")); case EZFS_VDEV_TOO_BIG: return (dgettext(TEXT_DOMAIN, "device exceeds supported size")); + case EZFS_TOOMANY: + return (dgettext(TEXT_DOMAIN, "argument list too long")); + case EZFS_INITIALIZING: + return (dgettext(TEXT_DOMAIN, "currently initializing")); + case EZFS_NO_INITIALIZE: + return (dgettext(TEXT_DOMAIN, "there is no active " + "initialization")); case EZFS_UNKNOWN: return (dgettext(TEXT_DOMAIN, "unknown error")); default: Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c Tue Jul 31 18:49:07 2018 (r336991) @@ -1038,3 +1038,40 @@ lzc_channel_program_nosync(const char *pool, const cha return (lzc_channel_program_impl(pool, program, B_FALSE, timeout, memlimit, argnvl, outnvl)); } + +/* + * Changes initializing state. + * + * vdevs should be a list of (, guid) where guid is a uint64 vdev GUID. + * The key is ignored. + * + * If there are errors related to vdev arguments, per-vdev errors are returned + * in an nvlist with the key "vdevs". Each error is a (guid, errno) pair where + * guid is stringified with PRIu64, and errno is one of the following as + * an int64_t: + * - ENODEV if the device was not found + * - EINVAL if the devices is not a leaf or is not concrete (e.g. missing) + * - EROFS if the device is not writeable + * - EBUSY start requested but the device is already being initialized + * - ESRCH cancel/suspend requested but device is not being initialized + * + * If the errlist is empty, then return value will be: + * - EINVAL if one or more arguments was invalid + * - Other spa_open failures + * - 0 if the operation succeeded + */ +int +lzc_initialize(const char *poolname, pool_initialize_func_t cmd_type, + nvlist_t *vdevs, nvlist_t **errlist) +{ + int error; + nvlist_t *args = fnvlist_alloc(); + fnvlist_add_uint64(args, ZPOOL_INITIALIZE_COMMAND, (uint64_t)cmd_type); + fnvlist_add_nvlist(args, ZPOOL_INITIALIZE_VDEVS, vdevs); + + error = lzc_ioctl(ZFS_IOC_POOL_INITIALIZE, poolname, args, errlist); + + fnvlist_free(args); + + return (error); +} Modified: vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h ============================================================================== --- vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h Tue Jul 31 18:49:07 2018 (r336991) @@ -31,7 +31,9 @@ #include #include #include +#include + #ifdef __cplusplus extern "C" { #endif @@ -56,6 +58,8 @@ int lzc_destroy_snaps(nvlist_t *, boolean_t, nvlist_t int lzc_bookmark(nvlist_t *, nvlist_t **); int lzc_get_bookmarks(const char *, nvlist_t *, nvlist_t **); int lzc_destroy_bookmarks(nvlist_t *, nvlist_t **); +int lzc_initialize(const char *, pool_initialize_func_t, nvlist_t *, + nvlist_t **); int lzc_snaprange_space(const char *, const char *, uint64_t *); Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Tue Jul 31 18:32:57 2018 (r336990) +++ vendor/illumos/dist/man/man1m/zpool.1m Tue Jul 31 18:49:07 2018 (r336991) @@ -105,6 +105,11 @@ .Ar pool Ns | Ns Ar id .Op Ar newpool .Nm +.Cm initialize +.Op Fl cs +.Ar pool +.Op Ar device Ns ... +.Nm .Cm iostat .Op Fl v .Op Fl T Sy u Ns | Ns Sy d @@ -1333,6 +1338,32 @@ mounting option is enabled. In this case, the checkpointed state of the pool is opened and an administrator can see how the pool would look like if they were to fully rewind. +.El +.It Xo +.Nm +.Cm initialize +.Op Fl cs +.Ar pool +.Op Ar device Ns ... +.Xc +Begins initializing by writing to all unallocated regions on the specified +devices, or all eligible devices in the pool if no individual devices are +specified. +Only leaf data or log devices may be initialized. +.Bl -tag -width Ds +.It Fl c, -cancel +Cancel initializing on the specified devices, or all eligible devices if none +are specified. +If one or more target devices are invalid or are not currently being +initialized, the command will fail and no cancellation will occur on any device. +.It Fl s -suspend +Suspend initializing on the specified devices, or all eligible devices if none +are specified. +If one or more target devices are invalid or are not currently being +initialized, the command will fail and no suspension will occur on any device. +Initializing can then be resumed by running +.Nm zpool Cm initialize +with no flags on the relevant target devices. .El .It Xo .Nm From owner-svn-src-vendor@freebsd.org Tue Jul 31 18:49:12 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 298D9106173C; Tue, 31 Jul 2018 18:49:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D1D5E76C18; Tue, 31 Jul 2018 18:49:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B41C3177BF; Tue, 31 Jul 2018 18:49:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VInBCk065559; Tue, 31 Jul 2018 18:49:11 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VIn7Xt065532; Tue, 31 Jul 2018 18:49:07 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807311849.w6VIn7Xt065532@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 18:49:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r336991 - vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/cmd/zpool vendo... X-SVN-Commit-Revision: 336991 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 18:49:12 -0000 Author: mav Date: Tue Jul 31 18:49:07 2018 New Revision: 336991 URL: https://svnweb.freebsd.org/changeset/base/336991 Log: 9102 zfs should be able to initialize storage devices The first access to a disk block can incur a performance penalty on some platforms (e.g. AWS's EBS, VMware VMDKs). Therefore it is recommended that volumes be "thick provisioned", where supported by the platform (VMware). Thick provisioning is time consuming and often is ignored. If the thick provision step is omitted, customers will see suboptimal performance until we have written to all parts of the LUN. ZFS should be able to initialize any unused storage to remove any first-write penalty that exists. illumos/illumos-gate@094e47e980b0796b94b1b8f51f462a64d246e516 Reviewed by: John Wren Kennedy Reviewed by: Matthew Ahrens Reviewed by: Pavel Zakharov Reviewed by: Prakash Surya Approved by: Richard Lowe Modified: vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_priority.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/lib/libzfs/common/libzfs_util.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.c vendor/illumos/dist/lib/libzfs_core/common/libzfs_core.h vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor-sys/illumos/dist/uts/common/Makefile.files ============================================================================== --- vendor-sys/illumos/dist/uts/common/Makefile.files Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/Makefile.files Tue Jul 31 18:49:07 2018 (r336991) @@ -1398,6 +1398,7 @@ ZFS_COMMON_OBJS += \ vdev.o \ vdev_cache.o \ vdev_file.o \ + vdev_initialize.o \ vdev_label.o \ vdev_mirror.o \ vdev_missing.o \ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Tue Jul 31 18:49:07 2018 (r336991) @@ -635,6 +635,8 @@ metaslab_group_create(metaslab_class_t *mc, vdev_t *vd mg = kmem_zalloc(sizeof (metaslab_group_t), KM_SLEEP); mutex_init(&mg->mg_lock, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&mg->mg_ms_initialize_lock, NULL, MUTEX_DEFAULT, NULL); + cv_init(&mg->mg_ms_initialize_cv, NULL, CV_DEFAULT, NULL); mg->mg_primaries = kmem_zalloc(allocators * sizeof (metaslab_t *), KM_SLEEP); mg->mg_secondaries = kmem_zalloc(allocators * sizeof (metaslab_t *), @@ -681,6 +683,8 @@ metaslab_group_destroy(metaslab_group_t *mg) kmem_free(mg->mg_secondaries, mg->mg_allocators * sizeof (metaslab_t *)); mutex_destroy(&mg->mg_lock); + mutex_destroy(&mg->mg_ms_initialize_lock); + cv_destroy(&mg->mg_ms_initialize_cv); for (int i = 0; i < mg->mg_allocators; i++) { refcount_destroy(&mg->mg_alloc_queue_depth[i]); @@ -1541,6 +1545,7 @@ metaslab_init(metaslab_group_t *mg, uint64_t id, uint6 mutex_init(&ms->ms_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&ms->ms_sync_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&ms->ms_load_cv, NULL, CV_DEFAULT, NULL); + ms->ms_id = id; ms->ms_start = id << vd->vdev_ms_shift; ms->ms_size = 1ULL << vd->vdev_ms_shift; @@ -2717,6 +2722,7 @@ metaslab_sync_done(metaslab_t *msp, uint64_t txg) * from it in 'metaslab_unload_delay' txgs, then unload it. */ if (msp->ms_loaded && + msp->ms_initializing == 0 && msp->ms_selected_txg + metaslab_unload_delay < txg) { for (int t = 1; t < TXG_CONCURRENT_STATES; t++) { VERIFY0(range_tree_space( @@ -2966,6 +2972,7 @@ metaslab_block_alloc(metaslab_t *msp, uint64_t size, u metaslab_class_t *mc = msp->ms_group->mg_class; VERIFY(!msp->ms_condensing); + VERIFY0(msp->ms_initializing); start = mc->mc_ops->msop_alloc(msp, size); if (start != -1ULL) { @@ -3026,9 +3033,10 @@ find_valid_metaslab(metaslab_group_t *mg, uint64_t act } /* - * If the selected metaslab is condensing, skip it. + * If the selected metaslab is condensing or being + * initialized, skip it. */ - if (msp->ms_condensing) + if (msp->ms_condensing || msp->ms_initializing > 0) continue; *was_active = msp->ms_allocator != -1; @@ -3193,11 +3201,20 @@ metaslab_group_alloc_normal(metaslab_group_t *mg, zio_ /* * If this metaslab is currently condensing then pick again as * we can't manipulate this metaslab until it's committed - * to disk. + * to disk. If this metaslab is being initialized, we shouldn't + * allocate from it since the allocated region might be + * overwritten after allocation. */ if (msp->ms_condensing) { metaslab_trace_add(zal, mg, msp, asize, d, TRACE_CONDENSING, allocator); + metaslab_passivate(msp, msp->ms_weight & + ~METASLAB_ACTIVE_MASK); + mutex_exit(&msp->ms_lock); + continue; + } else if (msp->ms_initializing > 0) { + metaslab_trace_add(zal, mg, msp, asize, d, + TRACE_INITIALIZING, allocator); metaslab_passivate(msp, msp->ms_weight & ~METASLAB_ACTIVE_MASK); mutex_exit(&msp->ms_lock); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Tue Jul 31 18:49:07 2018 (r336991) @@ -54,6 +54,7 @@ #include #include #include +#include #include #include #include @@ -413,8 +414,9 @@ spa_prop_get(spa_t *spa, nvlist_t **nvp) dp = spa_get_dsl(spa); dsl_pool_config_enter(dp, FTAG); - if (err = dsl_dataset_hold_obj(dp, - za.za_first_integer, FTAG, &ds)) { + err = dsl_dataset_hold_obj(dp, + za.za_first_integer, FTAG, &ds); + if (err != 0) { dsl_pool_config_exit(dp, FTAG); break; } @@ -569,7 +571,8 @@ spa_prop_validate(spa_t *spa, nvlist_t *props) break; } - if (error = dmu_objset_hold(strval, FTAG, &os)) + error = dmu_objset_hold(strval, FTAG, &os); + if (error != 0) break; /* @@ -1155,8 +1158,10 @@ spa_activate(spa_t *spa, int mode) spa_create_zio_taskqs(spa); } - for (size_t i = 0; i < TXG_SIZE; i++) - spa->spa_txg_zio[i] = zio_root(spa, NULL, NULL, 0); + for (size_t i = 0; i < TXG_SIZE; i++) { + spa->spa_txg_zio[i] = zio_root(spa, NULL, NULL, + ZIO_FLAG_CANFAIL); + } list_create(&spa->spa_config_dirty_list, sizeof (vdev_t), offsetof(vdev_t, vdev_config_dirty_node)); @@ -1315,6 +1320,11 @@ spa_unload(spa_t *spa) */ spa_async_suspend(spa); + if (spa->spa_root_vdev) { + vdev_initialize_stop_all(spa->spa_root_vdev, + VDEV_INITIALIZE_ACTIVE); + } + /* * Stop syncing. */ @@ -1330,10 +1340,10 @@ spa_unload(spa_t *spa) * calling taskq_wait(mg_taskq). */ if (spa->spa_root_vdev != NULL) { - spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); + spa_config_enter(spa, SCL_ALL, spa, RW_WRITER); for (int c = 0; c < spa->spa_root_vdev->vdev_children; c++) vdev_metaslab_fini(spa->spa_root_vdev->vdev_child[c]); - spa_config_exit(spa, SCL_ALL, FTAG); + spa_config_exit(spa, SCL_ALL, spa); } /* @@ -1367,7 +1377,7 @@ spa_unload(spa_t *spa) bpobj_close(&spa->spa_deferred_bpobj); - spa_config_enter(spa, SCL_ALL, FTAG, RW_WRITER); + spa_config_enter(spa, SCL_ALL, spa, RW_WRITER); /* * Close all vdevs. @@ -1429,7 +1439,7 @@ spa_unload(spa_t *spa) spa->spa_comment = NULL; } - spa_config_exit(spa, SCL_ALL, FTAG); + spa_config_exit(spa, SCL_ALL, spa); } /* @@ -3866,6 +3876,10 @@ spa_load_impl(spa_t *spa, spa_import_type_t type, char spa_restart_removal(spa); spa_spawn_aux_threads(spa); + + spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); + vdev_initialize_restart(spa->spa_root_vdev); + spa_config_exit(spa, SCL_CONFIG, FTAG); } spa_load_note(spa, "LOADED"); @@ -5347,6 +5361,7 @@ spa_export_common(char *pool, int new_state, nvlist_t * in which case we can modify its state. */ if (spa->spa_state != POOL_STATE_UNINITIALIZED && spa->spa_sync_on) { + /* * Objsets may be open only because they're dirty, so we * have to force it to sync before checking spa_refcnt. @@ -5381,6 +5396,18 @@ spa_export_common(char *pool, int new_state, nvlist_t } /* + * We're about to export or destroy this pool. Make sure + * we stop all initializtion activity here before we + * set the spa_final_txg. This will ensure that all + * dirty data resulting from the initialization is + * committed to disk before we unload the pool. + */ + if (spa->spa_root_vdev != NULL) { + vdev_initialize_stop_all(spa->spa_root_vdev, + VDEV_INITIALIZE_ACTIVE); + } + + /* * We want this to be reflected on every label, * so mark them all dirty. spa_unload() will do the * final sync that pushes these changes out. @@ -6070,6 +6097,86 @@ spa_vdev_detach(spa_t *spa, uint64_t guid, uint64_t pg return (error); } +int +spa_vdev_initialize(spa_t *spa, uint64_t guid, uint64_t cmd_type) +{ + /* + * We hold the namespace lock through the whole function + * to prevent any changes to the pool while we're starting or + * stopping initialization. The config and state locks are held so that + * we can properly assess the vdev state before we commit to + * the initializing operation. + */ + mutex_enter(&spa_namespace_lock); + spa_config_enter(spa, SCL_CONFIG | SCL_STATE, FTAG, RW_READER); + + /* Look up vdev and ensure it's a leaf. */ + vdev_t *vd = spa_lookup_by_guid(spa, guid, B_FALSE); + if (vd == NULL || vd->vdev_detached) { + spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); + mutex_exit(&spa_namespace_lock); + return (SET_ERROR(ENODEV)); + } else if (!vd->vdev_ops->vdev_op_leaf || !vdev_is_concrete(vd)) { + spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); + mutex_exit(&spa_namespace_lock); + return (SET_ERROR(EINVAL)); + } else if (!vdev_writeable(vd)) { + spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); + mutex_exit(&spa_namespace_lock); + return (SET_ERROR(EROFS)); + } + mutex_enter(&vd->vdev_initialize_lock); + spa_config_exit(spa, SCL_CONFIG | SCL_STATE, FTAG); + + /* + * When we activate an initialize action we check to see + * if the vdev_initialize_thread is NULL. We do this instead + * of using the vdev_initialize_state since there might be + * a previous initialization process which has completed but + * the thread is not exited. + */ + if (cmd_type == POOL_INITIALIZE_DO && + (vd->vdev_initialize_thread != NULL || + vd->vdev_top->vdev_removing)) { + mutex_exit(&vd->vdev_initialize_lock); + mutex_exit(&spa_namespace_lock); + return (SET_ERROR(EBUSY)); + } else if (cmd_type == POOL_INITIALIZE_CANCEL && + (vd->vdev_initialize_state != VDEV_INITIALIZE_ACTIVE && + vd->vdev_initialize_state != VDEV_INITIALIZE_SUSPENDED)) { + mutex_exit(&vd->vdev_initialize_lock); + mutex_exit(&spa_namespace_lock); + return (SET_ERROR(ESRCH)); + } else if (cmd_type == POOL_INITIALIZE_SUSPEND && + vd->vdev_initialize_state != VDEV_INITIALIZE_ACTIVE) { + mutex_exit(&vd->vdev_initialize_lock); + mutex_exit(&spa_namespace_lock); + return (SET_ERROR(ESRCH)); + } + + switch (cmd_type) { + case POOL_INITIALIZE_DO: + vdev_initialize(vd); + break; + case POOL_INITIALIZE_CANCEL: + vdev_initialize_stop(vd, VDEV_INITIALIZE_CANCELED); + break; + case POOL_INITIALIZE_SUSPEND: + vdev_initialize_stop(vd, VDEV_INITIALIZE_SUSPENDED); + break; + default: + panic("invalid cmd_type %llu", (unsigned long long)cmd_type); + } + mutex_exit(&vd->vdev_initialize_lock); + + /* Sync out the initializing state */ + txg_wait_synced(spa->spa_dsl_pool, 0); + mutex_exit(&spa_namespace_lock); + + return (0); +} + + /* * Split a set of devices from their mirrors, and create a new pool from them. */ @@ -6277,6 +6384,19 @@ spa_vdev_split_mirror(spa_t *spa, char *newname, nvlis spa_activate(newspa, spa_mode_global); spa_async_suspend(newspa); + for (c = 0; c < children; c++) { + if (vml[c] != NULL) { + /* + * Temporarily stop the initializing activity. We set + * the state to ACTIVE so that we know to resume + * the initializing once the split has completed. + */ + mutex_enter(&vml[c]->vdev_initialize_lock); + vdev_initialize_stop(vml[c], VDEV_INITIALIZE_ACTIVE); + mutex_exit(&vml[c]->vdev_initialize_lock); + } + } + newspa->spa_config_source = SPA_CONFIG_SRC_SPLIT; /* create the new pool from the disks of the original pool */ @@ -6364,6 +6484,10 @@ out: if (vml[c] != NULL) vml[c]->vdev_offline = B_FALSE; } + + /* restart initializing disks as necessary */ + spa_async_request(spa, SPA_ASYNC_INITIALIZE_RESTART); + vdev_reopen(spa->spa_root_vdev); nvlist_free(spa->spa_config_splitting); @@ -6739,6 +6863,14 @@ spa_async_thread(void *arg) if (tasks & SPA_ASYNC_RESILVER) dsl_resilver_restart(spa->spa_dsl_pool, 0); + if (tasks & SPA_ASYNC_INITIALIZE_RESTART) { + mutex_enter(&spa_namespace_lock); + spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); + vdev_initialize_restart(spa->spa_root_vdev); + spa_config_exit(spa, SCL_CONFIG, FTAG); + mutex_exit(&spa_namespace_lock); + } + /* * Let the world know that we're done. */ @@ -7384,8 +7516,9 @@ spa_sync(spa_t *spa, uint64_t txg) * Wait for i/os issued in open context that need to complete * before this txg syncs. */ - VERIFY0(zio_wait(spa->spa_txg_zio[txg & TXG_MASK])); - spa->spa_txg_zio[txg & TXG_MASK] = zio_root(spa, NULL, NULL, 0); + (void) zio_wait(spa->spa_txg_zio[txg & TXG_MASK]); + spa->spa_txg_zio[txg & TXG_MASK] = zio_root(spa, NULL, NULL, + ZIO_FLAG_CANFAIL); /* * Lock out configuration changes. @@ -7674,7 +7807,8 @@ spa_sync(spa_t *spa, uint64_t txg) /* * Update usable space statistics. */ - while (vd = txg_list_remove(&spa->spa_vdev_txg_list, TXG_CLEAN(txg))) + while ((vd = txg_list_remove(&spa->spa_vdev_txg_list, TXG_CLEAN(txg))) + != NULL) vdev_sync_done(vd, txg); spa_update_dspace(spa); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Tue Jul 31 18:49:07 2018 (r336991) @@ -39,6 +39,7 @@ #include #include #include +#include #include #include #include @@ -1196,6 +1197,12 @@ spa_vdev_config_exit(spa_t *spa, vdev_t *vd, uint64_t if (vd != NULL) { ASSERT(!vd->vdev_detached || vd->vdev_dtl_sm == NULL); + if (vd->vdev_ops->vdev_op_leaf) { + mutex_enter(&vd->vdev_initialize_lock); + vdev_initialize_stop(vd, VDEV_INITIALIZE_CANCELED); + mutex_exit(&vd->vdev_initialize_lock); + } + spa_config_enter(spa, SCL_ALL, spa, RW_WRITER); vdev_free(vd); spa_config_exit(spa, SCL_ALL, spa); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/metaslab_impl.h Tue Jul 31 18:49:07 2018 (r336991) @@ -68,7 +68,8 @@ typedef enum trace_alloc_type { TRACE_GROUP_FAILURE = -5ULL, TRACE_ENOSPC = -6ULL, TRACE_CONDENSING = -7ULL, - TRACE_VDEV_ERROR = -8ULL + TRACE_VDEV_ERROR = -8ULL, + TRACE_INITIALIZING = -9ULL } trace_alloc_type_t; #define METASLAB_WEIGHT_PRIMARY (1ULL << 63) @@ -270,6 +271,11 @@ struct metaslab_group { uint64_t mg_failed_allocations; uint64_t mg_fragmentation; uint64_t mg_histogram[RANGE_TREE_HISTOGRAM_SIZE]; + + int mg_ms_initializing; + boolean_t mg_initialize_updating; + kmutex_t mg_ms_initialize_lock; + kcondvar_t mg_ms_initialize_cv; }; /* @@ -359,6 +365,8 @@ struct metaslab { boolean_t ms_condensing; /* condensing? */ boolean_t ms_condense_wanted; uint64_t ms_condense_checked_txg; + + uint64_t ms_initializing; /* leaves initializing this ms */ /* * We must hold both ms_lock and ms_group->mg_lock in order to Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Tue Jul 31 18:49:07 2018 (r336991) @@ -650,6 +650,7 @@ extern int spa_scan_get_stats(spa_t *spa, pool_scan_st #define SPA_ASYNC_AUTOEXPAND 0x20 #define SPA_ASYNC_REMOVE_DONE 0x40 #define SPA_ASYNC_REMOVE_STOP 0x80 +#define SPA_ASYNC_INITIALIZE_RESTART 0x100 /* * Controls the behavior of spa_vdev_remove(). @@ -665,6 +666,7 @@ extern int spa_vdev_detach(spa_t *spa, uint64_t guid, int replace_done); extern int spa_vdev_remove(spa_t *spa, uint64_t guid, boolean_t unspare); extern boolean_t spa_vdev_remove_active(spa_t *spa); +extern int spa_vdev_initialize(spa_t *spa, uint64_t guid, uint64_t cmd_type); extern int spa_vdev_setpath(spa_t *spa, uint64_t guid, const char *newpath); extern int spa_vdev_setfru(spa_t *spa, uint64_t guid, const char *newfru); extern int spa_vdev_split_mirror(spa_t *spa, char *newname, nvlist_t *config, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h Tue Jul 31 18:49:07 2018 (r336991) @@ -79,6 +79,12 @@ typedef void vdev_remap_cb_t(uint64_t inner_offset, vd uint64_t offset, uint64_t size, void *arg); typedef void vdev_remap_func_t(vdev_t *vd, uint64_t offset, uint64_t size, vdev_remap_cb_t callback, void *arg); +/* + * Given a target vdev, translates the logical range "in" to the physical + * range "res" + */ +typedef void vdev_xlation_func_t(vdev_t *cvd, const range_seg_t *in, + range_seg_t *res); typedef struct vdev_ops { vdev_open_func_t *vdev_op_open; @@ -90,6 +96,11 @@ typedef struct vdev_ops { vdev_hold_func_t *vdev_op_hold; vdev_rele_func_t *vdev_op_rele; vdev_remap_func_t *vdev_op_remap; + /* + * For translating ranges from non-leaf vdevs (e.g. raidz) to leaves. + * Used when initializing vdevs. Isn't used by leaf ops. + */ + vdev_xlation_func_t *vdev_op_xlate; char vdev_op_type[16]; boolean_t vdev_op_leaf; } vdev_ops_t; @@ -231,7 +242,25 @@ struct vdev { /* pool checkpoint related */ space_map_t *vdev_checkpoint_sm; /* contains reserved blocks */ + + boolean_t vdev_initialize_exit_wanted; + vdev_initializing_state_t vdev_initialize_state; + kthread_t *vdev_initialize_thread; + /* Protects vdev_initialize_thread and vdev_initialize_state. */ + kmutex_t vdev_initialize_lock; + kcondvar_t vdev_initialize_cv; + uint64_t vdev_initialize_offset[TXG_SIZE]; + uint64_t vdev_initialize_last_offset; + range_tree_t *vdev_initialize_tree; /* valid while initializing */ + uint64_t vdev_initialize_bytes_est; + uint64_t vdev_initialize_bytes_done; + time_t vdev_initialize_action_time; /* start and end time */ + /* for limiting outstanding I/Os */ + kmutex_t vdev_initialize_io_lock; + kcondvar_t vdev_initialize_io_cv; + uint64_t vdev_initialize_inflight; + /* * Values stored in the config for an indirect or removing vdev. */ @@ -434,6 +463,8 @@ extern vdev_ops_t vdev_indirect_ops; /* * Common size functions */ +extern void vdev_default_xlate(vdev_t *vd, const range_seg_t *in, + range_seg_t *out); extern uint64_t vdev_default_asize(vdev_t *vd, uint64_t psize); extern uint64_t vdev_get_min_asize(vdev_t *vd); extern void vdev_set_min_asize(vdev_t *vd); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_priority.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_priority.h Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_priority.h Tue Jul 31 18:49:07 2018 (r336991) @@ -13,7 +13,7 @@ * CDDL HEADER END */ /* - * Copyright (c) 2014 by Delphix. All rights reserved. + * Copyright (c) 2014, 2016 by Delphix. All rights reserved. */ #ifndef _ZIO_PRIORITY_H #define _ZIO_PRIORITY_H @@ -29,6 +29,7 @@ typedef enum zio_priority { ZIO_PRIORITY_ASYNC_WRITE, /* spa_sync() */ ZIO_PRIORITY_SCRUB, /* asynchronous scrub/resilver reads */ ZIO_PRIORITY_REMOVAL, /* reads/writes for vdev removal */ + ZIO_PRIORITY_INITIALIZING, /* initializing I/O */ ZIO_PRIORITY_NUM_QUEUEABLE, ZIO_PRIORITY_NOW /* non-queued i/os (e.g. free) */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Tue Jul 31 18:49:07 2018 (r336991) @@ -49,6 +49,7 @@ #include #include #include +#include /* * Virtual device management. @@ -183,6 +184,14 @@ vdev_getops(const char *type) return (ops); } +/* ARGSUSED */ +void +vdev_default_xlate(vdev_t *vd, const range_seg_t *in, range_seg_t *res) +{ + res->rs_start = in->rs_start; + res->rs_end = in->rs_end; +} + /* * Default asize function: return the MAX of psize with the asize of * all children. This is what's used by anything other than RAID-Z. @@ -453,6 +462,11 @@ vdev_alloc_common(spa_t *spa, uint_t id, uint64_t guid mutex_init(&vd->vdev_stat_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&vd->vdev_probe_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&vd->vdev_queue_lock, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&vd->vdev_initialize_lock, NULL, MUTEX_DEFAULT, NULL); + mutex_init(&vd->vdev_initialize_io_lock, NULL, MUTEX_DEFAULT, NULL); + cv_init(&vd->vdev_initialize_cv, NULL, CV_DEFAULT, NULL); + cv_init(&vd->vdev_initialize_io_cv, NULL, CV_DEFAULT, NULL); + for (int t = 0; t < DTL_TYPES; t++) { vd->vdev_dtl[t] = range_tree_create(NULL, NULL); } @@ -725,6 +739,7 @@ void vdev_free(vdev_t *vd) { spa_t *spa = vd->vdev_spa; + ASSERT3P(vd->vdev_initialize_thread, ==, NULL); /* * vdev_free() implies closing the vdev first. This is simpler than @@ -743,6 +758,7 @@ vdev_free(vdev_t *vd) ASSERT(vd->vdev_child == NULL); ASSERT(vd->vdev_guid_sum == vd->vdev_guid); + ASSERT(vd->vdev_initialize_thread == NULL); /* * Discard allocation state. @@ -815,6 +831,10 @@ vdev_free(vdev_t *vd) mutex_destroy(&vd->vdev_dtl_lock); mutex_destroy(&vd->vdev_stat_lock); mutex_destroy(&vd->vdev_probe_lock); + mutex_destroy(&vd->vdev_initialize_lock); + mutex_destroy(&vd->vdev_initialize_io_lock); + cv_destroy(&vd->vdev_initialize_io_cv); + cv_destroy(&vd->vdev_initialize_cv); if (vd == spa->spa_root_vdev) spa->spa_root_vdev = NULL; @@ -2841,7 +2861,8 @@ vdev_sync_done(vdev_t *vd, uint64_t txg) ASSERT(vdev_is_concrete(vd)); - while (msp = txg_list_remove(&vd->vdev_ms_list, TXG_CLEAN(txg))) + while ((msp = txg_list_remove(&vd->vdev_ms_list, TXG_CLEAN(txg))) + != NULL) metaslab_sync_done(msp, txg); if (reassess) @@ -3067,6 +3088,15 @@ vdev_online(spa_t *spa, uint64_t guid, uint64_t flags, spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE); } + /* Restart initializing if necessary */ + mutex_enter(&vd->vdev_initialize_lock); + if (vdev_writeable(vd) && + vd->vdev_initialize_thread == NULL && + vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE) { + (void) vdev_initialize(vd); + } + mutex_exit(&vd->vdev_initialize_lock); + if (wasoffline || (oldstate < VDEV_STATE_DEGRADED && vd->vdev_state >= VDEV_STATE_DEGRADED)) @@ -3361,8 +3391,18 @@ vdev_get_stats(vdev_t *vd, vdev_stat_t *vs) vs->vs_timestamp = gethrtime() - vs->vs_timestamp; vs->vs_state = vd->vdev_state; vs->vs_rsize = vdev_get_min_asize(vd); - if (vd->vdev_ops->vdev_op_leaf) + if (vd->vdev_ops->vdev_op_leaf) { vs->vs_rsize += VDEV_LABEL_START_SIZE + VDEV_LABEL_END_SIZE; + /* + * Report intializing progress. Since we don't have the + * initializing locks held, this is only an estimate (although a + * fairly accurate one). + */ + vs->vs_initialize_bytes_done = vd->vdev_initialize_bytes_done; + vs->vs_initialize_bytes_est = vd->vdev_initialize_bytes_est; + vs->vs_initialize_state = vd->vdev_initialize_state; + vs->vs_initialize_action_time = vd->vdev_initialize_action_time; + } /* * Report expandable space on top-level, non-auxillary devices only. * The expandable space is reported in terms of metaslab sized units Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c Tue Jul 31 18:49:07 2018 (r336991) @@ -840,6 +840,7 @@ vdev_ops_t vdev_disk_ops = { vdev_disk_hold, vdev_disk_rele, NULL, + vdev_default_xlate, VDEV_TYPE_DISK, /* name of this vdev type */ B_TRUE /* leaf vdev */ }; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c Tue Jul 31 18:49:07 2018 (r336991) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2016 by Delphix. All rights reserved. */ #include @@ -263,6 +263,7 @@ vdev_ops_t vdev_file_ops = { vdev_file_hold, vdev_file_rele, NULL, + vdev_default_xlate, VDEV_TYPE_FILE, /* name of this vdev type */ B_TRUE /* leaf vdev */ }; @@ -282,6 +283,7 @@ vdev_ops_t vdev_disk_ops = { vdev_file_hold, vdev_file_rele, NULL, + vdev_default_xlate, VDEV_TYPE_DISK, /* name of this vdev type */ B_TRUE /* leaf vdev */ }; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_indirect.c Tue Jul 31 18:49:07 2018 (r336991) @@ -1628,6 +1628,7 @@ vdev_ops_t vdev_indirect_ops = { NULL, NULL, vdev_indirect_remap, + NULL, VDEV_TYPE_INDIRECT, /* name of this vdev type */ B_FALSE /* leaf vdev */ }; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c Tue Jul 31 18:49:07 2018 (r336991) @@ -564,6 +564,7 @@ vdev_ops_t vdev_mirror_ops = { NULL, NULL, NULL, + vdev_default_xlate, VDEV_TYPE_MIRROR, /* name of this vdev type */ B_FALSE /* not a leaf vdev */ }; @@ -578,6 +579,7 @@ vdev_ops_t vdev_replacing_ops = { NULL, NULL, NULL, + vdev_default_xlate, VDEV_TYPE_REPLACING, /* name of this vdev type */ B_FALSE /* not a leaf vdev */ }; @@ -592,6 +594,7 @@ vdev_ops_t vdev_spare_ops = { NULL, NULL, NULL, + vdev_default_xlate, VDEV_TYPE_SPARE, /* name of this vdev type */ B_FALSE /* not a leaf vdev */ }; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_missing.c Tue Jul 31 18:49:07 2018 (r336991) @@ -24,7 +24,7 @@ */ /* - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. */ /* @@ -89,6 +89,7 @@ vdev_ops_t vdev_missing_ops = { NULL, NULL, NULL, + NULL, VDEV_TYPE_MISSING, /* name of this vdev type */ B_TRUE /* leaf vdev */ }; @@ -99,6 +100,7 @@ vdev_ops_t vdev_hole_ops = { vdev_default_asize, vdev_missing_io_start, vdev_missing_io_done, + NULL, NULL, NULL, NULL, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c Tue Jul 31 18:49:07 2018 (r336991) @@ -150,6 +150,8 @@ uint32_t zfs_vdev_scrub_min_active = 1; uint32_t zfs_vdev_scrub_max_active = 2; uint32_t zfs_vdev_removal_min_active = 1; uint32_t zfs_vdev_removal_max_active = 2; +uint32_t zfs_vdev_initializing_min_active = 1; +uint32_t zfs_vdev_initializing_max_active = 1; /* * When the pool has less than zfs_vdev_async_write_active_min_dirty_percent @@ -407,6 +409,8 @@ vdev_queue_class_min_active(zio_priority_t p) return (zfs_vdev_scrub_min_active); case ZIO_PRIORITY_REMOVAL: return (zfs_vdev_removal_min_active); + case ZIO_PRIORITY_INITIALIZING: + return (zfs_vdev_initializing_min_active); default: panic("invalid priority %u", p); return (0); @@ -468,6 +472,8 @@ vdev_queue_class_max_active(spa_t *spa, zio_priority_t return (zfs_vdev_scrub_max_active); case ZIO_PRIORITY_REMOVAL: return (zfs_vdev_removal_max_active); + case ZIO_PRIORITY_INITIALIZING: + return (zfs_vdev_initializing_max_active); default: panic("invalid priority %u", p); return (0); @@ -688,8 +694,8 @@ again: } /* - * For LBA-ordered queues (async / scrub), issue the i/o which follows - * the most recently issued i/o in LBA (offset) order. + * For LBA-ordered queues (async / scrub / initializing), issue the + * i/o which follows the most recently issued i/o in LBA (offset) order. * * For FIFO queues (sync), issue the i/o with the lowest timestamp. */ @@ -745,13 +751,15 @@ vdev_queue_io(zio_t *zio) if (zio->io_priority != ZIO_PRIORITY_SYNC_READ && zio->io_priority != ZIO_PRIORITY_ASYNC_READ && zio->io_priority != ZIO_PRIORITY_SCRUB && - zio->io_priority != ZIO_PRIORITY_REMOVAL) + zio->io_priority != ZIO_PRIORITY_REMOVAL && + zio->io_priority != ZIO_PRIORITY_INITIALIZING) zio->io_priority = ZIO_PRIORITY_ASYNC_READ; } else { ASSERT(zio->io_type == ZIO_TYPE_WRITE); if (zio->io_priority != ZIO_PRIORITY_SYNC_WRITE && zio->io_priority != ZIO_PRIORITY_ASYNC_WRITE && - zio->io_priority != ZIO_PRIORITY_REMOVAL) + zio->io_priority != ZIO_PRIORITY_REMOVAL && + zio->io_priority != ZIO_PRIORITY_INITIALIZING) zio->io_priority = ZIO_PRIORITY_ASYNC_WRITE; } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c Tue Jul 31 18:49:07 2018 (r336991) @@ -38,6 +38,10 @@ #include #include +#ifdef ZFS_DEBUG +#include /* vdev_xlate testing */ +#endif + /* * Virtual device vector for RAID-Z. * @@ -1884,6 +1888,39 @@ vdev_raidz_child_done(zio_t *zio) rc->rc_skipped = 0; } +static void +vdev_raidz_io_verify(zio_t *zio, raidz_map_t *rm, int col) +{ +#ifdef ZFS_DEBUG + vdev_t *vd = zio->io_vd; + vdev_t *tvd = vd->vdev_top; + + range_seg_t logical_rs, physical_rs; + logical_rs.rs_start = zio->io_offset; + logical_rs.rs_end = logical_rs.rs_start + + vdev_raidz_asize(zio->io_vd, zio->io_size); + + raidz_col_t *rc = &rm->rm_col[col]; + vdev_t *cvd = vd->vdev_child[rc->rc_devidx]; + + vdev_xlate(cvd, &logical_rs, &physical_rs); + ASSERT3U(rc->rc_offset, ==, physical_rs.rs_start); + ASSERT3U(rc->rc_offset, <, physical_rs.rs_end); + /* + * It would be nice to assert that rs_end is equal + * to rc_offset + rc_size but there might be an + * optional I/O at the end that is not accounted in + * rc_size. + */ + if (physical_rs.rs_end > rc->rc_offset + rc->rc_size) { + ASSERT3U(physical_rs.rs_end, ==, rc->rc_offset + + rc->rc_size + (1 << tvd->vdev_ashift)); + } else { + ASSERT3U(physical_rs.rs_end, ==, rc->rc_offset + rc->rc_size); + } +#endif +} + /* * Start an IO operation on a RAIDZ VDev * @@ -1926,6 +1963,12 @@ vdev_raidz_io_start(zio_t *zio) for (c = 0; c < rm->rm_cols; c++) { rc = &rm->rm_col[c]; cvd = vd->vdev_child[rc->rc_devidx]; + + /* + * Verify physical to logical translation. + */ + vdev_raidz_io_verify(zio, rm, c); + zio_nowait(zio_vdev_child_io(zio, NULL, cvd, rc->rc_offset, rc->rc_abd, rc->rc_size, zio->io_type, zio->io_priority, 0, @@ -2555,6 +2598,37 @@ vdev_raidz_state_change(vdev_t *vd, int faulted, int d vdev_set_state(vd, B_FALSE, VDEV_STATE_HEALTHY, VDEV_AUX_NONE); } +static void +vdev_raidz_xlate(vdev_t *cvd, const range_seg_t *in, range_seg_t *res) +{ + vdev_t *raidvd = cvd->vdev_parent; + ASSERT(raidvd->vdev_ops == &vdev_raidz_ops); + + uint64_t width = raidvd->vdev_children; + uint64_t tgt_col = cvd->vdev_id; + uint64_t ashift = raidvd->vdev_top->vdev_ashift; + + /* make sure the offsets are block-aligned */ + ASSERT0(in->rs_start % (1 << ashift)); + ASSERT0(in->rs_end % (1 << ashift)); + uint64_t b_start = in->rs_start >> ashift; + uint64_t b_end = in->rs_end >> ashift; + + uint64_t start_row = 0; + if (b_start > tgt_col) /* avoid underflow */ + start_row = ((b_start - tgt_col - 1) / width) + 1; + + uint64_t end_row = 0; + if (b_end > tgt_col) + end_row = ((b_end - tgt_col - 1) / width) + 1; + + res->rs_start = start_row << ashift; + res->rs_end = end_row << ashift; + + ASSERT3U(res->rs_start, <=, in->rs_start); + ASSERT3U(res->rs_end - res->rs_start, <=, in->rs_end - in->rs_start); +} + vdev_ops_t vdev_raidz_ops = { vdev_raidz_open, vdev_raidz_close, @@ -2565,6 +2639,7 @@ vdev_ops_t vdev_raidz_ops = { NULL, NULL, NULL, + vdev_raidz_xlate, VDEV_TYPE_RAIDZ, /* name of this vdev type */ B_FALSE /* not a leaf vdev */ }; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Tue Jul 31 18:49:07 2018 (r336991) @@ -44,6 +44,7 @@ #include #include #include +#include /* * This file contains the necessary logic to remove vdevs from a @@ -1021,6 +1022,7 @@ vdev_remove_complete(spa_t *spa) txg_wait_synced(spa->spa_dsl_pool, 0); txg = spa_vdev_enter(spa); vdev_t *vd = vdev_lookup_top(spa, spa->spa_vdev_removal->svr_vdev_id); + ASSERT3P(vd->vdev_initialize_thread, ==, NULL); sysevent_t *ev = spa_event_create(spa, vd, NULL, ESC_ZFS_VDEV_REMOVE_DEV); @@ -1659,6 +1661,9 @@ spa_vdev_remove_log(vdev_t *vd, uint64_t *txg) /* Make sure these changes are sync'ed */ spa_vdev_config_exit(spa, NULL, *txg, 0, FTAG); + /* Stop initializing */ + (void) vdev_initialize_stop_all(vd, VDEV_INITIALIZE_CANCELED); + *txg = spa_vdev_config_enter(spa); sysevent_t *ev = spa_event_create(spa, vd, NULL, @@ -1819,6 +1824,13 @@ spa_vdev_remove_top(vdev_t *vd, uint64_t *txg) */ error = spa_reset_logs(spa); + /* + * We stop any initializing that is currently in progress but leave + * the state as "active". This will allow the initializing to resume + * if the removal is canceled sometime later. + */ + vdev_initialize_stop_all(vd, VDEV_INITIALIZE_ACTIVE); + *txg = spa_vdev_config_enter(spa); /* @@ -1830,6 +1842,7 @@ spa_vdev_remove_top(vdev_t *vd, uint64_t *txg) if (error != 0) { metaslab_group_activate(mg); + spa_async_request(spa, SPA_ASYNC_INITIALIZE_RESTART); return (error); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_root.c Tue Jul 31 18:49:07 2018 (r336991) @@ -24,7 +24,7 @@ */ /* - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2016 by Delphix. All rights reserved. */ #include @@ -146,6 +146,7 @@ vdev_ops_t vdev_root_ops = { NULL, /* io_start - not applicable to the root */ NULL, /* io_done - not applicable to the root */ vdev_root_state_change, + NULL, NULL, NULL, NULL, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Tue Jul 31 18:32:57 2018 (r336990) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Tue Jul 31 18:49:07 2018 (r336991) @@ -189,6 +189,8 @@ #include #include #include +#include +#include #include "zfs_namecheck.h" #include "zfs_prop.h" @@ -3707,6 +3709,80 @@ zfs_ioc_destroy(zfs_cmd_t *zc) } /* + * innvl: { + * vdevs: { + * guid 1, guid 2, ... + * }, + * func: POOL_INITIALIZE_{CANCEL|DO|SUSPEND} + * } + * + * outnvl: { + * [func: EINVAL (if provided command type didn't make sense)], + * [vdevs: { + * guid1: errno, (see function body for possible errnos) *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Tue Jul 31 19:23:28 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5459C1062AAE; Tue, 31 Jul 2018 19:23:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09BF8793F7; Tue, 31 Jul 2018 19:23:28 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DEDAC17EB2; Tue, 31 Jul 2018 19:23:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VJNRuV086560; Tue, 31 Jul 2018 19:23:27 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VJNR4h086559; Tue, 31 Jul 2018 19:23:27 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807311923.w6VJNR4h086559@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 19:23:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337001 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337001 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 19:23:28 -0000 Author: mav Date: Tue Jul 31 19:23:27 2018 New Revision: 337001 URL: https://svnweb.freebsd.org/changeset/base/337001 Log: Add new files missed in r336991. Added: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_initialize.h (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_initialize.c (contents, props changed) Added: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_initialize.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_initialize.h Tue Jul 31 19:23:27 2018 (r337001) @@ -0,0 +1,46 @@ +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ + +/* + * Copyright (c) 2016 by Delphix. All rights reserved. + */ + +#ifndef _SYS_VDEV_INITIALIZE_H +#define _SYS_VDEV_INITIALIZE_H + +#ifdef __cplusplus +extern "C" { +#endif + +extern void vdev_initialize(vdev_t *vd); +extern void vdev_initialize_stop(vdev_t *vd, + vdev_initializing_state_t tgt_state); +extern void vdev_initialize_stop_all(vdev_t *vd, + vdev_initializing_state_t tgt_state); +extern void vdev_initialize_restart(vdev_t *vd); +extern void vdev_xlate(vdev_t *vd, const range_seg_t *logical_rs, + range_seg_t *physical_rs); + +#ifdef __cplusplus +} +#endif + +#endif /* _SYS_VDEV_INITIALIZE_H */ Added: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_initialize.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_initialize.c Tue Jul 31 19:23:27 2018 (r337001) @@ -0,0 +1,791 @@ +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ + +/* + * Copyright (c) 2016 by Delphix. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * Maximum number of metaslabs per group that can be initialized + * simultaneously. + */ +int max_initialize_ms = 3; + +/* + * Value that is written to disk during initialization. + */ +uint64_t zfs_initialize_value = 0xdeadbeefdeadbeefULL; + +/* maximum number of I/Os outstanding per leaf vdev */ +int zfs_initialize_limit = 1; + +/* size of initializing writes; default 1MiB, see zfs_remove_max_segment */ +uint64_t zfs_initialize_chunk_size = 1024 * 1024; + +static boolean_t +vdev_initialize_should_stop(vdev_t *vd) +{ + return (vd->vdev_initialize_exit_wanted || !vdev_writeable(vd) || + vd->vdev_detached || vd->vdev_top->vdev_removing); +} + +static void +vdev_initialize_zap_update_sync(void *arg, dmu_tx_t *tx) +{ + /* + * We pass in the guid instead of the vdev_t since the vdev may + * have been freed prior to the sync task being processed. This + * happens when a vdev is detached as we call spa_config_vdev_exit(), + * stop the intializing thread, schedule the sync task, and free + * the vdev. Later when the scheduled sync task is invoked, it would + * find that the vdev has been freed. + */ + uint64_t guid = *(uint64_t *)arg; + uint64_t txg = dmu_tx_get_txg(tx); + kmem_free(arg, sizeof (uint64_t)); + + vdev_t *vd = spa_lookup_by_guid(tx->tx_pool->dp_spa, guid, B_FALSE); + if (vd == NULL || vd->vdev_top->vdev_removing || !vdev_is_concrete(vd)) + return; + + uint64_t last_offset = vd->vdev_initialize_offset[txg & TXG_MASK]; + vd->vdev_initialize_offset[txg & TXG_MASK] = 0; + + VERIFY(vd->vdev_leaf_zap != 0); + + objset_t *mos = vd->vdev_spa->spa_meta_objset; + + if (last_offset > 0) { + vd->vdev_initialize_last_offset = last_offset; + VERIFY0(zap_update(mos, vd->vdev_leaf_zap, + VDEV_LEAF_ZAP_INITIALIZE_LAST_OFFSET, + sizeof (last_offset), 1, &last_offset, tx)); + } + if (vd->vdev_initialize_action_time > 0) { + uint64_t val = (uint64_t)vd->vdev_initialize_action_time; + VERIFY0(zap_update(mos, vd->vdev_leaf_zap, + VDEV_LEAF_ZAP_INITIALIZE_ACTION_TIME, sizeof (val), + 1, &val, tx)); + } + + uint64_t initialize_state = vd->vdev_initialize_state; + VERIFY0(zap_update(mos, vd->vdev_leaf_zap, + VDEV_LEAF_ZAP_INITIALIZE_STATE, sizeof (initialize_state), 1, + &initialize_state, tx)); +} + +static void +vdev_initialize_change_state(vdev_t *vd, vdev_initializing_state_t new_state) +{ + ASSERT(MUTEX_HELD(&vd->vdev_initialize_lock)); + spa_t *spa = vd->vdev_spa; + + if (new_state == vd->vdev_initialize_state) + return; + + /* + * Copy the vd's guid, this will be freed by the sync task. + */ + uint64_t *guid = kmem_zalloc(sizeof (uint64_t), KM_SLEEP); + *guid = vd->vdev_guid; + + /* + * If we're suspending, then preserving the original start time. + */ + if (vd->vdev_initialize_state != VDEV_INITIALIZE_SUSPENDED) { + vd->vdev_initialize_action_time = gethrestime_sec(); + } + vd->vdev_initialize_state = new_state; + + dmu_tx_t *tx = dmu_tx_create_dd(spa_get_dsl(spa)->dp_mos_dir); + VERIFY0(dmu_tx_assign(tx, TXG_WAIT)); + dsl_sync_task_nowait(spa_get_dsl(spa), vdev_initialize_zap_update_sync, + guid, 2, ZFS_SPACE_CHECK_RESERVED, tx); + + switch (new_state) { + case VDEV_INITIALIZE_ACTIVE: + spa_history_log_internal(spa, "initialize", tx, + "vdev=%s activated", vd->vdev_path); + break; + case VDEV_INITIALIZE_SUSPENDED: + spa_history_log_internal(spa, "initialize", tx, + "vdev=%s suspended", vd->vdev_path); + break; + case VDEV_INITIALIZE_CANCELED: + spa_history_log_internal(spa, "initialize", tx, + "vdev=%s canceled", vd->vdev_path); + break; + case VDEV_INITIALIZE_COMPLETE: + spa_history_log_internal(spa, "initialize", tx, + "vdev=%s complete", vd->vdev_path); + break; + default: + panic("invalid state %llu", (unsigned long long)new_state); + } + + dmu_tx_commit(tx); +} + +static void +vdev_initialize_cb(zio_t *zio) +{ + vdev_t *vd = zio->io_vd; + mutex_enter(&vd->vdev_initialize_io_lock); + if (zio->io_error == ENXIO && !vdev_writeable(vd)) { + /* + * The I/O failed because the vdev was unavailable; roll the + * last offset back. (This works because spa_sync waits on + * spa_txg_zio before it runs sync tasks.) + */ + uint64_t *off = + &vd->vdev_initialize_offset[zio->io_txg & TXG_MASK]; + *off = MIN(*off, zio->io_offset); + } else { + /* + * Since initializing is best-effort, we ignore I/O errors and + * rely on vdev_probe to determine if the errors are more + * critical. + */ + if (zio->io_error != 0) + vd->vdev_stat.vs_initialize_errors++; + + vd->vdev_initialize_bytes_done += zio->io_orig_size; + } + ASSERT3U(vd->vdev_initialize_inflight, >, 0); + vd->vdev_initialize_inflight--; + cv_broadcast(&vd->vdev_initialize_io_cv); + mutex_exit(&vd->vdev_initialize_io_lock); + + spa_config_exit(vd->vdev_spa, SCL_STATE_ALL, vd); +} + +/* Takes care of physical writing and limiting # of concurrent ZIOs. */ +static int +vdev_initialize_write(vdev_t *vd, uint64_t start, uint64_t size, abd_t *data) +{ + spa_t *spa = vd->vdev_spa; + + /* Limit inflight initializing I/Os */ + mutex_enter(&vd->vdev_initialize_io_lock); + while (vd->vdev_initialize_inflight >= zfs_initialize_limit) { + cv_wait(&vd->vdev_initialize_io_cv, + &vd->vdev_initialize_io_lock); + } + vd->vdev_initialize_inflight++; + mutex_exit(&vd->vdev_initialize_io_lock); + + dmu_tx_t *tx = dmu_tx_create_dd(spa_get_dsl(spa)->dp_mos_dir); + VERIFY0(dmu_tx_assign(tx, TXG_WAIT)); + uint64_t txg = dmu_tx_get_txg(tx); + + spa_config_enter(spa, SCL_STATE_ALL, vd, RW_READER); + mutex_enter(&vd->vdev_initialize_lock); + + if (vd->vdev_initialize_offset[txg & TXG_MASK] == 0) { + uint64_t *guid = kmem_zalloc(sizeof (uint64_t), KM_SLEEP); + *guid = vd->vdev_guid; + + /* This is the first write of this txg. */ + dsl_sync_task_nowait(spa_get_dsl(spa), + vdev_initialize_zap_update_sync, guid, 2, + ZFS_SPACE_CHECK_RESERVED, tx); + } + + /* + * We know the vdev struct will still be around since all + * consumers of vdev_free must stop the initialization first. + */ + if (vdev_initialize_should_stop(vd)) { + mutex_enter(&vd->vdev_initialize_io_lock); + ASSERT3U(vd->vdev_initialize_inflight, >, 0); + vd->vdev_initialize_inflight--; + mutex_exit(&vd->vdev_initialize_io_lock); + spa_config_exit(vd->vdev_spa, SCL_STATE_ALL, vd); + mutex_exit(&vd->vdev_initialize_lock); + dmu_tx_commit(tx); + return (SET_ERROR(EINTR)); + } + mutex_exit(&vd->vdev_initialize_lock); + + vd->vdev_initialize_offset[txg & TXG_MASK] = start + size; + zio_nowait(zio_write_phys(spa->spa_txg_zio[txg & TXG_MASK], vd, start, + size, data, ZIO_CHECKSUM_OFF, vdev_initialize_cb, NULL, + ZIO_PRIORITY_INITIALIZING, ZIO_FLAG_CANFAIL, B_FALSE)); + /* vdev_initialize_cb releases SCL_STATE_ALL */ + + dmu_tx_commit(tx); + + return (0); +} + +/* + * Translate a logical range to the physical range for the specified vdev_t. + * This function is initially called with a leaf vdev and will walk each + * parent vdev until it reaches a top-level vdev. Once the top-level is + * reached the physical range is initialized and the recursive function + * begins to unwind. As it unwinds it calls the parent's vdev specific + * translation function to do the real conversion. + */ +void +vdev_xlate(vdev_t *vd, const range_seg_t *logical_rs, range_seg_t *physical_rs) +{ + /* + * Walk up the vdev tree + */ + if (vd != vd->vdev_top) { + vdev_xlate(vd->vdev_parent, logical_rs, physical_rs); + } else { + /* + * We've reached the top-level vdev, initialize the + * physical range to the logical range and start to + * unwind. + */ + physical_rs->rs_start = logical_rs->rs_start; + physical_rs->rs_end = logical_rs->rs_end; + return; + } + + vdev_t *pvd = vd->vdev_parent; + ASSERT3P(pvd, !=, NULL); + ASSERT3P(pvd->vdev_ops->vdev_op_xlate, !=, NULL); + + /* + * As this recursive function unwinds, translate the logical + * range into its physical components by calling the + * vdev specific translate function. + */ + range_seg_t intermediate = { 0 }; + pvd->vdev_ops->vdev_op_xlate(vd, physical_rs, &intermediate); + + physical_rs->rs_start = intermediate.rs_start; + physical_rs->rs_end = intermediate.rs_end; +} + +/* + * Callback to fill each ABD chunk with zfs_initialize_value. len must be + * divisible by sizeof (uint64_t), and buf must be 8-byte aligned. The ABD + * allocation will guarantee these for us. + */ +/* ARGSUSED */ +static int +vdev_initialize_block_fill(void *buf, size_t len, void *unused) +{ + ASSERT0(len % sizeof (uint64_t)); + for (uint64_t i = 0; i < len; i += sizeof (uint64_t)) { + *(uint64_t *)((char *)(buf) + i) = zfs_initialize_value; + } + return (0); +} + +static abd_t * +vdev_initialize_block_alloc() +{ + /* Allocate ABD for filler data */ + abd_t *data = abd_alloc_for_io(zfs_initialize_chunk_size, B_FALSE); + + ASSERT0(zfs_initialize_chunk_size % sizeof (uint64_t)); + (void) abd_iterate_func(data, 0, zfs_initialize_chunk_size, + vdev_initialize_block_fill, NULL); + + return (data); +} + +static void +vdev_initialize_block_free(abd_t *data) +{ + abd_free(data); +} + +static int +vdev_initialize_ranges(vdev_t *vd, abd_t *data) +{ + avl_tree_t *rt = &vd->vdev_initialize_tree->rt_root; + + for (range_seg_t *rs = avl_first(rt); rs != NULL; + rs = AVL_NEXT(rt, rs)) { + uint64_t size = rs->rs_end - rs->rs_start; + + /* Split range into legally-sized physical chunks */ + uint64_t writes_required = + ((size - 1) / zfs_initialize_chunk_size) + 1; + + for (uint64_t w = 0; w < writes_required; w++) { + int error; + + error = vdev_initialize_write(vd, + VDEV_LABEL_START_SIZE + rs->rs_start + + (w * zfs_initialize_chunk_size), + MIN(size - (w * zfs_initialize_chunk_size), + zfs_initialize_chunk_size), data); + if (error != 0) + return (error); + } + } + return (0); +} + +static void +vdev_initialize_ms_load(metaslab_t *msp) +{ + ASSERT(MUTEX_HELD(&msp->ms_lock)); + + metaslab_load_wait(msp); + if (!msp->ms_loaded) + VERIFY0(metaslab_load(msp)); +} + +static void +vdev_initialize_mg_wait(metaslab_group_t *mg) +{ + ASSERT(MUTEX_HELD(&mg->mg_ms_initialize_lock)); + while (mg->mg_initialize_updating) { + cv_wait(&mg->mg_ms_initialize_cv, &mg->mg_ms_initialize_lock); + } +} + +static void +vdev_initialize_mg_mark(metaslab_group_t *mg) +{ + ASSERT(MUTEX_HELD(&mg->mg_ms_initialize_lock)); + ASSERT(mg->mg_initialize_updating); + + while (mg->mg_ms_initializing >= max_initialize_ms) { + cv_wait(&mg->mg_ms_initialize_cv, &mg->mg_ms_initialize_lock); + } + mg->mg_ms_initializing++; + ASSERT3U(mg->mg_ms_initializing, <=, max_initialize_ms); +} + +/* + * Mark the metaslab as being initialized to prevent any allocations + * on this metaslab. We must also track how many metaslabs are currently + * being initialized within a metaslab group and limit them to prevent + * allocation failures from occurring because all metaslabs are being + * initialized. + */ +static void +vdev_initialize_ms_mark(metaslab_t *msp) +{ + ASSERT(!MUTEX_HELD(&msp->ms_lock)); + metaslab_group_t *mg = msp->ms_group; + + mutex_enter(&mg->mg_ms_initialize_lock); + + /* + * To keep an accurate count of how many threads are initializing + * a specific metaslab group, we only allow one thread to mark + * the metaslab group at a time. This ensures that the value of + * ms_initializing will be accurate when we decide to mark a metaslab + * group as being initialized. To do this we force all other threads + * to wait till the metaslab's mg_initialize_updating flag is no + * longer set. + */ + vdev_initialize_mg_wait(mg); + mg->mg_initialize_updating = B_TRUE; + if (msp->ms_initializing == 0) { + vdev_initialize_mg_mark(mg); + } + mutex_enter(&msp->ms_lock); + msp->ms_initializing++; + mutex_exit(&msp->ms_lock); + + mg->mg_initialize_updating = B_FALSE; + cv_broadcast(&mg->mg_ms_initialize_cv); + mutex_exit(&mg->mg_ms_initialize_lock); +} + +static void +vdev_initialize_ms_unmark(metaslab_t *msp) +{ + ASSERT(!MUTEX_HELD(&msp->ms_lock)); + metaslab_group_t *mg = msp->ms_group; + mutex_enter(&mg->mg_ms_initialize_lock); + mutex_enter(&msp->ms_lock); + if (--msp->ms_initializing == 0) { + mg->mg_ms_initializing--; + cv_broadcast(&mg->mg_ms_initialize_cv); + } + mutex_exit(&msp->ms_lock); + mutex_exit(&mg->mg_ms_initialize_lock); +} + +static void +vdev_initialize_calculate_progress(vdev_t *vd) +{ + ASSERT(spa_config_held(vd->vdev_spa, SCL_CONFIG, RW_READER) || + spa_config_held(vd->vdev_spa, SCL_CONFIG, RW_WRITER)); + ASSERT(vd->vdev_leaf_zap != 0); + + vd->vdev_initialize_bytes_est = 0; + vd->vdev_initialize_bytes_done = 0; + + for (uint64_t i = 0; i < vd->vdev_top->vdev_ms_count; i++) { + metaslab_t *msp = vd->vdev_top->vdev_ms[i]; + mutex_enter(&msp->ms_lock); + + uint64_t ms_free = msp->ms_size - + space_map_allocated(msp->ms_sm); + + if (vd->vdev_top->vdev_ops == &vdev_raidz_ops) + ms_free /= vd->vdev_top->vdev_children; + + /* + * Convert the metaslab range to a physical range + * on our vdev. We use this to determine if we are + * in the middle of this metaslab range. + */ + range_seg_t logical_rs, physical_rs; + logical_rs.rs_start = msp->ms_start; + logical_rs.rs_end = msp->ms_start + msp->ms_size; + vdev_xlate(vd, &logical_rs, &physical_rs); + + if (vd->vdev_initialize_last_offset <= physical_rs.rs_start) { + vd->vdev_initialize_bytes_est += ms_free; + mutex_exit(&msp->ms_lock); + continue; + } else if (vd->vdev_initialize_last_offset > + physical_rs.rs_end) { + vd->vdev_initialize_bytes_done += ms_free; + vd->vdev_initialize_bytes_est += ms_free; + mutex_exit(&msp->ms_lock); + continue; + } + + /* + * If we get here, we're in the middle of initializing this + * metaslab. Load it and walk the free tree for more accurate + * progress estimation. + */ + vdev_initialize_ms_load(msp); + + for (range_seg_t *rs = avl_first(&msp->ms_allocatable->rt_root); rs; + rs = AVL_NEXT(&msp->ms_allocatable->rt_root, rs)) { + logical_rs.rs_start = rs->rs_start; + logical_rs.rs_end = rs->rs_end; + vdev_xlate(vd, &logical_rs, &physical_rs); + + uint64_t size = physical_rs.rs_end - + physical_rs.rs_start; + vd->vdev_initialize_bytes_est += size; + if (vd->vdev_initialize_last_offset > + physical_rs.rs_end) { + vd->vdev_initialize_bytes_done += size; + } else if (vd->vdev_initialize_last_offset > + physical_rs.rs_start && + vd->vdev_initialize_last_offset < + physical_rs.rs_end) { + vd->vdev_initialize_bytes_done += + vd->vdev_initialize_last_offset - + physical_rs.rs_start; + } + } + mutex_exit(&msp->ms_lock); + } +} + +static void +vdev_initialize_load(vdev_t *vd) +{ + ASSERT(spa_config_held(vd->vdev_spa, SCL_CONFIG, RW_READER) || + spa_config_held(vd->vdev_spa, SCL_CONFIG, RW_WRITER)); + ASSERT(vd->vdev_leaf_zap != 0); + + if (vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE || + vd->vdev_initialize_state == VDEV_INITIALIZE_SUSPENDED) { + int err = zap_lookup(vd->vdev_spa->spa_meta_objset, + vd->vdev_leaf_zap, VDEV_LEAF_ZAP_INITIALIZE_LAST_OFFSET, + sizeof (vd->vdev_initialize_last_offset), 1, + &vd->vdev_initialize_last_offset); + ASSERT(err == 0 || err == ENOENT); + } + + vdev_initialize_calculate_progress(vd); +} + + +/* + * Convert the logical range into a physcial range and add it to our + * avl tree. + */ +void +vdev_initialize_range_add(void *arg, uint64_t start, uint64_t size) +{ + vdev_t *vd = arg; + range_seg_t logical_rs, physical_rs; + logical_rs.rs_start = start; + logical_rs.rs_end = start + size; + + ASSERT(vd->vdev_ops->vdev_op_leaf); + vdev_xlate(vd, &logical_rs, &physical_rs); + + IMPLY(vd->vdev_top == vd, + logical_rs.rs_start == physical_rs.rs_start); + IMPLY(vd->vdev_top == vd, + logical_rs.rs_end == physical_rs.rs_end); + + /* Only add segments that we have not visited yet */ + if (physical_rs.rs_end <= vd->vdev_initialize_last_offset) + return; + + /* Pick up where we left off mid-range. */ + if (vd->vdev_initialize_last_offset > physical_rs.rs_start) { + zfs_dbgmsg("range write: vd %s changed (%llu, %llu) to " + "(%llu, %llu)", vd->vdev_path, + (u_longlong_t)physical_rs.rs_start, + (u_longlong_t)physical_rs.rs_end, + (u_longlong_t)vd->vdev_initialize_last_offset, + (u_longlong_t)physical_rs.rs_end); + ASSERT3U(physical_rs.rs_end, >, + vd->vdev_initialize_last_offset); + physical_rs.rs_start = vd->vdev_initialize_last_offset; + } + ASSERT3U(physical_rs.rs_end, >=, physical_rs.rs_start); + + /* + * With raidz, it's possible that the logical range does not live on + * this leaf vdev. We only add the physical range to this vdev's if it + * has a length greater than 0. + */ + if (physical_rs.rs_end > physical_rs.rs_start) { + range_tree_add(vd->vdev_initialize_tree, physical_rs.rs_start, + physical_rs.rs_end - physical_rs.rs_start); + } else { + ASSERT3U(physical_rs.rs_end, ==, physical_rs.rs_start); + } +} + +static void +vdev_initialize_thread(void *arg) +{ + vdev_t *vd = arg; + spa_t *spa = vd->vdev_spa; + int error = 0; + uint64_t ms_count = 0; + + ASSERT(vdev_is_concrete(vd)); + spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); + + vd->vdev_initialize_last_offset = 0; + vdev_initialize_load(vd); + + abd_t *deadbeef = vdev_initialize_block_alloc(); + + vd->vdev_initialize_tree = range_tree_create(NULL, NULL); + + for (uint64_t i = 0; !vd->vdev_detached && + i < vd->vdev_top->vdev_ms_count; i++) { + metaslab_t *msp = vd->vdev_top->vdev_ms[i]; + + /* + * If we've expanded the top-level vdev or it's our + * first pass, calculate our progress. + */ + if (vd->vdev_top->vdev_ms_count != ms_count) { + vdev_initialize_calculate_progress(vd); + ms_count = vd->vdev_top->vdev_ms_count; + } + + vdev_initialize_ms_mark(msp); + mutex_enter(&msp->ms_lock); + vdev_initialize_ms_load(msp); + + range_tree_walk(msp->ms_allocatable, vdev_initialize_range_add, + vd); + mutex_exit(&msp->ms_lock); + + spa_config_exit(spa, SCL_CONFIG, FTAG); + error = vdev_initialize_ranges(vd, deadbeef); + vdev_initialize_ms_unmark(msp); + spa_config_enter(spa, SCL_CONFIG, FTAG, RW_READER); + + range_tree_vacate(vd->vdev_initialize_tree, NULL, NULL); + if (error != 0) + break; + } + + spa_config_exit(spa, SCL_CONFIG, FTAG); + mutex_enter(&vd->vdev_initialize_io_lock); + while (vd->vdev_initialize_inflight > 0) { + cv_wait(&vd->vdev_initialize_io_cv, + &vd->vdev_initialize_io_lock); + } + mutex_exit(&vd->vdev_initialize_io_lock); + + range_tree_destroy(vd->vdev_initialize_tree); + vdev_initialize_block_free(deadbeef); + vd->vdev_initialize_tree = NULL; + + mutex_enter(&vd->vdev_initialize_lock); + if (!vd->vdev_initialize_exit_wanted && vdev_writeable(vd)) { + vdev_initialize_change_state(vd, VDEV_INITIALIZE_COMPLETE); + } + ASSERT(vd->vdev_initialize_thread != NULL || + vd->vdev_initialize_inflight == 0); + + /* + * Drop the vdev_initialize_lock while we sync out the + * txg since it's possible that a device might be trying to + * come online and must check to see if it needs to restart an + * initialization. That thread will be holding the spa_config_lock + * which would prevent the txg_wait_synced from completing. + */ + mutex_exit(&vd->vdev_initialize_lock); + txg_wait_synced(spa_get_dsl(spa), 0); + mutex_enter(&vd->vdev_initialize_lock); + + vd->vdev_initialize_thread = NULL; + cv_broadcast(&vd->vdev_initialize_cv); + mutex_exit(&vd->vdev_initialize_lock); +} + +/* + * Initiates a device. Caller must hold vdev_initialize_lock. + * Device must be a leaf and not already be initializing. + */ +void +vdev_initialize(vdev_t *vd) +{ + ASSERT(MUTEX_HELD(&vd->vdev_initialize_lock)); + ASSERT(vd->vdev_ops->vdev_op_leaf); + ASSERT(vdev_is_concrete(vd)); + ASSERT3P(vd->vdev_initialize_thread, ==, NULL); + ASSERT(!vd->vdev_detached); + ASSERT(!vd->vdev_initialize_exit_wanted); + ASSERT(!vd->vdev_top->vdev_removing); + + vdev_initialize_change_state(vd, VDEV_INITIALIZE_ACTIVE); + vd->vdev_initialize_thread = thread_create(NULL, 0, + vdev_initialize_thread, vd, 0, &p0, TS_RUN, maxclsyspri); +} + +/* + * Stop initializng a device, with the resultant initialing state being + * tgt_state. Blocks until the initializing thread has exited. + * Caller must hold vdev_initialize_lock and must not be writing to the spa + * config, as the initializing thread may try to enter the config as a reader + * before exiting. + */ +void +vdev_initialize_stop(vdev_t *vd, vdev_initializing_state_t tgt_state) +{ + spa_t *spa = vd->vdev_spa; + ASSERT(!spa_config_held(spa, SCL_CONFIG | SCL_STATE, RW_WRITER)); + + ASSERT(MUTEX_HELD(&vd->vdev_initialize_lock)); + ASSERT(vd->vdev_ops->vdev_op_leaf); + ASSERT(vdev_is_concrete(vd)); + + /* + * Allow cancel requests to proceed even if the initialize thread + * has stopped. + */ + if (vd->vdev_initialize_thread == NULL && + tgt_state != VDEV_INITIALIZE_CANCELED) { + return; + } + + vdev_initialize_change_state(vd, tgt_state); + vd->vdev_initialize_exit_wanted = B_TRUE; + while (vd->vdev_initialize_thread != NULL) + cv_wait(&vd->vdev_initialize_cv, &vd->vdev_initialize_lock); + + ASSERT3P(vd->vdev_initialize_thread, ==, NULL); + vd->vdev_initialize_exit_wanted = B_FALSE; +} + +static void +vdev_initialize_stop_all_impl(vdev_t *vd, vdev_initializing_state_t tgt_state) +{ + if (vd->vdev_ops->vdev_op_leaf && vdev_is_concrete(vd)) { + mutex_enter(&vd->vdev_initialize_lock); + vdev_initialize_stop(vd, tgt_state); + mutex_exit(&vd->vdev_initialize_lock); + return; + } + + for (uint64_t i = 0; i < vd->vdev_children; i++) { + vdev_initialize_stop_all_impl(vd->vdev_child[i], tgt_state); + } +} + +/* + * Convenience function to stop initializing of a vdev tree and set all + * initialize thread pointers to NULL. + */ +void +vdev_initialize_stop_all(vdev_t *vd, vdev_initializing_state_t tgt_state) +{ + vdev_initialize_stop_all_impl(vd, tgt_state); + + if (vd->vdev_spa->spa_sync_on) { + /* Make sure that our state has been synced to disk */ + txg_wait_synced(spa_get_dsl(vd->vdev_spa), 0); + } +} + +void +vdev_initialize_restart(vdev_t *vd) +{ + ASSERT(MUTEX_HELD(&spa_namespace_lock)); + ASSERT(!spa_config_held(vd->vdev_spa, SCL_ALL, RW_WRITER)); + + if (vd->vdev_leaf_zap != 0) { + mutex_enter(&vd->vdev_initialize_lock); + uint64_t initialize_state = VDEV_INITIALIZE_NONE; + int err = zap_lookup(vd->vdev_spa->spa_meta_objset, + vd->vdev_leaf_zap, VDEV_LEAF_ZAP_INITIALIZE_STATE, + sizeof (initialize_state), 1, &initialize_state); + ASSERT(err == 0 || err == ENOENT); + vd->vdev_initialize_state = initialize_state; + + uint64_t timestamp = 0; + err = zap_lookup(vd->vdev_spa->spa_meta_objset, + vd->vdev_leaf_zap, VDEV_LEAF_ZAP_INITIALIZE_ACTION_TIME, + sizeof (timestamp), 1, ×tamp); + ASSERT(err == 0 || err == ENOENT); + vd->vdev_initialize_action_time = (time_t)timestamp; + + if (vd->vdev_initialize_state == VDEV_INITIALIZE_SUSPENDED || + vd->vdev_offline) { + /* load progress for reporting, but don't resume */ + vdev_initialize_load(vd); + } else if (vd->vdev_initialize_state == + VDEV_INITIALIZE_ACTIVE && vdev_writeable(vd)) { + vdev_initialize(vd); + } + + mutex_exit(&vd->vdev_initialize_lock); + } + + for (uint64_t i = 0; i < vd->vdev_children; i++) { + vdev_initialize_restart(vd->vdev_child[i]); + } +} From owner-svn-src-vendor@freebsd.org Tue Jul 31 19:30:09 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC0B41062D28; Tue, 31 Jul 2018 19:30:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 82BB479691; Tue, 31 Jul 2018 19:30:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 64FF917EC7; Tue, 31 Jul 2018 19:30:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VJU9WZ086910; Tue, 31 Jul 2018 19:30:09 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VJU9H9086909; Tue, 31 Jul 2018 19:30:09 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807311930.w6VJU9H9086909@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 19:30:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337002 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337002 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 19:30:10 -0000 Author: mav Date: Tue Jul 31 19:30:08 2018 New Revision: 337002 URL: https://svnweb.freebsd.org/changeset/base/337002 Log: 9434 Speculative prefetch is blocked by device removal code illumos/illumos-gate@47b8d4b884f2dab4e4c6952b40fd740a8fc4ab32 Reviewed by: Matthew Ahrens Reviewed by: Prashanth Sreenivasa Approved by: Matt Ahrens Author: Alexander Motin Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Tue Jul 31 19:23:27 2018 (r337001) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Tue Jul 31 19:30:08 2018 (r337002) @@ -364,6 +364,7 @@ spa_remove_init(spa_t *spa) spa->spa_removing_phys.sr_state = DSS_NONE; spa->spa_removing_phys.sr_removing_vdev = -1; spa->spa_removing_phys.sr_prev_indirect_vdev = -1; + spa->spa_indirect_vdevs_loaded = B_TRUE; return (0); } else if (error != 0) { return (error); From owner-svn-src-vendor@freebsd.org Tue Jul 31 21:26:04 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AC651065855; Tue, 31 Jul 2018 21:26:04 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F2B4F7E14B; Tue, 31 Jul 2018 21:26:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CEF5B19307; Tue, 31 Jul 2018 21:26:03 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VLQ3mM048755; Tue, 31 Jul 2018 21:26:03 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VLQ3qw048754; Tue, 31 Jul 2018 21:26:03 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807312126.w6VLQ3qw048754@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 21:26:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337013 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337013 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 21:26:04 -0000 Author: mav Date: Tue Jul 31 21:26:03 2018 New Revision: 337013 URL: https://svnweb.freebsd.org/changeset/base/337013 Log: 9194 mechanism to override ashift at pool creation time illumos/illumos-gate@93a1902e519bf492c071b48ffb93e5c1c7b53fb9 Reviewed by: Serapheim Dimitropoulos Reviewed by: Prakash Surya Reviewed by: Allan Jude Reviewed by: Igor Kozhukhov Reviewed by: Richard Elling Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Tue Jul 31 21:09:32 2018 (r337012) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Tue Jul 31 21:26:03 2018 (r337013) @@ -96,6 +96,8 @@ int vdev_dtl_sm_blksz = (1 << 12); */ int vdev_standard_sm_blksz = (1 << 17); +int zfs_ashift_min; + /*PRINTFLIKE2*/ void vdev_dbgmsg(vdev_t *vd, const char *fmt, ...) @@ -1511,6 +1513,7 @@ vdev_open(vdev_t *vd) vd->vdev_asize = asize; vd->vdev_max_asize = max_asize; vd->vdev_ashift = MAX(ashift, vd->vdev_ashift); + vd->vdev_ashift = MAX(zfs_ashift_min, vd->vdev_ashift); } else { /* * Detect if the alignment requirement has increased. From owner-svn-src-vendor@freebsd.org Tue Jul 31 21:31:25 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A0561065A71; Tue, 31 Jul 2018 21:31:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3ED497E4C0; Tue, 31 Jul 2018 21:31:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2013B1944F; Tue, 31 Jul 2018 21:31:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VLVPOp053118; Tue, 31 Jul 2018 21:31:25 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VLVOjR053115; Tue, 31 Jul 2018 21:31:24 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807312131.w6VLVOjR053115@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 21:31:24 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337014 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 337014 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 21:31:25 -0000 Author: mav Date: Tue Jul 31 21:31:24 2018 New Revision: 337014 URL: https://svnweb.freebsd.org/changeset/base/337014 Log: 9421 zdb should detect and print out the number of "leaked" objects 9422 zfs diff and zdb should explicitly mark objects that are on the deleted queue illumos/illumos-gate@20b5dafb425396adaebd0267d29e1026fc4dc413 Reviewed by: Matt Ahrens Reviewed by: Pavel Zakharov Approved by: Matt Ahrens Author: Paul Dagnelie Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Tue Jul 31 21:26:03 2018 (r337013) +++ vendor/illumos/dist/cmd/zdb/zdb.c Tue Jul 31 21:31:24 2018 (r337014) @@ -108,6 +108,7 @@ uint64_t *zopt_object = NULL; static unsigned zopt_objects = 0; libzfs_handle_t *g_zfs; uint64_t max_inflight = 1000; +static int leaked_objects = 0; static void snprintf_blkptr_compact(char *, size_t, const blkptr_t *); @@ -1963,9 +1964,12 @@ dump_znode(objset_t *os, uint64_t object, void *data, if (dump_opt['d'] > 4) { error = zfs_obj_to_path(os, object, path, sizeof (path)); - if (error != 0) { + if (error == ESTALE) { + (void) snprintf(path, sizeof (path), "on delete queue"); + } else if (error != 0) { + leaked_objects++; (void) snprintf(path, sizeof (path), - "\?\?\?", (u_longlong_t)object); + "path not found, possibly leaked"); } (void) printf("\tpath %s\n", path); } @@ -2295,6 +2299,11 @@ dump_dir(objset_t *os) (void) fprintf(stderr, "dmu_object_next() = %d\n", error); abort(); } + if (leaked_objects != 0) { + (void) printf("%d potentially leaked objects detected\n", + leaked_objects); + leaked_objects = 0; + } } static void @@ -5364,5 +5373,5 @@ main(int argc, char **argv) libzfs_fini(g_zfs); kernel_fini(); - return (0); + return (error); } Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c Tue Jul 31 21:26:03 2018 (r337013) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c Tue Jul 31 21:31:24 2018 (r337014) @@ -22,7 +22,7 @@ /* * Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2015 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2015 by Delphix. All rights reserved. + * Copyright (c) 2015, 2017 by Delphix. All rights reserved. * Copyright 2016 Joyent, Inc. * Copyright 2016 Igor Kozhukhov */ @@ -103,7 +103,10 @@ get_stats_for_obj(differ_info_t *di, const char *dsnam return (0); } - if (di->zerr == EPERM) { + if (di->zerr == ESTALE) { + (void) snprintf(pn, maxlen, "(on_delete_queue)"); + return (0); + } else if (di->zerr == EPERM) { (void) snprintf(di->errbuf, sizeof (di->errbuf), dgettext(TEXT_DOMAIN, "The sys_config privilege or diff delegated permission " From owner-svn-src-vendor@freebsd.org Tue Jul 31 21:31:25 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 12C091065A6D; Tue, 31 Jul 2018 21:31:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BC96F7E4BE; Tue, 31 Jul 2018 21:31:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9DC9B1944E; Tue, 31 Jul 2018 21:31:24 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VLVOeX053110; Tue, 31 Jul 2018 21:31:24 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VLVOtP053109; Tue, 31 Jul 2018 21:31:24 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807312131.w6VLVOtP053109@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 21:31:24 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337014 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/cmd/zdb vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 337014 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 21:31:25 -0000 Author: mav Date: Tue Jul 31 21:31:24 2018 New Revision: 337014 URL: https://svnweb.freebsd.org/changeset/base/337014 Log: 9421 zdb should detect and print out the number of "leaked" objects 9422 zfs diff and zdb should explicitly mark objects that are on the deleted queue illumos/illumos-gate@20b5dafb425396adaebd0267d29e1026fc4dc413 Reviewed by: Matt Ahrens Reviewed by: Pavel Zakharov Approved by: Matt Ahrens Author: Paul Dagnelie Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/lib/libzfs/common/libzfs_diff.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Tue Jul 31 21:26:03 2018 (r337013) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Tue Jul 31 21:31:24 2018 (r337014) @@ -2036,6 +2036,17 @@ zfs_obj_to_path_impl(objset_t *osp, uint64_t obj, sa_h *path = '\0'; sa_hdl = hdl; + uint64_t deleteq_obj; + VERIFY0(zap_lookup(osp, MASTER_NODE_OBJ, + ZFS_UNLINKED_SET, sizeof (uint64_t), 1, &deleteq_obj)); + error = zap_lookup_int(osp, deleteq_obj, obj); + if (error == 0) { + return (ESTALE); + } else if (error != ENOENT) { + return (error); + } + error = 0; + for (;;) { uint64_t pobj; char component[MAXNAMELEN + 2]; From owner-svn-src-vendor@freebsd.org Tue Jul 31 21:42:32 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A0AC1065E7D; Tue, 31 Jul 2018 21:42:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 409527F27F; Tue, 31 Jul 2018 21:42:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 22CC71962E; Tue, 31 Jul 2018 21:42:32 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VLgWTD059002; Tue, 31 Jul 2018 21:42:32 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VLgW1g059001; Tue, 31 Jul 2018 21:42:32 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807312142.w6VLgW1g059001@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 21:42:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337016 - vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 337016 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 21:42:32 -0000 Author: mav Date: Tue Jul 31 21:42:31 2018 New Revision: 337016 URL: https://svnweb.freebsd.org/changeset/base/337016 Log: 9237 "zpool add" fails for very large pools illumos/illumos-gate@ae5ee1bd5e54409b4589217b160c8cf4b81e2fb8 Reviewed by: Matthew Ahrens Reviewed by: Andriy Gapon Approved by: Matt Ahrens Author: Alan Somers Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_import.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Tue Jul 31 21:40:13 2018 (r337015) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Tue Jul 31 21:42:31 2018 (r337016) @@ -895,6 +895,7 @@ zpool_read_label(int fd, nvlist_t **config) free(label); *config = NULL; + errno = ENOENT; return (-1); } From owner-svn-src-vendor@freebsd.org Tue Jul 31 22:59:49 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D1851067A01; Tue, 31 Jul 2018 22:59:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D7C2782486; Tue, 31 Jul 2018 22:59:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B501B1A190; Tue, 31 Jul 2018 22:59:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VMxmu0095461; Tue, 31 Jul 2018 22:59:48 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VMxm5t095460; Tue, 31 Jul 2018 22:59:48 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807312259.w6VMxm5t095460@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 22:59:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337020 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337020 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 22:59:49 -0000 Author: mav Date: Tue Jul 31 22:59:48 2018 New Revision: 337020 URL: https://svnweb.freebsd.org/changeset/base/337020 Log: 9443 panic when scrub a v10 pool illumos/illumos-gate@bb1f424574ac8e08069d0ba993c2a41ffe796794 Reviewed by: Serapheim Dimitropoulos Reviewed by: George Wilson Reviewed by: Andriy Gapon Reviewed by: Igor Kozhukhov Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Tue Jul 31 22:57:27 2018 (r337019) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Tue Jul 31 22:59:48 2018 (r337020) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2008, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2017 by Delphix. All rights reserved. + * Copyright (c) 2011, 2018 by Delphix. All rights reserved. * Copyright 2016 Gary Mills * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright 2017 Joyent, Inc. @@ -1185,7 +1185,8 @@ dsl_scan_visitds(dsl_scan_t *scn, uint64_t dsobj, dmu_ * block-sharing rules don't apply to it. */ if (DSL_SCAN_IS_SCRUB_RESILVER(scn) && !dsl_dataset_is_snapshot(ds) && - ds->ds_dir != dp->dp_origin_snap->ds_dir) { + (dp->dp_origin_snap == NULL || + ds->ds_dir != dp->dp_origin_snap->ds_dir)) { objset_t *os; if (dmu_objset_from_ds(ds, &os) != 0) { goto out; From owner-svn-src-vendor@freebsd.org Tue Jul 31 23:44:15 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E10C10687B9; Tue, 31 Jul 2018 23:44:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D585283DF8; Tue, 31 Jul 2018 23:44:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B5B731A99C; Tue, 31 Jul 2018 23:44:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6VNiEI7021633; Tue, 31 Jul 2018 23:44:14 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6VNiEem021630; Tue, 31 Jul 2018 23:44:14 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201807312344.w6VNiEem021630@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Tue, 31 Jul 2018 23:44:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337022 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337022 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Jul 2018 23:44:15 -0000 Author: mav Date: Tue Jul 31 23:44:13 2018 New Revision: 337022 URL: https://svnweb.freebsd.org/changeset/base/337022 Log: 9403 assertion failed in arc_buf_destroy() when concurrently reading block with checksum error This assertion (VERIFY) failure was reported when reading a block. Turns out the problem is that if we get an i/o error (ECKSUM in this case), and there are multiple concurrent ARC reads of the same block (from different clones), then the ARC will put multiple buf's on the same ANON hdr, which isn't supposed to happen, and then causes a panic when we try to arc_buf_destroy() the buf. illumos/illumos-gate@fa98e487a9619b7902f218663be219e787a57dad Reviewed by: George Wilson Reviewed by: Paul Dagnelie Reviewed by: Pavel Zakharov Approved by: Matt Ahrens Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio_compress.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Tue Jul 31 23:00:58 2018 (r337021) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Tue Jul 31 23:44:13 2018 (r337022) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2018, Joyent, Inc. - * Copyright (c) 2011, 2017 by Delphix. All rights reserved. + * Copyright (c) 2011, 2018 by Delphix. All rights reserved. * Copyright (c) 2014 by Saso Kiselkov. All rights reserved. * Copyright 2017 Nexenta Systems, Inc. All rights reserved. */ @@ -4717,12 +4717,13 @@ void arc_getbuf_func(zio_t *zio, arc_buf_t *buf, void *arg) { arc_buf_t **bufp = arg; - if (zio && zio->io_error) { - arc_buf_destroy(buf, arg); + if (buf == NULL) { + ASSERT(zio == NULL || zio->io_error != 0); *bufp = NULL; } else { + ASSERT(zio == NULL || zio->io_error == 0); *bufp = buf; - ASSERT(buf->b_data); + ASSERT(buf->b_data != NULL); } } @@ -4821,12 +4822,38 @@ arc_read_done(zio_t *zio) /* This is a demand read since prefetches don't use callbacks */ callback_cnt++; - int error = arc_buf_alloc_impl(hdr, acb->acb_private, - acb->acb_compressed, no_zio_error, &acb->acb_buf); if (no_zio_error) { - zio->io_error = error; + int error = arc_buf_alloc_impl(hdr, acb->acb_private, + acb->acb_compressed, zio->io_error == 0, + &acb->acb_buf); + if (error != 0) { + /* + * Decompression failed. Set io_error + * so that when we call acb_done (below), + * we will indicate that the read failed. + * Note that in the unusual case where one + * callback is compressed and another + * uncompressed, we will mark all of them + * as failed, even though the uncompressed + * one can't actually fail. In this case, + * the hdr will not be anonymous, because + * if there are multiple callbacks, it's + * because multiple threads found the same + * arc buf in the hash table. + */ + zio->io_error = error; + } } } + /* + * If there are multiple callbacks, we must have the hash lock, + * because the only way for multiple threads to find this hdr is + * in the hash table. This ensures that if there are multiple + * callbacks, the hdr is not anonymous. If it were anonymous, + * we couldn't use arc_buf_destroy() in the error case below. + */ + ASSERT(callback_cnt < 2 || hash_lock != NULL); + hdr->b_l1hdr.b_acb = NULL; arc_hdr_clear_flags(hdr, ARC_FLAG_IO_IN_PROGRESS); if (callback_cnt == 0) { @@ -4871,8 +4898,18 @@ arc_read_done(zio_t *zio) /* execute each callback and free its structure */ while ((acb = callback_list) != NULL) { - if (acb->acb_done) + if (acb->acb_done != NULL) { + if (zio->io_error != 0 && acb->acb_buf != NULL) { + /* + * If arc_buf_alloc_impl() fails during + * decompression, the buf will still be + * allocated, and needs to be freed here. + */ + arc_buf_destroy(acb->acb_buf, acb->acb_private); + acb->acb_buf = NULL; + } acb->acb_done(zio, acb->acb_buf, acb->acb_private); + } if (acb->acb_zio_dummy != NULL) { acb->acb_zio_dummy->io_error = zio->io_error; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Tue Jul 31 23:00:58 2018 (r337021) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Tue Jul 31 23:44:13 2018 (r337022) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2012, 2017 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. @@ -989,22 +989,26 @@ dbuf_read_done(zio_t *zio, arc_buf_t *buf, void *vdb) ASSERT(refcount_count(&db->db_holds) > 0); ASSERT(db->db_buf == NULL); ASSERT(db->db.db_data == NULL); - if (db->db_level == 0 && db->db_freed_in_flight) { - /* we were freed in flight; disregard any error */ + if (buf == NULL) { + /* i/o error */ + ASSERT(zio == NULL || zio->io_error != 0); + ASSERT(db->db_blkid != DMU_BONUS_BLKID); + ASSERT3P(db->db_buf, ==, NULL); + db->db_state = DB_UNCACHED; + } else if (db->db_level == 0 && db->db_freed_in_flight) { + /* freed in flight */ + ASSERT(zio == NULL || zio->io_error == 0); arc_release(buf, db); bzero(buf->b_data, db->db.db_size); arc_buf_freeze(buf); db->db_freed_in_flight = FALSE; dbuf_set_data(db, buf); db->db_state = DB_CACHED; - } else if (zio == NULL || zio->io_error == 0) { + } else { + /* success */ + ASSERT(zio == NULL || zio->io_error == 0); dbuf_set_data(db, buf); db->db_state = DB_CACHED; - } else { - ASSERT(db->db_blkid != DMU_BONUS_BLKID); - ASSERT3P(db->db_buf, ==, NULL); - arc_buf_destroy(buf, db); - db->db_state = DB_UNCACHED; } cv_broadcast(&db->db_changed); dbuf_rele_and_unlock(db, NULL); @@ -2417,6 +2421,13 @@ dbuf_prefetch_indirect_done(zio_t *zio, arc_buf_t *abu ASSERT3S(dpa->dpa_zb.zb_level, <, dpa->dpa_curlevel); ASSERT3S(dpa->dpa_curlevel, >, 0); + if (abuf == NULL) { + ASSERT(zio == NULL || zio->io_error != 0); + kmem_free(dpa, sizeof (*dpa)); + return; + } + ASSERT(zio == NULL || zio->io_error == 0); + /* * The dpa_dnode is only valid if we are called with a NULL * zio. This indicates that the arc_read() returned without @@ -2455,7 +2466,7 @@ dbuf_prefetch_indirect_done(zio_t *zio, arc_buf_t *abu (dpa->dpa_epbs * (dpa->dpa_curlevel - dpa->dpa_zb.zb_level)); blkptr_t *bp = ((blkptr_t *)abuf->b_data) + P2PHASE(nextblkid, 1ULL << dpa->dpa_epbs); - if (BP_IS_HOLE(bp) || (zio != NULL && zio->io_error != 0)) { + if (BP_IS_HOLE(bp)) { kmem_free(dpa, sizeof (*dpa)); } else if (dpa->dpa_curlevel == dpa->dpa_zb.zb_level) { ASSERT3U(nextblkid, ==, dpa->dpa_zb.zb_blkid); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zio_compress.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zio_compress.c Tue Jul 31 23:00:58 2018 (r337021) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zio_compress.c Tue Jul 31 23:44:13 2018 (r337022) @@ -25,7 +25,7 @@ */ /* * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. - * Copyright (c) 2013, 2016 by Delphix. All rights reserved. + * Copyright (c) 2013, 2018 by Delphix. All rights reserved. */ #include @@ -36,6 +36,12 @@ #include /* + * If nonzero, every 1/X decompression attempts will fail, simulating + * an undetected memory error. + */ +uint64_t zio_decompress_fail_fraction = 0; + +/* * Compression vectors. */ zio_compress_info_t zio_compress_table[ZIO_COMPRESS_FUNCTIONS] = { @@ -145,6 +151,16 @@ zio_decompress_data(enum zio_compress c, abd_t *src, v void *tmp = abd_borrow_buf_copy(src, s_len); int ret = zio_decompress_data_buf(c, tmp, dst, s_len, d_len); abd_return_buf(src, tmp, s_len); + + /* + * Decompression shouldn't fail, because we've already verifyied + * the checksum. However, for extra protection (e.g. against bitflips + * in non-ECC RAM), we handle this error (and test it). + */ + ASSERT0(ret); + if (zio_decompress_fail_fraction != 0 && + spa_get_random(zio_decompress_fail_fraction) == 0) + ret = SET_ERROR(EINVAL); return (ret); } From owner-svn-src-vendor@freebsd.org Wed Aug 1 03:03:17 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 210F4106DE5E; Wed, 1 Aug 2018 03:03:17 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C352B701A1; Wed, 1 Aug 2018 03:03:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A42D81CADE; Wed, 1 Aug 2018 03:03:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w7133GPV037373; Wed, 1 Aug 2018 03:03:16 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w7133FfN037368; Wed, 1 Aug 2018 03:03:15 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808010303.w7133FfN037368@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 1 Aug 2018 03:03:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337027 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337027 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 03:03:17 -0000 Author: mav Date: Wed Aug 1 03:03:15 2018 New Revision: 337027 URL: https://svnweb.freebsd.org/changeset/base/337027 Log: 9328 zap code can take advantage of c99 9329 panic in zap_leaf_lookup() due to concurrent zapification illumos/illumos-gate@bf26014c5541b6119f34e0d95294b7f2eb105ac2 Reviewed by: Steve Gonczi Reviewed by: George Wilson Reviewed by: Pavel Zakharov Reviewed by: Brad Lewis Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c vendor-sys/illumos/dist/uts/common/fs/zfs/zap.c vendor-sys/illumos/dist/uts/common/fs/zfs/zap_leaf.c vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Wed Aug 1 02:59:56 2018 (r337026) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Wed Aug 1 03:03:15 2018 (r337027) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013, 2015 by Delphix. All rights reserved. + * Copyright (c) 2013, 2017 by Delphix. All rights reserved. * Copyright 2014 HybridCluster. All rights reserved. */ @@ -204,12 +204,18 @@ dmu_object_zapify(objset_t *mos, uint64_t object, dmu_ } ASSERT3U(dn->dn_type, ==, old_type); ASSERT0(dn->dn_maxblkid); + + /* + * We must initialize the ZAP data before changing the type, + * so that concurrent calls to *_is_zapified() can determine if + * the object has been completely zapified by checking the type. + */ + mzap_create_impl(mos, object, 0, 0, tx); + dn->dn_next_type[tx->tx_txg & TXG_MASK] = dn->dn_type = DMU_OTN_ZAP_METADATA; dnode_setdirty(dn, tx); dnode_rele(dn, FTAG); - - mzap_create_impl(mos, object, 0, 0, tx); spa_feature_incr(dmu_objset_spa(mos), SPA_FEATURE_EXTENSIBLE_DATASET, tx); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zap.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zap.c Wed Aug 1 02:59:56 2018 (r337026) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zap.c Wed Aug 1 03:03:15 2018 (r337027) @@ -58,10 +58,8 @@ static uint64_t zap_allocate_blocks(zap_t *zap, int nb void fzap_byteswap(void *vbuf, size_t size) { - uint64_t block_type; + uint64_t block_type = *(uint64_t *)vbuf; - block_type = *(uint64_t *)vbuf; - if (block_type == ZBT_LEAF || block_type == BSWAP_64(ZBT_LEAF)) zap_leaf_byteswap(vbuf, size); else { @@ -73,11 +71,6 @@ fzap_byteswap(void *vbuf, size_t size) void fzap_upgrade(zap_t *zap, dmu_tx_t *tx, zap_flags_t flags) { - dmu_buf_t *db; - zap_leaf_t *l; - int i; - zap_phys_t *zp; - ASSERT(RW_WRITE_HELD(&zap->zap_rwlock)); zap->zap_ismicro = FALSE; @@ -87,7 +80,7 @@ fzap_upgrade(zap_t *zap, dmu_tx_t *tx, zap_flags_t fla mutex_init(&zap->zap_f.zap_num_entries_mtx, 0, 0, 0); zap->zap_f.zap_block_shift = highbit64(zap->zap_dbuf->db_size) - 1; - zp = zap_f_phys(zap); + zap_phys_t *zp = zap_f_phys(zap); /* * explicitly zero it since it might be coming from an * initialized microzap @@ -106,17 +99,18 @@ fzap_upgrade(zap_t *zap, dmu_tx_t *tx, zap_flags_t fla zp->zap_flags = flags; /* block 1 will be the first leaf */ - for (i = 0; i < (1<zap_ptrtbl.zt_shift); i++) + for (int i = 0; i < (1<zap_ptrtbl.zt_shift); i++) ZAP_EMBEDDED_PTRTBL_ENT(zap, i) = 1; /* * set up block 1 - the first leaf */ - VERIFY(0 == dmu_buf_hold(zap->zap_objset, zap->zap_object, + dmu_buf_t *db; + VERIFY0(dmu_buf_hold(zap->zap_objset, zap->zap_object, 1<l_dbuf = db; zap_leaf_init(l, zp->zap_normflags != 0); @@ -146,9 +140,7 @@ zap_table_grow(zap_t *zap, zap_table_phys_t *tbl, void (*transfer_func)(const uint64_t *src, uint64_t *dst, int n), dmu_tx_t *tx) { - uint64_t b, newblk; - dmu_buf_t *db_old, *db_new; - int err; + uint64_t newblk; int bs = FZAP_BLOCK_SHIFT(zap); int hepb = 1<<(bs-4); /* hepb = half the number of entries in a block */ @@ -172,21 +164,23 @@ zap_table_grow(zap_t *zap, zap_table_phys_t *tbl, * Copy the ptrtbl from the old to new location. */ - b = tbl->zt_blks_copied; - err = dmu_buf_hold(zap->zap_objset, zap->zap_object, + uint64_t b = tbl->zt_blks_copied; + dmu_buf_t *db_old; + int err = dmu_buf_hold(zap->zap_objset, zap->zap_object, (tbl->zt_blk + b) << bs, FTAG, &db_old, DMU_READ_NO_PREFETCH); - if (err) + if (err != 0) return (err); /* first half of entries in old[b] go to new[2*b+0] */ - VERIFY(0 == dmu_buf_hold(zap->zap_objset, zap->zap_object, + dmu_buf_t *db_new; + VERIFY0(dmu_buf_hold(zap->zap_objset, zap->zap_object, (newblk + 2*b+0) << bs, FTAG, &db_new, DMU_READ_NO_PREFETCH)); dmu_buf_will_dirty(db_new, tx); transfer_func(db_old->db_data, db_new->db_data, hepb); dmu_buf_rele(db_new, FTAG); /* second half of entries in old[b] go to new[2*b+1] */ - VERIFY(0 == dmu_buf_hold(zap->zap_objset, zap->zap_object, + VERIFY0(dmu_buf_hold(zap->zap_objset, zap->zap_object, (newblk + 2*b+1) << bs, FTAG, &db_new, DMU_READ_NO_PREFETCH)); dmu_buf_will_dirty(db_new, tx); transfer_func((uint64_t *)db_old->db_data + hepb, @@ -221,22 +215,20 @@ static int zap_table_store(zap_t *zap, zap_table_phys_t *tbl, uint64_t idx, uint64_t val, dmu_tx_t *tx) { - int err; - uint64_t blk, off; int bs = FZAP_BLOCK_SHIFT(zap); - dmu_buf_t *db; ASSERT(RW_LOCK_HELD(&zap->zap_rwlock)); ASSERT(tbl->zt_blk != 0); dprintf("storing %llx at index %llx\n", val, idx); - blk = idx >> (bs-3); - off = idx & ((1<<(bs-3))-1); + uint64_t blk = idx >> (bs-3); + uint64_t off = idx & ((1<<(bs-3))-1); - err = dmu_buf_hold(zap->zap_objset, zap->zap_object, + dmu_buf_t *db; + int err = dmu_buf_hold(zap->zap_objset, zap->zap_object, (tbl->zt_blk + blk) << bs, FTAG, &db, DMU_READ_NO_PREFETCH); - if (err) + if (err != 0) return (err); dmu_buf_will_dirty(db, tx); @@ -249,7 +241,7 @@ zap_table_store(zap_t *zap, zap_table_phys_t *tbl, uin err = dmu_buf_hold(zap->zap_objset, zap->zap_object, (tbl->zt_nextblk + blk2) << bs, FTAG, &db2, DMU_READ_NO_PREFETCH); - if (err) { + if (err != 0) { dmu_buf_rele(db, FTAG); return (err); } @@ -268,27 +260,24 @@ zap_table_store(zap_t *zap, zap_table_phys_t *tbl, uin static int zap_table_load(zap_t *zap, zap_table_phys_t *tbl, uint64_t idx, uint64_t *valp) { - uint64_t blk, off; - int err; - dmu_buf_t *db; - dnode_t *dn; int bs = FZAP_BLOCK_SHIFT(zap); ASSERT(RW_LOCK_HELD(&zap->zap_rwlock)); - blk = idx >> (bs-3); - off = idx & ((1<<(bs-3))-1); + uint64_t blk = idx >> (bs-3); + uint64_t off = idx & ((1<<(bs-3))-1); /* * Note: this is equivalent to dmu_buf_hold(), but we use * _dnode_enter / _by_dnode because it's faster because we don't * have to hold the dnode. */ - dn = dmu_buf_dnode_enter(zap->zap_dbuf); - err = dmu_buf_hold_by_dnode(dn, + dnode_t *dn = dmu_buf_dnode_enter(zap->zap_dbuf); + dmu_buf_t *db; + int err = dmu_buf_hold_by_dnode(dn, (tbl->zt_blk + blk) << bs, FTAG, &db, DMU_READ_NO_PREFETCH); dmu_buf_dnode_exit(zap->zap_dbuf); - if (err) + if (err != 0) return (err); *valp = ((uint64_t *)db->db_data)[off]; dmu_buf_rele(db, FTAG); @@ -319,11 +308,10 @@ zap_table_load(zap_t *zap, zap_table_phys_t *tbl, uint static void zap_ptrtbl_transfer(const uint64_t *src, uint64_t *dst, int n) { - int i; - for (i = 0; i < n; i++) { + for (int i = 0; i < n; i++) { uint64_t lb = src[i]; - dst[2*i+0] = lb; - dst[2*i+1] = lb; + dst[2 * i + 0] = lb; + dst[2 * i + 1] = lb; } } @@ -345,19 +333,16 @@ zap_grow_ptrtbl(zap_t *zap, dmu_tx_t *tx) * stored in the header block). Give it its own entire * block, which will double the size of the ptrtbl. */ - uint64_t newblk; - dmu_buf_t *db_new; - int err; - ASSERT3U(zap_f_phys(zap)->zap_ptrtbl.zt_shift, ==, ZAP_EMBEDDED_PTRTBL_SHIFT(zap)); ASSERT0(zap_f_phys(zap)->zap_ptrtbl.zt_blk); - newblk = zap_allocate_blocks(zap, 1); - err = dmu_buf_hold(zap->zap_objset, zap->zap_object, + uint64_t newblk = zap_allocate_blocks(zap, 1); + dmu_buf_t *db_new; + int err = dmu_buf_hold(zap->zap_objset, zap->zap_object, newblk << FZAP_BLOCK_SHIFT(zap), FTAG, &db_new, DMU_READ_NO_PREFETCH); - if (err) + if (err != 0) return (err); dmu_buf_will_dirty(db_new, tx); zap_ptrtbl_transfer(&ZAP_EMBEDDED_PTRTBL_ENT(zap, 0), @@ -392,9 +377,8 @@ zap_increment_num_entries(zap_t *zap, int delta, dmu_t static uint64_t zap_allocate_blocks(zap_t *zap, int nblocks) { - uint64_t newblk; ASSERT(RW_WRITE_HELD(&zap->zap_rwlock)); - newblk = zap_f_phys(zap)->zap_freeblk; + uint64_t newblk = zap_f_phys(zap)->zap_freeblk; zap_f_phys(zap)->zap_freeblk += nblocks; return (newblk); } @@ -411,7 +395,6 @@ zap_leaf_evict_sync(void *dbu) static zap_leaf_t * zap_create_leaf(zap_t *zap, dmu_tx_t *tx) { - void *winner; zap_leaf_t *l = kmem_zalloc(sizeof (zap_leaf_t), KM_SLEEP); ASSERT(RW_WRITE_HELD(&zap->zap_rwlock)); @@ -421,12 +404,11 @@ zap_create_leaf(zap_t *zap, dmu_tx_t *tx) l->l_blkid = zap_allocate_blocks(zap, 1); l->l_dbuf = NULL; - VERIFY(0 == dmu_buf_hold(zap->zap_objset, zap->zap_object, + VERIFY0(dmu_buf_hold(zap->zap_objset, zap->zap_object, l->l_blkid << FZAP_BLOCK_SHIFT(zap), NULL, &l->l_dbuf, DMU_READ_NO_PREFETCH)); dmu_buf_init_user(&l->l_dbu, zap_leaf_evict_sync, NULL, &l->l_dbuf); - winner = dmu_buf_set_user(l->l_dbuf, &l->l_dbu); - ASSERT(winner == NULL); + VERIFY3P(NULL, ==, dmu_buf_set_user(l->l_dbuf, &l->l_dbu)); dmu_buf_will_dirty(l->l_dbuf, tx); zap_leaf_init(l, zap->zap_normflags != 0); @@ -460,11 +442,9 @@ zap_put_leaf(zap_leaf_t *l) static zap_leaf_t * zap_open_leaf(uint64_t blkid, dmu_buf_t *db) { - zap_leaf_t *l, *winner; - ASSERT(blkid != 0); - l = kmem_zalloc(sizeof (zap_leaf_t), KM_SLEEP); + zap_leaf_t *l = kmem_zalloc(sizeof (zap_leaf_t), KM_SLEEP); rw_init(&l->l_rwlock, 0, 0, 0); rw_enter(&l->l_rwlock, RW_WRITER); l->l_blkid = blkid; @@ -472,7 +452,7 @@ zap_open_leaf(uint64_t blkid, dmu_buf_t *db) l->l_dbuf = db; dmu_buf_init_user(&l->l_dbu, zap_leaf_evict_sync, NULL, &l->l_dbuf); - winner = dmu_buf_set_user(db, &l->l_dbu); + zap_leaf_t *winner = dmu_buf_set_user(db, &l->l_dbu); rw_exit(&l->l_rwlock); if (winner != NULL) { @@ -510,17 +490,15 @@ zap_get_leaf_byblk(zap_t *zap, uint64_t blkid, dmu_tx_ zap_leaf_t **lp) { dmu_buf_t *db; - zap_leaf_t *l; - int bs = FZAP_BLOCK_SHIFT(zap); - int err; ASSERT(RW_LOCK_HELD(&zap->zap_rwlock)); + int bs = FZAP_BLOCK_SHIFT(zap); dnode_t *dn = dmu_buf_dnode_enter(zap->zap_dbuf); - err = dmu_buf_hold_by_dnode(dn, + int err = dmu_buf_hold_by_dnode(dn, blkid << bs, NULL, &db, DMU_READ_NO_PREFETCH); dmu_buf_dnode_exit(zap->zap_dbuf); - if (err) + if (err != 0) return (err); ASSERT3U(db->db_object, ==, zap->zap_object); @@ -528,7 +506,7 @@ zap_get_leaf_byblk(zap_t *zap, uint64_t blkid, dmu_tx_ ASSERT3U(db->db_size, ==, 1 << bs); ASSERT(blkid != 0); - l = dmu_buf_get_user(db); + zap_leaf_t *l = dmu_buf_get_user(db); if (l == NULL) l = zap_open_leaf(blkid, db); @@ -583,8 +561,7 @@ zap_set_idx_to_blk(zap_t *zap, uint64_t idx, uint64_t static int zap_deref_leaf(zap_t *zap, uint64_t h, dmu_tx_t *tx, krw_t lt, zap_leaf_t **lp) { - uint64_t idx, blk; - int err; + uint64_t blk; ASSERT(zap->zap_dbuf == NULL || zap_f_phys(zap) == zap->zap_dbuf->db_data); @@ -596,8 +573,8 @@ zap_deref_leaf(zap_t *zap, uint64_t h, dmu_tx_t *tx, k return (SET_ERROR(EIO)); } - idx = ZAP_HASH_IDX(h, zap_f_phys(zap)->zap_ptrtbl.zt_shift); - err = zap_idx_to_blk(zap, idx, &blk); + uint64_t idx = ZAP_HASH_IDX(h, zap_f_phys(zap)->zap_ptrtbl.zt_shift); + int err = zap_idx_to_blk(zap, idx, &blk); if (err != 0) return (err); err = zap_get_leaf_byblk(zap, blk, tx, lt, lp); @@ -614,9 +591,7 @@ zap_expand_leaf(zap_name_t *zn, zap_leaf_t *l, { zap_t *zap = zn->zn_zap; uint64_t hash = zn->zn_hash; - zap_leaf_t *nl; - int prefix_diff, i, err; - uint64_t sibling; + int err; int old_prefix_len = zap_leaf_phys(l)->l_hdr.lh_prefix_len; ASSERT3U(old_prefix_len, <=, zap_f_phys(zap)->zap_ptrtbl.zt_shift); @@ -636,19 +611,19 @@ zap_expand_leaf(zap_name_t *zn, zap_leaf_t *l, err = zap_lockdir(os, object, tx, RW_WRITER, FALSE, FALSE, tag, &zn->zn_zap); zap = zn->zn_zap; - if (err) + if (err != 0) return (err); ASSERT(!zap->zap_ismicro); while (old_prefix_len == zap_f_phys(zap)->zap_ptrtbl.zt_shift) { err = zap_grow_ptrtbl(zap, tx); - if (err) + if (err != 0) return (err); } err = zap_deref_leaf(zap, hash, tx, RW_WRITER, &l); - if (err) + if (err != 0) return (err); if (zap_leaf_phys(l)->l_hdr.lh_prefix_len != old_prefix_len) { @@ -662,25 +637,26 @@ zap_expand_leaf(zap_name_t *zn, zap_leaf_t *l, ASSERT3U(ZAP_HASH_IDX(hash, old_prefix_len), ==, zap_leaf_phys(l)->l_hdr.lh_prefix); - prefix_diff = zap_f_phys(zap)->zap_ptrtbl.zt_shift - + int prefix_diff = zap_f_phys(zap)->zap_ptrtbl.zt_shift - (old_prefix_len + 1); - sibling = (ZAP_HASH_IDX(hash, old_prefix_len + 1) | 1) << prefix_diff; + uint64_t sibling = + (ZAP_HASH_IDX(hash, old_prefix_len + 1) | 1) << prefix_diff; /* check for i/o errors before doing zap_leaf_split */ - for (i = 0; i < (1ULL<l_blkid); } - nl = zap_create_leaf(zap, tx); + zap_leaf_t *nl = zap_create_leaf(zap, tx); zap_leaf_split(l, nl, zap->zap_normflags != 0); /* set sibling pointers */ - for (i = 0; i < (1ULL << prefix_diff); i++) { - err = zap_set_idx_to_blk(zap, sibling+i, nl->l_blkid, tx); + for (int i = 0; i < (1ULL << prefix_diff); i++) { + err = zap_set_idx_to_blk(zap, sibling + i, nl->l_blkid, tx); ASSERT0(err); /* we checked for i/o errors above */ } @@ -708,8 +684,6 @@ zap_put_leaf_maybe_grow_ptrtbl(zap_name_t *zn, zap_lea zap_put_leaf(l); if (leaffull || zap_f_phys(zap)->zap_ptrtbl.zt_nextblk) { - int err; - /* * We are in the middle of growing the pointer table, or * this leaf will soon make us grow it. @@ -719,10 +693,10 @@ zap_put_leaf_maybe_grow_ptrtbl(zap_name_t *zn, zap_lea uint64_t zapobj = zap->zap_object; zap_unlockdir(zap, tag); - err = zap_lockdir(os, zapobj, tx, + int err = zap_lockdir(os, zapobj, tx, RW_WRITER, FALSE, FALSE, tag, &zn->zn_zap); zap = zn->zn_zap; - if (err) + if (err != 0) return; } @@ -763,9 +737,8 @@ fzap_checksize(uint64_t integer_size, uint64_t num_int static int fzap_check(zap_name_t *zn, uint64_t integer_size, uint64_t num_integers) { - int err; - - if ((err = fzap_checkname(zn)) != 0) + int err = fzap_checkname(zn); + if (err != 0) return (err); return (fzap_checksize(integer_size, num_integers)); } @@ -779,10 +752,10 @@ fzap_lookup(zap_name_t *zn, char *realname, int rn_len, boolean_t *ncp) { zap_leaf_t *l; - int err; zap_entry_handle_t zeh; - if ((err = fzap_checkname(zn)) != 0) + int err = fzap_checkname(zn); + if (err != 0) return (err); err = zap_deref_leaf(zn->zn_zap, zn->zn_hash, NULL, RW_READER, &l); @@ -870,7 +843,8 @@ fzap_update(zap_name_t *zn, void *tag, dmu_tx_t *tx) { zap_leaf_t *l; - int err, create; + int err; + boolean_t create; zap_entry_handle_t zeh; zap_t *zap = zn->zn_zap; @@ -923,9 +897,9 @@ fzap_length(zap_name_t *zn, if (err != 0) goto out; - if (integer_size) + if (integer_size != 0) *integer_size = zeh.zeh_integer_size; - if (num_integers) + if (num_integers != 0) *num_integers = zeh.zeh_num_integers; out: zap_put_leaf(l); @@ -954,15 +928,14 @@ fzap_remove(zap_name_t *zn, dmu_tx_t *tx) void fzap_prefetch(zap_name_t *zn) { - uint64_t idx, blk; + uint64_t blk; zap_t *zap = zn->zn_zap; - int bs; - idx = ZAP_HASH_IDX(zn->zn_hash, + uint64_t idx = ZAP_HASH_IDX(zn->zn_hash, zap_f_phys(zap)->zap_ptrtbl.zt_shift); if (zap_idx_to_blk(zap, idx, &blk) != 0) return; - bs = FZAP_BLOCK_SHIFT(zap); + int bs = FZAP_BLOCK_SHIFT(zap); dmu_prefetch(zap->zap_objset, zap->zap_object, 0, blk << bs, 1 << bs, ZIO_PRIORITY_SYNC_READ); } @@ -975,9 +948,8 @@ uint64_t zap_create_link(objset_t *os, dmu_object_type_t ot, uint64_t parent_obj, const char *name, dmu_tx_t *tx) { - uint64_t new_obj; - - VERIFY((new_obj = zap_create(os, ot, DMU_OT_NONE, 0, tx)) > 0); + uint64_t new_obj = zap_create(os, ot, DMU_OT_NONE, 0, tx); + VERIFY(new_obj != 0); VERIFY0(zap_add(os, parent_obj, name, sizeof (uint64_t), 1, &new_obj, tx)); @@ -989,13 +961,12 @@ zap_value_search(objset_t *os, uint64_t zapobj, uint64 char *name) { zap_cursor_t zc; - zap_attribute_t *za; int err; if (mask == 0) mask = -1ULL; - za = kmem_alloc(sizeof (zap_attribute_t), KM_SLEEP); + zap_attribute_t *za = kmem_alloc(sizeof (*za), KM_SLEEP); for (zap_cursor_init(&zc, os, zapobj); (err = zap_cursor_retrieve(&zc, za)) == 0; zap_cursor_advance(&zc)) { @@ -1005,7 +976,7 @@ zap_value_search(objset_t *os, uint64_t zapobj, uint64 } } zap_cursor_fini(&zc); - kmem_free(za, sizeof (zap_attribute_t)); + kmem_free(za, sizeof (*za)); return (err); } @@ -1013,23 +984,23 @@ int zap_join(objset_t *os, uint64_t fromobj, uint64_t intoobj, dmu_tx_t *tx) { zap_cursor_t zc; - zap_attribute_t za; - int err; + int err = 0; - err = 0; + zap_attribute_t *za = kmem_alloc(sizeof (*za), KM_SLEEP); for (zap_cursor_init(&zc, os, fromobj); - zap_cursor_retrieve(&zc, &za) == 0; + zap_cursor_retrieve(&zc, za) == 0; (void) zap_cursor_advance(&zc)) { - if (za.za_integer_length != 8 || za.za_num_integers != 1) { + if (za->za_integer_length != 8 || za->za_num_integers != 1) { err = SET_ERROR(EINVAL); break; } - err = zap_add(os, intoobj, za.za_name, - 8, 1, &za.za_first_integer, tx); - if (err) + err = zap_add(os, intoobj, za->za_name, + 8, 1, &za->za_first_integer, tx); + if (err != 0) break; } zap_cursor_fini(&zc); + kmem_free(za, sizeof (*za)); return (err); } @@ -1038,23 +1009,23 @@ zap_join_key(objset_t *os, uint64_t fromobj, uint64_t uint64_t value, dmu_tx_t *tx) { zap_cursor_t zc; - zap_attribute_t za; - int err; + int err = 0; - err = 0; + zap_attribute_t *za = kmem_alloc(sizeof (*za), KM_SLEEP); for (zap_cursor_init(&zc, os, fromobj); - zap_cursor_retrieve(&zc, &za) == 0; + zap_cursor_retrieve(&zc, za) == 0; (void) zap_cursor_advance(&zc)) { - if (za.za_integer_length != 8 || za.za_num_integers != 1) { + if (za->za_integer_length != 8 || za->za_num_integers != 1) { err = SET_ERROR(EINVAL); break; } - err = zap_add(os, intoobj, za.za_name, + err = zap_add(os, intoobj, za->za_name, 8, 1, &value, tx); - if (err) + if (err != 0) break; } zap_cursor_fini(&zc); + kmem_free(za, sizeof (*za)); return (err); } @@ -1063,29 +1034,29 @@ zap_join_increment(objset_t *os, uint64_t fromobj, uin dmu_tx_t *tx) { zap_cursor_t zc; - zap_attribute_t za; - int err; + int err = 0; - err = 0; + zap_attribute_t *za = kmem_alloc(sizeof (*za), KM_SLEEP); for (zap_cursor_init(&zc, os, fromobj); - zap_cursor_retrieve(&zc, &za) == 0; + zap_cursor_retrieve(&zc, za) == 0; (void) zap_cursor_advance(&zc)) { uint64_t delta = 0; - if (za.za_integer_length != 8 || za.za_num_integers != 1) { + if (za->za_integer_length != 8 || za->za_num_integers != 1) { err = SET_ERROR(EINVAL); break; } - err = zap_lookup(os, intoobj, za.za_name, 8, 1, &delta); + err = zap_lookup(os, intoobj, za->za_name, 8, 1, &delta); if (err != 0 && err != ENOENT) break; - delta += za.za_first_integer; - err = zap_update(os, intoobj, za.za_name, 8, 1, &delta, tx); - if (err) + delta += za->za_first_integer; + err = zap_update(os, intoobj, za->za_name, 8, 1, &delta, tx); + if (err != 0) break; } zap_cursor_fini(&zc); + kmem_free(za, sizeof (*za)); return (err); } @@ -1150,12 +1121,11 @@ zap_increment(objset_t *os, uint64_t obj, const char * dmu_tx_t *tx) { uint64_t value = 0; - int err; if (delta == 0) return (0); - err = zap_lookup(os, obj, name, 8, 1, &value); + int err = zap_lookup(os, obj, name, 8, 1, &value); if (err != 0 && err != ENOENT) return (err); value += delta; @@ -1253,7 +1223,6 @@ again: static void zap_stats_ptrtbl(zap_t *zap, uint64_t *tbl, int len, zap_stats_t *zs) { - int i, err; uint64_t lastblk = 0; /* @@ -1261,14 +1230,14 @@ zap_stats_ptrtbl(zap_t *zap, uint64_t *tbl, int len, z * can hold, then it'll be accounted for more than once, since * we won't have lastblk. */ - for (i = 0; i < len; i++) { + for (int i = 0; i < len; i++) { zap_leaf_t *l; if (tbl[i] == lastblk) continue; lastblk = tbl[i]; - err = zap_get_leaf_byblk(zap, tbl[i], NULL, RW_READER, &l); + int err = zap_get_leaf_byblk(zap, tbl[i], NULL, RW_READER, &l); if (err == 0) { zap_leaf_stats(zap, l, zs); zap_put_leaf(l); @@ -1308,14 +1277,12 @@ fzap_get_stats(zap_t *zap, zap_stats_t *zs) zap_stats_ptrtbl(zap, &ZAP_EMBEDDED_PTRTBL_ENT(zap, 0), 1 << ZAP_EMBEDDED_PTRTBL_SHIFT(zap), zs); } else { - int b; - dmu_prefetch(zap->zap_objset, zap->zap_object, 0, zap_f_phys(zap)->zap_ptrtbl.zt_blk << bs, zap_f_phys(zap)->zap_ptrtbl.zt_numblks << bs, ZIO_PRIORITY_SYNC_READ); - for (b = 0; b < zap_f_phys(zap)->zap_ptrtbl.zt_numblks; + for (int b = 0; b < zap_f_phys(zap)->zap_ptrtbl.zt_numblks; b++) { dmu_buf_t *db; int err; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zap_leaf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zap_leaf.c Wed Aug 1 02:59:56 2018 (r337026) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zap_leaf.c Wed Aug 1 03:03:15 2018 (r337027) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2013, 2015 by Delphix. All rights reserved. + * Copyright (c) 2013, 2016 by Delphix. All rights reserved. * Copyright 2017 Nexenta Systems, Inc. */ @@ -107,7 +107,6 @@ ldv(int len, const void *addr) void zap_leaf_byteswap(zap_leaf_phys_t *buf, int size) { - int i; zap_leaf_t l; dmu_buf_t l_dbuf; @@ -123,10 +122,10 @@ zap_leaf_byteswap(zap_leaf_phys_t *buf, int size) buf->l_hdr.lh_prefix_len = BSWAP_16(buf->l_hdr.lh_prefix_len); buf->l_hdr.lh_freelist = BSWAP_16(buf->l_hdr.lh_freelist); - for (i = 0; i < ZAP_LEAF_HASH_NUMENTRIES(&l); i++) + for (int i = 0; i < ZAP_LEAF_HASH_NUMENTRIES(&l); i++) buf->l_hash[i] = BSWAP_16(buf->l_hash[i]); - for (i = 0; i < ZAP_LEAF_NUMCHUNKS(&l); i++) { + for (int i = 0; i < ZAP_LEAF_NUMCHUNKS(&l); i++) { zap_leaf_chunk_t *lc = &ZAP_LEAF_CHUNK(&l, i); struct zap_leaf_entry *le; @@ -162,14 +161,12 @@ zap_leaf_byteswap(zap_leaf_phys_t *buf, int size) void zap_leaf_init(zap_leaf_t *l, boolean_t sort) { - int i; - l->l_bs = highbit64(l->l_dbuf->db_size) - 1; zap_memset(&zap_leaf_phys(l)->l_hdr, 0, sizeof (struct zap_leaf_header)); zap_memset(zap_leaf_phys(l)->l_hash, CHAIN_END, 2*ZAP_LEAF_HASH_NUMENTRIES(l)); - for (i = 0; i < ZAP_LEAF_NUMCHUNKS(l); i++) { + for (int i = 0; i < ZAP_LEAF_NUMCHUNKS(l); i++) { ZAP_LEAF_CHUNK(l, i).l_free.lf_type = ZAP_CHUNK_FREE; ZAP_LEAF_CHUNK(l, i).l_free.lf_next = i+1; } @@ -188,11 +185,9 @@ zap_leaf_init(zap_leaf_t *l, boolean_t sort) static uint16_t zap_leaf_chunk_alloc(zap_leaf_t *l) { - int chunk; - ASSERT(zap_leaf_phys(l)->l_hdr.lh_nfree > 0); - chunk = zap_leaf_phys(l)->l_hdr.lh_freelist; + int chunk = zap_leaf_phys(l)->l_hdr.lh_freelist; ASSERT3U(chunk, <, ZAP_LEAF_NUMCHUNKS(l)); ASSERT3U(ZAP_LEAF_CHUNK(l, chunk).l_free.lf_type, ==, ZAP_CHUNK_FREE); @@ -232,7 +227,7 @@ zap_leaf_array_create(zap_leaf_t *l, const char *buf, uint16_t *chunkp = &chunk_head; int byten = 0; uint64_t value = 0; - int shift = (integer_size-1)*8; + int shift = (integer_size - 1) * 8; int len = num_integers; ASSERT3U(num_integers * integer_size, <, MAX_ARRAY_BYTES); @@ -240,10 +235,9 @@ zap_leaf_array_create(zap_leaf_t *l, const char *buf, while (len > 0) { uint16_t chunk = zap_leaf_chunk_alloc(l); struct zap_leaf_array *la = &ZAP_LEAF_CHUNK(l, chunk).l_array; - int i; la->la_type = ZAP_CHUNK_ARRAY; - for (i = 0; i < ZAP_LEAF_ARRAY_BYTES; i++) { + for (int i = 0; i < ZAP_LEAF_ARRAY_BYTES; i++) { if (byten == 0) value = ldv(integer_size, buf); la->la_array[i] = value >> shift; @@ -321,10 +315,9 @@ zap_leaf_array_read(zap_leaf_t *l, uint16_t chunk, while (len > 0) { struct zap_leaf_array *la = &ZAP_LEAF_CHUNK(l, chunk).l_array; - int i; ASSERT3U(chunk, <, ZAP_LEAF_NUMCHUNKS(l)); - for (i = 0; i < ZAP_LEAF_ARRAY_BYTES && len > 0; i++) { + for (int i = 0; i < ZAP_LEAF_ARRAY_BYTES && len > 0; i++) { value = (value << 8) | la->la_array[i]; byten++; if (byten == array_int_len) { @@ -347,16 +340,13 @@ zap_leaf_array_match(zap_leaf_t *l, zap_name_t *zn, int bseen = 0; if (zap_getflags(zn->zn_zap) & ZAP_FLAG_UINT64_KEY) { - uint64_t *thiskey; - boolean_t match; - + uint64_t *thiskey = + kmem_alloc(array_numints * sizeof (*thiskey), KM_SLEEP); ASSERT(zn->zn_key_intlen == sizeof (*thiskey)); - thiskey = kmem_alloc(array_numints * sizeof (*thiskey), - KM_SLEEP); zap_leaf_array_read(l, chunk, sizeof (*thiskey), array_numints, sizeof (*thiskey), array_numints, thiskey); - match = bcmp(thiskey, zn->zn_key_orig, + boolean_t match = bcmp(thiskey, zn->zn_key_orig, array_numints * sizeof (*thiskey)) == 0; kmem_free(thiskey, array_numints * sizeof (*thiskey)); return (match); @@ -365,11 +355,10 @@ zap_leaf_array_match(zap_leaf_t *l, zap_name_t *zn, ASSERT(zn->zn_key_intlen == 1); if (zn->zn_matchtype & MT_NORMALIZE) { char *thisname = kmem_alloc(array_numints, KM_SLEEP); - boolean_t match; zap_leaf_array_read(l, chunk, sizeof (char), array_numints, sizeof (char), array_numints, thisname); - match = zap_match(zn, thisname); + boolean_t match = zap_match(zn, thisname); kmem_free(thisname, array_numints); return (match); } @@ -400,12 +389,11 @@ zap_leaf_array_match(zap_leaf_t *l, zap_name_t *zn, int zap_leaf_lookup(zap_leaf_t *l, zap_name_t *zn, zap_entry_handle_t *zeh) { - uint16_t *chunkp; struct zap_leaf_entry *le; ASSERT3U(zap_leaf_phys(l)->l_hdr.lh_magic, ==, ZAP_LEAF_MAGIC); - for (chunkp = LEAF_HASH_ENTPTR(l, zn->zn_hash); + for (uint16_t *chunkp = LEAF_HASH_ENTPTR(l, zn->zn_hash); *chunkp != CHAIN_END; chunkp = &le->le_next) { uint16_t chunk = *chunkp; le = ZAP_LEAF_ENTRY(l, chunk); @@ -446,17 +434,15 @@ int zap_leaf_lookup_closest(zap_leaf_t *l, uint64_t h, uint32_t cd, zap_entry_handle_t *zeh) { - uint16_t chunk; uint64_t besth = -1ULL; uint32_t bestcd = -1U; uint16_t bestlh = ZAP_LEAF_HASH_NUMENTRIES(l)-1; - uint16_t lh; struct zap_leaf_entry *le; ASSERT3U(zap_leaf_phys(l)->l_hdr.lh_magic, ==, ZAP_LEAF_MAGIC); - for (lh = LEAF_HASH(l, h); lh <= bestlh; lh++) { - for (chunk = zap_leaf_phys(l)->l_hash[lh]; + for (uint16_t lh = LEAF_HASH(l, h); lh <= bestlh; lh++) { + for (uint16_t chunk = zap_leaf_phys(l)->l_hash[lh]; chunk != CHAIN_END; chunk = le->le_next) { le = ZAP_LEAF_ENTRY(l, chunk); @@ -529,11 +515,10 @@ int zap_entry_update(zap_entry_handle_t *zeh, uint8_t integer_size, uint64_t num_integers, const void *buf) { - int delta_chunks; zap_leaf_t *l = zeh->zeh_leaf; struct zap_leaf_entry *le = ZAP_LEAF_ENTRY(l, *zeh->zeh_chunkp); - delta_chunks = ZAP_LEAF_ARRAY_NCHUNKS(num_integers * integer_size) - + int delta_chunks = ZAP_LEAF_ARRAY_NCHUNKS(num_integers * integer_size) - ZAP_LEAF_ARRAY_NCHUNKS(le->le_value_numints * le->le_value_intlen); if ((int)zap_leaf_phys(l)->l_hdr.lh_nfree < delta_chunks) @@ -550,14 +535,12 @@ zap_entry_update(zap_entry_handle_t *zeh, void zap_entry_remove(zap_entry_handle_t *zeh) { - uint16_t entry_chunk; - struct zap_leaf_entry *le; zap_leaf_t *l = zeh->zeh_leaf; ASSERT3P(zeh->zeh_chunkp, !=, &zeh->zeh_fakechunk); - entry_chunk = *zeh->zeh_chunkp; - le = ZAP_LEAF_ENTRY(l, entry_chunk); + uint16_t entry_chunk = *zeh->zeh_chunkp; + struct zap_leaf_entry *le = ZAP_LEAF_ENTRY(l, entry_chunk); ASSERT3U(le->le_type, ==, ZAP_CHUNK_ENTRY); zap_leaf_array_free(l, &le->le_name_chunk); @@ -575,15 +558,12 @@ zap_entry_create(zap_leaf_t *l, zap_name_t *zn, uint32 zap_entry_handle_t *zeh) { uint16_t chunk; - uint16_t *chunkp; struct zap_leaf_entry *le; - uint64_t valuelen; - int numchunks; uint64_t h = zn->zn_hash; - valuelen = integer_size * num_integers; + uint64_t valuelen = integer_size * num_integers; - numchunks = 1 + ZAP_LEAF_ARRAY_NCHUNKS(zn->zn_key_orig_numints * + int numchunks = 1 + ZAP_LEAF_ARRAY_NCHUNKS(zn->zn_key_orig_numints * zn->zn_key_intlen) + ZAP_LEAF_ARRAY_NCHUNKS(valuelen); if (numchunks > ZAP_LEAF_NUMCHUNKS(l)) return (E2BIG); @@ -645,7 +625,7 @@ zap_entry_create(zap_leaf_t *l, zap_name_t *zn, uint32 /* link it into the hash chain */ /* XXX if we did the search above, we could just use that */ - chunkp = zap_leaf_rehash_entry(l, chunk); + uint16_t *chunkp = zap_leaf_rehash_entry(l, chunk); zap_leaf_phys(l)->l_hdr.lh_nentries++; @@ -673,14 +653,13 @@ boolean_t zap_entry_normalization_conflict(zap_entry_handle_t *zeh, zap_name_t *zn, const char *name, zap_t *zap) { - uint64_t chunk; struct zap_leaf_entry *le; boolean_t allocdzn = B_FALSE; if (zap->zap_normflags == 0) return (B_FALSE); - for (chunk = *LEAF_HASH_ENTPTR(zeh->zeh_leaf, zeh->zeh_hash); + for (uint16_t chunk = *LEAF_HASH_ENTPTR(zeh->zeh_leaf, zeh->zeh_hash); chunk != CHAIN_END; chunk = le->le_next) { le = ZAP_LEAF_ENTRY(zeh->zeh_leaf, chunk); if (le->le_hash != zeh->zeh_hash) @@ -763,14 +742,11 @@ zap_leaf_transfer_array(zap_leaf_t *l, uint16_t chunk, static void zap_leaf_transfer_entry(zap_leaf_t *l, int entry, zap_leaf_t *nl) { - struct zap_leaf_entry *le, *nle; - uint16_t chunk; - - le = ZAP_LEAF_ENTRY(l, entry); + struct zap_leaf_entry *le = ZAP_LEAF_ENTRY(l, entry); ASSERT3U(le->le_type, ==, ZAP_CHUNK_ENTRY); - chunk = zap_leaf_chunk_alloc(nl); - nle = ZAP_LEAF_ENTRY(nl, chunk); + uint16_t chunk = zap_leaf_chunk_alloc(nl); + struct zap_leaf_entry *nle = ZAP_LEAF_ENTRY(nl, chunk); *nle = *le; /* structure assignment */ (void) zap_leaf_rehash_entry(nl, chunk); @@ -791,7 +767,6 @@ zap_leaf_transfer_entry(zap_leaf_t *l, int entry, zap_ void zap_leaf_split(zap_leaf_t *l, zap_leaf_t *nl, boolean_t sort) { - int i; int bit = 64 - 1 - zap_leaf_phys(l)->l_hdr.lh_prefix_len; /* set new prefix and prefix_len */ @@ -818,7 +793,7 @@ zap_leaf_split(zap_leaf_t *l, zap_leaf_t *nl, boolean_ * but this accesses memory more sequentially, and when we're * called, the block is usually pretty full. */ - for (i = 0; i < ZAP_LEAF_NUMCHUNKS(l); i++) { + for (int i = 0; i < ZAP_LEAF_NUMCHUNKS(l); i++) { struct zap_leaf_entry *le = ZAP_LEAF_ENTRY(l, i); if (le->le_type != ZAP_CHUNK_ENTRY) continue; @@ -833,9 +808,7 @@ zap_leaf_split(zap_leaf_t *l, zap_leaf_t *nl, boolean_ void zap_leaf_stats(zap_t *zap, zap_leaf_t *l, zap_stats_t *zs) { - int i, n; - - n = zap_f_phys(zap)->zap_ptrtbl.zt_shift - + int n = zap_f_phys(zap)->zap_ptrtbl.zt_shift - zap_leaf_phys(l)->l_hdr.lh_prefix_len; n = MIN(n, ZAP_HISTOGRAM_SIZE-1); zs->zs_leafs_with_2n_pointers[n]++; @@ -851,7 +824,7 @@ zap_leaf_stats(zap_t *zap, zap_leaf_t *l, zap_stats_t n = MIN(n, ZAP_HISTOGRAM_SIZE-1); zs->zs_blocks_n_tenths_full[n]++; - for (i = 0; i < ZAP_LEAF_HASH_NUMENTRIES(l); i++) { + for (int i = 0; i < ZAP_LEAF_HASH_NUMENTRIES(l); i++) { int nentries = 0; int chunk = zap_leaf_phys(l)->l_hash[i]; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c Wed Aug 1 02:59:56 2018 (r337026) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zap_micro.c Wed Aug 1 03:03:15 2018 (r337027) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2016 by Delphix. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2017 Nexenta Systems, Inc. @@ -89,22 +89,20 @@ zap_hash(zap_name_t *zn) ASSERT(zfs_crc64_table[128] == ZFS_CRC64_POLY); if (zap_getflags(zap) & ZAP_FLAG_UINT64_KEY) { *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Aug 1 03:19:31 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C0ED1045224; Wed, 1 Aug 2018 03:19:31 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5FB07082B; Wed, 1 Aug 2018 03:19:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9143D1CC89; Wed, 1 Aug 2018 03:19:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w713JUxJ042947; Wed, 1 Aug 2018 03:19:30 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w713JUkI042946; Wed, 1 Aug 2018 03:19:30 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808010319.w713JUkI042946@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 1 Aug 2018 03:19:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337029 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337029 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 03:19:31 -0000 Author: mav Date: Wed Aug 1 03:19:30 2018 New Revision: 337029 URL: https://svnweb.freebsd.org/changeset/base/337029 Log: 9426 metaslab size can exceed offset addressable by spacemap metaslab size can exceed offset addressable by spacemap. The vdev can address up to 2^63 * SPA_MAXBLOCKSIZE (512). A metaslab can address up to 2^47 * 2^vdev_ashift. Therefore we may need to increase the number of metaslabs so that the maximum metaslab size is capped at the amount that can be addressed by the spacemap. This should happen in vdev_metaslab_set_size(). illumos/illumos-gate@b4bf0cf0458759c67920a031021a9d96cd683cfe Reviewed by: Paul Dagnelie Reviewed by: Matt Ahrens Approved by: Dan McDonald Author: Don Brady Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Wed Aug 1 03:07:33 2018 (r337028) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Wed Aug 1 03:19:30 2018 (r337029) @@ -72,15 +72,21 @@ static vdev_ops_t *vdev_ops_table[] = { /* maximum scrub/resilver I/O queue per leaf vdev */ int zfs_scrub_limit = 10; -/* maximum number of metaslabs per top-level vdev */ +/* target number of metaslabs per top-level vdev */ int vdev_max_ms_count = 200; -/* minimum amount of metaslabs per top-level vdev */ +/* minimum number of metaslabs per top-level vdev */ int vdev_min_ms_count = 16; -/* see comment in vdev_metaslab_set_size() */ +/* practical upper limit of total metaslabs per top-level vdev */ +int vdev_ms_count_limit = 1ULL << 17; + +/* lower limit for metaslab size (512M) */ int vdev_default_ms_shift = 29; +/* upper limit for metaslab size (256G) */ +int vdev_max_ms_shift = 38; + boolean_t vdev_validate_skip = B_FALSE; /* @@ -2028,34 +2034,53 @@ void vdev_metaslab_set_size(vdev_t *vd) { uint64_t asize = vd->vdev_asize; - uint64_t ms_shift = 0; + uint64_t ms_count = asize >> vdev_default_ms_shift; + uint64_t ms_shift; /* - * For vdevs that are bigger than 8G the metaslab size varies in - * a way that the number of metaslabs increases in powers of two, - * linearly in terms of vdev_asize, starting from 16 metaslabs. - * So for vdev_asize of 8G we get 16 metaslabs, for 16G, we get 32, - * and so on, until we hit the maximum metaslab count limit - * [vdev_max_ms_count] from which point the metaslab count stays - * the same. + * There are two dimensions to the metaslab sizing calculation: + * the size of the metaslab and the count of metaslabs per vdev. + * In general, we aim for vdev_max_ms_count (200) metaslabs. The + * range of the dimensions are as follows: + * + * 2^29 <= ms_size <= 2^38 + * 16 <= ms_count <= 131,072 + * + * On the lower end of vdev sizes, we aim for metaslabs sizes of + * at least 512MB (2^29) to minimize fragmentation effects when + * testing with smaller devices. However, the count constraint + * of at least 16 metaslabs will override this minimum size goal. + * + * On the upper end of vdev sizes, we aim for a maximum metaslab + * size of 256GB. However, we will cap the total count to 2^17 + * metaslabs to keep our memory footprint in check. + * + * The net effect of applying above constrains is summarized below. + * + * vdev size metaslab count + * -------------|----------------- + * < 8GB ~16 + * 8GB - 100GB one per 512MB + * 100GB - 50TB ~200 + * 50TB - 32PB one per 256GB + * > 32PB ~131,072 + * ------------------------------- */ - ms_shift = vdev_default_ms_shift; - if ((asize >> ms_shift) < vdev_min_ms_count) { - /* - * For devices that are less than 8G we want to have - * exactly 16 metaslabs. We don't want less as integer - * division rounds down, so less metaslabs mean more - * wasted space. We don't want more as these vdevs are - * small and in the likely event that we are running - * out of space, the SPA will have a hard time finding - * space due to fragmentation. - */ + if (ms_count < vdev_min_ms_count) ms_shift = highbit64(asize / vdev_min_ms_count); - ms_shift = MAX(ms_shift, SPA_MAXBLOCKSHIFT); - - } else if ((asize >> ms_shift) > vdev_max_ms_count) { + else if (ms_count > vdev_max_ms_count) ms_shift = highbit64(asize / vdev_max_ms_count); + else + ms_shift = vdev_default_ms_shift; + + if (ms_shift < SPA_MAXBLOCKSHIFT) { + ms_shift = SPA_MAXBLOCKSHIFT; + } else if (ms_shift > vdev_max_ms_shift) { + ms_shift = vdev_max_ms_shift; + /* cap the total count to constrain memory footprint */ + if ((asize >> ms_shift) > vdev_ms_count_limit) + ms_shift = highbit64(asize / vdev_ms_count_limit); } vd->vdev_ms_shift = ms_shift; From owner-svn-src-vendor@freebsd.org Wed Aug 1 18:28:20 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B90BB1065A58; Wed, 1 Aug 2018 18:28:19 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6EBFE812B2; Wed, 1 Aug 2018 18:28:19 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 514E726240; Wed, 1 Aug 2018 18:28:19 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w71ISJa9011901; Wed, 1 Aug 2018 18:28:19 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w71ISIZa011894; Wed, 1 Aug 2018 18:28:18 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808011828.w71ISIZa011894@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 1 Aug 2018 18:28:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337047 - vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 337047 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 18:28:20 -0000 Author: mav Date: Wed Aug 1 18:28:17 2018 New Revision: 337047 URL: https://svnweb.freebsd.org/changeset/base/337047 Log: 8115 parallel zfs mount Mounting of filesystems in "filesystem/local" is done using `zfs mount -a`, which mounts each filesystems serially. The bottleneck for each mount is the I/O done to load metadata for each filesystem. As such, mounting filesystems using a parallel algorithm should be a big win, and bring down the runtime of "filesystem/local"'s start method. illumos/illumos-gate@591e0e133f9980083db5d64ac33a30bcc3382ff7 Reviewed by: Matthew Ahrens Reviewed by: Pavel Zakharov Reviewed by: Brad Lewis Reviewed by: George Wilson Reviewed by: Paul Dagnelie Reviewed by: Prashanth Sreenivasa Approved by: Matt Ahrens Author: Sebastien Roy Added: vendor/illumos/dist/lib/libzfs/common/libzfs_taskq.c (contents, props changed) vendor/illumos/dist/lib/libzfs/common/libzfs_taskq.h (contents, props changed) Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c ============================================================================== --- vendor/illumos/dist/cmd/zfs/zfs_main.c Wed Aug 1 17:40:17 2018 (r337046) +++ vendor/illumos/dist/cmd/zfs/zfs_main.c Wed Aug 1 18:28:17 2018 (r337047) @@ -60,6 +60,7 @@ #include #include #include +#include #include #include @@ -5732,8 +5733,13 @@ zfs_do_holds(int argc, char **argv) #define CHECK_SPINNER 30 #define SPINNER_TIME 3 /* seconds */ -#define MOUNT_TIME 5 /* seconds */ +#define MOUNT_TIME 1 /* seconds */ +typedef struct get_all_state { + boolean_t ga_verbose; + get_all_cb_t *ga_cbp; +} get_all_state_t; + static int get_one_dataset(zfs_handle_t *zhp, void *data) { @@ -5741,10 +5747,10 @@ get_one_dataset(zfs_handle_t *zhp, void *data) static int spinval = 0; static int spincheck = 0; static time_t last_spin_time = (time_t)0; - get_all_cb_t *cbp = data; + get_all_state_t *state = data; zfs_type_t type = zfs_get_type(zhp); - if (cbp->cb_verbose) { + if (state->ga_verbose) { if (--spincheck < 0) { time_t now = time(NULL); if (last_spin_time + SPINNER_TIME < now) { @@ -5770,26 +5776,24 @@ get_one_dataset(zfs_handle_t *zhp, void *data) zfs_close(zhp); return (0); } - libzfs_add_handle(cbp, zhp); - assert(cbp->cb_used <= cbp->cb_alloc); + libzfs_add_handle(state->ga_cbp, zhp); + assert(state->ga_cbp->cb_used <= state->ga_cbp->cb_alloc); return (0); } static void -get_all_datasets(zfs_handle_t ***dslist, size_t *count, boolean_t verbose) +get_all_datasets(get_all_cb_t *cbp, boolean_t verbose) { - get_all_cb_t cb = { 0 }; - cb.cb_verbose = verbose; - cb.cb_getone = get_one_dataset; + get_all_state_t state = { + .ga_verbose = verbose, + .ga_cbp = cbp + }; if (verbose) set_progress_header(gettext("Reading ZFS config")); - (void) zfs_iter_root(g_zfs, get_one_dataset, &cb); + (void) zfs_iter_root(g_zfs, get_one_dataset, &state); - *dslist = cb.cb_handles; - *count = cb.cb_used; - if (verbose) finish_progress(gettext("done.")); } @@ -5799,9 +5803,20 @@ get_all_datasets(zfs_handle_t ***dslist, size_t *count * similar, we have a common function with an extra parameter to determine which * mode we are using. */ -#define OP_SHARE 0x1 -#define OP_MOUNT 0x2 +typedef enum { OP_SHARE, OP_MOUNT } share_mount_op_t; +typedef struct share_mount_state { + share_mount_op_t sm_op; + boolean_t sm_verbose; + int sm_flags; + char *sm_options; + char *sm_proto; /* only valid for OP_SHARE */ + mutex_t sm_lock; /* protects the remaining fields */ + uint_t sm_total; /* number of filesystems to process */ + uint_t sm_done; /* number of filesystems processed */ + int sm_status; /* -1 if any of the share/mount operations failed */ +} share_mount_state_t; + /* * Share or mount a dataset. */ @@ -6042,6 +6057,29 @@ report_mount_progress(int current, int total) update_progress(info); } +/* + * zfs_foreach_mountpoint() callback that mounts or shares one filesystem and + * updates the progress meter. + */ +static int +share_mount_one_cb(zfs_handle_t *zhp, void *arg) +{ + share_mount_state_t *sms = arg; + int ret; + + ret = share_mount_one(zhp, sms->sm_op, sms->sm_flags, sms->sm_proto, + B_FALSE, sms->sm_options); + + mutex_enter(&sms->sm_lock); + if (ret != 0) + sms->sm_status = ret; + sms->sm_done++; + if (sms->sm_verbose) + report_mount_progress(sms->sm_done, sms->sm_total); + mutex_exit(&sms->sm_lock); + return (ret); +} + static void append_options(char *mntopts, char *newopts) { @@ -6114,8 +6152,6 @@ share_mount(int op, int argc, char **argv) /* check number of arguments */ if (do_all) { - zfs_handle_t **dslist = NULL; - size_t i, count = 0; char *protocol = NULL; if (op == OP_SHARE && argc > 0) { @@ -6136,33 +6172,44 @@ share_mount(int op, int argc, char **argv) } start_progress_timer(); - get_all_datasets(&dslist, &count, verbose); + get_all_cb_t cb = { 0 }; + get_all_datasets(&cb, verbose); - if (count == 0) + if (cb.cb_used == 0) return (0); - qsort(dslist, count, sizeof (void *), libzfs_dataset_cmp); - sa_init_selective_arg_t sharearg; - sharearg.zhandle_arr = dslist; - sharearg.zhandle_len = count; - if ((ret = zfs_init_libshare_arg(zfs_get_handle(dslist[0]), - SA_INIT_SHARE_API_SELECTIVE, &sharearg)) != SA_OK) { - (void) fprintf(stderr, - gettext("Could not initialize libshare, %d"), ret); - return (ret); + if (op == OP_SHARE) { + sa_init_selective_arg_t sharearg; + sharearg.zhandle_arr = cb.cb_handles; + sharearg.zhandle_len = cb.cb_used; + if ((ret = zfs_init_libshare_arg(g_zfs, + SA_INIT_SHARE_API_SELECTIVE, &sharearg)) != SA_OK) { + (void) fprintf(stderr, gettext( + "Could not initialize libshare, %d"), ret); + return (ret); + } } - for (i = 0; i < count; i++) { - if (verbose) - report_mount_progress(i, count); + share_mount_state_t share_mount_state = { 0 }; + share_mount_state.sm_op = op; + share_mount_state.sm_verbose = verbose; + share_mount_state.sm_flags = flags; + share_mount_state.sm_options = options; + share_mount_state.sm_proto = protocol; + share_mount_state.sm_total = cb.cb_used; + (void) mutex_init(&share_mount_state.sm_lock, + LOCK_NORMAL | LOCK_ERRORCHECK, NULL); + /* + * libshare isn't mt-safe, so only do the operation in parallel + * if we're mounting. + */ + zfs_foreach_mountpoint(g_zfs, cb.cb_handles, cb.cb_used, + share_mount_one_cb, &share_mount_state, op == OP_MOUNT); + ret = share_mount_state.sm_status; - if (share_mount_one(dslist[i], op, flags, protocol, - B_FALSE, options) != 0) - ret = 1; - zfs_close(dslist[i]); - } - - free(dslist); + for (int i = 0; i < cb.cb_used; i++) + zfs_close(cb.cb_handles[i]); + free(cb.cb_handles); } else if (argc == 0) { struct mnttab entry; Modified: vendor/illumos/dist/lib/libzfs/common/libzfs.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs.h Wed Aug 1 17:40:17 2018 (r337046) +++ vendor/illumos/dist/lib/libzfs/common/libzfs.h Wed Aug 1 18:28:17 2018 (r337047) @@ -576,12 +576,11 @@ typedef struct get_all_cb { zfs_handle_t **cb_handles; size_t cb_alloc; size_t cb_used; - boolean_t cb_verbose; - int (*cb_getone)(zfs_handle_t *, void *); } get_all_cb_t; +void zfs_foreach_mountpoint(libzfs_handle_t *, zfs_handle_t **, size_t, + zfs_iter_f, void *, boolean_t); void libzfs_add_handle(get_all_cb_t *, zfs_handle_t *); -int libzfs_dataset_cmp(const void *, const void *); /* * Functions to create and destroy datasets. Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Aug 1 17:40:17 2018 (r337046) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Wed Aug 1 18:28:17 2018 (r337047) @@ -54,6 +54,7 @@ #include #include #include +#include #include #include @@ -785,6 +786,8 @@ libzfs_mnttab_cache_compare(const void *arg1, const vo void libzfs_mnttab_init(libzfs_handle_t *hdl) { + (void) mutex_init(&hdl->libzfs_mnttab_cache_lock, + LOCK_NORMAL | LOCK_ERRORCHECK, NULL); assert(avl_numnodes(&hdl->libzfs_mnttab_cache) == 0); avl_create(&hdl->libzfs_mnttab_cache, libzfs_mnttab_cache_compare, sizeof (mnttab_node_t), offsetof(mnttab_node_t, mtn_node)); @@ -825,6 +828,7 @@ libzfs_mnttab_fini(libzfs_handle_t *hdl) free(mtn); } avl_destroy(&hdl->libzfs_mnttab_cache); + (void) mutex_destroy(&hdl->libzfs_mnttab_cache_lock); } void @@ -839,6 +843,7 @@ libzfs_mnttab_find(libzfs_handle_t *hdl, const char *f { mnttab_node_t find; mnttab_node_t *mtn; + int ret = ENOENT; if (!hdl->libzfs_mnttab_enable) { struct mnttab srch = { 0 }; @@ -854,6 +859,7 @@ libzfs_mnttab_find(libzfs_handle_t *hdl, const char *f return (ENOENT); } + mutex_enter(&hdl->libzfs_mnttab_cache_lock); if (avl_numnodes(&hdl->libzfs_mnttab_cache) == 0) libzfs_mnttab_update(hdl); @@ -861,9 +867,10 @@ libzfs_mnttab_find(libzfs_handle_t *hdl, const char *f mtn = avl_find(&hdl->libzfs_mnttab_cache, &find, NULL); if (mtn) { *entry = mtn->mtn_mt; - return (0); + ret = 0; } - return (ENOENT); + mutex_exit(&hdl->libzfs_mnttab_cache_lock); + return (ret); } void @@ -872,14 +879,16 @@ libzfs_mnttab_add(libzfs_handle_t *hdl, const char *sp { mnttab_node_t *mtn; - if (avl_numnodes(&hdl->libzfs_mnttab_cache) == 0) - return; - mtn = zfs_alloc(hdl, sizeof (mnttab_node_t)); - mtn->mtn_mt.mnt_special = zfs_strdup(hdl, special); - mtn->mtn_mt.mnt_mountp = zfs_strdup(hdl, mountp); - mtn->mtn_mt.mnt_fstype = zfs_strdup(hdl, MNTTYPE_ZFS); - mtn->mtn_mt.mnt_mntopts = zfs_strdup(hdl, mntopts); - avl_add(&hdl->libzfs_mnttab_cache, mtn); + mutex_enter(&hdl->libzfs_mnttab_cache_lock); + if (avl_numnodes(&hdl->libzfs_mnttab_cache) != 0) { + mtn = zfs_alloc(hdl, sizeof (mnttab_node_t)); + mtn->mtn_mt.mnt_special = zfs_strdup(hdl, special); + mtn->mtn_mt.mnt_mountp = zfs_strdup(hdl, mountp); + mtn->mtn_mt.mnt_fstype = zfs_strdup(hdl, MNTTYPE_ZFS); + mtn->mtn_mt.mnt_mntopts = zfs_strdup(hdl, mntopts); + avl_add(&hdl->libzfs_mnttab_cache, mtn); + } + mutex_exit(&hdl->libzfs_mnttab_cache_lock); } void @@ -888,6 +897,7 @@ libzfs_mnttab_remove(libzfs_handle_t *hdl, const char mnttab_node_t find; mnttab_node_t *ret; + mutex_enter(&hdl->libzfs_mnttab_cache_lock); find.mtn_mt.mnt_special = (char *)fsname; if ((ret = avl_find(&hdl->libzfs_mnttab_cache, (void *)&find, NULL)) != NULL) { @@ -898,6 +908,7 @@ libzfs_mnttab_remove(libzfs_handle_t *hdl, const char free(ret->mtn_mt.mnt_mntopts); free(ret); } + mutex_exit(&hdl->libzfs_mnttab_cache_lock); } int Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h Wed Aug 1 17:40:17 2018 (r337046) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h Wed Aug 1 18:28:17 2018 (r337047) @@ -22,7 +22,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2011 Pawel Jakub Dawidek. All rights reserved. - * Copyright (c) 2011, 2016 by Delphix. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. */ #ifndef _LIBZFS_IMPL_H @@ -33,6 +33,7 @@ #include #include #include +#include #include #include @@ -73,6 +74,13 @@ struct libzfs_handle { int libzfs_storeerr; /* stuff error messages into buffer */ void *libzfs_sharehdl; /* libshare handle */ boolean_t libzfs_mnttab_enable; + /* + * We need a lock to handle the case where parallel mount + * threads are populating the mnttab cache simultaneously. The + * lock only protects the integrity of the avl tree, and does + * not protect the contents of the mnttab entries themselves. + */ + mutex_t libzfs_mnttab_cache_lock; avl_tree_t libzfs_mnttab_cache; int libzfs_pool_iter; topo_hdl_t *libzfs_topo_hdl; Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Wed Aug 1 17:40:17 2018 (r337046) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Wed Aug 1 18:28:17 2018 (r337047) @@ -22,7 +22,7 @@ /* * Copyright 2015 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2014, 2016 by Delphix. All rights reserved. + * Copyright (c) 2014, 2017 by Delphix. All rights reserved. * Copyright 2016 Igor Kozhukhov * Copyright 2017 Joyent, Inc. * Copyright 2017 RackTop Systems. @@ -34,25 +34,25 @@ * they are used by mount and unmount and when changing a filesystem's * mountpoint. * - * zfs_is_mounted() - * zfs_mount() - * zfs_unmount() - * zfs_unmountall() + * zfs_is_mounted() + * zfs_mount() + * zfs_unmount() + * zfs_unmountall() * * This file also contains the functions used to manage sharing filesystems via * NFS and iSCSI: * - * zfs_is_shared() - * zfs_share() - * zfs_unshare() + * zfs_is_shared() + * zfs_share() + * zfs_unshare() * - * zfs_is_shared_nfs() - * zfs_is_shared_smb() - * zfs_share_proto() - * zfs_shareall(); - * zfs_unshare_nfs() - * zfs_unshare_smb() - * zfs_unshareall_nfs() + * zfs_is_shared_nfs() + * zfs_is_shared_smb() + * zfs_share_proto() + * zfs_shareall(); + * zfs_unshare_nfs() + * zfs_unshare_smb() + * zfs_unshareall_nfs() * zfs_unshareall_smb() * zfs_unshareall() * zfs_unshareall_bypath() @@ -60,8 +60,8 @@ * The following functions are available for pool consumers, and will * mount/unmount and share/unshare all datasets within pool: * - * zpool_enable_datasets() - * zpool_disable_datasets() + * zpool_enable_datasets() + * zpool_disable_datasets() */ #include @@ -83,11 +83,15 @@ #include #include "libzfs_impl.h" +#include "libzfs_taskq.h" #include #include #define MAXISALEN 257 /* based on sysinfo(2) man page */ +static int mount_tq_nthr = 512; /* taskq threads for multi-threaded mounting */ + +static void zfs_mount_task(void *); static int zfs_share_proto(zfs_handle_t *, zfs_share_proto_t *); zfs_share_type_t zfs_is_shared_proto(zfs_handle_t *, char **, zfs_share_proto_t); @@ -1077,25 +1081,32 @@ remove_mountpoint(zfs_handle_t *zhp) } } +/* + * Add the given zfs handle to the cb_handles array, dynamically reallocating + * the array if it is out of space. + */ void libzfs_add_handle(get_all_cb_t *cbp, zfs_handle_t *zhp) { if (cbp->cb_alloc == cbp->cb_used) { size_t newsz; - void *ptr; + zfs_handle_t **newhandles; - newsz = cbp->cb_alloc ? cbp->cb_alloc * 2 : 64; - ptr = zfs_realloc(zhp->zfs_hdl, - cbp->cb_handles, cbp->cb_alloc * sizeof (void *), - newsz * sizeof (void *)); - cbp->cb_handles = ptr; + newsz = cbp->cb_alloc != 0 ? cbp->cb_alloc * 2 : 64; + newhandles = zfs_realloc(zhp->zfs_hdl, + cbp->cb_handles, cbp->cb_alloc * sizeof (zfs_handle_t *), + newsz * sizeof (zfs_handle_t *)); + cbp->cb_handles = newhandles; cbp->cb_alloc = newsz; } cbp->cb_handles[cbp->cb_used++] = zhp; } +/* + * Recursive helper function used during file system enumeration + */ static int -mount_cb(zfs_handle_t *zhp, void *data) +zfs_iter_cb(zfs_handle_t *zhp, void *data) { get_all_cb_t *cbp = data; @@ -1121,104 +1132,350 @@ mount_cb(zfs_handle_t *zhp, void *data) } libzfs_add_handle(cbp, zhp); - if (zfs_iter_filesystems(zhp, mount_cb, cbp) != 0) { + if (zfs_iter_filesystems(zhp, zfs_iter_cb, cbp) != 0) { zfs_close(zhp); return (-1); } return (0); } +/* + * Sort comparator that compares two mountpoint paths. We sort these paths so + * that subdirectories immediately follow their parents. This means that we + * effectively treat the '/' character as the lowest value non-nul char. An + * example sorted list using this comparator would look like: + * + * /foo + * /foo/bar + * /foo/bar/baz + * /foo/baz + * /foo.bar + * + * The mounting code depends on this ordering to deterministically iterate + * over filesystems in order to spawn parallel mount tasks. + */ int -libzfs_dataset_cmp(const void *a, const void *b) +mountpoint_cmp(const void *arga, const void *argb) { - zfs_handle_t **za = (zfs_handle_t **)a; - zfs_handle_t **zb = (zfs_handle_t **)b; + zfs_handle_t *const *zap = arga; + zfs_handle_t *za = *zap; + zfs_handle_t *const *zbp = argb; + zfs_handle_t *zb = *zbp; char mounta[MAXPATHLEN]; char mountb[MAXPATHLEN]; + const char *a = mounta; + const char *b = mountb; boolean_t gota, gotb; - if ((gota = (zfs_get_type(*za) == ZFS_TYPE_FILESYSTEM)) != 0) - verify(zfs_prop_get(*za, ZFS_PROP_MOUNTPOINT, mounta, + gota = (zfs_get_type(za) == ZFS_TYPE_FILESYSTEM); + if (gota) { + verify(zfs_prop_get(za, ZFS_PROP_MOUNTPOINT, mounta, sizeof (mounta), NULL, NULL, 0, B_FALSE) == 0); - if ((gotb = (zfs_get_type(*zb) == ZFS_TYPE_FILESYSTEM)) != 0) - verify(zfs_prop_get(*zb, ZFS_PROP_MOUNTPOINT, mountb, + } + gotb = (zfs_get_type(zb) == ZFS_TYPE_FILESYSTEM); + if (gotb) { + verify(zfs_prop_get(zb, ZFS_PROP_MOUNTPOINT, mountb, sizeof (mountb), NULL, NULL, 0, B_FALSE) == 0); + } - if (gota && gotb) - return (strcmp(mounta, mountb)); + if (gota && gotb) { + while (*a != '\0' && (*a == *b)) { + a++; + b++; + } + if (*a == *b) + return (0); + if (*a == '\0') + return (-1); + if (*b == '\0') + return (1); + if (*a == '/') + return (-1); + if (*b == '/') + return (1); + return (*a < *b ? -1 : *a > *b); + } if (gota) return (-1); if (gotb) return (1); - return (strcmp(zfs_get_name(a), zfs_get_name(b))); + /* + * If neither filesystem has a mountpoint, revert to sorting by + * dataset name. + */ + return (strcmp(zfs_get_name(za), zfs_get_name(zb))); } /* + * Return true if path2 is a child of path1. + */ +static boolean_t +libzfs_path_contains(const char *path1, const char *path2) +{ + return (strstr(path2, path1) == path2 && path2[strlen(path1)] == '/'); +} + +/* + * Given a mountpoint specified by idx in the handles array, find the first + * non-descendent of that mountpoint and return its index. Descendant paths + * start with the parent's path. This function relies on the ordering + * enforced by mountpoint_cmp(). + */ +static int +non_descendant_idx(zfs_handle_t **handles, size_t num_handles, int idx) +{ + char parent[ZFS_MAXPROPLEN]; + char child[ZFS_MAXPROPLEN]; + int i; + + verify(zfs_prop_get(handles[idx], ZFS_PROP_MOUNTPOINT, parent, + sizeof (parent), NULL, NULL, 0, B_FALSE) == 0); + + for (i = idx + 1; i < num_handles; i++) { + verify(zfs_prop_get(handles[i], ZFS_PROP_MOUNTPOINT, child, + sizeof (child), NULL, NULL, 0, B_FALSE) == 0); + if (!libzfs_path_contains(parent, child)) + break; + } + return (i); +} + +typedef struct mnt_param { + libzfs_handle_t *mnt_hdl; + zfs_taskq_t *mnt_tq; + zfs_handle_t **mnt_zhps; /* filesystems to mount */ + size_t mnt_num_handles; + int mnt_idx; /* Index of selected entry to mount */ + zfs_iter_f mnt_func; + void *mnt_data; +} mnt_param_t; + +/* + * Allocate and populate the parameter struct for mount function, and + * schedule mounting of the entry selected by idx. + */ +static void +zfs_dispatch_mount(libzfs_handle_t *hdl, zfs_handle_t **handles, + size_t num_handles, int idx, zfs_iter_f func, void *data, zfs_taskq_t *tq) +{ + mnt_param_t *mnt_param = zfs_alloc(hdl, sizeof (mnt_param_t)); + + mnt_param->mnt_hdl = hdl; + mnt_param->mnt_tq = tq; + mnt_param->mnt_zhps = handles; + mnt_param->mnt_num_handles = num_handles; + mnt_param->mnt_idx = idx; + mnt_param->mnt_func = func; + mnt_param->mnt_data = data; + + (void) zfs_taskq_dispatch(tq, zfs_mount_task, (void*)mnt_param, + ZFS_TQ_SLEEP); +} + +/* + * This is the structure used to keep state of mounting or sharing operations + * during a call to zpool_enable_datasets(). + */ +typedef struct mount_state { + /* + * ms_mntstatus is set to -1 if any mount fails. While multiple threads + * could update this variable concurrently, no synchronization is + * needed as it's only ever set to -1. + */ + int ms_mntstatus; + int ms_mntflags; + const char *ms_mntopts; +} mount_state_t; + +static int +zfs_mount_one(zfs_handle_t *zhp, void *arg) +{ + mount_state_t *ms = arg; + int ret = 0; + + if (zfs_mount(zhp, ms->ms_mntopts, ms->ms_mntflags) != 0) + ret = ms->ms_mntstatus = -1; + return (ret); +} + +static int +zfs_share_one(zfs_handle_t *zhp, void *arg) +{ + mount_state_t *ms = arg; + int ret = 0; + + if (zfs_share(zhp) != 0) + ret = ms->ms_mntstatus = -1; + return (ret); +} + +/* + * Task queue function to mount one file system. On completion, it finds and + * schedules its children to be mounted. This depends on the sorting done in + * zfs_foreach_mountpoint(). Note that the degenerate case (chain of entries + * each descending from the previous) will have no parallelism since we always + * have to wait for the parent to finish mounting before we can schedule + * its children. + */ +static void +zfs_mount_task(void *arg) +{ + mnt_param_t *mp = arg; + int idx = mp->mnt_idx; + zfs_handle_t **handles = mp->mnt_zhps; + size_t num_handles = mp->mnt_num_handles; + char mountpoint[ZFS_MAXPROPLEN]; + + verify(zfs_prop_get(handles[idx], ZFS_PROP_MOUNTPOINT, mountpoint, + sizeof (mountpoint), NULL, NULL, 0, B_FALSE) == 0); + + if (mp->mnt_func(handles[idx], mp->mnt_data) != 0) + return; + + /* + * We dispatch tasks to mount filesystems with mountpoints underneath + * this one. We do this by dispatching the next filesystem with a + * descendant mountpoint of the one we just mounted, then skip all of + * its descendants, dispatch the next descendant mountpoint, and so on. + * The non_descendant_idx() function skips over filesystems that are + * descendants of the filesystem we just dispatched. + */ + for (int i = idx + 1; i < num_handles; + i = non_descendant_idx(handles, num_handles, i)) { + char child[ZFS_MAXPROPLEN]; + verify(zfs_prop_get(handles[i], ZFS_PROP_MOUNTPOINT, + child, sizeof (child), NULL, NULL, 0, B_FALSE) == 0); + + if (!libzfs_path_contains(mountpoint, child)) + break; /* not a descendant, return */ + zfs_dispatch_mount(mp->mnt_hdl, handles, num_handles, i, + mp->mnt_func, mp->mnt_data, mp->mnt_tq); + } + free(mp); +} + +/* + * Issue the func callback for each ZFS handle contained in the handles + * array. This function is used to mount all datasets, and so this function + * guarantees that filesystems for parent mountpoints are called before their + * children. As such, before issuing any callbacks, we first sort the array + * of handles by mountpoint. + * + * Callbacks are issued in one of two ways: + * + * 1. Sequentially: If the parallel argument is B_FALSE or the ZFS_SERIAL_MOUNT + * environment variable is set, then we issue callbacks sequentially. + * + * 2. In parallel: If the parallel argument is B_TRUE and the ZFS_SERIAL_MOUNT + * environment variable is not set, then we use a taskq to dispatch threads + * to mount filesystems is parallel. This function dispatches tasks to mount + * the filesystems at the top-level mountpoints, and these tasks in turn + * are responsible for recursively mounting filesystems in their children + * mountpoints. + */ +void +zfs_foreach_mountpoint(libzfs_handle_t *hdl, zfs_handle_t **handles, + size_t num_handles, zfs_iter_f func, void *data, boolean_t parallel) +{ + /* + * The ZFS_SERIAL_MOUNT environment variable is an undocumented + * variable that can be used as a convenience to do a/b comparison + * of serial vs. parallel mounting. + */ + boolean_t serial_mount = !parallel || + (getenv("ZFS_SERIAL_MOUNT") != NULL); + + /* + * Sort the datasets by mountpoint. See mountpoint_cmp for details + * of how these are sorted. + */ + qsort(handles, num_handles, sizeof (zfs_handle_t *), mountpoint_cmp); + + if (serial_mount) { + for (int i = 0; i < num_handles; i++) { + func(handles[i], data); + } + return; + } + + /* + * Issue the callback function for each dataset using a parallel + * algorithm that uses a taskq to manage threads. + */ + zfs_taskq_t *tq = zfs_taskq_create("mount_taskq", mount_tq_nthr, 0, + mount_tq_nthr, mount_tq_nthr, ZFS_TASKQ_PREPOPULATE); + + /* + * There may be multiple "top level" mountpoints outside of the pool's + * root mountpoint, e.g.: /foo /bar. Dispatch a mount task for each of + * these. + */ + for (int i = 0; i < num_handles; + i = non_descendant_idx(handles, num_handles, i)) { + zfs_dispatch_mount(hdl, handles, num_handles, i, func, data, + tq); + } + + zfs_taskq_wait(tq); /* wait for all scheduled mounts to complete */ + zfs_taskq_destroy(tq); +} + +/* * Mount and share all datasets within the given pool. This assumes that no - * datasets within the pool are currently mounted. Because users can create - * complicated nested hierarchies of mountpoints, we first gather all the - * datasets and mountpoints within the pool, and sort them by mountpoint. Once - * we have the list of all filesystems, we iterate over them in order and mount - * and/or share each one. + * datasets within the pool are currently mounted. */ #pragma weak zpool_mount_datasets = zpool_enable_datasets int zpool_enable_datasets(zpool_handle_t *zhp, const char *mntopts, int flags) { get_all_cb_t cb = { 0 }; - libzfs_handle_t *hdl = zhp->zpool_hdl; + mount_state_t ms = { 0 }; zfs_handle_t *zfsp; - int i, ret = -1; - int *good; + sa_init_selective_arg_t sharearg; + int ret = 0; - /* - * Gather all non-snap datasets within the pool. - */ - if ((zfsp = zfs_open(hdl, zhp->zpool_name, ZFS_TYPE_DATASET)) == NULL) + if ((zfsp = zfs_open(zhp->zpool_hdl, zhp->zpool_name, + ZFS_TYPE_DATASET)) == NULL) goto out; - libzfs_add_handle(&cb, zfsp); - if (zfs_iter_filesystems(zfsp, mount_cb, &cb) != 0) - goto out; - /* - * Sort the datasets by mountpoint. - */ - qsort(cb.cb_handles, cb.cb_used, sizeof (void *), - libzfs_dataset_cmp); /* - * And mount all the datasets, keeping track of which ones - * succeeded or failed. + * Gather all non-snapshot datasets within the pool. Start by adding + * the root filesystem for this pool to the list, and then iterate + * over all child filesystems. */ - if ((good = zfs_alloc(zhp->zpool_hdl, - cb.cb_used * sizeof (int))) == NULL) + libzfs_add_handle(&cb, zfsp); + if (zfs_iter_filesystems(zfsp, zfs_iter_cb, &cb) != 0) goto out; - ret = 0; - for (i = 0; i < cb.cb_used; i++) { - if (zfs_mount(cb.cb_handles[i], mntopts, flags) != 0) - ret = -1; - else - good[i] = 1; - } + ms.ms_mntopts = mntopts; + ms.ms_mntflags = flags; + zfs_foreach_mountpoint(zhp->zpool_hdl, cb.cb_handles, cb.cb_used, + zfs_mount_one, &ms, B_TRUE); + if (ms.ms_mntstatus != 0) + ret = ms.ms_mntstatus; /* - * Then share all the ones that need to be shared. This needs - * to be a separate pass in order to avoid excessive reloading - * of the configuration. Good should never be NULL since - * zfs_alloc is supposed to exit if memory isn't available. + * Share all filesystems that need to be shared. This needs to be + * a separate pass because libshare is not mt-safe, and so we need + * to share serially. */ - for (i = 0; i < cb.cb_used; i++) { - if (good[i] && zfs_share(cb.cb_handles[i]) != 0) - ret = -1; - } + sharearg.zhandle_arr = cb.cb_handles; + sharearg.zhandle_len = cb.cb_used; + if ((ret = zfs_init_libshare_arg(zhp->zpool_hdl, + SA_INIT_SHARE_API_SELECTIVE, &sharearg)) != 0) + goto out; - free(good); + ms.ms_mntstatus = 0; + zfs_foreach_mountpoint(zhp->zpool_hdl, cb.cb_handles, cb.cb_used, + zfs_share_one, &ms, B_FALSE); + if (ms.ms_mntstatus != 0) + ret = ms.ms_mntstatus; out: - for (i = 0; i < cb.cb_used; i++) + for (int i = 0; i < cb.cb_used; i++) zfs_close(cb.cb_handles[i]); free(cb.cb_handles); Added: vendor/illumos/dist/lib/libzfs/common/libzfs_taskq.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_taskq.c Wed Aug 1 18:28:17 2018 (r337047) @@ -0,0 +1,297 @@ +/* + * CDDL HEADER START + * + * The contents of this file are subject to the terms of the + * Common Development and Distribution License (the "License"). + * You may not use this file except in compliance with the License. + * + * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE + * or http://www.opensolaris.org/os/licensing. + * See the License for the specific language governing permissions + * and limitations under the License. + * + * When distributing Covered Code, include this CDDL HEADER in each + * file and include the License file at usr/src/OPENSOLARIS.LICENSE. + * If applicable, add the following below this CDDL HEADER, with the + * fields enclosed by brackets "[]" replaced with your own identifying + * information: Portions Copyright [yyyy] [name of copyright owner] + * + * CDDL HEADER END + */ +/* + * Copyright 2010 Sun Microsystems, Inc. All rights reserved. + * Use is subject to license terms. + */ +/* + * Copyright 2011 Nexenta Systems, Inc. All rights reserved. + * Copyright 2012 Garrett D'Amore . All rights reserved. + * Copyright (c) 2014, 2018 by Delphix. All rights reserved. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include "libzfs_taskq.h" + +#define ZFS_TASKQ_ACTIVE 0x00010000 +#define ZFS_TASKQ_NAMELEN 31 + +typedef struct zfs_taskq_ent { + struct zfs_taskq_ent *ztqent_next; + struct zfs_taskq_ent *ztqent_prev; + ztask_func_t *ztqent_func; + void *ztqent_arg; + uintptr_t ztqent_flags; +} zfs_taskq_ent_t; + +struct zfs_taskq { + char ztq_name[ZFS_TASKQ_NAMELEN + 1]; + mutex_t ztq_lock; + rwlock_t ztq_threadlock; + cond_t ztq_dispatch_cv; + cond_t ztq_wait_cv; + thread_t *ztq_threadlist; + int ztq_flags; + int ztq_active; + int ztq_nthreads; + int ztq_nalloc; + int ztq_minalloc; + int ztq_maxalloc; + cond_t ztq_maxalloc_cv; + int ztq_maxalloc_wait; + zfs_taskq_ent_t *ztq_freelist; + zfs_taskq_ent_t ztq_task; +}; + +static zfs_taskq_ent_t * +ztask_alloc(zfs_taskq_t *ztq, int ztqflags) +{ + zfs_taskq_ent_t *t; + timestruc_t ts; + int err; + +again: if ((t = ztq->ztq_freelist) != NULL && + ztq->ztq_nalloc >= ztq->ztq_minalloc) { + ztq->ztq_freelist = t->ztqent_next; + } else { + if (ztq->ztq_nalloc >= ztq->ztq_maxalloc) { + if (!(ztqflags & UMEM_NOFAIL)) + return (NULL); + + /* + * We don't want to exceed ztq_maxalloc, but we can't + * wait for other tasks to complete (and thus free up + * task structures) without risking deadlock with + * the caller. So, we just delay for one second + * to throttle the allocation rate. If we have tasks + * complete before one second timeout expires then + * zfs_taskq_ent_free will signal us and we will + * immediately retry the allocation. + */ + ztq->ztq_maxalloc_wait++; + + ts.tv_sec = 1; + ts.tv_nsec = 0; + err = cond_reltimedwait(&ztq->ztq_maxalloc_cv, + &ztq->ztq_lock, &ts); + + ztq->ztq_maxalloc_wait--; + if (err == 0) + goto again; /* signaled */ + } + mutex_exit(&ztq->ztq_lock); + + t = umem_alloc(sizeof (zfs_taskq_ent_t), ztqflags); + + mutex_enter(&ztq->ztq_lock); + if (t != NULL) + ztq->ztq_nalloc++; + } + return (t); +} + +static void +ztask_free(zfs_taskq_t *ztq, zfs_taskq_ent_t *t) +{ + if (ztq->ztq_nalloc <= ztq->ztq_minalloc) { + t->ztqent_next = ztq->ztq_freelist; + ztq->ztq_freelist = t; + } else { + ztq->ztq_nalloc--; + mutex_exit(&ztq->ztq_lock); + umem_free(t, sizeof (zfs_taskq_ent_t)); + mutex_enter(&ztq->ztq_lock); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Wed Aug 1 18:28:18 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 777161065A54; Wed, 1 Aug 2018 18:28:18 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 28909812B0; Wed, 1 Aug 2018 18:28:18 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0A56B2623F; Wed, 1 Aug 2018 18:28:18 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w71ISHr8011889; Wed, 1 Aug 2018 18:28:17 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w71ISHVj011888; Wed, 1 Aug 2018 18:28:17 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808011828.w71ISHVj011888@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 1 Aug 2018 18:28:17 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337047 - vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/zfs vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 337047 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Aug 2018 18:28:18 -0000 Author: mav Date: Wed Aug 1 18:28:17 2018 New Revision: 337047 URL: https://svnweb.freebsd.org/changeset/base/337047 Log: 8115 parallel zfs mount Mounting of filesystems in "filesystem/local" is done using `zfs mount -a`, which mounts each filesystems serially. The bottleneck for each mount is the I/O done to load metadata for each filesystem. As such, mounting filesystems using a parallel algorithm should be a big win, and bring down the runtime of "filesystem/local"'s start method. illumos/illumos-gate@591e0e133f9980083db5d64ac33a30bcc3382ff7 Reviewed by: Matthew Ahrens Reviewed by: Pavel Zakharov Reviewed by: Brad Lewis Reviewed by: George Wilson Reviewed by: Paul Dagnelie Reviewed by: Prashanth Sreenivasa Approved by: Matt Ahrens Author: Sebastien Roy Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h Changes in other areas also in this revision: Added: vendor/illumos/dist/lib/libzfs/common/libzfs_taskq.c (contents, props changed) vendor/illumos/dist/lib/libzfs/common/libzfs_taskq.h (contents, props changed) Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/lib/libzfs/common/libzfs_impl.h vendor/illumos/dist/lib/libzfs/common/libzfs_mount.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h Wed Aug 1 17:40:17 2018 (r337046) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dsl_pool.h Wed Aug 1 18:28:17 2018 (r337047) @@ -89,7 +89,7 @@ typedef struct dsl_pool { struct dsl_dir *dp_leak_dir; struct dsl_dataset *dp_origin_snap; uint64_t dp_root_dir_obj; - struct taskq *dp_vnrele_taskq; + taskq_t *dp_vnrele_taskq; /* No lock needed - sync context only */ blkptr_t dp_meta_rootbp; From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:32:56 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43238106D689; Thu, 2 Aug 2018 17:32:56 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E41117210A; Thu, 2 Aug 2018 17:32:55 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ACFF114D2E; Thu, 2 Aug 2018 17:32:55 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HWtHO029235; Thu, 2 Aug 2018 17:32:55 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HWtgt029234; Thu, 2 Aug 2018 17:32:55 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021732.w72HWtgt029234@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:32:55 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337138 - vendor/llvm/llvm-trunk-r338536 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/llvm/llvm-trunk-r338536 X-SVN-Commit-Revision: 337138 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:32:56 -0000 Author: dim Date: Thu Aug 2 17:32:55 2018 New Revision: 337138 URL: https://svnweb.freebsd.org/changeset/base/337138 Log: Tag llvm trunk r338536. Added: vendor/llvm/llvm-trunk-r338536/ - copied from r337137, vendor/llvm/dist/ From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:32:52 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56D81106D680; Thu, 2 Aug 2018 17:32:52 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09229720FF; Thu, 2 Aug 2018 17:32:52 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C4F7214D2D; Thu, 2 Aug 2018 17:32:51 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HWpGD029189; Thu, 2 Aug 2018 17:32:51 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HWiRj029156; Thu, 2 Aug 2018 17:32:44 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021732.w72HWiRj029156@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:32:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337137 - in vendor/llvm/dist: cmake/modules docs docs/CommandGuide include/llvm include/llvm/ADT include/llvm/Analysis include/llvm/BinaryFormat include/llvm/CodeGen include/llvm/CodeG... X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/llvm/dist: cmake/modules docs docs/CommandGuide include/llvm include/llvm/ADT include/llvm/Analysis include/llvm/BinaryFormat include/llvm/CodeGen include/llvm/CodeGen/GlobalISel include/llv... X-SVN-Commit-Revision: 337137 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:32:53 -0000 Author: dim Date: Thu Aug 2 17:32:43 2018 New Revision: 337137 URL: https://svnweb.freebsd.org/changeset/base/337137 Log: Vendor import of llvm trunk r338536: https://llvm.org/svn/llvm-project/llvm/trunk@338536 Added: vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDebugAddr.h (contents, props changed) vendor/llvm/dist/lib/DebugInfo/DWARF/DWARFDebugAddr.cpp (contents, props changed) vendor/llvm/dist/lib/Transforms/Vectorize/VPlanDominatorTree.h (contents, props changed) vendor/llvm/dist/lib/Transforms/Vectorize/VPlanLoopInfo.h (contents, props changed) vendor/llvm/dist/test/Analysis/BasicAA/phi-values-usage.ll vendor/llvm/dist/test/CodeGen/AArch64/GlobalISel/irtranslator-block-order.ll vendor/llvm/dist/test/CodeGen/AArch64/GlobalISel/legalize-blockaddress.mir vendor/llvm/dist/test/CodeGen/AArch64/GlobalISel/select-blockaddress.mir vendor/llvm/dist/test/CodeGen/AArch64/machine-outliner-default.mir vendor/llvm/dist/test/CodeGen/AArch64/machine-outliner-regsave.mir vendor/llvm/dist/test/CodeGen/AMDGPU/fmaxnum.r600.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fminnum.r600.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.image.sample.ltolz.ll vendor/llvm/dist/test/CodeGen/Hexagon/bit-cmp0.mir vendor/llvm/dist/test/CodeGen/Mips/GlobalISel/instruction-select/gloal_address.mir vendor/llvm/dist/test/CodeGen/Mips/GlobalISel/irtranslator/global_address.ll vendor/llvm/dist/test/CodeGen/Mips/GlobalISel/legalizer/global_address.mir vendor/llvm/dist/test/CodeGen/Mips/GlobalISel/llvm-ir/global_address.ll vendor/llvm/dist/test/CodeGen/Mips/GlobalISel/regbankselect/global_address.mir vendor/llvm/dist/test/CodeGen/X86/pmaddubsw.ll vendor/llvm/dist/test/CodeGen/X86/win_coreclr_chkstk_liveins.mir vendor/llvm/dist/test/DebugInfo/PDB/pdb-invalid-type.test vendor/llvm/dist/test/DebugInfo/PDB/using-namespace.test vendor/llvm/dist/test/DebugInfo/RISCV/ vendor/llvm/dist/test/DebugInfo/RISCV/lit.local.cfg vendor/llvm/dist/test/DebugInfo/RISCV/relax-debug-line.ll vendor/llvm/dist/test/DebugInfo/X86/debug_addr.ll vendor/llvm/dist/test/Demangle/ms-cxx11.test vendor/llvm/dist/test/Demangle/ms-nested-scopes.test vendor/llvm/dist/test/Demangle/ms-return-qualifiers.test vendor/llvm/dist/test/Demangle/ms-template-callback.test vendor/llvm/dist/test/MC/AArch64/SVE/brka-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brka.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkas-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkas.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkb-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkb.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkbs-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkbs.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkn-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkn.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkns-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkns.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpa-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpa.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpas-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpas.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpb-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpb.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpbs-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/brkpbs.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/ctermeq-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/ctermeq.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/ctermne-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/ctermne.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/movprfx-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/movprfx.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/pfalse-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/pfalse.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/pfirst-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/pfirst.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/pnext-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/pnext.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/ptest-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/ptest.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/rev-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/sel-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/uqdecp-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilele-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilele.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilelo-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilelo.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilels-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilels.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilelt-diagnostics.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/SVE/whilelt.s (contents, props changed) vendor/llvm/dist/test/MC/AArch64/inst-directive-other.s (contents, props changed) vendor/llvm/dist/test/MC/ARM/inst-directive-other.s (contents, props changed) vendor/llvm/dist/test/MC/ARM/inst-thumb-suffixes-auto.s (contents, props changed) vendor/llvm/dist/test/Transforms/GlobalOpt/globalsra-multigep.ll vendor/llvm/dist/test/Transforms/InstCombine/pow-cbrt.ll vendor/llvm/dist/test/Transforms/InstCombine/select-binop-icmp.ll vendor/llvm/dist/test/Transforms/InstSimplify/select-and-cmp.ll vendor/llvm/dist/test/Transforms/InstSimplify/select-or-cmp.ll vendor/llvm/dist/test/Transforms/SLPVectorizer/AArch64/PR38339.ll vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_64bit_address.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_absent.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_address_size_mismatch.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_address_size_not_multiple.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_dwarf4.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_dwarf64.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_empty.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_invalid_addr_size.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_segment_selector.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_small_length_field.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_too_small_for_length_field.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_too_small_for_section.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_unsupported_version.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-dwarfdump/X86/debug_addr_version_mismatch.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/Atom/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/Broadwell/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/BtVer2/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/Generic/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/Haswell/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/SLM/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/SandyBridge/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/SkylakeClient/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/SkylakeServer/resources-x86_32.s (contents, props changed) vendor/llvm/dist/test/tools/llvm-mca/X86/Znver1/resources-x86_32.s (contents, props changed) vendor/llvm/dist/unittests/Transforms/Vectorize/VPlanDominatorTreeTest.cpp (contents, props changed) vendor/llvm/dist/unittests/Transforms/Vectorize/VPlanLoopInfoTest.cpp (contents, props changed) Deleted: vendor/llvm/dist/tools/llvm-mca/README.txt Modified: vendor/llvm/dist/cmake/modules/AddLLVM.cmake vendor/llvm/dist/docs/CommandGuide/llvm-mca.rst vendor/llvm/dist/docs/GettingStarted.rst vendor/llvm/dist/docs/LangRef.rst vendor/llvm/dist/docs/SourceLevelDebugging.rst vendor/llvm/dist/include/llvm/ADT/DenseSet.h vendor/llvm/dist/include/llvm/Analysis/BasicAliasAnalysis.h vendor/llvm/dist/include/llvm/Analysis/LoopAccessAnalysis.h vendor/llvm/dist/include/llvm/Analysis/MemoryDependenceAnalysis.h vendor/llvm/dist/include/llvm/Analysis/MustExecute.h vendor/llvm/dist/include/llvm/Analysis/TargetTransformInfoImpl.h vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h vendor/llvm/dist/include/llvm/BinaryFormat/Dwarf.def vendor/llvm/dist/include/llvm/BinaryFormat/ELF.h vendor/llvm/dist/include/llvm/CodeGen/GCStrategy.h vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/LegalizerInfo.h vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h vendor/llvm/dist/include/llvm/CodeGen/MachORelocation.h vendor/llvm/dist/include/llvm/CodeGen/MachineModuleInfo.h vendor/llvm/dist/include/llvm/CodeGen/MachineOutliner.h vendor/llvm/dist/include/llvm/CodeGen/ScheduleDAG.h vendor/llvm/dist/include/llvm/CodeGen/StackMaps.h vendor/llvm/dist/include/llvm/CodeGen/TargetLowering.h vendor/llvm/dist/include/llvm/CodeGen/TargetPassConfig.h vendor/llvm/dist/include/llvm/CodeGen/TargetRegisterInfo.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/CodeViewSymbols.def vendor/llvm/dist/include/llvm/DebugInfo/CodeView/SymbolRecord.h vendor/llvm/dist/include/llvm/DebugInfo/DIContext.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFContext.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDataExtractor.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDie.h vendor/llvm/dist/include/llvm/ExecutionEngine/Orc/RPCSerialization.h vendor/llvm/dist/include/llvm/IR/Attributes.td vendor/llvm/dist/include/llvm/IR/Instruction.h vendor/llvm/dist/include/llvm/IR/Instructions.h vendor/llvm/dist/include/llvm/IR/Intrinsics.td vendor/llvm/dist/include/llvm/IR/IntrinsicsAMDGPU.td vendor/llvm/dist/include/llvm/IR/IntrinsicsARM.td vendor/llvm/dist/include/llvm/IR/IntrinsicsPowerPC.td vendor/llvm/dist/include/llvm/IR/LegacyPassManagers.h vendor/llvm/dist/include/llvm/IR/Statepoint.h vendor/llvm/dist/include/llvm/IR/User.h vendor/llvm/dist/include/llvm/LinkAllIR.h vendor/llvm/dist/include/llvm/MC/MCDwarf.h vendor/llvm/dist/include/llvm/MC/MCFragment.h vendor/llvm/dist/include/llvm/MC/MCInstrAnalysis.h vendor/llvm/dist/include/llvm/MC/MCParser/AsmCond.h vendor/llvm/dist/include/llvm/MC/MCStreamer.h vendor/llvm/dist/include/llvm/Object/MachO.h vendor/llvm/dist/include/llvm/PassAnalysisSupport.h vendor/llvm/dist/include/llvm/PassRegistry.h vendor/llvm/dist/include/llvm/ProfileData/Coverage/CoverageMapping.h vendor/llvm/dist/include/llvm/Support/ARMBuildAttributes.h vendor/llvm/dist/include/llvm/Support/DataExtractor.h vendor/llvm/dist/include/llvm/Support/GenericDomTree.h vendor/llvm/dist/include/llvm/Support/MemoryBuffer.h vendor/llvm/dist/include/llvm/Support/SmallVectorMemoryBuffer.h vendor/llvm/dist/include/llvm/Support/TargetOpcodes.def vendor/llvm/dist/include/llvm/Support/xxhash.h vendor/llvm/dist/include/llvm/Target/GenericOpcodes.td vendor/llvm/dist/include/llvm/Target/TargetCallingConv.td vendor/llvm/dist/include/llvm/Target/TargetInstrPredicate.td vendor/llvm/dist/include/llvm/Transforms/Scalar/SpeculativeExecution.h vendor/llvm/dist/include/llvm/Transforms/Utils/CodeExtractor.h vendor/llvm/dist/include/llvm/Transforms/Utils/FunctionComparator.h vendor/llvm/dist/include/llvm/Transforms/Utils/SymbolRewriter.h vendor/llvm/dist/lib/Analysis/AliasSetTracker.cpp vendor/llvm/dist/lib/Analysis/BasicAliasAnalysis.cpp vendor/llvm/dist/lib/Analysis/CFGPrinter.cpp vendor/llvm/dist/lib/Analysis/CallGraph.cpp vendor/llvm/dist/lib/Analysis/CallGraphSCCPass.cpp vendor/llvm/dist/lib/Analysis/DemandedBits.cpp vendor/llvm/dist/lib/Analysis/GlobalsModRef.cpp vendor/llvm/dist/lib/Analysis/InstructionSimplify.cpp vendor/llvm/dist/lib/Analysis/LazyValueInfo.cpp vendor/llvm/dist/lib/Analysis/LoopAccessAnalysis.cpp vendor/llvm/dist/lib/Analysis/MemDepPrinter.cpp vendor/llvm/dist/lib/Analysis/MemoryDependenceAnalysis.cpp vendor/llvm/dist/lib/Analysis/MustExecute.cpp vendor/llvm/dist/lib/Analysis/ScalarEvolution.cpp vendor/llvm/dist/lib/Analysis/TargetTransformInfo.cpp vendor/llvm/dist/lib/Analysis/ValueTracking.cpp vendor/llvm/dist/lib/AsmParser/LLParser.cpp vendor/llvm/dist/lib/Bitcode/Writer/BitcodeWriter.cpp vendor/llvm/dist/lib/CodeGen/AntiDepBreaker.h vendor/llvm/dist/lib/CodeGen/AsmPrinter/AddressPool.cpp vendor/llvm/dist/lib/CodeGen/AsmPrinter/AddressPool.h vendor/llvm/dist/lib/CodeGen/AsmPrinter/DwarfDebug.cpp vendor/llvm/dist/lib/CodeGen/AsmPrinter/DwarfDebug.h vendor/llvm/dist/lib/CodeGen/AsmPrinter/DwarfExpression.h vendor/llvm/dist/lib/CodeGen/AsmPrinter/DwarfFile.cpp vendor/llvm/dist/lib/CodeGen/AsmPrinter/DwarfUnit.cpp vendor/llvm/dist/lib/CodeGen/AtomicExpandPass.cpp vendor/llvm/dist/lib/CodeGen/BuiltinGCs.cpp vendor/llvm/dist/lib/CodeGen/CriticalAntiDepBreaker.cpp vendor/llvm/dist/lib/CodeGen/GCMetadata.cpp vendor/llvm/dist/lib/CodeGen/GlobalISel/IRTranslator.cpp vendor/llvm/dist/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp vendor/llvm/dist/lib/CodeGen/GlobalMerge.cpp vendor/llvm/dist/lib/CodeGen/IntrinsicLowering.cpp vendor/llvm/dist/lib/CodeGen/LiveDebugValues.cpp vendor/llvm/dist/lib/CodeGen/MachineModuleInfo.cpp vendor/llvm/dist/lib/CodeGen/MachineOutliner.cpp vendor/llvm/dist/lib/CodeGen/MachineRegisterInfo.cpp vendor/llvm/dist/lib/CodeGen/MachineSSAUpdater.cpp vendor/llvm/dist/lib/CodeGen/MachineSink.cpp vendor/llvm/dist/lib/CodeGen/MachineTraceMetrics.cpp vendor/llvm/dist/lib/CodeGen/MachineVerifier.cpp vendor/llvm/dist/lib/CodeGen/RegisterScavenging.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/DAGCombiner.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/FastISel.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/FunctionLoweringInfo.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/LegalizeTypes.h vendor/llvm/dist/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h vendor/llvm/dist/lib/CodeGen/SelectionDAG/StatepointLowering.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/TargetLowering.cpp vendor/llvm/dist/lib/CodeGen/ShadowStackGCLowering.cpp vendor/llvm/dist/lib/CodeGen/SplitKit.h vendor/llvm/dist/lib/CodeGen/TargetLoweringBase.cpp vendor/llvm/dist/lib/CodeGen/TargetLoweringObjectFileImpl.cpp vendor/llvm/dist/lib/CodeGen/TargetPassConfig.cpp vendor/llvm/dist/lib/CodeGen/WinEHPrepare.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/RecordName.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/SymbolDumper.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/SymbolRecordMapping.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/TypeIndexDiscovery.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/TypeStreamMerger.cpp vendor/llvm/dist/lib/DebugInfo/DWARF/CMakeLists.txt vendor/llvm/dist/lib/DebugInfo/DWARF/DWARFAbbreviationDeclaration.cpp vendor/llvm/dist/lib/DebugInfo/DWARF/DWARFContext.cpp vendor/llvm/dist/lib/DebugInfo/Symbolize/SymbolizableObjectFile.cpp vendor/llvm/dist/lib/Demangle/ItaniumDemangle.cpp vendor/llvm/dist/lib/Demangle/MicrosoftDemangle.cpp vendor/llvm/dist/lib/Demangle/StringView.h vendor/llvm/dist/lib/ExecutionEngine/ExecutionEngineBindings.cpp vendor/llvm/dist/lib/ExecutionEngine/IntelJITEvents/ittnotify_config.h vendor/llvm/dist/lib/ExecutionEngine/IntelJITEvents/jitprofiling.h vendor/llvm/dist/lib/ExecutionEngine/Interpreter/Execution.cpp vendor/llvm/dist/lib/ExecutionEngine/Interpreter/Interpreter.h vendor/llvm/dist/lib/ExecutionEngine/RuntimeDyld/RTDyldMemoryManager.cpp vendor/llvm/dist/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp vendor/llvm/dist/lib/FuzzMutate/FuzzerCLI.cpp vendor/llvm/dist/lib/IR/Attributes.cpp vendor/llvm/dist/lib/IR/AutoUpgrade.cpp vendor/llvm/dist/lib/IR/Function.cpp vendor/llvm/dist/lib/IR/InlineAsm.cpp vendor/llvm/dist/lib/IR/Instructions.cpp vendor/llvm/dist/lib/IR/LLVMContextImpl.h vendor/llvm/dist/lib/IR/SymbolTableListTraitsImpl.h vendor/llvm/dist/lib/IR/ValueSymbolTable.cpp vendor/llvm/dist/lib/LTO/ThinLTOCodeGenerator.cpp vendor/llvm/dist/lib/MC/MCAsmStreamer.cpp vendor/llvm/dist/lib/MC/MCAssembler.cpp vendor/llvm/dist/lib/MC/MCDisassembler/Disassembler.cpp vendor/llvm/dist/lib/MC/MCDisassembler/Disassembler.h vendor/llvm/dist/lib/MC/MCDwarf.cpp vendor/llvm/dist/lib/MC/MCInstrAnalysis.cpp vendor/llvm/dist/lib/MC/MCObjectFileInfo.cpp vendor/llvm/dist/lib/MC/MCParser/ELFAsmParser.cpp vendor/llvm/dist/lib/MC/MCStreamer.cpp vendor/llvm/dist/lib/MC/MachObjectWriter.cpp vendor/llvm/dist/lib/Object/COFFObjectFile.cpp vendor/llvm/dist/lib/ObjectYAML/CodeViewYAMLSymbols.cpp vendor/llvm/dist/lib/Support/APFloat.cpp vendor/llvm/dist/lib/Support/ConvertUTF.cpp vendor/llvm/dist/lib/Support/CrashRecoveryContext.cpp vendor/llvm/dist/lib/Support/DAGDeltaAlgorithm.cpp vendor/llvm/dist/lib/Support/Errno.cpp vendor/llvm/dist/lib/Support/FoldingSet.cpp vendor/llvm/dist/lib/Support/FormattedStream.cpp vendor/llvm/dist/lib/Support/ManagedStatic.cpp vendor/llvm/dist/lib/Support/MemoryBuffer.cpp vendor/llvm/dist/lib/Support/Path.cpp vendor/llvm/dist/lib/Support/PrettyStackTrace.cpp vendor/llvm/dist/lib/Support/SourceMgr.cpp vendor/llvm/dist/lib/Support/StringPool.cpp vendor/llvm/dist/lib/Support/StringRef.cpp vendor/llvm/dist/lib/Support/TargetRegistry.cpp vendor/llvm/dist/lib/Support/Windows/Path.inc vendor/llvm/dist/lib/Support/YAMLParser.cpp vendor/llvm/dist/lib/Support/regex_impl.h vendor/llvm/dist/lib/Support/xxhash.cpp vendor/llvm/dist/lib/TableGen/StringMatcher.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64FastISel.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64ISelLowering.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64ISelLowering.h vendor/llvm/dist/lib/Target/AArch64/AArch64InstrFormats.td vendor/llvm/dist/lib/Target/AArch64/AArch64InstrInfo.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64InstrInfo.h vendor/llvm/dist/lib/Target/AArch64/AArch64InstructionSelector.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64LegalizerInfo.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64MachineFunctionInfo.h vendor/llvm/dist/lib/Target/AArch64/AArch64RegisterInfo.td vendor/llvm/dist/lib/Target/AArch64/AArch64SVEInstrInfo.td vendor/llvm/dist/lib/Target/AArch64/AArch64TargetMachine.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64TargetTransformInfo.cpp vendor/llvm/dist/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp vendor/llvm/dist/lib/Target/AArch64/MCTargetDesc/AArch64TargetStreamer.cpp vendor/llvm/dist/lib/Target/AArch64/SVEInstrFormats.td vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUISelLowering.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUInstrInfo.td vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUInstructions.td vendor/llvm/dist/lib/Target/AMDGPU/AMDGPULowerKernelArguments.cpp vendor/llvm/dist/lib/Target/AMDGPU/MIMGInstructions.td vendor/llvm/dist/lib/Target/AMDGPU/SIISelLowering.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIISelLowering.h vendor/llvm/dist/lib/Target/AMDGPU/SIInsertSkips.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIInstrInfo.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIInstrInfo.h vendor/llvm/dist/lib/Target/AMDGPU/SIInstructions.td vendor/llvm/dist/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp vendor/llvm/dist/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h vendor/llvm/dist/lib/Target/AMDGPU/VOP3PInstructions.td vendor/llvm/dist/lib/Target/ARM/ARMAsmPrinter.cpp vendor/llvm/dist/lib/Target/ARM/ARMBaseRegisterInfo.cpp vendor/llvm/dist/lib/Target/ARM/ARMCallingConv.h vendor/llvm/dist/lib/Target/ARM/ARMConstantIslandPass.cpp vendor/llvm/dist/lib/Target/ARM/ARMConstantPoolValue.h vendor/llvm/dist/lib/Target/ARM/ARMFastISel.cpp vendor/llvm/dist/lib/Target/ARM/ARMFrameLowering.cpp vendor/llvm/dist/lib/Target/ARM/ARMISelDAGToDAG.cpp vendor/llvm/dist/lib/Target/ARM/ARMISelLowering.cpp vendor/llvm/dist/lib/Target/ARM/ARMLoadStoreOptimizer.cpp vendor/llvm/dist/lib/Target/ARM/ARMMachineFunctionInfo.h vendor/llvm/dist/lib/Target/ARM/ARMSelectionDAGInfo.cpp vendor/llvm/dist/lib/Target/ARM/ARMTargetTransformInfo.cpp vendor/llvm/dist/lib/Target/ARM/ARMTargetTransformInfo.h vendor/llvm/dist/lib/Target/ARM/AsmParser/ARMAsmParser.cpp vendor/llvm/dist/lib/Target/ARM/Disassembler/ARMDisassembler.cpp vendor/llvm/dist/lib/Target/ARM/InstPrinter/ARMInstPrinter.cpp vendor/llvm/dist/lib/Target/ARM/MCTargetDesc/ARMELFObjectWriter.cpp vendor/llvm/dist/lib/Target/ARM/MCTargetDesc/ARMMCCodeEmitter.cpp vendor/llvm/dist/lib/Target/ARM/MCTargetDesc/ARMTargetStreamer.cpp vendor/llvm/dist/lib/Target/ARM/MLxExpansionPass.cpp vendor/llvm/dist/lib/Target/ARM/Thumb1FrameLowering.cpp vendor/llvm/dist/lib/Target/AVR/AVRISelLowering.cpp vendor/llvm/dist/lib/Target/Hexagon/HexagonBitSimplify.cpp vendor/llvm/dist/lib/Target/Hexagon/HexagonBitTracker.cpp vendor/llvm/dist/lib/Target/Hexagon/HexagonBitTracker.h vendor/llvm/dist/lib/Target/MSP430/MCTargetDesc/MSP430MCAsmInfo.h vendor/llvm/dist/lib/Target/Mips/AsmParser/MipsAsmParser.cpp vendor/llvm/dist/lib/Target/Mips/MCTargetDesc/MipsFixupKinds.h vendor/llvm/dist/lib/Target/Mips/MipsAsmPrinter.cpp vendor/llvm/dist/lib/Target/Mips/MipsCallLowering.cpp vendor/llvm/dist/lib/Target/Mips/MipsConstantIslandPass.cpp vendor/llvm/dist/lib/Target/Mips/MipsFastISel.cpp vendor/llvm/dist/lib/Target/Mips/MipsISelLowering.cpp vendor/llvm/dist/lib/Target/Mips/MipsISelLowering.h vendor/llvm/dist/lib/Target/Mips/MipsInstructionSelector.cpp vendor/llvm/dist/lib/Target/Mips/MipsLegalizerInfo.cpp vendor/llvm/dist/lib/Target/Mips/MipsRegisterBankInfo.cpp vendor/llvm/dist/lib/Target/Mips/MipsSubtarget.h vendor/llvm/dist/lib/Target/NVPTX/NVPTXAsmPrinter.h vendor/llvm/dist/lib/Target/NVPTX/NVPTXImageOptimizer.cpp vendor/llvm/dist/lib/Target/NVPTX/NVPTXMachineFunctionInfo.h vendor/llvm/dist/lib/Target/PowerPC/InstPrinter/PPCInstPrinter.cpp vendor/llvm/dist/lib/Target/PowerPC/InstPrinter/PPCInstPrinter.h vendor/llvm/dist/lib/Target/PowerPC/MCTargetDesc/PPCMCAsmInfo.cpp vendor/llvm/dist/lib/Target/PowerPC/MCTargetDesc/PPCMCCodeEmitter.cpp vendor/llvm/dist/lib/Target/PowerPC/MCTargetDesc/PPCPredicates.h vendor/llvm/dist/lib/Target/PowerPC/PPC.h vendor/llvm/dist/lib/Target/PowerPC/PPCBranchSelector.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCEarlyReturn.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCFastISel.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCFrameLowering.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCHazardRecognizers.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCISelLowering.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCISelLowering.h vendor/llvm/dist/lib/Target/PowerPC/PPCInstrInfo.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCLoopPreIncPrep.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCMCInstLower.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCMIPeephole.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCMachineFunctionInfo.h vendor/llvm/dist/lib/Target/PowerPC/PPCRegisterInfo.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCTargetTransformInfo.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCVSXSwapRemoval.cpp vendor/llvm/dist/lib/Target/Sparc/AsmParser/SparcAsmParser.cpp vendor/llvm/dist/lib/Target/Sparc/Disassembler/SparcDisassembler.cpp vendor/llvm/dist/lib/Target/Sparc/InstPrinter/SparcInstPrinter.cpp vendor/llvm/dist/lib/Target/Sparc/Sparc.h vendor/llvm/dist/lib/Target/Sparc/SparcISelLowering.h vendor/llvm/dist/lib/Target/Sparc/SparcInstrInfo.cpp vendor/llvm/dist/lib/Target/Sparc/SparcTargetMachine.cpp vendor/llvm/dist/lib/Target/SystemZ/SystemZHazardRecognizer.cpp vendor/llvm/dist/lib/Target/SystemZ/SystemZHazardRecognizer.h vendor/llvm/dist/lib/Target/SystemZ/SystemZISelLowering.cpp vendor/llvm/dist/lib/Target/SystemZ/SystemZISelLowering.h vendor/llvm/dist/lib/Target/SystemZ/SystemZInstrInfo.td vendor/llvm/dist/lib/Target/SystemZ/SystemZMachineScheduler.cpp vendor/llvm/dist/lib/Target/SystemZ/SystemZMachineScheduler.h vendor/llvm/dist/lib/Target/SystemZ/SystemZOperands.td vendor/llvm/dist/lib/Target/SystemZ/SystemZOperators.td vendor/llvm/dist/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp vendor/llvm/dist/lib/Target/Target.cpp vendor/llvm/dist/lib/Target/TargetLoweringObjectFile.cpp vendor/llvm/dist/lib/Target/X86/AsmParser/X86AsmParser.cpp vendor/llvm/dist/lib/Target/X86/InstPrinter/X86ATTInstPrinter.cpp vendor/llvm/dist/lib/Target/X86/MCTargetDesc/X86MCTargetDesc.cpp vendor/llvm/dist/lib/Target/X86/X86CallingConv.h vendor/llvm/dist/lib/Target/X86/X86CmovConversion.cpp vendor/llvm/dist/lib/Target/X86/X86FastISel.cpp vendor/llvm/dist/lib/Target/X86/X86FixupLEAs.cpp vendor/llvm/dist/lib/Target/X86/X86FlagsCopyLowering.cpp vendor/llvm/dist/lib/Target/X86/X86FloatingPoint.cpp vendor/llvm/dist/lib/Target/X86/X86FrameLowering.cpp vendor/llvm/dist/lib/Target/X86/X86ISelLowering.cpp vendor/llvm/dist/lib/Target/X86/X86ISelLowering.h vendor/llvm/dist/lib/Target/X86/X86InstrFoldTables.cpp vendor/llvm/dist/lib/Target/X86/X86InstrInfo.cpp vendor/llvm/dist/lib/Target/X86/X86InstrInfo.td vendor/llvm/dist/lib/Target/X86/X86InstrShiftRotate.td vendor/llvm/dist/lib/Target/X86/X86SchedBroadwell.td vendor/llvm/dist/lib/Target/X86/X86SchedHaswell.td vendor/llvm/dist/lib/Target/X86/X86SchedSandyBridge.td vendor/llvm/dist/lib/Target/X86/X86SchedSkylakeClient.td vendor/llvm/dist/lib/Target/X86/X86SchedSkylakeServer.td vendor/llvm/dist/lib/Target/X86/X86Schedule.td vendor/llvm/dist/lib/Target/X86/X86ScheduleAtom.td vendor/llvm/dist/lib/Target/X86/X86ScheduleBtVer2.td vendor/llvm/dist/lib/Target/X86/X86ScheduleSLM.td vendor/llvm/dist/lib/Target/X86/X86ScheduleZnver1.td vendor/llvm/dist/lib/Target/X86/X86Subtarget.h vendor/llvm/dist/lib/Target/X86/X86TargetTransformInfo.cpp vendor/llvm/dist/lib/Target/XCore/XCoreAsmPrinter.cpp vendor/llvm/dist/lib/Target/XCore/XCoreInstrInfo.cpp vendor/llvm/dist/lib/Target/XCore/XCoreMachineFunctionInfo.h vendor/llvm/dist/lib/Target/XCore/XCoreRegisterInfo.cpp vendor/llvm/dist/lib/Target/XCore/XCoreRegisterInfo.h vendor/llvm/dist/lib/Target/XCore/XCoreSubtarget.h vendor/llvm/dist/lib/Transforms/IPO/DeadArgumentElimination.cpp vendor/llvm/dist/lib/Transforms/IPO/FunctionAttrs.cpp vendor/llvm/dist/lib/Transforms/IPO/GlobalOpt.cpp vendor/llvm/dist/lib/Transforms/IPO/IPConstantPropagation.cpp vendor/llvm/dist/lib/Transforms/IPO/MergeFunctions.cpp vendor/llvm/dist/lib/Transforms/IPO/PruneEH.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineAddSub.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineAndOrXor.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineCasts.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineLoadStoreAlloca.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineSelect.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineShifts.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineVectorOps.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstructionCombining.cpp vendor/llvm/dist/lib/Transforms/Instrumentation/AddressSanitizer.cpp vendor/llvm/dist/lib/Transforms/Instrumentation/GCOVProfiling.cpp vendor/llvm/dist/lib/Transforms/Instrumentation/InstrProfiling.cpp vendor/llvm/dist/lib/Transforms/Scalar/AlignmentFromAssumptions.cpp vendor/llvm/dist/lib/Transforms/Scalar/ConstantHoisting.cpp vendor/llvm/dist/lib/Transforms/Scalar/CorrelatedValuePropagation.cpp vendor/llvm/dist/lib/Transforms/Scalar/DeadStoreElimination.cpp vendor/llvm/dist/lib/Transforms/Scalar/EarlyCSE.cpp vendor/llvm/dist/lib/Transforms/Scalar/GVNSink.cpp vendor/llvm/dist/lib/Transforms/Scalar/GuardWidening.cpp vendor/llvm/dist/lib/Transforms/Scalar/InductiveRangeCheckElimination.cpp vendor/llvm/dist/lib/Transforms/Scalar/LICM.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopIdiomRecognize.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopPredication.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopUnrollPass.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopUnswitch.cpp vendor/llvm/dist/lib/Transforms/Scalar/NewGVN.cpp vendor/llvm/dist/lib/Transforms/Scalar/Reassociate.cpp vendor/llvm/dist/lib/Transforms/Scalar/RewriteStatepointsForGC.cpp vendor/llvm/dist/lib/Transforms/Scalar/SROA.cpp vendor/llvm/dist/lib/Transforms/Scalar/SimpleLoopUnswitch.cpp vendor/llvm/dist/lib/Transforms/Utils/BuildLibCalls.cpp vendor/llvm/dist/lib/Transforms/Utils/CallPromotionUtils.cpp vendor/llvm/dist/lib/Transforms/Utils/CloneFunction.cpp vendor/llvm/dist/lib/Transforms/Utils/CloneModule.cpp vendor/llvm/dist/lib/Transforms/Utils/CodeExtractor.cpp vendor/llvm/dist/lib/Transforms/Utils/InlineFunction.cpp vendor/llvm/dist/lib/Transforms/Utils/IntegerDivision.cpp vendor/llvm/dist/lib/Transforms/Utils/LCSSA.cpp vendor/llvm/dist/lib/Transforms/Utils/LoopUnrollPeel.cpp vendor/llvm/dist/lib/Transforms/Utils/MetaRenamer.cpp vendor/llvm/dist/lib/Transforms/Utils/SSAUpdater.cpp vendor/llvm/dist/lib/Transforms/Utils/SimplifyIndVar.cpp vendor/llvm/dist/lib/Transforms/Utils/SimplifyLibCalls.cpp vendor/llvm/dist/lib/Transforms/Utils/SymbolRewriter.cpp vendor/llvm/dist/lib/Transforms/Utils/UnifyFunctionExitNodes.cpp vendor/llvm/dist/lib/Transforms/Vectorize/LoopVectorize.cpp vendor/llvm/dist/lib/Transforms/Vectorize/SLPVectorizer.cpp vendor/llvm/dist/lib/Transforms/Vectorize/VPlan.cpp vendor/llvm/dist/lib/Transforms/Vectorize/VPlan.h vendor/llvm/dist/lib/Transforms/Vectorize/VPlanHCFGBuilder.cpp vendor/llvm/dist/lib/Transforms/Vectorize/VPlanHCFGBuilder.h vendor/llvm/dist/test/Analysis/BasicAA/invalidation.ll vendor/llvm/dist/test/Analysis/BasicAA/phi-aa.ll vendor/llvm/dist/test/CodeGen/AArch64/GlobalISel/arm64-fallback.ll vendor/llvm/dist/test/CodeGen/AArch64/GlobalISel/arm64-irtranslator.ll vendor/llvm/dist/test/CodeGen/AArch64/O3-pipeline.ll vendor/llvm/dist/test/CodeGen/AArch64/arm64-cse.ll vendor/llvm/dist/test/CodeGen/AArch64/arm64-memset-to-bzero.ll vendor/llvm/dist/test/CodeGen/AArch64/arm64-opt-remarks-lazy-bfi.ll vendor/llvm/dist/test/CodeGen/AArch64/cond-sel.ll vendor/llvm/dist/test/CodeGen/AArch64/machine-outliner-flags.ll vendor/llvm/dist/test/CodeGen/AArch64/machine-outliner.ll vendor/llvm/dist/test/CodeGen/AArch64/machine-outliner.mir vendor/llvm/dist/test/CodeGen/AArch64/max-jump-table.ll vendor/llvm/dist/test/CodeGen/AArch64/rotate-extract.ll vendor/llvm/dist/test/CodeGen/AArch64/signbit-shift.ll vendor/llvm/dist/test/CodeGen/AMDGPU/bfi_int.ll vendor/llvm/dist/test/CodeGen/AMDGPU/call-argument-types.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fcanonicalize-elimination.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fcanonicalize.f16.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fcanonicalize.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fmax3.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fmaxnum.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fmin3.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fminnum.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fmul-2-combine-multi-use.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fneg-combines.ll vendor/llvm/dist/test/CodeGen/AMDGPU/function-args.ll vendor/llvm/dist/test/CodeGen/AMDGPU/function-returns.ll vendor/llvm/dist/test/CodeGen/AMDGPU/kernel-args.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.exp.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.sdot2.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.sdot4.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.sdot8.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.sendmsg.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.udot2.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.udot4.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.udot8.ll vendor/llvm/dist/test/CodeGen/AMDGPU/lower-kernargs.ll vendor/llvm/dist/test/CodeGen/AMDGPU/mad-mix-lo.ll vendor/llvm/dist/test/CodeGen/AMDGPU/mad-mix.ll vendor/llvm/dist/test/CodeGen/AMDGPU/mul.i16.ll vendor/llvm/dist/test/CodeGen/AMDGPU/r600.extract-lowbits.ll vendor/llvm/dist/test/CodeGen/AMDGPU/skip-if-dead.ll vendor/llvm/dist/test/CodeGen/AMDGPU/store-global.ll vendor/llvm/dist/test/CodeGen/AMDGPU/store-private.ll vendor/llvm/dist/test/CodeGen/AMDGPU/zero_extend.ll vendor/llvm/dist/test/CodeGen/ARM/aggregate-padding.ll vendor/llvm/dist/test/CodeGen/ARM/inline-asm-operand-implicit-cast.ll vendor/llvm/dist/test/CodeGen/ARM/inlineasm-64bit.ll vendor/llvm/dist/test/CodeGen/ARM/machine-cse-cmp.ll vendor/llvm/dist/test/CodeGen/Mips/const-mult.ll vendor/llvm/dist/test/CodeGen/PowerPC/signbit-shift.ll vendor/llvm/dist/test/CodeGen/RISCV/tail-calls.ll vendor/llvm/dist/test/CodeGen/SystemZ/shift-12.ll vendor/llvm/dist/test/CodeGen/SystemZ/vec-cmp-cmp-logic-select.ll vendor/llvm/dist/test/CodeGen/X86/atom-fixup-lea2.ll vendor/llvm/dist/test/CodeGen/X86/combine-sdiv.ll vendor/llvm/dist/test/CodeGen/X86/combine-shl.ll vendor/llvm/dist/test/CodeGen/X86/dagcombine-select.ll vendor/llvm/dist/test/CodeGen/X86/fast-isel-fold-mem.ll vendor/llvm/dist/test/CodeGen/X86/fast-isel-select.ll vendor/llvm/dist/test/CodeGen/X86/fast-isel-sext-zext.ll vendor/llvm/dist/test/CodeGen/X86/flags-copy-lowering.mir vendor/llvm/dist/test/CodeGen/X86/lea-opt.ll vendor/llvm/dist/test/CodeGen/X86/machine-outliner-tailcalls.ll vendor/llvm/dist/test/CodeGen/X86/mul-constant-i16.ll vendor/llvm/dist/test/CodeGen/X86/mul-constant-i32.ll vendor/llvm/dist/test/CodeGen/X86/mul-constant-i64.ll vendor/llvm/dist/test/CodeGen/X86/pku.ll vendor/llvm/dist/test/CodeGen/X86/rem.ll vendor/llvm/dist/test/CodeGen/X86/rotate-extract-vector.ll vendor/llvm/dist/test/CodeGen/X86/rotate-extract.ll vendor/llvm/dist/test/CodeGen/X86/signbit-shift.ll vendor/llvm/dist/test/CodeGen/X86/speculative-load-hardening.ll vendor/llvm/dist/test/CodeGen/X86/vector-idiv-sdiv-128.ll vendor/llvm/dist/test/CodeGen/X86/vector-idiv-sdiv-256.ll vendor/llvm/dist/test/CodeGen/X86/vector-idiv-sdiv-512.ll vendor/llvm/dist/test/CodeGen/X86/vector-idiv-udiv-128.ll vendor/llvm/dist/test/CodeGen/X86/vector-idiv-udiv-256.ll vendor/llvm/dist/test/CodeGen/X86/vector-idiv-udiv-512.ll vendor/llvm/dist/test/CodeGen/X86/vector-shift-lshr-128.ll vendor/llvm/dist/test/CodeGen/X86/vector-shift-lshr-256.ll vendor/llvm/dist/test/CodeGen/X86/win_coreclr_chkstk.ll vendor/llvm/dist/test/DebugInfo/X86/accel-tables-dwarf5.ll vendor/llvm/dist/test/DebugInfo/X86/accel-tables.ll vendor/llvm/dist/test/Demangle/ms-mangle.test vendor/llvm/dist/test/Instrumentation/InstrProfiling/linkage.ll vendor/llvm/dist/test/Instrumentation/InstrProfiling/platform.ll vendor/llvm/dist/test/MC/AArch64/SVE/abs.s vendor/llvm/dist/test/MC/AArch64/SVE/add-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/add.s vendor/llvm/dist/test/MC/AArch64/SVE/adr-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/and-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/and.s vendor/llvm/dist/test/MC/AArch64/SVE/andv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/asr-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/asr.s vendor/llvm/dist/test/MC/AArch64/SVE/asrd.s vendor/llvm/dist/test/MC/AArch64/SVE/asrr.s vendor/llvm/dist/test/MC/AArch64/SVE/bic-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/bic.s vendor/llvm/dist/test/MC/AArch64/SVE/clasta-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/clasta.s vendor/llvm/dist/test/MC/AArch64/SVE/clastb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/clastb.s vendor/llvm/dist/test/MC/AArch64/SVE/cls.s vendor/llvm/dist/test/MC/AArch64/SVE/clz.s vendor/llvm/dist/test/MC/AArch64/SVE/cmpeq-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmpge-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmpgt-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmphi-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmphs-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmple-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmplo-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmpls-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmplt-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cmpne-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cnot.s vendor/llvm/dist/test/MC/AArch64/SVE/cnt.s vendor/llvm/dist/test/MC/AArch64/SVE/compact-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/cpy.s vendor/llvm/dist/test/MC/AArch64/SVE/decp-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/decp.s vendor/llvm/dist/test/MC/AArch64/SVE/dup-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/dupm-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/eon-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/eon.s vendor/llvm/dist/test/MC/AArch64/SVE/eor-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/eor.s vendor/llvm/dist/test/MC/AArch64/SVE/eorv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ext-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ext.s vendor/llvm/dist/test/MC/AArch64/SVE/fabd.s vendor/llvm/dist/test/MC/AArch64/SVE/fabs.s vendor/llvm/dist/test/MC/AArch64/SVE/facge-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/facgt-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/facle-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/faclt-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fadd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fadd.s vendor/llvm/dist/test/MC/AArch64/SVE/fadda-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/faddv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcadd.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmeq-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmge-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmgt-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmla-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmla.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmle-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmlt-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmne-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcmuo-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fcpy.s vendor/llvm/dist/test/MC/AArch64/SVE/fcvt.s vendor/llvm/dist/test/MC/AArch64/SVE/fcvtzs.s vendor/llvm/dist/test/MC/AArch64/SVE/fcvtzu.s vendor/llvm/dist/test/MC/AArch64/SVE/fdiv.s vendor/llvm/dist/test/MC/AArch64/SVE/fdivr.s vendor/llvm/dist/test/MC/AArch64/SVE/fdup-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fexpa-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmad.s vendor/llvm/dist/test/MC/AArch64/SVE/fmax.s vendor/llvm/dist/test/MC/AArch64/SVE/fmaxnm.s vendor/llvm/dist/test/MC/AArch64/SVE/fmaxnmv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmaxv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmin.s vendor/llvm/dist/test/MC/AArch64/SVE/fminnm.s vendor/llvm/dist/test/MC/AArch64/SVE/fminnmv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fminv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmla-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmla.s vendor/llvm/dist/test/MC/AArch64/SVE/fmls-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmls.s vendor/llvm/dist/test/MC/AArch64/SVE/fmov-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmov.s vendor/llvm/dist/test/MC/AArch64/SVE/fmsb.s vendor/llvm/dist/test/MC/AArch64/SVE/fmul-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fmul.s vendor/llvm/dist/test/MC/AArch64/SVE/fmulx.s vendor/llvm/dist/test/MC/AArch64/SVE/fneg.s vendor/llvm/dist/test/MC/AArch64/SVE/fnmad.s vendor/llvm/dist/test/MC/AArch64/SVE/fnmla.s vendor/llvm/dist/test/MC/AArch64/SVE/fnmls.s vendor/llvm/dist/test/MC/AArch64/SVE/fnmsb.s vendor/llvm/dist/test/MC/AArch64/SVE/frecpe-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/frecps-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/frecpx.s vendor/llvm/dist/test/MC/AArch64/SVE/frinta.s vendor/llvm/dist/test/MC/AArch64/SVE/frinti.s vendor/llvm/dist/test/MC/AArch64/SVE/frintm.s vendor/llvm/dist/test/MC/AArch64/SVE/frintn.s vendor/llvm/dist/test/MC/AArch64/SVE/frintp.s vendor/llvm/dist/test/MC/AArch64/SVE/frintx.s vendor/llvm/dist/test/MC/AArch64/SVE/frintz.s vendor/llvm/dist/test/MC/AArch64/SVE/frsqrte-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/frsqrts-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fscale.s vendor/llvm/dist/test/MC/AArch64/SVE/fsqrt.s vendor/llvm/dist/test/MC/AArch64/SVE/fsub-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/fsub.s vendor/llvm/dist/test/MC/AArch64/SVE/fsubr.s vendor/llvm/dist/test/MC/AArch64/SVE/ftmad-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ftmad.s vendor/llvm/dist/test/MC/AArch64/SVE/ftsmul-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ftssel-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/incd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/incd.s vendor/llvm/dist/test/MC/AArch64/SVE/inch-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/inch.s vendor/llvm/dist/test/MC/AArch64/SVE/incp-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/incp.s vendor/llvm/dist/test/MC/AArch64/SVE/incw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/incw.s vendor/llvm/dist/test/MC/AArch64/SVE/index-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/insr-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/insr.s vendor/llvm/dist/test/MC/AArch64/SVE/lasta-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/lastb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rh-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rqb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rqd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rqh-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rqw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rsb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rsh-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rsw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1rw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1sb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1sh-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1sw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld1w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld2b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld2d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld2h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld2w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld3b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld3d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld3h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld3w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld4b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld4d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld4h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ld4w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldff1b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldff1d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldff1h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldff1sb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldff1sh-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldff1sw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldff1w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnf1b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnf1d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnf1h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnf1sb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnf1sh-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnf1sw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnf1w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnt1b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnt1d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnt1h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ldnt1w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/lsl-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/lsl.s vendor/llvm/dist/test/MC/AArch64/SVE/lslr.s vendor/llvm/dist/test/MC/AArch64/SVE/lsr-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/lsr.s vendor/llvm/dist/test/MC/AArch64/SVE/lsrr.s vendor/llvm/dist/test/MC/AArch64/SVE/mad.s vendor/llvm/dist/test/MC/AArch64/SVE/mla.s vendor/llvm/dist/test/MC/AArch64/SVE/mls.s vendor/llvm/dist/test/MC/AArch64/SVE/mov-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/mov.s vendor/llvm/dist/test/MC/AArch64/SVE/msb.s vendor/llvm/dist/test/MC/AArch64/SVE/mul-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/mul.s vendor/llvm/dist/test/MC/AArch64/SVE/neg.s vendor/llvm/dist/test/MC/AArch64/SVE/not.s vendor/llvm/dist/test/MC/AArch64/SVE/orn-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/orn.s vendor/llvm/dist/test/MC/AArch64/SVE/orr-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/orr.s vendor/llvm/dist/test/MC/AArch64/SVE/orv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/prfb-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/prfd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/prfh-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/prfw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/rbit.s vendor/llvm/dist/test/MC/AArch64/SVE/revb.s vendor/llvm/dist/test/MC/AArch64/SVE/revh.s vendor/llvm/dist/test/MC/AArch64/SVE/revw.s vendor/llvm/dist/test/MC/AArch64/SVE/sabd.s vendor/llvm/dist/test/MC/AArch64/SVE/saddv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/scvtf.s vendor/llvm/dist/test/MC/AArch64/SVE/sdiv.s vendor/llvm/dist/test/MC/AArch64/SVE/sdivr.s vendor/llvm/dist/test/MC/AArch64/SVE/sdot-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sdot.s vendor/llvm/dist/test/MC/AArch64/SVE/smax-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/smax.s vendor/llvm/dist/test/MC/AArch64/SVE/smaxv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/smin-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/smin.s vendor/llvm/dist/test/MC/AArch64/SVE/sminv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/smulh.s vendor/llvm/dist/test/MC/AArch64/SVE/splice-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/splice.s vendor/llvm/dist/test/MC/AArch64/SVE/sqadd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqadd.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdecd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdecd.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdech-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdech.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdecp-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdecp.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdecw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqdecw.s vendor/llvm/dist/test/MC/AArch64/SVE/sqincd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqincd.s vendor/llvm/dist/test/MC/AArch64/SVE/sqinch-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqinch.s vendor/llvm/dist/test/MC/AArch64/SVE/sqincp-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqincp.s vendor/llvm/dist/test/MC/AArch64/SVE/sqincw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqincw.s vendor/llvm/dist/test/MC/AArch64/SVE/sqsub-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sqsub.s vendor/llvm/dist/test/MC/AArch64/SVE/st1b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st1d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st1h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st1w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st2b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st2d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st2h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st2w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st3b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st3d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st3h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st3w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st4b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st4d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st4h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/st4w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/stnt1b-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/stnt1d-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/stnt1h-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/stnt1w-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sub-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sub.s vendor/llvm/dist/test/MC/AArch64/SVE/subr-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/subr.s vendor/llvm/dist/test/MC/AArch64/SVE/sunpkhi-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sunpklo-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/sxtb.s vendor/llvm/dist/test/MC/AArch64/SVE/sxth.s vendor/llvm/dist/test/MC/AArch64/SVE/sxtw.s vendor/llvm/dist/test/MC/AArch64/SVE/tbl-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/trn1-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/trn2-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uabd.s vendor/llvm/dist/test/MC/AArch64/SVE/uaddv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/ucvtf.s vendor/llvm/dist/test/MC/AArch64/SVE/udiv.s vendor/llvm/dist/test/MC/AArch64/SVE/udivr.s vendor/llvm/dist/test/MC/AArch64/SVE/udot-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/udot.s vendor/llvm/dist/test/MC/AArch64/SVE/umax-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/umax.s vendor/llvm/dist/test/MC/AArch64/SVE/umaxv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/umin-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/umin.s vendor/llvm/dist/test/MC/AArch64/SVE/uminv-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/umulh.s vendor/llvm/dist/test/MC/AArch64/SVE/uqadd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqadd.s vendor/llvm/dist/test/MC/AArch64/SVE/uqdecd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqdecd.s vendor/llvm/dist/test/MC/AArch64/SVE/uqdech-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqdech.s vendor/llvm/dist/test/MC/AArch64/SVE/uqdecp.s vendor/llvm/dist/test/MC/AArch64/SVE/uqdecw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqdecw.s vendor/llvm/dist/test/MC/AArch64/SVE/uqincd-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqincd.s vendor/llvm/dist/test/MC/AArch64/SVE/uqinch-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqinch.s vendor/llvm/dist/test/MC/AArch64/SVE/uqincp-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqincp.s vendor/llvm/dist/test/MC/AArch64/SVE/uqincw-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqincw.s vendor/llvm/dist/test/MC/AArch64/SVE/uqsub-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uqsub.s vendor/llvm/dist/test/MC/AArch64/SVE/uunpkhi-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uunpklo-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uxtb.s vendor/llvm/dist/test/MC/AArch64/SVE/uxth.s vendor/llvm/dist/test/MC/AArch64/SVE/uxtw.s vendor/llvm/dist/test/MC/AArch64/SVE/uzp1-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/uzp2-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/zip1-diagnostics.s vendor/llvm/dist/test/MC/AArch64/SVE/zip2-diagnostics.s vendor/llvm/dist/test/MC/AArch64/arm64-directive_loh.s vendor/llvm/dist/test/MC/ARM/directive-unsupported.s vendor/llvm/dist/test/MC/ARM/inst-thumb-suffixes.s vendor/llvm/dist/test/MC/WebAssembly/debug-info.ll vendor/llvm/dist/test/Other/new-pm-defaults.ll vendor/llvm/dist/test/Other/new-pm-lto-defaults.ll vendor/llvm/dist/test/Other/new-pm-thinlto-defaults.ll vendor/llvm/dist/test/Other/opt-O2-pipeline.ll vendor/llvm/dist/test/Other/opt-O3-pipeline.ll vendor/llvm/dist/test/Other/opt-Os-pipeline.ll vendor/llvm/dist/test/Transforms/GVN/PRE/pre-after-rle.ll vendor/llvm/dist/test/Transforms/GlobalOpt/globalsra-partial.ll vendor/llvm/dist/test/Transforms/Inline/attributes.ll vendor/llvm/dist/test/Transforms/InstCombine/and-xor-or.ll vendor/llvm/dist/test/Transforms/InstCombine/and2.ll vendor/llvm/dist/test/Transforms/InstCombine/double-float-shrink-1.ll vendor/llvm/dist/test/Transforms/InstCombine/gep-addrspace.ll vendor/llvm/dist/test/Transforms/InstCombine/pow-1.ll vendor/llvm/dist/test/Transforms/InstCombine/pow-sqrt.ll vendor/llvm/dist/test/Transforms/InstCombine/sub-not.ll vendor/llvm/dist/test/Transforms/InstCombine/xor.ll vendor/llvm/dist/test/Transforms/InstSimplify/AndOrXor.ll vendor/llvm/dist/test/Transforms/InstSimplify/call.ll vendor/llvm/dist/test/Transforms/InstSimplify/shift.ll vendor/llvm/dist/test/Transforms/LCSSA/basictest.ll vendor/llvm/dist/test/Transforms/SCCP/preserve-analysis.ll vendor/llvm/dist/test/Transforms/SimplifyCFG/merge-cond-stores.ll vendor/llvm/dist/test/tools/dsymutil/X86/accelerator.test vendor/llvm/dist/test/tools/dsymutil/X86/update-one-CU.test vendor/llvm/dist/test/tools/dsymutil/X86/update.test vendor/llvm/dist/test/tools/llvm-mca/X86/Atom/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/Broadwell/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/BtVer2/dependency-breaking-cmp.s vendor/llvm/dist/test/tools/llvm-mca/X86/BtVer2/dependency-breaking-pcmpeq.s vendor/llvm/dist/test/tools/llvm-mca/X86/BtVer2/dependency-breaking-sbb-2.s vendor/llvm/dist/test/tools/llvm-mca/X86/BtVer2/one-idioms.s vendor/llvm/dist/test/tools/llvm-mca/X86/BtVer2/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/Generic/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/Haswell/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/SLM/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/SandyBridge/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/SkylakeClient/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/SkylakeServer/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-mca/X86/Znver1/resources-x86_64.s vendor/llvm/dist/test/tools/llvm-objcopy/strip-debug.test vendor/llvm/dist/tools/dsymutil/DwarfLinker.cpp vendor/llvm/dist/tools/dsymutil/DwarfLinker.h vendor/llvm/dist/tools/dsymutil/MachOUtils.cpp vendor/llvm/dist/tools/dsymutil/MachOUtils.h vendor/llvm/dist/tools/dsymutil/dsymutil.cpp vendor/llvm/dist/tools/llvm-mca/DispatchStage.cpp vendor/llvm/dist/tools/llvm-mca/DispatchStage.h vendor/llvm/dist/tools/llvm-mca/InstrBuilder.cpp vendor/llvm/dist/tools/llvm-mca/Instruction.h vendor/llvm/dist/tools/llvm-mca/RetireControlUnit.h vendor/llvm/dist/tools/llvm-mca/RetireStage.cpp vendor/llvm/dist/tools/llvm-mca/llvm-mca.cpp vendor/llvm/dist/tools/llvm-objcopy/llvm-objcopy.cpp vendor/llvm/dist/tools/llvm-pdbutil/MinimalSymbolDumper.cpp vendor/llvm/dist/unittests/DebugInfo/CodeView/TypeIndexDiscoveryTest.cpp vendor/llvm/dist/unittests/DebugInfo/DWARF/DWARFDebugInfoTest.cpp vendor/llvm/dist/unittests/ExecutionEngine/Orc/RPCUtilsTest.cpp vendor/llvm/dist/unittests/Transforms/Vectorize/CMakeLists.txt vendor/llvm/dist/unittests/Transforms/Vectorize/VPlanTestBase.h vendor/llvm/dist/utils/LLVMVisualizers/llvm.natvis vendor/llvm/dist/utils/TableGen/CodeGenDAGPatterns.cpp Modified: vendor/llvm/dist/cmake/modules/AddLLVM.cmake ============================================================================== --- vendor/llvm/dist/cmake/modules/AddLLVM.cmake Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/cmake/modules/AddLLVM.cmake Thu Aug 2 17:32:43 2018 (r337137) @@ -867,6 +867,7 @@ if(NOT LLVM_TOOLCHAIN_TOOLS) llvm-ranlib llvm-lib llvm-objdump + llvm-rc ) endif() Modified: vendor/llvm/dist/docs/CommandGuide/llvm-mca.rst ============================================================================== --- vendor/llvm/dist/docs/CommandGuide/llvm-mca.rst Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/docs/CommandGuide/llvm-mca.rst Thu Aug 2 17:32:43 2018 (r337137) @@ -114,8 +114,8 @@ option specifies "``-``", then the output will also be .. option:: -register-file-size= Specify the size of the register file. When specified, this flag limits how - many temporary registers are available for register renaming purposes. A value - of zero for this flag means "unlimited number of temporary registers". + many physical registers are available for register renaming purposes. A value + of zero for this flag means "unlimited number of physical registers". .. option:: -iterations= @@ -207,23 +207,23 @@ EXIT STATUS :program:`llvm-mca` returns 0 on success. Otherwise, an error message is printed to standard error, and the tool returns 1. -HOW MCA WORKS -------------- +HOW LLVM-MCA WORKS +------------------ -MCA takes assembly code as input. The assembly code is parsed into a sequence -of MCInst with the help of the existing LLVM target assembly parsers. The -parsed sequence of MCInst is then analyzed by a ``Pipeline`` module to generate -a performance report. +:program:`llvm-mca` takes assembly code as input. The assembly code is parsed +into a sequence of MCInst with the help of the existing LLVM target assembly +parsers. The parsed sequence of MCInst is then analyzed by a ``Pipeline`` module +to generate a performance report. The Pipeline module simulates the execution of the machine code sequence in a loop of iterations (default is 100). During this process, the pipeline collects a number of execution related statistics. At the end of this process, the pipeline generates and prints a report from the collected statistics. -Here is an example of a performance report generated by MCA for a dot-product -of two packed float vectors of four elements. The analysis is conducted for -target x86, cpu btver2. The following result can be produced via the following -command using the example located at +Here is an example of a performance report generated by the tool for a +dot-product of two packed float vectors of four elements. The analysis is +conducted for target x86, cpu btver2. The following result can be produced via +the following command using the example located at ``test/tools/llvm-mca/X86/BtVer2/dot-product.s``: .. code-block:: bash @@ -287,11 +287,31 @@ for a total of 900 dynamically executed instructions. The report is structured in three main sections. The first section collects a few performance numbers; the goal of this section is to give a very quick overview of the performance throughput. In this example, the two important -performance indicators are the predicted total number of cycles, and the IPC. -IPC is probably the most important throughput indicator. A big delta between -the Dispatch Width and the computed IPC is an indicator of potential -performance issues. +performance indicators are **IPC** and **Block RThroughput** (Block Reciprocal +Throughput). +IPC is computed dividing the total number of simulated instructions by the total +number of cycles. A delta between Dispatch Width and IPC is an indicator of a +performance issue. In the absence of loop-carried data dependencies, the +observed IPC tends to a theoretical maximum which can be computed by dividing +the number of instructions of a single iteration by the *Block RThroughput*. + +IPC is bounded from above by the dispatch width. That is because the dispatch +width limits the maximum size of a dispatch group. IPC is also limited by the +amount of hardware parallelism. The availability of hardware resources affects +the resource pressure distribution, and it limits the number of instructions +that can be executed in parallel every cycle. A delta between Dispatch +Width and the theoretical maximum IPC is an indicator of a performance +bottleneck caused by the lack of hardware resources. In general, the lower the +Block RThroughput, the better. + +In this example, ``Instructions per iteration/Block RThroughput`` is 1.50. Since +there are no loop-carried dependencies, the observed IPC is expected to approach +1.50 when the number of iterations tends to infinity. The delta between the +Dispatch Width (2.00), and the theoretical maximum IPC (1.50) is an indicator of +a performance bottleneck caused by the lack of hardware resources, and the +*Resource pressure view* can help to identify the problematic resource usage. + The second section of the report shows the latency and reciprocal throughput of every instruction in the sequence. That section also reports extra information related to the number of micro opcodes, and opcode properties @@ -316,7 +336,7 @@ pressure should be uniformly distributed between multi Timeline View ^^^^^^^^^^^^^ -MCA's timeline view produces a detailed report of each instruction's state +The timeline view produces a detailed report of each instruction's state transitions through an instruction pipeline. This view is enabled by the command line option ``-timeline``. As instructions transition through the various stages of the pipeline, their states are depicted in the view report. @@ -331,7 +351,7 @@ These states are represented by the following characte Below is the timeline view for a subset of the dot-product example located in ``test/tools/llvm-mca/X86/BtVer2/dot-product.s`` and processed by -MCA using the following command: +:program:`llvm-mca` using the following command: .. code-block:: bash @@ -366,7 +386,7 @@ MCA using the following command: 2. 3 5.7 0.0 0.0 vhaddps %xmm3, %xmm3, %xmm4 The timeline view is interesting because it shows instruction state changes -during execution. It also gives an idea of how MCA processes instructions +during execution. It also gives an idea of how the tool processes instructions executed on the target, and how their timing information might be calculated. The timeline view is structured in two tables. The first table shows @@ -411,12 +431,12 @@ Parallelism). In the dot-product example, there are anti-dependencies introduced by instructions from different iterations. However, those dependencies can be removed at register renaming stage (at the cost of allocating register aliases, -and therefore consuming temporary registers). +and therefore consuming physical registers). Table *Average Wait times* helps diagnose performance issues that are caused by the presence of long latency instructions and potentially long data dependencies -which may limit the ILP. Note that MCA, by default, assumes at least 1cy -between the dispatch event and the issue event. +which may limit the ILP. Note that :program:`llvm-mca`, by default, assumes at +least 1cy between the dispatch event and the issue event. When the performance is limited by data dependencies and/or long latency instructions, the number of cycles spent while in the *ready* state is expected @@ -549,3 +569,177 @@ statistics are displayed by using the command option ` In this example, we can conclude that the IPC is mostly limited by data dependencies, and not by resource pressure. + +Instruction Flow +^^^^^^^^^^^^^^^^ +This section describes the instruction flow through MCA's default out-of-order +pipeline, as well as the functional units involved in the process. + +The default pipeline implements the following sequence of stages used to +process instructions. + +* Dispatch (Instruction is dispatched to the schedulers). +* Issue (Instruction is issued to the processor pipelines). +* Write Back (Instruction is executed, and results are written back). +* Retire (Instruction is retired; writes are architecturally committed). + +The default pipeline only models the out-of-order portion of a processor. +Therefore, the instruction fetch and decode stages are not modeled. Performance +bottlenecks in the frontend are not diagnosed. MCA assumes that instructions +have all been decoded and placed into a queue. Also, MCA does not model branch +prediction. + +Instruction Dispatch +"""""""""""""""""""" +During the dispatch stage, instructions are picked in program order from a +queue of already decoded instructions, and dispatched in groups to the +simulated hardware schedulers. + +The size of a dispatch group depends on the availability of the simulated +hardware resources. The processor dispatch width defaults to the value +of the ``IssueWidth`` in LLVM's scheduling model. + +An instruction can be dispatched if: + +* The size of the dispatch group is smaller than processor's dispatch width. +* There are enough entries in the reorder buffer. +* There are enough physical registers to do register renaming. +* The schedulers are not full. + +Scheduling models can optionally specify which register files are available on +the processor. MCA uses that information to initialize register file +descriptors. Users can limit the number of physical registers that are +globally available for register renaming by using the command option +``-register-file-size``. A value of zero for this option means *unbounded*. +By knowing how many registers are available for renaming, MCA can predict +dispatch stalls caused by the lack of registers. + +The number of reorder buffer entries consumed by an instruction depends on the +number of micro-opcodes specified by the target scheduling model. MCA's +reorder buffer's purpose is to track the progress of instructions that are +"in-flight," and to retire instructions in program order. The number of +entries in the reorder buffer defaults to the `MicroOpBufferSize` provided by +the target scheduling model. + +Instructions that are dispatched to the schedulers consume scheduler buffer +entries. :program:`llvm-mca` queries the scheduling model to determine the set +of buffered resources consumed by an instruction. Buffered resources are +treated like scheduler resources. + +Instruction Issue +""""""""""""""""" +Each processor scheduler implements a buffer of instructions. An instruction +has to wait in the scheduler's buffer until input register operands become +available. Only at that point, does the instruction becomes eligible for +execution and may be issued (potentially out-of-order) for execution. +Instruction latencies are computed by :program:`llvm-mca` with the help of the +scheduling model. + +:program:`llvm-mca`'s scheduler is designed to simulate multiple processor +schedulers. The scheduler is responsible for tracking data dependencies, and +dynamically selecting which processor resources are consumed by instructions. +It delegates the management of processor resource units and resource groups to a +resource manager. The resource manager is responsible for selecting resource +units that are consumed by instructions. For example, if an instruction +consumes 1cy of a resource group, the resource manager selects one of the +available units from the group; by default, the resource manager uses a +round-robin selector to guarantee that resource usage is uniformly distributed +between all units of a group. + +:program:`llvm-mca`'s scheduler implements three instruction queues: + +* WaitQueue: a queue of instructions whose operands are not ready. +* ReadyQueue: a queue of instructions ready to execute. +* IssuedQueue: a queue of instructions executing. + +Depending on the operand availability, instructions that are dispatched to the +scheduler are either placed into the WaitQueue or into the ReadyQueue. + +Every cycle, the scheduler checks if instructions can be moved from the +WaitQueue to the ReadyQueue, and if instructions from the ReadyQueue can be +issued to the underlying pipelines. The algorithm prioritizes older instructions +over younger instructions. + +Write-Back and Retire Stage +""""""""""""""""""""""""""" +Issued instructions are moved from the ReadyQueue to the IssuedQueue. There, +instructions wait until they reach the write-back stage. At that point, they +get removed from the queue and the retire control unit is notified. + +When instructions are executed, the retire control unit flags the +instruction as "ready to retire." + +Instructions are retired in program order. The register file is notified of +the retirement so that it can free the physical registers that were allocated +for the instruction during the register renaming stage. + +Load/Store Unit and Memory Consistency Model +"""""""""""""""""""""""""""""""""""""""""""" +To simulate an out-of-order execution of memory operations, :program:`llvm-mca` +utilizes a simulated load/store unit (LSUnit) to simulate the speculative +execution of loads and stores. + +Each load (or store) consumes an entry in the load (or store) queue. Users can +specify flags ``-lqueue`` and ``-squeue`` to limit the number of entries in the +load and store queues respectively. The queues are unbounded by default. + +The LSUnit implements a relaxed consistency model for memory loads and stores. +The rules are: + +1. A younger load is allowed to pass an older load only if there are no + intervening stores or barriers between the two loads. +2. A younger load is allowed to pass an older store provided that the load does + not alias with the store. +3. A younger store is not allowed to pass an older store. +4. A younger store is not allowed to pass an older load. + +By default, the LSUnit optimistically assumes that loads do not alias +(`-noalias=true`) store operations. Under this assumption, younger loads are +always allowed to pass older stores. Essentially, the LSUnit does not attempt +to run any alias analysis to predict when loads and stores do not alias with +each other. + +Note that, in the case of write-combining memory, rule 3 could be relaxed to +allow reordering of non-aliasing store operations. That being said, at the +moment, there is no way to further relax the memory model (``-noalias`` is the +only option). Essentially, there is no option to specify a different memory +type (e.g., write-back, write-combining, write-through; etc.) and consequently +to weaken, or strengthen, the memory model. + +Other limitations are: + +* The LSUnit does not know when store-to-load forwarding may occur. +* The LSUnit does not know anything about cache hierarchy and memory types. +* The LSUnit does not know how to identify serializing operations and memory + fences. + +The LSUnit does not attempt to predict if a load or store hits or misses the L1 +cache. It only knows if an instruction "MayLoad" and/or "MayStore." For +loads, the scheduling model provides an "optimistic" load-to-use latency (which +usually matches the load-to-use latency for when there is a hit in the L1D). + +:program:`llvm-mca` does not know about serializing operations or memory-barrier +like instructions. The LSUnit conservatively assumes that an instruction which +has both "MayLoad" and unmodeled side effects behaves like a "soft" +load-barrier. That means, it serializes loads without forcing a flush of the +load queue. Similarly, instructions that "MayStore" and have unmodeled side +effects are treated like store barriers. A full memory barrier is a "MayLoad" +and "MayStore" instruction with unmodeled side effects. This is inaccurate, but +it is the best that we can do at the moment with the current information +available in LLVM. + +A load/store barrier consumes one entry of the load/store queue. A load/store +barrier enforces ordering of loads/stores. A younger load cannot pass a load +barrier. Also, a younger store cannot pass a store barrier. A younger load +has to wait for the memory/load barrier to execute. A load/store barrier is +"executed" when it becomes the oldest entry in the load/store queue(s). That +also means, by construction, all of the older loads/stores have been executed. + +In conclusion, the full set of load/store consistency rules are: + +#. A store may not pass a previous store. +#. A store may not pass a previous load (regardless of ``-noalias``). +#. A store has to wait until an older store barrier is fully executed. +#. A load may pass a previous load. +#. A load may not pass a previous store unless ``-noalias`` is set. +#. A load has to wait until an older load barrier is fully executed. Modified: vendor/llvm/dist/docs/GettingStarted.rst ============================================================================== --- vendor/llvm/dist/docs/GettingStarted.rst Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/docs/GettingStarted.rst Thu Aug 2 17:32:43 2018 (r337137) @@ -838,7 +838,7 @@ To configure LLVM, follow these steps: .. code-block:: console - % cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=prefix=/install/path + % cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=/install/path [other options] SRC_ROOT Compiling the LLVM Suite Source Code Modified: vendor/llvm/dist/docs/LangRef.rst ============================================================================== --- vendor/llvm/dist/docs/LangRef.rst Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/docs/LangRef.rst Thu Aug 2 17:32:43 2018 (r337137) @@ -4588,9 +4588,12 @@ DIExpression ``DIExpression`` nodes represent expressions that are inspired by the DWARF expression language. They are used in :ref:`debug intrinsics` (such as ``llvm.dbg.declare`` and ``llvm.dbg.value``) to describe how the -referenced LLVM variable relates to the source language variable. +referenced LLVM variable relates to the source language variable. Debug +intrinsics are interpreted left-to-right: start by pushing the value/address +operand of the intrinsic onto a stack, then repeatedly push and evaluate +opcodes from the DIExpression until the final variable description is produced. -The current supported vocabulary is limited: +The current supported opcode vocabulary is limited: - ``DW_OP_deref`` dereferences the top of the expression stack. - ``DW_OP_plus`` pops the last two entries from the expression stack, adds @@ -4610,12 +4613,30 @@ The current supported vocabulary is limited: - ``DW_OP_stack_value`` marks a constant value. DWARF specifies three kinds of simple location descriptions: Register, memory, -and implicit location descriptions. Register and memory location descriptions -describe the *location* of a source variable (in the sense that a debugger might -modify its value), whereas implicit locations describe merely the *value* of a -source variable. DIExpressions also follow this model: A DIExpression that -doesn't have a trailing ``DW_OP_stack_value`` will describe an *address* when -combined with a concrete location. +and implicit location descriptions. Note that a location description is +defined over certain ranges of a program, i.e the location of a variable may +change over the course of the program. Register and memory location +descriptions describe the *concrete location* of a source variable (in the +sense that a debugger might modify its value), whereas *implicit locations* +describe merely the actual *value* of a source variable which might not exist +in registers or in memory (see ``DW_OP_stack_value``). + +A ``llvm.dbg.addr`` or ``llvm.dbg.declare`` intrinsic describes an indirect +value (the address) of a source variable. The first operand of the intrinsic +must be an address of some kind. A DIExpression attached to the intrinsic +refines this address to produce a concrete location for the source variable. + +A ``llvm.dbg.value`` intrinsic describes the direct value of a source variable. +The first operand of the intrinsic may be a direct or indirect value. A +DIExpresion attached to the intrinsic refines the first operand to produce a +direct value. For example, if the first operand is an indirect value, it may be +necessary to insert ``DW_OP_deref`` into the DIExpresion in order to produce a +valid debug intrinsic. + +.. note:: + + A DIExpression is interpreted in the same way regardless of which kind of + debug intrinsic it's attached to. .. code-block:: text Modified: vendor/llvm/dist/docs/SourceLevelDebugging.rst ============================================================================== --- vendor/llvm/dist/docs/SourceLevelDebugging.rst Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/docs/SourceLevelDebugging.rst Thu Aug 2 17:32:43 2018 (r337137) @@ -244,6 +244,11 @@ argument is a `local variable `_. +An `llvm.dbg.value` intrinsic describes the *value* of a source variable +directly, not its address. Note that the value operand of this intrinsic may +be indirect (i.e, a pointer to the source variable), provided that interpreting +the complex expression derives the direct value. + Object lifetimes and scoping ============================ Modified: vendor/llvm/dist/include/llvm/ADT/DenseSet.h ============================================================================== --- vendor/llvm/dist/include/llvm/ADT/DenseSet.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/ADT/DenseSet.h Thu Aug 2 17:32:43 2018 (r337137) @@ -17,7 +17,7 @@ #include "llvm/ADT/DenseMap.h" #include "llvm/ADT/DenseMapInfo.h" #include "llvm/Support/type_traits.h" -#include +#include #include #include #include Modified: vendor/llvm/dist/include/llvm/Analysis/BasicAliasAnalysis.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/BasicAliasAnalysis.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/Analysis/BasicAliasAnalysis.h Thu Aug 2 17:32:43 2018 (r337137) @@ -43,6 +43,7 @@ class LoopInfo; class PHINode; class SelectInst; class TargetLibraryInfo; +class PhiValues; class Value; /// This is the AA result object for the basic, local, and stateless alias @@ -60,19 +61,22 @@ class BasicAAResult : public AAResultBase VL, const DataL SmallVectorImpl &SortedIndices); /// Returns true if the memory operations \p A and \p B are consecutive. -/// This is a simple API that does not depend on the analysis pass. +/// This is a simple API that does not depend on the analysis pass. bool isConsecutiveAccess(Value *A, Value *B, const DataLayout &DL, ScalarEvolution &SE, bool CheckType = true); @@ -734,7 +734,7 @@ class LoopAccessLegacyAnalysis : public FunctionPass { /// accesses of a loop. /// /// It runs the analysis for a loop on demand. This can be initiated by -/// querying the loop access info via AM.getResult. +/// querying the loop access info via AM.getResult. /// getResult return a LoopAccessInfo object. See this class for the /// specifics of what information is provided. class LoopAccessAnalysis Modified: vendor/llvm/dist/include/llvm/Analysis/MemoryDependenceAnalysis.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/MemoryDependenceAnalysis.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/Analysis/MemoryDependenceAnalysis.h Thu Aug 2 17:32:43 2018 (r337137) @@ -44,6 +44,7 @@ class Instruction; class LoadInst; class PHITransAddr; class TargetLibraryInfo; +class PhiValues; class Value; /// A memory dependence query can return one of three different answers. @@ -360,13 +361,14 @@ class MemoryDependenceResults { (private) AssumptionCache &AC; const TargetLibraryInfo &TLI; DominatorTree &DT; + PhiValues &PV; PredIteratorCache PredCache; public: MemoryDependenceResults(AliasAnalysis &AA, AssumptionCache &AC, const TargetLibraryInfo &TLI, - DominatorTree &DT) - : AA(AA), AC(AC), TLI(TLI), DT(DT) {} + DominatorTree &DT, PhiValues &PV) + : AA(AA), AC(AC), TLI(TLI), DT(DT), PV(PV) {} /// Handle invalidation in the new PM. bool invalidate(Function &F, const PreservedAnalyses &PA, Modified: vendor/llvm/dist/include/llvm/Analysis/MustExecute.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/MustExecute.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/Analysis/MustExecute.h Thu Aug 2 17:32:43 2018 (r337137) @@ -10,7 +10,7 @@ /// Contains a collection of routines for determining if a given instruction is /// guaranteed to execute if a given point in control flow is reached. The most /// common example is an instruction within a loop being provably executed if we -/// branch to the header of it's containing loop. +/// branch to the header of it's containing loop. /// //===----------------------------------------------------------------------===// @@ -58,7 +58,7 @@ void computeLoopSafetyInfo(LoopSafetyInfo *, Loop *); bool isGuaranteedToExecute(const Instruction &Inst, const DominatorTree *DT, const Loop *CurLoop, const LoopSafetyInfo *SafetyInfo); - + } #endif Modified: vendor/llvm/dist/include/llvm/Analysis/TargetTransformInfoImpl.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/TargetTransformInfoImpl.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/Analysis/TargetTransformInfoImpl.h Thu Aug 2 17:32:43 2018 (r337137) @@ -326,7 +326,7 @@ class TargetTransformInfoImplBase { (public) bool haveFastSqrt(Type *Ty) { return false; } bool isFCmpOrdCheaperThanFCmpZero(Type *Ty) { return true; } - + unsigned getFPOpCost(Type *Ty) { return TargetTransformInfo::TCC_Basic; } int getIntImmCodeSizeCost(unsigned Opcode, unsigned Idx, const APInt &Imm, Modified: vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h Thu Aug 2 17:32:43 2018 (r337137) @@ -464,7 +464,7 @@ class Value; /// This is equivelent to saying that all instructions within the basic block /// are guaranteed to transfer execution to their successor within the basic /// block. This has the same assumptions w.r.t. undefined behavior as the - /// instruction variant of this function. + /// instruction variant of this function. bool isGuaranteedToTransferExecutionToSuccessor(const BasicBlock *BB); /// Return true if this function can prove that the instruction I Modified: vendor/llvm/dist/include/llvm/BinaryFormat/Dwarf.def ============================================================================== --- vendor/llvm/dist/include/llvm/BinaryFormat/Dwarf.def Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/BinaryFormat/Dwarf.def Thu Aug 2 17:32:43 2018 (r337137) @@ -856,6 +856,7 @@ HANDLE_DW_UT(0x06, split_type) // TODO: Add Mach-O and COFF names. // Official DWARF sections. HANDLE_DWARF_SECTION(DebugAbbrev, ".debug_abbrev", "debug-abbrev") +HANDLE_DWARF_SECTION(DebugAddr, ".debug_addr", "debug-addr") HANDLE_DWARF_SECTION(DebugAranges, ".debug_aranges", "debug-aranges") HANDLE_DWARF_SECTION(DebugInfo, ".debug_info", "debug-info") HANDLE_DWARF_SECTION(DebugTypes, ".debug_types", "debug-types") Modified: vendor/llvm/dist/include/llvm/BinaryFormat/ELF.h ============================================================================== --- vendor/llvm/dist/include/llvm/BinaryFormat/ELF.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/BinaryFormat/ELF.h Thu Aug 2 17:32:43 2018 (r337137) @@ -413,8 +413,10 @@ enum { // ARM Specific e_flags enum : unsigned { - EF_ARM_SOFT_FLOAT = 0x00000200U, - EF_ARM_VFP_FLOAT = 0x00000400U, + EF_ARM_SOFT_FLOAT = 0x00000200U, // Legacy pre EABI_VER5 + EF_ARM_ABI_FLOAT_SOFT = 0x00000200U, // EABI_VER5 + EF_ARM_VFP_FLOAT = 0x00000400U, // Legacy pre EABI_VER5 + EF_ARM_ABI_FLOAT_HARD = 0x00000400U, // EABI_VER5 EF_ARM_EABI_UNKNOWN = 0x00000000U, EF_ARM_EABI_VER1 = 0x01000000U, EF_ARM_EABI_VER2 = 0x02000000U, Modified: vendor/llvm/dist/include/llvm/CodeGen/GCStrategy.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/GCStrategy.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/GCStrategy.h Thu Aug 2 17:32:43 2018 (r337137) @@ -104,12 +104,12 @@ class GCStrategy { (public) const std::string &getName() const { return Name; } /// By default, write barriers are replaced with simple store - /// instructions. If true, you must provide a custom pass to lower + /// instructions. If true, you must provide a custom pass to lower /// calls to \@llvm.gcwrite. bool customWriteBarrier() const { return CustomWriteBarriers; } /// By default, read barriers are replaced with simple load - /// instructions. If true, you must provide a custom pass to lower + /// instructions. If true, you must provide a custom pass to lower /// calls to \@llvm.gcread. bool customReadBarrier() const { return CustomReadBarriers; } @@ -146,7 +146,7 @@ class GCStrategy { (public) } /// By default, roots are left for the code generator so it can generate a - /// stack map. If true, you must provide a custom pass to lower + /// stack map. If true, you must provide a custom pass to lower /// calls to \@llvm.gcroot. bool customRoots() const { return CustomRoots; } Modified: vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/LegalizerInfo.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/LegalizerInfo.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/LegalizerInfo.h Thu Aug 2 17:32:43 2018 (r337137) @@ -786,7 +786,7 @@ class LegalizerInfo { (public) /// setAction ({G_ADD, 0, LLT::scalar(32)}, Legal); /// setLegalizeScalarToDifferentSizeStrategy( /// G_ADD, 0, widenToLargerTypesAndNarrowToLargest); - /// will end up defining getAction({G_ADD, 0, T}) to return the following + /// will end up defining getAction({G_ADD, 0, T}) to return the following /// actions for different scalar types T: /// LLT::scalar(1)..LLT::scalar(31): {WidenScalar, 0, LLT::scalar(32)} /// LLT::scalar(32): {Legal, 0, LLT::scalar(32)} @@ -814,7 +814,7 @@ class LegalizerInfo { (public) VectorElementSizeChangeStrategies[OpcodeIdx][TypeIdx] = S; } - /// A SizeChangeStrategy for the common case where legalization for a + /// A SizeChangeStrategy for the common case where legalization for a /// particular operation consists of only supporting a specific set of type /// sizes. E.g. /// setAction ({G_DIV, 0, LLT::scalar(32)}, Legal); Modified: vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h Thu Aug 2 17:32:43 2018 (r337137) @@ -942,6 +942,16 @@ class MachineIRBuilderBase { (public) /// \return a MachineInstrBuilder for the newly created instruction. MachineInstrBuilder buildAtomicRMWUmin(unsigned OldValRes, unsigned Addr, unsigned Val, MachineMemOperand &MMO); + + /// Build and insert \p Res = G_BLOCK_ADDR \p BA + /// + /// G_BLOCK_ADDR computes the address of a basic block. + /// + /// \pre setBasicBlock or setMI must have been called. + /// \pre \p Res must be a generic virtual register of a pointer type. + /// + /// \return The newly created instruction. + MachineInstrBuilder buildBlockAddress(unsigned Res, const BlockAddress *BA); }; /// A CRTP class that contains methods for building instructions that can Modified: vendor/llvm/dist/include/llvm/CodeGen/MachORelocation.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/MachORelocation.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/MachORelocation.h Thu Aug 2 17:32:43 2018 (r337137) @@ -27,15 +27,15 @@ namespace llvm { uint32_t r_symbolnum; // symbol index if r_extern == 1 else section index bool r_pcrel; // was relocated pc-relative already uint8_t r_length; // length = 2 ^ r_length - bool r_extern; // + bool r_extern; // uint8_t r_type; // if not 0, machine-specific relocation type. bool r_scattered; // 1 = scattered, 0 = non-scattered int32_t r_value; // the value the item to be relocated is referring // to. - public: + public: uint32_t getPackedFields() const { if (r_scattered) - return (1 << 31) | (r_pcrel << 30) | ((r_length & 3) << 28) | + return (1 << 31) | (r_pcrel << 30) | ((r_length & 3) << 28) | ((r_type & 15) << 24) | (r_address & 0x00FFFFFF); else return (r_symbolnum << 8) | (r_pcrel << 7) | ((r_length & 3) << 5) | @@ -45,8 +45,8 @@ namespace llvm { uint32_t getRawAddress() const { return r_address; } MachORelocation(uint32_t addr, uint32_t index, bool pcrel, uint8_t len, - bool ext, uint8_t type, bool scattered = false, - int32_t value = 0) : + bool ext, uint8_t type, bool scattered = false, + int32_t value = 0) : r_address(addr), r_symbolnum(index), r_pcrel(pcrel), r_length(len), r_extern(ext), r_type(type), r_scattered(scattered), r_value(value) {} }; Modified: vendor/llvm/dist/include/llvm/CodeGen/MachineModuleInfo.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/MachineModuleInfo.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/MachineModuleInfo.h Thu Aug 2 17:32:43 2018 (r337137) @@ -105,7 +105,7 @@ class MachineModuleInfo : public ImmutablePass { /// basic block's address of label. MMIAddrLabelMap *AddrLabelSymbols; - // TODO: Ideally, what we'd like is to have a switch that allows emitting + // TODO: Ideally, what we'd like is to have a switch that allows emitting // synchronous (precise at call-sites only) CFA into .eh_frame. However, // even under this switch, we'd like .debug_frame to be precise when using // -g. At this moment, there's no way to specify that some CFI directives Modified: vendor/llvm/dist/include/llvm/CodeGen/MachineOutliner.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/MachineOutliner.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/MachineOutliner.h Thu Aug 2 17:32:43 2018 (r337137) @@ -19,6 +19,7 @@ #include "llvm/CodeGen/LiveRegUnits.h" #include "llvm/CodeGen/MachineFunction.h" #include "llvm/CodeGen/TargetRegisterInfo.h" +#include "llvm/CodeGen/LivePhysRegs.h" namespace llvm { namespace outliner { @@ -74,6 +75,13 @@ struct Candidate { (public) /// cost model information. LiveRegUnits LRU; + /// Contains the accumulated register liveness information for the + /// instructions in this \p Candidate. + /// + /// This is optionally used by the target to determine which registers have + /// been used across the sequence. + LiveRegUnits UsedInSequence; + /// Return the number of instructions in this Candidate. unsigned getLength() const { return Len; } @@ -137,6 +145,12 @@ struct Candidate { (public) // outlining candidate. std::for_each(MBB->rbegin(), (MachineBasicBlock::reverse_iterator)front(), [this](MachineInstr &MI) { LRU.stepBackward(MI); }); + + // Walk over the sequence itself and figure out which registers were used + // in the sequence. + UsedInSequence.init(TRI); + std::for_each(front(), std::next(back()), + [this](MachineInstr &MI) { UsedInSequence.accumulate(MI); }); } }; Modified: vendor/llvm/dist/include/llvm/CodeGen/ScheduleDAG.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/ScheduleDAG.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/ScheduleDAG.h Thu Aug 2 17:32:43 2018 (r337137) @@ -252,7 +252,7 @@ class TargetRegisterInfo; MachineInstr *Instr = nullptr; ///< Alternatively, a MachineInstr. public: - SUnit *OrigNode = nullptr; ///< If not this, the node from which this node + SUnit *OrigNode = nullptr; ///< If not this, the node from which this node /// was cloned. (SD scheduling only) const MCSchedClassDesc *SchedClass = Modified: vendor/llvm/dist/include/llvm/CodeGen/StackMaps.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/StackMaps.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/StackMaps.h Thu Aug 2 17:32:43 2018 (r337137) @@ -156,7 +156,7 @@ class StatepointOpers { // TODO:: we should change the STATEPOINT representation so that CC and // Flags should be part of meta operands, with args and deopt operands, and // gc operands all prefixed by their length and a type code. This would be - // much more consistent. + // much more consistent. public: // These values are aboolute offsets into the operands of the statepoint // instruction. Modified: vendor/llvm/dist/include/llvm/CodeGen/TargetLowering.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/TargetLowering.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/TargetLowering.h Thu Aug 2 17:32:43 2018 (r337137) @@ -718,7 +718,7 @@ class TargetLoweringBase { (public) /// always broken down into scalars in some contexts. This occurs even if the /// vector type is legal. virtual unsigned getVectorTypeBreakdownForCallingConv( - LLVMContext &Context, EVT VT, EVT &IntermediateVT, + LLVMContext &Context, CallingConv::ID CC, EVT VT, EVT &IntermediateVT, unsigned &NumIntermediates, MVT &RegisterVT) const { return getVectorTypeBreakdown(Context, VT, IntermediateVT, NumIntermediates, RegisterVT); @@ -1174,7 +1174,7 @@ class TargetLoweringBase { (public) /// are legal for some operations and not for other operations. /// For MIPS all vector types must be passed through the integer register set. virtual MVT getRegisterTypeForCallingConv(LLVMContext &Context, - EVT VT) const { + CallingConv::ID CC, EVT VT) const { return getRegisterType(Context, VT); } @@ -1182,6 +1182,7 @@ class TargetLoweringBase { (public) /// this occurs when a vector type is used, as vector are passed through the /// integer register set. virtual unsigned getNumRegistersForCallingConv(LLVMContext &Context, + CallingConv::ID CC, EVT VT) const { return getNumRegisters(Context, VT); } @@ -3489,10 +3490,10 @@ class TargetLowering : public TargetLoweringBase { (pu // SDValue BuildSDIV(SDNode *N, const APInt &Divisor, SelectionDAG &DAG, bool IsAfterLegalization, - std::vector *Created) const; + SmallVectorImpl &Created) const; SDValue BuildUDIV(SDNode *N, const APInt &Divisor, SelectionDAG &DAG, bool IsAfterLegalization, - std::vector *Created) const; + SmallVectorImpl &Created) const; /// Targets may override this function to provide custom SDIV lowering for /// power-of-2 denominators. If the target returns an empty SDValue, LLVM @@ -3500,7 +3501,7 @@ class TargetLowering : public TargetLoweringBase { (pu /// operations. virtual SDValue BuildSDIVPow2(SDNode *N, const APInt &Divisor, SelectionDAG &DAG, - std::vector *Created) const; + SmallVectorImpl &Created) const; /// Indicate whether this target prefers to combine FDIVs with the same /// divisor. If the transform should never be done, return zero. If the @@ -3690,7 +3691,7 @@ class TargetLowering : public TargetLoweringBase { (pu /// Given an LLVM IR type and return type attributes, compute the return value /// EVTs and flags, and optionally also the offsets, if the return value is /// being lowered to memory. -void GetReturnInfo(Type *ReturnType, AttributeList attr, +void GetReturnInfo(CallingConv::ID CC, Type *ReturnType, AttributeList attr, SmallVectorImpl &Outs, const TargetLowering &TLI, const DataLayout &DL); Modified: vendor/llvm/dist/include/llvm/CodeGen/TargetPassConfig.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/TargetPassConfig.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/TargetPassConfig.h Thu Aug 2 17:32:43 2018 (r337137) @@ -16,7 +16,7 @@ #include "llvm/Pass.h" #include "llvm/Support/CodeGen.h" -#include +#include #include namespace llvm { Modified: vendor/llvm/dist/include/llvm/CodeGen/TargetRegisterInfo.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/TargetRegisterInfo.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/CodeGen/TargetRegisterInfo.h Thu Aug 2 17:32:43 2018 (r337137) @@ -456,7 +456,7 @@ class TargetRegisterInfo : public MCRegisterInfo { (pu /// stack frame offset. The first register is closest to the incoming stack /// pointer if stack grows down, and vice versa. /// Notice: This function does not take into account disabled CSRs. - /// In most cases you will want to use instead the function + /// In most cases you will want to use instead the function /// getCalleeSavedRegs that is implemented in MachineRegisterInfo. virtual const MCPhysReg* getCalleeSavedRegs(const MachineFunction *MF) const = 0; @@ -518,7 +518,7 @@ class TargetRegisterInfo : public MCRegisterInfo { (pu /// guaranteed to be restored before any uses. This is useful for targets that /// have call sequences where a GOT register may be updated by the caller /// prior to a call and is guaranteed to be restored (also by the caller) - /// after the call. + /// after the call. virtual bool isCallerPreservedPhysReg(unsigned PhysReg, const MachineFunction &MF) const { return false; Modified: vendor/llvm/dist/include/llvm/DebugInfo/CodeView/CodeViewSymbols.def ============================================================================== --- vendor/llvm/dist/include/llvm/DebugInfo/CodeView/CodeViewSymbols.def Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/DebugInfo/CodeView/CodeViewSymbols.def Thu Aug 2 17:32:43 2018 (r337137) @@ -143,7 +143,6 @@ CV_SYMBOL(S_MANSLOT , 0x1120) CV_SYMBOL(S_MANMANYREG , 0x1121) CV_SYMBOL(S_MANREGREL , 0x1122) CV_SYMBOL(S_MANMANYREG2 , 0x1123) -CV_SYMBOL(S_UNAMESPACE , 0x1124) CV_SYMBOL(S_DATAREF , 0x1126) CV_SYMBOL(S_ANNOTATIONREF , 0x1128) CV_SYMBOL(S_TOKENREF , 0x1129) @@ -255,6 +254,7 @@ SYMBOL_RECORD_ALIAS(S_GMANDATA , 0x111d, ManagedG SYMBOL_RECORD(S_LTHREAD32 , 0x1112, ThreadLocalDataSym) SYMBOL_RECORD_ALIAS(S_GTHREAD32 , 0x1113, GlobalTLS, ThreadLocalDataSym) +SYMBOL_RECORD(S_UNAMESPACE , 0x1124, UsingNamespaceSym) #undef CV_SYMBOL #undef SYMBOL_RECORD Modified: vendor/llvm/dist/include/llvm/DebugInfo/CodeView/SymbolRecord.h ============================================================================== --- vendor/llvm/dist/include/llvm/DebugInfo/CodeView/SymbolRecord.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/DebugInfo/CodeView/SymbolRecord.h Thu Aug 2 17:32:43 2018 (r337137) @@ -942,6 +942,19 @@ class ThreadLocalDataSym : public SymbolRecord { (publ uint32_t RecordOffset; }; +// S_UNAMESPACE +class UsingNamespaceSym : public SymbolRecord { +public: + explicit UsingNamespaceSym(SymbolRecordKind Kind) : SymbolRecord(Kind) {} + explicit UsingNamespaceSym(uint32_t RecordOffset) + : SymbolRecord(SymbolRecordKind::RegRelativeSym), + RecordOffset(RecordOffset) {} + + StringRef Name; + + uint32_t RecordOffset; +}; + // S_ANNOTATION using CVSymbol = CVRecord; Modified: vendor/llvm/dist/include/llvm/DebugInfo/DIContext.h ============================================================================== --- vendor/llvm/dist/include/llvm/DebugInfo/DIContext.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/DebugInfo/DIContext.h Thu Aug 2 17:32:43 2018 (r337137) @@ -154,6 +154,8 @@ enum DIDumpType : unsigned { struct DIDumpOptions { unsigned DumpType = DIDT_All; unsigned RecurseDepth = -1U; + uint16_t Version = 0; // DWARF version to assume when extracting. + uint8_t AddrSize = 4; // Address byte size to assume when extracting. bool ShowAddresses = true; bool ShowChildren = false; bool ShowParents = false; Modified: vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFContext.h ============================================================================== --- vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFContext.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFContext.h Thu Aug 2 17:32:43 2018 (r337137) @@ -323,6 +323,10 @@ class DWARFContext : public DIContext { (public) /// have initialized the relevant target descriptions. Error loadRegisterInfo(const object::ObjectFile &Obj); + /// Get address size from CUs. + /// TODO: refactor compile_units() to make this const. + uint8_t getCUAddrSize(); + private: /// Return the compile unit which contains instruction with provided /// address. Modified: vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDataExtractor.h ============================================================================== --- vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDataExtractor.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDataExtractor.h Thu Aug 2 17:32:43 2018 (r337137) @@ -51,6 +51,8 @@ class DWARFDataExtractor : public DataExtractor { (pub /// reflect the absolute address of this pointer. Optional getEncodedPointer(uint32_t *Offset, uint8_t Encoding, uint64_t AbsPosOffset = 0) const; + + size_t size() const { return Section == nullptr ? 0 : Section->Data.size(); } }; } // end namespace llvm Added: vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDebugAddr.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDebugAddr.h Thu Aug 2 17:32:43 2018 (r337137) @@ -0,0 +1,98 @@ +//===- DWARFDebugAddr.h -------------------------------------*- C++ -*-===// +// +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +// +//===------------------------------------------------------------------===// + +#ifndef LLVM_DEBUGINFO_DWARFDEBUGADDR_H +#define LLVM_DEBUGINFO_DWARFDEBUGADDR_H + +#include "llvm/BinaryFormat/Dwarf.h" +#include "llvm/DebugInfo/DIContext.h" +#include "llvm/DebugInfo/DWARF/DWARFDataExtractor.h" +#include "llvm/Support/Errc.h" +#include "llvm/Support/Error.h" +#include +#include +#include + +namespace llvm { + +class Error; +class raw_ostream; + +/// A class representing an address table as specified in DWARF v5. +/// The table consists of a header followed by an array of address values from +/// .debug_addr section. +class DWARFDebugAddrTable { +public: + struct Header { + /// The total length of the entries for this table, not including the length + /// field itself. + uint32_t Length = 0; + /// The DWARF version number. + uint16_t Version = 5; + /// The size in bytes of an address on the target architecture. For + /// segmented addressing, this is the size of the offset portion of the + /// address. + uint8_t AddrSize; + /// The size in bytes of a segment selector on the target architecture. + /// If the target system uses a flat address space, this value is 0. + uint8_t SegSize = 0; + }; + +private: + dwarf::DwarfFormat Format; + uint32_t HeaderOffset; + Header HeaderData; + uint32_t DataSize = 0; + std::vector Addrs; + +public: + void clear(); + + /// Extract an entire table, including all addresses. + Error extract(DWARFDataExtractor Data, uint32_t *OffsetPtr, + uint16_t Version, uint8_t AddrSize, + std::function WarnCallback); + + uint32_t getHeaderOffset() const { return HeaderOffset; } + uint8_t getAddrSize() const { return HeaderData.AddrSize; } + void dump(raw_ostream &OS, DIDumpOptions DumpOpts = {}) const; + + /// Return the address based on a given index. + Expected getAddrEntry(uint32_t Index) const; + + /// Return the size of the table header including the length + /// but not including the addresses. + uint8_t getHeaderSize() const { + switch (Format) { + case dwarf::DwarfFormat::DWARF32: + return 8; // 4 + 2 + 1 + 1 + case dwarf::DwarfFormat::DWARF64: + return 16; // 12 + 2 + 1 + 1 + } + llvm_unreachable("Invalid DWARF format (expected DWARF32 or DWARF64)"); + } + + /// Returns the length of this table, including the length field, or 0 if the + /// length has not been determined (e.g. because the table has not yet been + /// parsed, or there was a problem in parsing). + uint32_t getLength() const; + + /// Verify that the given length is valid for this table. + bool hasValidLength() const { return getLength() != 0; } + + /// Invalidate Length field to stop further processing. + void invalidateLength() { HeaderData.Length = 0; } + + /// Returns the length of the array of addresses. + uint32_t getDataSize() const; +}; + +} // end namespace llvm + +#endif // LLVM_DEBUGINFO_DWARFDEBUGADDR_H Modified: vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDie.h ============================================================================== --- vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDie.h Thu Aug 2 17:06:03 2018 (r337136) +++ vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFDie.h Thu Aug 2 17:32:43 2018 (r337137) @@ -46,7 +46,7 @@ class DWARFDie { public: DWARFDie() = default; - DWARFDie(DWARFUnit *Unit, const DWARFDebugInfoEntry * D) : U(Unit), Die(D) {} + DWARFDie(DWARFUnit *Unit, const DWARFDebugInfoEntry *D) : U(Unit), Die(D) {} bool isValid() const { return U && Die; } explicit operator bool() const { return isValid(); } @@ -82,9 +82,7 @@ class DWARFDie { *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:17 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 550FB106D6E7; Thu, 2 Aug 2018 17:33:17 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F16C97233D; Thu, 2 Aug 2018 17:33:16 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A155C14D30; Thu, 2 Aug 2018 17:33:16 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXGdw029371; Thu, 2 Aug 2018 17:33:16 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXGwx029370; Thu, 2 Aug 2018 17:33:16 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXGwx029370@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337140 - vendor/clang/clang-trunk-r338536 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/clang/clang-trunk-r338536 X-SVN-Commit-Revision: 337140 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:17 -0000 Author: dim Date: Thu Aug 2 17:33:16 2018 New Revision: 337140 URL: https://svnweb.freebsd.org/changeset/base/337140 Log: Tag clang trunk r338536. Added: vendor/clang/clang-trunk-r338536/ - copied from r337139, vendor/clang/dist/ From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:14 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E781106D6E0; Thu, 2 Aug 2018 17:33:13 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 387577230C; Thu, 2 Aug 2018 17:33:13 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1961F14D2F; Thu, 2 Aug 2018 17:33:13 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXDnN029324; Thu, 2 Aug 2018 17:33:13 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXCXH029319; Thu, 2 Aug 2018 17:33:12 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXCXH029319@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:12 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337139 - in vendor/clang/dist: docs include/clang-c include/clang/ARCMigrate include/clang/AST include/clang/ASTMatchers include/clang/Analysis include/clang/Analysis/Analyses include/... X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/clang/dist: docs include/clang-c include/clang/ARCMigrate include/clang/AST include/clang/ASTMatchers include/clang/Analysis include/clang/Analysis/Analyses include/clang/Analysis/DomainSpec... X-SVN-Commit-Revision: 337139 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:14 -0000 Author: dim Date: Thu Aug 2 17:33:11 2018 New Revision: 337139 URL: https://svnweb.freebsd.org/changeset/base/337139 Log: Vendor import of clang trunk r338536: https://llvm.org/svn/llvm-project/cfe/trunk@338536 Added: vendor/clang/dist/lib/Driver/ToolChains/RISCV.cpp (contents, props changed) vendor/clang/dist/lib/Driver/ToolChains/RISCV.h (contents, props changed) vendor/clang/dist/test/Analysis/cfg-rich-constructors.mm vendor/clang/dist/test/Analysis/copy-elision.cpp (contents, props changed) vendor/clang/dist/test/Analysis/cstring-ranges.c (contents, props changed) vendor/clang/dist/test/Analysis/lifetime-extension.mm vendor/clang/dist/test/Analysis/temporaries.mm vendor/clang/dist/test/CodeGen/aapcs-align.cc (contents, props changed) vendor/clang/dist/test/CodeGen/aapcs64-align.cc (contents, props changed) vendor/clang/dist/test/CodeGen/catch-implicit-integer-truncations.c (contents, props changed) vendor/clang/dist/test/CodeGenCXX/Inputs/override-layout-nameless-struct-union.layout vendor/clang/dist/test/CodeGenCXX/Inputs/override-layout-packed-base.layout vendor/clang/dist/test/CodeGenCXX/castexpr-basepathsize-threshold.cpp (contents, props changed) vendor/clang/dist/test/CodeGenCXX/catch-implicit-integer-truncations.cpp (contents, props changed) vendor/clang/dist/test/CodeGenCXX/override-layout-nameless-struct-union.cpp (contents, props changed) vendor/clang/dist/test/CodeGenCXX/override-layout-packed-base.cpp (contents, props changed) vendor/clang/dist/test/CodeGenCXX/ubsan-new-checks.cpp (contents, props changed) vendor/clang/dist/test/CodeGenOpenCL/builtins-amdgcn-dl-insts-err-clamp.cl vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/bin/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/bin/riscv32-unknown-elf-ld (contents, props changed) vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/lib/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/lib/gcc/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/lib/gcc/riscv32-unknown-elf/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/lib/gcc/riscv32-unknown-elf/8.0.1/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/lib/gcc/riscv32-unknown-elf/8.0.1/crtbegin.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/lib/gcc/riscv32-unknown-elf/8.0.1/crtend.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/riscv32-unknown-elf/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/riscv32-unknown-elf/include/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/riscv32-unknown-elf/include/c++/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/riscv32-unknown-elf/include/c++/8.0.1/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/riscv32-unknown-elf/include/c++/8.0.1/.keep vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/riscv32-unknown-elf/lib/ vendor/clang/dist/test/Driver/Inputs/basic_riscv32_tree/riscv32-unknown-elf/lib/crt0.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/include/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/include/c++/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/include/c++/6.3.0/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/include/c++/6.3.0/backward/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/include/c++/6.3.0/backward/.keep vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/aarch64-oe-linux/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/aarch64-oe-linux/6.3.0/ vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/aarch64-oe-linux/6.3.0/crtbegin.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/aarch64-oe-linux/6.3.0/crtend.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/crt1.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/crti.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_aarch64_linux_tree/usr/lib64/crtn.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/include/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/include/c++/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/include/c++/6.3.0/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/include/c++/6.3.0/backward/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/include/c++/6.3.0/backward/.keep vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/arm-oe-linux-gnueabi/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/arm-oe-linux-gnueabi/6.3.0/ vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/arm-oe-linux-gnueabi/6.3.0/crtbegin.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/arm-oe-linux-gnueabi/6.3.0/crtend.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/crt1.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/crti.o (contents, props changed) vendor/clang/dist/test/Driver/Inputs/openembedded_arm_linux_tree/usr/lib/crtn.o (contents, props changed) vendor/clang/dist/test/Driver/cuda-unsupported-debug-options.cu vendor/clang/dist/test/Driver/openmp-unsupported-debug-options.c (contents, props changed) vendor/clang/dist/test/Index/complete-access-checks-crash.cpp (contents, props changed) vendor/clang/dist/test/SemaCXX/attr-lifetimebound.cpp (contents, props changed) vendor/clang/dist/test/SemaObjCXX/class-templ-error-null-init.mm Deleted: vendor/clang/dist/test/Analysis/cxx17-mandatory-elision.cpp vendor/clang/dist/test/Driver/Inputs/resource_dir/lib/darwin/ Modified: vendor/clang/dist/docs/ClangFormatStyleOptions.rst vendor/clang/dist/docs/ReleaseNotes.rst vendor/clang/dist/docs/UndefinedBehaviorSanitizer.rst vendor/clang/dist/docs/UsersManual.rst vendor/clang/dist/include/clang-c/Index.h vendor/clang/dist/include/clang/ARCMigrate/FileRemapper.h vendor/clang/dist/include/clang/AST/ASTContext.h vendor/clang/dist/include/clang/AST/ASTDiagnostic.h vendor/clang/dist/include/clang/AST/ASTImporter.h vendor/clang/dist/include/clang/AST/ASTLambda.h vendor/clang/dist/include/clang/AST/ASTMutationListener.h vendor/clang/dist/include/clang/AST/Attr.h vendor/clang/dist/include/clang/AST/AttrIterator.h vendor/clang/dist/include/clang/AST/BaseSubobject.h vendor/clang/dist/include/clang/AST/CXXInheritance.h vendor/clang/dist/include/clang/AST/CanonicalType.h vendor/clang/dist/include/clang/AST/CharUnits.h vendor/clang/dist/include/clang/AST/Comment.h vendor/clang/dist/include/clang/AST/CommentCommandTraits.h vendor/clang/dist/include/clang/AST/CommentLexer.h vendor/clang/dist/include/clang/AST/CommentSema.h vendor/clang/dist/include/clang/AST/Decl.h vendor/clang/dist/include/clang/AST/DeclBase.h vendor/clang/dist/include/clang/AST/DeclCXX.h vendor/clang/dist/include/clang/AST/DeclFriend.h vendor/clang/dist/include/clang/AST/DeclLookups.h vendor/clang/dist/include/clang/AST/DeclObjC.h vendor/clang/dist/include/clang/AST/DeclTemplate.h vendor/clang/dist/include/clang/AST/DeclarationName.h vendor/clang/dist/include/clang/AST/DependentDiagnostic.h vendor/clang/dist/include/clang/AST/EvaluatedExprVisitor.h vendor/clang/dist/include/clang/AST/Expr.h vendor/clang/dist/include/clang/AST/ExprCXX.h vendor/clang/dist/include/clang/AST/ExprObjC.h vendor/clang/dist/include/clang/AST/ExternalASTMerger.h vendor/clang/dist/include/clang/AST/ExternalASTSource.h vendor/clang/dist/include/clang/AST/GlobalDecl.h vendor/clang/dist/include/clang/AST/LambdaCapture.h vendor/clang/dist/include/clang/AST/Mangle.h vendor/clang/dist/include/clang/AST/NSAPI.h vendor/clang/dist/include/clang/AST/NestedNameSpecifier.h vendor/clang/dist/include/clang/AST/OperationKinds.def vendor/clang/dist/include/clang/AST/OperationKinds.h vendor/clang/dist/include/clang/AST/PrettyPrinter.h vendor/clang/dist/include/clang/AST/RecordLayout.h vendor/clang/dist/include/clang/AST/Stmt.h vendor/clang/dist/include/clang/AST/StmtIterator.h vendor/clang/dist/include/clang/AST/StmtObjC.h vendor/clang/dist/include/clang/AST/StmtOpenMP.h vendor/clang/dist/include/clang/AST/TemplateBase.h vendor/clang/dist/include/clang/AST/TemplateName.h vendor/clang/dist/include/clang/AST/Type.h vendor/clang/dist/include/clang/AST/TypeLoc.h vendor/clang/dist/include/clang/AST/TypeOrdering.h vendor/clang/dist/include/clang/AST/UnresolvedSet.h vendor/clang/dist/include/clang/AST/VTTBuilder.h vendor/clang/dist/include/clang/AST/VTableBuilder.h vendor/clang/dist/include/clang/ASTMatchers/ASTMatchers.h vendor/clang/dist/include/clang/Analysis/Analyses/CFGReachabilityAnalysis.h vendor/clang/dist/include/clang/Analysis/Analyses/Consumed.h vendor/clang/dist/include/clang/Analysis/Analyses/Dominators.h vendor/clang/dist/include/clang/Analysis/Analyses/FormatString.h vendor/clang/dist/include/clang/Analysis/Analyses/LiveVariables.h vendor/clang/dist/include/clang/Analysis/Analyses/ReachableCode.h vendor/clang/dist/include/clang/Analysis/Analyses/ThreadSafetyCommon.h vendor/clang/dist/include/clang/Analysis/AnalysisDeclContext.h vendor/clang/dist/include/clang/Analysis/CFG.h vendor/clang/dist/include/clang/Analysis/CFGStmtMap.h vendor/clang/dist/include/clang/Analysis/CloneDetection.h vendor/clang/dist/include/clang/Analysis/ConstructionContext.h vendor/clang/dist/include/clang/Analysis/DomainSpecific/CocoaConventions.h vendor/clang/dist/include/clang/Analysis/DomainSpecific/ObjCNoReturn.h vendor/clang/dist/include/clang/Analysis/ProgramPoint.h vendor/clang/dist/include/clang/Analysis/Support/BumpVector.h vendor/clang/dist/include/clang/Basic/ABI.h vendor/clang/dist/include/clang/Basic/AllDiagnostics.h vendor/clang/dist/include/clang/Basic/Attr.td vendor/clang/dist/include/clang/Basic/AttrDocs.td vendor/clang/dist/include/clang/Basic/Builtins.def vendor/clang/dist/include/clang/Basic/BuiltinsAMDGPU.def vendor/clang/dist/include/clang/Basic/BuiltinsPPC.def vendor/clang/dist/include/clang/Basic/Diagnostic.h vendor/clang/dist/include/clang/Basic/DiagnosticASTKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticCommentKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticCommonKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticDriverKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticFrontendKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticGroups.td vendor/clang/dist/include/clang/Basic/DiagnosticIDs.h vendor/clang/dist/include/clang/Basic/DiagnosticLexKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticOptions.def vendor/clang/dist/include/clang/Basic/DiagnosticOptions.h vendor/clang/dist/include/clang/Basic/DiagnosticParseKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticSemaKinds.td vendor/clang/dist/include/clang/Basic/DiagnosticSerializationKinds.td vendor/clang/dist/include/clang/Basic/FileSystemStatCache.h vendor/clang/dist/include/clang/Basic/IdentifierTable.h vendor/clang/dist/include/clang/Basic/LLVM.h vendor/clang/dist/include/clang/Basic/LangOptions.h vendor/clang/dist/include/clang/Basic/Linkage.h vendor/clang/dist/include/clang/Basic/Module.h vendor/clang/dist/include/clang/Basic/ObjCRuntime.h vendor/clang/dist/include/clang/Basic/OperatorKinds.h vendor/clang/dist/include/clang/Basic/Sanitizers.def vendor/clang/dist/include/clang/Basic/Sanitizers.h vendor/clang/dist/include/clang/Basic/SourceLocation.h vendor/clang/dist/include/clang/Basic/SourceManager.h vendor/clang/dist/include/clang/Basic/SourceManagerInternals.h vendor/clang/dist/include/clang/Basic/Specifiers.h vendor/clang/dist/include/clang/Basic/StmtNodes.td vendor/clang/dist/include/clang/Basic/TargetInfo.h vendor/clang/dist/include/clang/Basic/TemplateKinds.h vendor/clang/dist/include/clang/Basic/arm_neon.td vendor/clang/dist/include/clang/Basic/arm_neon_incl.td vendor/clang/dist/include/clang/Driver/CC1Options.td vendor/clang/dist/include/clang/Driver/Driver.h vendor/clang/dist/include/clang/Driver/ToolChain.h vendor/clang/dist/include/clang/Edit/Commit.h vendor/clang/dist/include/clang/Edit/EditedSource.h vendor/clang/dist/include/clang/Edit/Rewriters.h vendor/clang/dist/include/clang/Frontend/ASTUnit.h vendor/clang/dist/include/clang/Frontend/CompilerInstance.h vendor/clang/dist/include/clang/Frontend/CompilerInvocation.h vendor/clang/dist/include/clang/Frontend/DiagnosticRenderer.h vendor/clang/dist/include/clang/Frontend/FrontendActions.h vendor/clang/dist/include/clang/Frontend/LayoutOverrideSource.h vendor/clang/dist/include/clang/Frontend/LogDiagnosticPrinter.h vendor/clang/dist/include/clang/Frontend/PCHContainerOperations.h vendor/clang/dist/include/clang/Frontend/VerifyDiagnosticConsumer.h vendor/clang/dist/include/clang/Lex/CodeCompletionHandler.h vendor/clang/dist/include/clang/Lex/DirectoryLookup.h vendor/clang/dist/include/clang/Lex/ExternalPreprocessorSource.h vendor/clang/dist/include/clang/Lex/HeaderSearch.h vendor/clang/dist/include/clang/Lex/HeaderSearchOptions.h vendor/clang/dist/include/clang/Lex/LiteralSupport.h vendor/clang/dist/include/clang/Lex/MacroArgs.h vendor/clang/dist/include/clang/Lex/MacroInfo.h vendor/clang/dist/include/clang/Lex/ModuleLoader.h vendor/clang/dist/include/clang/Lex/ModuleMap.h vendor/clang/dist/include/clang/Lex/PPCallbacks.h vendor/clang/dist/include/clang/Lex/PPConditionalDirectiveRecord.h vendor/clang/dist/include/clang/Lex/PTHLexer.h vendor/clang/dist/include/clang/Lex/Pragma.h vendor/clang/dist/include/clang/Lex/PreprocessingRecord.h vendor/clang/dist/include/clang/Lex/Preprocessor.h vendor/clang/dist/include/clang/Lex/PreprocessorLexer.h vendor/clang/dist/include/clang/Lex/PreprocessorOptions.h vendor/clang/dist/include/clang/Lex/Token.h vendor/clang/dist/include/clang/Lex/TokenConcatenation.h vendor/clang/dist/include/clang/Lex/TokenLexer.h vendor/clang/dist/include/clang/Lex/VariadicMacroSupport.h vendor/clang/dist/include/clang/Parse/ParseAST.h vendor/clang/dist/include/clang/Parse/Parser.h vendor/clang/dist/include/clang/Parse/RAIIObjectsForParser.h vendor/clang/dist/include/clang/Sema/DeclSpec.h vendor/clang/dist/include/clang/Sema/DelayedDiagnostic.h vendor/clang/dist/include/clang/Sema/ExternalSemaSource.h vendor/clang/dist/include/clang/Sema/IdentifierResolver.h vendor/clang/dist/include/clang/Sema/Initialization.h vendor/clang/dist/include/clang/Sema/Lookup.h vendor/clang/dist/include/clang/Sema/MultiplexExternalSemaSource.h vendor/clang/dist/include/clang/Sema/Overload.h vendor/clang/dist/include/clang/Sema/ParsedAttr.h vendor/clang/dist/include/clang/Sema/ParsedTemplate.h vendor/clang/dist/include/clang/Sema/Scope.h vendor/clang/dist/include/clang/Sema/ScopeInfo.h vendor/clang/dist/include/clang/Sema/SemaInternal.h vendor/clang/dist/include/clang/Sema/SemaLambda.h vendor/clang/dist/include/clang/Sema/Template.h vendor/clang/dist/include/clang/Serialization/ASTBitCodes.h vendor/clang/dist/include/clang/Serialization/ASTReader.h vendor/clang/dist/include/clang/Serialization/ASTWriter.h vendor/clang/dist/include/clang/Serialization/ContinuousRangeMap.h vendor/clang/dist/include/clang/Serialization/GlobalModuleIndex.h vendor/clang/dist/include/clang/Serialization/Module.h vendor/clang/dist/include/clang/Serialization/ModuleFileExtension.h vendor/clang/dist/include/clang/Serialization/ModuleManager.h vendor/clang/dist/include/clang/StaticAnalyzer/Checkers/Checkers.td vendor/clang/dist/include/clang/StaticAnalyzer/Checkers/ObjCRetainCount.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/BugReporter/BugReporter.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/BugReporter/BugReporterVisitors.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/BugReporter/BugType.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/BugReporter/PathDiagnostic.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/Checker.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/CheckerManager.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/CheckerOptInfo.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/CheckerRegistry.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/IssueHash.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/APSIntType.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/AnalysisManager.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/BasicValueFactory.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/BlockCounter.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/CallEvent.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/CheckerContext.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/CoreEngine.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/Environment.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/ExplodedGraph.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/ExprEngine.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/MemRegion.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/ProgramState.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/SValBuilder.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/SVals.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/Store.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/StoreRef.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/SubEngine.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/SummaryManager.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/SymbolManager.h vendor/clang/dist/include/clang/StaticAnalyzer/Core/PathSensitive/WorkList.h vendor/clang/dist/lib/ARCMigrate/ARCMT.cpp vendor/clang/dist/lib/ARCMigrate/FileRemapper.cpp vendor/clang/dist/lib/ARCMigrate/Internals.h vendor/clang/dist/lib/ARCMigrate/ObjCMT.cpp vendor/clang/dist/lib/ARCMigrate/PlistReporter.cpp vendor/clang/dist/lib/ARCMigrate/TransARCAssign.cpp vendor/clang/dist/lib/ARCMigrate/TransAutoreleasePool.cpp vendor/clang/dist/lib/ARCMigrate/TransBlockObjCVariable.cpp vendor/clang/dist/lib/ARCMigrate/TransEmptyStatementsAndDealloc.cpp vendor/clang/dist/lib/ARCMigrate/TransGCAttrs.cpp vendor/clang/dist/lib/ARCMigrate/TransProperties.cpp vendor/clang/dist/lib/ARCMigrate/TransProtectedScope.cpp vendor/clang/dist/lib/ARCMigrate/TransRetainReleaseDealloc.cpp vendor/clang/dist/lib/ARCMigrate/TransUnbridgedCasts.cpp vendor/clang/dist/lib/ARCMigrate/TransZeroOutPropsInDealloc.cpp vendor/clang/dist/lib/ARCMigrate/TransformActions.cpp vendor/clang/dist/lib/ARCMigrate/Transforms.cpp vendor/clang/dist/lib/ARCMigrate/Transforms.h vendor/clang/dist/lib/AST/ASTContext.cpp vendor/clang/dist/lib/AST/ASTDiagnostic.cpp vendor/clang/dist/lib/AST/ASTDumper.cpp vendor/clang/dist/lib/AST/ASTImporter.cpp vendor/clang/dist/lib/AST/ASTStructuralEquivalence.cpp vendor/clang/dist/lib/AST/CXXInheritance.cpp vendor/clang/dist/lib/AST/Comment.cpp vendor/clang/dist/lib/AST/CommentLexer.cpp vendor/clang/dist/lib/AST/Decl.cpp vendor/clang/dist/lib/AST/DeclBase.cpp vendor/clang/dist/lib/AST/DeclCXX.cpp vendor/clang/dist/lib/AST/DeclObjC.cpp vendor/clang/dist/lib/AST/DeclPrinter.cpp vendor/clang/dist/lib/AST/DeclTemplate.cpp vendor/clang/dist/lib/AST/DeclarationName.cpp vendor/clang/dist/lib/AST/Expr.cpp vendor/clang/dist/lib/AST/ExprCXX.cpp vendor/clang/dist/lib/AST/ExprClassification.cpp vendor/clang/dist/lib/AST/ExprConstant.cpp vendor/clang/dist/lib/AST/ExternalASTSource.cpp vendor/clang/dist/lib/AST/ItaniumCXXABI.cpp vendor/clang/dist/lib/AST/ItaniumMangle.cpp vendor/clang/dist/lib/AST/Mangle.cpp vendor/clang/dist/lib/AST/MicrosoftMangle.cpp vendor/clang/dist/lib/AST/NSAPI.cpp vendor/clang/dist/lib/AST/NestedNameSpecifier.cpp vendor/clang/dist/lib/AST/ParentMap.cpp vendor/clang/dist/lib/AST/RecordLayout.cpp vendor/clang/dist/lib/AST/RecordLayoutBuilder.cpp vendor/clang/dist/lib/AST/StmtOpenMP.cpp vendor/clang/dist/lib/AST/StmtPrinter.cpp vendor/clang/dist/lib/AST/StmtProfile.cpp vendor/clang/dist/lib/AST/TemplateBase.cpp vendor/clang/dist/lib/AST/TemplateName.cpp vendor/clang/dist/lib/AST/Type.cpp vendor/clang/dist/lib/AST/TypeLoc.cpp vendor/clang/dist/lib/AST/TypePrinter.cpp vendor/clang/dist/lib/AST/VTTBuilder.cpp vendor/clang/dist/lib/AST/VTableBuilder.cpp vendor/clang/dist/lib/Analysis/AnalysisDeclContext.cpp vendor/clang/dist/lib/Analysis/BodyFarm.cpp vendor/clang/dist/lib/Analysis/CFG.cpp vendor/clang/dist/lib/Analysis/CFGReachabilityAnalysis.cpp vendor/clang/dist/lib/Analysis/CFGStmtMap.cpp vendor/clang/dist/lib/Analysis/CallGraph.cpp vendor/clang/dist/lib/Analysis/CocoaConventions.cpp vendor/clang/dist/lib/Analysis/ConstructionContext.cpp vendor/clang/dist/lib/Analysis/Consumed.cpp vendor/clang/dist/lib/Analysis/FormatString.cpp vendor/clang/dist/lib/Analysis/FormatStringParsing.h vendor/clang/dist/lib/Analysis/LiveVariables.cpp vendor/clang/dist/lib/Analysis/ObjCNoReturn.cpp vendor/clang/dist/lib/Analysis/PostOrderCFGView.cpp vendor/clang/dist/lib/Analysis/PrintfFormatString.cpp vendor/clang/dist/lib/Analysis/ProgramPoint.cpp vendor/clang/dist/lib/Analysis/ReachableCode.cpp vendor/clang/dist/lib/Analysis/ScanfFormatString.cpp vendor/clang/dist/lib/Analysis/UninitializedValues.cpp vendor/clang/dist/lib/Basic/Diagnostic.cpp vendor/clang/dist/lib/Basic/DiagnosticIDs.cpp vendor/clang/dist/lib/Basic/FileManager.cpp vendor/clang/dist/lib/Basic/FileSystemStatCache.cpp vendor/clang/dist/lib/Basic/IdentifierTable.cpp vendor/clang/dist/lib/Basic/Module.cpp vendor/clang/dist/lib/Basic/ObjCRuntime.cpp vendor/clang/dist/lib/Basic/SourceLocation.cpp vendor/clang/dist/lib/Basic/SourceManager.cpp vendor/clang/dist/lib/Basic/Targets/ARM.cpp vendor/clang/dist/lib/Basic/Targets/X86.cpp vendor/clang/dist/lib/Basic/Targets/X86.h vendor/clang/dist/lib/Basic/Warnings.cpp vendor/clang/dist/lib/CodeGen/CGBlocks.cpp vendor/clang/dist/lib/CodeGen/CGBlocks.h vendor/clang/dist/lib/CodeGen/CGBuilder.h vendor/clang/dist/lib/CodeGen/CGBuiltin.cpp vendor/clang/dist/lib/CodeGen/CGCXX.cpp vendor/clang/dist/lib/CodeGen/CGCXXABI.cpp vendor/clang/dist/lib/CodeGen/CGCall.cpp vendor/clang/dist/lib/CodeGen/CGCall.h vendor/clang/dist/lib/CodeGen/CGClass.cpp vendor/clang/dist/lib/CodeGen/CGCleanup.cpp vendor/clang/dist/lib/CodeGen/CGCleanup.h vendor/clang/dist/lib/CodeGen/CGDebugInfo.cpp vendor/clang/dist/lib/CodeGen/CGDeclCXX.cpp vendor/clang/dist/lib/CodeGen/CGException.cpp vendor/clang/dist/lib/CodeGen/CGExpr.cpp vendor/clang/dist/lib/CodeGen/CGExprAgg.cpp vendor/clang/dist/lib/CodeGen/CGExprCXX.cpp vendor/clang/dist/lib/CodeGen/CGExprConstant.cpp vendor/clang/dist/lib/CodeGen/CGExprScalar.cpp vendor/clang/dist/lib/CodeGen/CGLoopInfo.cpp vendor/clang/dist/lib/CodeGen/CGObjC.cpp vendor/clang/dist/lib/CodeGen/CGObjCGNU.cpp vendor/clang/dist/lib/CodeGen/CGObjCMac.cpp vendor/clang/dist/lib/CodeGen/CGObjCRuntime.cpp vendor/clang/dist/lib/CodeGen/CGObjCRuntime.h vendor/clang/dist/lib/CodeGen/CGOpenMPRuntime.cpp vendor/clang/dist/lib/CodeGen/CGRecordLayoutBuilder.cpp vendor/clang/dist/lib/CodeGen/CGVTT.cpp vendor/clang/dist/lib/CodeGen/CGVTables.cpp vendor/clang/dist/lib/CodeGen/CGVTables.h vendor/clang/dist/lib/CodeGen/CGValue.h vendor/clang/dist/lib/CodeGen/CodeGenAction.cpp vendor/clang/dist/lib/CodeGen/CodeGenFunction.h vendor/clang/dist/lib/CodeGen/CodeGenModule.cpp vendor/clang/dist/lib/CodeGen/CodeGenModule.h vendor/clang/dist/lib/CodeGen/CodeGenTypes.cpp vendor/clang/dist/lib/CodeGen/CodeGenTypes.h vendor/clang/dist/lib/CodeGen/ConstantInitBuilder.cpp vendor/clang/dist/lib/CodeGen/ItaniumCXXABI.cpp vendor/clang/dist/lib/CodeGen/MicrosoftCXXABI.cpp vendor/clang/dist/lib/CodeGen/SwiftCallingConv.cpp vendor/clang/dist/lib/CodeGen/TargetInfo.cpp vendor/clang/dist/lib/Driver/CMakeLists.txt vendor/clang/dist/lib/Driver/Driver.cpp vendor/clang/dist/lib/Driver/SanitizerArgs.cpp vendor/clang/dist/lib/Driver/ToolChain.cpp vendor/clang/dist/lib/Driver/ToolChains/Arch/RISCV.cpp vendor/clang/dist/lib/Driver/ToolChains/Clang.cpp vendor/clang/dist/lib/Driver/ToolChains/Cuda.cpp vendor/clang/dist/lib/Driver/ToolChains/Cuda.h vendor/clang/dist/lib/Driver/ToolChains/Darwin.cpp vendor/clang/dist/lib/Driver/ToolChains/Darwin.h vendor/clang/dist/lib/Driver/ToolChains/Gnu.cpp vendor/clang/dist/lib/Driver/ToolChains/Linux.cpp vendor/clang/dist/lib/Edit/Commit.cpp vendor/clang/dist/lib/Edit/RewriteObjCFoundationAPI.cpp vendor/clang/dist/lib/Format/BreakableToken.cpp vendor/clang/dist/lib/Format/BreakableToken.h vendor/clang/dist/lib/Format/ContinuationIndenter.cpp vendor/clang/dist/lib/Format/ContinuationIndenter.h vendor/clang/dist/lib/Format/Format.cpp vendor/clang/dist/lib/Frontend/ASTConsumers.cpp vendor/clang/dist/lib/Frontend/ASTMerge.cpp vendor/clang/dist/lib/Frontend/ChainedIncludesSource.cpp vendor/clang/dist/lib/Frontend/CompilerInstance.cpp vendor/clang/dist/lib/Frontend/DependencyFile.cpp vendor/clang/dist/lib/Frontend/DependencyGraph.cpp vendor/clang/dist/lib/Frontend/DiagnosticRenderer.cpp vendor/clang/dist/lib/Frontend/FrontendAction.cpp vendor/clang/dist/lib/Frontend/FrontendActions.cpp vendor/clang/dist/lib/Frontend/LayoutOverrideSource.cpp vendor/clang/dist/lib/Frontend/MultiplexConsumer.cpp vendor/clang/dist/lib/Frontend/PrintPreprocessedOutput.cpp vendor/clang/dist/lib/Frontend/Rewrite/FixItRewriter.cpp vendor/clang/dist/lib/Frontend/Rewrite/FrontendActions.cpp vendor/clang/dist/lib/Frontend/Rewrite/RewriteModernObjC.cpp vendor/clang/dist/lib/Frontend/Rewrite/RewriteObjC.cpp vendor/clang/dist/lib/Frontend/SerializedDiagnosticPrinter.cpp vendor/clang/dist/lib/Frontend/TestModuleFileExtension.cpp vendor/clang/dist/lib/Frontend/TextDiagnostic.cpp vendor/clang/dist/lib/FrontendTool/ExecuteCompilerInvocation.cpp vendor/clang/dist/lib/Headers/avx512fintrin.h vendor/clang/dist/lib/Headers/avx512vlcdintrin.h vendor/clang/dist/lib/Headers/clzerointrin.h vendor/clang/dist/lib/Headers/lwpintrin.h vendor/clang/dist/lib/Headers/opencl-c.h vendor/clang/dist/lib/Index/IndexBody.cpp vendor/clang/dist/lib/Index/IndexTypeSourceInfo.cpp vendor/clang/dist/lib/Index/IndexingContext.cpp vendor/clang/dist/lib/Index/USRGeneration.cpp vendor/clang/dist/lib/Lex/HeaderSearch.cpp vendor/clang/dist/lib/Lex/LiteralSupport.cpp vendor/clang/dist/lib/Lex/MacroArgs.cpp vendor/clang/dist/lib/Lex/ModuleMap.cpp vendor/clang/dist/lib/Lex/PPDirectives.cpp vendor/clang/dist/lib/Lex/PPExpressions.cpp vendor/clang/dist/lib/Lex/PPLexerChange.cpp vendor/clang/dist/lib/Lex/PPMacroExpansion.cpp vendor/clang/dist/lib/Lex/Pragma.cpp vendor/clang/dist/lib/Lex/PreprocessingRecord.cpp vendor/clang/dist/lib/Lex/Preprocessor.cpp vendor/clang/dist/lib/Lex/TokenLexer.cpp vendor/clang/dist/lib/Parse/ParseAST.cpp vendor/clang/dist/lib/Parse/ParseCXXInlineMethods.cpp vendor/clang/dist/lib/Parse/ParseDecl.cpp vendor/clang/dist/lib/Parse/ParseDeclCXX.cpp vendor/clang/dist/lib/Parse/ParseExpr.cpp vendor/clang/dist/lib/Parse/ParseExprCXX.cpp vendor/clang/dist/lib/Parse/ParseInit.cpp vendor/clang/dist/lib/Parse/ParseObjc.cpp vendor/clang/dist/lib/Parse/ParsePragma.cpp vendor/clang/dist/lib/Parse/ParseStmtAsm.cpp vendor/clang/dist/lib/Parse/ParseTemplate.cpp vendor/clang/dist/lib/Parse/ParseTentative.cpp vendor/clang/dist/lib/Parse/Parser.cpp vendor/clang/dist/lib/Rewrite/DeltaTree.cpp vendor/clang/dist/lib/Rewrite/HTMLRewrite.cpp vendor/clang/dist/lib/Rewrite/Rewriter.cpp vendor/clang/dist/lib/Sema/AnalysisBasedWarnings.cpp vendor/clang/dist/lib/Sema/CMakeLists.txt vendor/clang/dist/lib/Sema/CodeCompleteConsumer.cpp vendor/clang/dist/lib/Sema/DeclSpec.cpp vendor/clang/dist/lib/Sema/DelayedDiagnostic.cpp vendor/clang/dist/lib/Sema/IdentifierResolver.cpp vendor/clang/dist/lib/Sema/MultiplexExternalSemaSource.cpp vendor/clang/dist/lib/Sema/Sema.cpp vendor/clang/dist/lib/Sema/SemaAccess.cpp vendor/clang/dist/lib/Sema/SemaAttr.cpp vendor/clang/dist/lib/Sema/SemaCXXScopeSpec.cpp vendor/clang/dist/lib/Sema/SemaCast.cpp vendor/clang/dist/lib/Sema/SemaChecking.cpp vendor/clang/dist/lib/Sema/SemaCodeComplete.cpp vendor/clang/dist/lib/Sema/SemaDecl.cpp vendor/clang/dist/lib/Sema/SemaDeclAttr.cpp vendor/clang/dist/lib/Sema/SemaDeclObjC.cpp vendor/clang/dist/lib/Sema/SemaExceptionSpec.cpp vendor/clang/dist/lib/Sema/SemaExpr.cpp vendor/clang/dist/lib/Sema/SemaExprMember.cpp vendor/clang/dist/lib/Sema/SemaExprObjC.cpp vendor/clang/dist/lib/Sema/SemaInit.cpp vendor/clang/dist/lib/Sema/SemaLambda.cpp vendor/clang/dist/lib/Sema/SemaLookup.cpp vendor/clang/dist/lib/Sema/SemaObjCProperty.cpp vendor/clang/dist/lib/Sema/SemaPseudoObject.cpp vendor/clang/dist/lib/Sema/SemaStmt.cpp vendor/clang/dist/lib/Sema/SemaStmtAsm.cpp vendor/clang/dist/lib/Sema/SemaTemplate.cpp vendor/clang/dist/lib/Sema/SemaTemplateDeduction.cpp vendor/clang/dist/lib/Sema/SemaTemplateInstantiate.cpp vendor/clang/dist/lib/Sema/SemaTemplateInstantiateDecl.cpp vendor/clang/dist/lib/Sema/SemaTemplateVariadic.cpp vendor/clang/dist/lib/Sema/SemaType.cpp vendor/clang/dist/lib/Sema/TreeTransform.h vendor/clang/dist/lib/Sema/TypeLocBuilder.h vendor/clang/dist/lib/Serialization/ASTReader.cpp vendor/clang/dist/lib/Serialization/ASTReaderDecl.cpp vendor/clang/dist/lib/Serialization/ASTReaderInternals.h vendor/clang/dist/lib/Serialization/ASTWriterDecl.cpp vendor/clang/dist/lib/Serialization/ASTWriterStmt.cpp vendor/clang/dist/lib/Serialization/Module.cpp vendor/clang/dist/lib/Serialization/ModuleManager.cpp vendor/clang/dist/lib/StaticAnalyzer/Checkers/CStringChecker.cpp vendor/clang/dist/lib/StaticAnalyzer/Checkers/InnerPointerChecker.cpp vendor/clang/dist/lib/StaticAnalyzer/Checkers/IteratorChecker.cpp vendor/clang/dist/lib/StaticAnalyzer/Checkers/MallocChecker.cpp vendor/clang/dist/lib/StaticAnalyzer/Checkers/RetainCountChecker.cpp vendor/clang/dist/lib/StaticAnalyzer/Checkers/RunLoopAutoreleaseLeakChecker.cpp vendor/clang/dist/lib/StaticAnalyzer/Checkers/StackAddrEscapeChecker.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/BugReporterVisitors.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/CallEvent.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/ExprEngine.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/ExprEngineC.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/ExprEngineCXX.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/IssueHash.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/ProgramState.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/SimpleSValBuilder.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/Store.cpp vendor/clang/dist/test/Analysis/Checkers/RunLoopAutoreleaseLeakChecker.m vendor/clang/dist/test/Analysis/casts.c vendor/clang/dist/test/Analysis/cfg-rich-constructors.cpp vendor/clang/dist/test/Analysis/inner-pointer.cpp vendor/clang/dist/test/Analysis/temporaries.cpp vendor/clang/dist/test/CodeGen/arm-arguments.c vendor/clang/dist/test/CodeGen/builtin-memfns.c vendor/clang/dist/test/CodeGen/exceptions-seh-finally.c vendor/clang/dist/test/CodeGen/exceptions-seh.c vendor/clang/dist/test/CodeGen/mips-unsigned-ext-var.c vendor/clang/dist/test/CodeGen/mips-varargs.c vendor/clang/dist/test/CodeGen/mips-vector-arg.c vendor/clang/dist/test/CodeGenCUDA/device-var-init.cu vendor/clang/dist/test/CodeGenCXX/ctor-dtor-alias.cpp vendor/clang/dist/test/CodeGenCXX/float16-declarations.cpp vendor/clang/dist/test/CodeGenOpenCL/blocks.cl vendor/clang/dist/test/CodeGenOpenCL/builtins-amdgcn-dl-insts-err.cl vendor/clang/dist/test/CodeGenOpenCL/builtins-amdgcn-dl-insts.cl vendor/clang/dist/test/Driver/aarch64-cpus.c vendor/clang/dist/test/Driver/arm-cortex-cpus.c vendor/clang/dist/test/Driver/baremetal.cpp vendor/clang/dist/test/Driver/cuda-dwarf-2.cu vendor/clang/dist/test/Driver/darwin-asan-nofortify.c vendor/clang/dist/test/Driver/darwin-sanitizer-ld.c vendor/clang/dist/test/Driver/fsanitize.c vendor/clang/dist/test/Driver/fuzzer.c vendor/clang/dist/test/Driver/linux-header-search.cpp vendor/clang/dist/test/Driver/linux-ld.c vendor/clang/dist/test/Driver/openmp-offload-gpu.c vendor/clang/dist/test/Driver/riscv32-toolchain.c vendor/clang/dist/test/Driver/sanitizer-ld.c vendor/clang/dist/test/Frontend/clang-abi-compat.cpp vendor/clang/dist/test/Modules/resolution-change.m vendor/clang/dist/test/OpenMP/declare_target_codegen.cpp vendor/clang/dist/test/OpenMP/nvptx_declare_target_var_ctor_dtor_codegen.cpp vendor/clang/dist/test/OpenMP/target_codegen.cpp vendor/clang/dist/test/OpenMP/target_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_parallel_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_for_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_for_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_parallel_for_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_for_simd_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_for_simd_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_parallel_for_simd_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_if_codegen.cpp vendor/clang/dist/test/OpenMP/target_parallel_num_threads_codegen.cpp vendor/clang/dist/test/OpenMP/target_simd_codegen.cpp vendor/clang/dist/test/OpenMP/target_simd_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_simd_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_teams_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_parallel_for_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_parallel_for_simd_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_parallel_for_simd_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_simd_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp vendor/clang/dist/test/OpenMP/target_teams_distribute_simd_depend_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_num_teams_codegen.cpp vendor/clang/dist/test/OpenMP/target_teams_thread_limit_codegen.cpp vendor/clang/dist/test/PCH/coroutines.cpp vendor/clang/dist/test/PCH/cxx-templates.cpp vendor/clang/dist/test/PCH/cxx-templates.h vendor/clang/dist/test/Parser/cxx1z-fold-expressions.cpp vendor/clang/dist/test/SemaCUDA/device-var-init.cu vendor/clang/dist/test/SemaCXX/constexpr-string.cpp vendor/clang/dist/test/SemaCXX/cxx1z-class-template-argument-deduction.cpp vendor/clang/dist/test/SemaCXX/warn-dangling-local.cpp vendor/clang/dist/test/SemaObjCXX/noescape.mm vendor/clang/dist/test/SemaOpenCL/invalid-kernel-parameters.cl vendor/clang/dist/unittests/Format/FormatTestComments.cpp vendor/clang/dist/unittests/Format/FormatTestJS.cpp vendor/clang/dist/utils/TableGen/ClangAttrEmitter.cpp vendor/clang/dist/utils/analyzer/CmpRuns.py vendor/clang/dist/utils/analyzer/SATestBuild.py Modified: vendor/clang/dist/docs/ClangFormatStyleOptions.rst ============================================================================== --- vendor/clang/dist/docs/ClangFormatStyleOptions.rst Thu Aug 2 17:32:55 2018 (r337138) +++ vendor/clang/dist/docs/ClangFormatStyleOptions.rst Thu Aug 2 17:33:11 2018 (r337139) @@ -1134,17 +1134,21 @@ the configuration (without a prefix: ``Auto``). .. code-block:: c++ true: - SomeClass::Constructor() - : aaaaaaaa(aaaaaaaa), aaaaaaaa(aaaaaaaa), aaaaaaaa(aaaaaaaaaaaaaaaaaaaaaaaaa) { - return 0; - } + FitsOnOneLine::Constructor() + : aaaaaaaaaaaaa(aaaaaaaaaaaaaa), aaaaaaaaaaaaa(aaaaaaaaaaaaaa) {} + DoesntFit::Constructor() + : aaaaaaaaaaaaa(aaaaaaaaaaaaaa), + aaaaaaaaaaaaa(aaaaaaaaaaaaaa), + aaaaaaaaaaaaa(aaaaaaaaaaaaaa) {} + false: - SomeClass::Constructor() - : aaaaaaaa(aaaaaaaa), aaaaaaaa(aaaaaaaa), - aaaaaaaa(aaaaaaaaaaaaaaaaaaaaaaaaa) { - return 0; - } + FitsOnOneLine::Constructor() + : aaaaaaaaaaaaa(aaaaaaaaaaaaaa), aaaaaaaaaaaaa(aaaaaaaaaaaaaa) {} + + DoesntFit::Constructor() + : aaaaaaaaaaaaa(aaaaaaaaaaaaaa), aaaaaaaaaaaaa(aaaaaaaaaaaaaa), + aaaaaaaaaaaaa(aaaaaaaaaaaaaa) {} **ConstructorInitializerIndentWidth** (``unsigned``) The number of characters to use for indentation of constructor Modified: vendor/clang/dist/docs/ReleaseNotes.rst ============================================================================== --- vendor/clang/dist/docs/ReleaseNotes.rst Thu Aug 2 17:32:55 2018 (r337138) +++ vendor/clang/dist/docs/ReleaseNotes.rst Thu Aug 2 17:33:11 2018 (r337139) @@ -46,7 +46,9 @@ sections with improvements to Clang's support for thos Major New Features ------------------ -- ... +- A new Implicit Conversion Sanitizer (``-fsanitize=implicit-conversion``) group + was added. Please refer to the :ref:`release-notes-ubsan` section of the + release notes for the details. Improvements to Clang's diagnostics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -280,10 +282,36 @@ Static Analyzer ... +.. _release-notes-ubsan: + Undefined Behavior Sanitizer (UBSan) ------------------------------------ -* ... +* A new Implicit Conversion Sanitizer (``-fsanitize=implicit-conversion``) group + was added. + + Currently, only one type of issues is caught - implicit integer truncation + (``-fsanitize=implicit-integer-truncation``), also known as integer demotion. + While there is a ``-Wconversion`` diagnostic group that catches this kind of + issues, it is both noisy, and does not catch **all** the cases. + + .. code-block:: c++ + + unsigned char store = 0; + + bool consume(unsigned int val); + + void test(unsigned long val) { + if (consume(val)) // the value may have been silently truncated. + store = store + 768; // before addition, 'store' was promoted to int. + (void)consume((unsigned int)val); // OK, the truncation is explicit. + } + + Just like other ``-fsanitize=integer`` checks, these issues are **not** + undefined behaviour. But they are not *always* intentional, and are somewhat + hard to track down. This group is **not** enabled by ``-fsanitize=undefined``, + but the ``-fsanitize=implicit-integer-truncation`` check + is enabled by ``-fsanitize=integer``. Core Analysis Improvements ========================== Modified: vendor/clang/dist/docs/UndefinedBehaviorSanitizer.rst ============================================================================== --- vendor/clang/dist/docs/UndefinedBehaviorSanitizer.rst Thu Aug 2 17:32:55 2018 (r337138) +++ vendor/clang/dist/docs/UndefinedBehaviorSanitizer.rst Thu Aug 2 17:33:11 2018 (r337139) @@ -89,6 +89,12 @@ Available checks are: - ``-fsanitize=function``: Indirect call of a function through a function pointer of the wrong type (Darwin/Linux, C++ and x86/x86_64 only). + - ``-fsanitize=implicit-integer-truncation``: Implicit conversion from + integer of larger bit width to smaller bit width, if that results in data + loss. That is, if the demoted value, after casting back to the original + width, is not equal to the original value before the downcast. + Issues caught by this sanitizer are not undefined behavior, + but are often unintentional. - ``-fsanitize=integer-divide-by-zero``: Integer division by zero. - ``-fsanitize=nonnull-attribute``: Passing null pointer as a function parameter which is declared to never be null. @@ -121,15 +127,21 @@ Available checks are: unsigned overflow in C++. You can use ``-fsanitize=shift-base`` or ``-fsanitize=shift-exponent`` to check only left-hand side or right-hand side of shift operation, respectively. - - ``-fsanitize=signed-integer-overflow``: Signed integer overflow, - including all the checks added by ``-ftrapv``, and checking for - overflow in signed division (``INT_MIN / -1``). + - ``-fsanitize=signed-integer-overflow``: Signed integer overflow, where the + result of a signed integer computation cannot be represented in its type. + This includes all the checks covered by ``-ftrapv``, as well as checks for + signed division overflow (``INT_MIN/-1``), but not checks for + lossy implicit conversions performed before the computation + (see ``-fsanitize=implicit-conversion``). Both of these two issues are + handled by ``-fsanitize=implicit-conversion`` group of checks. - ``-fsanitize=unreachable``: If control flow reaches an unreachable program point. - - ``-fsanitize=unsigned-integer-overflow``: Unsigned integer - overflows. Note that unlike signed integer overflow, unsigned integer - is not undefined behavior. However, while it has well-defined semantics, - it is often unintentional, so UBSan offers to catch it. + - ``-fsanitize=unsigned-integer-overflow``: Unsigned integer overflow, where + the result of an unsigned integer computation cannot be represented in its + type. Unlike signed integer overflow, this is not undefined behavior, but + it is often unintentional. This sanitizer does not check for lossy implicit + conversions performed before such a computation + (see ``-fsanitize=implicit-conversion``). - ``-fsanitize=vla-bound``: A variable-length array whose bound does not evaluate to a positive value. - ``-fsanitize=vptr``: Use of an object whose vptr indicates that it is of @@ -140,11 +152,17 @@ Available checks are: You can also use the following check groups: - ``-fsanitize=undefined``: All of the checks listed above other than - ``unsigned-integer-overflow`` and the ``nullability-*`` checks. + ``unsigned-integer-overflow``, ``implicit-conversion`` and the + ``nullability-*`` group of checks. - ``-fsanitize=undefined-trap``: Deprecated alias of ``-fsanitize=undefined``. - ``-fsanitize=integer``: Checks for undefined or suspicious integer behavior (e.g. unsigned integer overflow). + Enables ``signed-integer-overflow``, ``unsigned-integer-overflow``, + ``shift``, ``integer-divide-by-zero``, and ``implicit-integer-truncation``. + - ``-fsanitize=implicit-conversion``: Checks for suspicious behaviours of + implicit conversions. + Currently, only ``-fsanitize=implicit-integer-truncation`` is implemented. - ``-fsanitize=nullability``: Enables ``nullability-arg``, ``nullability-assign``, and ``nullability-return``. While violating nullability does not have undefined behavior, it is often unintentional, Modified: vendor/clang/dist/docs/UsersManual.rst ============================================================================== --- vendor/clang/dist/docs/UsersManual.rst Thu Aug 2 17:32:55 2018 (r337138) +++ vendor/clang/dist/docs/UsersManual.rst Thu Aug 2 17:33:11 2018 (r337139) @@ -2155,13 +2155,8 @@ Objective-C++ Language Features OpenMP Features =============== -Clang supports all OpenMP 3.1 directives and clauses. In addition, some -features of OpenMP 4.0 are supported. For example, ``#pragma omp simd``, -``#pragma omp for simd``, ``#pragma omp parallel for simd`` directives, extended -set of atomic constructs, ``proc_bind`` clause for all parallel-based -directives, ``depend`` clause for ``#pragma omp task`` directive (except for -array sections), ``#pragma omp cancel`` and ``#pragma omp cancellation point`` -directives, and ``#pragma omp taskgroup`` directive. +Clang supports all OpenMP 4.5 directives and clauses. See :doc:`OpenMPSupport` +for additional details. Use `-fopenmp` to enable OpenMP. Support for OpenMP can be disabled with `-fno-openmp`. @@ -2783,6 +2778,7 @@ Execute ``clang-cl /?`` to see a list of supported opt /Brepro Emit an object file which can be reproduced over time /C Don't discard comments when preprocessing /c Compile only + /d1PP Retain macro definitions in /E mode /d1reportAllClassLayout Dump record layout information /diagnostics:caret Enable caret and column diagnostics (on by default) /diagnostics:classic Disable column and caret diagnostics @@ -2816,6 +2812,7 @@ Execute ``clang-cl /?`` to see a list of supported opt /GS- Disable buffer security check /GS Enable buffer security check /Gs Set stack probe size + /guard: Enable Control Flow Guard with /guard:cf /Gv Set __vectorcall as a default calling convention /Gw- Don't put each data item in its own section /Gw Put each data item in its own section @@ -2871,6 +2868,7 @@ Execute ``clang-cl /?`` to see a list of supported opt /WX- Do not treat warnings as errors /WX Treat warnings as errors /w Disable all warnings + /X Don't add %INCLUDE% to the include search path /Y- Disable precompiled headers, overrides /Yc and /Yu /Yc Generate a pch file for all code up to and including /Yu Load a pch file and use it instead of all code up to and including @@ -2894,8 +2892,15 @@ Execute ``clang-cl /?`` to see a list of supported opt OPTIONS: -### Print (but do not run) the commands to run for this compilation --analyze Run the static analyzer + -faddrsig Emit an address-significance table -fansi-escape-codes Use ANSI escape codes for diagnostics + -fblocks Enable the 'blocks' language feature + -fcf-protection= Instrument control-flow architecture protection. Options: return, branch, full, none. + -fcf-protection Enable cf-protection in 'full' mode -fcolor-diagnostics Use colors in diagnostics + -fcomplete-member-pointers + Require member pointer base types to be complete if they would be significant under the Microsoft ABI + -fcoverage-mapping Generate coverage mapping to enable code coverage analysis -fdebug-macro Emit macro debug information -fdelayed-template-parsing Parse templated function definitions at the end of the translation unit @@ -2905,6 +2910,7 @@ Execute ``clang-cl /?`` to see a list of supported opt Print fix-its in machine parseable form -flto= Set LTO mode to either 'full' or 'thin' -flto Enable LTO in 'full' mode + -fmerge-all-constants Allow merging of constants -fms-compatibility-version= Dot-separated value representing the Microsoft compiler version number to report in _MSC_VER (0 = don't define it (default)) @@ -2912,9 +2918,17 @@ Execute ``clang-cl /?`` to see a list of supported opt -fms-extensions Accept some non-standard constructs supported by the Microsoft compiler -fmsc-version= Microsoft compiler version number to report in _MSC_VER (0 = don't define it (default)) + -fno-addrsig Don't emit an address-significance table + -fno-builtin- Disable implicit builtin knowledge of a specific function + -fno-builtin Disable implicit builtin knowledge of functions + -fno-complete-member-pointers + Do not require member pointer base types to be complete if they would be significant under the Microsoft ABI + -fno-coverage-mapping Disable code coverage analysis -fno-debug-macro Do not emit macro debug information -fno-delayed-template-parsing Disable delayed template parsing + -fno-sanitize-address-poison-class-member-array-new-cookie + Disable poisoning array cookies when using class member operator new[] in AddressSanitizer -fno-sanitize-address-use-after-scope Disable use-after-scope detection in AddressSanitizer -fno-sanitize-blacklist Don't use blacklist file for sanitizers @@ -2950,6 +2964,8 @@ Execute ``clang-cl /?`` to see a list of supported opt Level of field padding for AddressSanitizer -fsanitize-address-globals-dead-stripping Enable linker dead stripping of globals in AddressSanitizer + -fsanitize-address-poison-class-member-array-new-cookie + Enable poisoning array cookies when using class member operator new[] in AddressSanitizer -fsanitize-address-use-after-scope Enable use-after-scope detection in AddressSanitizer -fsanitize-blacklist= Modified: vendor/clang/dist/include/clang-c/Index.h ============================================================================== --- vendor/clang/dist/include/clang-c/Index.h Thu Aug 2 17:32:55 2018 (r337138) +++ vendor/clang/dist/include/clang-c/Index.h Thu Aug 2 17:33:11 2018 (r337139) @@ -160,7 +160,7 @@ typedef struct CXVersion { int Major; /** * The minor version number, e.g., the '7' in '10.7.3'. This value - * will be negative if no minor version number was provided, e.g., for + * will be negative if no minor version number was provided, e.g., for * version '10'. */ int Minor; @@ -387,7 +387,7 @@ CINDEX_LINKAGE int clang_getFileUniqueID(CXFile file, * multiple inclusions, either with the conventional * \#ifndef/\#define/\#endif macro guards or with \#pragma once. */ -CINDEX_LINKAGE unsigned +CINDEX_LINKAGE unsigned clang_isFileMultipleIncludeGuarded(CXTranslationUnit tu, CXFile file); /** @@ -786,7 +786,7 @@ typedef void *CXDiagnostic; * A group of CXDiagnostics. */ typedef void *CXDiagnosticSet; - + /** * Determine the number of diagnostics in a CXDiagnosticSet. */ @@ -802,7 +802,7 @@ CINDEX_LINKAGE unsigned clang_getNumDiagnosticsInSet(C * via a call to \c clang_disposeDiagnostic(). */ CINDEX_LINKAGE CXDiagnostic clang_getDiagnosticInSet(CXDiagnosticSet Diags, - unsigned Index); + unsigned Index); /** * Describes the kind of error that occurred (if any) in a call to @@ -813,26 +813,26 @@ enum CXLoadDiag_Error { * Indicates that no error occurred. */ CXLoadDiag_None = 0, - + /** * Indicates that an unknown error occurred while attempting to * deserialize diagnostics. */ CXLoadDiag_Unknown = 1, - + /** * Indicates that the file containing the serialized diagnostics * could not be opened. */ CXLoadDiag_CannotLoad = 2, - + /** * Indicates that the serialized diagnostics file is invalid or * corrupt. */ CXLoadDiag_InvalidFile = 3 }; - + /** * Deserialize a set of diagnostics from a Clang diagnostics bitcode * file. @@ -856,7 +856,7 @@ CINDEX_LINKAGE CXDiagnosticSet clang_loadDiagnostics(c CINDEX_LINKAGE void clang_disposeDiagnosticSet(CXDiagnosticSet Diags); /** - * Retrieve the child diagnostics of a CXDiagnostic. + * Retrieve the child diagnostics of a CXDiagnostic. * * This CXDiagnosticSet does not need to be released by * clang_disposeDiagnosticSet. @@ -888,7 +888,7 @@ CINDEX_LINKAGE CXDiagnostic clang_getDiagnostic(CXTran * \param Unit the translation unit to query. */ CINDEX_LINKAGE CXDiagnosticSet - clang_getDiagnosticSetFromTU(CXTranslationUnit Unit); + clang_getDiagnosticSetFromTU(CXTranslationUnit Unit); /** * Destroy a diagnostic. @@ -934,7 +934,7 @@ enum CXDiagnosticDisplayOptions { * \c -fdiagnostics-print-source-range-info. */ CXDiagnostic_DisplaySourceRanges = 0x04, - + /** * Display the option name associated with this diagnostic, if any. * @@ -943,12 +943,12 @@ enum CXDiagnosticDisplayOptions { * \c -fdiagnostics-show-option. */ CXDiagnostic_DisplayOption = 0x08, - + /** * Display the category number associated with this diagnostic, if any. * * The category number is displayed within brackets after the diagnostic text. - * This option corresponds to the clang flag + * This option corresponds to the clang flag * \c -fdiagnostics-show-category=id. */ CXDiagnostic_DisplayCategoryId = 0x10, @@ -957,7 +957,7 @@ enum CXDiagnosticDisplayOptions { * Display the category name associated with this diagnostic, if any. * * The category name is displayed within brackets after the diagnostic text. - * This option corresponds to the clang flag + * This option corresponds to the clang flag * \c -fdiagnostics-show-category=name. */ CXDiagnostic_DisplayCategoryName = 0x20 @@ -1019,7 +1019,7 @@ CINDEX_LINKAGE CXString clang_getDiagnosticSpelling(CX * diagnostic (if any). * * \returns A string that contains the command-line option used to enable this - * warning, such as "-Wconversion" or "-pedantic". + * warning, such as "-Wconversion" or "-pedantic". */ CINDEX_LINKAGE CXString clang_getDiagnosticOption(CXDiagnostic Diag, CXString *Disable); @@ -1028,7 +1028,7 @@ CINDEX_LINKAGE CXString clang_getDiagnosticOption(CXDi * Retrieve the category number for this diagnostic. * * Diagnostics can be categorized into groups along with other, related - * diagnostics (e.g., diagnostics under the same warning flag). This routine + * diagnostics (e.g., diagnostics under the same warning flag). This routine * retrieves the category number for the given diagnostic. * * \returns The number of the category that contains this diagnostic, or zero @@ -1041,7 +1041,7 @@ CINDEX_LINKAGE unsigned clang_getDiagnosticCategory(CX * is now deprecated. Use clang_getDiagnosticCategoryText() * instead. * - * \param Category A diagnostic category number, as returned by + * \param Category A diagnostic category number, as returned by * \c clang_getDiagnosticCategory(). * * \returns The name of the given diagnostic category. @@ -1055,7 +1055,7 @@ CXString clang_getDiagnosticCategoryName(unsigned Cate * \returns The text of the given diagnostic category. */ CINDEX_LINKAGE CXString clang_getDiagnosticCategoryText(CXDiagnostic); - + /** * Determine the number of source ranges associated with the given * diagnostic. @@ -1242,9 +1242,9 @@ enum CXTranslationUnit_Flags { * intent of producing a precompiled header. */ CXTranslationUnit_Incomplete = 0x02, - + /** - * Used to indicate that the translation unit should be built with an + * Used to indicate that the translation unit should be built with an * implicit precompiled header for the preamble. * * An implicit precompiled header is used as an optimization when a @@ -1258,7 +1258,7 @@ enum CXTranslationUnit_Flags { * precompiled header to improve parsing performance. */ CXTranslationUnit_PrecompiledPreamble = 0x04, - + /** * Used to indicate that the translation unit should cache some * code-completion results with each reparse of the source file. @@ -1343,7 +1343,7 @@ enum CXTranslationUnit_Flags { * to indicate that the translation unit is likely to be reparsed many times, * either explicitly (via \c clang_reparseTranslationUnit()) or implicitly * (e.g., by code completion (\c clang_codeCompletionAt())). The returned flag - * set contains an unspecified set of optimizations (e.g., the precompiled + * set contains an unspecified set of optimizations (e.g., the precompiled * preamble) geared toward improving the performance of these routines. The * set of optimizations enabled may change from one version to the next. */ @@ -1374,7 +1374,7 @@ clang_parseTranslationUnit(CXIndex CIdx, * command-line arguments so that the compilation can be configured in the same * way that the compiler is configured on the command line. * - * \param CIdx The index object with which the translation unit will be + * \param CIdx The index object with which the translation unit will be * associated. * * \param source_filename The name of the source file to load, or NULL if the @@ -1383,7 +1383,7 @@ clang_parseTranslationUnit(CXIndex CIdx, * \param command_line_args The command-line arguments that would be * passed to the \c clang executable if it were being invoked out-of-process. * These command-line options will be parsed and will affect how the translation - * unit is parsed. Note that the following options are ignored: '-c', + * unit is parsed. Note that the following options are ignored: '-c', * '-emit-ast', '-fsyntax-only' (which is the default), and '-o \'. * * \param num_command_line_args The number of command-line arguments in @@ -1463,32 +1463,32 @@ enum CXSaveError { * Indicates that no error occurred while saving a translation unit. */ CXSaveError_None = 0, - + /** * Indicates that an unknown error occurred while attempting to save * the file. * - * This error typically indicates that file I/O failed when attempting to + * This error typically indicates that file I/O failed when attempting to * write the file. */ CXSaveError_Unknown = 1, - + /** * Indicates that errors during translation prevented this attempt * to save the translation unit. - * + * * Errors that prevent the translation unit from being saved can be * extracted using \c clang_getNumDiagnostics() and \c clang_getDiagnostic(). */ CXSaveError_TranslationErrors = 2, - + /** * Indicates that the translation unit to be saved was somehow * invalid (e.g., NULL). */ CXSaveError_InvalidTU = 3 }; - + /** * Saves a translation unit into a serialized representation of * that translation unit on disk. @@ -1509,7 +1509,7 @@ enum CXSaveError { * CXSaveTranslationUnit_XXX flags. * * \returns A value that will match one of the enumerators of the CXSaveError - * enumeration. Zero (CXSaveError_None) indicates that the translation unit was + * enumeration. Zero (CXSaveError_None) indicates that the translation unit was * saved successfully, while a non-zero value indicates that a problem occurred. */ CINDEX_LINKAGE int clang_saveTranslationUnit(CXTranslationUnit TU, @@ -1543,7 +1543,7 @@ enum CXReparse_Flags { */ CXReparse_None = 0x0 }; - + /** * Returns the set of flags that is suitable for reparsing a translation * unit. @@ -1551,7 +1551,7 @@ enum CXReparse_Flags { * The set of flags returned provide options for * \c clang_reparseTranslationUnit() by default. The returned flag * set contains an unspecified set of optimizations geared toward common uses - * of reparsing. The set of optimizations enabled may change from one version + * of reparsing. The set of optimizations enabled may change from one version * to the next. */ CINDEX_LINKAGE unsigned clang_defaultReparseOptions(CXTranslationUnit TU); @@ -1563,17 +1563,17 @@ CINDEX_LINKAGE unsigned clang_defaultReparseOptions(CX * created the given translation unit, for example because those source files * have changed (either on disk or as passed via \p unsaved_files). The * source code will be reparsed with the same command-line options as it - * was originally parsed. + * was originally parsed. * * Reparsing a translation unit invalidates all cursors and source locations * that refer into that translation unit. This makes reparsing a translation * unit semantically equivalent to destroying the translation unit and then * creating a new translation unit with the same command-line arguments. - * However, it may be more efficient to reparse a translation + * However, it may be more efficient to reparse a translation * unit using this routine. * * \param TU The translation unit whose contents will be re-parsed. The - * translation unit must originally have been built with + * translation unit must originally have been built with * \c clang_createTranslationUnitFromSourceFile(). * * \param num_unsaved_files The number of unsaved file entries in \p @@ -1584,7 +1584,7 @@ CINDEX_LINKAGE unsigned clang_defaultReparseOptions(CX * those files. The contents and name of these files (as specified by * CXUnsavedFile) are copied when necessary, so the client only needs to * guarantee their validity until the call to this function returns. - * + * * \param options A bitset of options composed of the flags in CXReparse_Flags. * The function \c clang_defaultReparseOptions() produces a default set of * options recommended for most uses, based on the translation unit. @@ -1612,8 +1612,8 @@ enum CXTUResourceUsageKind { CXTUResourceUsage_AST_SideTables = 6, CXTUResourceUsage_SourceManager_Membuffer_Malloc = 7, CXTUResourceUsage_SourceManager_Membuffer_MMap = 8, - CXTUResourceUsage_ExternalASTSource_Membuffer_Malloc = 9, - CXTUResourceUsage_ExternalASTSource_Membuffer_MMap = 10, + CXTUResourceUsage_ExternalASTSource_Membuffer_Malloc = 9, + CXTUResourceUsage_ExternalASTSource_Membuffer_MMap = 10, CXTUResourceUsage_Preprocessor = 11, CXTUResourceUsage_PreprocessingRecord = 12, CXTUResourceUsage_SourceManager_DataStructures = 13, @@ -1635,8 +1635,8 @@ const char *clang_getTUResourceUsageName(enum CXTUReso typedef struct CXTUResourceUsageEntry { /* The memory usage category. */ - enum CXTUResourceUsageKind kind; - /* Amount of resources used. + enum CXTUResourceUsageKind kind; + /* Amount of resources used. The units will depend on the resource kind. */ unsigned long amount; } CXTUResourceUsageEntry; @@ -1819,7 +1819,7 @@ enum CXCursorKind { */ CXCursor_TypeRef = 43, CXCursor_CXXBaseSpecifier = 44, - /** + /** * A reference to a class template, function template, template * template parameter, or class template partial specialization. */ @@ -1829,14 +1829,14 @@ enum CXCursorKind { */ CXCursor_NamespaceRef = 46, /** - * A reference to a member of a struct, union, or class that occurs in + * A reference to a member of a struct, union, or class that occurs in * some non-expression context, e.g., a designated initializer. */ CXCursor_MemberRef = 47, /** * A reference to a labeled statement. * - * This cursor kind is used to describe the jump to "start_over" in the + * This cursor kind is used to describe the jump to "start_over" in the * goto statement in the following example: * * \code @@ -1849,7 +1849,7 @@ enum CXCursorKind { * A label reference cursor refers to a label statement. */ CXCursor_LabelRef = 48, - + /** * A reference to a set of overloaded functions or function templates * that has not yet been resolved to a specific function or function template. @@ -1882,18 +1882,18 @@ enum CXCursorKind { * argument-dependent lookup (e.g., the "swap" function at the end of the * example). * - * The functions \c clang_getNumOverloadedDecls() and + * The functions \c clang_getNumOverloadedDecls() and * \c clang_getOverloadedDecl() can be used to retrieve the definitions * referenced by this cursor. */ CXCursor_OverloadedDeclRef = 49, - + /** - * A reference to a variable that occurs in some non-expression + * A reference to a variable that occurs in some non-expression * context, e.g., a C++ lambda capture list. */ CXCursor_VariableRef = 50, - + CXCursor_LastRef = CXCursor_VariableRef, /* Error conditions */ @@ -2153,7 +2153,7 @@ enum CXCursorKind { * \endcode */ CXCursor_LambdaExpr = 144, - + /** Objective-c Boolean Literal. */ CXCursor_ObjCBoolLiteralExpr = 145, @@ -2189,10 +2189,10 @@ enum CXCursorKind { * reported. */ CXCursor_UnexposedStmt = 200, - - /** A labelled statement in a function. + + /** A labelled statement in a function. * - * This cursor kind is used to describe the "start_over:" label statement in + * This cursor kind is used to describe the "start_over:" label statement in * the following example: * * \code @@ -2650,7 +2650,7 @@ CINDEX_LINKAGE int clang_Cursor_isNull(CXCursor cursor * Compute a hash value for the given cursor. */ CINDEX_LINKAGE unsigned clang_hashCursor(CXCursor); - + /** * Retrieve the kind of the given cursor. */ @@ -2718,7 +2718,7 @@ CINDEX_LINKAGE unsigned clang_isTranslationUnit(enum C * element, such as a preprocessor directive or macro instantiation. */ CINDEX_LINKAGE unsigned clang_isPreprocessing(enum CXCursorKind); - + /*** * Determine whether the given cursor represents a currently * unexposed piece of the AST (e.g., CXCursor_UnexposedStmt). @@ -2785,7 +2785,7 @@ CINDEX_LINKAGE enum CXVisibilityKind clang_getCursorVi * * \returns The availability of the cursor. */ -CINDEX_LINKAGE enum CXAvailabilityKind +CINDEX_LINKAGE enum CXAvailabilityKind clang_getCursorAvailability(CXCursor cursor); /** @@ -2831,10 +2831,10 @@ typedef struct CXPlatformAvailability { * * \param cursor The cursor to query. * - * \param always_deprecated If non-NULL, will be set to indicate whether the + * \param always_deprecated If non-NULL, will be set to indicate whether the * entity is deprecated on all platforms. * - * \param deprecated_message If non-NULL, will be set to the message text + * \param deprecated_message If non-NULL, will be set to the message text * provided along with the unconditional deprecation of this entity. The client * is responsible for deallocating this string. * @@ -2842,7 +2842,7 @@ typedef struct CXPlatformAvailability { * entity is unavailable on all platforms. * * \param unavailable_message If non-NULL, will be set to the message text - * provided along with the unconditional unavailability of this entity. The + * provided along with the unconditional unavailability of this entity. The * client is responsible for deallocating this string. * * \param availability If non-NULL, an array of CXPlatformAvailability instances @@ -2850,15 +2850,15 @@ typedef struct CXPlatformAvailability { * the number of platforms for which availability information is available (as * returned by this function) or \c availability_size, whichever is smaller. * - * \param availability_size The number of elements available in the + * \param availability_size The number of elements available in the * \c availability array. * * \returns The number of platforms (N) for which availability information is * available (which is unrelated to \c availability_size). * - * Note that the client is responsible for calling - * \c clang_disposeCXPlatformAvailability to free each of the - * platform-availability structures returned. There are + * Note that the client is responsible for calling + * \c clang_disposeCXPlatformAvailability to free each of the + * platform-availability structures returned. There are * \c min(N, availability_size) such structures. */ CINDEX_LINKAGE int @@ -2875,7 +2875,7 @@ clang_getCursorPlatformAvailability(CXCursor cursor, */ CINDEX_LINKAGE void clang_disposeCXPlatformAvailability(CXPlatformAvailability *availability); - + /** * Describe the "language" of the entity referred to by a cursor. */ @@ -2948,7 +2948,7 @@ CINDEX_LINKAGE unsigned clang_CXCursorSet_insert(CXCur * * The semantic parent of a cursor is the cursor that semantically contains * the given \p cursor. For many declarations, the lexical and semantic parents - * are equivalent (the lexical parent is returned by + * are equivalent (the lexical parent is returned by * \c clang_getCursorLexicalParent()). They diverge when declarations or * definitions are provided out-of-line. For example: * @@ -2983,7 +2983,7 @@ CINDEX_LINKAGE CXCursor clang_getCursorSemanticParent( * * The lexical parent of a cursor is the cursor in which the given \p cursor * was actually written. For many declarations, the lexical and semantic parents - * are equivalent (the semantic parent is returned by + * are equivalent (the semantic parent is returned by * \c clang_getCursorSemanticParent()). They diverge when declarations or * definitions are provided out-of-line. For example: * @@ -3046,18 +3046,18 @@ CINDEX_LINKAGE CXCursor clang_getCursorLexicalParent(C * \param cursor A cursor representing an Objective-C or C++ * method. This routine will compute the set of methods that this * method overrides. - * + * * \param overridden A pointer whose pointee will be replaced with a * pointer to an array of cursors, representing the set of overridden * methods. If there are no overridden methods, the pointee will be - * set to NULL. The pointee must be freed via a call to + * set to NULL. The pointee must be freed via a call to * \c clang_disposeOverriddenCursors(). * * \param num_overridden A pointer to the number of overridden * functions, will be set to the number of overridden functions in the * array pointed to by \p overridden. */ -CINDEX_LINKAGE void clang_getOverriddenCursors(CXCursor cursor, +CINDEX_LINKAGE void clang_getOverriddenCursors(CXCursor cursor, CXCursor **overridden, unsigned *num_overridden); @@ -3072,7 +3072,7 @@ CINDEX_LINKAGE void clang_disposeOverriddenCursors(CXC * cursor. */ CINDEX_LINKAGE CXFile clang_getIncludedFile(CXCursor cursor); - + /** * @} */ @@ -3133,7 +3133,7 @@ CINDEX_LINKAGE CXSourceRange clang_getCursorExtent(CXC /** * @} */ - + /** * \defgroup CINDEX_TYPES Type information for CXCursors * @@ -3582,7 +3582,7 @@ CINDEX_LINKAGE CXString clang_getDeclObjCTypeEncoding( /** * Returns the Objective-C type encoding for the specified CXType. */ -CINDEX_LINKAGE CXString clang_Type_getObjCEncoding(CXType type); +CINDEX_LINKAGE CXString clang_Type_getObjCEncoding(CXType type); /** * Retrieve the spelling of a given CXTypeKind. @@ -3842,7 +3842,7 @@ CINDEX_LINKAGE unsigned clang_Cursor_isBitField(CXCurs * CX_CXXBaseSpecifier is virtual. */ CINDEX_LINKAGE unsigned clang_isVirtualBase(CXCursor); - + /** * Represents the C++ access control level to a base class for a * cursor with kind CX_CXXBaseSpecifier. @@ -3887,7 +3887,7 @@ enum CX_StorageClass { CINDEX_LINKAGE enum CX_StorageClass clang_Cursor_getStorageClass(CXCursor); /** - * Determine the number of overloaded declarations referenced by a + * Determine the number of overloaded declarations referenced by a * \c CXCursor_OverloadedDeclRef cursor. * * \param cursor The cursor whose overloaded declarations are being queried. @@ -3906,18 +3906,18 @@ CINDEX_LINKAGE unsigned clang_getNumOverloadedDecls(CX * \param index The zero-based index into the set of overloaded declarations in * the cursor. * - * \returns A cursor representing the declaration referenced by the given - * \c cursor at the specified \c index. If the cursor does not have an + * \returns A cursor representing the declaration referenced by the given + * \c cursor at the specified \c index. If the cursor does not have an * associated set of overloaded declarations, or if the index is out of bounds, * returns \c clang_getNullCursor(); */ -CINDEX_LINKAGE CXCursor clang_getOverloadedDecl(CXCursor cursor, +CINDEX_LINKAGE CXCursor clang_getOverloadedDecl(CXCursor cursor, unsigned index); - + /** * @} */ - + /** * \defgroup CINDEX_ATTRIBUTES Information for attributes * @@ -4021,7 +4021,7 @@ CINDEX_LINKAGE unsigned clang_visitChildren(CXCursor p * The visitor should return one of the \c CXChildVisitResult values * to direct clang_visitChildrenWithBlock(). */ -typedef enum CXChildVisitResult +typedef enum CXChildVisitResult (^CXCursorVisitorBlock)(CXCursor cursor, CXCursor parent); /** @@ -4109,10 +4109,10 @@ CINDEX_LINKAGE CXString clang_getCursorSpelling(CXCurs * Most of the times there is only one range for the complete spelling but for * Objective-C methods and Objective-C message expressions, there are multiple * pieces for each selector identifier. - * + * * \param pieceIndex the index of the spelling name piece. If this is greater * than the actual number of pieces, it will return a NULL (invalid) range. - * + * * \param options Reserved. */ CINDEX_LINKAGE CXSourceRange clang_Cursor_getSpellingNameRange(CXCursor, @@ -4206,11 +4206,11 @@ CINDEX_LINKAGE CXString clang_getCursorPrettyPrinted(C * Retrieve the display name for the entity referenced by this cursor. * * The display name contains extra information that helps identify the cursor, - * such as the parameters of a function or template or the arguments of a + * such as the parameters of a function or template or the arguments of a * class template specialization. */ CINDEX_LINKAGE CXString clang_getCursorDisplayName(CXCursor); - + /** For a cursor that is a reference, retrieve a cursor representing the * entity that it references. * @@ -4274,10 +4274,10 @@ CINDEX_LINKAGE unsigned clang_isCursorDefinition(CXCur * }; * \endcode * - * The declarations and the definition of \c X are represented by three - * different cursors, all of which are declarations of the same underlying + * The declarations and the definition of \c X are represented by three + * different cursors, all of which are declarations of the same underlying * entity. One of these cursor is considered the "canonical" cursor, which - * is effectively the representative for the underlying entity. One can + * is effectively the representative for the underlying entity. One can * determine if two cursors are declarations of the same underlying entity by * comparing their canonical cursors. * @@ -4301,11 +4301,11 @@ CINDEX_LINKAGE int clang_Cursor_getObjCSelectorIndex(C /** * Given a cursor pointing to a C++ method call or an Objective-C * message, returns non-zero if the method/message is "dynamic", meaning: - * + * * For a C++ method: the call is virtual. * For an Objective-C message: the receiver is an object instance, not 'super' * or a specific class. - * + * * If the method/message is "static" or the cursor does not point to a * method/message, it will return zero. */ @@ -4575,7 +4575,7 @@ CINDEX_LINKAGE unsigned clang_CXXMethod_isDefaulted(CX CINDEX_LINKAGE unsigned clang_CXXMethod_isPureVirtual(CXCursor C); /** - * Determine if a C++ member function or member function template is + * Determine if a C++ member function or member function template is * declared 'static'. */ CINDEX_LINKAGE unsigned clang_CXXMethod_isStatic(CXCursor C); @@ -4622,16 +4622,16 @@ CINDEX_LINKAGE unsigned clang_CXXMethod_isConst(CXCurs * \c CXCursor_NoDeclFound. */ CINDEX_LINKAGE enum CXCursorKind clang_getTemplateCursorKind(CXCursor C); - + /** * Given a cursor that may represent a specialization or instantiation * of a template, retrieve the cursor that represents the template that it * specializes or from which it was instantiated. * - * This routine determines the template involved both for explicit + * This routine determines the template involved both for explicit * specializations of templates and for implicit instantiations of the template, * both of which are referred to as "specializations". For a class template - * specialization (e.g., \c std::vector), this routine will return + * specialization (e.g., \c std::vector), this routine will return * either the primary template (\c std::vector) or, if the specialization was * instantiated from a class template partial specialization, the class template * partial specialization. For a class template partial specialization and a @@ -4639,7 +4639,7 @@ CINDEX_LINKAGE enum CXCursorKind clang_getTemplateCurs * this routine will return the specialized template. * * For members of a class template (e.g., member functions, member classes, or - * static data members), returns the specialized or instantiated member. + * static data members), returns the specialized or instantiated member. * Although not strictly "templates" in the C++ language, members of class * templates have the same notions of specializations and instantiations that * templates do, so this routine treats them similarly. @@ -4647,7 +4647,7 @@ CINDEX_LINKAGE enum CXCursorKind clang_getTemplateCurs * \param C A cursor that may be a specialization of a template or a member * of a template. * - * \returns If the given cursor is a specialization or instantiation of a + * \returns If the given cursor is a specialization or instantiation of a * template or a member thereof, the template or member that it specializes or * from which it was instantiated. Otherwise, returns a NULL cursor. */ @@ -4659,11 +4659,11 @@ CINDEX_LINKAGE CXCursor clang_getSpecializedCursorTemp * * \param C A cursor pointing to a member reference, a declaration reference, or * an operator call. - * \param NameFlags A bitset with three independent flags: + * \param NameFlags A bitset with three independent flags: * CXNameRange_WantQualifier, CXNameRange_WantTemplateArgs, and * CXNameRange_WantSinglePiece. - * \param PieceIndex For contiguous names or when passing the flag - * CXNameRange_WantSinglePiece, only one piece with index 0 is + * \param PieceIndex For contiguous names or when passing the flag + * CXNameRange_WantSinglePiece, only one piece with index 0 is * available. When the CXNameRange_WantSinglePiece flag is not passed for a * non-contiguous names, this index can be used to retrieve the individual * pieces of the name. See also CXNameRange_WantSinglePiece. @@ -4672,7 +4672,7 @@ CINDEX_LINKAGE CXCursor clang_getSpecializedCursorTemp * name, or if the PieceIndex is out-of-range, a null-cursor will be returned. */ CINDEX_LINKAGE CXSourceRange clang_getCursorReferenceNameRange(CXCursor C, - unsigned NameFlags, + unsigned NameFlags, unsigned PieceIndex); enum CXNameRefFlags { @@ -4681,7 +4681,7 @@ enum CXNameRefFlags { * range. */ CXNameRange_WantQualifier = 0x1, - + /** * Include the explicit template arguments, e.g. \ in x.f, * in the range. @@ -4700,7 +4700,7 @@ enum CXNameRefFlags { */ CXNameRange_WantSinglePiece = 0x4 }; - + /** * @} */ @@ -5153,7 +5153,7 @@ clang_getNumCompletionChunks(CXCompletionString comple /** * Determine the priority of this code completion. * - * The priority of a code completion indicates how likely it is that this + * The priority of a code completion indicates how likely it is that this * particular completion is the completion that the user will select. The * priority is selected by various internal heuristics. * @@ -5164,7 +5164,7 @@ clang_getNumCompletionChunks(CXCompletionString comple */ CINDEX_LINKAGE unsigned clang_getCompletionPriority(CXCompletionString completion_string); - + /** * Determine the availability of the entity that this code-completion *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:29 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47096106D753; Thu, 2 Aug 2018 17:33:29 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 57BF572505; Thu, 2 Aug 2018 17:33:27 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0654114D32; Thu, 2 Aug 2018 17:33:27 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXQww029457; Thu, 2 Aug 2018 17:33:26 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXJNY029420; Thu, 2 Aug 2018 17:33:19 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXJNY029420@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337141 - in vendor/compiler-rt/dist: lib/asan lib/asan/tests lib/builtins lib/profile lib/sanitizer_common lib/ubsan lib/ubsan_minimal lib/xray lib/xray/tests/unit test/asan/TestCases ... X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/compiler-rt/dist: lib/asan lib/asan/tests lib/builtins lib/profile lib/sanitizer_common lib/ubsan lib/ubsan_minimal lib/xray lib/xray/tests/unit test/asan/TestCases test/fuzzer test/profile ... X-SVN-Commit-Revision: 337141 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:29 -0000 Author: dim Date: Thu Aug 2 17:33:19 2018 New Revision: 337141 URL: https://svnweb.freebsd.org/changeset/base/337141 Log: Vendor import of compiler-rt trunk r338536: https://llvm.org/svn/llvm-project/compiler-rt/trunk@338536 Added: vendor/compiler-rt/dist/test/fuzzer/ImplicitIntegerTruncationTest.cpp (contents, props changed) vendor/compiler-rt/dist/test/fuzzer/fuzzer-implicit-integer-truncation.test vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-fork.c (contents, props changed) vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-fork.c.gcov vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch1.c (contents, props changed) vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch1.c.gcov vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch2.c (contents, props changed) vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch2.c.gcov vendor/compiler-rt/dist/test/profile/Posix/ vendor/compiler-rt/dist/test/profile/Posix/instrprof-gcov-fork.test vendor/compiler-rt/dist/test/profile/instrprof-gcov-switch.test vendor/compiler-rt/dist/test/profile/instrprof-set-dir-mode.c (contents, props changed) vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitCast/ vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/ vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation-blacklist.c (contents, props changed) vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation-summary.cpp (contents, props changed) vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation.c (contents, props changed) vendor/compiler-rt/dist/test/ubsan_minimal/TestCases/implicit-integer-truncation.c (contents, props changed) Modified: vendor/compiler-rt/dist/lib/asan/asan_mapping.h vendor/compiler-rt/dist/lib/asan/tests/asan_test.cc vendor/compiler-rt/dist/lib/builtins/CMakeLists.txt vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.c vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.h vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mutex.h vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc vendor/compiler-rt/dist/lib/ubsan/ubsan_checks.inc vendor/compiler-rt/dist/lib/ubsan/ubsan_diag.h vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.cc vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.h vendor/compiler-rt/dist/lib/ubsan/ubsan_interface.inc vendor/compiler-rt/dist/lib/ubsan_minimal/ubsan_minimal_handlers.cc vendor/compiler-rt/dist/lib/xray/tests/unit/profile_collector_test.cc vendor/compiler-rt/dist/lib/xray/xray_buffer_queue.cc vendor/compiler-rt/dist/lib/xray/xray_profile_collector.cc vendor/compiler-rt/dist/lib/xray/xray_profiling.cc vendor/compiler-rt/dist/test/asan/TestCases/intercept-rethrow-exception.cc vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-multi-threaded.cc vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-single-threaded.cc Modified: vendor/compiler-rt/dist/lib/asan/asan_mapping.h ============================================================================== --- vendor/compiler-rt/dist/lib/asan/asan_mapping.h Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/asan/asan_mapping.h Thu Aug 2 17:33:19 2018 (r337141) @@ -122,7 +122,7 @@ // || `[0x400000000000, 0x47ffffffffff]` || LowShadow || // || `[0x000000000000, 0x3fffffffffff]` || LowMem || // -// Shadow mapping on NerBSD/i386 with SHADOW_OFFSET == 0x40000000: +// Shadow mapping on NetBSD/i386 with SHADOW_OFFSET == 0x40000000: // || `[0x60000000, 0xfffff000]` || HighMem || // || `[0x4c000000, 0x5fffffff]` || HighShadow || // || `[0x48000000, 0x4bffffff]` || ShadowGap || Modified: vendor/compiler-rt/dist/lib/asan/tests/asan_test.cc ============================================================================== --- vendor/compiler-rt/dist/lib/asan/tests/asan_test.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/asan/tests/asan_test.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -963,7 +963,7 @@ TEST(AddressSanitizer, ThreadNamesTest) { #if ASAN_NEEDS_SEGV TEST(AddressSanitizer, ShadowGapTest) { #if SANITIZER_WORDSIZE == 32 - char *addr = (char*)0x22000000; + char *addr = (char*)0x23000000; #else # if defined(__powerpc64__) char *addr = (char*)0x024000800000; Modified: vendor/compiler-rt/dist/lib/builtins/CMakeLists.txt ============================================================================== --- vendor/compiler-rt/dist/lib/builtins/CMakeLists.txt Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/builtins/CMakeLists.txt Thu Aug 2 17:33:19 2018 (r337141) @@ -567,6 +567,15 @@ else () set(_arch "arm") endif() + # For ARM archs, exclude any VFP builtins if VFP is not supported + if (${arch} MATCHES "^(arm|armhf|armv7|armv7s|armv7k|armv7m|armv7em)$") + string(REPLACE ";" " " _TARGET_${arch}_CFLAGS "${TARGET_${arch}_CFLAGS}") + check_compile_definition(__VFP_FP__ "${CMAKE_C_FLAGS} ${_TARGET_${arch}_CFLAGS}" COMPILER_RT_HAS_${arch}_VFP) + if(NOT COMPILER_RT_HAS_${arch}_VFP) + list(REMOVE_ITEM ${arch}_SOURCES ${arm_Thumb1_VFPv2_SOURCES} ${arm_Thumb1_SjLj_EH_SOURCES}) + endif() + endif() + # Filter out generic versions of routines that are re-implemented in # architecture specific manner. This prevents multiple definitions of the # same symbols, making the symbol selection non-deterministic. Modified: vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.c ============================================================================== --- vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.c Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.c Thu Aug 2 17:33:19 2018 (r337141) @@ -35,6 +35,8 @@ #include "InstrProfiling.h" #include "InstrProfilingUtil.h" +COMPILER_RT_WEAK unsigned lprofDirMode = 0755; + COMPILER_RT_VISIBILITY void __llvm_profile_recursive_mkdir(char *path) { int i; @@ -47,11 +49,18 @@ void __llvm_profile_recursive_mkdir(char *path) { #ifdef _WIN32 _mkdir(path); #else - mkdir(path, 0755); /* Some of these will fail, ignore it. */ + /* Some of these will fail, ignore it. */ + mkdir(path, __llvm_profile_get_dir_mode()); #endif path[i] = save; } } + +COMPILER_RT_VISIBILITY +void __llvm_profile_set_dir_mode(unsigned Mode) { lprofDirMode = Mode; } + +COMPILER_RT_VISIBILITY +unsigned __llvm_profile_get_dir_mode(void) { return lprofDirMode; } #if COMPILER_RT_HAS_ATOMICS != 1 COMPILER_RT_VISIBILITY Modified: vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.h ============================================================================== --- vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.h Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/profile/InstrProfilingUtil.h Thu Aug 2 17:33:19 2018 (r337141) @@ -16,6 +16,12 @@ /*! \brief Create a directory tree. */ void __llvm_profile_recursive_mkdir(char *Pathname); +/*! Set the mode used when creating profile directories. */ +void __llvm_profile_set_dir_mode(unsigned Mode); + +/*! Return the directory creation mode. */ +unsigned __llvm_profile_get_dir_mode(void); + int lprofLockFd(int fd); int lprofUnlockFd(int fd); Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mutex.h ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mutex.h Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mutex.h Thu Aug 2 17:33:19 2018 (r337141) @@ -73,13 +73,8 @@ class SpinMutex : public StaticSpinMutex { class BlockingMutex { public: -#if SANITIZER_WINDOWS - // Windows does not currently support LinkerInitialized - explicit BlockingMutex(LinkerInitialized); -#else explicit constexpr BlockingMutex(LinkerInitialized) - : opaque_storage_ {0, }, owner_(0) {} -#endif + : opaque_storage_ {0, }, owner_ {0} {} BlockingMutex(); void Lock(); void Unlock(); Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -767,43 +767,22 @@ void *internal_start_thread(void (*func)(void *arg), v void internal_join_thread(void *th) { } // ---------------------- BlockingMutex ---------------- {{{1 -const uptr LOCK_UNINITIALIZED = 0; -const uptr LOCK_READY = (uptr)-1; -BlockingMutex::BlockingMutex(LinkerInitialized li) { - // FIXME: see comments in BlockingMutex::Lock() for the details. - CHECK(li == LINKER_INITIALIZED || owner_ == LOCK_UNINITIALIZED); - - CHECK(sizeof(CRITICAL_SECTION) <= sizeof(opaque_storage_)); - InitializeCriticalSection((LPCRITICAL_SECTION)opaque_storage_); - owner_ = LOCK_READY; -} - BlockingMutex::BlockingMutex() { - CHECK(sizeof(CRITICAL_SECTION) <= sizeof(opaque_storage_)); - InitializeCriticalSection((LPCRITICAL_SECTION)opaque_storage_); - owner_ = LOCK_READY; + CHECK(sizeof(SRWLOCK) <= sizeof(opaque_storage_)); + internal_memset(this, 0, sizeof(*this)); } void BlockingMutex::Lock() { - if (owner_ == LOCK_UNINITIALIZED) { - // FIXME: hm, global BlockingMutex objects are not initialized?!? - // This might be a side effect of the clang+cl+link Frankenbuild... - new(this) BlockingMutex((LinkerInitialized)(LINKER_INITIALIZED + 1)); - - // FIXME: If it turns out the linker doesn't invoke our - // constructors, we should probably manually Lock/Unlock all the global - // locks while we're starting in one thread to avoid double-init races. - } - EnterCriticalSection((LPCRITICAL_SECTION)opaque_storage_); - CHECK_EQ(owner_, LOCK_READY); + AcquireSRWLockExclusive((PSRWLOCK)opaque_storage_); + CHECK_EQ(owner_, 0); owner_ = GetThreadSelf(); } void BlockingMutex::Unlock() { - CHECK_EQ(owner_, GetThreadSelf()); - owner_ = LOCK_READY; - LeaveCriticalSection((LPCRITICAL_SECTION)opaque_storage_); + CheckLocked(); + owner_ = 0; + ReleaseSRWLockExclusive((PSRWLOCK)opaque_storage_); } void BlockingMutex::CheckLocked() { Modified: vendor/compiler-rt/dist/lib/ubsan/ubsan_checks.inc ============================================================================== --- vendor/compiler-rt/dist/lib/ubsan/ubsan_checks.inc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/ubsan/ubsan_checks.inc Thu Aug 2 17:33:19 2018 (r337141) @@ -30,6 +30,8 @@ UBSAN_CHECK(IntegerDivideByZero, "integer-divide-by-ze "integer-divide-by-zero") UBSAN_CHECK(FloatDivideByZero, "float-divide-by-zero", "float-divide-by-zero") UBSAN_CHECK(InvalidBuiltin, "invalid-builtin-use", "invalid-builtin-use") +UBSAN_CHECK(ImplicitIntegerTruncation, "implicit-integer-truncation", + "implicit-integer-truncation") UBSAN_CHECK(InvalidShiftBase, "invalid-shift-base", "shift-base") UBSAN_CHECK(InvalidShiftExponent, "invalid-shift-exponent", "shift-exponent") UBSAN_CHECK(OutOfBoundsIndex, "out-of-bounds-index", "bounds") Modified: vendor/compiler-rt/dist/lib/ubsan/ubsan_diag.h ============================================================================== --- vendor/compiler-rt/dist/lib/ubsan/ubsan_diag.h Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/ubsan/ubsan_diag.h Thu Aug 2 17:33:19 2018 (r337141) @@ -178,7 +178,7 @@ class Diag { (public) }; private: - static const unsigned MaxArgs = 5; + static const unsigned MaxArgs = 8; static const unsigned MaxRanges = 1; /// The arguments which have been added to this diagnostic so far. Modified: vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.cc ============================================================================== --- vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -451,6 +451,49 @@ void __ubsan::__ubsan_handle_load_invalid_value_abort( Die(); } +static void handleImplicitConversion(ImplicitConversionData *Data, + ReportOptions Opts, ValueHandle Src, + ValueHandle Dst) { + SourceLocation Loc = Data->Loc.acquire(); + ErrorType ET = ErrorType::GenericUB; + + switch (Data->Kind) { + case ICCK_IntegerTruncation: + ET = ErrorType::ImplicitIntegerTruncation; + break; + } + + if (ignoreReport(Loc, Opts, ET)) + return; + + const TypeDescriptor &SrcTy = Data->FromType; + const TypeDescriptor &DstTy = Data->ToType; + + ScopedReport R(Opts, Loc, ET); + + // FIXME: is it possible to dump the values as hex with fixed width? + + Diag(Loc, DL_Error, ET, + "implicit conversion from type %0 of value %1 (%2-bit, %3signed) to " + "type %4 changed the value to %5 (%6-bit, %7signed)") + << SrcTy << Value(SrcTy, Src) << SrcTy.getIntegerBitWidth() + << (SrcTy.isSignedIntegerTy() ? "" : "un") << DstTy << Value(DstTy, Dst) + << DstTy.getIntegerBitWidth() << (DstTy.isSignedIntegerTy() ? "" : "un"); +} + +void __ubsan::__ubsan_handle_implicit_conversion(ImplicitConversionData *Data, + ValueHandle Src, + ValueHandle Dst) { + GET_REPORT_OPTIONS(false); + handleImplicitConversion(Data, Opts, Src, Dst); +} +void __ubsan::__ubsan_handle_implicit_conversion_abort( + ImplicitConversionData *Data, ValueHandle Src, ValueHandle Dst) { + GET_REPORT_OPTIONS(true); + handleImplicitConversion(Data, Opts, Src, Dst); + Die(); +} + static void handleInvalidBuiltin(InvalidBuiltinData *Data, ReportOptions Opts) { SourceLocation Loc = Data->Loc.acquire(); ErrorType ET = ErrorType::InvalidBuiltin; Modified: vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.h ============================================================================== --- vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.h Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/ubsan/ubsan_handlers.h Thu Aug 2 17:33:19 2018 (r337141) @@ -122,6 +122,23 @@ struct InvalidValueData { /// \brief Handle a load of an invalid value for the type. RECOVERABLE(load_invalid_value, InvalidValueData *Data, ValueHandle Val) +/// Known implicit conversion check kinds. +/// Keep in sync with the enum of the same name in CGExprScalar.cpp +enum ImplicitConversionCheckKind : unsigned char { + ICCK_IntegerTruncation = 0, +}; + +struct ImplicitConversionData { + SourceLocation Loc; + const TypeDescriptor &FromType; + const TypeDescriptor &ToType; + /* ImplicitConversionCheckKind */ unsigned char Kind; +}; + +/// \brief Implict conversion that changed the value. +RECOVERABLE(implicit_conversion, ImplicitConversionData *Data, ValueHandle Src, + ValueHandle Dst) + /// Known builtin check kinds. /// Keep in sync with the enum of the same name in CodeGenFunction.h enum BuiltinCheckKind : unsigned char { Modified: vendor/compiler-rt/dist/lib/ubsan/ubsan_interface.inc ============================================================================== --- vendor/compiler-rt/dist/lib/ubsan/ubsan_interface.inc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/ubsan/ubsan_interface.inc Thu Aug 2 17:33:19 2018 (r337141) @@ -22,6 +22,8 @@ INTERFACE_FUNCTION(__ubsan_handle_float_cast_overflow) INTERFACE_FUNCTION(__ubsan_handle_float_cast_overflow_abort) INTERFACE_FUNCTION(__ubsan_handle_function_type_mismatch) INTERFACE_FUNCTION(__ubsan_handle_function_type_mismatch_abort) +INTERFACE_FUNCTION(__ubsan_handle_implicit_conversion) +INTERFACE_FUNCTION(__ubsan_handle_implicit_conversion_abort) INTERFACE_FUNCTION(__ubsan_handle_invalid_builtin) INTERFACE_FUNCTION(__ubsan_handle_invalid_builtin_abort) INTERFACE_FUNCTION(__ubsan_handle_load_invalid_value) Modified: vendor/compiler-rt/dist/lib/ubsan_minimal/ubsan_minimal_handlers.cc ============================================================================== --- vendor/compiler-rt/dist/lib/ubsan_minimal/ubsan_minimal_handlers.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/ubsan_minimal/ubsan_minimal_handlers.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -109,6 +109,7 @@ HANDLER(float_cast_overflow, "float-cast-overflow") HANDLER(load_invalid_value, "load-invalid-value") HANDLER(invalid_builtin, "invalid-builtin") HANDLER(function_type_mismatch, "function-type-mismatch") +HANDLER(implicit_conversion, "implicit-conversion") HANDLER(nonnull_arg, "nonnull-arg") HANDLER(nonnull_return, "nonnull-return") HANDLER(nullability_arg, "nullability-arg") Modified: vendor/compiler-rt/dist/lib/xray/tests/unit/profile_collector_test.cc ============================================================================== --- vendor/compiler-rt/dist/lib/xray/tests/unit/profile_collector_test.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/xray/tests/unit/profile_collector_test.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -15,6 +15,8 @@ #include "xray_profile_collector.h" #include "xray_profiling_flags.h" #include +#include +#include #include #include #include @@ -24,6 +26,29 @@ namespace { static constexpr auto kHeaderSize = 16u; +constexpr uptr ExpectedProfilingVersion = 0x20180424; + +struct ExpectedProfilingFileHeader { + const u64 MagicBytes = 0x7872617970726f66; // Identifier for XRay profiling + // files 'xrayprof' in hex. + const u64 Version = ExpectedProfilingVersion; + u64 Timestamp = 0; + u64 PID = 0; +}; + +void ValidateFileHeaderBlock(XRayBuffer B) { + ASSERT_NE(static_cast(B.Data), nullptr); + ASSERT_EQ(B.Size, sizeof(ExpectedProfilingFileHeader)); + typename std::aligned_storage::type + FileHeaderStorage; + ExpectedProfilingFileHeader ExpectedHeader; + std::memcpy(&FileHeaderStorage, B.Data, B.Size); + auto &FileHeader = + *reinterpret_cast(&FileHeaderStorage); + ASSERT_EQ(ExpectedHeader.MagicBytes, FileHeader.MagicBytes); + ASSERT_EQ(ExpectedHeader.Version, FileHeader.Version); +} + void ValidateBlock(XRayBuffer B) { profilingFlags()->setDefaults(); ASSERT_NE(static_cast(B.Data), nullptr); @@ -107,9 +132,13 @@ TEST(profileCollectorServiceTest, PostSerializeCollect // Then we serialize the data. profileCollectorService::serialize(); - // Then we go through a single buffer to see whether we're getting the data we - // expect. + // Then we go through two buffers to see whether we're getting the data we + // expect. The first block must always be as large as a file header, which + // will have a fixed size. auto B = profileCollectorService::nextBuffer({nullptr, 0}); + ValidateFileHeaderBlock(B); + + B = profileCollectorService::nextBuffer(B); ValidateBlock(B); u32 BlockSize; u32 BlockNum; @@ -169,6 +198,9 @@ TEST(profileCollectorServiceTest, PostSerializeCollect // Ensure that we see two buffers. auto B = profileCollectorService::nextBuffer({nullptr, 0}); + ValidateFileHeaderBlock(B); + + B = profileCollectorService::nextBuffer(B); ValidateBlock(B); B = profileCollectorService::nextBuffer(B); Modified: vendor/compiler-rt/dist/lib/xray/xray_buffer_queue.cc ============================================================================== --- vendor/compiler-rt/dist/lib/xray/xray_buffer_queue.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/xray/xray_buffer_queue.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -13,17 +13,50 @@ // //===----------------------------------------------------------------------===// #include "xray_buffer_queue.h" -#include "sanitizer_common/sanitizer_allocator_internal.h" #include "sanitizer_common/sanitizer_common.h" #include "sanitizer_common/sanitizer_libc.h" +#include "sanitizer_common/sanitizer_posix.h" #include +#include +#ifndef MAP_NORESERVE +// no-op on NetBSD (at least), unsupported flag on FreeBSD +#define MAP_NORESERVE 0 +#endif + using namespace __xray; using namespace __sanitizer; +template static T *allocRaw(size_t N) { + // TODO: Report errors? + // We use MAP_NORESERVE on platforms where it's supported to ensure that the + // pages we're allocating for XRay never end up in pages that can be swapped + // in/out. We're doing this because for FDR mode, we want to ensure that + // writes to the buffers stay resident in memory to prevent XRay itself from + // causing swapping/thrashing. + // + // In the case when XRay pages cannot be swapped in/out or there's not enough + // RAM to back these pages, we're willing to cause a segmentation fault + // instead of introducing latency in the measurement. We assume here that + // there are enough pages that are swappable in/out outside of the buffers + // being used by FDR mode (which are bounded and configurable anyway) to allow + // us to keep using always-resident memory. + // + // TODO: Make this configurable? + void *A = reinterpret_cast( + internal_mmap(NULL, N * sizeof(T), PROT_WRITE | PROT_READ, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_NORESERVE, -1, 0)); + return (A == MAP_FAILED) ? nullptr : reinterpret_cast(A); +} + +template static void deallocRaw(T *ptr, size_t N) { + // TODO: Report errors? + if (ptr != nullptr) + internal_munmap(ptr, N); +} + template static T *initArray(size_t N) { - auto A = reinterpret_cast( - InternalAlloc(N * sizeof(T), nullptr, kCacheLineSize)); + auto A = allocRaw(N); if (A != nullptr) while (N > 0) new (A + (--N)) T(); @@ -42,19 +75,19 @@ BufferQueue::BufferQueue(size_t B, size_t N, bool &Suc // Clean up the buffers we've already allocated. for (auto B = Buffers, E = Buffers + BufferCount; B != E; ++B) B->~BufferRep(); - InternalFree(Buffers); + deallocRaw(Buffers, N); Success = false; return; }; for (size_t i = 0; i < N; ++i) { auto &T = Buffers[i]; - void *Tmp = InternalAlloc(BufferSize, nullptr, 64); + void *Tmp = allocRaw(BufferSize); if (Tmp == nullptr) { Success = false; return; } - void *Extents = InternalAlloc(sizeof(BufferExtents), nullptr, 64); + auto *Extents = allocRaw(1); if (Extents == nullptr) { Success = false; return; @@ -62,7 +95,7 @@ BufferQueue::BufferQueue(size_t B, size_t N, bool &Suc auto &Buf = T.Buff; Buf.Data = Tmp; Buf.Size = B; - Buf.Extents = reinterpret_cast(Extents); + Buf.Extents = Extents; OwnedBuffers[i] = Tmp; } Success = true; @@ -128,11 +161,11 @@ BufferQueue::~BufferQueue() { for (auto I = Buffers, E = Buffers + BufferCount; I != E; ++I) { auto &T = *I; auto &Buf = T.Buff; - InternalFree(Buf.Data); - InternalFree(Buf.Extents); + deallocRaw(Buf.Data, Buf.Size); + deallocRaw(Buf.Extents, 1); } for (auto B = Buffers, E = Buffers + BufferCount; B != E; ++B) B->~BufferRep(); - InternalFree(Buffers); - InternalFree(OwnedBuffers); + deallocRaw(Buffers, BufferCount); + deallocRaw(OwnedBuffers, BufferCount); } Modified: vendor/compiler-rt/dist/lib/xray/xray_profile_collector.cc ============================================================================== --- vendor/compiler-rt/dist/lib/xray/xray_profile_collector.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/xray/xray_profile_collector.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -37,6 +37,19 @@ struct ProfileBuffer { size_t Size; }; +// Current version of the profile format. +constexpr u64 XRayProfilingVersion = 0x20180424; + +// Identifier for XRay profiling files 'xrayprof' in hex. +constexpr u64 XRayMagicBytes = 0x7872617970726f66; + +struct XRayProfilingFileHeader { + const u64 MagicBytes = XRayMagicBytes; + const u64 Version = XRayProfilingVersion; + u64 Timestamp = 0; // System time in nanoseconds. + u64 PID = 0; // Process ID. +}; + struct BlockHeader { u32 BlockSize; u32 BlockNum; @@ -302,7 +315,22 @@ XRayBuffer nextBuffer(XRayBuffer B) { if (ProfileBuffers == nullptr || ProfileBuffers->Size() == 0) return {nullptr, 0}; - if (B.Data == nullptr) + static pthread_once_t Once = PTHREAD_ONCE_INIT; + static typename std::aligned_storage::type + FileHeaderStorage; + pthread_once(&Once, + +[] { new (&FileHeaderStorage) XRayProfilingFileHeader{}; }); + + if (UNLIKELY(B.Data == nullptr)) { + // The first buffer should always contain the file header information. + auto &FileHeader = + *reinterpret_cast(&FileHeaderStorage); + FileHeader.Timestamp = NanoTime(); + FileHeader.PID = internal_getpid(); + return {&FileHeaderStorage, sizeof(XRayProfilingFileHeader)}; + } + + if (UNLIKELY(B.Data == &FileHeaderStorage)) return {(*ProfileBuffers)[0].Data, (*ProfileBuffers)[0].Size}; BlockHeader Header; Modified: vendor/compiler-rt/dist/lib/xray/xray_profiling.cc ============================================================================== --- vendor/compiler-rt/dist/lib/xray/xray_profiling.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/lib/xray/xray_profiling.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -32,16 +32,6 @@ namespace __xray { namespace { -constexpr uptr XRayProfilingVersion = 0x20180424; - -struct XRayProfilingFileHeader { - const u64 MagicBytes = 0x7872617970726f66; // Identifier for XRay profiling - // files 'xrayprof' in hex. - const uptr Version = XRayProfilingVersion; - uptr Timestamp = 0; // System time in nanoseconds. - uptr PID = 0; // Process ID. -}; - atomic_sint32_t ProfilerLogFlushStatus = { XRayLogFlushStatus::XRAY_LOG_NOT_FLUSHING}; @@ -144,14 +134,7 @@ XRayLogFlushStatus profilingFlush() XRAY_NEVER_INSTRUM if (Verbosity()) Report("profiling: Failed to flush to file, dropping data.\n"); } else { - XRayProfilingFileHeader Header; - Header.Timestamp = NanoTime(); - Header.PID = internal_getpid(); - retryingWriteAll(Fd, reinterpret_cast(&Header), - reinterpret_cast(&Header) + - sizeof(Header)); - - // Now for each of the threads, write out the profile data as we would + // Now for each of the buffers, write out the profile data as we would // see it in memory, verbatim. while (B.Data != nullptr && B.Size != 0) { retryingWriteAll(Fd, reinterpret_cast(B.Data), Modified: vendor/compiler-rt/dist/test/asan/TestCases/intercept-rethrow-exception.cc ============================================================================== --- vendor/compiler-rt/dist/test/asan/TestCases/intercept-rethrow-exception.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/test/asan/TestCases/intercept-rethrow-exception.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -1,7 +1,7 @@ // Regression test for // https://bugs.llvm.org/show_bug.cgi?id=32434 -// RUN: %clangxx_asan -O0 %s -o %t +// RUN: %clangxx_asan -fexceptions -O0 %s -o %t // RUN: %run %t #include Added: vendor/compiler-rt/dist/test/fuzzer/ImplicitIntegerTruncationTest.cpp ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/fuzzer/ImplicitIntegerTruncationTest.cpp Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,27 @@ +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. + +// Test for signed-integer-overflow. +#include +#include +#include +#include +#include +#include + +static volatile int Sink; +static unsigned char Large = UINT8_MAX; + +extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) { + assert(Data); + if (Size > 0 && Data[0] == 'H') { + Sink = 1; + if (Size > 1 && Data[1] == 'i') { + Sink = 2; + if (Size > 2 && Data[2] == '!') { + Large = Large + 1; // 'char overflow'. + } + } + } + return 0; +} Added: vendor/compiler-rt/dist/test/fuzzer/fuzzer-implicit-integer-truncation.test ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/fuzzer/fuzzer-implicit-integer-truncation.test Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,5 @@ +RUN: rm -f %t-ImplicitIntegerTruncationTest-Ubsan +RUN: %cpp_compiler -fsanitize=implicit-integer-truncation -fno-sanitize-recover=all %S/ImplicitIntegerTruncationTest.cpp -o %t-ImplicitIntegerTruncationTest-Ubsan +RUN: not %run %t-ImplicitIntegerTruncationTest-Ubsan 2>&1 | FileCheck %s +CHECK: runtime error: implicit conversion from type 'int' of value 256 (32-bit, signed) to type 'unsigned char' changed the value to 0 (8-bit, unsigned) +CHECK: Test unit written to ./crash- Added: vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-fork.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-fork.c Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,15 @@ +#include + +void func1() {} +void func2() {} + +int main(void) +{ + func1(); + + fork(); + + func2(); + + return 0; +} Added: vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-fork.c.gcov ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-fork.c.gcov Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,23 @@ +// CHECK: -: 0:Source:{{.*}}Inputs/instrprof-gcov-fork.c +// CHECK-NEXT: -: 0:Graph:instrprof-gcov-fork.gcno +// CHECK-NEXT: -: 0:Data:instrprof-gcov-fork.gcda +// CHECK-NEXT: -: 0:Runs:1 +// CHECK-NEXT: -: 0:Programs:1 +// CHECK-NEXT: -: 1:#include +// CHECK-NEXT: -: 2: +// CHECK-NEXT:function func1 called 1 returned 100% blocks executed 100% +// CHECK-NEXT: 1: 3:void func1() {} +// CHECK-NEXT:function func2 called 2 returned 100% blocks executed 100% +// CHECK-NEXT: 2: 4:void func2() {} +// CHECK-NEXT: -: 5: +// CHECK-NEXT:function main called 1 returned 100% blocks executed 100% +// CHECK-NEXT: -: 6:int main(void) +// CHECK-NEXT: -: 7:{ +// CHECK-NEXT: 1: 8: func1(); +// CHECK-NEXT: -: 9: +// CHECK-NEXT: 1: 10: fork(); +// CHECK-NEXT: -: 11: +// CHECK-NEXT: 2: 12: func2(); +// CHECK-NEXT: -: 13: +// CHECK-NEXT: 2: 14: return 0; +// CHECK-NEXT: -: 15:} Added: vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch1.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch1.c Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,18 @@ +int main(void) +{ + int i = 22; + + switch (i) { + case 7: + break; + + case 22: + i = 7; + break; + + case 42: + break; + } + + return 0; +} Added: vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch1.c.gcov ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch1.c.gcov Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,23 @@ +// CHECK: -: 0:Source:{{.*}}Inputs/instrprof-gcov-switch1.c +// CHECK-NEXT: -: 0:Graph:instrprof-gcov-switch1.gcno +// CHECK-NEXT: -: 0:Data:instrprof-gcov-switch1.gcda +// CHECK-NEXT: -: 0:Runs:1 +// CHECK-NEXT: -: 0:Programs:1 +// CHECK-NEXT: -: 1:int main(void) +// CHECK-NEXT: -: 2:{ +// CHECK-NEXT: 2: 3: int i = 22; +// CHECK-NEXT: -: 4: +// CHECK-NEXT: 2: 5: switch (i) { +// CHECK-NEXT: -: 6: case 7: +// CHECK-NEXT: #####: 7: break; +// CHECK-NEXT: -: 8: +// CHECK-NEXT: -: 9: case 22: +// CHECK-NEXT: 1: 10: i = 7; +// CHECK-NEXT: 1: 11: break; +// CHECK-NEXT: -: 12: +// CHECK-NEXT: -: 13: case 42: +// CHECK-NEXT: #####: 14: break; +// CHECK-NEXT: -: 15: } +// CHECK-NEXT: -: 16: +// CHECK-NEXT: 1: 17: return 0; +// CHECK-NEXT: -: 18:} Added: vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch2.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch2.c Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,18 @@ +int main(void) +{ + int i = 22; + + switch (i) { + case 7: + break; + + case 22: + i = 7; + + case 42: + i = 22; + break; + } + + return 0; +} Added: vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch2.c.gcov ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/Inputs/instrprof-gcov-switch2.c.gcov Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,23 @@ +// CHECK: -: 0:Source:{{.*}}Inputs/instrprof-gcov-switch2.c +// CHECK-NEXT: -: 0:Graph:instrprof-gcov-switch2.gcno +// CHECK-NEXT: -: 0:Data:instrprof-gcov-switch2.gcda +// CHECK-NEXT: -: 0:Runs:1 +// CHECK-NEXT: -: 0:Programs:1 +// CHECK-NEXT: -: 1:int main(void) +// CHECK-NEXT: -: 2:{ +// CHECK-NEXT: 3: 3: int i = 22; +// CHECK-NEXT: -: 4: +// CHECK-NEXT: 3: 5: switch (i) { +// CHECK-NEXT: -: 6: case 7: +// CHECK-NEXT: #####: 7: break; +// CHECK-NEXT: -: 8: +// CHECK-NEXT: -: 9: case 22: +// CHECK-NEXT: 1: 10: i = 7; +// CHECK-NEXT: -: 11: +// CHECK-NEXT: -: 12: case 42: +// CHECK-NEXT: 1: 13: i = 22; +// CHECK-NEXT: 1: 14: break; +// CHECK-NEXT: -: 15: } +// CHECK-NEXT: -: 16: +// CHECK-NEXT: 1: 17: return 0; +// CHECK-NEXT: -: 18:} Added: vendor/compiler-rt/dist/test/profile/Posix/instrprof-gcov-fork.test ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/Posix/instrprof-gcov-fork.test Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,12 @@ +XFAIL: * + +RUN: mkdir -p %t.d +RUN: cd %t.d + +RUN: %clang --coverage -o %t %S/../Inputs/instrprof-gcov-fork.c +RUN: test -f instrprof-gcov-fork.gcno + +RUN: rm -f instrprof-gcov-fork.gcda +RUN: %run %t +RUN: llvm-cov gcov -b -c instrprof-gcov-fork.gcda +RUN: FileCheck --match-full-lines --strict-whitespace --input-file instrprof-gcov-fork.c.gcov %S/../Inputs/instrprof-gcov-fork.c.gcov Added: vendor/compiler-rt/dist/test/profile/instrprof-gcov-switch.test ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/instrprof-gcov-switch.test Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,16 @@ +RUN: mkdir -p %t.d +RUN: cd %t.d + +RUN: %clang --coverage -o %t %S/Inputs/instrprof-gcov-switch1.c +RUN: test -f instrprof-gcov-switch1.gcno +RUN: rm -f instrprof-gcov-switch1.gcda +RUN: %run %t +RUN: llvm-cov gcov instrprof-gcov-switch1.gcda +RUN: FileCheck --match-full-lines --strict-whitespace --input-file instrprof-gcov-switch1.c.gcov %S/Inputs/instrprof-gcov-switch1.c.gcov + +RUN: %clang --coverage -o %t %S/Inputs/instrprof-gcov-switch2.c +RUN: test -f instrprof-gcov-switch2.gcno +RUN: rm -f instrprof-gcov-switch2.gcda +RUN: %run %t +RUN: llvm-cov gcov instrprof-gcov-switch2.gcda +RUN: FileCheck --match-full-lines --strict-whitespace --input-file instrprof-gcov-switch2.c.gcov %S/Inputs/instrprof-gcov-switch2.c.gcov Added: vendor/compiler-rt/dist/test/profile/instrprof-set-dir-mode.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/profile/instrprof-set-dir-mode.c Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,48 @@ +// UNSUPPORTED: windows +// RUN: %clang_pgogen -o %t.bin %s -DTESTPATH=\"%t.dir\" +// RUN: rm -rf %t.dir +// RUN: %run %t.bin + +#include +#include +#include +#include + +void __llvm_profile_set_dir_mode(unsigned Mode); +unsigned __llvm_profile_get_dir_mode(void); +void __llvm_profile_recursive_mkdir(char *Path); + +static int test(unsigned Mode, const char *TestDir) { + int Ret = 0; + + /* Create a dir and set the mode accordingly. */ + char *Dir = strdup(TestDir); + if (!Dir) + return -1; + __llvm_profile_set_dir_mode(Mode); + __llvm_profile_recursive_mkdir(Dir); + + if (Mode != __llvm_profile_get_dir_mode()) + Ret = -1; + else { + const unsigned Expected = ~umask(0) & Mode; + struct stat DirSt; + if (stat(Dir, &DirSt) == -1) + Ret = -1; + else if (DirSt.st_mode != Expected) { + printf("Modes do not match: Expected %o but found %o (%s)\n", Expected, + DirSt.st_mode, Dir); + Ret = -1; + } + } + + free(Dir); + return Ret; +} + +int main(void) { + if (test(S_IFDIR | 0777, TESTPATH "/foo/bar/baz/") || + test(S_IFDIR | 0666, TESTPATH "/foo/bar/qux/")) + return -1; + return 0; +} Added: vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation-blacklist.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation-blacklist.c Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,20 @@ +// FIXME: https://code.google.com/p/address-sanitizer/issues/detail?id=316 +// I'm not sure this is actually *that* issue, but this seems oddly similar to the other XFAIL'ed cases. +// XFAIL: android +// UNSUPPORTED: ios + +// RUN: rm -f %tmp +// RUN: echo "[implicit-integer-truncation]" >> %tmp +// RUN: echo "fun:*implicitTruncation*" >> %tmp +// RUN: %clang -fsanitize=implicit-integer-truncation -fno-sanitize-recover=implicit-integer-truncation -fsanitize-blacklist=%tmp -O0 %s -o %t && not %run %t 2>&1 +// RUN: %clang -fsanitize=implicit-integer-truncation -fno-sanitize-recover=implicit-integer-truncation -fsanitize-blacklist=%tmp -O1 %s -o %t && not %run %t 2>&1 +// RUN: %clang -fsanitize=implicit-integer-truncation -fno-sanitize-recover=implicit-integer-truncation -fsanitize-blacklist=%tmp -O2 %s -o %t && not %run %t 2>&1 +// RUN: %clang -fsanitize=implicit-integer-truncation -fno-sanitize-recover=implicit-integer-truncation -fsanitize-blacklist=%tmp -O3 %s -o %t && not %run %t 2>&1 + +unsigned char implicitTruncation(unsigned int argc) { + return argc; // BOOM +} + +int main(int argc, char **argv) { + return implicitTruncation(~0U); +} Added: vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation-summary.cpp ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation-summary.cpp Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,13 @@ +// RUN: %clangxx -fsanitize=implicit-integer-truncation %s -o %t +// RUN: %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-NOTYPE +// RUN: %env_ubsan_opts=report_error_type=1 %run %t 2>&1 | FileCheck %s --check-prefix=CHECK-TYPE +// REQUIRES: !ubsan-standalone && !ubsan-standalone-static + +#include + +int main() { + uint8_t t0 = (~(uint32_t(0))); + // CHECK-NOTYPE: SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior {{.*}}summary.cpp:[[@LINE-1]]:16 + // CHECK-TYPE: SUMMARY: UndefinedBehaviorSanitizer: implicit-integer-truncation {{.*}}summary.cpp:[[@LINE-2]]:16 + return 0; +} Added: vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/ubsan/TestCases/ImplicitConversion/integer-truncation.c Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,63 @@ +// RUN: %clang -x c -fsanitize=implicit-integer-truncation %s -o %t && %run %t 2>&1 | FileCheck %s --check-prefixes=CHECK +// RUN: %clangxx -x c++ -fsanitize=implicit-integer-truncation %s -o %t && %run %t 2>&1 | FileCheck %s --check-prefixes=CHECK + +#include + +#if !defined(__cplusplus) +#define bool _Bool +#endif + +int main() { +// CHECK-NOT: integer-truncation.c + + // Negative tests. Even if they produce unexpected results, this sanitizer does not care. + int8_t n0 = (~((uint32_t)0)); // ~0 -> -1, but do not warn. + uint8_t n2 = 128; + uint8_t n3 = 255; + // Bools do not count + bool b0 = (~((uint32_t)0)); + bool b1 = 255; + + // Explicit and-ing of bits will silence it. + uint8_t nc0 = (~((uint32_t)0)) & 255; + + // Explicit casts + uint8_t i0 = (uint8_t)(~((uint32_t)0)); + +#if defined(__cplusplus) + uint8_t i1 = uint8_t(~(uint32_t(0))); + uint8_t i2 = static_cast(~(uint32_t(0))); +#endif + + // Positive tests. + + uint8_t t_b0 = (~((uint16_t)(0))); +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:18: runtime error: implicit conversion from type 'int' of value -1 (32-bit, signed) to type 'uint8_t' (aka 'unsigned char') changed the value to 255 (8-bit, unsigned) + + uint8_t t_b1 = (~((uint32_t)0)); +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:18: runtime error: implicit conversion from type 'uint32_t' (aka 'unsigned int') of value 4294967295 (32-bit, unsigned) to type 'uint8_t' (aka 'unsigned char') changed the value to 255 (8-bit, unsigned) + uint16_t t_b2 = (~((uint32_t)0)); +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:19: runtime error: implicit conversion from type 'uint32_t' (aka 'unsigned int') of value 4294967295 (32-bit, unsigned) to type 'uint16_t' (aka 'unsigned short') changed the value to 65535 (16-bit, unsigned) + + uint8_t t_b3 = ~((uint64_t)0); +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:18: runtime error: implicit conversion from type 'uint64_t' (aka 'unsigned long{{[^']*}}') of value 18446744073709551615 (64-bit, unsigned) to type 'uint8_t' (aka 'unsigned char') changed the value to 255 (8-bit, unsigned) + uint16_t t_b4 = ~((uint64_t)0); +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:19: runtime error: implicit conversion from type 'uint64_t' (aka 'unsigned long{{[^']*}}') of value 18446744073709551615 (64-bit, unsigned) to type 'uint16_t' (aka 'unsigned short') changed the value to 65535 (16-bit, unsigned) + uint32_t t_b5 = ~((uint64_t)0); +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:19: runtime error: implicit conversion from type 'uint64_t' (aka 'unsigned long{{[^']*}}') of value 18446744073709551615 (64-bit, unsigned) to type 'uint32_t' (aka 'unsigned int') changed the value to 4294967295 (32-bit, unsigned) + + int8_t t1 = 255; +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:15: runtime error: implicit conversion from type 'int' of value 255 (32-bit, signed) to type 'int8_t' (aka 'signed char') changed the value to -1 (8-bit, signed) + uint8_t t2 = 256; +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:16: runtime error: implicit conversion from type 'int' of value 256 (32-bit, signed) to type 'uint8_t' (aka 'unsigned char') changed the value to 0 (8-bit, unsigned) + int8_t t3 = 256; +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:15: runtime error: implicit conversion from type 'int' of value 256 (32-bit, signed) to type 'int8_t' (aka 'signed char') changed the value to 0 (8-bit, signed) + uint8_t t4 = 257; +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:16: runtime error: implicit conversion from type 'int' of value 257 (32-bit, signed) to type 'uint8_t' (aka 'unsigned char') changed the value to 1 (8-bit, unsigned) + int8_t t5 = 257; +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:15: runtime error: implicit conversion from type 'int' of value 257 (32-bit, signed) to type 'int8_t' (aka 'signed char') changed the value to 1 (8-bit, signed) + int8_t t6 = 128; +// CHECK: {{.*}}integer-truncation.c:[[@LINE-1]]:15: runtime error: implicit conversion from type 'int' of value 128 (32-bit, signed) to type 'int8_t' (aka 'signed char') changed the value to -128 (8-bit, signed) + + return 0; +} Added: vendor/compiler-rt/dist/test/ubsan_minimal/TestCases/implicit-integer-truncation.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/compiler-rt/dist/test/ubsan_minimal/TestCases/implicit-integer-truncation.c Thu Aug 2 17:33:19 2018 (r337141) @@ -0,0 +1,24 @@ +// RUN: %clang -fsanitize=implicit-integer-truncation %s -o %t && %run %t 2>&1 | FileCheck %s --check-prefixes=CHECK + +#include + +int main() { +// CHECK-NOT: integer-truncation.c + + // Negative tests. Even if they produce unexpected results, this sanitizer does not care. + int8_t n0 = (~((uint32_t)(0))); // ~0 -> -1, but do not warn. + uint8_t n2 = 128; + uint8_t n3 = 255; + // Bools do not count + _Bool b0 = (~((uint32_t)(0))); + _Bool b1 = 255; + + // Explicit and-ing of bits will silence it. + uint8_t nc0 = ((~((uint32_t)(0))) & 255); + + // Positive tests. + uint8_t t0 = (~((uint32_t)(0))); +// CHECK: implicit-conversion + + return 0; +} Modified: vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-multi-threaded.cc ============================================================================== --- vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-multi-threaded.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-multi-threaded.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -51,7 +51,8 @@ volatile int buffer_counter = 0; assert(__xray_log_finalize() == XRayLogInitStatus::XRAY_LOG_FINALIZED); assert(__xray_log_process_buffers(process_buffer) == XRayLogFlushStatus::XRAY_LOG_FLUSHED); - // We're running three threds, so we expect three buffers. - assert(buffer_counter == 3); + // We're running three threads, so we expect four buffers (including the file + // header buffer). + assert(buffer_counter == 4); assert(__xray_log_flushLog() == XRayLogFlushStatus::XRAY_LOG_FLUSHED); } Modified: vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-single-threaded.cc ============================================================================== --- vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-single-threaded.cc Thu Aug 2 17:33:16 2018 (r337140) +++ vendor/compiler-rt/dist/test/xray/TestCases/Posix/profiling-single-threaded.cc Thu Aug 2 17:33:19 2018 (r337141) @@ -47,7 +47,10 @@ volatile int buffer_counter = 0; f0(); assert(__xray_log_process_buffers(process_buffer) == XRayLogFlushStatus::XRAY_LOG_FLUSHED); - assert(buffer_counter == 1); + // There's always at least one buffer, containing the profile file header. We + // assert that we have two, to indicate that we're expecting exactly one + // thread's worth of data. + assert(buffer_counter == 2); assert(__xray_log_flushLog() == XRayLogFlushStatus::XRAY_LOG_FLUSHED); // Let's reset the counter. @@ -60,6 +63,6 @@ volatile int buffer_counter = 0; f0(); assert(__xray_log_process_buffers(process_buffer) == XRayLogFlushStatus::XRAY_LOG_FLUSHED); - assert(buffer_counter == 1); + assert(buffer_counter == 2); assert(__xray_log_flushLog() == XRayLogFlushStatus::XRAY_LOG_FLUSHED); } From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:30 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CECC2106D760; Thu, 2 Aug 2018 17:33:30 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 825D672521; Thu, 2 Aug 2018 17:33:30 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 484E014D33; Thu, 2 Aug 2018 17:33:30 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXUbr029506; Thu, 2 Aug 2018 17:33:30 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXUB1029505; Thu, 2 Aug 2018 17:33:30 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXUB1029505@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337142 - vendor/compiler-rt/compiler-rt-trunk-r338536 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/compiler-rt/compiler-rt-trunk-r338536 X-SVN-Commit-Revision: 337142 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:31 -0000 Author: dim Date: Thu Aug 2 17:33:29 2018 New Revision: 337142 URL: https://svnweb.freebsd.org/changeset/base/337142 Log: Tag compiler-rt trunk r338536. Added: vendor/compiler-rt/compiler-rt-trunk-r338536/ - copied from r337141, vendor/compiler-rt/dist/ From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:38 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71045106D7AE; Thu, 2 Aug 2018 17:33:38 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B68F7265F; Thu, 2 Aug 2018 17:33:36 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2685814D34; Thu, 2 Aug 2018 17:33:36 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXaiQ029565; Thu, 2 Aug 2018 17:33:36 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXY3u029552; Thu, 2 Aug 2018 17:33:34 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXY3u029552@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337143 - in vendor/libc++/dist: include include/experimental include/support/newlib lib/abi src test/libcxx test/libcxx/language.support test/std/containers/associative/map/map.modifie... X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/libc++/dist: include include/experimental include/support/newlib lib/abi src test/libcxx test/libcxx/language.support test/std/containers/associative/map/map.modifiers test/std/containers/as... X-SVN-Commit-Revision: 337143 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:38 -0000 Author: dim Date: Thu Aug 2 17:33:33 2018 New Revision: 337143 URL: https://svnweb.freebsd.org/changeset/base/337143 Log: Vendor import of libc++ trunk r338536: https://llvm.org/svn/llvm-project/libcxx/trunk@338536 Added: vendor/libc++/dist/include/__node_handle vendor/libc++/dist/include/charconv vendor/libc++/dist/src/charconv.cpp (contents, props changed) vendor/libc++/dist/test/libcxx/language.support/has_c11_features.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/map/map.modifiers/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/map/map.modifiers/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/map/map.modifiers/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/map/map.modifiers/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multimap/multimap.modifiers/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multimap/multimap.modifiers/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multimap/multimap.modifiers/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multimap/multimap.modifiers/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multiset/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multiset/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multiset/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/multiset/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/set/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/set/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/set/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/associative/set/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/container.node/ vendor/libc++/dist/test/std/containers/container.node/node_handle.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.map/unord.map.modifiers/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.map/unord.map.modifiers/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.map/unord.map.modifiers/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.map/unord.map.modifiers/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multimap/unord.multimap.modifiers/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multimap/unord.multimap.modifiers/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multimap/unord.multimap.modifiers/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multimap/unord.multimap.modifiers/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multiset/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multiset/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multiset/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.multiset/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.set/extract_iterator.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.set/extract_key.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.set/insert_node_type.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/containers/unord/unord.set/insert_node_type_hint.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/experimental/simd/simd.abi/ vendor/libc++/dist/test/std/experimental/simd/simd.abi/vector_extension.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/experimental/simd/simd.access/ vendor/libc++/dist/test/std/experimental/simd/simd.access/default.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/experimental/simd/simd.cons/default.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/experimental/simd/simd.cons/load.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/experimental/simd/simd.mem/ vendor/libc++/dist/test/std/experimental/simd/simd.mem/load.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/experimental/simd/simd.mem/store.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/utilities/charconv/ vendor/libc++/dist/test/std/utilities/charconv/charconv.from.chars/ vendor/libc++/dist/test/std/utilities/charconv/charconv.from.chars/integral.bool.fail.cpp (contents, props changed) vendor/libc++/dist/test/std/utilities/charconv/charconv.from.chars/integral.pass.cpp (contents, props changed) vendor/libc++/dist/test/std/utilities/charconv/charconv.to.chars/ vendor/libc++/dist/test/std/utilities/charconv/charconv.to.chars/integral.bool.fail.cpp (contents, props changed) vendor/libc++/dist/test/std/utilities/charconv/charconv.to.chars/integral.pass.cpp (contents, props changed) vendor/libc++/dist/test/support/charconv_test_helpers.h (contents, props changed) Modified: vendor/libc++/dist/include/CMakeLists.txt vendor/libc++/dist/include/__config vendor/libc++/dist/include/__functional_base vendor/libc++/dist/include/__hash_table vendor/libc++/dist/include/__mutex_base vendor/libc++/dist/include/__tree vendor/libc++/dist/include/cfloat vendor/libc++/dist/include/ctime vendor/libc++/dist/include/experimental/__config vendor/libc++/dist/include/experimental/simd vendor/libc++/dist/include/float.h vendor/libc++/dist/include/functional vendor/libc++/dist/include/map vendor/libc++/dist/include/memory vendor/libc++/dist/include/module.modulemap vendor/libc++/dist/include/new vendor/libc++/dist/include/set vendor/libc++/dist/include/shared_mutex vendor/libc++/dist/include/support/newlib/xlocale.h vendor/libc++/dist/include/system_error vendor/libc++/dist/include/unordered_map vendor/libc++/dist/include/unordered_set vendor/libc++/dist/include/utility vendor/libc++/dist/lib/abi/x86_64-unknown-linux-gnu.v1.abilist vendor/libc++/dist/src/bind.cpp vendor/libc++/dist/src/future.cpp vendor/libc++/dist/src/memory.cpp vendor/libc++/dist/src/mutex.cpp vendor/libc++/dist/src/new.cpp vendor/libc++/dist/src/shared_mutex.cpp vendor/libc++/dist/src/system_error.cpp vendor/libc++/dist/src/utility.cpp vendor/libc++/dist/test/libcxx/double_include.sh.cpp vendor/libc++/dist/test/std/depr/depr.c.headers/float_h.pass.cpp vendor/libc++/dist/test/std/depr/depr.c.headers/stdlib_h.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.casts/simd_cast.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.casts/static_simd_cast.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.cons/broadcast.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.cons/generator.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.traits/abi_for_size.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.traits/is_abi_tag.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.traits/is_simd.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.traits/is_simd_flag_type.pass.cpp vendor/libc++/dist/test/std/experimental/simd/simd.traits/is_simd_mask.pass.cpp vendor/libc++/dist/test/std/language.support/support.limits/c.limits/cfloat.pass.cpp vendor/libc++/dist/test/std/language.support/support.runtime/cstdlib.pass.cpp vendor/libc++/dist/test/std/language.support/support.runtime/ctime.pass.cpp vendor/libc++/dist/test/std/utilities/time/date.time/ctime.pass.cpp vendor/libc++/dist/test/support/Counter.h vendor/libc++/dist/test/support/test_macros.h vendor/libc++/dist/www/cxx1z_status.html Modified: vendor/libc++/dist/include/CMakeLists.txt ============================================================================== --- vendor/libc++/dist/include/CMakeLists.txt Thu Aug 2 17:33:29 2018 (r337142) +++ vendor/libc++/dist/include/CMakeLists.txt Thu Aug 2 17:33:33 2018 (r337143) @@ -11,6 +11,7 @@ set(files __libcpp_version __locale __mutex_base + __node_handle __nullptr __split_buffer __sso_allocator @@ -31,6 +32,7 @@ set(files cerrno cfenv cfloat + charconv chrono cinttypes ciso646 Modified: vendor/libc++/dist/include/__config ============================================================================== --- vendor/libc++/dist/include/__config Thu Aug 2 17:33:29 2018 (r337142) +++ vendor/libc++/dist/include/__config Thu Aug 2 17:33:33 2018 (r337143) @@ -510,7 +510,7 @@ namespace std { #define _LIBCPP_HAS_IS_BASE_OF #endif -#if !__EXCEPTIONS +#if !__EXCEPTIONS && !defined(_LIBCPP_NO_EXCEPTIONS) #define _LIBCPP_NO_EXCEPTIONS #endif @@ -620,6 +620,8 @@ namespace std { #define _LIBCPP_ALWAYS_INLINE __forceinline +#define _LIBCPP_HAS_NO_VECTOR_EXTENSION + #elif defined(_LIBCPP_COMPILER_IBM) #define _ALIGNAS(x) __attribute__((__aligned__(x))) @@ -651,6 +653,8 @@ namespace std { #define _LIBCPP_HAS_NO_ASAN #define _LIBCPP_ALWAYS_INLINE __attribute__ ((__always_inline__)) + +#define _LIBCPP_HAS_NO_VECTOR_EXTENSION #endif // _LIBCPP_COMPILER_[CLANG|GCC|MSVC|IBM] Modified: vendor/libc++/dist/include/__functional_base ============================================================================== --- vendor/libc++/dist/include/__functional_base Thu Aug 2 17:33:29 2018 (r337142) +++ vendor/libc++/dist/include/__functional_base Thu Aug 2 17:33:33 2018 (r337143) @@ -561,7 +561,7 @@ struct __is_transparent<_Tp, _Up, struct _LIBCPP_TEMPLATE_VIS allocator_arg_t { }; -#if defined(_LIBCPP_CXX03_LANG) || defined(_LIBCPP_BUILDING_MEMORY) +#if defined(_LIBCPP_CXX03_LANG) || defined(_LIBCPP_BUILDING_LIBRARY) extern const allocator_arg_t allocator_arg; #else /* _LIBCPP_INLINE_VAR */ constexpr allocator_arg_t allocator_arg = allocator_arg_t(); Modified: vendor/libc++/dist/include/__hash_table ============================================================================== --- vendor/libc++/dist/include/__hash_table Thu Aug 2 17:33:29 2018 (r337142) +++ vendor/libc++/dist/include/__hash_table Thu Aug 2 17:33:33 2018 (r337143) @@ -859,7 +859,18 @@ class __hash_node_destructor (public) template friend class __hash_map_node_destructor; }; +#if _LIBCPP_STD_VER > 14 +template +struct __generic_container_node_destructor; +template +struct __generic_container_node_destructor<__hash_node<_Tp, _VoidPtr>, _Alloc> + : __hash_node_destructor<_Alloc> +{ + using __hash_node_destructor<_Alloc>::__hash_node_destructor; +}; +#endif + #ifndef _LIBCPP_CXX03_LANG template struct __diagnose_hash_table_helper { @@ -1151,6 +1162,30 @@ class __hash_table (public) return __emplace_unique_key_args(_NodeTypes::__get_key(__x), __x); } +#if _LIBCPP_STD_VER > 14 + template + _LIBCPP_INLINE_VISIBILITY + _InsertReturnType __node_handle_insert_unique(_NodeHandle&& __nh); + template + _LIBCPP_INLINE_VISIBILITY + iterator __node_handle_insert_unique(const_iterator __hint, + _NodeHandle&& __nh); + + template + _LIBCPP_INLINE_VISIBILITY + iterator __node_handle_insert_multi(_NodeHandle&& __nh); + template + _LIBCPP_INLINE_VISIBILITY + iterator __node_handle_insert_multi(const_iterator __hint, _NodeHandle&& __nh); + + template + _LIBCPP_INLINE_VISIBILITY + _NodeHandle __node_handle_extract(key_type const& __key); + template + _LIBCPP_INLINE_VISIBILITY + _NodeHandle __node_handle_extract(const_iterator __it); +#endif + void clear() _NOEXCEPT; void rehash(size_type __n); _LIBCPP_INLINE_VISIBILITY void reserve(size_type __n) @@ -2125,6 +2160,91 @@ __hash_table<_Tp, _Hash, _Equal, _Alloc>::__insert_mul } #endif // _LIBCPP_CXX03_LANG + +#if _LIBCPP_STD_VER > 14 +template +template +_LIBCPP_INLINE_VISIBILITY +_InsertReturnType +__hash_table<_Tp, _Hash, _Equal, _Alloc>::__node_handle_insert_unique( + _NodeHandle&& __nh) +{ + if (__nh.empty()) + return _InsertReturnType{end(), false, _NodeHandle()}; + pair __result = __node_insert_unique(__nh.__ptr_); + if (__result.second) + __nh.__release(); + return _InsertReturnType{__result.first, __result.second, _VSTD::move(__nh)}; +} + +template +template +_LIBCPP_INLINE_VISIBILITY +typename __hash_table<_Tp, _Hash, _Equal, _Alloc>::iterator +__hash_table<_Tp, _Hash, _Equal, _Alloc>::__node_handle_insert_unique( + const_iterator, _NodeHandle&& __nh) +{ + if (__nh.empty()) + return end(); + pair __result = __node_insert_unique(__nh.__ptr_); + if (__result.second) + __nh.__release(); + return __result.first; +} + +template +template +_LIBCPP_INLINE_VISIBILITY +_NodeHandle +__hash_table<_Tp, _Hash, _Equal, _Alloc>::__node_handle_extract( + key_type const& __key) +{ + iterator __i = find(__key); + if (__i == end()) + return _NodeHandle(); + return __node_handle_extract<_NodeHandle>(__i); +} + +template +template +_LIBCPP_INLINE_VISIBILITY +_NodeHandle +__hash_table<_Tp, _Hash, _Equal, _Alloc>::__node_handle_extract( + const_iterator __p) +{ + allocator_type __alloc(__node_alloc()); + return _NodeHandle(remove(__p).release(), __alloc); +} + +template +template +_LIBCPP_INLINE_VISIBILITY +typename __hash_table<_Tp, _Hash, _Equal, _Alloc>::iterator +__hash_table<_Tp, _Hash, _Equal, _Alloc>::__node_handle_insert_multi( + _NodeHandle&& __nh) +{ + if (__nh.empty()) + return end(); + iterator __result = __node_insert_multi(__nh.__ptr_); + __nh.__release(); + return __result; +} + +template +template +_LIBCPP_INLINE_VISIBILITY +typename __hash_table<_Tp, _Hash, _Equal, _Alloc>::iterator +__hash_table<_Tp, _Hash, _Equal, _Alloc>::__node_handle_insert_multi( + const_iterator __hint, _NodeHandle&& __nh) +{ + if (__nh.empty()) + return end(); + iterator __result = __node_insert_multi(__hint, __nh.__ptr_); + __nh.__release(); + return __result; +} + +#endif // _LIBCPP_STD_VER > 14 template void Modified: vendor/libc++/dist/include/__mutex_base ============================================================================== --- vendor/libc++/dist/include/__mutex_base Thu Aug 2 17:33:29 2018 (r337142) +++ vendor/libc++/dist/include/__mutex_base Thu Aug 2 17:33:33 2018 (r337143) @@ -74,7 +74,7 @@ struct _LIBCPP_TYPE_VIS defer_lock_t {}; struct _LIBCPP_TYPE_VIS try_to_lock_t {}; struct _LIBCPP_TYPE_VIS adopt_lock_t {}; -#if defined(_LIBCPP_CXX03_LANG) || defined(_LIBCPP_BUILDING_MUTEX) +#if defined(_LIBCPP_CXX03_LANG) || defined(_LIBCPP_BUILDING_LIBRARY) extern const defer_lock_t defer_lock; extern const try_to_lock_t try_to_lock; Added: vendor/libc++/dist/include/__node_handle ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libc++/dist/include/__node_handle Thu Aug 2 17:33:33 2018 (r337143) @@ -0,0 +1,212 @@ +// -*- C++ -*- +//===----------------------------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is dual licensed under the MIT and the University of Illinois Open +// Source Licenses. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// + +#ifndef _LIBCPP___NODE_HANDLE +#define _LIBCPP___NODE_HANDLE + +#include <__config> +#include +#include + +#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER) +#pragma GCC system_header +#endif + +_LIBCPP_PUSH_MACROS +#include <__undef_macros> + +_LIBCPP_BEGIN_NAMESPACE_STD + +#if _LIBCPP_STD_VER > 14 + +#define __cpp_lib_node_extract 201606L + +// Specialized in __tree & __hash_table for their _NodeType. +template +struct __generic_container_node_destructor; + +template class _MapOrSetSpecifics> +class _LIBCPP_TEMPLATE_VIS __basic_node_handle + : public _MapOrSetSpecifics< + _NodeType, + __basic_node_handle<_NodeType, _Alloc, _MapOrSetSpecifics>> +{ + template + friend class __tree; + template + friend class __hash_table; + friend struct _MapOrSetSpecifics< + _NodeType, __basic_node_handle<_NodeType, _Alloc, _MapOrSetSpecifics>>; + + typedef allocator_traits<_Alloc> __alloc_traits; + typedef typename __rebind_pointer::type + __node_pointer_type; + +public: + typedef _Alloc allocator_type; + +private: + __node_pointer_type __ptr_ = nullptr; + optional __alloc_; + + _LIBCPP_INLINE_VISIBILITY + void __release() + { + __ptr_ = nullptr; + __alloc_ = _VSTD::nullopt; + } + + _LIBCPP_INLINE_VISIBILITY + void __destroy_node_pointer() + { + if (__ptr_ != nullptr) + { + typedef typename __allocator_traits_rebind< + allocator_type, _NodeType>::type __node_alloc_type; + __node_alloc_type __alloc(*__alloc_); + __generic_container_node_destructor<_NodeType, __node_alloc_type>( + __alloc, true)(__ptr_); + __ptr_ = nullptr; + } + } + + _LIBCPP_INLINE_VISIBILITY + __basic_node_handle(__node_pointer_type __ptr, + allocator_type const& __alloc) + : __ptr_(__ptr), __alloc_(__alloc) + { + } + +public: + _LIBCPP_INLINE_VISIBILITY + __basic_node_handle() = default; + + _LIBCPP_INLINE_VISIBILITY + __basic_node_handle(__basic_node_handle&& __other) noexcept + : __ptr_(__other.__ptr_), + __alloc_(_VSTD::move(__other.__alloc_)) + { + __other.__ptr_ = nullptr; + __other.__alloc_ = _VSTD::nullopt; + } + + _LIBCPP_INLINE_VISIBILITY + __basic_node_handle& operator=(__basic_node_handle&& __other) + { + _LIBCPP_ASSERT( + __alloc_ == _VSTD::nullopt || + __alloc_traits::propagate_on_container_move_assignment::value || + __alloc_ == __other.__alloc_, + "node_type with incompatible allocator passed to " + "node_type::operator=(node_type&&)"); + + __destroy_node_pointer(); + __ptr_ = __other.__ptr_; + + if (__alloc_traits::propagate_on_container_move_assignment::value || + __alloc_ == _VSTD::nullopt) + __alloc_ = _VSTD::move(__other.__alloc_); + + __other.__ptr_ = nullptr; + __other.__alloc_ = _VSTD::nullopt; + + return *this; + } + + _LIBCPP_INLINE_VISIBILITY + allocator_type get_allocator() const { return *__alloc_; } + + _LIBCPP_INLINE_VISIBILITY + explicit operator bool() const { return __ptr_ != nullptr; } + + _LIBCPP_NODISCARD_AFTER_CXX17 _LIBCPP_INLINE_VISIBILITY + bool empty() const { return __ptr_ == nullptr; } + + _LIBCPP_INLINE_VISIBILITY + void swap(__basic_node_handle& __other) noexcept( + __alloc_traits::propagate_on_container_swap::value || + __alloc_traits::is_always_equal::value) + { + using _VSTD::swap; + swap(__ptr_, __other.__ptr_); + if (__alloc_traits::propagate_on_container_swap::value || + __alloc_ == _VSTD::nullopt || __other.__alloc_ == _VSTD::nullopt) + swap(__alloc_, __other.__alloc_); + } + + _LIBCPP_INLINE_VISIBILITY + friend void swap(__basic_node_handle& __a, __basic_node_handle& __b) + noexcept(noexcept(__a.swap(__b))) { __a.swap(__b); } + + _LIBCPP_INLINE_VISIBILITY + ~__basic_node_handle() + { + __destroy_node_pointer(); + } +}; + +template +struct __set_node_handle_specifics +{ + typedef typename _NodeType::__node_value_type value_type; + + _LIBCPP_INLINE_VISIBILITY + value_type& value() const + { + return static_cast<_Derived const*>(this)->__ptr_->__value_; + } +}; + +template +struct __map_node_handle_specifics +{ + typedef typename _NodeType::__node_value_type::key_type key_type; + typedef typename _NodeType::__node_value_type::mapped_type mapped_type; + + _LIBCPP_INLINE_VISIBILITY + key_type& key() const + { + return static_cast<_Derived const*>(this)-> + __ptr_->__value_.__ref().first; + } + + _LIBCPP_INLINE_VISIBILITY + mapped_type& mapped() const + { + return static_cast<_Derived const*>(this)-> + __ptr_->__value_.__ref().second; + } +}; + +template +using __set_node_handle = + __basic_node_handle< _NodeType, _Alloc, __set_node_handle_specifics>; + +template +using __map_node_handle = + __basic_node_handle< _NodeType, _Alloc, __map_node_handle_specifics>; + +template +_LIBCPP_TEMPLATE_VIS +struct __insert_return_type +{ + _Iterator position; + bool inserted; + _NodeType node; +}; + +#endif // _LIBCPP_STD_VER > 14 + +_LIBCPP_END_NAMESPACE_STD +_LIBCPP_POP_MACROS + +#endif Modified: vendor/libc++/dist/include/__tree ============================================================================== --- vendor/libc++/dist/include/__tree Thu Aug 2 17:33:29 2018 (r337142) +++ vendor/libc++/dist/include/__tree Thu Aug 2 17:33:33 2018 (r337143) @@ -796,6 +796,16 @@ class __tree_node_destructor (public) template friend class __map_node_destructor; }; +#if _LIBCPP_STD_VER > 14 +template +struct __generic_container_node_destructor; +template +struct __generic_container_node_destructor<__tree_node<_Tp, _VoidPtr>, _Alloc> + : __tree_node_destructor<_Alloc> +{ + using __tree_node_destructor<_Alloc>::__tree_node_destructor; +}; +#endif template class _LIBCPP_TEMPLATE_VIS __tree_iterator @@ -1338,6 +1348,33 @@ class __tree (public) iterator __node_insert_multi(__node_pointer __nd); iterator __node_insert_multi(const_iterator __p, __node_pointer __nd); + + _LIBCPP_INLINE_VISIBILITY iterator __remove_node_pointer(__node_pointer); + +#if _LIBCPP_STD_VER > 14 + template + _LIBCPP_INLINE_VISIBILITY + _InsertReturnType __node_handle_insert_unique(_NodeHandle&&); + template + _LIBCPP_INLINE_VISIBILITY + iterator __node_handle_insert_unique(const_iterator, _NodeHandle&&); + + template + _LIBCPP_INLINE_VISIBILITY + iterator __node_handle_insert_multi(_NodeHandle&&); + template + _LIBCPP_INLINE_VISIBILITY + iterator __node_handle_insert_multi(const_iterator, _NodeHandle&&); + + + template + _LIBCPP_INLINE_VISIBILITY + _NodeHandle __node_handle_extract(key_type const&); + template + _LIBCPP_INLINE_VISIBILITY + _NodeHandle __node_handle_extract(const_iterator); +#endif + iterator erase(const_iterator __p); iterator erase(const_iterator __f, const_iterator __l); template @@ -2347,17 +2384,138 @@ __tree<_Tp, _Compare, _Allocator>::__node_insert_multi template typename __tree<_Tp, _Compare, _Allocator>::iterator -__tree<_Tp, _Compare, _Allocator>::erase(const_iterator __p) +__tree<_Tp, _Compare, _Allocator>::__remove_node_pointer(__node_pointer __ptr) { - __node_pointer __np = __p.__get_np(); - iterator __r(__p.__ptr_); + iterator __r(__ptr); ++__r; - if (__begin_node() == __p.__ptr_) + if (__begin_node() == __ptr) __begin_node() = __r.__ptr_; --size(); - __node_allocator& __na = __node_alloc(); __tree_remove(__end_node()->__left_, - static_cast<__node_base_pointer>(__np)); + static_cast<__node_base_pointer>(__ptr)); + return __r; +} + +#if _LIBCPP_STD_VER > 14 +template +template +_LIBCPP_INLINE_VISIBILITY +_InsertReturnType +__tree<_Tp, _Compare, _Allocator>::__node_handle_insert_unique( + _NodeHandle&& __nh) +{ + if (__nh.empty()) + return _InsertReturnType{end(), false, _NodeHandle()}; + + __node_pointer __ptr = __nh.__ptr_; + __parent_pointer __parent; + __node_base_pointer& __child = __find_equal(__parent, + __ptr->__value_); + if (__child != nullptr) + return _InsertReturnType{ + iterator(static_cast<__node_pointer>(__child)), + false, _VSTD::move(__nh)}; + + __insert_node_at(__parent, __child, + static_cast<__node_base_pointer>(__ptr)); + __nh.__release(); + return _InsertReturnType{iterator(__ptr), true, _NodeHandle()}; +} + +template +template +_LIBCPP_INLINE_VISIBILITY +typename __tree<_Tp, _Compare, _Allocator>::iterator +__tree<_Tp, _Compare, _Allocator>::__node_handle_insert_unique( + const_iterator __hint, _NodeHandle&& __nh) +{ + if (__nh.empty()) + return end(); + + __node_pointer __ptr = __nh.__ptr_; + __parent_pointer __parent; + __node_base_pointer __dummy; + __node_base_pointer& __child = __find_equal(__hint, __parent, __dummy, + __ptr->__value_); + __node_pointer __r = static_cast<__node_pointer>(__child); + if (__child == nullptr) + { + __insert_node_at(__parent, __child, + static_cast<__node_base_pointer>(__ptr)); + __r = __ptr; + __nh.__release(); + } + return iterator(__r); +} + +template +template +_LIBCPP_INLINE_VISIBILITY +_NodeHandle +__tree<_Tp, _Compare, _Allocator>::__node_handle_extract(key_type const& __key) +{ + iterator __it = find(__key); + if (__it == end()) + return _NodeHandle(); + return __node_handle_extract<_NodeHandle>(__it); +} + +template +template +_LIBCPP_INLINE_VISIBILITY +_NodeHandle +__tree<_Tp, _Compare, _Allocator>::__node_handle_extract(const_iterator __p) +{ + __node_pointer __np = __p.__get_np(); + __remove_node_pointer(__np); + return _NodeHandle(__np, __alloc()); +} + +template +template +_LIBCPP_INLINE_VISIBILITY +typename __tree<_Tp, _Compare, _Allocator>::iterator +__tree<_Tp, _Compare, _Allocator>::__node_handle_insert_multi(_NodeHandle&& __nh) +{ + if (__nh.empty()) + return end(); + __node_pointer __ptr = __nh.__ptr_; + __parent_pointer __parent; + __node_base_pointer& __child = __find_leaf_high( + __parent, _NodeTypes::__get_key(__ptr->__value_)); + __insert_node_at(__parent, __child, static_cast<__node_base_pointer>(__ptr)); + __nh.__release(); + return iterator(__ptr); +} + +template +template +_LIBCPP_INLINE_VISIBILITY +typename __tree<_Tp, _Compare, _Allocator>::iterator +__tree<_Tp, _Compare, _Allocator>::__node_handle_insert_multi( + const_iterator __hint, _NodeHandle&& __nh) +{ + if (__nh.empty()) + return end(); + + __node_pointer __ptr = __nh.__ptr_; + __parent_pointer __parent; + __node_base_pointer& __child = __find_leaf(__hint, __parent, + _NodeTypes::__get_key(__ptr->__value_)); + __insert_node_at(__parent, __child, static_cast<__node_base_pointer>(__ptr)); + __nh.__release(); + return iterator(__ptr); +} + +#endif // _LIBCPP_STD_VER > 14 + +template +typename __tree<_Tp, _Compare, _Allocator>::iterator +__tree<_Tp, _Compare, _Allocator>::erase(const_iterator __p) +{ + __node_pointer __np = __p.__get_np(); + iterator __r = __remove_node_pointer(__np); + __node_allocator& __na = __node_alloc(); __node_traits::destroy(__na, _NodeTypes::__get_ptr( const_cast<__node_value_type&>(*__p))); __node_traits::deallocate(__na, __np, 1); Modified: vendor/libc++/dist/include/cfloat ============================================================================== --- vendor/libc++/dist/include/cfloat Thu Aug 2 17:33:29 2018 (r337142) +++ vendor/libc++/dist/include/cfloat Thu Aug 2 17:33:33 2018 (r337143) @@ -20,11 +20,18 @@ Macros: FLT_EVAL_METHOD // C99 FLT_RADIX + FLT_HAS_SUBNORM // C11 + DBL_HAS_SUBNORM // C11 + LDBL_HAS_SUBNORM // C11 + FLT_MANT_DIG DBL_MANT_DIG LDBL_MANT_DIG DECIMAL_DIG // C99 + FLT_DECIMAL_DIG // C11 + DBL_DECIMAL_DIG // C11 + LDBL_DECIMAL_DIG // C11 FLT_DIG DBL_DIG @@ -58,6 +65,9 @@ Macros: DBL_MIN LDBL_MIN + FLT_TRUE_MIN // C11 + DBL_TRUE_MIN // C11 + LDBL_TRUE_MIN // C11 */ #include <__config> Added: vendor/libc++/dist/include/charconv ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libc++/dist/include/charconv Thu Aug 2 17:33:33 2018 (r337143) @@ -0,0 +1,610 @@ +// -*- C++ -*- +//===------------------------------ charconv ------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is dual licensed under the MIT and the University of Illinois Open +// Source Licenses. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// + +#ifndef _LIBCPP_CHARCONV +#define _LIBCPP_CHARCONV + +/* + charconv synopsis + +namespace std { + + // floating-point format for primitive numerical conversion + enum class chars_format { + scientific = unspecified, + fixed = unspecified, + hex = unspecified, + general = fixed | scientific + }; + + // 23.20.2, primitive numerical output conversion + struct to_chars_result { + char* ptr; + errc ec; + }; + + to_chars_result to_chars(char* first, char* last, see below value, + int base = 10); + + to_chars_result to_chars(char* first, char* last, float value); + to_chars_result to_chars(char* first, char* last, double value); + to_chars_result to_chars(char* first, char* last, long double value); + + to_chars_result to_chars(char* first, char* last, float value, + chars_format fmt); + to_chars_result to_chars(char* first, char* last, double value, + chars_format fmt); + to_chars_result to_chars(char* first, char* last, long double value, + chars_format fmt); + + to_chars_result to_chars(char* first, char* last, float value, + chars_format fmt, int precision); + to_chars_result to_chars(char* first, char* last, double value, + chars_format fmt, int precision); + to_chars_result to_chars(char* first, char* last, long double value, + chars_format fmt, int precision); + + // 23.20.3, primitive numerical input conversion + struct from_chars_result { + const char* ptr; + errc ec; + }; + + from_chars_result from_chars(const char* first, const char* last, + see below& value, int base = 10); + + from_chars_result from_chars(const char* first, const char* last, + float& value, + chars_format fmt = chars_format::general); + from_chars_result from_chars(const char* first, const char* last, + double& value, + chars_format fmt = chars_format::general); + from_chars_result from_chars(const char* first, const char* last, + long double& value, + chars_format fmt = chars_format::general); + +} // namespace std + +*/ + +#include <__errc> +#include +#include +#include +#include +#include + +#include <__debug> + +#if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER) +#pragma GCC system_header +#endif + +_LIBCPP_BEGIN_NAMESPACE_STD + +#if _LIBCPP_STD_VER > 11 + +enum class _LIBCPP_ENUM_VIS chars_format +{ + scientific = 0x1, + fixed = 0x2, + hex = 0x4, + general = fixed | scientific +}; + +struct _LIBCPP_TYPE_VIS to_chars_result +{ + char* ptr; + errc ec; +}; + +struct _LIBCPP_TYPE_VIS from_chars_result +{ + const char* ptr; + errc ec; +}; + +void to_chars(char*, char*, bool, int = 10) = delete; +void from_chars(const char*, const char*, bool, int = 10) = delete; + +namespace __itoa +{ + +static constexpr uint64_t __pow10_64[] = { + UINT64_C(0), + UINT64_C(10), + UINT64_C(100), + UINT64_C(1000), + UINT64_C(10000), + UINT64_C(100000), + UINT64_C(1000000), + UINT64_C(10000000), + UINT64_C(100000000), + UINT64_C(1000000000), + UINT64_C(10000000000), + UINT64_C(100000000000), + UINT64_C(1000000000000), + UINT64_C(10000000000000), + UINT64_C(100000000000000), + UINT64_C(1000000000000000), + UINT64_C(10000000000000000), + UINT64_C(100000000000000000), + UINT64_C(1000000000000000000), + UINT64_C(10000000000000000000), +}; + +static constexpr uint32_t __pow10_32[] = { + UINT32_C(0), UINT32_C(10), UINT32_C(100), + UINT32_C(1000), UINT32_C(10000), UINT32_C(100000), + UINT32_C(1000000), UINT32_C(10000000), UINT32_C(100000000), + UINT32_C(1000000000), +}; + +_LIBCPP_FUNC_VIS char* __u64toa(uint64_t __value, char* __buffer); +_LIBCPP_FUNC_VIS char* __u32toa(uint32_t __value, char* __buffer); + +template +struct _LIBCPP_HIDDEN __traits_base +{ + using type = uint64_t; + +#if !defined(_LIBCPP_COMPILER_MSVC) + static _LIBCPP_INLINE_VISIBILITY int __width(_Tp __v) + { + auto __t = (64 - __builtin_clzll(__v | 1)) * 1233 >> 12; + return __t - (__v < __pow10_64[__t]) + 1; + } +#endif + + static _LIBCPP_INLINE_VISIBILITY char* __convert(_Tp __v, char* __p) + { + return __u64toa(__v, __p); + } + + static _LIBCPP_INLINE_VISIBILITY auto& __pow() { return __pow10_64; } +}; + +template +struct _LIBCPP_HIDDEN + __traits_base<_Tp, decltype(void(uint32_t{declval<_Tp>()}))> +{ + using type = uint32_t; + +#if !defined(_LIBCPP_COMPILER_MSVC) + static _LIBCPP_INLINE_VISIBILITY int __width(_Tp __v) + { + auto __t = (32 - __builtin_clz(__v | 1)) * 1233 >> 12; + return __t - (__v < __pow10_32[__t]) + 1; + } +#endif + + static _LIBCPP_INLINE_VISIBILITY char* __convert(_Tp __v, char* __p) + { + return __u32toa(__v, __p); + } + + static _LIBCPP_INLINE_VISIBILITY auto& __pow() { return __pow10_32; } +}; + +template +inline _LIBCPP_INLINE_VISIBILITY bool +__mul_overflowed(unsigned char __a, _Tp __b, unsigned char& __r) +{ + auto __c = __a * __b; + __r = __c; + return __c > (numeric_limits::max)(); +} + +template +inline _LIBCPP_INLINE_VISIBILITY bool +__mul_overflowed(unsigned short __a, _Tp __b, unsigned short& __r) +{ + auto __c = __a * __b; + __r = __c; + return __c > (numeric_limits::max)(); +} + +template +inline _LIBCPP_INLINE_VISIBILITY bool +__mul_overflowed(_Tp __a, _Tp __b, _Tp& __r) +{ + static_assert(is_unsigned<_Tp>::value, ""); +#if !defined(_LIBCPP_COMPILER_MSVC) + return __builtin_mul_overflow(__a, __b, &__r); +#else + bool __did = __b && ((numeric_limits<_Tp>::max)() / __b) < __a; + __r = __a * __b; + return __did; +#endif +} + +template +inline _LIBCPP_INLINE_VISIBILITY bool +__mul_overflowed(_Tp __a, _Up __b, _Tp& __r) +{ + return __mul_overflowed(__a, static_cast<_Tp>(__b), __r); +} + +template +struct _LIBCPP_HIDDEN __traits : __traits_base<_Tp> +{ + static constexpr int digits = numeric_limits<_Tp>::digits10 + 1; + using __traits_base<_Tp>::__pow; + using typename __traits_base<_Tp>::type; + + // precondition: at least one non-zero character available + static _LIBCPP_INLINE_VISIBILITY char const* + __read(char const* __p, char const* __ep, type& __a, type& __b) + { + type __cprod[digits]; + int __j = digits - 1; + int __i = digits; + do + { + if (!('0' <= *__p && *__p <= '9')) + break; + __cprod[--__i] = *__p++ - '0'; + } while (__p != __ep && __i != 0); + + __a = __inner_product(__cprod + __i + 1, __cprod + __j, __pow() + 1, + __cprod[__i]); + if (__mul_overflowed(__cprod[__j], __pow()[__j - __i], __b)) + --__p; + return __p; + } + + template + static _LIBCPP_INLINE_VISIBILITY _Up + __inner_product(_It1 __first1, _It1 __last1, _It2 __first2, _Up __init) + { + for (; __first1 < __last1; ++__first1, ++__first2) + __init = __init + *__first1 * *__first2; + return __init; + } +}; + +} // namespace __itoa + +template +inline _LIBCPP_INLINE_VISIBILITY _Tp +__complement(_Tp __x) +{ + static_assert(is_unsigned<_Tp>::value, "cast to unsigned first"); + return _Tp(~__x + 1); +} + +template +inline _LIBCPP_INLINE_VISIBILITY auto +__to_unsigned(_Tp __x) +{ + return static_cast>(__x); +} + +template +inline _LIBCPP_INLINE_VISIBILITY to_chars_result +__to_chars_itoa(char* __first, char* __last, _Tp __value, true_type) +{ + auto __x = __to_unsigned(__value); + if (__value < 0 && __first != __last) + { + *__first++ = '-'; + __x = __complement(__x); + } + + return __to_chars_itoa(__first, __last, __x, false_type()); +} + +template +inline _LIBCPP_INLINE_VISIBILITY to_chars_result +__to_chars_itoa(char* __first, char* __last, _Tp __value, false_type) +{ + using __tx = __itoa::__traits<_Tp>; + auto __diff = __last - __first; + +#if !defined(_LIBCPP_COMPILER_MSVC) + if (__tx::digits <= __diff || __tx::__width(__value) <= __diff) + return {__tx::__convert(__value, __first), {}}; + else + return {__last, errc::value_too_large}; +#else + if (__tx::digits <= __diff) + return {__tx::__convert(__value, __first), {}}; + else + { + char __buf[__tx::digits]; + auto __p = __tx::__convert(__value, __buf); + auto __len = __p - __buf; + if (__len <= __diff) + { + memcpy(__first, __buf, __len); + return {__first + __len, {}}; + } + else + return {__last, errc::value_too_large}; + } +#endif +} + +template +inline _LIBCPP_INLINE_VISIBILITY to_chars_result +__to_chars_integral(char* __first, char* __last, _Tp __value, int __base, + true_type) +{ + auto __x = __to_unsigned(__value); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:42 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7CE83106D7B7; Thu, 2 Aug 2018 17:33:42 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18AFA726F1; Thu, 2 Aug 2018 17:33:39 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6449B14D35; Thu, 2 Aug 2018 17:33:39 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXdHl029611; Thu, 2 Aug 2018 17:33:39 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXdHN029610; Thu, 2 Aug 2018 17:33:39 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXdHN029610@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337144 - vendor/libc++/libc++-trunk-r338536 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/libc++/libc++-trunk-r338536 X-SVN-Commit-Revision: 337144 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:42 -0000 Author: dim Date: Thu Aug 2 17:33:38 2018 New Revision: 337144 URL: https://svnweb.freebsd.org/changeset/base/337144 Log: Tag libc++ trunk r338536. Added: vendor/libc++/libc++-trunk-r338536/ - copied from r337143, vendor/libc++/dist/ From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:49 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EEF5106D805; Thu, 2 Aug 2018 17:33:49 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 04A8072818; Thu, 2 Aug 2018 17:33:47 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5A3E214D37; Thu, 2 Aug 2018 17:33:47 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXlpg029685; Thu, 2 Aug 2018 17:33:47 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXgga029660; Thu, 2 Aug 2018 17:33:42 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXgga029660@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337145 - in vendor/lld/dist: COFF ELF ELF/Arch docs test/ELF test/ELF/Inputs test/ELF/linkerscript test/ELF/lto test/ELF/lto/Inputs X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/lld/dist: COFF ELF ELF/Arch docs test/ELF test/ELF/Inputs test/ELF/linkerscript test/ELF/lto test/ELF/lto/Inputs X-SVN-Commit-Revision: 337145 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:50 -0000 Author: dim Date: Thu Aug 2 17:33:42 2018 New Revision: 337145 URL: https://svnweb.freebsd.org/changeset/base/337145 Log: Vendor import of lld trunk r338536: https://llvm.org/svn/llvm-project/lld/trunk@338536 Added: vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-base.s (contents, props changed) vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-compat.s (contents, props changed) vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-toolchain.s (contents, props changed) vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-vfp.s (contents, props changed) vendor/lld/dist/test/ELF/Inputs/icf-absolute2.s (contents, props changed) vendor/lld/dist/test/ELF/arm-tag-vfp-args-errs.s (contents, props changed) vendor/lld/dist/test/ELF/arm-tag-vfp-args-illegal.s (contents, props changed) vendor/lld/dist/test/ELF/arm-tag-vfp-args.s (contents, props changed) vendor/lld/dist/test/ELF/execute-only-mixed-data.s (contents, props changed) vendor/lld/dist/test/ELF/execute-only.s (contents, props changed) vendor/lld/dist/test/ELF/icf-absolute2.s (contents, props changed) vendor/lld/dist/test/ELF/icf17.s (contents, props changed) vendor/lld/dist/test/ELF/lto/Inputs/libcall-archive.ll vendor/lld/dist/test/ELF/lto/libcall-archive.ll Modified: vendor/lld/dist/COFF/ICF.cpp vendor/lld/dist/ELF/Arch/ARM.cpp vendor/lld/dist/ELF/Arch/Hexagon.cpp vendor/lld/dist/ELF/Config.h vendor/lld/dist/ELF/Driver.cpp vendor/lld/dist/ELF/ICF.cpp vendor/lld/dist/ELF/InputFiles.cpp vendor/lld/dist/ELF/InputSection.cpp vendor/lld/dist/ELF/Options.td vendor/lld/dist/ELF/SyntheticSections.cpp vendor/lld/dist/ELF/SyntheticSections.h vendor/lld/dist/ELF/Writer.cpp vendor/lld/dist/docs/index.rst vendor/lld/dist/docs/ld.lld.1 vendor/lld/dist/docs/windows_support.rst vendor/lld/dist/test/ELF/arm-eabi-version.s vendor/lld/dist/test/ELF/hexagon.s vendor/lld/dist/test/ELF/icf-safe.s vendor/lld/dist/test/ELF/linkerscript/orphan-report.s vendor/lld/dist/test/ELF/oformat-binary.s vendor/lld/dist/test/ELF/relocatable-many-sections.s Modified: vendor/lld/dist/COFF/ICF.cpp ============================================================================== --- vendor/lld/dist/COFF/ICF.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/COFF/ICF.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -27,6 +27,7 @@ #include "llvm/Support/Debug.h" #include "llvm/Support/Parallel.h" #include "llvm/Support/raw_ostream.h" +#include "llvm/Support/xxhash.h" #include #include #include @@ -65,13 +66,6 @@ class ICF { (private) std::atomic Repeat = {false}; }; -// Returns a hash value for S. -uint32_t ICF::getHash(SectionChunk *C) { - return hash_combine(C->getOutputCharacteristics(), C->SectionName, - C->Relocs.size(), uint32_t(C->Header->SizeOfRawData), - C->Checksum, C->getContents()); -} - // Returns true if section S is subject of ICF. // // Microsoft's documentation @@ -265,7 +259,7 @@ void ICF::run(ArrayRef Vec) { // Initially, we use hash values to partition sections. for_each(parallel::par, Chunks.begin(), Chunks.end(), [&](SectionChunk *SC) { // Set MSB to 1 to avoid collisions with non-hash classs. - SC->Class[0] = getHash(SC) | (1 << 31); + SC->Class[0] = xxHash64(SC->getContents()) | (1 << 31); }); // From now on, sections in Chunks are ordered so that sections in Modified: vendor/lld/dist/ELF/Arch/ARM.cpp ============================================================================== --- vendor/lld/dist/ELF/Arch/ARM.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/Arch/ARM.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -97,10 +97,19 @@ ARM::ARM() { } uint32_t ARM::calcEFlags() const { + // The ABIFloatType is used by loaders to detect the floating point calling + // convention. + uint32_t ABIFloatType = 0; + if (Config->ARMVFPArgs == ARMVFPArgKind::Base || + Config->ARMVFPArgs == ARMVFPArgKind::Default) + ABIFloatType = EF_ARM_ABI_FLOAT_SOFT; + else if (Config->ARMVFPArgs == ARMVFPArgKind::VFP) + ABIFloatType = EF_ARM_ABI_FLOAT_HARD; + // We don't currently use any features incompatible with EF_ARM_EABI_VER5, // but we don't have any firm guarantees of conformance. Linux AArch64 // kernels (as of 2016) require an EABI version to be set. - return EF_ARM_EABI_VER5; + return EF_ARM_EABI_VER5 | ABIFloatType; } RelExpr ARM::getRelExpr(RelType Type, const Symbol &S, Modified: vendor/lld/dist/ELF/Arch/Hexagon.cpp ============================================================================== --- vendor/lld/dist/ELF/Arch/Hexagon.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/Arch/Hexagon.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -70,6 +70,12 @@ void Hexagon::relocateOne(uint8_t *Loc, RelType Type, switch (Type) { case R_HEX_NONE: break; + case R_HEX_12_X: + or32le(Loc, applyMask(0x000007e0, Val)); + break; + case R_HEX_32_6_X: + or32le(Loc, applyMask(0x0fff3fff, Val >> 6)); + break; case R_HEX_B15_PCREL: or32le(Loc, applyMask(0x00df20fe, Val >> 2)); break; Modified: vendor/lld/dist/ELF/Config.h ============================================================================== --- vendor/lld/dist/ELF/Config.h Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/Config.h Thu Aug 2 17:33:42 2018 (r337145) @@ -58,6 +58,9 @@ enum class SortSectionPolicy { Default, None, Alignmen // For --target2 enum class Target2Policy { Abs, Rel, GotRel }; +// For tracking ARM Float Argument PCS +enum class ARMVFPArgKind { Default, Base, VFP, ToolChain }; + struct SymbolVersion { llvm::StringRef Name; bool IsExternCpp; @@ -133,6 +136,7 @@ struct Configuration { bool EhFrameHdr; bool EmitRelocs; bool EnableNewDtags; + bool ExecuteOnly; bool ExportDynamic; bool FixCortexA53Errata843419; bool GcSections; @@ -195,6 +199,7 @@ struct Configuration { StripPolicy Strip; UnresolvedPolicy UnresolvedSymbols; Target2Policy Target2; + ARMVFPArgKind ARMVFPArgs = ARMVFPArgKind::Default; BuildIdKind BuildId = BuildIdKind::None; ELFKind EKind = ELFNoneKind; uint16_t DefaultSymbolVersion = llvm::ELF::VER_NDX_GLOBAL; Modified: vendor/lld/dist/ELF/Driver.cpp ============================================================================== --- vendor/lld/dist/ELF/Driver.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/Driver.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -302,6 +302,14 @@ static void checkOptions(opt::InputArgList &Args) { if (Config->Pie) error("-r and -pie may not be used together"); } + + if (Config->ExecuteOnly) { + if (Config->EMachine != EM_AARCH64) + error("-execute-only is only supported on AArch64 targets"); + + if (Config->SingleRoRx && !Script->HasSectionsCommand) + error("-execute-only and -no-rosegment cannot be used together"); + } } static const char *getReproduceOption(opt::InputArgList &Args) { @@ -493,6 +501,8 @@ static bool isOutputFormatBinary(opt::InputArgList &Ar StringRef S = Arg->getValue(); if (S == "binary") return true; + if (S.startswith("elf")) + return false; error("unknown --oformat value: " + S); } return false; @@ -747,6 +757,8 @@ void LinkerDriver::readConfigs(opt::InputArgList &Args Config->EnableNewDtags = Args.hasFlag(OPT_enable_new_dtags, OPT_disable_new_dtags, true); Config->Entry = Args.getLastArgValue(OPT_entry); + Config->ExecuteOnly = + Args.hasFlag(OPT_execute_only, OPT_no_execute_only, false); Config->ExportDynamic = Args.hasFlag(OPT_export_dynamic, OPT_no_export_dynamic, false); Config->FilterList = args::getStrings(Args, OPT_filter); @@ -1303,6 +1315,12 @@ static void findKeepUniqueSections(opt::InputArgList & } } +static const char *LibcallRoutineNames[] = { +#define HANDLE_LIBCALL(code, name) name, +#include "llvm/IR/RuntimeLibcalls.def" +#undef HANDLE_LIBCALL +}; + // Do actual linking. Note that when this function is called, // all linker scripts have already been parsed. template void LinkerDriver::link(opt::InputArgList &Args) { @@ -1369,10 +1387,20 @@ template void LinkerDriver::link(opt::Inp for (StringRef S : Config->Undefined) handleUndefined(S); - // If an entry symbol is in a static archive, pull out that file now - // to complete the symbol table. After this, no new names except a - // few linker-synthesized ones will be added to the symbol table. + // If an entry symbol is in a static archive, pull out that file now. handleUndefined(Config->Entry); + + // If any of our inputs are bitcode files, the LTO code generator may create + // references to certain library functions that might not be explicit in the + // bitcode file's symbol table. If any of those library functions are defined + // in a bitcode file in an archive member, we need to arrange to use LTO to + // compile those archive members by adding them to the link beforehand. + // + // With this the symbol table should be complete. After this, no new names + // except a few linker-synthesized ones will be added to the symbol table. + if (!BitcodeFiles.empty()) + for (const char *S : LibcallRoutineNames) + handleUndefined(S); // Return if there were name resolution errors. if (errorCount()) Modified: vendor/lld/dist/ELF/ICF.cpp ============================================================================== --- vendor/lld/dist/ELF/ICF.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/ICF.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -80,9 +80,10 @@ #include "SyntheticSections.h" #include "Writer.h" #include "lld/Common/Threads.h" -#include "llvm/ADT/Hashing.h" +#include "llvm/ADT/StringExtras.h" #include "llvm/BinaryFormat/ELF.h" #include "llvm/Object/ELF.h" +#include "llvm/Support/xxhash.h" #include #include @@ -155,12 +156,6 @@ template class ICF { (private) }; } -// Returns a hash value for S. Note that the information about -// relocation targets is not included in the hash value. -template static uint32_t getHash(InputSection *S) { - return hash_combine(S->Flags, S->getSize(), S->NumRelocations, S->Data); -} - // Returns true if section S is subject of ICF. static bool isEligible(InputSection *S) { if (!S->Live || S->KeepUnique || !(S->Flags & SHF_ALLOC)) @@ -441,7 +436,7 @@ template void ICF::run() { // Initially, we use hash values to partition sections. parallelForEach(Sections, [&](InputSection *S) { // Set MSB to 1 to avoid collisions with non-hash IDs. - S->Class[0] = getHash(S) | (1U << 31); + S->Class[0] = xxHash64(S->Data) | (1U << 31); }); // From now on, sections in Sections vector are ordered so that sections Modified: vendor/lld/dist/ELF/InputFiles.cpp ============================================================================== --- vendor/lld/dist/ELF/InputFiles.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/InputFiles.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -494,6 +494,46 @@ void ObjFile::initializeSections( } } +// For ARM only, to set the EF_ARM_ABI_FLOAT_SOFT or EF_ARM_ABI_FLOAT_HARD +// flag in the ELF Header we need to look at Tag_ABI_VFP_args to find out how +// the input objects have been compiled. +static void updateARMVFPArgs(const ARMAttributeParser &Attributes, + const InputFile *F) { + if (!Attributes.hasAttribute(ARMBuildAttrs::ABI_VFP_args)) + // If an ABI tag isn't present then it is implicitly given the value of 0 + // which maps to ARMBuildAttrs::BaseAAPCS. However many assembler files, + // including some in glibc that don't use FP args (and should have value 3) + // don't have the attribute so we do not consider an implicit value of 0 + // as a clash. + return; + + unsigned VFPArgs = Attributes.getAttributeValue(ARMBuildAttrs::ABI_VFP_args); + ARMVFPArgKind Arg; + switch (VFPArgs) { + case ARMBuildAttrs::BaseAAPCS: + Arg = ARMVFPArgKind::Base; + break; + case ARMBuildAttrs::HardFPAAPCS: + Arg = ARMVFPArgKind::VFP; + break; + case ARMBuildAttrs::ToolChainFPPCS: + // Tool chain specific convention that conforms to neither AAPCS variant. + Arg = ARMVFPArgKind::ToolChain; + break; + case ARMBuildAttrs::CompatibleFPAAPCS: + // Object compatible with all conventions. + return; + default: + error(toString(F) + ": unknown Tag_ABI_VFP_args value: " + Twine(VFPArgs)); + return; + } + // Follow ld.bfd and error if there is a mix of calling conventions. + if (Config->ARMVFPArgs != Arg && Config->ARMVFPArgs != ARMVFPArgKind::Default) + error(toString(F) + ": incompatible Tag_ABI_VFP_args"); + else + Config->ARMVFPArgs = Arg; +} + // The ARM support in lld makes some use of instructions that are not available // on all ARM architectures. Namely: // - Use of BLX instruction for interworking between ARM and Thumb state. @@ -573,6 +613,8 @@ InputSectionBase *ObjFile::createInputSection(co ArrayRef Contents = check(this->getObj().getSectionContents(&Sec)); Attributes.Parse(Contents, /*isLittle*/ Config->EKind == ELF32LEKind); updateSupportedARMFeatures(Attributes); + updateARMVFPArgs(Attributes, this); + // FIXME: Retain the first attribute section we see. The eglibc ARM // dynamic loaders require the presence of an attribute section for dlopen // to work. In a full implementation we would merge all attribute sections. Modified: vendor/lld/dist/ELF/InputSection.cpp ============================================================================== --- vendor/lld/dist/ELF/InputSection.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/InputSection.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -221,8 +221,8 @@ template Defined *InputSectionBase::getEnclosingFunction(uint64_t Offset) { for (Symbol *B : File->getSymbols()) if (Defined *D = dyn_cast(B)) - if (D->Section == this && D->Type == STT_FUNC && - D->Value <= Offset && Offset < D->Value + D->Size) + if (D->Section == this && D->Type == STT_FUNC && D->Value <= Offset && + Offset < D->Value + D->Size) return D; return nullptr; } @@ -671,7 +671,7 @@ static uint64_t getRelocTargetVA(const InputFile *File case R_TLSLD_GOT_FROM_END: return InX::Got->getTlsIndexOff() + A - InX::Got->getSize(); case R_TLSLD_GOT: - return InX::Got->getTlsIndexOff() + A; + return InX::Got->getTlsIndexOff() + A; case R_TLSLD_PC: return InX::Got->getTlsIndexVA() + A - P; } @@ -842,8 +842,7 @@ void InputSectionBase::relocateAlloc(uint8_t *Buf, uin // For each function-defining prologue, find any calls to __morestack, // and replace them with calls to __morestack_non_split. static void switchMorestackCallsToMorestackNonSplit( - llvm::DenseSet& Prologues, - std::vector& MorestackCalls) { + DenseSet &Prologues, std::vector &MorestackCalls) { // If the target adjusted a function's prologue, all calls to // __morestack inside that function should be switched to @@ -873,9 +872,8 @@ static void switchMorestackCallsToMorestackNonSplit( } } -static bool -enclosingPrologueAdjusted(uint64_t Offset, - const llvm::DenseSet &Prologues) { +static bool enclosingPrologueAdjusted(uint64_t Offset, + const DenseSet &Prologues) { for (Defined *F : Prologues) if (F->Value <= Offset && Offset < F->Value + F->Size) return true; @@ -891,7 +889,7 @@ void InputSectionBase::adjustSplitStackFunctionPrologu uint8_t *End) { if (!getFile()->SplitStack) return; - llvm::DenseSet AdjustedPrologues; + DenseSet AdjustedPrologues; std::vector MorestackCalls; for (Relocation &Rel : Relocations) { @@ -1071,8 +1069,7 @@ void MergeInputSection::splitNonStrings(ArrayRef Modified: vendor/lld/dist/ELF/Options.td ============================================================================== --- vendor/lld/dist/ELF/Options.td Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/Options.td Thu Aug 2 17:33:42 2018 (r337145) @@ -58,8 +58,8 @@ defm allow_multiple_definition: B<"allow-multiple-defi "Do not allow multiple definitions (default)">; defm apply_dynamic_relocs: B<"apply-dynamic-relocs", - "Apply dynamic relocations to place", - "Do not apply dynamic relocations to place">; + "Apply link-time values for dynamic relocations", + "Do not apply link-time values for dynamic relocations (default)">; defm as_needed: B<"as-needed", "Only set DT_NEEDED for shared libraries if used", @@ -130,6 +130,10 @@ def error_unresolved_symbols: F<"error-unresolved-symb HelpText<"Report unresolved symbols as errors">; defm exclude_libs: Eq<"exclude-libs", "Exclude static libraries from automatic export">; + +defm execute_only: B<"execute-only", + "Do not mark executable sections readable", + "Mark executable sections readable (default)">; defm export_dynamic: B<"export-dynamic", "Put symbols in the dynamic symbol table", Modified: vendor/lld/dist/ELF/SyntheticSections.cpp ============================================================================== --- vendor/lld/dist/ELF/SyntheticSections.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/SyntheticSections.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -335,7 +335,7 @@ void BuildIdSection::writeBuildId(ArrayRef Bu switch (Config->BuildId) { case BuildIdKind::Fast: computeHash(Buf, [](uint8_t *Dest, ArrayRef Arr) { - write64le(Dest, xxHash64(toStringRef(Arr))); + write64le(Dest, xxHash64(Arr)); }); break; case BuildIdKind::Md5: @@ -1935,6 +1935,23 @@ SymbolTableSection::SymbolTableSection(StringTab this->Entsize = sizeof(Elf_Sym); } +static BssSection *getCommonSec(Symbol *Sym) { + if (!Config->DefineCommon) + if (auto *D = dyn_cast(Sym)) + return dyn_cast_or_null(D->Section); + return nullptr; +} + +static uint32_t getSymSectionIndex(Symbol *Sym) { + if (getCommonSec(Sym)) + return SHN_COMMON; + if (!isa(Sym) || Sym->NeedsPltAddr) + return SHN_UNDEF; + if (const OutputSection *OS = Sym->getOutputSection()) + return OS->SectionIndex >= SHN_LORESERVE ? SHN_XINDEX : OS->SectionIndex; + return SHN_ABS; +} + // Write the internal symbol table contents to the output symbol table. template void SymbolTableSection::writeTo(uint8_t *Buf) { // The first entry is a null entry as per the ELF spec. @@ -1956,23 +1973,8 @@ template void SymbolTableSection::w } ESym->st_name = Ent.StrTabOffset; + ESym->st_shndx = getSymSectionIndex(Ent.Sym); - // Set a section index. - BssSection *CommonSec = nullptr; - if (!Config->DefineCommon) - if (auto *D = dyn_cast(Sym)) - CommonSec = dyn_cast_or_null(D->Section); - if (CommonSec) - ESym->st_shndx = SHN_COMMON; - else if (Sym->NeedsPltAddr) - ESym->st_shndx = SHN_UNDEF; - else if (const OutputSection *OutSec = Sym->getOutputSection()) - ESym->st_shndx = OutSec->SectionIndex; - else if (isa(Sym)) - ESym->st_shndx = SHN_ABS; - else - ESym->st_shndx = SHN_UNDEF; - // Copy symbol size if it is a defined symbol. st_size is not significant // for undefined symbols, so whether copying it or not is up to us if that's // the case. We'll leave it as zero because by not setting a value, we can @@ -1986,7 +1988,7 @@ template void SymbolTableSection::w // st_value is usually an address of a symbol, but that has a // special meaining for uninstantiated common symbols (this can // occur if -r is given). - if (CommonSec) + if (BssSection *CommonSec = getCommonSec(Ent.Sym)) ESym->st_value = CommonSec->Alignment; else ESym->st_value = Sym->getVA(); @@ -2026,6 +2028,44 @@ template void SymbolTableSection::w } } +SymtabShndxSection::SymtabShndxSection() + : SyntheticSection(0, SHT_SYMTAB_SHNDX, 4, ".symtab_shndxr") { + this->Entsize = 4; +} + +void SymtabShndxSection::writeTo(uint8_t *Buf) { + // We write an array of 32 bit values, where each value has 1:1 association + // with an entry in .symtab. If the corresponding entry contains SHN_XINDEX, + // we need to write actual index, otherwise, we must write SHN_UNDEF(0). + Buf += 4; // Ignore .symtab[0] entry. + for (const SymbolTableEntry &Entry : InX::SymTab->getSymbols()) { + if (getSymSectionIndex(Entry.Sym) == SHN_XINDEX) + write32(Buf, Entry.Sym->getOutputSection()->SectionIndex); + Buf += 4; + } +} + +bool SymtabShndxSection::empty() const { + // SHT_SYMTAB can hold symbols with section indices values up to + // SHN_LORESERVE. If we need more, we want to use extension SHT_SYMTAB_SHNDX + // section. Problem is that we reveal the final section indices a bit too + // late, and we do not know them here. For simplicity, we just always create + // a .symtab_shndxr section when the amount of output sections is huge. + size_t Size = 0; + for (BaseCommand *Base : Script->SectionCommands) + if (isa(Base)) + ++Size; + return Size < SHN_LORESERVE; +} + +void SymtabShndxSection::finalizeContents() { + getParent()->Link = InX::SymTab->getParent()->SectionIndex; +} + +size_t SymtabShndxSection::getSize() const { + return InX::SymTab->getNumSymbols() * 4; +} + // .hash and .gnu.hash sections contain on-disk hash tables that map // symbol names to their dynamic symbol table indices. Their purpose // is to help the dynamic linker resolve symbols quickly. If ELF files @@ -3025,6 +3065,7 @@ RelocationBaseSection *InX::RelaIplt; StringTableSection *InX::ShStrTab; StringTableSection *InX::StrTab; SymbolTableBaseSection *InX::SymTab; +SymtabShndxSection *InX::SymTabShndx; template GdbIndexSection *GdbIndexSection::create(); template GdbIndexSection *GdbIndexSection::create(); Modified: vendor/lld/dist/ELF/SyntheticSections.h ============================================================================== --- vendor/lld/dist/ELF/SyntheticSections.h Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/SyntheticSections.h Thu Aug 2 17:33:42 2018 (r337145) @@ -588,6 +588,16 @@ class SymbolTableSection final : public SymbolTableBas void writeTo(uint8_t *Buf) override; }; +class SymtabShndxSection final : public SyntheticSection { +public: + SymtabShndxSection(); + + void writeTo(uint8_t *Buf) override; + size_t getSize() const override; + bool empty() const override; + void finalizeContents() override; +}; + // Outputs GNU Hash section. For detailed explanation see: // https://blogs.oracle.com/ali/entry/gnu_hash_elf_sections class GnuHashTableSection final : public SyntheticSection { @@ -992,6 +1002,7 @@ struct InX { static StringTableSection *ShStrTab; static StringTableSection *StrTab; static SymbolTableBaseSection *SymTab; + static SymtabShndxSection* SymTabShndx; }; template struct In { Modified: vendor/lld/dist/ELF/Writer.cpp ============================================================================== --- vendor/lld/dist/ELF/Writer.cpp Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/ELF/Writer.cpp Thu Aug 2 17:33:42 2018 (r337145) @@ -287,6 +287,7 @@ template static void createSyntheticSecti if (Config->Strip != StripPolicy::All) { InX::StrTab = make(".strtab", false); InX::SymTab = make>(*InX::StrTab); + InX::SymTabShndx = make(); } if (Config->BuildId != BuildIdKind::None) { @@ -409,6 +410,8 @@ template static void createSyntheticSecti if (InX::SymTab) Add(InX::SymTab); + if (InX::SymTabShndx) + Add(InX::SymTabShndx); Add(InX::ShStrTab); if (InX::StrTab) Add(InX::StrTab); @@ -518,7 +521,6 @@ static bool shouldKeepInSymtab(SectionBase *Sec, Strin if (B.isSection()) return false; - if (Config->Discard == DiscardPolicy::None) return true; @@ -1605,6 +1607,15 @@ template void Writer::finalizeSecti if (auto *Sec = dyn_cast(Base)) OutputSections.push_back(Sec); + // Ensure data sections are not mixed with executable sections when + // -execute-only is used. + if (Config->ExecuteOnly) + for (OutputSection *OS : OutputSections) + if (OS->Flags & SHF_EXECINSTR) + for (InputSection *IS : getInputSections(OS)) + if (!(IS->Flags & SHF_EXECINSTR)) + error("-execute-only does not support intermingling data and code"); + // Prefer command line supplied address over other constraints. for (OutputSection *Sec : OutputSections) { auto I = Config->SectionStartMap.find(Sec->Name); @@ -1639,12 +1650,13 @@ template void Writer::finalizeSecti // Dynamic section must be the last one in this list and dynamic // symbol table section (DynSymTab) must be the first one. applySynthetic( - {InX::DynSymTab, InX::Bss, InX::BssRelRo, InX::GnuHashTab, - InX::HashTab, InX::SymTab, InX::ShStrTab, InX::StrTab, - In::VerDef, InX::DynStrTab, InX::Got, InX::MipsGot, - InX::IgotPlt, InX::GotPlt, InX::RelaDyn, InX::RelrDyn, - InX::RelaIplt, InX::RelaPlt, InX::Plt, InX::Iplt, - InX::EhFrameHdr, In::VerSym, In::VerNeed, InX::Dynamic}, + {InX::DynSymTab, InX::Bss, InX::BssRelRo, InX::GnuHashTab, + InX::HashTab, InX::SymTab, InX::SymTabShndx, InX::ShStrTab, + InX::StrTab, In::VerDef, InX::DynStrTab, InX::Got, + InX::MipsGot, InX::IgotPlt, InX::GotPlt, InX::RelaDyn, + InX::RelrDyn, InX::RelaIplt, InX::RelaPlt, InX::Plt, + InX::Iplt, InX::EhFrameHdr, In::VerSym, In::VerNeed, + InX::Dynamic}, [](SyntheticSection *SS) { SS->finalizeContents(); }); if (!Script->HasSectionsCommand && !Config->Relocatable) @@ -1763,6 +1775,8 @@ static bool needsPtLoad(OutputSection *Sec) { static uint64_t computeFlags(uint64_t Flags) { if (Config->Omagic) return PF_R | PF_W | PF_X; + if (Config->ExecuteOnly && (Flags & PF_X)) + return Flags & ~PF_R; if (Config->SingleRoRx && !(Flags & PF_W)) return Flags | PF_X; return Flags; Modified: vendor/lld/dist/docs/index.rst ============================================================================== --- vendor/lld/dist/docs/index.rst Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/docs/index.rst Thu Aug 2 17:33:42 2018 (r337145) @@ -8,7 +8,7 @@ features that are useful for toolchain developers. The linker supports ELF (Unix), PE/COFF (Windows), Mach-O (macOS) and WebAssembly in descending order of completeness. Internally, LLD consists of several different linkers. The ELF port is the one that will be described in -this document. The PE/COFF port is almost complete except the lack of the +this document. The PE/COFF port is complete, including Windows debug info (PDB) support. The WebAssembly port is still a work in progress (See :doc:`WebAssembly`). The Mach-O port is built based on a different architecture than the others. For the details about Mach-O, please Modified: vendor/lld/dist/docs/ld.lld.1 ============================================================================== --- vendor/lld/dist/docs/ld.lld.1 Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/docs/ld.lld.1 Thu Aug 2 17:33:42 2018 (r337145) @@ -3,7 +3,7 @@ .\" .\" This man page documents only lld's ELF linking support, obtained originally .\" from FreeBSD. -.Dd April 28, 2018 +.Dd July 30, 2018 .Dt LD.LLD 1 .Os .Sh NAME @@ -30,6 +30,8 @@ These options are available: .It Fl -allow-multiple-definition Do not error if a symbol is defined multiple times. The first definition will be used. +.It Fl -apply-dynamic-relocs +Apply link-time values for dynamic relocations. .It Fl -as-needed Only set .Dv DT_NEEDED @@ -38,14 +40,14 @@ for shared libraries if used. Set the .Dv DT_AUXILIARY field to the specified name. -.It Fl -Bdynamic +.It Fl -Bdynamic , Fl -dy Link against shared libraries. -.It Fl -Bstatic +.It Fl -Bstatic , Fl -static , Fl -dn Do not link against shared libraries. -.It Fl -Bsymbolic-functions -Bind defined function symbols locally. .It Fl -Bsymbolic Bind defined symbols locally. +.It Fl -Bsymbolic-functions +Bind defined function symbols locally. .It Fl -build-id Ns = Ns Ar value Generate a build ID note. .Ar value @@ -93,7 +95,9 @@ may be .Cm none or .Cm zlib . -.It Fl -define-common +.It Fl -cref +Output cross reference table. +.It Fl -define-common , Fl d Assign space to common symbols. .It Fl -defsym Ns = Ns Ar symbol Ns = Ns Ar expression Define a symbol alias. @@ -107,9 +111,9 @@ or Demangle symbol names. .It Fl -disable-new-dtags Disable new dynamic tags. -.It Fl -discard-all +.It Fl -discard-all , Fl x Delete all local symbols. -.It Fl -discard-locals +.It Fl -discard-locals , Fl X Delete temporary local symbols. .It Fl -discard-none Keep all symbols in the symbol table. @@ -126,7 +130,7 @@ Request creation of section and .Dv PT_GNU_EH_FRAME segment header. -.It Fl -emit-relocs +.It Fl -emit-relocs , Fl q Generate relocations in the output. .It Fl -enable-new-dtags Enable new dynamic tags. @@ -142,21 +146,21 @@ A value of zero indicates that there is no limit. Report unresolved symbols as errors. .It Fl -exclude-libs Ns = Ns Ar value Exclude static libraries from automatic export. +.It Fl -export-dynamic , Fl E +Put symbols in the dynamic symbol table. .It Fl -export-dynamic-symbol Ns = Ns Ar symbol Include .Ar symbol in the dynamic symbol table. -.It Fl -export-dynamic -Put symbols in the dynamic symbol table. .It Fl -fatal-warnings Treat warnings as errors. -.It Fl -filter Ns = Ns Ar value +.It Fl -filter Ns = Ns Ar value , Fl F Ar value Set the .Dv DT_FILTER field to the specified value. .It Fl -fini Ns = Ns Ar symbol Specify a finalizer function. -.It Fl -format Ns = Ns Ar input-format +.It Fl -format Ns = Ns Ar input-format , Fl b Ar input-format Specify the format of the inputs following this option. .Ar input-format may be one of @@ -196,6 +200,14 @@ Set the base address to .Ar value . .It Fl -init Ns = Ns Ar symbol Specify an initializer function. +.It Fl -keep-unique Ns = Ns Ar symbol +Do not fold +.Ar symbol +during ICF. +.It Fl l Ar libName, Fl -library Ns = Ns Ar libName +Root name of library to use. +.It Fl L Ar dir , Fl -library-path Ns = Ns Ar dir +Add a directory to the library search path. .It Fl -lto-aa-pipeline Ns = Ns Ar value AA pipeline to run during LTO. Used in conjunction with @@ -206,15 +218,11 @@ Passes to run during LTO. Optimization level for LTO. .It Fl -lto-partitions Ns = Ns Ar value Number of LTO codegen partitions. -.It Fl L Ar dir -Add a directory to the library search path. -.It Fl l Ar libName -Root name of library to use. -.It Fl -Map Ns = Ns Ar file -Print a link map to -.Ar file . .It Fl m Ar value Set target emulation. +.It Fl -Map Ns = Ns Ar file , Fl M Ar file +Print a link map to +.Ar file . .It Fl -no-as-needed Always set .Dv DT_NEEDED @@ -243,26 +251,18 @@ Report version scripts that refer undefined symbols. Report unresolved symbols even if the linker is creating a shared library. .It Fl -no-whole-archive Restores the default behavior of loading archive members. -.It Fl -noinhibit-exec -Retain the executable output file whenever it is still usable. .It Fl -no-pie Do not create a position independent executable. +.It Fl -noinhibit-exec +Retain the executable output file whenever it is still usable. .It Fl -nostdlib Only search directories specified on the command line. -.It Fl -oformat Ns = Ns Ar format -Specify the format for the output object file. -The only supported -.Ar format -is -.Cm binary , -which produces output with no ELF header. -.It Fl -omagic -Set the text and data sections to be readable and writable. -.It Fl -opt-remarks-filename Ar file -Write optimization remarks in YAML format to -.Ar file . -.It Fl -opt-remarks-with-hotness -Include hotness information in the optimization remarks file. +.It Fl o Ar path +Write the output executable, library, or object to +.Ar path . +If not specified, +.Dv a.out +is used as a default. .It Fl O Ns Ar value Optimize output file size. .Ar value @@ -279,12 +279,20 @@ Enable string tail merging. .Pp .Fl O Ns Cm 1 is the default. -.It Fl o Ar path -Write the output executable, library, or object to -.Ar path . -If not specified, -.Dv a.out -is used as a default. +.It Fl -oformat Ns = Ns Ar format +Specify the format for the output object file. +The only supported +.Ar format +is +.Cm binary , +which produces output with no ELF header. +.It Fl -omagic , Fl N +Set the text and data sections to be readable and writable. +.It Fl -opt-remarks-filename Ar file +Write optimization remarks in YAML format to +.Ar file . +.It Fl -opt-remarks-with-hotness +Include hotness information in the optimization remarks file. .It Fl -pie Create a position independent executable. .It Fl -print-gc-sections @@ -300,13 +308,13 @@ and .It Fl -pop-state Undo the effect of .Fl -push-state. -.It Fl -relocatable +.It Fl -relocatable , Fl r Create relocatable object file. .It Fl -reproduce Ns = Ns Ar value Dump linker invocation and input files for debugging. .It Fl -retain-symbols-file Ns = Ns Ar file Retain only the symbols listed in the file. -.It Fl -rpath Ns = Ns Ar value +.It Fl -rpath Ns = Ns Ar value , Fl R Ar value Add a .Dv DT_RUNPATH to the output. @@ -316,14 +324,14 @@ The supported values are .Cm windows and .Cm posix . -.It Fl -script Ns = Ns Ar file +.It Fl -script Ns = Ns Ar file , Fl T Ar file Read linker script from .Ar file . -.It Fl -section-start Ns = Ar section Ns = Ns Ar address +.It Fl -section-start Ns = Ns Ar section Ns = Ns Ar address Set address of section. -.It Fl -shared +.It Fl -shared , Fl -Bsharable Build a shared object. -.It Fl -soname Ns = Ns Ar value +.It Fl -soname Ns = Ns Ar value , Fl h Ar value Set .Dv DT_SONAME to @@ -333,9 +341,9 @@ Specifies sections sorting rule when linkerscript is u .It Fl -start-lib Start a grouping of objects that should be treated as if they were together in an archive. -.It Fl -strip-all +.It Fl -strip-all , Fl s Strip all symbols. -.It Fl -strip-debug +.It Fl -strip-debug , Fl S Strip debugging information. .It Fl -symbol-ordering-file Ns = Ns Ar file Lay out sections in the order specified by @@ -376,6 +384,12 @@ Same as with .Li .data as the sectionname. +.It Fl -Ttext Ns = Ns Ar value +Same as +.Fl -section-start +with +.Li .text +as the sectionname. .It Fl -thinlto-cache-dir Ns = Ns Ar value Path to ThinLTO cached object file directory. .It Fl -thinlto-cache-policy Ns = Ns Ar value @@ -385,33 +399,27 @@ Number of ThinLTO jobs. .It Fl -threads Run the linker multi-threaded. This option is enabled by default. -.It Fl -trace-symbol Ns = Ns Ar symbol -Trace references to -.Ar symbol . .It Fl -trace Print the names of the input files. -.It Fl -Ttext Ns = Ns Ar value -Same as -.Fl -section-start -with -.Li .text -as the sectionname. -.It Fl -undefined Ns = Ns Ar symbol +.It Fl -trace-symbol Ns = Ns Ar symbol , Fl y Ar symbol +Trace references to +.Ar symbol . +.It Fl -undefined Ns = Ns Ar symbol , Fl u Ar symbol Force .Ar symbol to be an undefined symbol during linking. .It Fl -unresolved-symbols Ns = Ns Ar value Determine how to handle unresolved symbols. +.It Fl v +Display the version number and proceed with linking if object files are +specified. +.It Fl V , Fl -version +Display the version number and exit. .It Fl -verbose Verbose mode. .It Fl -version-script Ns = Ns Ar file Read version script from .Ar file . -.It Fl V , Fl -version -Display the version number and exit. -.It Fl v -Display the version number and proceed with linking if object files are -specified. .It Fl -warn-backrefs Warn about reverse or cyclic dependencies to or between static archives. This can be used to ensure linker invocation remains compatible with @@ -432,6 +440,10 @@ Make the main stack executable. Stack permissions are recorded in the .Dv PT_GNU_STACK segment. +.It Cm initfirst +Sets the +.Dv DF_1_INITFIRST +flag to indicate the module should be initialized first. .It Cm muldefs Do not error if a symbol is defined multiple times. The first definition will be used. Modified: vendor/lld/dist/docs/windows_support.rst ============================================================================== --- vendor/lld/dist/docs/windows_support.rst Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/docs/windows_support.rst Thu Aug 2 17:33:42 2018 (r337145) @@ -60,6 +60,13 @@ Debug info link.exe. However, LLD does not support /DEBUG:FASTLINK. +Downloading LLD +=============== + +The Windows version of LLD is included in the `pre-built binaries of LLVM's +releases `_ and in the `LLVM Snapshot +Builds `_. + Building LLD ============ Added: vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-base.s ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-base.s Thu Aug 2 17:33:42 2018 (r337145) @@ -0,0 +1,16 @@ + .arch armv7-a + .eabi_attribute 20, 1 + .eabi_attribute 21, 1 + .eabi_attribute 23, 3 + .eabi_attribute 24, 1 + .eabi_attribute 25, 1 + .eabi_attribute 26, 2 + .eabi_attribute 30, 6 + .eabi_attribute 34, 1 + .eabi_attribute 18, 4 + .eabi_attribute 28, 0 // Tag_ABI_VFP_args = 0 (AAPCS, Base variant) + + .syntax unified + .global f0 + .type f0, %function +f0: bx lr Added: vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-compat.s ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-compat.s Thu Aug 2 17:33:42 2018 (r337145) @@ -0,0 +1,16 @@ + .arch armv7-a + .eabi_attribute 20, 1 + .eabi_attribute 21, 1 + .eabi_attribute 23, 3 + .eabi_attribute 24, 1 + .eabi_attribute 25, 1 + .eabi_attribute 26, 2 + .eabi_attribute 30, 6 + .eabi_attribute 34, 1 + .eabi_attribute 18, 4 + .eabi_attribute 28, 3 // Tag_ABI_VFP_args = 3 (Compatible with all) + + .syntax unified + .global f3 + .type f3, %function +f3: bx lr Added: vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-toolchain.s ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-toolchain.s Thu Aug 2 17:33:42 2018 (r337145) @@ -0,0 +1,15 @@ + .arch armv7-a + .eabi_attribute 20, 1 + .eabi_attribute 21, 1 + .eabi_attribute 23, 3 + .eabi_attribute 24, 1 + .eabi_attribute 25, 1 + .eabi_attribute 26, 2 + .eabi_attribute 30, 6 + .eabi_attribute 34, 1 + .eabi_attribute 18, 4 + .eabi_attribute 28, 2 // Tag_ABI_VFP_args = 2 (Toolchain specific) + .syntax unified + .global f2 + .type f1, %function +f2: bx lr Added: vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-vfp.s ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/lld/dist/test/ELF/Inputs/arm-vfp-arg-vfp.s Thu Aug 2 17:33:42 2018 (r337145) @@ -0,0 +1,15 @@ + .arch armv7-a + .eabi_attribute 20, 1 + .eabi_attribute 21, 1 + .eabi_attribute 23, 3 + .eabi_attribute 24, 1 + .eabi_attribute 25, 1 + .eabi_attribute 26, 2 + .eabi_attribute 30, 6 + .eabi_attribute 34, 1 + .eabi_attribute 18, 4 + .eabi_attribute 28, 1 // Tag_ABI_VFP_args = 1 (AAPCS, VFP variant) + .syntax unified + .global f1 + .type f1, %function +f1: bx lr Added: vendor/lld/dist/test/ELF/Inputs/icf-absolute2.s ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/lld/dist/test/ELF/Inputs/icf-absolute2.s Thu Aug 2 17:33:42 2018 (r337145) @@ -0,0 +1,3 @@ +.globl a1, a2 +a1 = 1 +a2 = 2 Modified: vendor/lld/dist/test/ELF/arm-eabi-version.s ============================================================================== --- vendor/lld/dist/test/ELF/arm-eabi-version.s Thu Aug 2 17:33:38 2018 (r337144) +++ vendor/lld/dist/test/ELF/arm-eabi-version.s Thu Aug 2 17:33:42 2018 (r337145) @@ -9,6 +9,7 @@ _start: bx lr // CHECK: Flags [ +// CHECK-NEXT: 0x200 // CHECK-NEXT: 0x1000000 // CHECK-NEXT: 0x4000000 // CHECK-NEXT: ] Added: vendor/lld/dist/test/ELF/arm-tag-vfp-args-errs.s ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/lld/dist/test/ELF/arm-tag-vfp-args-errs.s Thu Aug 2 17:33:42 2018 (r337145) @@ -0,0 +1,29 @@ *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:33:54 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44B00106D825; Thu, 2 Aug 2018 17:33:54 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 58BFB728B8; Thu, 2 Aug 2018 17:33:51 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AEAB714D38; Thu, 2 Aug 2018 17:33:50 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HXoD2029732; Thu, 2 Aug 2018 17:33:50 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXo1l029731; Thu, 2 Aug 2018 17:33:50 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXo1l029731@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:50 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337146 - vendor/lld/lld-trunk-r338536 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lld/lld-trunk-r338536 X-SVN-Commit-Revision: 337146 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:33:54 -0000 Author: dim Date: Thu Aug 2 17:33:50 2018 New Revision: 337146 URL: https://svnweb.freebsd.org/changeset/base/337146 Log: Tag lld trunk r338536. Added: vendor/lld/lld-trunk-r338536/ - copied from r337145, vendor/lld/dist/ From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:34:08 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E64D106D887; Thu, 2 Aug 2018 17:34:08 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 33DB472AF4; Thu, 2 Aug 2018 17:34:08 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EF90A14D3A; Thu, 2 Aug 2018 17:34:07 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HY7el029881; Thu, 2 Aug 2018 17:34:07 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HY7h4029880; Thu, 2 Aug 2018 17:34:07 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021734.w72HY7h4029880@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:34:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337148 - vendor/lldb/lldb-trunk-r338536 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lldb/lldb-trunk-r338536 X-SVN-Commit-Revision: 337148 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:34:08 -0000 Author: dim Date: Thu Aug 2 17:34:07 2018 New Revision: 337148 URL: https://svnweb.freebsd.org/changeset/base/337148 Log: Tag lldb trunk r338536. Added: vendor/lldb/lldb-trunk-r338536/ - copied from r337147, vendor/lldb/dist/ From owner-svn-src-vendor@freebsd.org Thu Aug 2 17:34:07 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29664106D87E; Thu, 2 Aug 2018 17:34:07 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 96A7072AE9; Thu, 2 Aug 2018 17:34:05 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B095C14D39; Thu, 2 Aug 2018 17:34:04 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72HY4Ts029832; Thu, 2 Aug 2018 17:34:04 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72HXtup029781; Thu, 2 Aug 2018 17:33:55 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808021733.w72HXtup029781@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Thu, 2 Aug 2018 17:33:55 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337147 - in vendor/lldb/dist: include/lldb/Utility packages/Python/lldbsuite/test packages/Python/lldbsuite/test/api/multiple-debuggers packages/Python/lldbsuite/test/api/multithreaded... X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/lldb/dist: include/lldb/Utility packages/Python/lldbsuite/test packages/Python/lldbsuite/test/api/multiple-debuggers packages/Python/lldbsuite/test/api/multithreaded packages/Python/lldbsuit... X-SVN-Commit-Revision: 337147 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 17:34:07 -0000 Author: dim Date: Thu Aug 2 17:33:54 2018 New Revision: 337147 URL: https://svnweb.freebsd.org/changeset/base/337147 Log: Vendor import of lldb trunk r338536: https://llvm.org/svn/llvm-project/lldb/trunk@338536 Added: vendor/lldb/dist/unittests/Utility/StreamTest.cpp (contents, props changed) Modified: vendor/lldb/dist/include/lldb/Utility/CompletionRequest.h vendor/lldb/dist/include/lldb/Utility/Stream.h vendor/lldb/dist/packages/Python/lldbsuite/test/api/multiple-debuggers/TestMultipleDebuggers.py vendor/lldb/dist/packages/Python/lldbsuite/test/api/multithreaded/TestMultithreaded.py vendor/lldb/dist/packages/Python/lldbsuite/test/decorators.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-function/TestCallStdStringFunction.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-restarts/TestCallThatRestarts.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-throws/TestCallThatThrows.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/char/TestExprsChar.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/fixits/TestFixIts.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/issue_11588/Test11588.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/pr35310/TestExprsBug35310.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/save_jit_objects/TestSaveJITObjects.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/unwind_expression/TestUnwindExpression.py vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/xvalue/TestXValuePrinting.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/address_breakpoints/TestBadAddressBreakpoints.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/auto_continue/TestBreakpointAutoContinue.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_locations/TestBreakpointLocations.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_names/TestBreakpointNames.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/hardware_breakpoints/hardware_breakpoint_on_multiple_threads/TestHWBreakMultiThread.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/serialize/TestBreakpointSerialization.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/step_over_breakpoint/TestStepOverBreakpoint.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/TestCompletion.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/main.cpp vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/bitset/TestDataFormatterLibcxxBitset.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/multiset/TestDataFormatterLibcxxMultiSet.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/tuple/TestDataFormatterStdTuple.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/exec/TestExec.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame-language/TestGuessLanguage.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame_var/TestFrameVar.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/TestTargetXMLArch.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/gdbclientutils.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/history/TestHistoryRecall.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_using_paths/TestLoadUsingPaths.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/pre_run_dylibs/TestPreRunDylibs.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/register/intel_avx/TestZMMRegister.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/return-value/TestReturnValue.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/create_during_step/TestCreateDuringStep.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/num_threads/TestNumThreads.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/step_until/TestStepUntil.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/watchpoint/multi_watchpoint_slots/TestWatchpointMultipleSlots.py vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/watchpoint/watchpoint_disable/TestWatchpointDisable.py vendor/lldb/dist/packages/Python/lldbsuite/test/lang/c/find_struct_type/TestFindStructTypes.py vendor/lldb/dist/packages/Python/lldbsuite/test/lang/cpp/dynamic-value-same-basename/TestDynamicValueSameBase.py vendor/lldb/dist/packages/Python/lldbsuite/test/lang/cpp/incomplete-types/TestCppIncompleteTypes.py vendor/lldb/dist/packages/Python/lldbsuite/test/lang/cpp/llvm-style/TestLLVMStyle.py vendor/lldb/dist/packages/Python/lldbsuite/test/lang/cpp/scope/TestCppScope.py vendor/lldb/dist/packages/Python/lldbsuite/test/lang/cpp/trivial_abi/TestTrivialABI.py vendor/lldb/dist/packages/Python/lldbsuite/test/lldbdwarf.py vendor/lldb/dist/packages/Python/lldbsuite/test/lldbtest.py vendor/lldb/dist/packages/Python/lldbsuite/test/lldbutil.py vendor/lldb/dist/packages/Python/lldbsuite/test/macosx/find-app-in-bundle/TestFindAppInBundle.py vendor/lldb/dist/packages/Python/lldbsuite/test/macosx/find-dsym/bundle-with-dot-in-filename/TestBundleWithDotInFilename.py vendor/lldb/dist/packages/Python/lldbsuite/test/macosx/find-dsym/deep-bundle/TestDeepBundle.py vendor/lldb/dist/packages/Python/lldbsuite/test/macosx/thread-names/TestInterruptThreadNames.py vendor/lldb/dist/packages/Python/lldbsuite/test/python_api/name_lookup/TestNameLookup.py vendor/lldb/dist/packages/Python/lldbsuite/test/python_api/sbdata/TestSBData.py vendor/lldb/dist/packages/Python/lldbsuite/test/sample_test/TestSampleTest.py vendor/lldb/dist/packages/Python/lldbsuite/test/settings/TestSettings.py vendor/lldb/dist/packages/Python/lldbsuite/test/tools/lldb-mi/startup_options/TestMiStartupOptions.py vendor/lldb/dist/packages/Python/lldbsuite/test/tools/lldb-server/TestGdbRemote_qThreadStopInfo.py vendor/lldb/dist/packages/Python/lldbsuite/test/tools/lldb-server/TestLldbGdbServer.py vendor/lldb/dist/source/Commands/CommandCompletions.cpp vendor/lldb/dist/source/Commands/CommandObjectCommands.cpp vendor/lldb/dist/source/Commands/CommandObjectFrame.cpp vendor/lldb/dist/source/Commands/CommandObjectMultiword.cpp vendor/lldb/dist/source/Commands/CommandObjectPlatform.cpp vendor/lldb/dist/source/Commands/CommandObjectPlugin.cpp vendor/lldb/dist/source/Commands/CommandObjectProcess.cpp vendor/lldb/dist/source/Commands/CommandObjectSettings.cpp vendor/lldb/dist/source/Commands/CommandObjectTarget.cpp vendor/lldb/dist/source/Core/FormatEntity.cpp vendor/lldb/dist/source/Core/IOHandler.cpp vendor/lldb/dist/source/Interpreter/CommandInterpreter.cpp vendor/lldb/dist/source/Interpreter/CommandObject.cpp vendor/lldb/dist/source/Interpreter/CommandObjectRegexCommand.cpp vendor/lldb/dist/source/Interpreter/OptionValue.cpp vendor/lldb/dist/source/Interpreter/OptionValueArch.cpp vendor/lldb/dist/source/Interpreter/OptionValueBoolean.cpp vendor/lldb/dist/source/Interpreter/OptionValueEnumeration.cpp vendor/lldb/dist/source/Interpreter/OptionValueFileSpec.cpp vendor/lldb/dist/source/Interpreter/OptionValueUUID.cpp vendor/lldb/dist/source/Interpreter/Options.cpp vendor/lldb/dist/source/Plugins/Platform/Android/PlatformAndroid.cpp vendor/lldb/dist/source/Plugins/Platform/Windows/PlatformWindows.cpp vendor/lldb/dist/source/Plugins/SymbolFile/DWARF/DWARFCompileUnit.h vendor/lldb/dist/source/Plugins/SymbolFile/DWARF/DWARFUnit.h vendor/lldb/dist/source/Symbol/Variable.cpp vendor/lldb/dist/source/Utility/ArchSpec.cpp vendor/lldb/dist/source/Utility/CompletionRequest.cpp vendor/lldb/dist/source/Utility/Stream.cpp vendor/lldb/dist/unittests/Utility/CMakeLists.txt vendor/lldb/dist/unittests/Utility/CompletionRequestTest.cpp Modified: vendor/lldb/dist/include/lldb/Utility/CompletionRequest.h ============================================================================== --- vendor/lldb/dist/include/lldb/Utility/CompletionRequest.h Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/include/lldb/Utility/CompletionRequest.h Thu Aug 2 17:33:54 2018 (r337147) @@ -13,6 +13,7 @@ #include "lldb/Utility/Args.h" #include "lldb/Utility/StringList.h" #include "llvm/ADT/StringRef.h" +#include "llvm/ADT/StringSet.h" namespace lldb_private { @@ -77,9 +78,30 @@ class CompletionRequest { (public) void SetWordComplete(bool v) { m_word_complete = v; } - /// The array of matches returned. - StringList &GetMatches() { return *m_matches; } + /// Adds a possible completion string. If the completion was already + /// suggested before, it will not be added to the list of results. A copy of + /// the suggested completion is stored, so the given string can be free'd + /// afterwards. + /// + /// @param match The suggested completion. + void AddCompletion(llvm::StringRef completion) { + // Add the completion if we haven't seen the same value before. + if (m_match_set.insert(completion).second) + m_matches->AppendString(completion); + } + /// Adds multiple possible completion strings. + /// + /// \param completions The list of completions. + /// + /// @see AddCompletion + void AddCompletions(const StringList &completions) { + for (std::size_t i = 0; i < completions.GetSize(); ++i) + AddCompletion(completions.GetStringAtIndex(i)); + } + + std::size_t GetNumberOfMatches() const { return m_matches->GetSize(); } + llvm::StringRef GetCursorArgument() const { return GetParsedLine().GetArgumentAtIndex(GetCursorIndex()); } @@ -111,8 +133,15 @@ class CompletionRequest { (public) /// \btrue if this is a complete option value (a space will be inserted /// after the completion.) \bfalse otherwise. bool m_word_complete = false; - // We don't own the list. + + // Note: This list is kept private. This is by design to prevent that any + // completion depends on any already computed completion from another backend. + // Note: We don't own the list. It's owned by the creator of the + // CompletionRequest object. StringList *m_matches; + + /// List of added completions so far. Used to filter out duplicates. + llvm::StringSet<> m_match_set; }; } // namespace lldb_private Modified: vendor/lldb/dist/include/lldb/Utility/Stream.h ============================================================================== --- vendor/lldb/dist/include/lldb/Utility/Stream.h Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/include/lldb/Utility/Stream.h Thu Aug 2 17:33:54 2018 (r337147) @@ -524,8 +524,6 @@ class Stream { (public) //------------------------------------------------------------------ size_t PutULEB128(uint64_t uval); - static void UnitTest(Stream *s); - protected: //------------------------------------------------------------------ // Member variables Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/api/multiple-debuggers/TestMultipleDebuggers.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/api/multiple-debuggers/TestMultipleDebuggers.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/api/multiple-debuggers/TestMultipleDebuggers.py Thu Aug 2 17:33:54 2018 (r337147) @@ -18,7 +18,7 @@ class TestMultipleSimultaneousDebuggers(TestBase): mydir = TestBase.compute_mydir(__file__) - # This test case fails non-deterministically. + # This test case fails non-deterministically. @skipIfNoSBHeaders @expectedFailureAll(bugnumber="llvm.org/pr20282") def test_multiple_debuggers(self): Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/api/multithreaded/TestMultithreaded.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/api/multithreaded/TestMultithreaded.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/api/multithreaded/TestMultithreaded.py Thu Aug 2 17:33:54 2018 (r337147) @@ -16,7 +16,7 @@ import subprocess class SBBreakpointCallbackCase(TestBase): NO_DEBUG_INFO_TESTCASE = True - + def setUp(self): TestBase.setUp(self) self.generateSource('driver.cpp') Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/decorators.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/decorators.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/decorators.py Thu Aug 2 17:33:54 2018 (r337147) @@ -791,7 +791,7 @@ def skipIfLLVMTargetMissing(target): if targets.GetItemAtIndex(i).GetStringValue(99) == target: found = True break - + return unittest2.skipIf(not found, "requires " + target) # Call sysctl on darwin to see if a specified hardware feature is available on this machine. Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-function/TestCallStdStringFunction.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-function/TestCallStdStringFunction.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-function/TestCallStdStringFunction.py Thu Aug 2 17:33:54 2018 (r337147) @@ -46,7 +46,7 @@ class ExprCommandCallFunctionTestCase(TestBase): # Calling this function now succeeds, but we follow the typedef return type through to # const char *, and thus don't invoke the Summary formatter. - # clang's libstdc++ on ios arm64 inlines std::string::c_str() always; + # clang's libstdc++ on ios arm64 inlines std::string::c_str() always; # skip this part of the test. triple = self.dbg.GetSelectedPlatform().GetTriple() do_cstr_test = True Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-restarts/TestCallThatRestarts.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-restarts/TestCallThatRestarts.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-restarts/TestCallThatRestarts.py Thu Aug 2 17:33:54 2018 (r337147) @@ -48,7 +48,7 @@ class ExprCommandThatRestartsTestCase(TestBase): "Restored the zeroth frame correctly") def call_function(self): - (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, 'Stop here in main.', self.main_source_spec) # Make sure the SIGCHLD behavior is pass/no-stop/no-notify: Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-throws/TestCallThatThrows.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-throws/TestCallThatThrows.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/call-throws/TestCallThatThrows.py Thu Aug 2 17:33:54 2018 (r337147) @@ -37,7 +37,7 @@ class ExprCommandWithThrowTestCase(TestBase): def call_function(self): """Test calling function that throws.""" - (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, 'I am about to throw.', self.main_source_spec) options = lldb.SBExpressionOptions() Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/char/TestExprsChar.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/char/TestExprsChar.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/char/TestExprsChar.py Thu Aug 2 17:33:54 2018 (r337147) @@ -22,7 +22,7 @@ class ExprCharTestCase(TestBase): """These basic expression commands should work as expected.""" self.build(dictionary=dictionary) - (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, '// Break here', self.main_source_spec) frame = thread.GetFrameAtIndex(0) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/fixits/TestFixIts.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/fixits/TestFixIts.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/fixits/TestFixIts.py Thu Aug 2 17:33:54 2018 (r337147) @@ -37,7 +37,7 @@ class ExprCommandWithFixits(TestBase): def try_expressions(self): """Test calling expressions with errors that can be fixed by the FixIts.""" - (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, 'Stop here to evaluate expressions', self.main_source_spec) options = lldb.SBExpressionOptions() Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/issue_11588/Test11588.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/issue_11588/Test11588.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/issue_11588/Test11588.py Thu Aug 2 17:33:54 2018 (r337147) @@ -32,8 +32,8 @@ class Issue11581TestCase(TestBase): """valobj.AddressOf() should return correct values.""" self.build() - (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, - 'Set breakpoint here.', + (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + 'Set breakpoint here.', lldb.SBFileSpec("main.cpp", False)) self.runCmd("command script import --allow-reload s11588.py") self.runCmd( Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/pr35310/TestExprsBug35310.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/pr35310/TestExprsBug35310.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/pr35310/TestExprsBug35310.py Thu Aug 2 17:33:54 2018 (r337147) @@ -25,7 +25,7 @@ class ExprBug35310(TestBase): """ self.build() - (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, '// Break here', self.main_source_spec) frame = thread.GetFrameAtIndex(0) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/save_jit_objects/TestSaveJITObjects.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/save_jit_objects/TestSaveJITObjects.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/save_jit_objects/TestSaveJITObjects.py Thu Aug 2 17:33:54 2018 (r337147) @@ -16,7 +16,7 @@ class SaveJITObjectsTestCase(TestBase): def enumerateJITFiles(self): return [f for f in os.listdir(self.getBuildDir()) if f.startswith("jit")] - + def countJITFiles(self): return len(self.enumerateJITFiles()) @@ -31,7 +31,7 @@ class SaveJITObjectsTestCase(TestBase): os.chdir(self.getBuildDir()) src_file = "main.c" src_file_spec = lldb.SBFileSpec(src_file) - + (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint( self, "break", src_file_spec) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/unwind_expression/TestUnwindExpression.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/unwind_expression/TestUnwindExpression.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/unwind_expression/TestUnwindExpression.py Thu Aug 2 17:33:54 2018 (r337147) @@ -23,7 +23,7 @@ class UnwindFromExpressionTest(TestBase): def build_and_run_to_bkpt(self): self.build() - (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + (target, process, self.thread, bkpt) = lldbutil.run_to_source_breakpoint(self, "// Set a breakpoint here to get started", self.main_spec) # Next set a breakpoint in this function, set up Expression options to stop on Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/xvalue/TestXValuePrinting.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/xvalue/TestXValuePrinting.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/expression_command/xvalue/TestXValuePrinting.py Thu Aug 2 17:33:54 2018 (r337147) @@ -22,7 +22,7 @@ class ExprXValuePrintingTestCase(TestBase): """Printing an xvalue should work.""" self.build(dictionary=dictionary) - (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, '// Break here', self.main_source_spec) frame = thread.GetFrameAtIndex(0) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/address_breakpoints/TestBadAddressBreakpoints.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/address_breakpoints/TestBadAddressBreakpoints.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/address_breakpoints/TestBadAddressBreakpoints.py Thu Aug 2 17:33:54 2018 (r337147) @@ -31,7 +31,7 @@ class BadAddressBreakpointTestCase(TestBase): def address_breakpoints(self): """Test that breakpoints set on a bad address say they are bad.""" target, process, thread, bkpt = \ - lldbutil.run_to_source_breakpoint(self, + lldbutil.run_to_source_breakpoint(self, "Set a breakpoint here", lldb.SBFileSpec("main.c")) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/auto_continue/TestBreakpointAutoContinue.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/auto_continue/TestBreakpointAutoContinue.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/auto_continue/TestBreakpointAutoContinue.py Thu Aug 2 17:33:54 2018 (r337147) @@ -34,17 +34,17 @@ class BreakpointAutoContinue(TestBase): self.build() self.auto_continue_location() - def make_target_and_bkpt(self, additional_options=None, num_expected_loc=1, + def make_target_and_bkpt(self, additional_options=None, num_expected_loc=1, pattern="Set a breakpoint here"): exe = self.getBuildArtifact("a.out") self.target = self.dbg.CreateTarget(exe) self.assertTrue(self.target.IsValid(), "Target is not valid") - + extra_options_txt = "--auto-continue 1 " if additional_options: extra_options_txt += additional_options - bpno = lldbutil.run_break_set_by_source_regexp(self, pattern, - extra_options = extra_options_txt, + bpno = lldbutil.run_break_set_by_source_regexp(self, pattern, + extra_options = extra_options_txt, num_expected_locations = num_expected_loc) return bpno Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_locations/TestBreakpointLocations.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_locations/TestBreakpointLocations.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_locations/TestBreakpointLocations.py Thu Aug 2 17:33:54 2018 (r337147) @@ -46,12 +46,12 @@ class BreakpointLocationsTestCase(TestBase): self.assertTrue(target, "Target %s is not valid"%(exe)) # This should create a breakpoint with 3 locations. - + bkpt = target.BreakpointCreateByLocation("main.c", self.line) # The breakpoint list should show 3 locations. self.assertEqual(bkpt.GetNumLocations(), 3, "Wrong number of locations") - + self.expect( "breakpoint list -f", "Breakpoint locations shown correctly", @@ -62,7 +62,7 @@ class BreakpointLocationsTestCase(TestBase): "where = a.out`func_inlined .+unresolved, hit count = 0", "where = a.out`main .+\[inlined\].+unresolved, hit count = 0"]) - return bkpt + return bkpt def shadowed_bkpt_cond_test(self): """Test that options set on the breakpoint and location behave correctly.""" @@ -106,7 +106,7 @@ class BreakpointLocationsTestCase(TestBase): commands = ["AAAAAA", "BBBBBB", "CCCCCC"] str_list = lldb.SBStringList() str_list.AppendList(commands, len(commands)) - + bkpt.SetCommandLineCommands(str_list) cmd_list = lldb.SBStringList() bkpt.GetCommandLineCommands(cmd_list) @@ -123,7 +123,7 @@ class BreakpointLocationsTestCase(TestBase): bkpt.location[1].GetCommandLineCommands(loc_cmd_list) loc_list_size = loc_list.GetSize() - + # Check that the location has the right commands: self.assertEqual(loc_cmd_list.GetSize() , loc_list_size, "Added the right number of commands to location") for i in range(0,loc_list_size): Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_names/TestBreakpointNames.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_names/TestBreakpointNames.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/breakpoint_names/TestBreakpointNames.py Thu Aug 2 17:33:54 2018 (r337147) @@ -64,7 +64,7 @@ class BreakpointNames(TestBase): self.target = self.dbg.CreateTarget(exe) self.assertTrue(self.target, VALID_TARGET) self.main_file_spec = lldb.SBFileSpec(os.path.join(self.getSourceDir(), "main.c")) - + def check_name_in_target(self, bkpt_name): name_list = lldb.SBStringList() self.target.GetBreakpointNames(name_list) @@ -74,7 +74,7 @@ class BreakpointNames(TestBase): found_it = True break self.assertTrue(found_it, "Didn't find the name %s in the target's name list:"%(bkpt_name)) - + def setUp(self): # Call super's setUp(). TestBase.setUp(self) @@ -107,14 +107,14 @@ class BreakpointNames(TestBase): matches = bkpt.MatchesName(bkpt_name) self.assertTrue(matches, "We didn't match the name we just set") - + # Make sure we don't match irrelevant names: matches = bkpt.MatchesName("NotABreakpoint") self.assertTrue(not matches, "We matched a name we didn't set.") # Make sure the name is also in the target: self.check_name_in_target(bkpt_name) - + # Add another name, make sure that works too: bkpt.AddName(other_bkpt_name) @@ -132,7 +132,7 @@ class BreakpointNames(TestBase): bkpt.GetNames(name_list) num_names = name_list.GetSize() self.assertTrue(num_names == 1, "Name list has %d items, expected 1."%(num_names)) - + name = name_list.GetStringAtIndex(0) self.assertTrue(name == other_bkpt_name, "Remaining name was: %s expected %s."%(name, other_bkpt_name)) @@ -156,7 +156,7 @@ class BreakpointNames(TestBase): def do_check_using_names(self): """Use Python APIs to check names work in place of breakpoint ID's.""" - + bkpt = self.target.BreakpointCreateByLocation(self.main_file_spec, 10) bkpt_name = "ABreakpoint" other_bkpt_name= "_AnotherBreakpoint" @@ -255,12 +255,12 @@ class BreakpointNames(TestBase): bp_name.SetAutoContinue(new_auto_continue) self.assertEqual(bp_name.GetAutoContinue(), new_auto_continue, "Couldn't change auto-continue on the name") self.assertEqual(bkpt.GetAutoContinue(), new_auto_continue, "Option didn't propagate to the breakpoint.") - + # Now make this same breakpoint name - but from the command line - cmd_str = "breakpoint name configure %s -o %d -i %d -c '%s' -G %d -t %d -x %d -T '%s' -q '%s' -H '%s'"%(cl_bp_name_string, - self.is_one_shot, - self.ignore_count, - self.condition, + cmd_str = "breakpoint name configure %s -o %d -i %d -c '%s' -G %d -t %d -x %d -T '%s' -q '%s' -H '%s'"%(cl_bp_name_string, + self.is_one_shot, + self.ignore_count, + self.condition, self.auto_continue, self.tid, self.tidx, @@ -269,7 +269,7 @@ class BreakpointNames(TestBase): self.help_string) for cmd in self.cmd_list: cmd_str += " -C '%s'"%(cmd) - + self.runCmd(cmd_str, check=True) # Now look up this name again and check its options: cl_name = lldb.SBBreakpointName(self.target, cl_bp_name_string) @@ -280,14 +280,14 @@ class BreakpointNames(TestBase): new_help = "I do something even more interesting" cl_name.SetHelpString(new_help) self.assertEqual(new_help, cl_name.GetHelpString(), "SetHelpString didn't") - + # We should have three names now, make sure the target can list them: name_list = lldb.SBStringList() self.target.GetBreakpointNames(name_list) for name_string in [self.bp_name_string, other_bp_name_string, cl_bp_name_string]: self.assertTrue(name_string in name_list, "Didn't find %s in names"%(name_string)) - # Delete the name from the current target. Make sure that works and deletes the + # Delete the name from the current target. Make sure that works and deletes the # name from the breakpoint as well: self.target.DeleteBreakpointName(self.bp_name_string) name_list.Clear() @@ -305,7 +305,7 @@ class BreakpointNames(TestBase): self.assertTrue(self.bp_name_string not in name_list, "Didn't delete %s from the dummy target"%(self.bp_name_string)) # Also make sure the name got removed from breakpoints holding it: self.assertFalse(bkpt.MatchesName(self.bp_name_string), "Didn't remove the name from the breakpoint.") - + def check_permission_results(self, bp_name): self.assertEqual(bp_name.GetAllowDelete(), False, "Didn't set allow delete.") protected_bkpt = self.target.BreakpointCreateByLocation(self.main_file_spec, 10) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/hardware_breakpoints/hardware_breakpoint_on_multiple_threads/TestHWBreakMultiThread.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/hardware_breakpoints/hardware_breakpoint_on_multiple_threads/TestHWBreakMultiThread.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/hardware_breakpoints/hardware_breakpoint_on_multiple_threads/TestHWBreakMultiThread.py Thu Aug 2 17:33:54 2018 (r337147) @@ -25,7 +25,7 @@ class HardwareBreakpointMultiThreadTestCase(TestBase): @expectedFailureAndroid def test_hw_break_set_delete_multi_thread(self): self.build() - self.setTearDownCleanup() + self.setTearDownCleanup() self.break_multi_thread('delete') # LLDB supports hardware breakpoints for arm and aarch64 architectures. @@ -33,7 +33,7 @@ class HardwareBreakpointMultiThreadTestCase(TestBase): @expectedFailureAndroid def test_hw_break_set_disable_multi_thread(self): self.build() - self.setTearDownCleanup() + self.setTearDownCleanup() self.break_multi_thread('disable') def setUp(self): @@ -74,7 +74,7 @@ class HardwareBreakpointMultiThreadTestCase(TestBase): count = 0 while count < 2 : - + self.runCmd("process continue") # We should be stopped in hw_break_function Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/serialize/TestBreakpointSerialization.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/serialize/TestBreakpointSerialization.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/serialize/TestBreakpointSerialization.py Thu Aug 2 17:33:54 2018 (r337147) @@ -65,7 +65,7 @@ class BreakpointSerialization(TestBase): # Create the targets we are making breakpoints in and copying them to: self.orig_target = self.dbg.CreateTarget(exe) self.assertTrue(self.orig_target, VALID_TARGET) - + self.copy_target = self.dbg.CreateTarget(exe) self.assertTrue(self.copy_target, VALID_TARGET) @@ -91,7 +91,7 @@ class BreakpointSerialization(TestBase): num_source_bps = source_bps.GetSize() num_copy_bps = copy_bps.GetSize() self.assertTrue(num_source_bps == num_copy_bps, "Didn't get same number of input and output breakpoints - orig: %d copy: %d"%(num_source_bps, num_copy_bps)) - + for i in range(0, num_source_bps): source_bp = source_bps.GetBreakpointAtIndex(i) source_desc = lldb.SBStream() @@ -132,7 +132,7 @@ class BreakpointSerialization(TestBase): source_bps.Append(self.orig_target.BreakpointCreateByName("blubby", lldb.eFunctionNameTypeAuto, empty_module_list, empty_cu_list)) source_bps.Append(self.orig_target.BreakpointCreateByName("blubby", lldb.eFunctionNameTypeFull, empty_module_list,empty_cu_list)) source_bps.Append(self.orig_target.BreakpointCreateBySourceRegex("dont really care", blubby_file_spec)) - + # And some number greater than one: self.check_equivalence(source_bps) @@ -185,7 +185,7 @@ class BreakpointSerialization(TestBase): bkpt.SetOneShot(True) bkpt.SetThreadID(10) source_bps.Append(bkpt) - + # Make sure we get one right: self.check_equivalence(source_bps) source_bps.Clear() @@ -232,7 +232,7 @@ class BreakpointSerialization(TestBase): bkpt.SetThreadID(10) source_bps.Append(bkpt) all_bps.Append(bkpt) - + error = lldb.SBError() error = self.orig_target.BreakpointsWriteToFile(self.bkpts_file_spec, source_bps) self.assertTrue(error.Success(), "Failed writing breakpoints to file: %s."%(error.GetCString())) @@ -265,7 +265,7 @@ class BreakpointSerialization(TestBase): write_bps = lldb.SBBreakpointList(self.orig_target) bkpt.AddName(good_bkpt_name) write_bps.Append(bkpt) - + error = lldb.SBError() error = self.orig_target.BreakpointsWriteToFile(self.bkpts_file_spec, write_bps) self.assertTrue(error.Success(), "Failed writing breakpoints to file: %s."%(error.GetCString())) @@ -282,7 +282,3 @@ class BreakpointSerialization(TestBase): error = self.copy_target.BreakpointsCreateFromFile(self.bkpts_file_spec, names_list, copy_bps) self.assertTrue(error.Success(), "Failed reading breakpoints from file: %s"%(error.GetCString())) self.assertTrue(copy_bps.GetSize() == 1, "Found the matching breakpoint.") - - - - Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/step_over_breakpoint/TestStepOverBreakpoint.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/step_over_breakpoint/TestStepOverBreakpoint.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/breakpoint/step_over_breakpoint/TestStepOverBreakpoint.py Thu Aug 2 17:33:54 2018 (r337147) @@ -1,8 +1,8 @@ """ Test that breakpoints do not affect stepping. -Check for correct StopReason when stepping to the line with breakpoint +Check for correct StopReason when stepping to the line with breakpoint which chould be eStopReasonBreakpoint in general, -and eStopReasonPlanComplete when breakpoint's condition fails. +and eStopReasonPlanComplete when breakpoint's condition fails. """ from __future__ import print_function @@ -19,7 +19,7 @@ class StepOverBreakpointsTestCase(TestBase): def setUp(self): TestBase.setUp(self) - + self.build() exe = self.getBuildArtifact("a.out") src = lldb.SBFileSpec("main.cpp") @@ -32,7 +32,7 @@ class StepOverBreakpointsTestCase(TestBase): self.line1 = line_number('main.cpp', "breakpoint_1") self.line4 = line_number('main.cpp', "breakpoint_4") - self.breakpoint1 = self.target.BreakpointCreateByLocation(src, self.line1) + self.breakpoint1 = self.target.BreakpointCreateByLocation(src, self.line1) self.assertTrue( self.breakpoint1 and self.breakpoint1.GetNumLocations() == 1, VALID_BREAKPOINT) @@ -52,7 +52,7 @@ class StepOverBreakpointsTestCase(TestBase): self.thread = lldbutil.get_one_thread_stopped_at_breakpoint(self.process, self.breakpoint1) self.assertIsNotNone(self.thread, "Didn't stop at breakpoint 1.") - def test_step_instruction(self): + def test_step_instruction(self): # Count instructions between breakpoint_1 and breakpoint_4 contextList = self.target.FindFunctions('main', lldb.eFunctionNameTypeAuto) self.assertEquals(contextList.GetSize(), 1) @@ -89,7 +89,7 @@ class StepOverBreakpointsTestCase(TestBase): @skipIf(bugnumber="llvm.org/pr31972", hostoslist=["windows"]) def test_step_over(self): #lldb.DBG.EnableLog("lldb", ["step","breakpoint"]) - + self.thread.StepOver() # We should be stopped at the breakpoint_2 line with stop plan complete reason self.assertEquals(self.process.GetState(), lldb.eStateStopped) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/TestCompletion.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/TestCompletion.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/TestCompletion.py Thu Aug 2 17:33:54 2018 (r337147) @@ -39,6 +39,46 @@ class CommandLineCompletionTestCase(TestBase): self.complete_from_to('de', 'detach ') @skipIfFreeBSD # timing out on the FreeBSD buildbot + def test_frame_variable(self): + self.build() + self.main_source = "main.cpp" + self.main_source_spec = lldb.SBFileSpec(self.main_source) + self.dbg.CreateTarget(self.getBuildArtifact("a.out")) + + (target, process, thread, bkpt) = lldbutil.run_to_source_breakpoint(self, + '// Break here', self.main_source_spec) + self.assertEquals(process.GetState(), lldb.eStateStopped) + # FIXME: This pulls in the debug information to make the completions work, + # but the completions should also work without. + self.runCmd("frame variable fooo") + + self.complete_from_to('frame variable fo', + 'frame variable fooo') + self.complete_from_to('frame variable fooo.', + 'frame variable fooo.') + self.complete_from_to('frame variable fooo.dd', + 'frame variable fooo.dd') + + self.complete_from_to('frame variable ptr_fooo->', + 'frame variable ptr_fooo->') + self.complete_from_to('frame variable ptr_fooo->dd', + 'frame variable ptr_fooo->dd') + + self.complete_from_to('frame variable cont', + 'frame variable container') + self.complete_from_to('frame variable container.', + 'frame variable container.MemberVar') + self.complete_from_to('frame variable container.Mem', + 'frame variable container.MemberVar') + + self.complete_from_to('frame variable ptr_cont', + 'frame variable ptr_container') + self.complete_from_to('frame variable ptr_container->', + 'frame variable ptr_container->MemberVar') + self.complete_from_to('frame variable ptr_container->Mem', + 'frame variable ptr_container->MemberVar') + + @skipIfFreeBSD # timing out on the FreeBSD buildbot def test_process_attach_dash_dash_con(self): """Test that 'process attach --con' completes to 'process attach --continue '.""" self.complete_from_to( @@ -261,7 +301,7 @@ class CommandLineCompletionTestCase(TestBase): common_match = match_strings.GetStringAtIndex(0) if num_matches == 0: compare_string = str_input - else: + else: if common_match != None and len(common_match) > 0: compare_string = str_input + common_match else: @@ -273,8 +313,8 @@ class CommandLineCompletionTestCase(TestBase): if turn_off_re_match: self.expect( compare_string, msg=COMPLETION_MSG( - str_input, p), exe=False, substrs=[p]) + str_input, p, match_strings), exe=False, substrs=[p]) else: self.expect( compare_string, msg=COMPLETION_MSG( - str_input, p), exe=False, patterns=[p]) + str_input, p, match_strings), exe=False, patterns=[p]) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/main.cpp ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/main.cpp Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/completion/main.cpp Thu Aug 2 17:33:54 2018 (r337147) @@ -7,8 +7,15 @@ class Foo (public) } }; +struct Container { int MemberVar; }; + int main() { - Foo f; - f.Bar(1, 2); + Foo fooo; + Foo *ptr_fooo = &fooo; + fooo.Bar(1, 2); + + Container container; + Container *ptr_container = &container; + return container.MemberVar = 3; // Break here } Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/bitset/TestDataFormatterLibcxxBitset.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/bitset/TestDataFormatterLibcxxBitset.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/bitset/TestDataFormatterLibcxxBitset.py Thu Aug 2 17:33:54 2018 (r337147) @@ -49,7 +49,7 @@ class TestDataFormatterLibcxxBitset(TestBase): def test_ptr_and_ref(self): """Test that ref and ptr to std::bitset is displayed correctly""" self.build() - (_, process, _, bkpt) = lldbutil.run_to_source_breakpoint(self, + (_, process, _, bkpt) = lldbutil.run_to_source_breakpoint(self, 'Check ref and ptr', lldb.SBFileSpec("main.cpp", False)) @@ -57,6 +57,6 @@ class TestDataFormatterLibcxxBitset(TestBase): self.check("ptr", 13) lldbutil.continue_to_breakpoint(process, bkpt) - + self.check("ref", 200) self.check("ptr", 200) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/list/TestDataFormatterLibcxxList.py Thu Aug 2 17:33:54 2018 (r337147) @@ -90,7 +90,7 @@ class LibcxxListDataFormatterTestCase(TestBase): self.runCmd("n") # This gets up past the printf self.runCmd("n") # Now advance over the first push_back. - + self.expect("frame variable numbers_list", substrs=['list has 1 items', '[0] = ', Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/multiset/TestDataFormatterLibcxxMultiSet.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/multiset/TestDataFormatterLibcxxMultiSet.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/multiset/TestDataFormatterLibcxxMultiSet.py Thu Aug 2 17:33:54 2018 (r337147) @@ -134,7 +134,7 @@ class LibcxxMultiSetDataFormatterTestCase(TestBase): """Test that the data formatters work on ref and ptr.""" self.build() (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint( - self, "Stop here to check by ref and ptr.", + self, "Stop here to check by ref and ptr.", lldb.SBFileSpec("main.cpp", False)) # The reference should print just like the value: self.check_ii("ref") Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/set/TestDataFormatterLibcxxSet.py Thu Aug 2 17:33:54 2018 (r337147) @@ -130,7 +130,7 @@ class LibcxxSetDataFormatterTestCase(TestBase): """Test that the data formatters work on ref and ptr.""" self.build() (self.target, process, _, bkpt) = lldbutil.run_to_source_breakpoint( - self, "Stop here to check by ref and ptr.", + self, "Stop here to check by ref and ptr.", lldb.SBFileSpec("main.cpp", False)) # The reference should print just like the value: self.check_ii("ref") @@ -139,4 +139,4 @@ class LibcxxSetDataFormatterTestCase(TestBase): substrs=["ptr =", "size=7"]) self.expect("expr ptr", substrs=["size=7"]) - + Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libcxx/vector/TestDataFormatterLibcxxVector.py Thu Aug 2 17:33:54 2018 (r337147) @@ -192,7 +192,7 @@ class LibcxxVectorDataFormatterTestCase(TestBase): self.check_numbers("ref") # The pointer should just show the right number of elements: - + self.expect("frame variable ptr", substrs=['ptr =', ' size=7']) self.expect("p ptr", substrs=['$', 'size=7']) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/tuple/TestDataFormatterStdTuple.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/tuple/TestDataFormatterStdTuple.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/data-formatter/data-formatter-stl/libstdcpp/tuple/TestDataFormatterStdTuple.py Thu Aug 2 17:33:54 2018 (r337147) @@ -40,7 +40,7 @@ class StdTupleDataFormatterTestCase(TestBase): self.assertEqual('"foobar"', frame.GetValueForVariablePath("ts[0]").GetSummary()) self.assertFalse(frame.GetValueForVariablePath("ts[1]").IsValid()) - + self.assertEqual(1, frame.GetValueForVariablePath("tt[0]").GetValueAsUnsigned()) self.assertEqual('"baz"', frame.GetValueForVariablePath("tt[1]").GetSummary()) self.assertEqual(2, frame.GetValueForVariablePath("tt[2]").GetValueAsUnsigned()) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/exec/TestExec.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/exec/TestExec.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/exec/TestExec.py Thu Aug 2 17:33:54 2018 (r337147) @@ -96,7 +96,7 @@ class ExecTestCase(TestBase): if not skip_exec: self.assertTrue(process.GetState() == lldb.eStateStopped, "Process should be stopped at __dyld_start") - + threads = lldbutil.get_stopped_threads( process, lldb.eStopReasonExec) self.assertTrue( Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame-language/TestGuessLanguage.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame-language/TestGuessLanguage.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame-language/TestGuessLanguage.py Thu Aug 2 17:33:54 2018 (r337147) @@ -18,7 +18,7 @@ class TestFrameGuessLanguage(TestBase): mydir = TestBase.compute_mydir(__file__) - # If your test case doesn't stress debug info, the + # If your test case doesn't stress debug info, the # set this to true. That way it won't be run once for # each debug info format. NO_DEBUG_INFO_TESTCASE = True @@ -84,6 +84,6 @@ class TestFrameGuessLanguage(TestBase): self.check_language(thread, 0, c_frame_language) self.check_language(thread, 1, lldb.eLanguageTypeC_plus_plus) self.check_language(thread, 2, lldb.eLanguageTypeC_plus_plus) - - + + Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame_var/TestFrameVar.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame_var/TestFrameVar.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/frame_var/TestFrameVar.py Thu Aug 2 17:33:54 2018 (r337147) @@ -17,7 +17,7 @@ class TestFrameVar(TestBase): mydir = TestBase.compute_mydir(__file__) - # If your test case doesn't stress debug info, the + # If your test case doesn't stress debug info, the # set this to true. That way it won't be run once for # each debug info format. NO_DEBUG_INFO_TESTCASE = True @@ -67,7 +67,7 @@ class TestFrameVar(TestBase): frame = threads[0].GetFrameAtIndex(0) command_result = lldb.SBCommandReturnObject() interp = self.dbg.GetCommandInterpreter() - + # Just get args: result = interp.HandleCommand("frame var -l", command_result) self.assertEqual(result, lldb.eReturnStatusSuccessFinishResult, "frame var -a didn't succeed") @@ -85,7 +85,7 @@ class TestFrameVar(TestBase): self.assertTrue("argv" not in output, "Locals found argv") self.assertTrue("test_var" in output, "Locals didn't find test_var") self.assertTrue("g_var" not in output, "Locals found a global") - + # Get the file statics: result = interp.HandleCommand("frame var -l -a -g", command_result) self.assertEqual(result, lldb.eReturnStatusSuccessFinishResult, "frame var -a didn't succeed") @@ -94,6 +94,6 @@ class TestFrameVar(TestBase): self.assertTrue("argv" not in output, "Globals found argv") self.assertTrue("test_var" not in output, "Globals found test_var") self.assertTrue("g_var" in output, "Globals didn't find g_var") - - + + Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/TestTargetXMLArch.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/TestTargetXMLArch.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/TestTargetXMLArch.py Thu Aug 2 17:33:54 2018 (r337147) @@ -98,12 +98,12 @@ class TestTargetXMLArch(GDBRemoteTestBase): return "T05thread:00000001;06:9038d60f00700000;07:98b4062680ffffff;10:c0d7bf1b80ffffff;" def readRegister(self, register): - regs = {0x0: "00b0060000610000", - 0xa: "68fe471c80ffffff", - 0xc: "60574a1c80ffffff", - 0xd: "18f3042680ffffff", - 0xe: "be8a4d7142000000", - 0xf: "50df471c80ffffff", + regs = {0x0: "00b0060000610000", + 0xa: "68fe471c80ffffff", + 0xc: "60574a1c80ffffff", + 0xd: "18f3042680ffffff", + 0xe: "be8a4d7142000000", + 0xf: "50df471c80ffffff", 0x10: "c0d7bf1b80ffffff" } if register in regs: return regs[register] Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/gdbclientutils.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/gdbclientutils.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/gdb_remote_client/gdbclientutils.py Thu Aug 2 17:33:54 2018 (r337147) @@ -467,7 +467,7 @@ class GDBRemoteTestBase(TestBase): i = 0 j = 0 log = self.server.responder.packetLog - + while i < len(packets) and j < len(log): if log[j] == packets[i]: i += 1 Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/history/TestHistoryRecall.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/history/TestHistoryRecall.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/history/TestHistoryRecall.py Thu Aug 2 17:33:54 2018 (r337147) @@ -17,7 +17,7 @@ class TestHistoryRecall(TestBase): mydir = TestBase.compute_mydir(__file__) - # If your test case doesn't stress debug info, the + # If your test case doesn't stress debug info, the # set this to true. That way it won't be run once for # each debug info format. NO_DEBUG_INFO_TESTCASE = True @@ -35,7 +35,7 @@ class TestHistoryRecall(TestBase): result = lldb.SBCommandReturnObject() interp.HandleCommand("command history", result, True) interp.HandleCommand("platform list", result, True) - + interp.HandleCommand("!0", result, False) self.assertTrue(result.Succeeded(), "!0 command did not work: %s"%(result.GetError())) self.assertTrue("command history" in result.GetOutput(), "!0 didn't rerun command history") Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_unload/TestLoadUnload.py Thu Aug 2 17:33:54 2018 (r337147) @@ -20,7 +20,7 @@ class LoadUnloadTestCase(TestBase): mydir = TestBase.compute_mydir(__file__) NO_DEBUG_INFO_TESTCASE = True - + def setUp(self): # Call super's setUp(). TestBase.setUp(self) @@ -210,7 +210,7 @@ class LoadUnloadTestCase(TestBase): if not self.platformIsDarwin(): env_cmd_string += ":" + wd self.runCmd(env_cmd_string) - + # This time, the hidden library should be picked up. self.expect("run", substrs=["return", "12345"]) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_using_paths/TestLoadUsingPaths.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_using_paths/TestLoadUsingPaths.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/load_using_paths/TestLoadUsingPaths.py Thu Aug 2 17:33:54 2018 (r337147) @@ -20,7 +20,7 @@ class LoadUsingPathsTestCase(TestBase): mydir = TestBase.compute_mydir(__file__) NO_DEBUG_INFO_TESTCASE = True - + def setUp(self): # Call super's setUp(). TestBase.setUp(self) @@ -52,8 +52,8 @@ class LoadUsingPathsTestCase(TestBase): # The directory with the dynamic library we did not link to. path_dir = os.path.join(self.getBuildDir(), "hidden") - (target, process, thread, - _) = lldbutil.run_to_source_breakpoint(self, + (target, process, thread, + _) = lldbutil.run_to_source_breakpoint(self, "Break here to do the load using paths", lldb.SBFileSpec("main.cpp")) error = lldb.SBError() @@ -63,18 +63,18 @@ class LoadUsingPathsTestCase(TestBase): paths.AppendString(os.path.join(self.wd, "no_such_dir")) out_spec = lldb.SBFileSpec() - + # First try with no correct directories on the path, and make sure that doesn't blow up: token = process.LoadImageUsingPaths(lib_spec, paths, out_spec, error) self.assertEqual(token, lldb.LLDB_INVALID_IMAGE_TOKEN, "Only looked on the provided path.") - + # Now add the correct dir to the paths list and try again: paths.AppendString(self.hidden_dir) token = process.LoadImageUsingPaths(lib_spec, paths, out_spec, error) self.assertNotEqual(token, lldb.LLDB_INVALID_IMAGE_TOKEN, "Got a valid token") self.assertEqual(out_spec, lldb.SBFileSpec(self.hidden_lib), "Found the expected library") - + # Make sure this really is in the image list: loaded_module = target.FindModule(out_spec) @@ -89,11 +89,11 @@ class LoadUsingPathsTestCase(TestBase): # Make sure the token works to unload it: process.UnloadImage(token) - # Make sure this really is no longer in the image list: + # Make sure this really is no longer in the image list: loaded_module = target.FindModule(out_spec) self.assertFalse(loaded_module.IsValid(), "The unloaded module is no longer in the image list.") - + # Make sure a relative path also works: paths.Clear() paths.AppendString(os.path.join(self.wd, "no_such_dir")) @@ -107,7 +107,7 @@ class LoadUsingPathsTestCase(TestBase): self.assertEqual(out_spec, lldb.SBFileSpec(self.hidden_lib), "Found the expected library with relative path") process.UnloadImage(token) - + # Make sure the presence of an empty path doesn't mess anything up: paths.Clear() paths.AppendString("") @@ -122,9 +122,9 @@ class LoadUsingPathsTestCase(TestBase): self.assertEqual(out_spec, lldb.SBFileSpec(self.hidden_lib), "Found the expected library with included empty path") process.UnloadImage(token) - + # Finally, passing in an absolute path should work like the basename: # This should NOT work because we've taken hidden_dir off the paths: abs_spec = lldb.SBFileSpec(os.path.join(self.hidden_dir, self.lib_name)) @@ -140,4 +140,4 @@ class LoadUsingPathsTestCase(TestBase): self.assertNotEqual(token, lldb.LLDB_INVALID_IMAGE_TOKEN, "Got a valid token") self.assertEqual(out_spec, lldb.SBFileSpec(self.hidden_lib), "Found the expected library") - + Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/pre_run_dylibs/TestPreRunDylibs.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/pre_run_dylibs/TestPreRunDylibs.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/pre_run_dylibs/TestPreRunDylibs.py Thu Aug 2 17:33:54 2018 (r337147) @@ -35,4 +35,4 @@ class TestPreRunLibraries(TestBase): self.assertTrue(found_it, "Couldn't find unlikely_to_occur_name in loaded libraries.") - + Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/register/intel_avx/TestZMMRegister.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/register/intel_avx/TestZMMRegister.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/register/intel_avx/TestZMMRegister.py Thu Aug 2 17:33:54 2018 (r337147) @@ -81,7 +81,7 @@ class TestYMMRegister(TestBase): self.expect( "register read ymm" + str(i), substrs=[pattern]) - + self.expect("continue", PROCESS_STOPPED, substrs=['stopped']) # Check stop reason; Should be either signal SIGTRAP or EXC_BREAKPOINT Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/return-value/TestReturnValue.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/return-value/TestReturnValue.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/return-value/TestReturnValue.py Thu Aug 2 17:33:54 2018 (r337147) @@ -68,7 +68,7 @@ class ReturnValueTestCase(TestBase): # Run again and we will stop in inner_sint the second time outer_sint is called. # Then test stepping out two frames at once: - + thread_list = lldbutil.continue_to_breakpoint(self.process, inner_sint_bkpt) self.assertTrue(len(thread_list) == 1) thread = thread_list[0] Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/create_during_step/TestCreateDuringStep.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/create_during_step/TestCreateDuringStep.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/create_during_step/TestCreateDuringStep.py Thu Aug 2 17:33:54 2018 (r337147) @@ -90,7 +90,7 @@ class CreateDuringStepTestCase(TestBase): target = self.dbg.GetSelectedTarget() # This should create a breakpoint in the stepping thread. - self.bkpt = target.BreakpointCreateByLocation("main.cpp", self.breakpoint) + self.bkpt = target.BreakpointCreateByLocation("main.cpp", self.breakpoint) # Run the program. self.runCmd("run", RUN_SUCCEEDED) Modified: vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/num_threads/TestNumThreads.py ============================================================================== --- vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/num_threads/TestNumThreads.py Thu Aug 2 17:33:50 2018 (r337146) +++ vendor/lldb/dist/packages/Python/lldbsuite/test/functionalities/thread/num_threads/TestNumThreads.py Thu Aug 2 17:33:54 2018 (r337147) @@ -92,7 +92,7 @@ class NumberOfThreadsTestCase(TestBase): self.assertTrue( num_threads >= 10, 'Number of expected threads and actual threads do not match.') - + # Attempt to walk each of the thread's executing the thread3 function to # the same breakpoint. def is_thread3(thread): *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu Aug 2 19:09:14 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B8D0104AF83; Thu, 2 Aug 2018 19:09:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F2B0276C51; Thu, 2 Aug 2018 19:09:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id B601415C6E; Thu, 2 Aug 2018 19:09:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72J9D4I076485; Thu, 2 Aug 2018 19:09:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72J9DfR076484; Thu, 2 Aug 2018 19:09:13 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808021909.w72J9DfR076484@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 19:09:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337161 - in vendor/illumos/dist: cmd/zfs lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor/illumos/dist: cmd/zfs lib/libzfs/common X-SVN-Commit-Revision: 337161 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 19:09:14 -0000 Author: mav Date: Thu Aug 2 19:09:13 2018 New Revision: 337161 URL: https://svnweb.freebsd.org/changeset/base/337161 Log: 9512 zfs remap poolname@snapname coredumps Only filesystems and volumes are valid "zfs remap" parameters: when passed a snapshot name zfs_remap_indirects() does not handle the EINVAL returned from libzfs_core, which results in failing an assertion and consequently crashing. illumos/illumos-gate@0b2e8253986c5c761129b58cfdac46d204903de1 Reviewed by: Matthew Ahrens Reviewed by: John Wren Kennedy Reviewed by: Sara Hartse Approved by: Matt Ahrens Author: loli10K Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Modified: vendor/illumos/dist/cmd/zfs/zfs_main.c ============================================================================== --- vendor/illumos/dist/cmd/zfs/zfs_main.c Thu Aug 2 18:55:55 2018 (r337160) +++ vendor/illumos/dist/cmd/zfs/zfs_main.c Thu Aug 2 19:09:13 2018 (r337161) @@ -6924,11 +6924,28 @@ zfs_do_diff(int argc, char **argv) return (err != 0); } +/* + * zfs remap + * + * Remap the indirect blocks in the given fileystem or volume. + */ static int zfs_do_remap(int argc, char **argv) { const char *fsname; int err = 0; + int c; + + /* check options */ + while ((c = getopt(argc, argv, "")) != -1) { + switch (c) { + case '?': + (void) fprintf(stderr, + gettext("invalid option '%c'\n"), optopt); + usage(B_FALSE); + } + } + if (argc != 2) { (void) fprintf(stderr, gettext("wrong number of arguments\n")); usage(B_FALSE); Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Thu Aug 2 18:55:55 2018 (r337160) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Thu Aug 2 19:09:13 2018 (r337161) @@ -3877,12 +3877,24 @@ zfs_remap_indirects(libzfs_handle_t *hdl, const char * char errbuf[1024]; (void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN, - "cannot remap filesystem '%s' "), fs); + "cannot remap dataset '%s'"), fs); err = lzc_remap(fs); if (err != 0) { - (void) zfs_standard_error(hdl, err, errbuf); + switch (err) { + case ENOTSUP: + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "pool must be upgraded")); + (void) zfs_error(hdl, EZFS_BADVERSION, errbuf); + break; + case EINVAL: + (void) zfs_error(hdl, EZFS_BADTYPE, errbuf); + break; + default: + (void) zfs_standard_error(hdl, err, errbuf); + break; + } } return (err); From owner-svn-src-vendor@freebsd.org Thu Aug 2 19:37:14 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65800104D7F0; Thu, 2 Aug 2018 19:37:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1BD3C786B9; Thu, 2 Aug 2018 19:37:14 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id F12761615A; Thu, 2 Aug 2018 19:37:13 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72JbDel092498; Thu, 2 Aug 2018 19:37:13 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72JbDrZ092495; Thu, 2 Aug 2018 19:37:13 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808021937.w72JbDrZ092495@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 19:37:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337167 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337167 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 19:37:14 -0000 Author: mav Date: Thu Aug 2 19:37:13 2018 New Revision: 337167 URL: https://svnweb.freebsd.org/changeset/base/337167 Log: 9442 decrease indirect block size of spacemaps Updates to indirect blocks of spacemaps can contribute significantly to write inflation. Therefore we want to reduce the indirect block size of spacemaps from 128K to 16K. illumos/illumos-gate@221813c13b43ef48330b03725e00edee85108cf1 Reviewed by: Serapheim Dimitropoulos Reviewed by: George Wilson Reviewed by: Albert Lee Reviewed by: Igor Kozhukhov Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Thu Aug 2 19:32:32 2018 (r337166) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Thu Aug 2 19:37:13 2018 (r337167) @@ -32,7 +32,8 @@ #include uint64_t -dmu_object_alloc(objset_t *os, dmu_object_type_t ot, int blocksize, +dmu_object_alloc_ibs(objset_t *os, dmu_object_type_t ot, int blocksize, + int indirect_blockshift, dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx) { uint64_t object; @@ -92,13 +93,22 @@ dmu_object_alloc(objset_t *os, dmu_object_type_t ot, i os->os_obj_next = object - 1; } - dnode_allocate(dn, ot, blocksize, 0, bonustype, bonuslen, tx); + dnode_allocate(dn, ot, blocksize, indirect_blockshift, + bonustype, bonuslen, tx); mutex_exit(&os->os_obj_lock); dmu_tx_add_new_object(tx, dn); dnode_rele(dn, FTAG); return (object); +} + +uint64_t +dmu_object_alloc(objset_t *os, dmu_object_type_t ot, int blocksize, + dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx) +{ + return (dmu_object_alloc_ibs(os, ot, blocksize, 0, + bonustype, bonuslen, tx)); } int Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c Thu Aug 2 19:32:32 2018 (r337166) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/space_map.c Thu Aug 2 19:37:13 2018 (r337167) @@ -52,6 +52,14 @@ */ boolean_t zfs_force_some_double_word_sm_entries = B_FALSE; +/* + * Override the default indirect block size of 128K, instead using 16K for + * spacemaps (2^14 bytes). This dramatically reduces write inflation since + * appending to a spacemap typically has to write one data block (4KB) and one + * or two indirect blocks (16K-32K, rather than 128K). + */ +int space_map_ibs = 14; + boolean_t sm_entry_is_debug(uint64_t e) { @@ -674,8 +682,8 @@ space_map_write_impl(space_map_t *sm, range_tree_t *rt * * [1] The feature is enabled. * [2] The offset or run is too big for a single-word entry, - * or the vdev_id is set (meaning not equal to - * SM_NO_VDEVID). + * or the vdev_id is set (meaning not equal to + * SM_NO_VDEVID). * * Note that for purposes of testing we've added the case that * we write two-word entries occasionally when the feature is @@ -834,7 +842,8 @@ space_map_truncate(space_map_t *sm, int blocksize, dmu */ if ((spa_feature_is_enabled(spa, SPA_FEATURE_SPACEMAP_HISTOGRAM) && doi.doi_bonus_size != sizeof (space_map_phys_t)) || - doi.doi_data_block_size != blocksize) { + doi.doi_data_block_size != blocksize || + doi.doi_metadata_block_size != 1 << space_map_ibs) { zfs_dbgmsg("txg %llu, spa %s, sm %p, reallocating " "object[%llu]: old bonus %u, old blocksz %u", dmu_tx_get_txg(tx), spa_name(spa), sm, sm->sm_object, @@ -890,8 +899,8 @@ space_map_alloc(objset_t *os, int blocksize, dmu_tx_t bonuslen = SPACE_MAP_SIZE_V0; } - object = dmu_object_alloc(os, DMU_OT_SPACE_MAP, blocksize, - DMU_OT_SPACE_MAP_HEADER, bonuslen, tx); + object = dmu_object_alloc_ibs(os, DMU_OT_SPACE_MAP, blocksize, + space_map_ibs, DMU_OT_SPACE_MAP_HEADER, bonuslen, tx); return (object); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Thu Aug 2 19:32:32 2018 (r337166) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Thu Aug 2 19:37:13 2018 (r337167) @@ -354,6 +354,9 @@ typedef struct dmu_buf { */ uint64_t dmu_object_alloc(objset_t *os, dmu_object_type_t ot, int blocksize, dmu_object_type_t bonus_type, int bonus_len, dmu_tx_t *tx); +uint64_t dmu_object_alloc_ibs(objset_t *os, dmu_object_type_t ot, int blocksize, + int indirect_blockshift, + dmu_object_type_t bonustype, int bonuslen, dmu_tx_t *tx); int dmu_object_claim(objset_t *os, uint64_t object, dmu_object_type_t ot, int blocksize, dmu_object_type_t bonus_type, int bonus_len, dmu_tx_t *tx); int dmu_object_reclaim(objset_t *os, uint64_t object, dmu_object_type_t ot, From owner-svn-src-vendor@freebsd.org Thu Aug 2 20:17:39 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C2BE104EA93; Thu, 2 Aug 2018 20:17:39 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E62CE7A271; Thu, 2 Aug 2018 20:17:38 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C4289167DF; Thu, 2 Aug 2018 20:17:38 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72KHch3013257; Thu, 2 Aug 2018 20:17:38 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72KHcoJ013254; Thu, 2 Aug 2018 20:17:38 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022017.w72KHcoJ013254@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 20:17:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337171 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337171 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 20:17:39 -0000 Author: mav Date: Thu Aug 2 20:17:37 2018 New Revision: 337171 URL: https://svnweb.freebsd.org/changeset/base/337171 Log: 9464 txg_kick() fails to see that we are quiescing, forcing transactions to their next stages without leaving them accumulate changes Ideally we would like txg_kick() to get triggered only when we are sure that we are not syncing AND not quiescing any txg. This way we can kick an open TXG to the quiescing state when we are sure that there is nothing going on and we would benefit from the different states running concurrently. illumos/illumos-gate@fa41d87de9ec9000964c605eb01d6dc19e4a1abe Reviewed by: Matt Ahrens Reviewed by: Brad Lewis Reviewed by: Andriy Gapon Approved by: Dan McDonald Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Thu Aug 2 20:10:59 2018 (r337170) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Thu Aug 2 20:17:37 2018 (r337171) @@ -1085,7 +1085,12 @@ dmu_tx_wait(dmu_tx_t *tx) mutex_exit(&dn->dn_mtx); tx->tx_needassign_txh = NULL; } else { - txg_wait_open(tx->tx_pool, tx->tx_lasttried_txg + 1); + /* + * If we have a lot of dirty data just wait until we sync + * out a TXG at which point we'll hopefully have synced + * a portion of the changes. + */ + txg_wait_synced(dp, spa_last_synced_txg(spa) + 1); } } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg_impl.h Thu Aug 2 20:10:59 2018 (r337170) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg_impl.h Thu Aug 2 20:17:37 2018 (r337171) @@ -25,7 +25,7 @@ */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2013, 2017 by Delphix. All rights reserved. */ #ifndef _SYS_TXG_IMPL_H @@ -92,6 +92,7 @@ typedef struct tx_state { kmutex_t tx_sync_lock; /* protects the rest of this struct */ uint64_t tx_open_txg; /* currently open txg id */ + uint64_t tx_quiescing_txg; /* currently quiescing txg id */ uint64_t tx_quiesced_txg; /* quiesced txg waiting for sync */ uint64_t tx_syncing_txg; /* currently syncing txg id */ uint64_t tx_synced_txg; /* last synced txg id */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Thu Aug 2 20:10:59 2018 (r337170) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Thu Aug 2 20:17:37 2018 (r337171) @@ -444,6 +444,30 @@ txg_dispatch_callbacks(dsl_pool_t *dp, uint64_t txg) } } +static boolean_t +txg_is_syncing(dsl_pool_t *dp) +{ + tx_state_t *tx = &dp->dp_tx; + ASSERT(MUTEX_HELD(&tx->tx_sync_lock)); + return (tx->tx_syncing_txg != 0); +} + +static boolean_t +txg_is_quiescing(dsl_pool_t *dp) +{ + tx_state_t *tx = &dp->dp_tx; + ASSERT(MUTEX_HELD(&tx->tx_sync_lock)); + return (tx->tx_quiescing_txg != 0); +} + +static boolean_t +txg_has_quiesced_to_sync(dsl_pool_t *dp) +{ + tx_state_t *tx = &dp->dp_tx; + ASSERT(MUTEX_HELD(&tx->tx_sync_lock)); + return (tx->tx_quiesced_txg != 0); +} + static void txg_sync_thread(void *arg) { @@ -470,7 +494,7 @@ txg_sync_thread(void *arg) while (!dsl_scan_active(dp->dp_scan) && !tx->tx_exiting && timer > 0 && tx->tx_synced_txg >= tx->tx_sync_txg_waiting && - tx->tx_quiesced_txg == 0 && + !txg_has_quiesced_to_sync(dp) && dp->dp_dirty_total < zfs_dirty_data_sync) { dprintf("waiting; tx_synced=%llu waiting=%llu dp=%p\n", tx->tx_synced_txg, tx->tx_sync_txg_waiting, dp); @@ -483,7 +507,7 @@ txg_sync_thread(void *arg) * Wait until the quiesce thread hands off a txg to us, * prompting it to do so if necessary. */ - while (!tx->tx_exiting && tx->tx_quiesced_txg == 0) { + while (!tx->tx_exiting && !txg_has_quiesced_to_sync(dp)) { if (tx->tx_quiesce_txg_waiting < tx->tx_open_txg+1) tx->tx_quiesce_txg_waiting = tx->tx_open_txg+1; cv_broadcast(&tx->tx_quiesce_more_cv); @@ -498,6 +522,7 @@ txg_sync_thread(void *arg) * us. This may cause the quiescing thread to now be * able to quiesce another txg, so we must signal it. */ + ASSERT(tx->tx_quiesced_txg != 0); txg = tx->tx_quiesced_txg; tx->tx_quiesced_txg = 0; tx->tx_syncing_txg = txg; @@ -546,7 +571,7 @@ txg_quiesce_thread(void *arg) */ while (!tx->tx_exiting && (tx->tx_open_txg >= tx->tx_quiesce_txg_waiting || - tx->tx_quiesced_txg != 0)) + txg_has_quiesced_to_sync(dp))) txg_thread_wait(tx, &cpr, &tx->tx_quiesce_more_cv, 0); if (tx->tx_exiting) @@ -556,6 +581,8 @@ txg_quiesce_thread(void *arg) dprintf("txg=%llu quiesce_txg=%llu sync_txg=%llu\n", txg, tx->tx_quiesce_txg_waiting, tx->tx_sync_txg_waiting); + tx->tx_quiescing_txg = txg; + mutex_exit(&tx->tx_sync_lock); txg_quiesce(dp, txg); mutex_enter(&tx->tx_sync_lock); @@ -564,6 +591,7 @@ txg_quiesce_thread(void *arg) * Hand this txg off to the sync thread. */ dprintf("quiesce done, handing off txg %llu\n", txg); + tx->tx_quiescing_txg = 0; tx->tx_quiesced_txg = txg; DTRACE_PROBE2(txg__quiesced, dsl_pool_t *, dp, uint64_t, txg); cv_broadcast(&tx->tx_sync_more_cv); @@ -661,7 +689,8 @@ txg_kick(dsl_pool_t *dp) ASSERT(!dsl_pool_config_held(dp)); mutex_enter(&tx->tx_sync_lock); - if (tx->tx_syncing_txg == 0 && + if (!txg_is_syncing(dp) && + !txg_is_quiescing(dp) && tx->tx_quiesce_txg_waiting <= tx->tx_open_txg && tx->tx_sync_txg_waiting <= tx->tx_synced_txg && tx->tx_quiesced_txg <= tx->tx_synced_txg) { From owner-svn-src-vendor@freebsd.org Thu Aug 2 20:29:59 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD4ED104F162; Thu, 2 Aug 2018 20:29:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 727387AC22; Thu, 2 Aug 2018 20:29:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 539C416992; Thu, 2 Aug 2018 20:29:59 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72KTxx1018624; Thu, 2 Aug 2018 20:29:59 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72KTxuW018623; Thu, 2 Aug 2018 20:29:59 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022029.w72KTxuW018623@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 20:29:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337175 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337175 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 20:30:00 -0000 Author: mav Date: Thu Aug 2 20:29:58 2018 New Revision: 337175 URL: https://svnweb.freebsd.org/changeset/base/337175 Log: 9487 Free objects when receiving full stream as clone All objects after the last written or freed object are not supposed to exist after receiving the stream. We should free them accordingly, as if a freeobjects record for them had been included in the stream. zfsonlinux/zfs@48fbb9ddbf2281911560dfbc2821aa8b74127315 illumos/illumos-gate@7864b8192b8d30471fa2240466d516292e5765b8 Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: Paul Dagnelie Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_send.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Thu Aug 2 20:28:09 2018 (r337174) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c Thu Aug 2 20:29:58 2018 (r337175) @@ -1757,6 +1757,7 @@ dmu_recv_begin(char *tofs, char *tosnap, dmu_replay_re drc->drc_force = force; drc->drc_resumable = resumable; drc->drc_cred = CRED(); + drc->drc_clone = (origin != NULL); if (drc->drc_drrb->drr_magic == BSWAP_64(DMU_BACKUP_MAGIC)) { drc->drc_byteswap = B_TRUE; @@ -1817,7 +1818,9 @@ struct receive_writer_arg { /* A map from guid to dataset to help handle dedup'd streams. */ avl_tree_t *guid_to_ds_map; boolean_t resumable; - uint64_t last_object, last_offset; + uint64_t last_object; + uint64_t last_offset; + uint64_t max_object; /* highest object ID referenced in stream */ uint64_t bytes_read; /* bytes read when current record created */ }; @@ -2089,6 +2092,9 @@ receive_object(struct receive_writer_arg *rwa, struct return (SET_ERROR(EINVAL)); object = err == 0 ? drro->drr_object : DMU_NEW_OBJECT; + if (drro->drr_object > rwa->max_object) + rwa->max_object = drro->drr_object; + /* * If we are losing blkptrs or changing the block size this must * be a new file instance. We must clear out the previous file @@ -2184,6 +2190,9 @@ receive_freeobjects(struct receive_writer_arg *rwa, err = dmu_free_long_object(rwa->os, obj); if (err != 0) return (err); + + if (obj > rwa->max_object) + rwa->max_object = obj; } if (next_err != ESRCH) return (next_err); @@ -2213,6 +2222,9 @@ receive_write(struct receive_writer_arg *rwa, struct d rwa->last_object = drrw->drr_object; rwa->last_offset = drrw->drr_offset; + if (rwa->last_object > rwa->max_object) + rwa->max_object = rwa->last_object; + if (dmu_object_info(rwa->os, drrw->drr_object, NULL) != 0) return (SET_ERROR(EINVAL)); @@ -2289,6 +2301,9 @@ receive_write_byref(struct receive_writer_arg *rwa, ref_os = rwa->os; } + if (drrwbr->drr_object > rwa->max_object) + rwa->max_object = drrwbr->drr_object; + err = dmu_buf_hold(ref_os, drrwbr->drr_refobject, drrwbr->drr_refoffset, FTAG, &dbp, DMU_READ_PREFETCH); if (err != 0) @@ -2331,6 +2346,9 @@ receive_write_embedded(struct receive_writer_arg *rwa, if (drrwe->drr_compression >= ZIO_COMPRESS_FUNCTIONS) return (EINVAL); + if (drrwe->drr_object > rwa->max_object) + rwa->max_object = drrwe->drr_object; + tx = dmu_tx_create(rwa->os); dmu_tx_hold_write(tx, drrwe->drr_object, @@ -2367,6 +2385,9 @@ receive_spill(struct receive_writer_arg *rwa, struct d if (dmu_object_info(rwa->os, drrs->drr_object, NULL) != 0) return (SET_ERROR(EINVAL)); + if (drrs->drr_object > rwa->max_object) + rwa->max_object = drrs->drr_object; + VERIFY0(dmu_bonus_hold(rwa->os, drrs->drr_object, FTAG, &db)); if ((err = dmu_spill_hold_by_bonus(db, FTAG, &db_spill)) != 0) { dmu_buf_rele(db, FTAG); @@ -2411,6 +2432,9 @@ receive_free(struct receive_writer_arg *rwa, struct dr if (dmu_object_info(rwa->os, drrf->drr_object, NULL) != 0) return (SET_ERROR(EINVAL)); + if (drrf->drr_object > rwa->max_object) + rwa->max_object = drrf->drr_object; + err = dmu_free_long_range(rwa->os, drrf->drr_object, drrf->drr_offset, drrf->drr_length); @@ -3028,6 +3052,41 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, vnode_t *vp, o cv_wait(&rwa.cv, &rwa.mutex); } mutex_exit(&rwa.mutex); + + /* + * If we are receiving a full stream as a clone, all object IDs which + * are greater than the maximum ID referenced in the stream are + * by definition unused and must be freed. Note that it's possible that + * we've resumed this send and the first record we received was the END + * record. In that case, max_object would be 0, but we shouldn't start + * freeing all objects from there; instead we should start from the + * resumeobj. + */ + if (drc->drc_clone && drc->drc_drrb->drr_fromguid == 0) { + uint64_t obj; + if (nvlist_lookup_uint64(begin_nvl, "resume_object", &obj) != 0) + obj = 0; + if (rwa.max_object > obj) + obj = rwa.max_object; + obj++; + int free_err = 0; + int next_err = 0; + + while (next_err == 0) { + free_err = dmu_free_long_object(rwa.os, obj); + if (free_err != 0 && free_err != ENOENT) + break; + + next_err = dmu_object_next(rwa.os, &obj, FALSE, 0); + } + + if (err == 0) { + if (free_err != 0 && free_err != ENOENT) + err = free_err; + else if (next_err != ESRCH) + err = next_err; + } + } cv_destroy(&rwa.cv); mutex_destroy(&rwa.mutex); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_send.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_send.h Thu Aug 2 20:28:09 2018 (r337174) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu_send.h Thu Aug 2 20:29:58 2018 (r337175) @@ -63,6 +63,7 @@ typedef struct dmu_recv_cookie { boolean_t drc_byteswap; boolean_t drc_force; boolean_t drc_resumable; + boolean_t drc_clone; struct avl_tree *drc_guid_to_ds_map; zio_cksum_t drc_cksum; uint64_t drc_newsnapobj; From owner-svn-src-vendor@freebsd.org Thu Aug 2 20:37:54 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83DF0104FAB6; Thu, 2 Aug 2018 20:37:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A2FF7B4E2; Thu, 2 Aug 2018 20:37:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1733516B25; Thu, 2 Aug 2018 20:37:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72KbrWN023637; Thu, 2 Aug 2018 20:37:53 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72Kbro6023636; Thu, 2 Aug 2018 20:37:53 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022037.w72Kbro6023636@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 20:37:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337178 - vendor/illumos/dist/cmd/zdb X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/cmd/zdb X-SVN-Commit-Revision: 337178 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 20:37:54 -0000 Author: mav Date: Thu Aug 2 20:37:53 2018 New Revision: 337178 URL: https://svnweb.freebsd.org/changeset/base/337178 Log: 9523 Large alloc in zdb can cause trouble 16MB alloc in zdb_embedded_block() can cause cores in certain situations (clang, gcc55). OsX commit: https://github.com/openzfsonosx/zfs/commit/ced236a5da6e72ea7bf6d2919fe14e17cffe10f1 FreeBSD commit: https://svnweb.freebsd.org/base?view=revision&revision=326150 illumos/illumos-gate@03a4c2f4bfaca30115963b76445279b36468a614 Reviewed by: Igor Kozhukhov Reviewed by: Andriy Gapon Reviewed by: Matthew Ahrens Approved by: Dan McDonald Author: Jorgen Lundman Modified: vendor/illumos/dist/cmd/zdb/zdb.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Thu Aug 2 20:33:13 2018 (r337177) +++ vendor/illumos/dist/cmd/zdb/zdb.c Thu Aug 2 20:37:53 2018 (r337178) @@ -4906,7 +4906,7 @@ zdb_embedded_block(char *thing) { blkptr_t bp; unsigned long long *words = (void *)&bp; - char buf[SPA_MAXBLOCKSIZE]; + char *buf; int err; bzero(&bp, sizeof (bp)); @@ -4917,16 +4917,22 @@ zdb_embedded_block(char *thing) words + 8, words + 9, words + 10, words + 11, words + 12, words + 13, words + 14, words + 15); if (err != 16) { - (void) printf("invalid input format\n"); + (void) fprintf(stderr, "invalid input format\n"); exit(1); } ASSERT3U(BPE_GET_LSIZE(&bp), <=, SPA_MAXBLOCKSIZE); + buf = malloc(SPA_MAXBLOCKSIZE); + if (buf == NULL) { + (void) fprintf(stderr, "out of memory\n"); + exit(1); + } err = decode_embedded_bp(&bp, buf, BPE_GET_LSIZE(&bp)); if (err != 0) { - (void) printf("decode failed: %u\n", err); + (void) fprintf(stderr, "decode failed: %u\n", err); exit(1); } zdb_dump_block_raw(buf, BPE_GET_LSIZE(&bp), 0); + free(buf); } static boolean_t From owner-svn-src-vendor@freebsd.org Thu Aug 2 20:49:09 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6FF3105022C; Thu, 2 Aug 2018 20:49:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C4887BD9D; Thu, 2 Aug 2018 20:49:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4D4A816CF1; Thu, 2 Aug 2018 20:49:09 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72Kn9AF028853; Thu, 2 Aug 2018 20:49:09 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72Kn8ph028850; Thu, 2 Aug 2018 20:49:08 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022049.w72Kn8ph028850@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 20:49:08 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337180 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337180 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 20:49:10 -0000 Author: mav Date: Thu Aug 2 20:49:08 2018 New Revision: 337180 URL: https://svnweb.freebsd.org/changeset/base/337180 Log: 9539 Make zvol operations use _by_dnode routines Continues what was started in 7801 add more by-dnode routines by fully converting zvols to avoid unnecessary dnode_hold() calls. This saves a small amount of CPU time and slightly improves latencies of operations on zvols. illumos/illumos-gate@8dfe5547fbf0979fc1065a8b6fddc1e940a7cf4f Reviewed by: Matthew Ahrens Reviewed by: Brian Behlendorf Reviewed by: Rick McNeal Approved by: Dan McDonald Author: Richard Yao Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Thu Aug 2 20:44:07 2018 (r337179) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c Thu Aug 2 20:49:08 2018 (r337180) @@ -443,7 +443,7 @@ dmu_spill_hold_by_bonus(dmu_buf_t *bonus, void *tag, d * and can induce severe lock contention when writing to several files * whose dnodes are in the same block. */ -static int +int dmu_buf_hold_array_by_dnode(dnode_t *dn, uint64_t offset, uint64_t length, boolean_t read, void *tag, int *numbufsp, dmu_buf_t ***dbpp, uint32_t flags) { @@ -1302,7 +1302,7 @@ xuio_stat_wbuf_nocopy(void) } #ifdef _KERNEL -static int +int dmu_read_uio_dnode(dnode_t *dn, uio_t *uio, uint64_t size) { dmu_buf_t **dbp; @@ -1411,7 +1411,7 @@ dmu_read_uio(objset_t *os, uint64_t object, uio_t *uio return (err); } -static int +int dmu_write_uio_dnode(dnode_t *dn, uio_t *uio, uint64_t size, dmu_tx_t *tx) { dmu_buf_t **dbp; @@ -1600,22 +1600,17 @@ dmu_return_arcbuf(arc_buf_t *buf) * dmu_write(). */ void -dmu_assign_arcbuf(dmu_buf_t *handle, uint64_t offset, arc_buf_t *buf, +dmu_assign_arcbuf_dnode(dnode_t *dn, uint64_t offset, arc_buf_t *buf, dmu_tx_t *tx) { - dmu_buf_impl_t *dbuf = (dmu_buf_impl_t *)handle; - dnode_t *dn; dmu_buf_impl_t *db; uint32_t blksz = (uint32_t)arc_buf_lsize(buf); uint64_t blkid; - DB_DNODE_ENTER(dbuf); - dn = DB_DNODE(dbuf); rw_enter(&dn->dn_struct_rwlock, RW_READER); blkid = dbuf_whichblock(dn, 0, offset); VERIFY((db = dbuf_hold(dn, blkid, FTAG)) != NULL); rw_exit(&dn->dn_struct_rwlock); - DB_DNODE_EXIT(dbuf); /* * We can only assign if the offset is aligned, the arc buf is the @@ -1632,17 +1627,25 @@ dmu_assign_arcbuf(dmu_buf_t *handle, uint64_t offset, ASSERT3U(arc_get_compression(buf), ==, ZIO_COMPRESS_OFF); ASSERT(!(buf->b_flags & ARC_BUF_FLAG_COMPRESSED)); - DB_DNODE_ENTER(dbuf); - dn = DB_DNODE(dbuf); os = dn->dn_objset; object = dn->dn_object; - DB_DNODE_EXIT(dbuf); dbuf_rele(db, FTAG); dmu_write(os, object, offset, blksz, buf->b_data, tx); dmu_return_arcbuf(buf); XUIOSTAT_BUMP(xuiostat_wbuf_copied); } +} + +void +dmu_assign_arcbuf(dmu_buf_t *handle, uint64_t offset, arc_buf_t *buf, + dmu_tx_t *tx) +{ + dmu_buf_impl_t *dbuf = (dmu_buf_impl_t *)handle; + + DB_DNODE_ENTER(dbuf); + dmu_assign_arcbuf_dnode(DB_DNODE(dbuf), offset, buf, tx); + DB_DNODE_EXIT(dbuf); } typedef struct { Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Thu Aug 2 20:44:07 2018 (r337179) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dmu.h Thu Aug 2 20:49:08 2018 (r337180) @@ -517,6 +517,9 @@ uint64_t dmu_buf_refcount(dmu_buf_t *db); int dmu_buf_hold_array_by_bonus(dmu_buf_t *db, uint64_t offset, uint64_t length, boolean_t read, void *tag, int *numbufsp, dmu_buf_t ***dbpp); +int dmu_buf_hold_array_by_dnode(dnode_t *dn, uint64_t offset, uint64_t length, + boolean_t read, void *tag, int *numbufsp, dmu_buf_t ***dbpp, + uint32_t flags); void dmu_buf_rele_array(dmu_buf_t **, int numbufs, void *tag); typedef void dmu_buf_evict_func_t(void *user_ptr); @@ -755,14 +758,19 @@ void dmu_prealloc(objset_t *os, uint64_t object, uint6 dmu_tx_t *tx); int dmu_read_uio(objset_t *os, uint64_t object, struct uio *uio, uint64_t size); int dmu_read_uio_dbuf(dmu_buf_t *zdb, struct uio *uio, uint64_t size); +int dmu_read_uio_dnode(dnode_t *dn, struct uio *uio, uint64_t size); int dmu_write_uio(objset_t *os, uint64_t object, struct uio *uio, uint64_t size, dmu_tx_t *tx); int dmu_write_uio_dbuf(dmu_buf_t *zdb, struct uio *uio, uint64_t size, dmu_tx_t *tx); +int dmu_write_uio_dnode(dnode_t *dn, struct uio *uio, uint64_t size, + dmu_tx_t *tx); int dmu_write_pages(objset_t *os, uint64_t object, uint64_t offset, uint64_t size, struct page *pp, dmu_tx_t *tx); struct arc_buf *dmu_request_arcbuf(dmu_buf_t *handle, int size); void dmu_return_arcbuf(struct arc_buf *buf); +void dmu_assign_arcbuf_dnode(dnode_t *handle, uint64_t offset, + struct arc_buf *buf, dmu_tx_t *tx); void dmu_assign_arcbuf(dmu_buf_t *handle, uint64_t offset, struct arc_buf *buf, dmu_tx_t *tx); int dmu_xuio_init(struct xuio *uio, int niov); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Thu Aug 2 20:44:07 2018 (r337179) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Thu Aug 2 20:49:08 2018 (r337180) @@ -128,7 +128,7 @@ typedef struct zvol_state { zilog_t *zv_zilog; /* ZIL handle */ list_t zv_extents; /* List of extents for dump */ znode_t zv_znode; /* for range locking */ - dmu_buf_t *zv_dbuf; /* bonus handle */ + dnode_t *zv_dn; /* dnode hold */ } zvol_state_t; /* @@ -646,7 +646,7 @@ zvol_first_open(zvol_state_t *zv) return (error); } - error = dmu_bonus_hold(os, ZVOL_OBJ, zvol_tag, &zv->zv_dbuf); + error = dnode_hold(os, ZVOL_OBJ, zvol_tag, &zv->zv_dn); if (error) { dmu_objset_disown(os, zvol_tag); return (error); @@ -671,8 +671,8 @@ zvol_last_close(zvol_state_t *zv) zil_close(zv->zv_zilog); zv->zv_zilog = NULL; - dmu_buf_rele(zv->zv_dbuf, zvol_tag); - zv->zv_dbuf = NULL; + dnode_rele(zv->zv_dn, zvol_tag); + zv->zv_dn = NULL; /* * Evict cached data @@ -993,8 +993,6 @@ static int zvol_get_data(void *arg, lr_write_t *lr, char *buf, struct lwb *lwb, zio_t *zio) { zvol_state_t *zv = arg; - objset_t *os = zv->zv_objset; - uint64_t object = ZVOL_OBJ; uint64_t offset = lr->lr_offset; uint64_t size = lr->lr_length; /* length of user data */ dmu_buf_t *db; @@ -1018,7 +1016,7 @@ zvol_get_data(void *arg, lr_write_t *lr, char *buf, st if (buf != NULL) { /* immediate write */ zgd->zgd_rl = zfs_range_lock(&zv->zv_znode, offset, size, RL_READER); - error = dmu_read(os, object, offset, size, buf, + error = dmu_read_by_dnode(zv->zv_dn, offset, size, buf, DMU_READ_NO_PREFETCH); } else { /* indirect write */ /* @@ -1031,7 +1029,7 @@ zvol_get_data(void *arg, lr_write_t *lr, char *buf, st offset = P2ALIGN(offset, size); zgd->zgd_rl = zfs_range_lock(&zv->zv_znode, offset, size, RL_READER); - error = dmu_buf_hold(os, object, offset, zgd, &db, + error = dmu_buf_hold_by_dnode(zv->zv_dn, offset, zgd, &db, DMU_READ_NO_PREFETCH); if (error == 0) { blkptr_t *bp = &lr->lr_blkptr; @@ -1098,8 +1096,8 @@ zvol_log_write(zvol_state_t *zv, dmu_tx_t *tx, offset_ itx = zil_itx_create(TX_WRITE, sizeof (*lr) + (wr_state == WR_COPIED ? len : 0)); lr = (lr_write_t *)&itx->itx_lr; - if (wr_state == WR_COPIED && dmu_read(zv->zv_objset, - ZVOL_OBJ, off, len, lr + 1, DMU_READ_NO_PREFETCH) != 0) { + if (wr_state == WR_COPIED && dmu_read_by_dnode(zv->zv_dn, + off, len, lr + 1, DMU_READ_NO_PREFETCH) != 0) { zil_itx_destroy(itx); itx = zil_itx_create(TX_WRITE, sizeof (*lr)); lr = (lr_write_t *)&itx->itx_lr; @@ -1468,7 +1466,7 @@ zvol_write(dev_t dev, uio_t *uio, cred_t *cr) dmu_tx_abort(tx); break; } - error = dmu_write_uio_dbuf(zv->zv_dbuf, uio, bytes, tx); + error = dmu_write_uio_dnode(zv->zv_dn, uio, bytes, tx); if (error == 0) zvol_log_write(zv, tx, off, bytes, sync); dmu_tx_commit(tx); @@ -1549,7 +1547,7 @@ zvol_getefi(void *arg, int flag, uint64_t vs, uint8_t int zvol_get_volume_params(minor_t minor, uint64_t *blksize, uint64_t *max_xfer_len, void **minor_hdl, void **objset_hdl, void **zil_hdl, - void **rl_hdl, void **bonus_hdl) + void **rl_hdl, void **dnode_hdl) { zvol_state_t *zv; @@ -1560,7 +1558,7 @@ zvol_get_volume_params(minor_t minor, uint64_t *blksiz return (SET_ERROR(ENXIO)); ASSERT(blksize && max_xfer_len && minor_hdl && - objset_hdl && zil_hdl && rl_hdl && bonus_hdl); + objset_hdl && zil_hdl && rl_hdl && dnode_hdl); *blksize = zv->zv_volblocksize; *max_xfer_len = (uint64_t)zvol_maxphys; @@ -1568,7 +1566,7 @@ zvol_get_volume_params(minor_t minor, uint64_t *blksiz *objset_hdl = zv->zv_objset; *zil_hdl = zv->zv_zilog; *rl_hdl = &zv->zv_znode; - *bonus_hdl = zv->zv_dbuf; + *dnode_hdl = zv->zv_dn; return (0); } From owner-svn-src-vendor@freebsd.org Thu Aug 2 21:12:54 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E0411050C0F; Thu, 2 Aug 2018 21:12:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0E6187CC20; Thu, 2 Aug 2018 21:12:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DE988171BD; Thu, 2 Aug 2018 21:12:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72LCrLT043807; Thu, 2 Aug 2018 21:12:53 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72LCrDt043803; Thu, 2 Aug 2018 21:12:53 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022112.w72LCrDt043803@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 21:12:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337182 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Commit-Revision: 337182 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 21:12:54 -0000 Author: mav Date: Thu Aug 2 21:12:52 2018 New Revision: 337182 URL: https://svnweb.freebsd.org/changeset/base/337182 Log: 9330 stack overflow when creating a deeply nested dataset Datasets that are deeply nested (~100 levels) are impractical. We just put a limit of 50 levels to newly created datasets. Existing datasets should work without a problem. illumos/illumos-gate@5ac95da7d61660aa299c287a39277cb0372be959 Reviewed by: John Kennedy Reviewed by: Matt Ahrens Approved by: Garrett D'Amore Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/common/zfs/zfs_namecheck.c vendor-sys/illumos/dist/common/zfs/zfs_namecheck.h vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/man/man1m/zfs.1m Modified: vendor-sys/illumos/dist/common/zfs/zfs_namecheck.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_namecheck.c Thu Aug 2 21:07:04 2018 (r337181) +++ vendor-sys/illumos/dist/common/zfs/zfs_namecheck.c Thu Aug 2 21:12:52 2018 (r337182) @@ -23,7 +23,7 @@ * Use is subject to license terms. */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2013, 2016 by Delphix. All rights reserved. */ /* @@ -34,8 +34,6 @@ * name is invalid. In the kernel, we only care whether it's valid or not. * Each routine therefore takes a 'namecheck_err_t' which describes exactly why * the name failed to validate. - * - * Each function returns 0 on success, -1 on error. */ #if defined(_KERNEL) @@ -50,6 +48,14 @@ #include "zfs_namecheck.h" #include "zfs_deleg.h" +/* + * Deeply nested datasets can overflow the stack, so we put a limit + * in the amount of nesting a path can have. zfs_max_dataset_nesting + * can be tuned temporarily to fix existing datasets that exceed our + * predefined limit. + */ +int zfs_max_dataset_nesting = 50; + static int valid_char(char c) { @@ -60,10 +66,35 @@ valid_char(char c) } /* + * Looks at a path and returns its level of nesting (depth). + */ +int +get_dataset_depth(const char *path) +{ + const char *loc = path; + int nesting = 0; + + /* + * Keep track of nesting until you hit the end of the + * path or found the snapshot/bookmark seperator. + */ + for (int i = 0; loc[i] != '\0' && + loc[i] != '@' && + loc[i] != '#'; i++) { + if (loc[i] == '/') + nesting++; + } + + return (nesting); +} + +/* * Snapshot names must be made up of alphanumeric characters plus the following * characters: * - * [-_.: ] + * [-_.: ] + * + * Returns 0 on success, -1 on error. */ int zfs_component_namecheck(const char *path, namecheck_err_t *why, char *what) @@ -99,6 +130,8 @@ zfs_component_namecheck(const char *path, namecheck_er * Permissions set name must start with the letter '@' followed by the * same character restrictions as snapshot names, except that the name * cannot exceed 64 characters. + * + * Returns 0 on success, -1 on error. */ int permset_namecheck(const char *path, namecheck_err_t *why, char *what) @@ -121,28 +154,40 @@ permset_namecheck(const char *path, namecheck_err_t *w } /* + * Dataset paths should not be deeper than zfs_max_dataset_nesting + * in terms of nesting. + * + * Returns 0 on success, -1 on error. + */ +int +dataset_nestcheck(const char *path) +{ + return ((get_dataset_depth(path) < zfs_max_dataset_nesting) ? 0 : -1); +} + +/* * Entity names must be of the following form: * - * [component/]*[component][(@|#)component]? + * [component/]*[component][(@|#)component]? * * Where each component is made up of alphanumeric characters plus the following * characters: * - * [-_.:%] + * [-_.:%] * * We allow '%' here as we use that character internally to create unique * names for temporary clones (for online recv). + * + * Returns 0 on success, -1 on error. */ int entity_namecheck(const char *path, namecheck_err_t *why, char *what) { - const char *start, *end; - int found_delim; + const char *end; /* * Make sure the name is not too long. */ - if (strlen(path) >= ZFS_MAX_DATASET_NAME_LEN) { if (why) *why = NAME_ERR_TOOLONG; @@ -162,8 +207,8 @@ entity_namecheck(const char *path, namecheck_err_t *wh return (-1); } - start = path; - found_delim = 0; + const char *start = path; + boolean_t found_delim = B_FALSE; for (;;) { /* Find the end of this component */ end = start; @@ -198,7 +243,7 @@ entity_namecheck(const char *path, namecheck_err_t *wh return (-1); } - found_delim = 1; + found_delim = B_TRUE; } /* Zero-length components are not allowed */ @@ -250,6 +295,8 @@ dataset_namecheck(const char *path, namecheck_err_t *w * mountpoint names must be of the following form: * * /[component][/]*[component][/] + * + * Returns 0 on success, -1 on error. */ int mountpoint_namecheck(const char *path, namecheck_err_t *why) @@ -294,6 +341,8 @@ mountpoint_namecheck(const char *path, namecheck_err_t * dataset names, with the additional restriction that the pool name must begin * with a letter. The pool names 'raidz' and 'mirror' are also reserved names * that cannot be used. + * + * Returns 0 on success, -1 on error. */ int pool_namecheck(const char *pool, namecheck_err_t *why, char *what) Modified: vendor-sys/illumos/dist/common/zfs/zfs_namecheck.h ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_namecheck.h Thu Aug 2 21:07:04 2018 (r337181) +++ vendor-sys/illumos/dist/common/zfs/zfs_namecheck.h Thu Aug 2 21:12:52 2018 (r337182) @@ -23,7 +23,7 @@ * Use is subject to license terms. */ /* - * Copyright (c) 2013 by Delphix. All rights reserved. + * Copyright (c) 2013, 2016 by Delphix. All rights reserved. */ #ifndef _ZFS_NAMECHECK_H @@ -48,9 +48,13 @@ typedef enum { #define ZFS_PERMSET_MAXLEN 64 +extern int zfs_max_dataset_nesting; + +int get_dataset_depth(const char *); int pool_namecheck(const char *, namecheck_err_t *, char *); int entity_namecheck(const char *, namecheck_err_t *, char *); int dataset_namecheck(const char *, namecheck_err_t *, char *); +int dataset_nestcheck(const char *); int mountpoint_namecheck(const char *, namecheck_err_t *); int zfs_component_namecheck(const char *, namecheck_err_t *, char *); int permset_namecheck(const char *, namecheck_err_t *, char *); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Thu Aug 2 21:07:04 2018 (r337181) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c Thu Aug 2 21:12:52 2018 (r337182) @@ -54,6 +54,7 @@ #include #include #include +#include "zfs_namecheck.h" /* * Needed to close a window in dnode_move() that allows the objset to be freed @@ -909,6 +910,9 @@ dmu_objset_create_check(void *arg, dmu_tx_t *tx) return (SET_ERROR(EINVAL)); if (strlen(doca->doca_name) >= ZFS_MAX_DATASET_NAME_LEN) + return (SET_ERROR(ENAMETOOLONG)); + + if (dataset_nestcheck(doca->doca_name) != 0) return (SET_ERROR(ENAMETOOLONG)); error = dsl_dir_hold(dp, doca->doca_name, FTAG, &pdd, &tail); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c Thu Aug 2 21:07:04 2018 (r337181) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c Thu Aug 2 21:12:52 2018 (r337182) @@ -1810,17 +1810,29 @@ typedef struct dsl_dir_rename_arg { cred_t *ddra_cred; } dsl_dir_rename_arg_t; +typedef struct dsl_valid_rename_arg { + int char_delta; + int nest_delta; +} dsl_valid_rename_arg_t; + /* ARGSUSED */ static int dsl_valid_rename(dsl_pool_t *dp, dsl_dataset_t *ds, void *arg) { - int *deltap = arg; + dsl_valid_rename_arg_t *dvra = arg; char namebuf[ZFS_MAX_DATASET_NAME_LEN]; dsl_dataset_name(ds, namebuf); - if (strlen(namebuf) + *deltap >= ZFS_MAX_DATASET_NAME_LEN) + ASSERT3U(strnlen(namebuf, ZFS_MAX_DATASET_NAME_LEN), + <, ZFS_MAX_DATASET_NAME_LEN); + int namelen = strlen(namebuf) + dvra->char_delta; + int depth = get_dataset_depth(namebuf) + dvra->nest_delta; + + if (namelen >= ZFS_MAX_DATASET_NAME_LEN) return (SET_ERROR(ENAMETOOLONG)); + if (dvra->nest_delta > 0 && depth >= zfs_max_dataset_nesting) + return (SET_ERROR(ENAMETOOLONG)); return (0); } @@ -1830,9 +1842,9 @@ dsl_dir_rename_check(void *arg, dmu_tx_t *tx) dsl_dir_rename_arg_t *ddra = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dir_t *dd, *newparent; + dsl_valid_rename_arg_t dvra; const char *mynewname; int error; - int delta = strlen(ddra->ddra_newname) - strlen(ddra->ddra_oldname); /* target dir should exist */ error = dsl_dir_hold(dp, ddra->ddra_oldname, FTAG, &dd, NULL); @@ -1861,10 +1873,19 @@ dsl_dir_rename_check(void *arg, dmu_tx_t *tx) return (SET_ERROR(EEXIST)); } + ASSERT3U(strnlen(ddra->ddra_newname, ZFS_MAX_DATASET_NAME_LEN), + <, ZFS_MAX_DATASET_NAME_LEN); + ASSERT3U(strnlen(ddra->ddra_oldname, ZFS_MAX_DATASET_NAME_LEN), + <, ZFS_MAX_DATASET_NAME_LEN); + dvra.char_delta = strlen(ddra->ddra_newname) + - strlen(ddra->ddra_oldname); + dvra.nest_delta = get_dataset_depth(ddra->ddra_newname) + - get_dataset_depth(ddra->ddra_oldname); + /* if the name length is growing, validate child name lengths */ - if (delta > 0) { + if (dvra.char_delta > 0 || dvra.nest_delta > 0) { error = dmu_objset_find_dp(dp, dd->dd_object, dsl_valid_rename, - &delta, DS_FIND_CHILDREN | DS_FIND_SNAPSHOTS); + &dvra, DS_FIND_CHILDREN | DS_FIND_SNAPSHOTS); if (error != 0) { dsl_dir_rele(newparent, FTAG); dsl_dir_rele(dd, FTAG); From owner-svn-src-vendor@freebsd.org Thu Aug 2 21:12:55 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB9FD1050C12; Thu, 2 Aug 2018 21:12:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 912747CC21; Thu, 2 Aug 2018 21:12:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7232F171BE; Thu, 2 Aug 2018 21:12:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72LCsmG043814; Thu, 2 Aug 2018 21:12:54 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72LCsT1043812; Thu, 2 Aug 2018 21:12:54 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022112.w72LCsT1043812@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 21:12:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337182 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Commit-Revision: 337182 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 21:12:55 -0000 Author: mav Date: Thu Aug 2 21:12:52 2018 New Revision: 337182 URL: https://svnweb.freebsd.org/changeset/base/337182 Log: 9330 stack overflow when creating a deeply nested dataset Datasets that are deeply nested (~100 levels) are impractical. We just put a limit of 50 levels to newly created datasets. Existing datasets should work without a problem. illumos/illumos-gate@5ac95da7d61660aa299c287a39277cb0372be959 Reviewed by: John Kennedy Reviewed by: Matt Ahrens Approved by: Garrett D'Amore Author: Serapheim Dimitropoulos Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/man/man1m/zfs.1m Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zfs_namecheck.c vendor-sys/illumos/dist/common/zfs/zfs_namecheck.h vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_objset.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Thu Aug 2 21:07:04 2018 (r337181) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Thu Aug 2 21:12:52 2018 (r337182) @@ -3409,8 +3409,22 @@ zfs_create_ancestors(libzfs_handle_t *hdl, const char { int prefix; char *path_copy; + char errbuf[1024]; int rc = 0; + (void) snprintf(errbuf, sizeof (errbuf), dgettext(TEXT_DOMAIN, + "cannot create '%s'"), path); + + /* + * Check that we are not passing the nesting limit + * before we start creating any ancestors. + */ + if (dataset_nestcheck(path) != 0) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "maximum name nesting depth exceeded")); + return (zfs_error(hdl, EZFS_INVALIDNAME, errbuf)); + } + if (check_parents(hdl, path, NULL, B_TRUE, &prefix) != 0) return (-1); @@ -3446,6 +3460,12 @@ zfs_create(libzfs_handle_t *hdl, const char *path, zfs if (!zfs_validate_name(hdl, path, type, B_TRUE)) return (zfs_error(hdl, EZFS_INVALIDNAME, errbuf)); + if (dataset_nestcheck(path) != 0) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "maximum name nesting depth exceeded")); + return (zfs_error(hdl, EZFS_INVALIDNAME, errbuf)); + } + /* validate parents exist */ if (check_parents(hdl, path, &zoned, B_FALSE, NULL) != 0) return (-1); @@ -4233,6 +4253,7 @@ zfs_rename(zfs_handle_t *zhp, const char *target, bool errbuf)); } } + if (!zfs_validate_name(hdl, target, zhp->zfs_type, B_TRUE)) return (zfs_error(hdl, EZFS_INVALIDNAME, errbuf)); } else { Modified: vendor/illumos/dist/man/man1m/zfs.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zfs.1m Thu Aug 2 21:07:04 2018 (r337181) +++ vendor/illumos/dist/man/man1m/zfs.1m Thu Aug 2 21:12:52 2018 (r337182) @@ -289,7 +289,8 @@ pool/{filesystem,volume,snapshot} .Pp where the maximum length of a dataset name is .Dv MAXNAMELEN -.Pq 256 bytes . +.Pq 256 bytes +and the maximum amount of nesting allowed in a path is 50 levels deep. .Pp A dataset can be one of the following: .Bl -tag -width "file system" From owner-svn-src-vendor@freebsd.org Thu Aug 2 21:24:05 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1A2110512B7; Thu, 2 Aug 2018 21:24:05 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 754DA7D5B9; Thu, 2 Aug 2018 21:24:05 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 567FC17355; Thu, 2 Aug 2018 21:24:05 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72LO5FX049470; Thu, 2 Aug 2018 21:24:05 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72LO50A049469; Thu, 2 Aug 2018 21:24:05 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022124.w72LO50A049469@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 21:24:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337184 - vendor/illumos/dist/lib/libzfs/common X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor/illumos/dist/lib/libzfs/common X-SVN-Commit-Revision: 337184 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 21:24:06 -0000 Author: mav Date: Thu Aug 2 21:24:04 2018 New Revision: 337184 URL: https://svnweb.freebsd.org/changeset/base/337184 Log: 9457 libzfs_import.c:add_config() has a memory leak A memory leak occurs on lines 209 and 213 because the config is not freed in the error case. The interface to add_config() seems less than ideal - it would be better if it copied any data necessary from the config and the caller freed it. illumos/illumos-gate@ddfe901b12348d31c500fb57f9174e88860a4061 Reviewed by: Matt Ahrens Reviewed by: Serapheim Dimitropoulos Approved by: Robert Mustacchi Author: sara hartse Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_import.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Thu Aug 2 21:19:35 2018 (r337183) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_import.c Thu Aug 2 21:24:04 2018 (r337184) @@ -33,7 +33,7 @@ * ZFS label of each device. If we successfully read the label, then we * organize the configuration information in the following hierarchy: * - * pool guid -> toplevel vdev guid -> label txg + * pool guid -> toplevel vdev guid -> label txg * * Duplicate entries matching this same tuple will be discarded. Once we have * examined every device, we pick the best label txg config for each toplevel @@ -217,7 +217,6 @@ add_config(libzfs_handle_t *hdl, pool_list_t *pl, cons ne->ne_next = pl->names; pl->names = ne; - nvlist_free(config); return (0); } @@ -237,7 +236,6 @@ add_config(libzfs_handle_t *hdl, pool_list_t *pl, cons &top_guid) != 0 || nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_TXG, &txg) != 0 || txg == 0) { - nvlist_free(config); return (0); } @@ -252,7 +250,6 @@ add_config(libzfs_handle_t *hdl, pool_list_t *pl, cons if (pe == NULL) { if ((pe = zfs_alloc(hdl, sizeof (pool_entry_t))) == NULL) { - nvlist_free(config); return (-1); } pe->pe_guid = pool_guid; @@ -271,7 +268,6 @@ add_config(libzfs_handle_t *hdl, pool_list_t *pl, cons if (ve == NULL) { if ((ve = zfs_alloc(hdl, sizeof (vdev_entry_t))) == NULL) { - nvlist_free(config); return (-1); } ve->ve_guid = top_guid; @@ -291,15 +287,12 @@ add_config(libzfs_handle_t *hdl, pool_list_t *pl, cons if (ce == NULL) { if ((ce = zfs_alloc(hdl, sizeof (config_entry_t))) == NULL) { - nvlist_free(config); return (-1); } ce->ce_txg = txg; - ce->ce_config = config; + ce->ce_config = fnvlist_dup(config); ce->ce_next = ve->ve_configs; ve->ve_configs = ce; - } else { - nvlist_free(config); } /* @@ -1235,9 +1228,7 @@ zpool_find_import_impl(libzfs_handle_t *hdl, importarg &this_guid) == 0 && iarg->guid == this_guid; } - if (!matched) { - nvlist_free(config); - } else { + if (matched) { /* * use the non-raw path for the config */ @@ -1247,6 +1238,7 @@ zpool_find_import_impl(libzfs_handle_t *hdl, importarg config) != 0) config_failed = B_TRUE; } + nvlist_free(config); } free(slice->rn_name); free(slice); From owner-svn-src-vendor@freebsd.org Thu Aug 2 21:52:48 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27EB1105207D; Thu, 2 Aug 2018 21:52:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC31D7E82A; Thu, 2 Aug 2018 21:52:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AAD3317844; Thu, 2 Aug 2018 21:52:47 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72Lqlx4064747; Thu, 2 Aug 2018 21:52:47 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72Lqlvh064746; Thu, 2 Aug 2018 21:52:47 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022152.w72Lqlvh064746@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 21:52:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337187 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337187 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 21:52:48 -0000 Author: mav Date: Thu Aug 2 21:52:47 2018 New Revision: 337187 URL: https://svnweb.freebsd.org/changeset/base/337187 Log: 9479 fix wrong format specifier for vdev_id illumos/illumos-gate@c7a7b2fa084facd68c3d1e860dfabe20c713877b Reviewed by: Matt Ahrens Approved by: Robert Mustacchi Author: Andriy Gapon Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Aug 2 21:33:45 2018 (r337186) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Aug 2 21:52:47 2018 (r337187) @@ -168,7 +168,7 @@ vdev_dbgmsg_print_tree(vdev_t *vd, int indent) } zfs_dbgmsg("%*svdev %u: %s%s, guid: %llu, path: %s, %s", indent, - "", vd->vdev_id, vd->vdev_ops->vdev_op_type, + "", (int)vd->vdev_id, vd->vdev_ops->vdev_op_type, vd->vdev_islog ? " (log)" : "", (u_longlong_t)vd->vdev_guid, vd->vdev_path ? vd->vdev_path : "N/A", state); From owner-svn-src-vendor@freebsd.org Thu Aug 2 21:58:01 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77F3F1052349; Thu, 2 Aug 2018 21:58:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E0037F310; Thu, 2 Aug 2018 21:58:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0AD4E17872; Thu, 2 Aug 2018 21:58:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72Lw0Og065328; Thu, 2 Aug 2018 21:58:00 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72Lw0G0065323; Thu, 2 Aug 2018 21:58:00 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022158.w72Lw0G0065323@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 21:58:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337190 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337190 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 21:58:01 -0000 Author: mav Date: Thu Aug 2 21:57:59 2018 New Revision: 337190 URL: https://svnweb.freebsd.org/changeset/base/337190 Log: 9486 reduce memory used by device removal on fragmented pools In the most fragmented real-world cases, this reduces memory used by the mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less fragmented cases will typically also see around 50-100MB of RAM per 1TB of storage. illumos/illumos-gate@cfd63e1b1bcf7ba4bf72f55ddbd87ce008d2986d Reviewed by: George Wilson Reviewed by: Serapheim Dimitropoulos Reviewed by: Brian Behlendorf Reviewed by: Tim Chase Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c Thu Aug 2 21:54:34 2018 (r337189) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/range_tree.c Thu Aug 2 21:57:59 2018 (r337190) @@ -298,7 +298,6 @@ range_tree_remove(void *arg, uint64_t start, uint64_t static range_seg_t * range_tree_find_impl(range_tree_t *rt, uint64_t start, uint64_t size) { - avl_index_t where; range_seg_t rsearch; uint64_t end = start + size; @@ -306,7 +305,7 @@ range_tree_find_impl(range_tree_t *rt, uint64_t start, rsearch.rs_start = start; rsearch.rs_end = end; - return (avl_find(&rt->rt_root, &rsearch, &where)); + return (avl_find(&rt->rt_root, &rsearch, NULL)); } static range_seg_t * @@ -406,4 +405,24 @@ range_tree_is_empty(range_tree_t *rt) { ASSERT(rt != NULL); return (range_tree_space(rt) == 0); +} + +uint64_t +range_tree_min(range_tree_t *rt) +{ + range_seg_t *rs = avl_first(&rt->rt_root); + return (rs != NULL ? rs->rs_start : 0); +} + +uint64_t +range_tree_max(range_tree_t *rt) +{ + range_seg_t *rs = avl_last(&rt->rt_root); + return (rs != NULL ? rs->rs_end : 0); +} + +uint64_t +range_tree_span(range_tree_t *rt) +{ + return (range_tree_max(rt) - range_tree_min(rt)); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h Thu Aug 2 21:54:34 2018 (r337189) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/range_tree.h Thu Aug 2 21:57:59 2018 (r337190) @@ -86,6 +86,9 @@ boolean_t range_tree_is_empty(range_tree_t *rt); void range_tree_verify(range_tree_t *rt, uint64_t start, uint64_t size); void range_tree_swap(range_tree_t **rtsrc, range_tree_t **rtdst); void range_tree_stat_verify(range_tree_t *rt); +uint64_t range_tree_min(range_tree_t *rt); +uint64_t range_tree_max(range_tree_t *rt); +uint64_t range_tree_span(range_tree_t *rt); void range_tree_add(void *arg, uint64_t start, uint64_t size); void range_tree_remove(void *arg, uint64_t start, uint64_t size); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h Thu Aug 2 21:54:34 2018 (r337189) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_removal.h Thu Aug 2 21:57:59 2018 (r337190) @@ -86,6 +86,9 @@ extern void spa_vdev_remove_suspend(spa_t *); extern int spa_vdev_remove_cancel(spa_t *); extern void spa_vdev_removal_destroy(spa_vdev_removal_t *svr); +extern int vdev_removal_max_span; +extern int zfs_remove_max_segment; + #ifdef __cplusplus } #endif Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Thu Aug 2 21:54:34 2018 (r337189) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c Thu Aug 2 21:57:59 2018 (r337190) @@ -33,15 +33,15 @@ * 1. Uniquely identify this device as part of a ZFS pool and confirm its * identity within the pool. * - * 2. Verify that all the devices given in a configuration are present + * 2. Verify that all the devices given in a configuration are present * within the pool. * - * 3. Determine the uberblock for the pool. + * 3. Determine the uberblock for the pool. * - * 4. In case of an import operation, determine the configuration of the + * 4. In case of an import operation, determine the configuration of the * toplevel vdev of which it is a part. * - * 5. If an import operation cannot find all the devices in the pool, + * 5. If an import operation cannot find all the devices in the pool, * provide enough information to the administrator to determine which * devices are missing. * @@ -77,9 +77,9 @@ * In order to identify which labels are valid, the labels are written in the * following manner: * - * 1. For each vdev, update 'L1' to the new label - * 2. Update the uberblock - * 3. For each vdev, update 'L2' to the new label + * 1. For each vdev, update 'L1' to the new label + * 2. Update the uberblock + * 3. For each vdev, update 'L2' to the new label * * Given arbitrary failure, we can determine the correct label to use based on * the transaction group. If we fail after updating L1 but before updating the @@ -117,19 +117,19 @@ * * The nvlist describing the pool and vdev contains the following elements: * - * version ZFS on-disk version - * name Pool name - * state Pool state - * txg Transaction group in which this label was written - * pool_guid Unique identifier for this pool - * vdev_tree An nvlist describing vdev tree. + * version ZFS on-disk version + * name Pool name + * state Pool state + * txg Transaction group in which this label was written + * pool_guid Unique identifier for this pool + * vdev_tree An nvlist describing vdev tree. * features_for_read * An nvlist of the features necessary for reading the MOS. * * Each leaf device label also contains the following: * - * top_guid Unique ID for top-level vdev in which this is contained - * guid Unique ID for the leaf vdev + * top_guid Unique ID for top-level vdev in which this is contained + * guid Unique ID for the leaf vdev * * The 'vs' configuration follows the format described in 'spa_config.c'. */ @@ -390,22 +390,33 @@ vdev_config_generate(spa_t *spa, vdev_t *vd, boolean_t * histograms. */ uint64_t seg_count = 0; + uint64_t to_alloc = vd->vdev_stat.vs_alloc; /* * There are the same number of allocated segments * as free segments, so we will have at least one - * entry per free segment. + * entry per free segment. However, small free + * segments (smaller than vdev_removal_max_span) + * will be combined with adjacent allocated segments + * as a single mapping. */ for (int i = 0; i < RANGE_TREE_HISTOGRAM_SIZE; i++) { - seg_count += vd->vdev_mg->mg_histogram[i]; + if (1ULL << (i + 1) < vdev_removal_max_span) { + to_alloc += + vd->vdev_mg->mg_histogram[i] << + i + 1; + } else { + seg_count += + vd->vdev_mg->mg_histogram[i]; + } } /* - * The maximum length of a mapping is SPA_MAXBLOCKSIZE, - * so we need at least one entry per SPA_MAXBLOCKSIZE - * of allocated data. + * The maximum length of a mapping is + * zfs_remove_max_segment, so we need at least one entry + * per zfs_remove_max_segment of allocated data. */ - seg_count += vd->vdev_stat.vs_alloc / SPA_MAXBLOCKSIZE; + seg_count += to_alloc / zfs_remove_max_segment; fnvlist_add_uint64(nv, ZPOOL_CONFIG_INDIRECT_SIZE, seg_count * Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Thu Aug 2 21:54:34 2018 (r337189) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_removal.c Thu Aug 2 21:57:59 2018 (r337190) @@ -106,6 +106,24 @@ int zfs_remove_max_copy_bytes = 64 * 1024 * 1024; int zfs_remove_max_segment = 1024 * 1024; /* + * Allow a remap segment to span free chunks of at most this size. The main + * impact of a larger span is that we will read and write larger, more + * contiguous chunks, with more "unnecessary" data -- trading off bandwidth + * for iops. The value here was chosen to align with + * zfs_vdev_read_gap_limit, which is a similar concept when doing regular + * reads (but there's no reason it has to be the same). + * + * Additionally, a higher span will have the following relatively minor + * effects: + * - the mapping will be smaller, since one entry can cover more allocated + * segments + * - more of the fragmentation in the removing device will be preserved + * - we'll do larger allocations, which may fail and fall back on smaller + * allocations + */ +int vdev_removal_max_span = 32 * 1024; + +/* * This is used by the test suite so that it can ensure that certain * actions happen while in the middle of a removal. */ @@ -726,13 +744,52 @@ vdev_mapping_sync(void *arg, dmu_tx_t *tx) spa_sync_removing_state(spa, tx); } +typedef struct vdev_copy_segment_arg { + spa_t *vcsa_spa; + dva_t *vcsa_dest_dva; + uint64_t vcsa_txg; + range_tree_t *vcsa_obsolete_segs; +} vdev_copy_segment_arg_t; + +static void +unalloc_seg(void *arg, uint64_t start, uint64_t size) +{ + vdev_copy_segment_arg_t *vcsa = arg; + spa_t *spa = vcsa->vcsa_spa; + blkptr_t bp = { 0 }; + + BP_SET_BIRTH(&bp, TXG_INITIAL, TXG_INITIAL); + BP_SET_LSIZE(&bp, size); + BP_SET_PSIZE(&bp, size); + BP_SET_COMPRESS(&bp, ZIO_COMPRESS_OFF); + BP_SET_CHECKSUM(&bp, ZIO_CHECKSUM_OFF); + BP_SET_TYPE(&bp, DMU_OT_NONE); + BP_SET_LEVEL(&bp, 0); + BP_SET_DEDUP(&bp, 0); + BP_SET_BYTEORDER(&bp, ZFS_HOST_BYTEORDER); + + DVA_SET_VDEV(&bp.blk_dva[0], DVA_GET_VDEV(vcsa->vcsa_dest_dva)); + DVA_SET_OFFSET(&bp.blk_dva[0], + DVA_GET_OFFSET(vcsa->vcsa_dest_dva) + start); + DVA_SET_ASIZE(&bp.blk_dva[0], size); + + zio_free(spa, vcsa->vcsa_txg, &bp); +} + /* * All reads and writes associated with a call to spa_vdev_copy_segment() * are done. */ static void -spa_vdev_copy_nullzio_done(zio_t *zio) +spa_vdev_copy_segment_done(zio_t *zio) { + vdev_copy_segment_arg_t *vcsa = zio->io_private; + + range_tree_vacate(vcsa->vcsa_obsolete_segs, + unalloc_seg, vcsa); + range_tree_destroy(vcsa->vcsa_obsolete_segs); + kmem_free(vcsa, sizeof (*vcsa)); + spa_config_exit(zio->io_spa, SCL_STATE, zio->io_spa); } @@ -849,7 +906,8 @@ spa_vdev_copy_one_child(vdev_copy_arg_t *vca, zio_t *n * read from the old location and write to the new location. */ static int -spa_vdev_copy_segment(vdev_t *vd, uint64_t start, uint64_t size, uint64_t txg, +spa_vdev_copy_segment(vdev_t *vd, range_tree_t *segs, + uint64_t maxalloc, uint64_t txg, vdev_copy_arg_t *vca, zio_alloc_list_t *zal) { metaslab_group_t *mg = vd->vdev_mg; @@ -857,9 +915,40 @@ spa_vdev_copy_segment(vdev_t *vd, uint64_t start, uint spa_vdev_removal_t *svr = spa->spa_vdev_removal; vdev_indirect_mapping_entry_t *entry; dva_t dst = { 0 }; + uint64_t start = range_tree_min(segs); - ASSERT3U(size, <=, SPA_MAXBLOCKSIZE); + ASSERT3U(maxalloc, <=, SPA_MAXBLOCKSIZE); + uint64_t size = range_tree_span(segs); + if (range_tree_span(segs) > maxalloc) { + /* + * We can't allocate all the segments. Prefer to end + * the allocation at the end of a segment, thus avoiding + * additional split blocks. + */ + range_seg_t search; + avl_index_t where; + search.rs_start = start + maxalloc; + search.rs_end = search.rs_start; + range_seg_t *rs = avl_find(&segs->rt_root, &search, &where); + if (rs == NULL) { + rs = avl_nearest(&segs->rt_root, where, AVL_BEFORE); + } else { + rs = AVL_PREV(&segs->rt_root, rs); + } + if (rs != NULL) { + size = rs->rs_end - start; + } else { + /* + * There are no segments that end before maxalloc. + * I.e. the first segment is larger than maxalloc, + * so we must split it. + */ + size = maxalloc; + } + } + ASSERT3U(size, <=, maxalloc); + /* * We use allocator 0 for this I/O because we don't expect device remap * to be the steady state of the system, so parallelizing is not as @@ -873,6 +962,31 @@ spa_vdev_copy_segment(vdev_t *vd, uint64_t start, uint return (error); /* + * Determine the ranges that are not actually needed. Offsets are + * relative to the start of the range to be copied (i.e. relative to the + * local variable "start"). + */ + range_tree_t *obsolete_segs = range_tree_create(NULL, NULL); + + range_seg_t *rs = avl_first(&segs->rt_root); + ASSERT3U(rs->rs_start, ==, start); + uint64_t prev_seg_end = rs->rs_end; + while ((rs = AVL_NEXT(&segs->rt_root, rs)) != NULL) { + if (rs->rs_start >= start + size) { + break; + } else { + range_tree_add(obsolete_segs, + prev_seg_end - start, + rs->rs_start - prev_seg_end); + } + prev_seg_end = rs->rs_end; + } + /* We don't end in the middle of an obsolete range */ + ASSERT3U(start + size, <=, prev_seg_end); + + range_tree_clear(segs, start, size); + + /* * We can't have any padding of the allocated size, otherwise we will * misunderstand what's allocated, and the size of the mapping. * The caller ensures this will be true by passing in a size that is @@ -883,13 +997,22 @@ spa_vdev_copy_segment(vdev_t *vd, uint64_t start, uint entry = kmem_zalloc(sizeof (vdev_indirect_mapping_entry_t), KM_SLEEP); DVA_MAPPING_SET_SRC_OFFSET(&entry->vime_mapping, start); entry->vime_mapping.vimep_dst = dst; + if (spa_feature_is_enabled(spa, SPA_FEATURE_OBSOLETE_COUNTS)) { + entry->vime_obsolete_count = range_tree_space(obsolete_segs); + } + vdev_copy_segment_arg_t *vcsa = kmem_zalloc(sizeof (*vcsa), KM_SLEEP); + vcsa->vcsa_dest_dva = &entry->vime_mapping.vimep_dst; + vcsa->vcsa_obsolete_segs = obsolete_segs; + vcsa->vcsa_spa = spa; + vcsa->vcsa_txg = txg; + /* * See comment before spa_vdev_copy_one_child(). */ spa_config_enter(spa, SCL_STATE, spa, RW_READER); zio_t *nzio = zio_null(spa->spa_txg_zio[txg & TXG_MASK], spa, NULL, - spa_vdev_copy_nullzio_done, NULL, 0); + spa_vdev_copy_segment_done, vcsa, 0); vdev_t *dest_vd = vdev_lookup_top(spa, DVA_GET_VDEV(&dst)); if (dest_vd->vdev_ops == &vdev_mirror_ops) { for (int i = 0; i < dest_vd->vdev_children; i++) { @@ -1092,39 +1215,78 @@ spa_vdev_copy_impl(vdev_t *vd, spa_vdev_removal_t *svr mutex_enter(&svr->svr_lock); - range_seg_t *rs = avl_first(&svr->svr_allocd_segs->rt_root); - if (rs == NULL) { + /* + * Determine how big of a chunk to copy. We can allocate up + * to max_alloc bytes, and we can span up to vdev_removal_max_span + * bytes of unallocated space at a time. "segs" will track the + * allocated segments that we are copying. We may also be copying + * free segments (of up to vdev_removal_max_span bytes). + */ + range_tree_t *segs = range_tree_create(NULL, NULL); + for (;;) { + range_seg_t *rs = avl_first(&svr->svr_allocd_segs->rt_root); + if (rs == NULL) + break; + + uint64_t seg_length; + + if (range_tree_is_empty(segs)) { + /* need to truncate the first seg based on max_alloc */ + seg_length = + MIN(rs->rs_end - rs->rs_start, *max_alloc); + } else { + if (rs->rs_start - range_tree_max(segs) > + vdev_removal_max_span) { + /* + * Including this segment would cause us to + * copy a larger unneeded chunk than is allowed. + */ + break; + } else if (rs->rs_end - range_tree_min(segs) > + *max_alloc) { + /* + * This additional segment would extend past + * max_alloc. Rather than splitting this + * segment, leave it for the next mapping. + */ + break; + } else { + seg_length = rs->rs_end - rs->rs_start; + } + } + + range_tree_add(segs, rs->rs_start, seg_length); + range_tree_remove(svr->svr_allocd_segs, + rs->rs_start, seg_length); + } + + if (range_tree_is_empty(segs)) { mutex_exit(&svr->svr_lock); + range_tree_destroy(segs); return; } - uint64_t offset = rs->rs_start; - uint64_t length = MIN(rs->rs_end - rs->rs_start, *max_alloc); - range_tree_remove(svr->svr_allocd_segs, offset, length); - if (svr->svr_max_offset_to_sync[txg & TXG_MASK] == 0) { dsl_sync_task_nowait(dmu_tx_pool(tx), vdev_mapping_sync, svr, 0, ZFS_SPACE_CHECK_NONE, tx); } - svr->svr_max_offset_to_sync[txg & TXG_MASK] = offset + length; + svr->svr_max_offset_to_sync[txg & TXG_MASK] = range_tree_max(segs); /* * Note: this is the amount of *allocated* space * that we are taking care of each txg. */ - svr->svr_bytes_done[txg & TXG_MASK] += length; + svr->svr_bytes_done[txg & TXG_MASK] += range_tree_space(segs); mutex_exit(&svr->svr_lock); zio_alloc_list_t zal; metaslab_trace_init(&zal); - uint64_t thismax = *max_alloc; - while (length > 0) { - uint64_t mylen = MIN(length, thismax); - + uint64_t thismax = SPA_MAXBLOCKSIZE; + while (!range_tree_is_empty(segs)) { int error = spa_vdev_copy_segment(vd, - offset, mylen, txg, vca, &zal); + segs, thismax, txg, vca, &zal); if (error == ENOSPC) { /* @@ -1138,18 +1300,17 @@ spa_vdev_copy_impl(vdev_t *vd, spa_vdev_removal_t *svr */ ASSERT3U(spa->spa_max_ashift, >=, SPA_MINBLOCKSHIFT); ASSERT3U(spa->spa_max_ashift, ==, spa->spa_min_ashift); - thismax = P2ROUNDUP(mylen / 2, + uint64_t attempted = + MIN(range_tree_span(segs), thismax); + thismax = P2ROUNDUP(attempted / 2, 1 << spa->spa_max_ashift); - ASSERT3U(thismax, <, mylen); /* * The minimum-size allocation can not fail. */ - ASSERT3U(mylen, >, 1 << spa->spa_max_ashift); - *max_alloc = mylen - (1 << spa->spa_max_ashift); + ASSERT3U(attempted, >, 1 << spa->spa_max_ashift); + *max_alloc = attempted - (1 << spa->spa_max_ashift); } else { ASSERT0(error); - length -= mylen; - offset += mylen; /* * We've performed an allocation, so reset the @@ -1160,6 +1321,7 @@ spa_vdev_copy_impl(vdev_t *vd, spa_vdev_removal_t *svr } } metaslab_trace_fini(&zal); + range_tree_destroy(segs); } /* From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:13:41 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5282E1053835; Thu, 2 Aug 2018 23:13:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC854814E2; Thu, 2 Aug 2018 23:13:40 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CCA3118604; Thu, 2 Aug 2018 23:13:40 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NDeC5006035; Thu, 2 Aug 2018 23:13:40 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NDeui006034; Thu, 2 Aug 2018 23:13:40 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022313.w72NDeui006034@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:13:40 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337193 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337193 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:13:41 -0000 Author: mav Date: Thu Aug 2 23:13:40 2018 New Revision: 337193 URL: https://svnweb.freebsd.org/changeset/base/337193 Log: 9424 ztest failure: "unprotected error in call to Lua API (Invalid value type 'function' for key 'error')" illumos/illumos-gate@fe3ba4d1227d8746116ece7240682b13595c3142 Reviewed by: Sebastien Roy Reviewed by: Paul Dagnelie Reviewed by: Don Brady Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zcp.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zcp.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zcp.c Thu Aug 2 22:52:05 2018 (r337192) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zcp.c Thu Aug 2 23:13:40 2018 (r337193) @@ -426,7 +426,7 @@ zcp_lua_to_nvlist_impl(lua_State *state, int index, nv /* * Convert a lua value to an nvpair, adding it to an nvlist with the given key. */ -void +static void zcp_lua_to_nvlist(lua_State *state, int index, nvlist_t *nvl, const char *key) { /* @@ -438,7 +438,7 @@ zcp_lua_to_nvlist(lua_State *state, int index, nvlist_ (void) lua_error(state); } -int +static int zcp_lua_to_nvlist_helper(lua_State *state) { nvlist_t *nv = (nvlist_t *)lua_touserdata(state, 2); @@ -447,11 +447,12 @@ zcp_lua_to_nvlist_helper(lua_State *state) return (0); } -void +static void zcp_convert_return_values(lua_State *state, nvlist_t *nvl, const char *key, zcp_eval_arg_t *evalargs) { int err; + VERIFY3U(1, ==, lua_gettop(state)); lua_pushcfunction(state, zcp_lua_to_nvlist_helper); lua_pushlightuserdata(state, (char *)key); lua_pushlightuserdata(state, nvl); @@ -897,6 +898,7 @@ zcp_eval_impl(dmu_tx_t *tx, boolean_t sync, zcp_eval_a ZCP_RET_RETURN, evalargs); } else if (return_count > 1) { evalargs->ea_result = SET_ERROR(ECHRNG); + lua_settop(state, 0); (void) lua_pushfstring(state, "Multiple return " "values not supported"); zcp_convert_return_values(state, evalargs->ea_outnvl, @@ -958,6 +960,7 @@ static void zcp_pool_error(zcp_eval_arg_t *evalargs, const char *poolname) { evalargs->ea_result = SET_ERROR(ECHRNG); + lua_settop(evalargs->ea_state, 0); (void) lua_pushfstring(evalargs->ea_state, "Could not open pool: %s", poolname); zcp_convert_return_values(evalargs->ea_state, evalargs->ea_outnvl, From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:17:12 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F7D11053A41; Thu, 2 Aug 2018 23:17:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8656F8187F; Thu, 2 Aug 2018 23:17:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 642F618608; Thu, 2 Aug 2018 23:17:11 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NHB11006288; Thu, 2 Aug 2018 23:17:11 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NHBv3006287; Thu, 2 Aug 2018 23:17:11 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022317.w72NHBv3006287@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:17:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337195 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337195 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:17:12 -0000 Author: mav Date: Thu Aug 2 23:17:10 2018 New Revision: 337195 URL: https://svnweb.freebsd.org/changeset/base/337195 Log: 9454 ::zfs_blkstats should count embedded blocks illumos/illumos-gate@dec267e7ea9828898b1c64462daa6636c4ef5e29 Reviewed by: Dan Kimmel Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Aug 2 23:15:10 2018 (r337194) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Thu Aug 2 23:17:10 2018 (r337195) @@ -1951,14 +1951,14 @@ dsl_scan_scrub_cb(dsl_pool_t *dp, int zio_flags = ZIO_FLAG_SCAN_THREAD | ZIO_FLAG_RAW | ZIO_FLAG_CANFAIL; int scan_delay = 0; + count_block(dp->dp_blkstats, bp); + if (phys_birth <= scn->scn_phys.scn_min_txg || phys_birth >= scn->scn_phys.scn_max_txg) return (0); - count_block(dp->dp_blkstats, bp); - - if (BP_IS_EMBEDDED(bp)) - return (0); + /* Embedded BP's have phys_birth==0, so we reject them above. */ + ASSERT(!BP_IS_EMBEDDED(bp)); ASSERT(DSL_SCAN_IS_SCRUB_RESILVER(scn)); if (scn->scn_phys.scn_func == POOL_SCAN_SCRUB) { From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:24:42 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E379E1053E68; Thu, 2 Aug 2018 23:24:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 894F381EB9; Thu, 2 Aug 2018 23:24:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 679961879A; Thu, 2 Aug 2018 23:24:41 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NOfUD011324; Thu, 2 Aug 2018 23:24:41 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NOfPG011323; Thu, 2 Aug 2018 23:24:41 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022324.w72NOfPG011323@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:24:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337197 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337197 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:24:42 -0000 Author: mav Date: Thu Aug 2 23:24:40 2018 New Revision: 337197 URL: https://svnweb.freebsd.org/changeset/base/337197 Log: 9456 ztest failure in zil_commit_waiter_timeout illumos/illumos-gate@b6031810da58df96413bf76e068638fcab1f228a Reviewed by: Matt Ahrens Reviewed by: Serapheim Dimitropoulos Approved by: Matt Ahrens Author: Prakash Surya Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Thu Aug 2 23:23:10 2018 (r337196) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Thu Aug 2 23:24:40 2018 (r337197) @@ -2290,7 +2290,7 @@ zil_commit_waiter_timeout(zilog_t *zilog, zil_commit_w */ lwb_t *nlwb = zil_lwb_write_issue(zilog, lwb); - ASSERT3S(lwb->lwb_state, !=, LWB_STATE_OPENED); + IMPLY(nlwb != NULL, lwb->lwb_state != LWB_STATE_OPENED); /* * Since the lwb's zio hadn't been issued by the time this thread From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:28:48 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA9621054013; Thu, 2 Aug 2018 23:28:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 761118218F; Thu, 2 Aug 2018 23:28:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 52EAF1879F; Thu, 2 Aug 2018 23:28:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NSmpr011596; Thu, 2 Aug 2018 23:28:48 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NSmHq011595; Thu, 2 Aug 2018 23:28:48 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022328.w72NSmHq011595@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:28:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337199 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/man/man1m vendor/illumos/dist/man/man5 X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/man/man1m vendor/illumos/dist/man/man5 X-SVN-Commit-Revision: 337199 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:28:49 -0000 Author: mav Date: Thu Aug 2 23:28:47 2018 New Revision: 337199 URL: https://svnweb.freebsd.org/changeset/base/337199 Log: 8906 uts: illumos rootfs should support salted cksum illumos/illumos-gate@0dd498c0020c4a0dd34d350b07f97399a475d102 Reviewed by: C Fraire Reviewed by: Robert Mustacchi Reviewed by: Yuri Pankov Approved by: Dan McDonald Author: Toomas Soome Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/man/man1m/zfs.1m vendor/illumos/dist/man/man5/zpool-features.5 Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Thu Aug 2 23:25:49 2018 (r337198) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Thu Aug 2 23:28:47 2018 (r337199) @@ -4069,16 +4069,7 @@ zfs_check_settable(const char *dsname, nvpair_t *pair, if ((err = spa_open(dsname, &spa, FTAG)) != 0) return (err); - /* - * Salted checksums are not supported on root pools. - */ - if (spa_bootfs(spa) != 0 && - intval < ZIO_CHECKSUM_FUNCTIONS && - (zio_checksum_table[intval].ci_flags & - ZCHECKSUM_FLAG_SALTED)) { - spa_close(spa, FTAG); - return (SET_ERROR(ERANGE)); - } + if (!spa_feature_is_enabled(spa, feature)) { spa_close(spa, FTAG); return (SET_ERROR(ENOTSUP)); From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:28:49 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5050F1054017; Thu, 2 Aug 2018 23:28:49 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB1F382190; Thu, 2 Aug 2018 23:28:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C8483187A0; Thu, 2 Aug 2018 23:28:48 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NSmGr011603; Thu, 2 Aug 2018 23:28:48 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NSmK5011601; Thu, 2 Aug 2018 23:28:48 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022328.w72NSmK5011601@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:28:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337199 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/man/man1m vendor/illumos/dist/man/man5 X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs vendor/illumos/dist/man/man1m vendor/illumos/dist/man/man5 X-SVN-Commit-Revision: 337199 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:28:49 -0000 Author: mav Date: Thu Aug 2 23:28:47 2018 New Revision: 337199 URL: https://svnweb.freebsd.org/changeset/base/337199 Log: 8906 uts: illumos rootfs should support salted cksum illumos/illumos-gate@0dd498c0020c4a0dd34d350b07f97399a475d102 Reviewed by: C Fraire Reviewed by: Robert Mustacchi Reviewed by: Yuri Pankov Approved by: Dan McDonald Author: Toomas Soome Modified: vendor/illumos/dist/man/man1m/zfs.1m vendor/illumos/dist/man/man5/zpool-features.5 Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ioctl.c Modified: vendor/illumos/dist/man/man1m/zfs.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zfs.1m Thu Aug 2 23:25:49 2018 (r337198) +++ vendor/illumos/dist/man/man1m/zfs.1m Thu Aug 2 23:28:47 2018 (r337199) @@ -29,7 +29,7 @@ .\" Copyright 2017 Nexenta Systems, Inc. .\" Copyright 2018 Joyent, Inc. .\" -.Dd December 6, 2017 +.Dd Feb 10, 2018 .Dt ZFS 1M .Os .Sh NAME @@ -1002,10 +1002,6 @@ Please see for more information on these algorithms. .Pp Changing this property affects only newly-written data. -.Pp -Salted checksum algorithms -.Pq Cm edonr , skein -are currently not supported for any filesystem on the boot pools. .It Xo .Sy compression Ns = Ns Sy on Ns | Ns Sy off Ns | Ns Sy gzip Ns | Ns .Sy gzip- Ns Em N Ns | Ns Sy lz4 Ns | Ns Sy lzjb Ns | Ns Sy zle Modified: vendor/illumos/dist/man/man5/zpool-features.5 ============================================================================== --- vendor/illumos/dist/man/man5/zpool-features.5 Thu Aug 2 23:25:49 2018 (r337198) +++ vendor/illumos/dist/man/man5/zpool-features.5 Thu Aug 2 23:28:47 2018 (r337199) @@ -15,7 +15,7 @@ .\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your .\" own identifying information: .\" Portions Copyright [yyyy] [name of copyright owner] -.TH ZPOOL-FEATURES 5 "Aug 27, 2013" +.TH ZPOOL-FEATURES 5 "Jun 8, 2018" .SH NAME zpool\-features \- ZFS pool feature descriptions .SH DESCRIPTION @@ -561,8 +561,7 @@ can turn on the \fBsha512\fR checksum on any dataset u and will return to being \fBenabled\fR once all filesystems that have ever had their checksum set to \fBsha512\fR are destroyed. -Booting off of pools utilizing SHA-512/256 is supported (provided that -the updated GRUB stage2 module is installed). +Booting off of pools utilizing SHA-512/256 is supported. .RE @@ -596,9 +595,7 @@ can turn on the \fBskein\fR checksum on any dataset us and will return to being \fBenabled\fR once all filesystems that have ever had their checksum set to \fBskein\fR are destroyed. -Booting off of pools using \fBskein\fR is \fBNOT\fR supported --- any attempt to enable \fBskein\fR on a root pool will fail with an -error. +Booting off of pools using \fBskein\fR is supported. .RE @@ -638,9 +635,7 @@ can turn on the \fBedonr\fR checksum on any dataset us and will return to being \fBenabled\fR once all filesystems that have ever had their checksum set to \fBedonr\fR are destroyed. -Booting off of pools using \fBedonr\fR is \fBNOT\fR supported --- any attempt to enable \fBedonr\fR on a root pool will fail with an -error. +Booting off of pools using \fBedonr\fR is supported. .SH "SEE ALSO" \fBzpool\fR(1M) From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:37:28 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D915A105437C; Thu, 2 Aug 2018 23:37:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8389682759; Thu, 2 Aug 2018 23:37:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 628E418936; Thu, 2 Aug 2018 23:37:27 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NbRfs016776; Thu, 2 Aug 2018 23:37:27 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NbQXB016770; Thu, 2 Aug 2018 23:37:26 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022337.w72NbQXB016770@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:37:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337200 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337200 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:37:28 -0000 Author: mav Date: Thu Aug 2 23:37:26 2018 New Revision: 337200 URL: https://svnweb.freebsd.org/changeset/base/337200 Log: 9438 Holes can lose birth time info if a block has a mix of birth times Ultimately, the problem here is that when you truncate and write a file in the same transaction group, the dbuf for the indirect block will be zeroed out to deal with the truncation, and then written for the write. During this process, we will lose hole birth time information for any holes in the range. In the case where a dnode is being freed, we need to determine whether the block should be converted to a higher-level hole in the zio pipeline, and if so do it when the dnode is being synced out. illumos/illumos-gate@738e2a3ce3b2579222d6855e7fe75b5bcfcddf8d Reviewed by: Matt Ahrens Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Paul Dagnelie Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Thu Aug 2 23:28:47 2018 (r337199) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_object.c Thu Aug 2 23:37:26 2018 (r337200) @@ -167,6 +167,10 @@ dmu_object_free(objset_t *os, uint64_t object, dmu_tx_ return (err); ASSERT(dn->dn_type != DMU_OT_NONE); + /* + * If we don't create this free range, we'll leak indirect blocks when + * we get to freeing the dnode in syncing context. + */ dnode_free_range(dn, 0, DMU_OBJECT_END, tx); dnode_free(dn, tx); dnode_rele(dn, FTAG); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:28:47 2018 (r337199) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:37:26 2018 (r337200) @@ -1507,6 +1507,72 @@ dnode_dirty_l1(dnode_t *dn, uint64_t l1blkid, dmu_tx_t } } +/* + * Dirty all the in-core level-1 dbufs in the range specified by start_blkid + * and end_blkid. + */ +static void +dnode_dirty_l1range(dnode_t *dn, uint64_t start_blkid, uint64_t end_blkid, + dmu_tx_t *tx) +{ + dmu_buf_impl_t db_search; + dmu_buf_impl_t *db; + avl_index_t where; + + mutex_enter(&dn->dn_dbufs_mtx); + + db_search.db_level = 1; + db_search.db_blkid = start_blkid + 1; + db_search.db_state = DB_SEARCH; + for (;;) { + + db = avl_find(&dn->dn_dbufs, &db_search, &where); + if (db == NULL) + db = avl_nearest(&dn->dn_dbufs, where, AVL_AFTER); + + if (db == NULL || db->db_level != 1 || + db->db_blkid >= end_blkid) { + break; + } + + /* + * Setup the next blkid we want to search for. + */ + db_search.db_blkid = db->db_blkid + 1; + ASSERT3U(db->db_blkid, >=, start_blkid); + + /* + * If the dbuf transitions to DB_EVICTING while we're trying + * to dirty it, then we will be unable to discover it in + * the dbuf hash table. This will result in a call to + * dbuf_create() which needs to acquire the dn_dbufs_mtx + * lock. To avoid a deadlock, we drop the lock before + * dirtying the level-1 dbuf. + */ + mutex_exit(&dn->dn_dbufs_mtx); + dnode_dirty_l1(dn, db->db_blkid, tx); + mutex_enter(&dn->dn_dbufs_mtx); + } + +#ifdef ZFS_DEBUG + /* + * Walk all the in-core level-1 dbufs and verify they have been dirtied. + */ + db_search.db_level = 1; + db_search.db_blkid = start_blkid + 1; + db_search.db_state = DB_SEARCH; + db = avl_find(&dn->dn_dbufs, &db_search, &where); + if (db == NULL) + db = avl_nearest(&dn->dn_dbufs, where, AVL_AFTER); + for (; db != NULL; db = AVL_NEXT(&dn->dn_dbufs, db)) { + if (db->db_level != 1 || db->db_blkid >= end_blkid) + break; + ASSERT(db->db_dirtycnt > 0); + } +#endif + mutex_exit(&dn->dn_dbufs_mtx); +} + void dnode_free_range(dnode_t *dn, uint64_t off, uint64_t len, dmu_tx_t *tx) { @@ -1657,6 +1723,8 @@ dnode_free_range(dnode_t *dn, uint64_t off, uint64_t l last = (blkid + nblks - 1) >> epbs; if (last != first) dnode_dirty_l1(dn, last, tx); + + dnode_dirty_l1range(dn, first, last, tx); int shift = dn->dn_datablkshift + dn->dn_indblkshift - SPA_BLKPTRSHIFT; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c Thu Aug 2 23:28:47 2018 (r337199) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c Thu Aug 2 23:37:26 2018 (r337200) @@ -229,9 +229,24 @@ free_verify(dmu_buf_impl_t *db, uint64_t start, uint64 } #endif +/* + * We don't usually free the indirect blocks here. If in one txg we have a + * free_range and a write to the same indirect block, it's important that we + * preserve the hole's birth times. Therefore, we don't free any any indirect + * blocks in free_children(). If an indirect block happens to turn into all + * holes, it will be freed by dbuf_write_children_ready, which happens at a + * point in the syncing process where we know for certain the contents of the + * indirect block. + * + * However, if we're freeing a dnode, its space accounting must go to zero + * before we actually try to free the dnode, or we will trip an assertion. In + * addition, we know the case described above cannot occur, because the dnode is + * being freed. Therefore, we free the indirect blocks immediately in that + * case. + */ static void free_children(dmu_buf_impl_t *db, uint64_t blkid, uint64_t nblks, - dmu_tx_t *tx) + boolean_t free_indirects, dmu_tx_t *tx) { dnode_t *dn; blkptr_t *bp; @@ -283,32 +298,16 @@ free_children(dmu_buf_impl_t *db, uint64_t blkid, uint rw_exit(&dn->dn_struct_rwlock); ASSERT3P(bp, ==, subdb->db_blkptr); - free_children(subdb, blkid, nblks, tx); + free_children(subdb, blkid, nblks, free_indirects, tx); dbuf_rele(subdb, FTAG); } } - /* If this whole block is free, free ourself too. */ - for (i = 0, bp = db->db.db_data; i < 1 << epbs; i++, bp++) { - if (!BP_IS_HOLE(bp)) - break; - } - if (i == 1 << epbs) { - /* - * We only found holes. Grab the rwlock to prevent - * anybody from reading the blocks we're about to - * zero out. - */ - rw_enter(&dn->dn_struct_rwlock, RW_WRITER); + if (free_indirects) { + for (i = 0, bp = db->db.db_data; i < 1 << epbs; i++, bp++) + ASSERT(BP_IS_HOLE(bp)); bzero(db->db.db_data, db->db.db_size); - rw_exit(&dn->dn_struct_rwlock); free_blocks(dn, db->db_blkptr, 1, tx); - } else { - /* - * Partial block free; must be marked dirty so that it - * will be written out. - */ - ASSERT(db->db_dirtycnt > 0); } DB_DNODE_EXIT(db); @@ -321,7 +320,7 @@ free_children(dmu_buf_impl_t *db, uint64_t blkid, uint */ static void dnode_sync_free_range_impl(dnode_t *dn, uint64_t blkid, uint64_t nblks, - dmu_tx_t *tx) + boolean_t free_indirects, dmu_tx_t *tx) { blkptr_t *bp = dn->dn_phys->dn_blkptr; int dnlevel = dn->dn_phys->dn_nlevels; @@ -361,7 +360,7 @@ dnode_sync_free_range_impl(dnode_t *dn, uint64_t blkid TRUE, FALSE, FTAG, &db)); rw_exit(&dn->dn_struct_rwlock); - free_children(db, blkid, nblks, tx); + free_children(db, blkid, nblks, free_indirects, tx); dbuf_rele(db, FTAG); } } @@ -380,6 +379,7 @@ dnode_sync_free_range_impl(dnode_t *dn, uint64_t blkid typedef struct dnode_sync_free_range_arg { dnode_t *dsfra_dnode; dmu_tx_t *dsfra_tx; + boolean_t dsfra_free_indirects; } dnode_sync_free_range_arg_t; static void @@ -389,7 +389,8 @@ dnode_sync_free_range(void *arg, uint64_t blkid, uint6 dnode_t *dn = dsfra->dsfra_dnode; mutex_exit(&dn->dn_mtx); - dnode_sync_free_range_impl(dn, blkid, nblks, dsfra->dsfra_tx); + dnode_sync_free_range_impl(dn, blkid, nblks, + dsfra->dsfra_free_indirects, dsfra->dsfra_tx); mutex_enter(&dn->dn_mtx); } @@ -670,6 +671,11 @@ dnode_sync(dnode_t *dn, dmu_tx_t *tx) dnode_sync_free_range_arg_t dsfra; dsfra.dsfra_dnode = dn; dsfra.dsfra_tx = tx; + dsfra.dsfra_free_indirects = freeing_dnode; + if (freeing_dnode) { + ASSERT(range_tree_contains(dn->dn_free_ranges[txgoff], + 0, dn->dn_maxblkid + 1)); + } mutex_enter(&dn->dn_mtx); range_tree_vacate(dn->dn_free_ranges[txgoff], dnode_sync_free_range, &dsfra); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Thu Aug 2 23:28:47 2018 (r337199) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Thu Aug 2 23:37:26 2018 (r337200) @@ -1603,7 +1603,8 @@ zfs_trunc(znode_t *zp, uint64_t end) return (0); } - error = dmu_free_long_range(zfsvfs->z_os, zp->z_id, end, -1); + error = dmu_free_long_range(zfsvfs->z_os, zp->z_id, end, + DMU_OBJECT_END); if (error) { zfs_range_unlock(rl); return (error); From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:45:26 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D80911054CE1; Thu, 2 Aug 2018 23:45:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8F4CA840FB; Thu, 2 Aug 2018 23:45:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5987F18B08; Thu, 2 Aug 2018 23:45:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NjPa6024275; Thu, 2 Aug 2018 23:45:25 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NjP41024273; Thu, 2 Aug 2018 23:45:25 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022345.w72NjP41024273@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:45:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337204 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337204 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:45:26 -0000 Author: mav Date: Thu Aug 2 23:45:24 2018 New Revision: 337204 URL: https://svnweb.freebsd.org/changeset/base/337204 Log: 9439 ZFS double-free due to failure to dirty indirect block illumos/illumos-gate@99a19144e82244f3426f055cc73af8a937c0135c Reviewed by: George Wilson Reviewed by: Paul Dagnelie Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:45:14 2018 (r337203) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:45:24 2018 (r337204) @@ -1605,13 +1605,11 @@ dnode_free_range(dnode_t *dn, uint64_t off, uint64_t l if (off == 0 && len >= blksz) { /* * Freeing the whole block; fast-track this request. - * Note that we won't dirty any indirect blocks, - * which is fine because we will be freeing the entire - * file and thus all indirect blocks will be freed - * by free_children(). */ blkid = 0; nblks = 1; + if (dn->dn_nlevels > 1) + dnode_dirty_l1(dn, 0, tx); goto done; } else if (off >= blksz) { /* Freeing past end-of-data */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c Thu Aug 2 23:45:14 2018 (r337203) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c Thu Aug 2 23:45:24 2018 (r337204) @@ -263,6 +263,24 @@ free_children(dmu_buf_impl_t *db, uint64_t blkid, uint if (db->db_state != DB_CACHED) (void) dbuf_read(db, NULL, DB_RF_MUST_SUCCEED); + /* + * If we modify this indirect block, and we are not freeing the + * dnode (!free_indirects), then this indirect block needs to get + * written to disk by dbuf_write(). If it is dirty, we know it will + * be written (otherwise, we would have incorrect on-disk state + * because the space would be freed but still referenced by the BP + * in this indirect block). Therefore we VERIFY that it is + * dirty. + * + * Our VERIFY covers some cases that do not actually have to be + * dirty, but the open-context code happens to dirty. E.g. if the + * blocks we are freeing are all holes, because in that case, we + * are only freeing part of this indirect block, so it is an + * ancestor of the first or last block to be freed. The first and + * last L1 indirect blocks are always dirtied by dnode_free_range(). + */ + VERIFY(BP_GET_FILL(db->db_blkptr) == 0 || db->db_dirtycnt > 0); + dbuf_release_bp(db); bp = db->db.db_data; From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:48:20 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16C8D1055000; Thu, 2 Aug 2018 23:48:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BFE80845F4; Thu, 2 Aug 2018 23:48:19 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9D39418B0B; Thu, 2 Aug 2018 23:48:19 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NmJ6b024487; Thu, 2 Aug 2018 23:48:19 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NmJDQ024486; Thu, 2 Aug 2018 23:48:19 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022348.w72NmJDQ024486@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:48:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337206 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337206 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:48:20 -0000 Author: mav Date: Thu Aug 2 23:48:19 2018 New Revision: 337206 URL: https://svnweb.freebsd.org/changeset/base/337206 Log: 9338 moved dnode has incorrect dn_next_type illumos/illumos-gate@c7fbe46df966ea665df63b6e6071808987e839d1 Reviewed by: Prashanth Sreenivasa Reviewed by: Serapheim Dimitropoulos Reviewed by: Dan Kimmel Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:46:30 2018 (r337205) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:48:19 2018 (r337206) @@ -733,6 +733,8 @@ dnode_move_impl(dnode_t *odn, dnode_t *ndn) ndn->dn_datablkszsec = odn->dn_datablkszsec; ndn->dn_datablksz = odn->dn_datablksz; ndn->dn_maxblkid = odn->dn_maxblkid; + bcopy(&odn->dn_next_type[0], &ndn->dn_next_type[0], + sizeof (odn->dn_next_type)); bcopy(&odn->dn_next_nblkptr[0], &ndn->dn_next_nblkptr[0], sizeof (odn->dn_next_nblkptr)); bcopy(&odn->dn_next_nlevels[0], &ndn->dn_next_nlevels[0], From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:52:30 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A56F3105547C; Thu, 2 Aug 2018 23:52:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B0A984D5E; Thu, 2 Aug 2018 23:52:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2219218CB0; Thu, 2 Aug 2018 23:52:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NqTLJ029166; Thu, 2 Aug 2018 23:52:29 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NqTvj029147; Thu, 2 Aug 2018 23:52:29 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022352.w72NqTvj029147@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:52:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337208 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Commit-Revision: 337208 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:52:31 -0000 Author: mav Date: Thu Aug 2 23:52:29 2018 New Revision: 337208 URL: https://svnweb.freebsd.org/changeset/base/337208 Log: 9591 ms_shift can be incorrectly changed in MOS config for indirect vdevs that have been historically expanded illumos/illumos-gate@11f6a9680e013a7c9c57dc0b64d3e91e2eee1a6b Reviewed by: Matthew Ahrens Reviewed by: George Wilson Reviewed by: John Kennedy Reviewed by: Prashanth Sreenivasa Reviewed by: Tim Chase Approved by: Richard Lowe Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c Thu Aug 2 23:50:03 2018 (r337207) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_config.c Thu Aug 2 23:52:29 2018 (r337208) @@ -22,7 +22,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2011, 2015 by Delphix. All rights reserved. + * Copyright (c) 2011, 2018 by Delphix. All rights reserved. * Copyright 2017 Joyent, Inc. */ @@ -510,6 +510,18 @@ spa_config_update(spa_t *spa, int what) */ for (c = 0; c < rvd->vdev_children; c++) { vdev_t *tvd = rvd->vdev_child[c]; + + /* + * Explicitly skip vdevs that are indirect or + * log vdevs that are being removed. The reason + * is that both of those can have vdev_ms_array + * set to 0 and we wouldn't want to change their + * metaslab size nor call vdev_expand() on them. + */ + if (!vdev_is_concrete(tvd) || + (tvd->vdev_islog && tvd->vdev_removing)) + continue; + if (tvd->vdev_ms_array == 0) vdev_metaslab_set_size(tvd); vdev_expand(tvd, txg); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Aug 2 23:50:03 2018 (r337207) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Thu Aug 2 23:52:29 2018 (r337208) @@ -4086,11 +4086,11 @@ vdev_expand(vdev_t *vd, uint64_t txg) { ASSERT(vd->vdev_top == vd); ASSERT(spa_config_held(vd->vdev_spa, SCL_ALL, RW_WRITER) == SCL_ALL); + ASSERT(vdev_is_concrete(vd)); vdev_set_deflate_ratio(vd); - if ((vd->vdev_asize >> vd->vdev_ms_shift) > vd->vdev_ms_count && - vdev_is_concrete(vd)) { + if ((vd->vdev_asize >> vd->vdev_ms_shift) > vd->vdev_ms_count) { VERIFY(vdev_metaslab_init(vd, txg) == 0); vdev_config_dirty(vd); } From owner-svn-src-vendor@freebsd.org Thu Aug 2 23:59:54 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 16B9D1055A55; Thu, 2 Aug 2018 23:59:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE6BB85365; Thu, 2 Aug 2018 23:59:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9F50B18CBD; Thu, 2 Aug 2018 23:59:53 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w72NxrvX029688; Thu, 2 Aug 2018 23:59:53 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w72NxqhY029681; Thu, 2 Aug 2018 23:59:52 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808022359.w72NxqhY029681@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Thu, 2 Aug 2018 23:59:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337210 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337210 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2018 23:59:54 -0000 Author: mav Date: Thu Aug 2 23:59:52 2018 New Revision: 337210 URL: https://svnweb.freebsd.org/changeset/base/337210 Log: 9577 remove zfs_dbuf_evict_key tsd The zfs_dbuf_evict_key TSD (thread-specific data) is not necessary - we can instead pass a flag down in a few places to prevent recursive dbuf eviction. Making this change has 3 benefits: 1. The code semantics are easier to understand. 2. On Linux, performance is improved, because creating/removing TSD values (by setting to NULL vs non-NULL) is expensive, and we do it very often. 3. According to Nexenta, the current semantics can cause a deadlock when concurrently calling dmu_objset_evict_dbufs() (which is rare today, but they are working on a "parallel unmount" change that triggers this more easily) illumos/illumos-gate@c2919acbea007fa95c709b60d073db9a24526e01 Reviewed by: George Wilson Reviewed by: Serapheim Dimitropoulos Reviewed by: Brian Behlendorf Reviewed by: Andy Stormont Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Thu Aug 2 23:56:07 2018 (r337209) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Thu Aug 2 23:59:52 2018 (r337210) @@ -51,8 +51,6 @@ #include #include -uint_t zfs_dbuf_evict_key; - static boolean_t dbuf_undirty(dmu_buf_impl_t *db, dmu_tx_t *tx); static void dbuf_write(dbuf_dirty_record_t *dr, arc_buf_t *data, dmu_tx_t *tx); @@ -524,14 +522,6 @@ dbuf_evict_one(void) ASSERT(!MUTEX_HELD(&dbuf_evict_lock)); - /* - * Set the thread's tsd to indicate that it's processing evictions. - * Once a thread stops evicting from the dbuf cache it will - * reset its tsd to NULL. - */ - ASSERT3P(tsd_get(zfs_dbuf_evict_key), ==, NULL); - (void) tsd_set(zfs_dbuf_evict_key, (void *)B_TRUE); - dmu_buf_impl_t *db = multilist_sublist_tail(mls); while (db != NULL && mutex_tryenter(&db->db_mtx) == 0) { db = multilist_sublist_prev(mls, db); @@ -551,7 +541,6 @@ dbuf_evict_one(void) } else { multilist_sublist_unlock(mls); } - (void) tsd_set(zfs_dbuf_evict_key, NULL); } /* @@ -605,30 +594,7 @@ dbuf_evict_thread(void *unused) static void dbuf_evict_notify(void) { - /* - * We use thread specific data to track when a thread has - * started processing evictions. This allows us to avoid deeply - * nested stacks that would have a call flow similar to this: - * - * dbuf_rele()-->dbuf_rele_and_unlock()-->dbuf_evict_notify() - * ^ | - * | | - * +-----dbuf_destroy()<--dbuf_evict_one()<--------+ - * - * The dbuf_eviction_thread will always have its tsd set until - * that thread exits. All other threads will only set their tsd - * if they are participating in the eviction process. This only - * happens if the eviction thread is unable to process evictions - * fast enough. To keep the dbuf cache size in check, other threads - * can evict from the dbuf cache directly. Those threads will set - * their tsd values so that we ensure that they only evict one dbuf - * from the dbuf cache. - */ - if (tsd_get(zfs_dbuf_evict_key) != NULL) - return; - - /* * We check if we should evict without holding the dbuf_evict_lock, * because it's OK to occasionally make the wrong decision here, * and grabbing the lock results in massive lock contention. @@ -704,7 +670,6 @@ retry: refcount_create(&dbuf_caches[dcs].size); } - tsd_create(&zfs_dbuf_evict_key, NULL); dbuf_evict_thread_exit = B_FALSE; mutex_init(&dbuf_evict_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&dbuf_evict_cv, NULL, CV_DEFAULT, NULL); @@ -731,7 +696,6 @@ dbuf_fini(void) cv_wait(&dbuf_evict_cv, &dbuf_evict_lock); } mutex_exit(&dbuf_evict_lock); - tsd_destroy(&zfs_dbuf_evict_key); mutex_destroy(&dbuf_evict_lock); cv_destroy(&dbuf_evict_cv); @@ -1011,7 +975,7 @@ dbuf_read_done(zio_t *zio, arc_buf_t *buf, void *vdb) db->db_state = DB_CACHED; } cv_broadcast(&db->db_changed); - dbuf_rele_and_unlock(db, NULL); + dbuf_rele_and_unlock(db, NULL, B_FALSE); } static void @@ -2171,7 +2135,8 @@ dbuf_destroy(dmu_buf_impl_t *db) * value in dnode_move(), since DB_DNODE_EXIT doesn't actually * release any lock. */ - dnode_rele(dn, db); + mutex_enter(&dn->dn_mtx); + dnode_rele_and_unlock(dn, db, B_TRUE); db->db_dnode_handle = NULL; dbuf_hash_remove(db); @@ -2198,8 +2163,10 @@ dbuf_destroy(dmu_buf_impl_t *db) * If this dbuf is referenced from an indirect dbuf, * decrement the ref count on the indirect dbuf. */ - if (parent && parent != dndb) - dbuf_rele(parent, db); + if (parent && parent != dndb) { + mutex_enter(&parent->db_mtx); + dbuf_rele_and_unlock(parent, db, B_TRUE); + } } /* @@ -2823,7 +2790,7 @@ void dbuf_rele(dmu_buf_impl_t *db, void *tag) { mutex_enter(&db->db_mtx); - dbuf_rele_and_unlock(db, tag); + dbuf_rele_and_unlock(db, tag, B_FALSE); } void @@ -2834,10 +2801,19 @@ dmu_buf_rele(dmu_buf_t *db, void *tag) /* * dbuf_rele() for an already-locked dbuf. This is necessary to allow - * db_dirtycnt and db_holds to be updated atomically. + * db_dirtycnt and db_holds to be updated atomically. The 'evicting' + * argument should be set if we are already in the dbuf-evicting code + * path, in which case we don't want to recursively evict. This allows us to + * avoid deeply nested stacks that would have a call flow similar to this: + * + * dbuf_rele()-->dbuf_rele_and_unlock()-->dbuf_evict_notify() + * ^ | + * | | + * +-----dbuf_destroy()<--dbuf_evict_one()<--------+ + * */ void -dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag) +dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag, boolean_t evicting) { int64_t holds; @@ -2940,7 +2916,8 @@ dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag) db->db.db_size, db); mutex_exit(&db->db_mtx); - if (db->db_caching_status == DB_DBUF_CACHE) { + if (db->db_caching_status == DB_DBUF_CACHE && + !evicting) { dbuf_evict_notify(); } } @@ -3203,7 +3180,7 @@ dbuf_sync_leaf(dbuf_dirty_record_t *dr, dmu_tx_t *tx) kmem_free(dr, sizeof (dbuf_dirty_record_t)); ASSERT(db->db_dirtycnt > 0); db->db_dirtycnt -= 1; - dbuf_rele_and_unlock(db, (void *)(uintptr_t)txg); + dbuf_rele_and_unlock(db, (void *)(uintptr_t)txg, B_FALSE); return; } @@ -3553,7 +3530,7 @@ dbuf_write_done(zio_t *zio, arc_buf_t *buf, void *vdb) ASSERT(db->db_dirtycnt > 0); db->db_dirtycnt -= 1; db->db_data_pending = NULL; - dbuf_rele_and_unlock(db, (void *)(uintptr_t)tx->tx_txg); + dbuf_rele_and_unlock(db, (void *)(uintptr_t)tx->tx_txg, B_FALSE); } static void Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:56:07 2018 (r337209) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode.c Thu Aug 2 23:59:52 2018 (r337210) @@ -1229,11 +1229,11 @@ void dnode_rele(dnode_t *dn, void *tag) { mutex_enter(&dn->dn_mtx); - dnode_rele_and_unlock(dn, tag); + dnode_rele_and_unlock(dn, tag, B_FALSE); } void -dnode_rele_and_unlock(dnode_t *dn, void *tag) +dnode_rele_and_unlock(dnode_t *dn, void *tag, boolean_t evicting) { uint64_t refs; /* Get while the hold prevents the dnode from moving. */ @@ -1264,7 +1264,8 @@ dnode_rele_and_unlock(dnode_t *dn, void *tag) * that the handle has zero references, but that will be * asserted anyway when the handle gets destroyed. */ - dbuf_rele(db, dnh); + mutex_enter(&db->db_mtx); + dbuf_rele_and_unlock(db, dnh, evicting); } } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c Thu Aug 2 23:56:07 2018 (r337209) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dnode_sync.c Thu Aug 2 23:59:52 2018 (r337210) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2017 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. */ @@ -439,6 +439,19 @@ dnode_evict_dbufs(dnode_t *dn) avl_insert_here(&dn->dn_dbufs, &db_marker, db, AVL_BEFORE); + /* + * We need to use the "marker" dbuf rather than + * simply getting the next dbuf, because + * dbuf_destroy() may actually remove multiple dbufs. + * It can call itself recursively on the parent dbuf, + * which may also be removed from dn_dbufs. The code + * flow would look like: + * + * dbuf_destroy(): + * dnode_rele_and_unlock(parent_dbuf, evicting=TRUE): + * if (!cacheable || pending_evict) + * dbuf_destroy() + */ dbuf_destroy(db); db_next = AVL_NEXT(&dn->dn_dbufs, &db_marker); @@ -497,7 +510,7 @@ dnode_undirty_dbufs(list_t *list) list_destroy(&dr->dt.di.dr_children); } kmem_free(dr, sizeof (dbuf_dirty_record_t)); - dbuf_rele_and_unlock(db, (void *)(uintptr_t)txg); + dbuf_rele_and_unlock(db, (void *)(uintptr_t)txg, B_FALSE); } } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h Thu Aug 2 23:56:07 2018 (r337209) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dbuf.h Thu Aug 2 23:59:52 2018 (r337210) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. * Copyright (c) 2013 by Saso Kiselkov. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. */ @@ -303,7 +303,7 @@ boolean_t dbuf_try_add_ref(dmu_buf_t *db, objset_t *os uint64_t dbuf_refcount(dmu_buf_impl_t *db); void dbuf_rele(dmu_buf_impl_t *db, void *tag); -void dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag); +void dbuf_rele_and_unlock(dmu_buf_impl_t *db, void *tag, boolean_t evicting); dmu_buf_impl_t *dbuf_find(struct objset *os, uint64_t object, uint8_t level, uint64_t blkid); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h Thu Aug 2 23:56:07 2018 (r337209) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/dnode.h Thu Aug 2 23:59:52 2018 (r337210) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2017 by Delphix. All rights reserved. + * Copyright (c) 2012, 2018 by Delphix. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. */ @@ -291,7 +291,7 @@ int dnode_hold_impl(struct objset *dd, uint64_t object void *ref, dnode_t **dnp); boolean_t dnode_add_ref(dnode_t *dn, void *ref); void dnode_rele(dnode_t *dn, void *ref); -void dnode_rele_and_unlock(dnode_t *dn, void *tag); +void dnode_rele_and_unlock(dnode_t *dn, void *tag, boolean_t evicting); void dnode_setdirty(dnode_t *dn, dmu_tx_t *tx); void dnode_sync(dnode_t *dn, dmu_tx_t *tx); void dnode_allocate(dnode_t *dn, dmu_object_type_t ot, int blocksize, int ibs, From owner-svn-src-vendor@freebsd.org Fri Aug 3 00:11:08 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90354105603E; Fri, 3 Aug 2018 00:11:08 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4503985CD1; Fri, 3 Aug 2018 00:11:08 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2618218E82; Fri, 3 Aug 2018 00:11:08 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w730B7A6036771; Fri, 3 Aug 2018 00:11:07 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w730B6vs036762; Fri, 3 Aug 2018 00:11:06 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030011.w730B6vs036762@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 00:11:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337212 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Commit-Revision: 337212 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 00:11:08 -0000 Author: mav Date: Fri Aug 3 00:11:06 2018 New Revision: 337212 URL: https://svnweb.freebsd.org/changeset/base/337212 Log: 9465 ARC check for 'anon_size > arc_c/2' can stall the system illumos/illumos-gate@abe1fd01ce5a83718c5a840daeab4abdaec1c104 Reviewed by: Sebastien Roy Reviewed by: Matt Ahrens Reviewed by: Prashanth Sreenivasa Approved by: Robert Mustacchi Author: Don Brady Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Fri Aug 3 00:01:48 2018 (r337211) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c Fri Aug 3 00:11:06 2018 (r337212) @@ -379,6 +379,13 @@ int zfs_arc_shrink_shift = 0; int zfs_arc_p_min_shift = 0; int zfs_arc_average_blocksize = 8 * 1024; /* 8KB */ +/* + * ARC dirty data constraints for arc_tempreserve_space() throttle + */ +uint_t zfs_arc_dirty_limit_percent = 50; /* total dirty data limit */ +uint_t zfs_arc_anon_limit_percent = 25; /* anon block dirty limit */ +uint_t zfs_arc_pool_dirty_percent = 20; /* each pool's anon allowance */ + boolean_t zfs_compressed_arc_enabled = B_TRUE; /* @@ -5839,12 +5846,10 @@ arc_write(zio_t *pio, spa_t *spa, uint64_t txg, blkptr } static int -arc_memory_throttle(uint64_t reserve, uint64_t txg) +arc_memory_throttle(spa_t *spa, uint64_t reserve, uint64_t txg) { #ifdef _KERNEL uint64_t available_memory = ptob(freemem); - static uint64_t page_load = 0; - static uint64_t last_txg = 0; #if defined(__i386) available_memory = @@ -5854,9 +5859,9 @@ arc_memory_throttle(uint64_t reserve, uint64_t txg) if (freemem > physmem * arc_lotsfree_percent / 100) return (0); - if (txg > last_txg) { - last_txg = txg; - page_load = 0; + if (txg > spa->spa_lowmem_last_txg) { + spa->spa_lowmem_last_txg = txg; + spa->spa_lowmem_page_load = 0; } /* * If we are in pageout, we know that memory is already tight, @@ -5864,18 +5869,19 @@ arc_memory_throttle(uint64_t reserve, uint64_t txg) * continue to let page writes occur as quickly as possible. */ if (curproc == proc_pageout) { - if (page_load > MAX(ptob(minfree), available_memory) / 4) + if (spa->spa_lowmem_page_load > + MAX(ptob(minfree), available_memory) / 4) return (SET_ERROR(ERESTART)); /* Note: reserve is inflated, so we deflate */ - page_load += reserve / 8; + atomic_add_64(&spa->spa_lowmem_page_load, reserve / 8); return (0); - } else if (page_load > 0 && arc_reclaim_needed()) { + } else if (spa->spa_lowmem_page_load > 0 && arc_reclaim_needed()) { /* memory is low, delay before restarting */ ARCSTAT_INCR(arcstat_memory_throttle_count, 1); return (SET_ERROR(EAGAIN)); } - page_load = 0; -#endif + spa->spa_lowmem_page_load = 0; +#endif /* _KERNEL */ return (0); } @@ -5887,7 +5893,7 @@ arc_tempreserve_clear(uint64_t reserve) } int -arc_tempreserve_space(uint64_t reserve, uint64_t txg) +arc_tempreserve_space(spa_t *spa, uint64_t reserve, uint64_t txg) { int error; uint64_t anon_size; @@ -5914,7 +5920,7 @@ arc_tempreserve_space(uint64_t reserve, uint64_t txg) * in order to compress/encrypt/etc the data. We therefore need to * make sure that there is sufficient available memory for this. */ - error = arc_memory_throttle(reserve, txg); + error = arc_memory_throttle(spa, reserve, txg); if (error != 0) return (error); @@ -5922,12 +5928,24 @@ arc_tempreserve_space(uint64_t reserve, uint64_t txg) * Throttle writes when the amount of dirty data in the cache * gets too large. We try to keep the cache less than half full * of dirty blocks so that our sync times don't grow too large. + * + * In the case of one pool being built on another pool, we want + * to make sure we don't end up throttling the lower (backing) + * pool when the upper pool is the majority contributor to dirty + * data. To insure we make forward progress during throttling, we + * also check the current pool's net dirty data and only throttle + * if it exceeds zfs_arc_pool_dirty_percent of the anonymous dirty + * data in the cache. + * * Note: if two requests come in concurrently, we might let them * both succeed, when one of them should fail. Not a huge deal. */ + uint64_t total_dirty = reserve + arc_tempreserve + anon_size; + uint64_t spa_dirty_anon = spa_dirty_data(spa); - if (reserve + arc_tempreserve + anon_size > arc_c / 2 && - anon_size > arc_c / 4) { + if (total_dirty > arc_c * zfs_arc_dirty_limit_percent / 100 && + anon_size > arc_c * zfs_arc_anon_limit_percent / 100 && + spa_dirty_anon > anon_size * zfs_arc_pool_dirty_percent / 100) { uint64_t meta_esize = refcount_count(&arc_anon->arcs_esize[ARC_BUFC_METADATA]); uint64_t data_esize = Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c Fri Aug 3 00:01:48 2018 (r337211) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_dir.c Fri Aug 3 00:11:06 2018 (r337212) @@ -1379,7 +1379,7 @@ dsl_dir_tempreserve_space(dsl_dir_t *dd, uint64_t lsiz offsetof(struct tempreserve, tr_node)); ASSERT3S(asize, >, 0); - err = arc_tempreserve_space(lsize, tx->tx_txg); + err = arc_tempreserve_space(dd->dd_pool->dp_spa, lsize, tx->tx_txg); if (err == 0) { struct tempreserve *tr; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Fri Aug 3 00:01:48 2018 (r337211) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa_misc.c Fri Aug 3 00:11:06 2018 (r337212) @@ -1910,6 +1910,12 @@ bp_get_dsize(spa_t *spa, const blkptr_t *bp) return (dsize); } +uint64_t +spa_dirty_data(spa_t *spa) +{ + return (spa->spa_dsl_pool->dp_dirty_total); +} + /* * ========================================================================== * Initialization and Termination Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h Fri Aug 3 00:01:48 2018 (r337211) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/arc.h Fri Aug 3 00:11:06 2018 (r337212) @@ -190,7 +190,7 @@ void arc_freed(spa_t *spa, const blkptr_t *bp); void arc_flush(spa_t *spa, boolean_t retry); void arc_tempreserve_clear(uint64_t reserve); -int arc_tempreserve_space(uint64_t reserve, uint64_t txg); +int arc_tempreserve_space(spa_t *spa, uint64_t reserve, uint64_t txg); uint64_t arc_max_bytes(void); void arc_init(void); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Fri Aug 3 00:01:48 2018 (r337211) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h Fri Aug 3 00:11:06 2018 (r337212) @@ -813,6 +813,7 @@ extern uint64_t spa_bootfs(spa_t *spa); extern uint64_t spa_delegation(spa_t *spa); extern objset_t *spa_meta_objset(spa_t *spa); extern uint64_t spa_deadman_synctime(spa_t *spa); +extern uint64_t spa_dirty_data(spa_t *spa); /* Miscellaneous support routines */ extern void spa_load_failed(spa_t *spa, const char *fmt, ...); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Fri Aug 3 00:01:48 2018 (r337211) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Fri Aug 3 00:11:06 2018 (r337212) @@ -373,6 +373,10 @@ struct spa { int spa_queued; } spa_queue_stats[ZIO_PRIORITY_NUM_QUEUEABLE]; + /* arc_memory_throttle() parameters during low memory condition */ + uint64_t spa_lowmem_page_load; /* memory load during txg */ + uint64_t spa_lowmem_last_txg; /* txg window start */ + hrtime_t spa_ccw_fail_time; /* Conf cache write fail time */ /* From owner-svn-src-vendor@freebsd.org Fri Aug 3 00:20:01 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5E111056469; Fri, 3 Aug 2018 00:20:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9848B867C6; Fri, 3 Aug 2018 00:20:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 756FF18FF6; Fri, 3 Aug 2018 00:20:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w730K0eG040492; Fri, 3 Aug 2018 00:20:00 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w730K08w040491; Fri, 3 Aug 2018 00:20:00 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030020.w730K08w040491@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 00:20:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337214 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Commit-Revision: 337214 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 00:20:01 -0000 Author: mav Date: Fri Aug 3 00:19:59 2018 New Revision: 337214 URL: https://svnweb.freebsd.org/changeset/base/337214 Log: 9621 Make createtxg and guid properties public illumos/illumos-gate@e8d4a73c868afb740396041be80ed2b141065e76 Reviewed by: Andy Stormont Reviewed by: Paul Dagnelie Reviewed by: Matt Ahrens Reviewed by: Yuri Pankov Approved by: Robert Mustacchi Author: Josh Paetzel Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/man/man1m/zfs.1m Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_prop.c Fri Aug 3 00:14:36 2018 (r337213) +++ vendor-sys/illumos/dist/common/zfs/zfs_prop.c Fri Aug 3 00:19:59 2018 (r337214) @@ -411,6 +411,10 @@ zfs_prop_init(void) zprop_register_number(ZFS_PROP_SNAPSHOT_COUNT, "snapshot_count", UINT64_MAX, PROP_DEFAULT, ZFS_TYPE_FILESYSTEM | ZFS_TYPE_VOLUME, "", "SSCOUNT"); + zprop_register_number(ZFS_PROP_GUID, "guid", 0, PROP_READONLY, + ZFS_TYPE_DATASET | ZFS_TYPE_BOOKMARK, "", "GUID"); + zprop_register_number(ZFS_PROP_CREATETXG, "createtxg", 0, PROP_READONLY, + ZFS_TYPE_DATASET | ZFS_TYPE_BOOKMARK, "", "CREATETXG"); /* inherit number properties */ zprop_register_number(ZFS_PROP_RECORDSIZE, "recordsize", @@ -418,8 +422,6 @@ zfs_prop_init(void) ZFS_TYPE_FILESYSTEM, "512 to 1M, power of 2", "RECSIZE"); /* hidden properties */ - zprop_register_hidden(ZFS_PROP_CREATETXG, "createtxg", PROP_TYPE_NUMBER, - PROP_READONLY, ZFS_TYPE_DATASET | ZFS_TYPE_BOOKMARK, "CREATETXG"); zprop_register_hidden(ZFS_PROP_REMAPTXG, "remaptxg", PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_DATASET, "REMAPTXG"); zprop_register_hidden(ZFS_PROP_NUMCLONES, "numclones", PROP_TYPE_NUMBER, @@ -431,8 +433,6 @@ zfs_prop_init(void) zprop_register_hidden(ZFS_PROP_STMF_SHAREINFO, "stmf_sbd_lu", PROP_TYPE_STRING, PROP_INHERIT, ZFS_TYPE_VOLUME, "STMF_SBD_LU"); - zprop_register_hidden(ZFS_PROP_GUID, "guid", PROP_TYPE_NUMBER, - PROP_READONLY, ZFS_TYPE_DATASET | ZFS_TYPE_BOOKMARK, "GUID"); zprop_register_hidden(ZFS_PROP_USERACCOUNTING, "useraccounting", PROP_TYPE_NUMBER, PROP_READONLY, ZFS_TYPE_DATASET, "USERACCOUNTING"); Modified: vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Fri Aug 3 00:14:36 2018 (r337213) +++ vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Fri Aug 3 00:19:59 2018 (r337214) @@ -115,7 +115,7 @@ typedef enum { ZFS_PROP_SNAPDIR, ZFS_PROP_ACLMODE, ZFS_PROP_ACLINHERIT, - ZFS_PROP_CREATETXG, /* not exposed to the user */ + ZFS_PROP_CREATETXG, ZFS_PROP_NAME, /* not exposed to the user */ ZFS_PROP_CANMOUNT, ZFS_PROP_ISCSIOPTIONS, /* not exposed to the user */ From owner-svn-src-vendor@freebsd.org Fri Aug 3 00:20:01 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D6EF1056471; Fri, 3 Aug 2018 00:20:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 30EC4867C7; Fri, 3 Aug 2018 00:20:01 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EEDF618FF7; Fri, 3 Aug 2018 00:20:00 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w730K0Dj040500; Fri, 3 Aug 2018 00:20:00 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w730K0Zs040498; Fri, 3 Aug 2018 00:20:00 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030020.w730K0Zs040498@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 00:20:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337214 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/sys/fs vendor/illumos/dist/lib/libzfs/common vendor/illumos/dist/man/man1m X-SVN-Commit-Revision: 337214 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 00:20:01 -0000 Author: mav Date: Fri Aug 3 00:19:59 2018 New Revision: 337214 URL: https://svnweb.freebsd.org/changeset/base/337214 Log: 9621 Make createtxg and guid properties public illumos/illumos-gate@e8d4a73c868afb740396041be80ed2b141065e76 Reviewed by: Andy Stormont Reviewed by: Paul Dagnelie Reviewed by: Matt Ahrens Reviewed by: Yuri Pankov Approved by: Robert Mustacchi Author: Josh Paetzel Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c vendor/illumos/dist/man/man1m/zfs.1m Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zfs_prop.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Fri Aug 3 00:14:36 2018 (r337213) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_dataset.c Fri Aug 3 00:19:59 2018 (r337214) @@ -2753,6 +2753,7 @@ zfs_prop_get(zfs_handle_t *zhp, zfs_prop_t prop, char break; case ZFS_PROP_GUID: + case ZFS_PROP_CREATETXG: /* * GUIDs are stored as numbers, but they are identifiers. * We don't want them to be pretty printed, because pretty Modified: vendor/illumos/dist/man/man1m/zfs.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zfs.1m Fri Aug 3 00:14:36 2018 (r337213) +++ vendor/illumos/dist/man/man1m/zfs.1m Fri Aug 3 00:19:59 2018 (r337214) @@ -550,6 +550,13 @@ Compression can be turned on by running: .Nm zfs Cm set Sy compression Ns = Ns Sy on Ar dataset . The default value is .Sy off . +.It Sy createtxg +The transaction group (txg) in which the dataset was created. +Bookmarks have the same +.Sy createtxg +as the snapshot they are initially tied to. +This property is suitable for ordering a list of snapshots, +e.g. for incremental send and receive. .It Sy creation The time this dataset was created. .It Sy clones @@ -581,6 +588,14 @@ the dataset tree. This value is only available when a .Sy filesystem_limit has been set somewhere in the tree under which the dataset resides. +.It Sy guid +The 64 bit GUID of this dataset or bookmark which does not change over its +entire lifetime. +When a snapshot is sent to another pool, the received snapshot has the same +GUID. +Thus, the +.Sy guid +is suitable to identify a snapshot across pools. .It Sy logicalreferenced The amount of space that is .Qq logically From owner-svn-src-vendor@freebsd.org Fri Aug 3 00:47:25 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C90921056F1C; Fri, 3 Aug 2018 00:47:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A166882BF; Fri, 3 Aug 2018 00:47:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 57DE9194C0; Fri, 3 Aug 2018 00:47:25 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w730lPcR055636; Fri, 3 Aug 2018 00:47:25 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w730lPL3055635; Fri, 3 Aug 2018 00:47:25 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030047.w730lPL3055635@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 00:47:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337216 - vendor-sys/illumos/dist/common/nvpair X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/nvpair X-SVN-Commit-Revision: 337216 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 00:47:26 -0000 Author: mav Date: Fri Aug 3 00:47:24 2018 New Revision: 337216 URL: https://svnweb.freebsd.org/changeset/base/337216 Log: 7263 deeply nested nvlist can overflow stack illumos/illumos-gate@9ca527c3d3dfa7c8f304b34a9e03b5eddace838f Reviewed by: Adam Leventhal Reviewed by: George Wilson Reviewed by: Robert Mustacchi Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c ============================================================================== --- vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 00:24:27 2018 (r337215) +++ vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 00:47:24 2018 (r337216) @@ -21,6 +21,7 @@ /* * Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved. + * Copyright (c) 2015, 2016 by Delphix. All rights reserved. */ #include @@ -138,6 +139,11 @@ static int nvlist_add_common(nvlist_t *nvl, const char #define NVPAIR2I_NVP(nvp) \ ((i_nvp_t *)((size_t)(nvp) - offsetof(i_nvp_t, nvi_nvp))) +#ifdef _KERNEL +int nvpair_max_recursion = 20; +#else +int nvpair_max_recursion = 100; +#endif int nv_alloc_init(nv_alloc_t *nva, const nv_alloc_ops_t *nvo, /* args */ ...) @@ -2014,6 +2020,7 @@ typedef struct { const nvs_ops_t *nvs_ops; void *nvs_private; nvpriv_t *nvs_priv; + int nvs_recursion; } nvstream_t; /* @@ -2165,9 +2172,16 @@ static int nvs_embedded(nvstream_t *nvs, nvlist_t *embedded) { switch (nvs->nvs_op) { - case NVS_OP_ENCODE: - return (nvs_operation(nvs, embedded, NULL)); + case NVS_OP_ENCODE: { + int err; + if (nvs->nvs_recursion >= nvpair_max_recursion) + return (EINVAL); + nvs->nvs_recursion++; + err = nvs_operation(nvs, embedded, NULL); + nvs->nvs_recursion--; + return (err); + } case NVS_OP_DECODE: { nvpriv_t *priv; int err; @@ -2180,8 +2194,12 @@ nvs_embedded(nvstream_t *nvs, nvlist_t *embedded) nvlist_init(embedded, embedded->nvl_nvflag, priv); + if (nvs->nvs_recursion >= nvpair_max_recursion) + return (EINVAL); + nvs->nvs_recursion++; if ((err = nvs_operation(nvs, embedded, NULL)) != 0) nvlist_free(embedded); + nvs->nvs_recursion--; return (err); } default: @@ -2269,6 +2287,7 @@ nvlist_common(nvlist_t *nvl, char *buf, size_t *buflen return (EINVAL); nvs.nvs_op = nvs_op; + nvs.nvs_recursion = 0; /* * For NVS_OP_ENCODE and NVS_OP_DECODE make sure an nvlist and From owner-svn-src-vendor@freebsd.org Fri Aug 3 01:17:15 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F28410578FD; Fri, 3 Aug 2018 01:17:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 420A888F9F; Fri, 3 Aug 2018 01:17:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1E34719988; Fri, 3 Aug 2018 01:17:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w731HEFp070868; Fri, 3 Aug 2018 01:17:14 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w731HE1Q070867; Fri, 3 Aug 2018 01:17:14 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030117.w731HE1Q070867@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 01:17:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337218 - vendor-sys/illumos/dist/common/nvpair X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/nvpair X-SVN-Commit-Revision: 337218 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 01:17:15 -0000 Author: mav Date: Fri Aug 3 01:17:14 2018 New Revision: 337218 URL: https://svnweb.freebsd.org/changeset/base/337218 Log: 7261 nvlist code should enforce name length limit illumos/illumos-gate@48dd5e630c9b1773b7b10d08a3b90b6c9062d713 Reviewed by: Sebastien Roy Reviewed by: George Wilson Reviewed by: Robert Mustacchi Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c ============================================================================== --- vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 01:09:12 2018 (r337217) +++ vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 01:17:14 2018 (r337218) @@ -910,6 +910,8 @@ nvlist_add_common(nvlist_t *nvl, const char *name, /* calculate sizes of the nvpair elements and the nvpair itself */ name_sz = strlen(name) + 1; + if (name_sz >= 1ULL << (sizeof (nvp->nvp_name_sz) * NBBY - 1)) + return (EINVAL); nvp_sz = NVP_SIZE_CALC(name_sz, value_sz); From owner-svn-src-vendor@freebsd.org Fri Aug 3 01:28:12 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A040D1057CE9; Fri, 3 Aug 2018 01:28:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 56843895DF; Fri, 3 Aug 2018 01:28:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3368719B37; Fri, 3 Aug 2018 01:28:12 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w731SBK3076054; Fri, 3 Aug 2018 01:28:11 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w731SBjN076053; Fri, 3 Aug 2018 01:28:11 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030128.w731SBjN076053@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 01:28:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337220 - vendor-sys/illumos/dist/common/nvpair X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/nvpair X-SVN-Commit-Revision: 337220 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 01:28:12 -0000 Author: mav Date: Fri Aug 3 01:28:11 2018 New Revision: 337220 URL: https://svnweb.freebsd.org/changeset/base/337220 Log: 8375 Kernel memory leak in nvpair code illumos/illumos-gate@843c2111b160463f014d325560ad4b051711928e Reviewed by: Pavel Zakharov Reviewed by: George Wilson Reviewed by: Prashanth Sreenivasa Reviewed by: Robert Mustacchi Approved by: Dan McDonald Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c ============================================================================== --- vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 01:26:07 2018 (r337219) +++ vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 01:28:11 2018 (r337220) @@ -21,7 +21,7 @@ /* * Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2015, 2016 by Delphix. All rights reserved. + * Copyright (c) 2015, 2017 by Delphix. All rights reserved. */ #include @@ -2196,8 +2196,10 @@ nvs_embedded(nvstream_t *nvs, nvlist_t *embedded) nvlist_init(embedded, embedded->nvl_nvflag, priv); - if (nvs->nvs_recursion >= nvpair_max_recursion) + if (nvs->nvs_recursion >= nvpair_max_recursion) { + nvlist_free(embedded); return (EINVAL); + } nvs->nvs_recursion++; if ((err = nvs_operation(nvs, embedded, NULL)) != 0) nvlist_free(embedded); From owner-svn-src-vendor@freebsd.org Fri Aug 3 01:37:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09CD3105801F; Fri, 3 Aug 2018 01:37:16 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B20F889C6E; Fri, 3 Aug 2018 01:37:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 94AAF19CCF; Fri, 3 Aug 2018 01:37:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w731bFXp081305; Fri, 3 Aug 2018 01:37:15 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w731bFvx081304; Fri, 3 Aug 2018 01:37:15 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030137.w731bFvx081304@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 01:37:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337223 - vendor-sys/illumos/dist/common/nvpair vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/lib/libnvpair X-SVN-Group: vendor X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/nvpair vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/lib/libnvpair X-SVN-Commit-Revision: 337223 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 01:37:16 -0000 Author: mav Date: Fri Aug 3 01:37:14 2018 New Revision: 337223 URL: https://svnweb.freebsd.org/changeset/base/337223 Log: 9580 Add a hash-table on top of nvlist to speed-up operations illumos/illumos-gate@2ec7644aab2a726a64681fa66c6db8731b160de1 Reviewed by: Matt Ahrens Reviewed by: Sebastien Roy Approved by: Robert Mustacchi Author: Serapheim Dimitropoulos Modified: vendor/illumos/dist/lib/libnvpair/nvpair_json.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c vendor-sys/illumos/dist/uts/common/sys/nvpair.h vendor-sys/illumos/dist/uts/common/sys/nvpair_impl.h Modified: vendor/illumos/dist/lib/libnvpair/nvpair_json.c ============================================================================== --- vendor/illumos/dist/lib/libnvpair/nvpair_json.c Fri Aug 3 01:37:00 2018 (r337222) +++ vendor/illumos/dist/lib/libnvpair/nvpair_json.c Fri Aug 3 01:37:14 2018 (r337223) @@ -10,6 +10,7 @@ */ /* * Copyright (c) 2014, Joyent, Inc. + * Copyright (c) 2017 by Delphix. All rights reserved. */ #include @@ -394,8 +395,10 @@ nvlist_print_json(FILE *fp, nvlist_t *nvl) } case DATA_TYPE_UNKNOWN: + case DATA_TYPE_DONTCARE: return (-1); } + } FPRINTF(fp, "}"); From owner-svn-src-vendor@freebsd.org Fri Aug 3 01:37:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C66ED105801E; Fri, 3 Aug 2018 01:37:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A72989C6D; Fri, 3 Aug 2018 01:37:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 42D7219CCE; Fri, 3 Aug 2018 01:37:15 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w731bF23081298; Fri, 3 Aug 2018 01:37:15 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w731bEir081295; Fri, 3 Aug 2018 01:37:14 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201808030137.w731bEir081295@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 3 Aug 2018 01:37:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337223 - vendor-sys/illumos/dist/common/nvpair vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/lib/libnvpair X-SVN-Group: vendor-sys X-SVN-Commit-Author: mav X-SVN-Commit-Paths: vendor-sys/illumos/dist/common/nvpair vendor-sys/illumos/dist/uts/common/sys vendor/illumos/dist/lib/libnvpair X-SVN-Commit-Revision: 337223 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Aug 2018 01:37:16 -0000 Author: mav Date: Fri Aug 3 01:37:14 2018 New Revision: 337223 URL: https://svnweb.freebsd.org/changeset/base/337223 Log: 9580 Add a hash-table on top of nvlist to speed-up operations illumos/illumos-gate@2ec7644aab2a726a64681fa66c6db8731b160de1 Reviewed by: Matt Ahrens Reviewed by: Sebastien Roy Approved by: Robert Mustacchi Author: Serapheim Dimitropoulos Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c vendor-sys/illumos/dist/uts/common/sys/nvpair.h vendor-sys/illumos/dist/uts/common/sys/nvpair_impl.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/lib/libnvpair/nvpair_json.c Modified: vendor-sys/illumos/dist/common/nvpair/nvpair.c ============================================================================== --- vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 01:37:00 2018 (r337222) +++ vendor-sys/illumos/dist/common/nvpair/nvpair.c Fri Aug 3 01:37:14 2018 (r337223) @@ -145,6 +145,8 @@ int nvpair_max_recursion = 20; int nvpair_max_recursion = 100; #endif +uint64_t nvlist_hashtable_init_size = (1 << 4); + int nv_alloc_init(nv_alloc_t *nva, const nv_alloc_ops_t *nvo, /* args */ ...) { @@ -252,7 +254,292 @@ nv_priv_alloc_embedded(nvpriv_t *priv) return (emb_priv); } +static int +nvt_tab_alloc(nvpriv_t *priv, uint64_t buckets) +{ + ASSERT3P(priv->nvp_hashtable, ==, NULL); + ASSERT0(priv->nvp_nbuckets); + ASSERT0(priv->nvp_nentries); + + i_nvp_t **tab = nv_mem_zalloc(priv, buckets * sizeof (i_nvp_t *)); + if (tab == NULL) + return (ENOMEM); + + priv->nvp_hashtable = tab; + priv->nvp_nbuckets = buckets; + return (0); +} + static void +nvt_tab_free(nvpriv_t *priv) +{ + i_nvp_t **tab = priv->nvp_hashtable; + if (tab == NULL) { + ASSERT0(priv->nvp_nbuckets); + ASSERT0(priv->nvp_nentries); + return; + } + + nv_mem_free(priv, tab, priv->nvp_nbuckets * sizeof (i_nvp_t *)); + + priv->nvp_hashtable = NULL; + priv->nvp_nbuckets = 0; + priv->nvp_nentries = 0; +} + +static uint32_t +nvt_hash(const char *p) +{ + uint32_t g, hval = 0; + + while (*p) { + hval = (hval << 4) + *p++; + if ((g = (hval & 0xf0000000)) != 0) + hval ^= g >> 24; + hval &= ~g; + } + return (hval); +} + +static boolean_t +nvt_nvpair_match(nvpair_t *nvp1, nvpair_t *nvp2, uint32_t nvflag) +{ + boolean_t match = B_FALSE; + if (nvflag & NV_UNIQUE_NAME_TYPE) { + if (strcmp(NVP_NAME(nvp1), NVP_NAME(nvp2)) == 0 && + NVP_TYPE(nvp1) == NVP_TYPE(nvp2)) + match = B_TRUE; + } else { + ASSERT(nvflag == 0 || nvflag & NV_UNIQUE_NAME); + if (strcmp(NVP_NAME(nvp1), NVP_NAME(nvp2)) == 0) + match = B_TRUE; + } + return (match); +} + +static nvpair_t * +nvt_lookup_name_type(nvlist_t *nvl, const char *name, data_type_t type) +{ + nvpriv_t *priv = (nvpriv_t *)(uintptr_t)nvl->nvl_priv; + ASSERT(priv != NULL); + + i_nvp_t **tab = priv->nvp_hashtable; + + if (tab == NULL) { + ASSERT3P(priv->nvp_list, ==, NULL); + ASSERT0(priv->nvp_nbuckets); + ASSERT0(priv->nvp_nentries); + return (NULL); + } else { + ASSERT(priv->nvp_nbuckets != 0); + } + + uint64_t hash = nvt_hash(name); + uint64_t index = hash & (priv->nvp_nbuckets - 1); + + ASSERT3U(index, <, priv->nvp_nbuckets); + i_nvp_t *entry = tab[index]; + + for (i_nvp_t *e = entry; e != NULL; e = e->nvi_hashtable_next) { + if (strcmp(NVP_NAME(&e->nvi_nvp), name) == 0 && + (type == DATA_TYPE_DONTCARE || + NVP_TYPE(&e->nvi_nvp) == type)) + return (&e->nvi_nvp); + } + return (NULL); +} + +static nvpair_t * +nvt_lookup_name(nvlist_t *nvl, const char *name) +{ + return (nvt_lookup_name_type(nvl, name, DATA_TYPE_DONTCARE)); +} + +static int +nvt_resize(nvpriv_t *priv, uint32_t new_size) +{ + i_nvp_t **tab = priv->nvp_hashtable; + + /* + * Migrate all the entries from the current table + * to a newly-allocated table with the new size by + * re-adjusting the pointers of their entries. + */ + uint32_t size = priv->nvp_nbuckets; + uint32_t new_mask = new_size - 1; + ASSERT(ISP2(new_size)); + + i_nvp_t **new_tab = nv_mem_zalloc(priv, new_size * sizeof (i_nvp_t *)); + if (new_tab == NULL) + return (ENOMEM); + + uint32_t nentries = 0; + for (uint32_t i = 0; i < size; i++) { + i_nvp_t *next, *e = tab[i]; + + while (e != NULL) { + next = e->nvi_hashtable_next; + + uint32_t hash = nvt_hash(NVP_NAME(&e->nvi_nvp)); + uint32_t index = hash & new_mask; + + e->nvi_hashtable_next = new_tab[index]; + new_tab[index] = e; + nentries++; + + e = next; + } + tab[i] = NULL; + } + ASSERT3U(nentries, ==, priv->nvp_nentries); + + nvt_tab_free(priv); + + priv->nvp_hashtable = new_tab; + priv->nvp_nbuckets = new_size; + priv->nvp_nentries = nentries; + + return (0); +} + +static boolean_t +nvt_needs_togrow(nvpriv_t *priv) +{ + /* + * Grow only when we have more elements than buckets + * and the # of buckets doesn't overflow. + */ + return (priv->nvp_nentries > priv->nvp_nbuckets && + (UINT32_MAX >> 1) >= priv->nvp_nbuckets); +} + +/* + * Allocate a new table that's twice the size of the old one, + * and migrate all the entries from the old one to the new + * one by re-adjusting their pointers. + */ +static int +nvt_grow(nvpriv_t *priv) +{ + uint32_t current_size = priv->nvp_nbuckets; + /* ensure we won't overflow */ + ASSERT3U(UINT32_MAX >> 1, >=, current_size); + return (nvt_resize(priv, current_size << 1)); +} + +static boolean_t +nvt_needs_toshrink(nvpriv_t *priv) +{ + /* + * Shrink only when the # of elements is less than or + * equal to 1/4 the # of buckets. Never shrink less than + * nvlist_hashtable_init_size. + */ + ASSERT3U(priv->nvp_nbuckets, >=, nvlist_hashtable_init_size); + if (priv->nvp_nbuckets == nvlist_hashtable_init_size) + return (B_FALSE); + return (priv->nvp_nentries <= (priv->nvp_nbuckets >> 2)); +} + +/* + * Allocate a new table that's half the size of the old one, + * and migrate all the entries from the old one to the new + * one by re-adjusting their pointers. + */ +static int +nvt_shrink(nvpriv_t *priv) +{ + uint32_t current_size = priv->nvp_nbuckets; + /* ensure we won't overflow */ + ASSERT3U(current_size, >=, nvlist_hashtable_init_size); + return (nvt_resize(priv, current_size >> 1)); +} + +static int +nvt_remove_nvpair(nvlist_t *nvl, nvpair_t *nvp) +{ + nvpriv_t *priv = (nvpriv_t *)(uintptr_t)nvl->nvl_priv; + + if (nvt_needs_toshrink(priv)) { + int err = nvt_shrink(priv); + if (err != 0) + return (err); + } + i_nvp_t **tab = priv->nvp_hashtable; + + char *name = NVP_NAME(nvp); + uint64_t hash = nvt_hash(name); + uint64_t index = hash & (priv->nvp_nbuckets - 1); + + ASSERT3U(index, <, priv->nvp_nbuckets); + i_nvp_t *bucket = tab[index]; + + for (i_nvp_t *prev = NULL, *e = bucket; + e != NULL; prev = e, e = e->nvi_hashtable_next) { + if (nvt_nvpair_match(&e->nvi_nvp, nvp, nvl->nvl_flag)) { + if (prev != NULL) { + prev->nvi_hashtable_next = + e->nvi_hashtable_next; + } else { + ASSERT3P(e, ==, bucket); + tab[index] = e->nvi_hashtable_next; + } + e->nvi_hashtable_next = NULL; + priv->nvp_nentries--; + break; + } + } + + return (0); +} + +static int +nvt_add_nvpair(nvlist_t *nvl, nvpair_t *nvp) +{ + nvpriv_t *priv = (nvpriv_t *)(uintptr_t)nvl->nvl_priv; + + /* initialize nvpair table now if it doesn't exist. */ + if (priv->nvp_hashtable == NULL) { + int err = nvt_tab_alloc(priv, nvlist_hashtable_init_size); + if (err != 0) + return (err); + } + + /* + * if we don't allow duplicate entries, make sure to + * unlink any existing entries from the table. + */ + if (nvl->nvl_nvflag != 0) { + int err = nvt_remove_nvpair(nvl, nvp); + if (err != 0) + return (err); + } + + if (nvt_needs_togrow(priv)) { + int err = nvt_grow(priv); + if (err != 0) + return (err); + } + i_nvp_t **tab = priv->nvp_hashtable; + + char *name = NVP_NAME(nvp); + uint64_t hash = nvt_hash(name); + uint64_t index = hash & (priv->nvp_nbuckets - 1); + + ASSERT3U(index, <, priv->nvp_nbuckets); + i_nvp_t *bucket = tab[index]; + + /* insert link at the beginning of the bucket */ + i_nvp_t *new_entry = NVPAIR2I_NVP(nvp); + ASSERT3P(new_entry->nvi_hashtable_next, ==, NULL); + new_entry->nvi_hashtable_next = bucket; + tab[index] = new_entry; + + priv->nvp_nentries++; + return (0); +} + +static void nvlist_init(nvlist_t *nvl, uint32_t nvflag, nvpriv_t *priv) { nvl->nvl_version = NV_VERSION; @@ -584,6 +871,7 @@ nvlist_free(nvlist_t *nvl) else nvl->nvl_priv = 0; + nvt_tab_free(priv); nv_mem_free(priv, priv, sizeof (nvpriv_t)); } @@ -644,26 +932,14 @@ nvlist_xdup(nvlist_t *nvl, nvlist_t **nvlp, nv_alloc_t int nvlist_remove_all(nvlist_t *nvl, const char *name) { - nvpriv_t *priv; - i_nvp_t *curr; int error = ENOENT; - if (nvl == NULL || name == NULL || - (priv = (nvpriv_t *)(uintptr_t)nvl->nvl_priv) == NULL) + if (nvl == NULL || name == NULL || nvl->nvl_priv == 0) return (EINVAL); - curr = priv->nvp_list; - while (curr != NULL) { - nvpair_t *nvp = &curr->nvi_nvp; - - curr = curr->nvi_next; - if (strcmp(name, NVP_NAME(nvp)) != 0) - continue; - - nvp_buf_unlink(nvl, nvp); - nvpair_free(nvp); - nvp_buf_free(nvl, nvp); - + nvpair_t *nvp; + while ((nvp = nvt_lookup_name(nvl, name)) != NULL) { + VERIFY0(nvlist_remove_nvpair(nvl, nvp)); error = 0; } @@ -676,28 +952,14 @@ nvlist_remove_all(nvlist_t *nvl, const char *name) int nvlist_remove(nvlist_t *nvl, const char *name, data_type_t type) { - nvpriv_t *priv; - i_nvp_t *curr; - - if (nvl == NULL || name == NULL || - (priv = (nvpriv_t *)(uintptr_t)nvl->nvl_priv) == NULL) + if (nvl == NULL || name == NULL || nvl->nvl_priv == 0) return (EINVAL); - curr = priv->nvp_list; - while (curr != NULL) { - nvpair_t *nvp = &curr->nvi_nvp; + nvpair_t *nvp = nvt_lookup_name_type(nvl, name, type); + if (nvp == NULL) + return (ENOENT); - if (strcmp(name, NVP_NAME(nvp)) == 0 && NVP_TYPE(nvp) == type) { - nvp_buf_unlink(nvl, nvp); - nvpair_free(nvp); - nvp_buf_free(nvl, nvp); - - return (0); - } - curr = curr->nvi_next; - } - - return (ENOENT); + return (nvlist_remove_nvpair(nvl, nvp)); } int @@ -706,6 +968,10 @@ nvlist_remove_nvpair(nvlist_t *nvl, nvpair_t *nvp) if (nvl == NULL || nvp == NULL) return (EINVAL); + int err = nvt_remove_nvpair(nvl, nvp); + if (err != 0) + return (err); + nvp_buf_unlink(nvl, nvp); nvpair_free(nvp); nvp_buf_free(nvl, nvp); @@ -983,6 +1249,12 @@ nvlist_add_common(nvlist_t *nvl, const char *name, else if (nvl->nvl_nvflag & NV_UNIQUE_NAME_TYPE) (void) nvlist_remove(nvl, name, type); + err = nvt_add_nvpair(nvl, nvp); + if (err != 0) { + nvpair_free(nvp); + nvp_buf_free(nvl, nvp); + return (err); + } nvp_buf_link(nvl, nvp); return (0); @@ -1332,25 +1604,17 @@ static int nvlist_lookup_common(nvlist_t *nvl, const char *name, data_type_t type, uint_t *nelem, void *data) { - nvpriv_t *priv; - nvpair_t *nvp; - i_nvp_t *curr; - - if (name == NULL || nvl == NULL || - (priv = (nvpriv_t *)(uintptr_t)nvl->nvl_priv) == NULL) + if (name == NULL || nvl == NULL || nvl->nvl_priv == 0) return (EINVAL); if (!(nvl->nvl_nvflag & (NV_UNIQUE_NAME | NV_UNIQUE_NAME_TYPE))) return (ENOTSUP); - for (curr = priv->nvp_list; curr != NULL; curr = curr->nvi_next) { - nvp = &curr->nvi_nvp; + nvpair_t *nvp = nvt_lookup_name_type(nvl, name, type); + if (nvp == NULL) + return (ENOENT); - if (strcmp(name, NVP_NAME(nvp)) == 0 && NVP_TYPE(nvp) == type) - return (nvpair_value_common(nvp, type, nelem, data)); - } - - return (ENOENT); + return (nvpair_value_common(nvp, type, nelem, data)); } int @@ -2108,6 +2372,12 @@ nvs_decode_pairs(nvstream_t *nvs, nvlist_t *nvl) return (EFAULT); } + err = nvt_add_nvpair(nvl, nvp); + if (err != 0) { + nvpair_free(nvp); + nvp_buf_free(nvl, nvp); + return (err); + } nvp_buf_link(nvl, nvp); } return (err); Modified: vendor-sys/illumos/dist/uts/common/sys/nvpair.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/nvpair.h Fri Aug 3 01:37:00 2018 (r337222) +++ vendor-sys/illumos/dist/uts/common/sys/nvpair.h Fri Aug 3 01:37:14 2018 (r337223) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. */ #ifndef _SYS_NVPAIR_H @@ -40,6 +40,7 @@ extern "C" { #endif typedef enum { + DATA_TYPE_DONTCARE = -1, DATA_TYPE_UNKNOWN = 0, DATA_TYPE_BOOLEAN, DATA_TYPE_BYTE, Modified: vendor-sys/illumos/dist/uts/common/sys/nvpair_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/nvpair_impl.h Fri Aug 3 01:37:00 2018 (r337222) +++ vendor-sys/illumos/dist/uts/common/sys/nvpair_impl.h Fri Aug 3 01:37:14 2018 (r337223) @@ -24,11 +24,13 @@ * Use is subject to license terms. */ +/* + * Copyright (c) 2017 by Delphix. All rights reserved. + */ + #ifndef _NVPAIR_IMPL_H #define _NVPAIR_IMPL_H -#pragma ident "%Z%%M% %I% %E% SMI" - #ifdef __cplusplus extern "C" { #endif @@ -47,16 +49,27 @@ typedef struct i_nvp i_nvp_t; struct i_nvp { union { - uint64_t _nvi_align; /* ensure alignment */ + /* ensure alignment */ + uint64_t _nvi_align; + struct { - i_nvp_t *_nvi_next; /* pointer to next nvpair */ - i_nvp_t *_nvi_prev; /* pointer to prev nvpair */ + /* pointer to next nvpair */ + i_nvp_t *_nvi_next; + + /* pointer to prev nvpair */ + i_nvp_t *_nvi_prev; + + /* next pair in table bucket */ + i_nvp_t *_nvi_hashtable_next; } _nvi; } _nvi_un; - nvpair_t nvi_nvp; /* nvpair */ + + /* nvpair */ + nvpair_t nvi_nvp; }; #define nvi_next _nvi_un._nvi._nvi_next #define nvi_prev _nvi_un._nvi._nvi_prev +#define nvi_hashtable_next _nvi_un._nvi._nvi_hashtable_next typedef struct { i_nvp_t *nvp_list; /* linked list of nvpairs */ @@ -64,6 +77,10 @@ typedef struct { i_nvp_t *nvp_curr; /* current walker nvpair */ nv_alloc_t *nvp_nva; /* pluggable allocator */ uint32_t nvp_stat; /* internal state */ + + i_nvp_t **nvp_hashtable; /* table of entries used for lookup */ + uint32_t nvp_nbuckets; /* # of buckets in hash table */ + uint32_t nvp_nentries; /* # of entries in hash table */ } nvpriv_t; #ifdef __cplusplus From owner-svn-src-vendor@freebsd.org Sat Aug 4 12:52:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 070A1106A45F; Sat, 4 Aug 2018 12:52:16 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AF439747EA; Sat, 4 Aug 2018 12:52:15 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 776BA7C03; Sat, 4 Aug 2018 12:52:15 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74CqF9T072747; Sat, 4 Aug 2018 12:52:15 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74CqFpO072746; Sat, 4 Aug 2018 12:52:15 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041252.w74CqFpO072746@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 12:52:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337292 - vendor/llvm/dist-release_70 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/llvm/dist-release_70 X-SVN-Commit-Revision: 337292 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 12:52:16 -0000 Author: dim Date: Sat Aug 4 12:52:15 2018 New Revision: 337292 URL: https://svnweb.freebsd.org/changeset/base/337292 Log: Branch vendor/llvm/dist to vendor/llvm/dist-release_70, to allow for independent merges of the upstream trunk and release_70 branches. Added: vendor/llvm/dist-release_70/ - copied from r337291, vendor/llvm/dist/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 12:52:22 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E83B106A4AB; Sat, 4 Aug 2018 12:52:22 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09517748F6; Sat, 4 Aug 2018 12:52:22 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C0E3A7C0E; Sat, 4 Aug 2018 12:52:21 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74CqLx6072845; Sat, 4 Aug 2018 12:52:21 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74CqLPg072844; Sat, 4 Aug 2018 12:52:21 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041252.w74CqLPg072844@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 12:52:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337294 - vendor/compiler-rt/dist-release_70 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/compiler-rt/dist-release_70 X-SVN-Commit-Revision: 337294 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 12:52:22 -0000 Author: dim Date: Sat Aug 4 12:52:21 2018 New Revision: 337294 URL: https://svnweb.freebsd.org/changeset/base/337294 Log: Branch vendor/compiler-rt/dist to vendor/compiler-rt/dist-release_70, to allow for independent merges of the upstream trunk and release_70 branches. Added: vendor/compiler-rt/dist-release_70/ - copied from r337293, vendor/compiler-rt/dist/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 12:52:21 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B5A1106A4A5; Sat, 4 Aug 2018 12:52:21 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E77A6748C9; Sat, 4 Aug 2018 12:52:20 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CB8AB7C07; Sat, 4 Aug 2018 12:52:18 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74CqI0i072793; Sat, 4 Aug 2018 12:52:18 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74CqIZG072792; Sat, 4 Aug 2018 12:52:18 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041252.w74CqIZG072792@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 12:52:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337293 - vendor/clang/dist-release_70 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/clang/dist-release_70 X-SVN-Commit-Revision: 337293 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 12:52:21 -0000 Author: dim Date: Sat Aug 4 12:52:18 2018 New Revision: 337293 URL: https://svnweb.freebsd.org/changeset/base/337293 Log: Branch vendor/clang/dist to vendor/clang/dist-release_70, to allow for independent merges of the upstream trunk and release_70 branches. Added: vendor/clang/dist-release_70/ - copied from r337292, vendor/clang/dist/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 12:52:26 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74B06106A4EA; Sat, 4 Aug 2018 12:52:26 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 987CB749B6; Sat, 4 Aug 2018 12:52:25 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EAD7B7C14; Sat, 4 Aug 2018 12:52:24 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74CqO92072892; Sat, 4 Aug 2018 12:52:24 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74CqOrT072891; Sat, 4 Aug 2018 12:52:24 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041252.w74CqOrT072891@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 12:52:24 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337295 - vendor/libc++/dist-release_70 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/libc++/dist-release_70 X-SVN-Commit-Revision: 337295 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 12:52:26 -0000 Author: dim Date: Sat Aug 4 12:52:24 2018 New Revision: 337295 URL: https://svnweb.freebsd.org/changeset/base/337295 Log: Branch vendor/libc++/dist to vendor/libc++/dist-release_70, to allow for independent merges of the upstream trunk and release_70 branches. Added: vendor/libc++/dist-release_70/ - copied from r337294, vendor/libc++/dist/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 12:52:30 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5EF54106A50D; Sat, 4 Aug 2018 12:52:30 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48DB874A3B; Sat, 4 Aug 2018 12:52:28 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 471AC7C17; Sat, 4 Aug 2018 12:52:28 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74CqRQq072941; Sat, 4 Aug 2018 12:52:27 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74CqRDq072940; Sat, 4 Aug 2018 12:52:27 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041252.w74CqRDq072940@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 12:52:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337296 - vendor/lld/dist-release_70 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lld/dist-release_70 X-SVN-Commit-Revision: 337296 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 12:52:30 -0000 Author: dim Date: Sat Aug 4 12:52:27 2018 New Revision: 337296 URL: https://svnweb.freebsd.org/changeset/base/337296 Log: Branch vendor/lld/dist to vendor/lld/dist-release_70, to allow for independent merges of the upstream trunk and release_70 branches. Added: vendor/lld/dist-release_70/ - copied from r337295, vendor/lld/dist/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 12:52:37 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D35C106A530; Sat, 4 Aug 2018 12:52:37 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D87A674AE2; Sat, 4 Aug 2018 12:52:31 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 442017C1F; Sat, 4 Aug 2018 12:52:31 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74CqVLc072991; Sat, 4 Aug 2018 12:52:31 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74CqVM2072990; Sat, 4 Aug 2018 12:52:31 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041252.w74CqVM2072990@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 12:52:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337297 - vendor/lldb/dist-release_70 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lldb/dist-release_70 X-SVN-Commit-Revision: 337297 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 12:52:37 -0000 Author: dim Date: Sat Aug 4 12:52:30 2018 New Revision: 337297 URL: https://svnweb.freebsd.org/changeset/base/337297 Log: Branch vendor/lldb/dist to vendor/lldb/dist-release_70, to allow for independent merges of the upstream trunk and release_70 branches. Added: vendor/lldb/dist-release_70/ - copied from r337296, vendor/lldb/dist/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:08:37 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BED7B106AB38; Sat, 4 Aug 2018 13:08:37 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 74543756B5; Sat, 4 Aug 2018 13:08:37 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 557C67DC5; Sat, 4 Aug 2018 13:08:37 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74D8bEd078224; Sat, 4 Aug 2018 13:08:37 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74D8b9i078223; Sat, 4 Aug 2018 13:08:37 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041308.w74D8b9i078223@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:08:37 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337298 - vendor/libcxxrt/dist X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/libcxxrt/dist X-SVN-Commit-Revision: 337298 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:08:37 -0000 Author: dim Date: Sat Aug 4 13:08:36 2018 New Revision: 337298 URL: https://svnweb.freebsd.org/changeset/base/337298 Log: Remove FreeBSD keyword properties. Modified: Directory Properties: vendor/libcxxrt/dist/CMakeLists.txt (props changed) vendor/libcxxrt/dist/abi_namespace.h (props changed) vendor/libcxxrt/dist/auxhelper.cc (props changed) vendor/libcxxrt/dist/cxxabi.h (props changed) vendor/libcxxrt/dist/dwarf_eh.h (props changed) vendor/libcxxrt/dist/dynamic_cast.cc (props changed) vendor/libcxxrt/dist/exception.cc (props changed) vendor/libcxxrt/dist/guard.cc (props changed) vendor/libcxxrt/dist/libelftc_dem_gnu3.c (props changed) vendor/libcxxrt/dist/memory.cc (props changed) vendor/libcxxrt/dist/stdexcept.cc (props changed) vendor/libcxxrt/dist/stdexcept.h (props changed) vendor/libcxxrt/dist/terminate.cc (props changed) vendor/libcxxrt/dist/typeinfo.cc (props changed) vendor/libcxxrt/dist/typeinfo.h (props changed) From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:15:09 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD2FB106AE86; Sat, 4 Aug 2018 13:15:08 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6F20975C2E; Sat, 4 Aug 2018 13:15:08 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4F5087F5B; Sat, 4 Aug 2018 13:15:08 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DF8bx083258; Sat, 4 Aug 2018 13:15:08 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DF61w083248; Sat, 4 Aug 2018 13:15:06 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041315.w74DF61w083248@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:15:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337299 - in vendor/llvm/dist-release_70: . docs include/llvm/Support lib/Support lib/Target/AArch64 lib/Target/PowerPC lib/Target/X86 test/CodeGen/AArch64 test/CodeGen/PowerPC test/Cod... X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/llvm/dist-release_70: . docs include/llvm/Support lib/Support lib/Target/AArch64 lib/Target/PowerPC lib/Target/X86 test/CodeGen/AArch64 test/CodeGen/PowerPC test/CodeGen/X86 test/tools/llvm-... X-SVN-Commit-Revision: 337299 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:15:09 -0000 Author: dim Date: Sat Aug 4 13:15:05 2018 New Revision: 337299 URL: https://svnweb.freebsd.org/changeset/base/337299 Log: Vendor import of llvm release_70 branch r338892: https://llvm.org/svn/llvm-project/llvm/branches/release_70@338892 Added: vendor/llvm/dist-release_70/test/CodeGen/X86/absolute-bit-mask-fastisel.ll Modified: vendor/llvm/dist-release_70/CMakeLists.txt vendor/llvm/dist-release_70/docs/ReleaseNotes.rst vendor/llvm/dist-release_70/include/llvm/Support/DebugCounter.h vendor/llvm/dist-release_70/lib/Support/DebugCounter.cpp vendor/llvm/dist-release_70/lib/Target/AArch64/AArch64InstrFormats.td vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCISelLowering.cpp vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCInstrVSX.td vendor/llvm/dist-release_70/lib/Target/X86/X86FastISel.cpp vendor/llvm/dist-release_70/test/CodeGen/AArch64/f16-instructions.ll vendor/llvm/dist-release_70/test/CodeGen/PowerPC/build-vector-tests.ll vendor/llvm/dist-release_70/test/tools/llvm-ar/invalid-command-line.test vendor/llvm/dist-release_70/tools/llvm-ar/llvm-ar.cpp vendor/llvm/dist-release_70/tools/llvm-shlib/CMakeLists.txt vendor/llvm/dist-release_70/utils/release/tag.sh Modified: vendor/llvm/dist-release_70/CMakeLists.txt ============================================================================== --- vendor/llvm/dist-release_70/CMakeLists.txt Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/CMakeLists.txt Sat Aug 4 13:15:05 2018 (r337299) @@ -32,7 +32,7 @@ if(NOT DEFINED LLVM_VERSION_PATCH) set(LLVM_VERSION_PATCH 0) endif() if(NOT DEFINED LLVM_VERSION_SUFFIX) - set(LLVM_VERSION_SUFFIX svn) + set(LLVM_VERSION_SUFFIX "") endif() if (NOT PACKAGE_VERSION) @@ -556,8 +556,6 @@ if(LLVM_LINK_LLVM_DYLIB OR LLVM_BUILD_LLVM_C_DYLIB) set(LLVM_BUILD_LLVM_DYLIB_default ON) endif() option(LLVM_BUILD_LLVM_DYLIB "Build libllvm dynamic library" ${LLVM_BUILD_LLVM_DYLIB_default}) - -option(LLVM_DYLIB_SYMBOL_VERSIONING OFF) option(LLVM_OPTIMIZED_TABLEGEN "Force TableGen to be built with optimization" OFF) if(CMAKE_CROSSCOMPILING OR (LLVM_OPTIMIZED_TABLEGEN AND (LLVM_ENABLE_ASSERTIONS OR CMAKE_CONFIGURATION_TYPES))) Modified: vendor/llvm/dist-release_70/docs/ReleaseNotes.rst ============================================================================== --- vendor/llvm/dist-release_70/docs/ReleaseNotes.rst Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/docs/ReleaseNotes.rst Sat Aug 4 13:15:05 2018 (r337299) @@ -65,7 +65,7 @@ Non-comprehensive list of changes in this release results for code that is relying on the undefined behavior of overflowing casts. The optimization can be disabled by specifying a function attribute: "strict-float-cast-overflow"="false". This attribute may be created by the - clang option :option:`-fno-strict-float-cast-overflow`. + clang option ``-fno-strict-float-cast-overflow``. Code sanitizers can be used to detect affected patterns. The option for detecting this problem alone is "-fsanitize=float-cast-overflow": @@ -108,6 +108,12 @@ Non-comprehensive list of changes in this release implementations saddle std::vector with and is better at using ``realloc``, it's now a better choice even on the heap (although when TinyPtrVector works, it's even smaller). + +* Preliminary/experimental support for DWARF v5 debugging information, + including the new .debug_names accelerator table. DWARF emitted at ``-O0`` + should be fully DWARF v5 compliant. Type units and split DWARF are known + not to be compliant, and higher optimization levels will still emit some + information in v4 format. * Note.. Modified: vendor/llvm/dist-release_70/include/llvm/Support/DebugCounter.h ============================================================================== --- vendor/llvm/dist-release_70/include/llvm/Support/DebugCounter.h Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/include/llvm/Support/DebugCounter.h Sat Aug 4 13:15:05 2018 (r337299) @@ -70,10 +70,9 @@ class DebugCounter { (public) return instance().addCounter(Name, Desc); } inline static bool shouldExecute(unsigned CounterName) { -// Compile to nothing when debugging is off -#ifdef NDEBUG - return true; -#else + if (!isCountingEnabled()) + return true; + auto &Us = instance(); auto Result = Us.Counters.find(CounterName); if (Result != Us.Counters.end()) { @@ -93,7 +92,6 @@ class DebugCounter { (public) } // Didn't find the counter, should we warn? return true; -#endif // NDEBUG } // Return true if a given counter had values set (either programatically or on @@ -142,7 +140,23 @@ class DebugCounter { (public) } CounterVector::const_iterator end() const { return RegisteredCounters.end(); } + // Force-enables counting all DebugCounters. + // + // Since DebugCounters are incompatible with threading (not only do they not + // make sense, but we'll also see data races), this should only be used in + // contexts where we're certain we won't spawn threads. + static void enableAllCounters() { instance().Enabled = true; } + private: + static bool isCountingEnabled() { +// Compile to nothing when debugging is off +#ifdef NDEBUG + return false; +#else + return instance().Enabled; +#endif + } + unsigned addCounter(const std::string &Name, const std::string &Desc) { unsigned Result = RegisteredCounters.insert(Name); Counters[Result] = {}; @@ -159,6 +173,10 @@ class DebugCounter { (public) }; DenseMap Counters; CounterVector RegisteredCounters; + + // Whether we should do DebugCounting at all. DebugCounters aren't + // thread-safe, so this should always be false in multithreaded scenarios. + bool Enabled = false; }; #define DEBUG_COUNTER(VARNAME, COUNTERNAME, DESC) \ Modified: vendor/llvm/dist-release_70/lib/Support/DebugCounter.cpp ============================================================================== --- vendor/llvm/dist-release_70/lib/Support/DebugCounter.cpp Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/lib/Support/DebugCounter.cpp Sat Aug 4 13:15:05 2018 (r337299) @@ -82,6 +82,7 @@ void DebugCounter::push_back(const std::string &Val) { << " is not a registered counter\n"; return; } + enableAllCounters(); Counters[CounterID].Skip = CounterVal; Counters[CounterID].IsSet = true; } else if (CounterPair.first.endswith("-count")) { @@ -92,6 +93,7 @@ void DebugCounter::push_back(const std::string &Val) { << " is not a registered counter\n"; return; } + enableAllCounters(); Counters[CounterID].StopAfter = CounterVal; Counters[CounterID].IsSet = true; } else { Modified: vendor/llvm/dist-release_70/lib/Target/AArch64/AArch64InstrFormats.td ============================================================================== --- vendor/llvm/dist-release_70/lib/Target/AArch64/AArch64InstrFormats.td Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/lib/Target/AArch64/AArch64InstrFormats.td Sat Aug 4 13:15:05 2018 (r337299) @@ -4639,7 +4639,9 @@ class BaseFPCondComparison { - def Hrr : BaseFPCondComparison { + def Hrr : BaseFPCondComparison { let Inst{23-22} = 0b11; let Predicates = [HasFullFP16]; } Modified: vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCISelLowering.cpp ============================================================================== --- vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCISelLowering.cpp Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCISelLowering.cpp Sat Aug 4 13:15:05 2018 (r337299) @@ -11761,6 +11761,14 @@ SDValue PPCTargetLowering::DAGCombineExtBoolTrunc(SDNo ShiftCst); } +// Is this an extending load from an f32 to an f64? +static bool isFPExtLoad(SDValue Op) { + if (LoadSDNode *LD = dyn_cast(Op.getNode())) + return LD->getExtensionType() == ISD::EXTLOAD && + Op.getValueType() == MVT::f64; + return false; +} + /// Reduces the number of fp-to-int conversion when building a vector. /// /// If this vector is built out of floating to integer conversions, @@ -11795,11 +11803,18 @@ combineElementTruncationToVectorTruncation(SDNode *N, SmallVector Ops; EVT TargetVT = N->getValueType(0); for (int i = 0, e = N->getNumOperands(); i < e; ++i) { - if (N->getOperand(i).getOpcode() != PPCISD::MFVSR) + SDValue NextOp = N->getOperand(i); + if (NextOp.getOpcode() != PPCISD::MFVSR) return SDValue(); - unsigned NextConversion = N->getOperand(i).getOperand(0).getOpcode(); + unsigned NextConversion = NextOp.getOperand(0).getOpcode(); if (NextConversion != FirstConversion) return SDValue(); + // If we are converting to 32-bit integers, we need to add an FP_ROUND. + // This is not valid if the input was originally double precision. It is + // also not profitable to do unless this is an extending load in which + // case doing this combine will allow us to combine consecutive loads. + if (Is32Bit && !isFPExtLoad(NextOp.getOperand(0).getOperand(0))) + return SDValue(); if (N->getOperand(i) != FirstInput) IsSplat = false; } @@ -11813,8 +11828,9 @@ combineElementTruncationToVectorTruncation(SDNode *N, // Now that we know we have the right type of node, get its operands for (int i = 0, e = N->getNumOperands(); i < e; ++i) { SDValue In = N->getOperand(i).getOperand(0); - // For 32-bit values, we need to add an FP_ROUND node. if (Is32Bit) { + // For 32-bit values, we need to add an FP_ROUND node (if we made it + // here, we know that all inputs are extending loads so this is safe). if (In.isUndef()) Ops.push_back(DAG.getUNDEF(SrcVT)); else { Modified: vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCInstrVSX.td ============================================================================== --- vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCInstrVSX.td Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/lib/Target/PowerPC/PPCInstrVSX.td Sat Aug 4 13:15:05 2018 (r337299) @@ -3494,6 +3494,17 @@ def DblToFlt { dag B1 = (f32 (fpround (f64 (extractelt v2f64:$B, 1)))); } +def ExtDbl { + dag A0S = (i32 (PPCmfvsr (f64 (PPCfctiwz (f64 (extractelt v2f64:$A, 0)))))); + dag A1S = (i32 (PPCmfvsr (f64 (PPCfctiwz (f64 (extractelt v2f64:$A, 1)))))); + dag B0S = (i32 (PPCmfvsr (f64 (PPCfctiwz (f64 (extractelt v2f64:$B, 0)))))); + dag B1S = (i32 (PPCmfvsr (f64 (PPCfctiwz (f64 (extractelt v2f64:$B, 1)))))); + dag A0U = (i32 (PPCmfvsr (f64 (PPCfctiwuz (f64 (extractelt v2f64:$A, 0)))))); + dag A1U = (i32 (PPCmfvsr (f64 (PPCfctiwuz (f64 (extractelt v2f64:$A, 1)))))); + dag B0U = (i32 (PPCmfvsr (f64 (PPCfctiwuz (f64 (extractelt v2f64:$B, 0)))))); + dag B1U = (i32 (PPCmfvsr (f64 (PPCfctiwuz (f64 (extractelt v2f64:$B, 1)))))); +} + def ByteToWord { dag LE_A0 = (i32 (sext_inreg (i32 (vector_extract v16i8:$A, 0)), i8)); dag LE_A1 = (i32 (sext_inreg (i32 (vector_extract v16i8:$A, 4)), i8)); @@ -3571,9 +3582,15 @@ def FltToULong { } def DblToInt { dag A = (i32 (PPCmfvsr (f64 (PPCfctiwz f64:$A)))); + dag B = (i32 (PPCmfvsr (f64 (PPCfctiwz f64:$B)))); + dag C = (i32 (PPCmfvsr (f64 (PPCfctiwz f64:$C)))); + dag D = (i32 (PPCmfvsr (f64 (PPCfctiwz f64:$D)))); } def DblToUInt { dag A = (i32 (PPCmfvsr (f64 (PPCfctiwuz f64:$A)))); + dag B = (i32 (PPCmfvsr (f64 (PPCfctiwuz f64:$B)))); + dag C = (i32 (PPCmfvsr (f64 (PPCfctiwuz f64:$C)))); + dag D = (i32 (PPCmfvsr (f64 (PPCfctiwuz f64:$D)))); } def DblToLong { dag A = (i64 (PPCmfvsr (f64 (PPCfctidz f64:$A)))); @@ -3612,6 +3629,47 @@ def MrgFP { dag BAlToFlt = (XVCVDPSP (XXPERMDI $B, $A, 3)); } +// Word-element merge dags - conversions from f64 to i32 merged into vectors. +def MrgWords { + // For big endian, we merge low and hi doublewords (A, B). + dag A0B0 = (v2f64 (XXPERMDI v2f64:$A, v2f64:$B, 0)); + dag A1B1 = (v2f64 (XXPERMDI v2f64:$A, v2f64:$B, 3)); + dag CVA1B1S = (v4i32 (XVCVDPSXWS A1B1)); + dag CVA0B0S = (v4i32 (XVCVDPSXWS A0B0)); + dag CVA1B1U = (v4i32 (XVCVDPUXWS A1B1)); + dag CVA0B0U = (v4i32 (XVCVDPUXWS A0B0)); + + // For little endian, we merge low and hi doublewords (B, A). + dag B1A1 = (v2f64 (XXPERMDI v2f64:$B, v2f64:$A, 0)); + dag B0A0 = (v2f64 (XXPERMDI v2f64:$B, v2f64:$A, 3)); + dag CVB1A1S = (v4i32 (XVCVDPSXWS B1A1)); + dag CVB0A0S = (v4i32 (XVCVDPSXWS B0A0)); + dag CVB1A1U = (v4i32 (XVCVDPUXWS B1A1)); + dag CVB0A0U = (v4i32 (XVCVDPUXWS B0A0)); + + // For big endian, we merge hi doublewords of (A, C) and (B, D), convert + // then merge. + dag AC = (v2f64 (XXPERMDI (COPY_TO_REGCLASS f64:$A, VSRC), + (COPY_TO_REGCLASS f64:$C, VSRC), 0)); + dag BD = (v2f64 (XXPERMDI (COPY_TO_REGCLASS f64:$B, VSRC), + (COPY_TO_REGCLASS f64:$D, VSRC), 0)); + dag CVACS = (v4i32 (XVCVDPSXWS AC)); + dag CVBDS = (v4i32 (XVCVDPSXWS BD)); + dag CVACU = (v4i32 (XVCVDPUXWS AC)); + dag CVBDU = (v4i32 (XVCVDPUXWS BD)); + + // For little endian, we merge hi doublewords of (D, B) and (C, A), convert + // then merge. + dag DB = (v2f64 (XXPERMDI (COPY_TO_REGCLASS f64:$D, VSRC), + (COPY_TO_REGCLASS f64:$B, VSRC), 0)); + dag CA = (v2f64 (XXPERMDI (COPY_TO_REGCLASS f64:$C, VSRC), + (COPY_TO_REGCLASS f64:$A, VSRC), 0)); + dag CVDBS = (v4i32 (XVCVDPSXWS DB)); + dag CVCAS = (v4i32 (XVCVDPSXWS CA)); + dag CVDBU = (v4i32 (XVCVDPUXWS DB)); + dag CVCAU = (v4i32 (XVCVDPUXWS CA)); +} + // Patterns for BUILD_VECTOR nodes. let AddedComplexity = 400 in { @@ -3679,6 +3737,20 @@ let AddedComplexity = 400 in { def : Pat<(v4f32 (build_vector DblToFlt.A0, DblToFlt.A1, DblToFlt.B0, DblToFlt.B1)), (v4f32 (VMRGEW MrgFP.ABhToFlt, MrgFP.ABlToFlt))>; + + // Convert 4 doubles to a vector of ints. + def : Pat<(v4i32 (build_vector DblToInt.A, DblToInt.B, + DblToInt.C, DblToInt.D)), + (v4i32 (VMRGEW MrgWords.CVACS, MrgWords.CVBDS))>; + def : Pat<(v4i32 (build_vector DblToUInt.A, DblToUInt.B, + DblToUInt.C, DblToUInt.D)), + (v4i32 (VMRGEW MrgWords.CVACU, MrgWords.CVBDU))>; + def : Pat<(v4i32 (build_vector ExtDbl.A0S, ExtDbl.A1S, + ExtDbl.B0S, ExtDbl.B1S)), + (v4i32 (VMRGEW MrgWords.CVA0B0S, MrgWords.CVA1B1S))>; + def : Pat<(v4i32 (build_vector ExtDbl.A0U, ExtDbl.A1U, + ExtDbl.B0U, ExtDbl.B1U)), + (v4i32 (VMRGEW MrgWords.CVA0B0U, MrgWords.CVA1B1U))>; } let Predicates = [IsLittleEndian, HasVSX] in { @@ -3693,6 +3765,20 @@ let AddedComplexity = 400 in { def : Pat<(v4f32 (build_vector DblToFlt.A0, DblToFlt.A1, DblToFlt.B0, DblToFlt.B1)), (v4f32 (VMRGEW MrgFP.BAhToFlt, MrgFP.BAlToFlt))>; + + // Convert 4 doubles to a vector of ints. + def : Pat<(v4i32 (build_vector DblToInt.A, DblToInt.B, + DblToInt.C, DblToInt.D)), + (v4i32 (VMRGEW MrgWords.CVDBS, MrgWords.CVCAS))>; + def : Pat<(v4i32 (build_vector DblToUInt.A, DblToUInt.B, + DblToUInt.C, DblToUInt.D)), + (v4i32 (VMRGEW MrgWords.CVDBU, MrgWords.CVCAU))>; + def : Pat<(v4i32 (build_vector ExtDbl.A0S, ExtDbl.A1S, + ExtDbl.B0S, ExtDbl.B1S)), + (v4i32 (VMRGEW MrgWords.CVB1A1S, MrgWords.CVB0A0S))>; + def : Pat<(v4i32 (build_vector ExtDbl.A0U, ExtDbl.A1U, + ExtDbl.B0U, ExtDbl.B1U)), + (v4i32 (VMRGEW MrgWords.CVB1A1U, MrgWords.CVB0A0U))>; } let Predicates = [HasDirectMove] in { Modified: vendor/llvm/dist-release_70/lib/Target/X86/X86FastISel.cpp ============================================================================== --- vendor/llvm/dist-release_70/lib/Target/X86/X86FastISel.cpp Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/lib/Target/X86/X86FastISel.cpp Sat Aug 4 13:15:05 2018 (r337299) @@ -738,6 +738,10 @@ bool X86FastISel::handleConstantAddresses(const Value if (GV->isThreadLocal()) return false; + // Can't handle !absolute_symbol references yet. + if (GV->isAbsoluteSymbolRef()) + return false; + // RIP-relative addresses can't have additional register operands, so if // we've already folded stuff into the addressing mode, just force the // global value into its own register, which we can use as the basereg. Modified: vendor/llvm/dist-release_70/test/CodeGen/AArch64/f16-instructions.ll ============================================================================== --- vendor/llvm/dist-release_70/test/CodeGen/AArch64/f16-instructions.ll Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/test/CodeGen/AArch64/f16-instructions.ll Sat Aug 4 13:15:05 2018 (r337299) @@ -456,6 +456,36 @@ define i1 @test_fcmp_ord(half %a, half %b) #0 { ret i1 %r } +; CHECK-COMMON-LABEL: test_fccmp: +; CHECK-CVT: fcvt s0, h0 +; CHECK-CVT-NEXT: fmov s1, #8.00000000 +; CHECK-CVT-NEXT: fmov s2, #5.00000000 +; CHECK-CVT-NEXT: fcmp s0, s1 +; CHECK-CVT-NEXT: cset w8, gt +; CHECK-CVT-NEXT: fcmp s0, s2 +; CHECK-CVT-NEXT: cset w9, mi +; CHECK-CVT-NEXT: tst w8, w9 +; CHECK-CVT-NEXT: fcsel s0, s0, s2, ne +; CHECK-CVT-NEXT: fcvt h0, s0 +; CHECK-CVT-NEXT: str h0, [x0] +; CHECK-CVT-NEXT: ret +; CHECK-FP16: fmov h1, #5.00000000 +; CHECK-FP16-NEXT: fcmp h0, h1 +; CHECK-FP16-NEXT: fmov h2, #8.00000000 +; CHECK-FP16-NEXT: fccmp h0, h2, #4, mi +; CHECK-FP16-NEXT: fcsel h0, h0, h1, gt +; CHECK-FP16-NEXT: str h0, [x0] +; CHECK-FP16-NEXT: ret + +define void @test_fccmp(half %in, half* %out) { + %cmp1 = fcmp ogt half %in, 0xH4800 + %cmp2 = fcmp olt half %in, 0xH4500 + %cond = and i1 %cmp1, %cmp2 + %result = select i1 %cond, half %in, half 0xH4500 + store half %result, half* %out + ret void +} + ; CHECK-CVT-LABEL: test_br_cc: ; CHECK-CVT-NEXT: fcvt s1, h1 ; CHECK-CVT-NEXT: fcvt s0, h0 Modified: vendor/llvm/dist-release_70/test/CodeGen/PowerPC/build-vector-tests.ll ============================================================================== --- vendor/llvm/dist-release_70/test/CodeGen/PowerPC/build-vector-tests.ll Sat Aug 4 13:08:36 2018 (r337298) +++ vendor/llvm/dist-release_70/test/CodeGen/PowerPC/build-vector-tests.ll Sat Aug 4 13:15:05 2018 (r337299) @@ -119,8 +119,8 @@ ;vector int spltCnstConvftoi() { // ; return (vector int) 4.74f; // ;} // -;// P8: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // -;// P9: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvdpsxws // +;// P8: 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // ;vector int fromRegsConvftoi(float a, float b, float c, float d) { // ; return (vector int) { a, b, c, d }; // ;} // @@ -139,15 +139,15 @@ ;vector int fromDiffMemConsDConvftoi(float *ptr) { // ; return (vector int) { ptr[3], ptr[2], ptr[1], ptr[0] }; // ;} // -;// P8: 4 x lxsspx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // -;// P9: 4 x lxssp, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // +;// P8: 4 x lxsspx, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: 4 x lxssp, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // ;// Note: if the consecutive loads learns to handle pre-inc, this can be: // ;// sldi 2, load, xvcvspuxws // ;vector int fromDiffMemVarAConvftoi(float *arr, int elem) { // ; return (vector int) { arr[elem], arr[elem+1], arr[elem+2], arr[elem+3] }; // ;} // -;// P8: 4 x lxsspx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // -;// P9: 4 x lxssp, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // +;// P8: 4 x lxsspx, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: 4 x lxssp, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // ;// Note: if the consecutive loads learns to handle pre-inc, this can be: // ;// sldi 2, 2 x load, vperm, xvcvspuxws // ;vector int fromDiffMemVarDConvftoi(float *arr, int elem) { // @@ -168,8 +168,8 @@ ;vector int spltCnstConvdtoi() { // ; return (vector int) 4.74; // ;} // -;// P8: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // -;// P9: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // +;// P8: 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // ;vector int fromRegsConvdtoi(double a, double b, double c, double d) { // ; return (vector int) { a, b, c, d }; // ;} // @@ -178,25 +178,23 @@ ;vector int fromDiffConstsConvdtoi() { // ; return (vector int) { 24.46, 234., 988.19, 422.39 }; // ;} // -;// P8: 2 x lxvd2x, 2 x xxswapd, xxmrgld, xxmrghd, 2 x xvcvdpsp, vmrgew, // -;// xvcvspsxws // -;// P9: 2 x lxvx, 2 x xxswapd, xxmrgld, xxmrghd, 2 x xvcvdpsp, vmrgew, // -;// xvcvspsxws // +;// P8: 2 x lxvd2x, 2 x xxswapd, xxmrgld, xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: 2 x lxvx, 2 x xxswapd, xxmrgld, xxmrghd, 2 x xvcvspsxws, vmrgew // ;vector int fromDiffMemConsAConvdtoi(double *ptr) { // ; return (vector int) { ptr[0], ptr[1], ptr[2], ptr[3] }; // ;} // -;// P8: 4 x lxsdx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // -;// P9: 4 x lfd, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // +;// P8: 4 x lxsdx, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: 4 x lfd, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // ;vector int fromDiffMemConsDConvdtoi(double *ptr) { // ; return (vector int) { ptr[3], ptr[2], ptr[1], ptr[0] }; // ;} // -;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // -;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // +;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // ;vector int fromDiffMemVarAConvdtoi(double *arr, int elem) { // ; return (vector int) { arr[elem], arr[elem+1], arr[elem+2], arr[elem+3] }; // ;} // -;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // -;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspsxws // +;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // +;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvspsxws, vmrgew // ;vector int fromDiffMemVarDConvdtoi(double *arr, int elem) { // ; return (vector int) { arr[elem], arr[elem-1], arr[elem-2], arr[elem-3] }; // ;} // @@ -296,8 +294,8 @@ ;vector unsigned int spltCnstConvftoui() { // ; return (vector unsigned int) 4.74f; // ;} // -;// P8: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // -;// P9: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // ;vector unsigned int fromRegsConvftoui(float a, float b, float c, float d) { // ; return (vector unsigned int) { a, b, c, d }; // ;} // @@ -316,16 +314,16 @@ ;vector unsigned int fromDiffMemConsDConvftoui(float *ptr) { // ; return (vector unsigned int) { ptr[3], ptr[2], ptr[1], ptr[0] }; // ;} // -;// P8: lfsux, 3 x lxsspx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // -;// P9: lfsux, 3 x lfs, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: lfsux, 3 x lxsspx, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: lfsux, 3 x lfs, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // ;// Note: if the consecutive loads learns to handle pre-inc, this can be: // ;// sldi 2, load, xvcvspuxws // ;vector unsigned int fromDiffMemVarAConvftoui(float *arr, int elem) { // ; return (vector unsigned int) { arr[elem], arr[elem+1], // ; arr[elem+2], arr[elem+3] }; // ;} // -;// P8: lfsux, 3 x lxsspx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // -;// P9: lfsux, 3 x lfs, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: lfsux, 3 x lxsspx, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: lfsux, 3 x lfs, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // ;// Note: if the consecutive loads learns to handle pre-inc, this can be: // ;// sldi 2, 2 x load, vperm, xvcvspuxws // ;vector unsigned int fromDiffMemVarDConvftoui(float *arr, int elem) { // @@ -347,8 +345,8 @@ ;vector unsigned int spltCnstConvdtoui() { // ; return (vector unsigned int) 4.74; // ;} // -;// P8: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // -;// P9: 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // ;vector unsigned int fromRegsConvdtoui(double a, double b, // ; double c, double d) { // ; return (vector unsigned int) { a, b, c, d }; // @@ -358,25 +356,24 @@ ;vector unsigned int fromDiffConstsConvdtoui() { // ; return (vector unsigned int) { 24.46, 234., 988.19, 422.39 }; // ;} // -;// P8: 2 x lxvd2x, 2 x xxswapd, xxmrgld, xxmrghd, 2 x xvcvdpsp, vmrgew, // -;// xvcvspuxws // -;// P9: 2 x lxvx, xxmrgld, xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: 2 x lxvd2x, 2 x xxswapd, xxmrgld, xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: 2 x lxvx, xxmrgld, xxmrghd, 2 x xvcvspuxws, vmrgew // ;vector unsigned int fromDiffMemConsAConvdtoui(double *ptr) { // ; return (vector unsigned int) { ptr[0], ptr[1], ptr[2], ptr[3] }; // ;} // -;// P8: 4 x lxsdx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // -;// P9: 4 x lfd, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: 4 x lxsdx, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: 4 x lfd, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // ;vector unsigned int fromDiffMemConsDConvdtoui(double *ptr) { // ; return (vector unsigned int) { ptr[3], ptr[2], ptr[1], ptr[0] }; // ;} // -;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // -;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // ;vector unsigned int fromDiffMemVarAConvdtoui(double *arr, int elem) { // ; return (vector unsigned int) { arr[elem], arr[elem+1], // ; arr[elem+2], arr[elem+3] }; // ;} // -;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // -;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvdpsp, vmrgew, xvcvspuxws // +;// P8: lfdux, 3 x lxsdx, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // +;// P9: lfdux, 3 x lfd, 2 x xxmrghd, 2 x xvcvspuxws, vmrgew // ;vector unsigned int fromDiffMemVarDConvdtoui(double *arr, int elem) { // ; return (vector unsigned int) { arr[elem], arr[elem-1], // ; arr[elem-2], arr[elem-3] }; // @@ -1253,28 +1250,24 @@ entry: ; P8LE-LABEL: fromRegsConvftoi ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P9BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9BE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9BE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9BE: vmrgew v2, [[REG3]], [[REG4]] -; P9BE: xvcvspsxws v2, v2 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P9LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9LE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9LE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9LE: vmrgew v2, [[REG4]], [[REG3]] -; P9LE: xvcvspsxws v2, v2 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P8BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8BE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8BE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8BE: vmrgew v2, [[REG3]], [[REG4]] -; P8BE: xvcvspsxws v2, v2 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P8LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8LE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8LE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8LE: vmrgew v2, [[REG4]], [[REG3]] -; P8LE: xvcvspsxws v2, v2 } ; Function Attrs: norecurse nounwind readnone @@ -1529,28 +1522,24 @@ entry: ; P8LE-LABEL: fromRegsConvdtoi ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P9BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9BE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9BE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9BE: vmrgew v2, [[REG3]], [[REG4]] -; P9BE: xvcvspsxws v2, v2 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P9LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9LE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9LE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9LE: vmrgew v2, [[REG4]], [[REG3]] -; P9LE: xvcvspsxws v2, v2 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P8BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8BE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8BE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8BE: vmrgew v2, [[REG3]], [[REG4]] -; P8BE: xvcvspsxws v2, v2 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P8LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8LE-DAG: xvcvdpsxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8LE-DAG: xvcvdpsxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8LE: vmrgew v2, [[REG4]], [[REG3]] -; P8LE: xvcvspsxws v2, v2 } ; Function Attrs: norecurse nounwind readnone @@ -1592,36 +1581,32 @@ entry: ; P9BE-DAG: lxv [[REG2:[vs0-9]+]], 16(r3) ; P9BE-DAG: xxmrgld [[REG3:[vs0-9]+]], [[REG1]], [[REG2]] ; P9BE-DAG: xxmrghd [[REG4:[vs0-9]+]], [[REG1]], [[REG2]] -; P9BE-DAG: xvcvdpsp [[REG5:[vs0-9]+]], [[REG3]] -; P9BE-DAG: xvcvdpsp [[REG6:[vs0-9]+]], [[REG4]] +; P9BE-DAG: xvcvdpsxws [[REG5:[vs0-9]+]], [[REG3]] +; P9BE-DAG: xvcvdpsxws [[REG6:[vs0-9]+]], [[REG4]] ; P9BE: vmrgew v2, [[REG6]], [[REG5]] -; P9BE: xvcvspsxws v2, v2 ; P9LE-DAG: lxv [[REG1:[vs0-9]+]], 0(r3) ; P9LE-DAG: lxv [[REG2:[vs0-9]+]], 16(r3) ; P9LE-DAG: xxmrgld [[REG3:[vs0-9]+]], [[REG2]], [[REG1]] ; P9LE-DAG: xxmrghd [[REG4:[vs0-9]+]], [[REG2]], [[REG1]] -; P9LE-DAG: xvcvdpsp [[REG5:[vs0-9]+]], [[REG3]] -; P9LE-DAG: xvcvdpsp [[REG6:[vs0-9]+]], [[REG4]] +; P9LE-DAG: xvcvdpsxws [[REG5:[vs0-9]+]], [[REG3]] +; P9LE-DAG: xvcvdpsxws [[REG6:[vs0-9]+]], [[REG4]] ; P9LE: vmrgew v2, [[REG6]], [[REG5]] -; P9LE: xvcvspsxws v2, v2 ; P8BE: lxvd2x [[REG1:[vs0-9]+]], 0, r3 ; P8BE: lxvd2x [[REG2:[vs0-9]+]], r3, r4 ; P8BE-DAG: xxmrgld [[REG3:[vs0-9]+]], [[REG1]], [[REG2]] ; P8BE-DAG: xxmrghd [[REG4:[vs0-9]+]], [[REG1]], [[REG2]] -; P8BE-DAG: xvcvdpsp [[REG5:[vs0-9]+]], [[REG3]] -; P8BE-DAG: xvcvdpsp [[REG6:[vs0-9]+]], [[REG4]] +; P8BE-DAG: xvcvdpsxws [[REG5:[vs0-9]+]], [[REG3]] +; P8BE-DAG: xvcvdpsxws [[REG6:[vs0-9]+]], [[REG4]] ; P8BE: vmrgew v2, [[REG6]], [[REG5]] -; P8BE: xvcvspsxws v2, v2 ; P8LE: lxvd2x [[REG1:[vs0-9]+]], 0, r3 ; P8LE: lxvd2x [[REG2:[vs0-9]+]], r3, r4 ; P8LE-DAG: xxswapd [[REG3:[vs0-9]+]], [[REG1]] ; P8LE-DAG: xxswapd [[REG4:[vs0-9]+]], [[REG2]] ; P8LE-DAG: xxmrgld [[REG5:[vs0-9]+]], [[REG4]], [[REG3]] ; P8LE-DAG: xxmrghd [[REG6:[vs0-9]+]], [[REG4]], [[REG3]] -; P8LE-DAG: xvcvdpsp [[REG7:[vs0-9]+]], [[REG5]] -; P8LE-DAG: xvcvdpsp [[REG8:[vs0-9]+]], [[REG6]] +; P8LE-DAG: xvcvdpsxws [[REG7:[vs0-9]+]], [[REG5]] +; P8LE-DAG: xvcvdpsxws [[REG8:[vs0-9]+]], [[REG6]] ; P8LE: vmrgew v2, [[REG8]], [[REG7]] -; P8LE: xvcvspsxws v2, v2 } ; Function Attrs: norecurse nounwind readonly @@ -1653,40 +1638,36 @@ entry: ; P9BE: lfd ; P9BE: xxmrghd ; P9BE: xxmrghd -; P9BE: xvcvdpsp -; P9BE: xvcvdpsp -; P9BE: vmrgew -; P9BE: xvcvspsxws v2 +; P9BE: xvcvdpsxws +; P9BE: xvcvdpsxws +; P9BE: vmrgew v2 ; P9LE: lfd ; P9LE: lfd ; P9LE: lfd ; P9LE: lfd ; P9LE: xxmrghd ; P9LE: xxmrghd -; P9LE: xvcvdpsp -; P9LE: xvcvdpsp -; P9LE: vmrgew -; P9LE: xvcvspsxws v2 +; P9LE: xvcvdpsxws +; P9LE: xvcvdpsxws +; P9LE: vmrgew v2 ; P8BE: lfdx ; P8BE: lfd ; P8BE: lfd ; P8BE: lfd ; P8BE: xxmrghd ; P8BE: xxmrghd -; P8BE: xvcvdpsp -; P8BE: xvcvdpsp -; P8BE: vmrgew -; P8BE: xvcvspsxws v2 +; P8BE: xvcvdpsxws +; P8BE: xvcvdpsxws +; P8BE: vmrgew v2 ; P8LE: lfdx ; P8LE: lfd ; P8LE: lfd ; P8LE: lfd ; P8LE: xxmrghd ; P8LE: xxmrghd -; P8LE: xvcvdpsp -; P8LE: xvcvdpsp -; P8LE: vmrgew -; P8LE: xvcvspsxws v2 +; P8LE: xvcvdpsxws +; P8LE: xvcvdpsxws +; P8LE: vmrgew v2 } ; Function Attrs: norecurse nounwind readonly @@ -1726,40 +1707,36 @@ entry: ; P9BE: lfd ; P9BE: xxmrghd ; P9BE: xxmrghd -; P9BE: xvcvdpsp -; P9BE: xvcvdpsp -; P9BE: vmrgew -; P9BE: xvcvspsxws v2 +; P9BE: xvcvdpsxws +; P9BE: xvcvdpsxws +; P9BE: vmrgew v2 ; P9LE: lfdux ; P9LE: lfd ; P9LE: lfd ; P9LE: lfd ; P9LE: xxmrghd ; P9LE: xxmrghd -; P9LE: xvcvdpsp -; P9LE: xvcvdpsp -; P9LE: vmrgew -; P9LE: xvcvspsxws v2 +; P9LE: xvcvdpsxws +; P9LE: xvcvdpsxws +; P9LE: vmrgew v2 ; P8BE: lfdux ; P8BE: lfd ; P8BE: lfd ; P8BE: lfd ; P8BE: xxmrghd ; P8BE: xxmrghd -; P8BE: xvcvdpsp -; P8BE: xvcvdpsp -; P8BE: vmrgew -; P8BE: xvcvspsxws v2 +; P8BE: xvcvdpsxws +; P8BE: xvcvdpsxws +; P8BE: vmrgew v2 ; P8LE: lfdux ; P8LE: lfd ; P8LE: lfd ; P8LE: lfd ; P8LE: xxmrghd ; P8LE: xxmrghd -; P8LE: xvcvdpsp -; P8LE: xvcvdpsp -; P8LE: vmrgew -; P8LE: xvcvspsxws v2 +; P8LE: xvcvdpsxws +; P8LE: xvcvdpsxws +; P8LE: vmrgew v2 } ; Function Attrs: norecurse nounwind readonly @@ -1799,40 +1776,36 @@ entry: ; P9BE: lfd ; P9BE: xxmrghd ; P9BE: xxmrghd -; P9BE: xvcvdpsp -; P9BE: xvcvdpsp -; P9BE: vmrgew -; P9BE: xvcvspsxws v2 +; P9BE: xvcvdpsxws +; P9BE: xvcvdpsxws +; P9BE: vmrgew v2 ; P9LE: lfdux ; P9LE: lfd ; P9LE: lfd ; P9LE: lfd ; P9LE: xxmrghd ; P9LE: xxmrghd -; P9LE: xvcvdpsp -; P9LE: xvcvdpsp -; P9LE: vmrgew -; P9LE: xvcvspsxws v2 +; P9LE: xvcvdpsxws +; P9LE: xvcvdpsxws +; P9LE: vmrgew v2 ; P8BE: lfdux ; P8BE: lfd ; P8BE: lfd ; P8BE: lfd ; P8BE: xxmrghd ; P8BE: xxmrghd -; P8BE: xvcvdpsp -; P8BE: xvcvdpsp -; P8BE: vmrgew -; P8BE: xvcvspsxws v2 +; P8BE: xvcvdpsxws +; P8BE: xvcvdpsxws +; P8BE: vmrgew v2 ; P8LE: lfdux ; P8LE: lfd ; P8LE: lfd ; P8LE: lfd ; P8LE: xxmrghd ; P8LE: xxmrghd -; P8LE: xvcvdpsp -; P8LE: xvcvdpsp -; P8LE: vmrgew -; P8LE: xvcvspsxws v2 +; P8LE: xvcvdpsxws +; P8LE: xvcvdpsxws +; P8LE: vmrgew v2 } ; Function Attrs: norecurse nounwind readnone @@ -2413,28 +2386,24 @@ entry: ; P8LE-LABEL: fromRegsConvftoui ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P9BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9BE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9BE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9BE: vmrgew v2, [[REG3]], [[REG4]] -; P9BE: xvcvspuxws v2, v2 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P9LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9LE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9LE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9LE: vmrgew v2, [[REG4]], [[REG3]] -; P9LE: xvcvspuxws v2, v2 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P8BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8BE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8BE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8BE: vmrgew v2, [[REG3]], [[REG4]] -; P8BE: xvcvspuxws v2, v2 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P8LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8LE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8LE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8LE: vmrgew v2, [[REG4]], [[REG3]] -; P8LE: xvcvspuxws v2, v2 } ; Function Attrs: norecurse nounwind readnone @@ -2689,28 +2658,24 @@ entry: ; P8LE-LABEL: fromRegsConvdtoui ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P9BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P9BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9BE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9BE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9BE: vmrgew v2, [[REG3]], [[REG4]] -; P9BE: xvcvspuxws v2, v2 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P9LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P9LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P9LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P9LE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P9LE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P9LE: vmrgew v2, [[REG4]], [[REG3]] -; P9LE: xvcvspuxws v2, v2 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs1, vs3 ; P8BE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs2, vs4 -; P8BE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8BE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8BE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8BE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8BE: vmrgew v2, [[REG3]], [[REG4]] -; P8BE: xvcvspuxws v2, v2 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG1:[0-9]+]], vs3, vs1 ; P8LE-DAG: xxmrghd {{[vs]+}}[[REG2:[0-9]+]], vs4, vs2 -; P8LE-DAG: xvcvdpsp [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] -; P8LE-DAG: xvcvdpsp [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] +; P8LE-DAG: xvcvdpuxws [[REG3:v[0-9]+]], {{[vs]+}}[[REG1]] +; P8LE-DAG: xvcvdpuxws [[REG4:v[0-9]+]], {{[vs]+}}[[REG2]] ; P8LE: vmrgew v2, [[REG4]], [[REG3]] -; P8LE: xvcvspuxws v2, v2 } ; Function Attrs: norecurse nounwind readnone @@ -2752,36 +2717,32 @@ entry: ; P9BE-DAG: lxv [[REG2:[vs0-9]+]], 16(r3) ; P9BE-DAG: xxmrgld [[REG3:[vs0-9]+]], [[REG1]], [[REG2]] ; P9BE-DAG: xxmrghd [[REG4:[vs0-9]+]], [[REG1]], [[REG2]] -; P9BE-DAG: xvcvdpsp [[REG5:[vs0-9]+]], [[REG3]] -; P9BE-DAG: xvcvdpsp [[REG6:[vs0-9]+]], [[REG4]] +; P9BE-DAG: xvcvdpuxws [[REG5:[vs0-9]+]], [[REG3]] +; P9BE-DAG: xvcvdpuxws [[REG6:[vs0-9]+]], [[REG4]] ; P9BE: vmrgew v2, [[REG6]], [[REG5]] -; P9BE: xvcvspuxws v2, v2 ; P9LE-DAG: lxv [[REG1:[vs0-9]+]], 0(r3) ; P9LE-DAG: lxv [[REG2:[vs0-9]+]], 16(r3) -; P9LE-DAG: xxmrgld [[REG3:[vs0-9]+]], [[REG2]], [[REG1]] ; P9LE-DAG: xxmrghd [[REG4:[vs0-9]+]], [[REG2]], [[REG1]] -; P9LE-DAG: xvcvdpsp [[REG5:[vs0-9]+]], [[REG3]] -; P9LE-DAG: xvcvdpsp [[REG6:[vs0-9]+]], [[REG4]] +; P9LE-DAG: xxmrgld [[REG3:[vs0-9]+]], [[REG2]], [[REG1]] +; P9LE-DAG: xvcvdpuxws [[REG5:[vs0-9]+]], [[REG3]] +; P9LE-DAG: xvcvdpuxws [[REG6:[vs0-9]+]], [[REG4]] ; P9LE: vmrgew v2, [[REG6]], [[REG5]] -; P9LE: xvcvspuxws v2, v2 ; P8BE: lxvd2x [[REG1:[vs0-9]+]], 0, r3 ; P8BE: lxvd2x [[REG2:[vs0-9]+]], r3, r4 ; P8BE-DAG: xxmrgld [[REG3:[vs0-9]+]], [[REG1]], [[REG2]] ; P8BE-DAG: xxmrghd [[REG4:[vs0-9]+]], [[REG1]], [[REG2]] -; P8BE-DAG: xvcvdpsp [[REG5:[vs0-9]+]], [[REG3]] -; P8BE-DAG: xvcvdpsp [[REG6:[vs0-9]+]], [[REG4]] +; P8BE-DAG: xvcvdpuxws [[REG5:[vs0-9]+]], [[REG3]] +; P8BE-DAG: xvcvdpuxws [[REG6:[vs0-9]+]], [[REG4]] ; P8BE: vmrgew v2, [[REG6]], [[REG5]] -; P8BE: xvcvspuxws v2, v2 ; P8LE: lxvd2x [[REG1:[vs0-9]+]], 0, r3 ; P8LE: lxvd2x [[REG2:[vs0-9]+]], r3, r4 ; P8LE-DAG: xxswapd [[REG3:[vs0-9]+]], [[REG1]] ; P8LE-DAG: xxswapd [[REG4:[vs0-9]+]], [[REG2]] ; P8LE-DAG: xxmrgld [[REG5:[vs0-9]+]], [[REG4]], [[REG3]] ; P8LE-DAG: xxmrghd [[REG6:[vs0-9]+]], [[REG4]], [[REG3]] -; P8LE-DAG: xvcvdpsp [[REG7:[vs0-9]+]], [[REG5]] -; P8LE-DAG: xvcvdpsp [[REG8:[vs0-9]+]], [[REG6]] +; P8LE-DAG: xvcvdpuxws [[REG7:[vs0-9]+]], [[REG5]] +; P8LE-DAG: xvcvdpuxws [[REG8:[vs0-9]+]], [[REG6]] ; P8LE: vmrgew v2, [[REG8]], [[REG7]] -; P8LE: xvcvspuxws v2, v2 } ; Function Attrs: norecurse nounwind readonly @@ -2813,40 +2774,36 @@ entry: ; P9BE: lfd ; P9BE: xxmrghd ; P9BE: xxmrghd -; P9BE: xvcvdpsp -; P9BE: xvcvdpsp -; P9BE: vmrgew -; P9BE: xvcvspuxws v2 +; P9BE: xvcvdpuxws +; P9BE: xvcvdpuxws +; P9BE: vmrgew v2 ; P9LE: lfd ; P9LE: lfd ; P9LE: lfd ; P9LE: lfd ; P9LE: xxmrghd ; P9LE: xxmrghd -; P9LE: xvcvdpsp -; P9LE: xvcvdpsp -; P9LE: vmrgew -; P9LE: xvcvspuxws v2 +; P9LE: xvcvdpuxws +; P9LE: xvcvdpuxws +; P9LE: vmrgew v2 ; P8BE: lfdx ; P8BE: lfd ; P8BE: lfd ; P8BE: lfd ; P8BE: xxmrghd ; P8BE: xxmrghd -; P8BE: xvcvdpsp -; P8BE: xvcvdpsp -; P8BE: vmrgew -; P8BE: xvcvspuxws v2 +; P8BE: xvcvdpuxws +; P8BE: xvcvdpuxws +; P8BE: vmrgew v2 ; P8LE: lfdx ; P8LE: lfd ; P8LE: lfd ; P8LE: lfd ; P8LE: xxmrghd ; P8LE: xxmrghd -; P8LE: xvcvdpsp -; P8LE: xvcvdpsp -; P8LE: vmrgew -; P8LE: xvcvspuxws v2 +; P8LE: xvcvdpuxws +; P8LE: xvcvdpuxws +; P8LE: vmrgew v2 } ; Function Attrs: norecurse nounwind readonly @@ -2886,40 +2843,36 @@ entry: ; P9BE: lfd ; P9BE: xxmrghd ; P9BE: xxmrghd -; P9BE: xvcvdpsp -; P9BE: xvcvdpsp -; P9BE: vmrgew -; P9BE: xvcvspuxws v2 +; P9BE: xvcvdpuxws +; P9BE: xvcvdpuxws +; P9BE: vmrgew v2 ; P9LE: lfdux ; P9LE: lfd ; P9LE: lfd ; P9LE: lfd ; P9LE: xxmrghd ; P9LE: xxmrghd -; P9LE: xvcvdpsp -; P9LE: xvcvdpsp -; P9LE: vmrgew -; P9LE: xvcvspuxws v2 +; P9LE: xvcvdpuxws +; P9LE: xvcvdpuxws +; P9LE: vmrgew v2 ; P8BE: lfdux ; P8BE: lfd ; P8BE: lfd ; P8BE: lfd ; P8BE: xxmrghd ; P8BE: xxmrghd -; P8BE: xvcvdpsp -; P8BE: xvcvdpsp -; P8BE: vmrgew *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:15:12 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A105106AE92; Sat, 4 Aug 2018 13:15:12 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B295C75C3C; Sat, 4 Aug 2018 13:15:11 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 7CAA17F5C; Sat, 4 Aug 2018 13:15:11 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DFBLQ083305; Sat, 4 Aug 2018 13:15:11 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DFBC6083304; Sat, 4 Aug 2018 13:15:11 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041315.w74DFBC6083304@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:15:11 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337300 - vendor/llvm/llvm-release_70-r338892 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/llvm/llvm-release_70-r338892 X-SVN-Commit-Revision: 337300 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:15:12 -0000 Author: dim Date: Sat Aug 4 13:15:11 2018 New Revision: 337300 URL: https://svnweb.freebsd.org/changeset/base/337300 Log: Tag llvm release_70 branch r338892. Added: vendor/llvm/llvm-release_70-r338892/ - copied from r337299, vendor/llvm/dist-release_70/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:15:50 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B84A106AF19; Sat, 4 Aug 2018 13:15:50 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E321575E55; Sat, 4 Aug 2018 13:15:49 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C57AF7F5D; Sat, 4 Aug 2018 13:15:49 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DFnZm083394; Sat, 4 Aug 2018 13:15:49 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DFnLM083392; Sat, 4 Aug 2018 13:15:49 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041315.w74DFnLM083392@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:15:49 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337301 - in vendor/clang/dist-release_70: docs include/clang/Basic lib/AST lib/Basic lib/Sema test/CodeGen test/Driver test/SemaCXX X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: in vendor/clang/dist-release_70: docs include/clang/Basic lib/AST lib/Basic lib/Sema test/CodeGen test/Driver test/SemaCXX X-SVN-Commit-Revision: 337301 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:15:50 -0000 Author: dim Date: Sat Aug 4 13:15:49 2018 New Revision: 337301 URL: https://svnweb.freebsd.org/changeset/base/337301 Log: Vendor import of clang release_70 branch r338892: https://llvm.org/svn/llvm-project/cfe/branches/release_70@338892 Modified: vendor/clang/dist-release_70/docs/AttributeReference.rst vendor/clang/dist-release_70/docs/ClangCommandLineReference.rst vendor/clang/dist-release_70/docs/ReleaseNotes.rst vendor/clang/dist-release_70/include/clang/Basic/Builtins.def vendor/clang/dist-release_70/include/clang/Basic/DiagnosticASTKinds.td vendor/clang/dist-release_70/lib/AST/ExprConstant.cpp vendor/clang/dist-release_70/lib/Basic/Version.cpp vendor/clang/dist-release_70/lib/Sema/SemaInit.cpp vendor/clang/dist-release_70/test/CodeGen/builtin-memfns.c vendor/clang/dist-release_70/test/Driver/darwin-infer-simulator-sdkroot.c vendor/clang/dist-release_70/test/Driver/integrated-as.c vendor/clang/dist-release_70/test/SemaCXX/constexpr-string.cpp Modified: vendor/clang/dist-release_70/docs/AttributeReference.rst ============================================================================== --- vendor/clang/dist-release_70/docs/AttributeReference.rst Sat Aug 4 13:15:11 2018 (r337300) +++ vendor/clang/dist-release_70/docs/AttributeReference.rst Sat Aug 4 13:15:49 2018 (r337301) @@ -1,13 +1,3899 @@ .. ------------------------------------------------------------------- NOTE: This file is automatically generated by running clang-tblgen - -gen-attr-docs. Do not edit this file by hand!! The contents for - this file are automatically generated by a server-side process. - - Please do not commit this file. The file exists for local testing - purposes only. + -gen-attr-docs. Do not edit this file by hand!! ------------------------------------------------------------------- =================== Attributes in Clang -=================== \ No newline at end of file +=================== +.. contents:: + :local: + +Introduction +============ + +This page lists the attributes currently supported by Clang. + +Function Attributes +=================== + + +#pragma omp declare simd +------------------------ +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "","","","","", "X", "" + +The `declare simd` construct can be applied to a function to enable the creation +of one or more versions that can process multiple arguments using SIMD +instructions from a single invocation in a SIMD loop. The `declare simd` +directive is a declarative directive. There may be multiple `declare simd` +directives for a function. The use of a `declare simd` construct on a function +enables the creation of SIMD versions of the associated function that can be +used to process multiple arguments from a single invocation from a SIMD loop +concurrently. +The syntax of the `declare simd` construct is as follows: + + .. code-block:: none + + #pragma omp declare simd [clause[[,] clause] ...] new-line + [#pragma omp declare simd [clause[[,] clause] ...] new-line] + [...] + function definition or declaration + +where clause is one of the following: + + .. code-block:: none + + simdlen(length) + linear(argument-list[:constant-linear-step]) + aligned(argument-list[:alignment]) + uniform(argument-list) + inbranch + notinbranch + + +#pragma omp declare target +-------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "","","","","", "X", "" + +The `declare target` directive specifies that variables and functions are mapped +to a device for OpenMP offload mechanism. + +The syntax of the declare target directive is as follows: + + .. code-block:: c + + #pragma omp declare target new-line + declarations-definition-seq + #pragma omp end declare target new-line + + +_Noreturn +--------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "","","","","X", "", "" + +A function declared as ``_Noreturn`` shall not return to its caller. The +compiler will generate a diagnostic for a function declared as ``_Noreturn`` +that appears to be capable of returning to its caller. + + +abi_tag (gnu::abi_tag) +---------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "X" + +The ``abi_tag`` attribute can be applied to a function, variable, class or +inline namespace declaration to modify the mangled name of the entity. It gives +the ability to distinguish between different versions of the same entity but +with different ABI versions supported. For example, a newer version of a class +could have a different set of data members and thus have a different size. Using +the ``abi_tag`` attribute, it is possible to have different mangled names for +a global variable of the class type. Therefore, the old code could keep using +the old manged name and the new code will use the new mangled name with tags. + + +acquire_capability (acquire_shared_capability, clang::acquire_capability, clang::acquire_shared_capability) +----------------------------------------------------------------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "" + +Marks a function as acquiring a capability. + + +alloc_align (gnu::alloc_align) +------------------------------ +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "" + +Use ``__attribute__((alloc_align())`` on a function +declaration to specify that the return value of the function (which must be a +pointer type) is at least as aligned as the value of the indicated parameter. The +parameter is given by its index in the list of formal parameters; the first +parameter has index 1 unless the function is a C++ non-static member function, +in which case the first parameter has index 2 to account for the implicit ``this`` +parameter. + +.. code-block:: c++ + + // The returned pointer has the alignment specified by the first parameter. + void *a(size_t align) __attribute__((alloc_align(1))); + + // The returned pointer has the alignment specified by the second parameter. + void *b(void *v, size_t align) __attribute__((alloc_align(2))); + + // The returned pointer has the alignment specified by the second visible + // parameter, however it must be adjusted for the implicit 'this' parameter. + void *Foo::b(void *v, size_t align) __attribute__((alloc_align(3))); + +Note that this attribute merely informs the compiler that a function always +returns a sufficiently aligned pointer. It does not cause the compiler to +emit code to enforce that alignment. The behavior is undefined if the returned +poitner is not sufficiently aligned. + + +alloc_size (gnu::alloc_size) +---------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "X" + +The ``alloc_size`` attribute can be placed on functions that return pointers in +order to hint to the compiler how many bytes of memory will be available at the +returned pointer. ``alloc_size`` takes one or two arguments. + +- ``alloc_size(N)`` implies that argument number N equals the number of + available bytes at the returned pointer. +- ``alloc_size(N, M)`` implies that the product of argument number N and + argument number M equals the number of available bytes at the returned + pointer. + +Argument numbers are 1-based. + +An example of how to use ``alloc_size`` + +.. code-block:: c + + void *my_malloc(int a) __attribute__((alloc_size(1))); + void *my_calloc(int a, int b) __attribute__((alloc_size(1, 2))); + + int main() { + void *const p = my_malloc(100); + assert(__builtin_object_size(p, 0) == 100); + void *const a = my_calloc(20, 5); + assert(__builtin_object_size(a, 0) == 100); + } + +.. Note:: This attribute works differently in clang than it does in GCC. + Specifically, clang will only trace ``const`` pointers (as above); we give up + on pointers that are not marked as ``const``. In the vast majority of cases, + this is unimportant, because LLVM has support for the ``alloc_size`` + attribute. However, this may cause mildly unintuitive behavior when used with + other attributes, such as ``enable_if``. + + +artificial (gnu::artificial) +---------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "" + +The ``artificial`` attribute can be applied to an inline function. If such a +function is inlined, the attribute indicates that debuggers should associate +the resulting instructions with the call site, rather than with the +corresponding line within the inlined callee. + + +assert_capability (assert_shared_capability, clang::assert_capability, clang::assert_shared_capability) +------------------------------------------------------------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "" + +Marks a function that dynamically tests whether a capability is held, and halts +the program if it is not held. + + +assume_aligned (gnu::assume_aligned) +------------------------------------ +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "X" + +Use ``__attribute__((assume_aligned([,]))`` on a function +declaration to specify that the return value of the function (which must be a +pointer type) has the specified offset, in bytes, from an address with the +specified alignment. The offset is taken to be zero if omitted. + +.. code-block:: c++ + + // The returned pointer value has 32-byte alignment. + void *a() __attribute__((assume_aligned (32))); + + // The returned pointer value is 4 bytes greater than an address having + // 32-byte alignment. + void *b() __attribute__((assume_aligned (32, 4))); + +Note that this attribute provides information to the compiler regarding a +condition that the code already ensures is true. It does not cause the compiler +to enforce the provided alignment assumption. + + +availability (clang::availability, clang::availability) +------------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","X","","", "", "X" + +The ``availability`` attribute can be placed on declarations to describe the +lifecycle of that declaration relative to operating system versions. Consider +the function declaration for a hypothetical function ``f``: + +.. code-block:: c++ + + void f(void) __attribute__((availability(macos,introduced=10.4,deprecated=10.6,obsoleted=10.7))); + +The availability attribute states that ``f`` was introduced in macOS 10.4, +deprecated in macOS 10.6, and obsoleted in macOS 10.7. This information +is used by Clang to determine when it is safe to use ``f``: for example, if +Clang is instructed to compile code for macOS 10.5, a call to ``f()`` +succeeds. If Clang is instructed to compile code for macOS 10.6, the call +succeeds but Clang emits a warning specifying that the function is deprecated. +Finally, if Clang is instructed to compile code for macOS 10.7, the call +fails because ``f()`` is no longer available. + +The availability attribute is a comma-separated list starting with the +platform name and then including clauses specifying important milestones in the +declaration's lifetime (in any order) along with additional information. Those +clauses can be: + +introduced=\ *version* + The first version in which this declaration was introduced. + +deprecated=\ *version* + The first version in which this declaration was deprecated, meaning that + users should migrate away from this API. + +obsoleted=\ *version* + The first version in which this declaration was obsoleted, meaning that it + was removed completely and can no longer be used. + +unavailable + This declaration is never available on this platform. + +message=\ *string-literal* + Additional message text that Clang will provide when emitting a warning or + error about use of a deprecated or obsoleted declaration. Useful to direct + users to replacement APIs. + +replacement=\ *string-literal* + Additional message text that Clang will use to provide Fix-It when emitting + a warning about use of a deprecated declaration. The Fix-It will replace + the deprecated declaration with the new declaration specified. + +Multiple availability attributes can be placed on a declaration, which may +correspond to different platforms. Only the availability attribute with the +platform corresponding to the target platform will be used; any others will be +ignored. If no availability attribute specifies availability for the current +target platform, the availability attributes are ignored. Supported platforms +are: + +``ios`` + Apple's iOS operating system. The minimum deployment target is specified by + the ``-mios-version-min=*version*`` or ``-miphoneos-version-min=*version*`` + command-line arguments. + +``macos`` + Apple's macOS operating system. The minimum deployment target is + specified by the ``-mmacosx-version-min=*version*`` command-line argument. + ``macosx`` is supported for backward-compatibility reasons, but it is + deprecated. + +``tvos`` + Apple's tvOS operating system. The minimum deployment target is specified by + the ``-mtvos-version-min=*version*`` command-line argument. + +``watchos`` + Apple's watchOS operating system. The minimum deployment target is specified by + the ``-mwatchos-version-min=*version*`` command-line argument. + +A declaration can typically be used even when deploying back to a platform +version prior to when the declaration was introduced. When this happens, the +declaration is `weakly linked +`_, +as if the ``weak_import`` attribute were added to the declaration. A +weakly-linked declaration may or may not be present a run-time, and a program +can determine whether the declaration is present by checking whether the +address of that declaration is non-NULL. + +The flag ``strict`` disallows using API when deploying back to a +platform version prior to when the declaration was introduced. An +attempt to use such API before its introduction causes a hard error. +Weakly-linking is almost always a better API choice, since it allows +users to query availability at runtime. + +If there are multiple declarations of the same entity, the availability +attributes must either match on a per-platform basis or later +declarations must not have availability attributes for that +platform. For example: + +.. code-block:: c + + void g(void) __attribute__((availability(macos,introduced=10.4))); + void g(void) __attribute__((availability(macos,introduced=10.4))); // okay, matches + void g(void) __attribute__((availability(ios,introduced=4.0))); // okay, adds a new platform + void g(void); // okay, inherits both macos and ios availability from above. + void g(void) __attribute__((availability(macos,introduced=10.5))); // error: mismatch + +When one method overrides another, the overriding method can be more widely available than the overridden method, e.g.,: + +.. code-block:: objc + + @interface A + - (id)method __attribute__((availability(macos,introduced=10.4))); + - (id)method2 __attribute__((availability(macos,introduced=10.4))); + @end + + @interface B : A + - (id)method __attribute__((availability(macos,introduced=10.3))); // okay: method moved into base class later + - (id)method __attribute__((availability(macos,introduced=10.5))); // error: this method was available via the base class in 10.4 + @end + +Starting with the macOS 10.12 SDK, the ``API_AVAILABLE`` macro from +```` can simplify the spelling: + +.. code-block:: objc + + @interface A + - (id)method API_AVAILABLE(macos(10.11))); + - (id)otherMethod API_AVAILABLE(macos(10.11), ios(11.0)); + @end + +Also see the documentation for `@available +`_ + + +carries_dependency +------------------ +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "X" + +The ``carries_dependency`` attribute specifies dependency propagation into and +out of functions. + +When specified on a function or Objective-C method, the ``carries_dependency`` +attribute means that the return value carries a dependency out of the function, +so that the implementation need not constrain ordering upon return from that +function. Implementations of the function and its caller may choose to preserve +dependencies instead of emitting memory ordering instructions such as fences. + +Note, this attribute does not change the meaning of the program, but may result +in generation of more efficient code. + + +code_seg +-------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "","","","X","", "", "" + +The ``__declspec(code_seg)`` attribute enables the placement of code into separate +named segments that can be paged or locked in memory individually. This attribute +is used to control the placement of instantiated templates and compiler-generated +code. See the documentation for `__declspec(code_seg)`_ on MSDN. + +.. _`__declspec(code_seg)`: http://msdn.microsoft.com/en-us/library/dn636922.aspx + + +convergent (clang::convergent, clang::convergent) +------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","X","","", "", "X" + +The ``convergent`` attribute can be placed on a function declaration. It is +translated into the LLVM ``convergent`` attribute, which indicates that the call +instructions of a function with this attribute cannot be made control-dependent +on any additional values. + +In languages designed for SPMD/SIMT programming model, e.g. OpenCL or CUDA, +the call instructions of a function with this attribute must be executed by +all work items or threads in a work group or sub group. + +This attribute is different from ``noduplicate`` because it allows duplicating +function calls if it can be proved that the duplicated function calls are +not made control-dependent on any additional values, e.g., unrolling a loop +executed by all work items. + +Sample usage: +.. code-block:: c + + void convfunc(void) __attribute__((convergent)); + // Setting it as a C++11 attribute is also valid in a C++ program. + // void convfunc(void) [[clang::convergent]]; + + +cpu_dispatch (clang::cpu_dispatch, clang::cpu_dispatch) +------------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","X","","", "", "X" + +The ``cpu_specific`` and ``cpu_dispatch`` attributes are used to define and +resolve multiversioned functions. This form of multiversioning provides a +mechanism for declaring versions across translation units and manually +specifying the resolved function list. A specified CPU defines a set of minimum +features that are required for the function to be called. The result of this is +that future processors execute the most restrictive version of the function the +new processor can execute. + +Function versions are defined with ``cpu_specific``, which takes one or more CPU +names as a parameter. For example: + +.. code-block:: c + + // Declares and defines the ivybridge version of single_cpu. + __attribute__((cpu_specific(ivybridge))) + void single_cpu(void){} + + // Declares and defines the atom version of single_cpu. + __attribute__((cpu_specific(atom))) + void single_cpu(void){} + + // Declares and defines both the ivybridge and atom version of multi_cpu. + __attribute__((cpu_specific(ivybridge, atom))) + void multi_cpu(void){} + +A dispatching (or resolving) function can be declared anywhere in a project's +source code with ``cpu_dispatch``. This attribute takes one or more CPU names +as a parameter (like ``cpu_specific``). Functions marked with ``cpu_dispatch`` +are not expected to be defined, only declared. If such a marked function has a +definition, any side effects of the function are ignored; trivial function +bodies are permissible for ICC compatibility. + +.. code-block:: c + + // Creates a resolver for single_cpu above. + __attribute__((cpu_dispatch(ivybridge, atom))) + void single_cpu(void){} + + // Creates a resolver for multi_cpu, but adds a 3rd version defined in another + // translation unit. + __attribute__((cpu_dispatch(ivybridge, atom, sandybridge))) + void multi_cpu(void){} + +Note that it is possible to have a resolving function that dispatches based on +more or fewer options than are present in the program. Specifying fewer will +result in the omitted options not being considered during resolution. Specifying +a version for resolution that isn't defined in the program will result in a +linking failure. + +It is also possible to specify a CPU name of ``generic`` which will be resolved +if the executing processor doesn't satisfy the features required in the CPU +name. The behavior of a program executing on a processor that doesn't satisfy +any option of a multiversioned function is undefined. + + +cpu_specific (clang::cpu_specific, clang::cpu_specific) +------------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","X","","", "", "X" + +The ``cpu_specific`` and ``cpu_dispatch`` attributes are used to define and +resolve multiversioned functions. This form of multiversioning provides a +mechanism for declaring versions across translation units and manually +specifying the resolved function list. A specified CPU defines a set of minimum +features that are required for the function to be called. The result of this is +that future processors execute the most restrictive version of the function the +new processor can execute. + +Function versions are defined with ``cpu_specific``, which takes one or more CPU +names as a parameter. For example: + +.. code-block:: c + + // Declares and defines the ivybridge version of single_cpu. + __attribute__((cpu_specific(ivybridge))) + void single_cpu(void){} + + // Declares and defines the atom version of single_cpu. + __attribute__((cpu_specific(atom))) + void single_cpu(void){} + + // Declares and defines both the ivybridge and atom version of multi_cpu. + __attribute__((cpu_specific(ivybridge, atom))) + void multi_cpu(void){} + +A dispatching (or resolving) function can be declared anywhere in a project's +source code with ``cpu_dispatch``. This attribute takes one or more CPU names +as a parameter (like ``cpu_specific``). Functions marked with ``cpu_dispatch`` +are not expected to be defined, only declared. If such a marked function has a +definition, any side effects of the function are ignored; trivial function +bodies are permissible for ICC compatibility. + +.. code-block:: c + + // Creates a resolver for single_cpu above. + __attribute__((cpu_dispatch(ivybridge, atom))) + void single_cpu(void){} + + // Creates a resolver for multi_cpu, but adds a 3rd version defined in another + // translation unit. + __attribute__((cpu_dispatch(ivybridge, atom, sandybridge))) + void multi_cpu(void){} + +Note that it is possible to have a resolving function that dispatches based on +more or fewer options than are present in the program. Specifying fewer will +result in the omitted options not being considered during resolution. Specifying +a version for resolution that isn't defined in the program will result in a +linking failure. + +It is also possible to specify a CPU name of ``generic`` which will be resolved +if the executing processor doesn't satisfy the features required in the CPU +name. The behavior of a program executing on a processor that doesn't satisfy +any option of a multiversioned function is undefined. + + +deprecated (gnu::deprecated) +---------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","X","X","", "", "" + +The ``deprecated`` attribute can be applied to a function, a variable, or a +type. This is useful when identifying functions, variables, or types that are +expected to be removed in a future version of a program. + +Consider the function declaration for a hypothetical function ``f``: + +.. code-block:: c++ + + void f(void) __attribute__((deprecated("message", "replacement"))); + +When spelled as `__attribute__((deprecated))`, the deprecated attribute can have +two optional string arguments. The first one is the message to display when +emitting the warning; the second one enables the compiler to provide a Fix-It +to replace the deprecated name with a new name. Otherwise, when spelled as +`[[gnu::deprecated]] or [[deprecated]]`, the attribute can have one optional +string argument which is the message to display when emitting the warning. + + +diagnose_if +----------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","","","","", "", "" + +The ``diagnose_if`` attribute can be placed on function declarations to emit +warnings or errors at compile-time if calls to the attributed function meet +certain user-defined criteria. For example: + +.. code-block:: c + + void abs(int a) + __attribute__((diagnose_if(a >= 0, "Redundant abs call", "warning"))); + void must_abs(int a) + __attribute__((diagnose_if(a >= 0, "Redundant abs call", "error"))); + + int val = abs(1); // warning: Redundant abs call + int val2 = must_abs(1); // error: Redundant abs call + int val3 = abs(val); + int val4 = must_abs(val); // Because run-time checks are not emitted for + // diagnose_if attributes, this executes without + // issue. + + +``diagnose_if`` is closely related to ``enable_if``, with a few key differences: + +* Overload resolution is not aware of ``diagnose_if`` attributes: they're + considered only after we select the best candidate from a given candidate set. +* Function declarations that differ only in their ``diagnose_if`` attributes are + considered to be redeclarations of the same function (not overloads). +* If the condition provided to ``diagnose_if`` cannot be evaluated, no + diagnostic will be emitted. + +Otherwise, ``diagnose_if`` is essentially the logical negation of ``enable_if``. + +As a result of bullet number two, ``diagnose_if`` attributes will stack on the +same function. For example: + +.. code-block:: c + + int foo() __attribute__((diagnose_if(1, "diag1", "warning"))); + int foo() __attribute__((diagnose_if(1, "diag2", "warning"))); + + int bar = foo(); // warning: diag1 + // warning: diag2 + int (*fooptr)(void) = foo; // warning: diag1 + // warning: diag2 + + constexpr int supportsAPILevel(int N) { return N < 5; } + int baz(int a) + __attribute__((diagnose_if(!supportsAPILevel(10), + "Upgrade to API level 10 to use baz", "error"))); + int baz(int a) + __attribute__((diagnose_if(!a, "0 is not recommended.", "warning"))); + + int (*bazptr)(int) = baz; // error: Upgrade to API level 10 to use baz + int v = baz(0); // error: Upgrade to API level 10 to use baz + +Query for this feature with ``__has_attribute(diagnose_if)``. + + +disable_tail_calls (clang::disable_tail_calls, clang::disable_tail_calls) +------------------------------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","X","","", "", "X" + +The ``disable_tail_calls`` attribute instructs the backend to not perform tail call optimization inside the marked function. + +For example: + + .. code-block:: c + + int callee(int); + + int foo(int a) __attribute__((disable_tail_calls)) { + return callee(a); // This call is not tail-call optimized. + } + +Marking virtual functions as ``disable_tail_calls`` is legal. + + .. code-block:: c++ + + int callee(int); + + class Base { + public: + [[clang::disable_tail_calls]] virtual int foo1() { + return callee(); // This call is not tail-call optimized. + } + }; + + class Derived1 : public Base { + public: + int foo1() override { + return callee(); // This call is tail-call optimized. + } + }; + + +enable_if +--------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","","","","", "", "X" + +.. Note:: Some features of this attribute are experimental. The meaning of + multiple enable_if attributes on a single declaration is subject to change in + a future version of clang. Also, the ABI is not standardized and the name + mangling may change in future versions. To avoid that, use asm labels. + +The ``enable_if`` attribute can be placed on function declarations to control +which overload is selected based on the values of the function's arguments. +When combined with the ``overloadable`` attribute, this feature is also +available in C. + +.. code-block:: c++ + + int isdigit(int c); + int isdigit(int c) __attribute__((enable_if(c <= -1 || c > 255, "chosen when 'c' is out of range"))) __attribute__((unavailable("'c' must have the value of an unsigned char or EOF"))); + + void foo(char c) { + isdigit(c); + isdigit(10); + isdigit(-10); // results in a compile-time error. + } + +The enable_if attribute takes two arguments, the first is an expression written +in terms of the function parameters, the second is a string explaining why this +overload candidate could not be selected to be displayed in diagnostics. The +expression is part of the function signature for the purposes of determining +whether it is a redeclaration (following the rules used when determining +whether a C++ template specialization is ODR-equivalent), but is not part of +the type. + +The enable_if expression is evaluated as if it were the body of a +bool-returning constexpr function declared with the arguments of the function +it is being applied to, then called with the parameters at the call site. If the +result is false or could not be determined through constant expression +evaluation, then this overload will not be chosen and the provided string may +be used in a diagnostic if the compile fails as a result. + +Because the enable_if expression is an unevaluated context, there are no global +state changes, nor the ability to pass information from the enable_if +expression to the function body. For example, suppose we want calls to +strnlen(strbuf, maxlen) to resolve to strnlen_chk(strbuf, maxlen, size of +strbuf) only if the size of strbuf can be determined: + +.. code-block:: c++ + + __attribute__((always_inline)) + static inline size_t strnlen(const char *s, size_t maxlen) + __attribute__((overloadable)) + __attribute__((enable_if(__builtin_object_size(s, 0) != -1))), + "chosen when the buffer size is known but 'maxlen' is not"))) + { + return strnlen_chk(s, maxlen, __builtin_object_size(s, 0)); + } + +Multiple enable_if attributes may be applied to a single declaration. In this +case, the enable_if expressions are evaluated from left to right in the +following manner. First, the candidates whose enable_if expressions evaluate to +false or cannot be evaluated are discarded. If the remaining candidates do not +share ODR-equivalent enable_if expressions, the overload resolution is +ambiguous. Otherwise, enable_if overload resolution continues with the next +enable_if attribute on the candidates that have not been discarded and have +remaining enable_if attributes. In this way, we pick the most specific +overload out of a number of viable overloads using enable_if. + +.. code-block:: c++ + + void f() __attribute__((enable_if(true, ""))); // #1 + void f() __attribute__((enable_if(true, ""))) __attribute__((enable_if(true, ""))); // #2 + + void g(int i, int j) __attribute__((enable_if(i, ""))); // #1 + void g(int i, int j) __attribute__((enable_if(j, ""))) __attribute__((enable_if(true))); // #2 + +In this example, a call to f() is always resolved to #2, as the first enable_if +expression is ODR-equivalent for both declarations, but #1 does not have another +enable_if expression to continue evaluating, so the next round of evaluation has +only a single candidate. In a call to g(1, 1), the call is ambiguous even though +#2 has more enable_if attributes, because the first enable_if expressions are +not ODR-equivalent. + +Query for this feature with ``__has_attribute(enable_if)``. + +Note that functions with one or more ``enable_if`` attributes may not have +their address taken, unless all of the conditions specified by said +``enable_if`` are constants that evaluate to ``true``. For example: + +.. code-block:: c + + const int TrueConstant = 1; + const int FalseConstant = 0; + int f(int a) __attribute__((enable_if(a > 0, ""))); + int g(int a) __attribute__((enable_if(a == 0 || a != 0, ""))); + int h(int a) __attribute__((enable_if(1, ""))); + int i(int a) __attribute__((enable_if(TrueConstant, ""))); + int j(int a) __attribute__((enable_if(FalseConstant, ""))); + + void fn() { + int (*ptr)(int); + ptr = &f; // error: 'a > 0' is not always true + ptr = &g; // error: 'a == 0 || a != 0' is not a truthy constant + ptr = &h; // OK: 1 is a truthy constant + ptr = &i; // OK: 'TrueConstant' is a truthy constant + ptr = &j; // error: 'FalseConstant' is a constant, but not truthy + } + +Because ``enable_if`` evaluation happens during overload resolution, +``enable_if`` may give unintuitive results when used with templates, depending +on when overloads are resolved. In the example below, clang will emit a +diagnostic about no viable overloads for ``foo`` in ``bar``, but not in ``baz``: + +.. code-block:: c++ + + double foo(int i) __attribute__((enable_if(i > 0, ""))); + void *foo(int i) __attribute__((enable_if(i <= 0, ""))); + template + auto bar() { return foo(I); } + + template + auto baz() { return foo(T::number); } + + struct WithNumber { constexpr static int number = 1; }; + void callThem() { + bar(); + baz(); + } + +This is because, in ``bar``, ``foo`` is resolved prior to template +instantiation, so the value for ``I`` isn't known (thus, both ``enable_if`` +conditions for ``foo`` fail). However, in ``baz``, ``foo`` is resolved during +template instantiation, so the value for ``T::number`` is known. + + +external_source_symbol (clang::external_source_symbol, clang::external_source_symbol) +------------------------------------------------------------------------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","X","","", "", "X" + +The ``external_source_symbol`` attribute specifies that a declaration originates +from an external source and describes the nature of that source. + +The fact that Clang is capable of recognizing declarations that were defined +externally can be used to provide better tooling support for mixed-language +projects or projects that rely on auto-generated code. For instance, an IDE that +uses Clang and that supports mixed-language projects can use this attribute to +provide a correct 'jump-to-definition' feature. For a concrete example, +consider a protocol that's defined in a Swift file: + +.. code-block:: swift + + @objc public protocol SwiftProtocol { + func method() + } + +This protocol can be used from Objective-C code by including a header file that +was generated by the Swift compiler. The declarations in that header can use +the ``external_source_symbol`` attribute to make Clang aware of the fact +that ``SwiftProtocol`` actually originates from a Swift module: + +.. code-block:: objc + + __attribute__((external_source_symbol(language="Swift",defined_in="module"))) + @protocol SwiftProtocol + @required + - (void) method; + @end + +Consequently, when 'jump-to-definition' is performed at a location that +references ``SwiftProtocol``, the IDE can jump to the original definition in +the Swift source file rather than jumping to the Objective-C declaration in the +auto-generated header file. + +The ``external_source_symbol`` attribute is a comma-separated list that includes +clauses that describe the origin and the nature of the particular declaration. +Those clauses can be: + +language=\ *string-literal* + The name of the source language in which this declaration was defined. + +defined_in=\ *string-literal* + The name of the source container in which the declaration was defined. The + exact definition of source container is language-specific, e.g. Swift's + source containers are modules, so ``defined_in`` should specify the Swift + module name. + +generated_declaration + This declaration was automatically generated by some tool. + +The clauses can be specified in any order. The clauses that are listed above are +all optional, but the attribute has to have at least one clause. + + +flatten (gnu::flatten) +---------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "X" + +The ``flatten`` attribute causes calls within the attributed function to +be inlined unless it is impossible to do so, for example if the body of the +callee is unavailable or if the callee has the ``noinline`` attribute. + + +force_align_arg_pointer (gnu::force_align_arg_pointer) +------------------------------------------------------ +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "" + +Use this attribute to force stack alignment. + +Legacy x86 code uses 4-byte stack alignment. Newer aligned SSE instructions +(like 'movaps') that work with the stack require operands to be 16-byte aligned. +This attribute realigns the stack in the function prologue to make sure the +stack can be used with SSE instructions. + +Note that the x86_64 ABI forces 16-byte stack alignment at the call site. +Because of this, 'force_align_arg_pointer' is not needed on x86_64, except in +rare cases where the caller does not align the stack properly (e.g. flow +jumps from i386 arch code). + + .. code-block:: c + + __attribute__ ((force_align_arg_pointer)) + void f () { + ... + } + + +format (gnu::format) +-------------------- +.. csv-table:: Supported Syntaxes + :header: "GNU", "C++11", "C2x", "__declspec", "Keyword", "Pragma", "Pragma clang attribute" + + "X","X","","","", "", "" + +Clang supports the ``format`` attribute, which indicates that the function +accepts a ``printf`` or ``scanf``-like format string and corresponding +arguments or a ``va_list`` that contains these arguments. + +Please see `GCC documentation about format attribute +`_ to find details +about attribute syntax. + +Clang implements two kinds of checks with this attribute. + +#. Clang checks that the function with the ``format`` attribute is called with + a format string that uses format specifiers that are allowed, and that + arguments match the format string. This is the ``-Wformat`` warning, it is + on by default. + +#. Clang checks that the format string argument is a literal string. This is + the ``-Wformat-nonliteral`` warning, it is off by default. + + Clang implements this mostly the same way as GCC, but there is a difference + for functions that accept a ``va_list`` argument (for example, ``vprintf``). + GCC does not emit ``-Wformat-nonliteral`` warning for calls to such + functions. Clang does not warn if the format string comes from a function + parameter, where the function is annotated with a compatible attribute, + otherwise it warns. For example: + + .. code-block:: c + + __attribute__((__format__ (__scanf__, 1, 3))) + void foo(const char* s, char *buf, ...) { + va_list ap; + va_start(ap, buf); + + vprintf(s, ap); // warning: format string is not a string literal + } + + In this case we warn because ``s`` contains a format string for a + ``scanf``-like function, but it is passed to a ``printf``-like function. + + If the attribute is removed, clang still warns, because the format string is + not a string literal. + + Another example: + + .. code-block:: c + + __attribute__((__format__ (__printf__, 1, 3))) + void foo(const char* s, char *buf, ...) { + va_list ap; + va_start(ap, buf); + + vprintf(s, ap); // warning + } + + In this case Clang does not warn because the format string ``s`` and + the corresponding arguments are annotated. If the arguments are + incorrect, the caller of ``foo`` will receive a warning. *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:15:53 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7CF46106AF1F; Sat, 4 Aug 2018 13:15:53 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 304C675E5C; Sat, 4 Aug 2018 13:15:53 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id EA8FE7F5E; Sat, 4 Aug 2018 13:15:52 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DFqlM083440; Sat, 4 Aug 2018 13:15:52 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DFqca083439; Sat, 4 Aug 2018 13:15:52 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041315.w74DFqca083439@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:15:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337302 - vendor/clang/clang-release_70-r338892 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/clang/clang-release_70-r338892 X-SVN-Commit-Revision: 337302 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:15:53 -0000 Author: dim Date: Sat Aug 4 13:15:52 2018 New Revision: 337302 URL: https://svnweb.freebsd.org/changeset/base/337302 Log: Tag clang release_70 branch r338892. Added: vendor/clang/clang-release_70-r338892/ - copied from r337301, vendor/clang/dist-release_70/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:15:59 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E70B106AF95; Sat, 4 Aug 2018 13:15:59 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A3E3975F47; Sat, 4 Aug 2018 13:15:58 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9AE317F5F; Sat, 4 Aug 2018 13:15:57 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DFvAI083491; Sat, 4 Aug 2018 13:15:57 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DFvSv083489; Sat, 4 Aug 2018 13:15:57 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041315.w74DFvSv083489@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:15:57 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337303 - vendor/compiler-rt/dist-release_70/lib/sanitizer_common X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/compiler-rt/dist-release_70/lib/sanitizer_common X-SVN-Commit-Revision: 337303 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:15:59 -0000 Author: dim Date: Sat Aug 4 13:15:56 2018 New Revision: 337303 URL: https://svnweb.freebsd.org/changeset/base/337303 Log: Vendor import of compiler-rt release_70 branch r338892: https://llvm.org/svn/llvm-project/compiler-rt/branches/release_70@338892 Modified: vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_common_interceptors.inc vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_linux_libcdep.cc Modified: vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_common_interceptors.inc ============================================================================== --- vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_common_interceptors.inc Sat Aug 4 13:15:52 2018 (r337302) +++ vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_common_interceptors.inc Sat Aug 4 13:15:56 2018 (r337303) @@ -4460,6 +4460,7 @@ INTERCEPTOR(int, random_r, void *buf, u32 *result) { // its metadata. See // https://github.com/google/sanitizers/issues/321. #if SANITIZER_INTERCEPT_PTHREAD_ATTR_GET || \ + SANITIZER_INTERCEPT_PTHREAD_ATTR_GET_SCHED || \ SANITIZER_INTERCEPT_PTHREAD_ATTR_GETINHERITSSCHED || \ SANITIZER_INTERCEPT_PTHREAD_MUTEXATTR_GET || \ SANITIZER_INTERCEPT_PTHREAD_RWLOCKATTR_GET || \ Modified: vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_linux_libcdep.cc ============================================================================== --- vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_linux_libcdep.cc Sat Aug 4 13:15:52 2018 (r337302) +++ vendor/compiler-rt/dist-release_70/lib/sanitizer_common/sanitizer_linux_libcdep.cc Sat Aug 4 13:15:56 2018 (r337303) @@ -292,7 +292,7 @@ uptr ThreadDescriptorSize() { val = FIRST_32_SECOND_64(1168, 1776); else if (minor == 11 || (minor == 12 && patch == 1)) val = FIRST_32_SECOND_64(1168, 2288); - else if (minor <= 13) + else if (minor <= 14) val = FIRST_32_SECOND_64(1168, 2304); else val = FIRST_32_SECOND_64(1216, 2304); From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:16:04 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D33E106AFC4; Sat, 4 Aug 2018 13:16:04 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 507CF75FD5; Sat, 4 Aug 2018 13:16:01 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ADBE47F60; Sat, 4 Aug 2018 13:16:00 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DG0nw083544; Sat, 4 Aug 2018 13:16:00 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DG0xG083543; Sat, 4 Aug 2018 13:16:00 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041316.w74DG0xG083543@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:16:00 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337304 - vendor/compiler-rt/compiler-rt-release_70-r338892 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/compiler-rt/compiler-rt-release_70-r338892 X-SVN-Commit-Revision: 337304 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:16:04 -0000 Author: dim Date: Sat Aug 4 13:16:00 2018 New Revision: 337304 URL: https://svnweb.freebsd.org/changeset/base/337304 Log: Tag compiler-rt release_70 branch r338892. Added: vendor/compiler-rt/compiler-rt-release_70-r338892/ - copied from r337303, vendor/compiler-rt/dist-release_70/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:16:09 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 77AE7106AFF9; Sat, 4 Aug 2018 13:16:09 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8C7F676105; Sat, 4 Aug 2018 13:16:07 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2E3427F61; Sat, 4 Aug 2018 13:16:07 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DG60e083594; Sat, 4 Aug 2018 13:16:06 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DG6FJ083593; Sat, 4 Aug 2018 13:16:06 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041316.w74DG6FJ083593@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:16:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337305 - vendor/libc++/dist-release_70 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/libc++/dist-release_70 X-SVN-Commit-Revision: 337305 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:16:09 -0000 Author: dim Date: Sat Aug 4 13:16:06 2018 New Revision: 337305 URL: https://svnweb.freebsd.org/changeset/base/337305 Log: Vendor import of libc++ release_70 branch r338892: https://llvm.org/svn/llvm-project/libcxx/branches/release_70@338892 Modified: vendor/libc++/dist-release_70/CMakeLists.txt Modified: vendor/libc++/dist-release_70/CMakeLists.txt ============================================================================== --- vendor/libc++/dist-release_70/CMakeLists.txt Sat Aug 4 13:16:00 2018 (r337304) +++ vendor/libc++/dist-release_70/CMakeLists.txt Sat Aug 4 13:16:06 2018 (r337305) @@ -27,7 +27,7 @@ if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR project(libcxx CXX C) set(PACKAGE_NAME libcxx) - set(PACKAGE_VERSION 7.0.0svn) + set(PACKAGE_VERSION 7.0.0) set(PACKAGE_STRING "${PACKAGE_NAME} ${PACKAGE_VERSION}") set(PACKAGE_BUGREPORT "llvm-bugs@lists.llvm.org") From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:16:14 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 29E6D106B01B; Sat, 4 Aug 2018 13:16:14 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7DE4E76139; Sat, 4 Aug 2018 13:16:10 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 327C57F62; Sat, 4 Aug 2018 13:16:10 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DGAPY083640; Sat, 4 Aug 2018 13:16:10 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DGA0p083639; Sat, 4 Aug 2018 13:16:10 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041316.w74DGA0p083639@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:16:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337306 - vendor/libc++/libc++-release_70-r338892 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/libc++/libc++-release_70-r338892 X-SVN-Commit-Revision: 337306 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:16:14 -0000 Author: dim Date: Sat Aug 4 13:16:09 2018 New Revision: 337306 URL: https://svnweb.freebsd.org/changeset/base/337306 Log: Tag libc++ release_70 branch r338892. Added: vendor/libc++/libc++-release_70-r338892/ - copied from r337305, vendor/libc++/dist-release_70/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:16:16 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95C50106B02C; Sat, 4 Aug 2018 13:16:16 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0DB8476201; Sat, 4 Aug 2018 13:16:14 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 990317F63; Sat, 4 Aug 2018 13:16:14 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DGEni083689; Sat, 4 Aug 2018 13:16:14 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DGEMD083688; Sat, 4 Aug 2018 13:16:14 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041316.w74DGEMD083688@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:16:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337307 - vendor/lld/lld-release_70-r338892 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lld/lld-release_70-r338892 X-SVN-Commit-Revision: 337307 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:16:16 -0000 Author: dim Date: Sat Aug 4 13:16:14 2018 New Revision: 337307 URL: https://svnweb.freebsd.org/changeset/base/337307 Log: Tag lld release_70 branch r338892. Added: vendor/lld/lld-release_70-r338892/ - copied from r337306, vendor/lld/dist-release_70/ From owner-svn-src-vendor@freebsd.org Sat Aug 4 13:16:25 2018 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D37F106B079; Sat, 4 Aug 2018 13:16:25 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B1F676368; Sat, 4 Aug 2018 13:16:21 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8171C7F64; Sat, 4 Aug 2018 13:16:21 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w74DGLZj083738; Sat, 4 Aug 2018 13:16:21 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w74DGLlt083737; Sat, 4 Aug 2018 13:16:21 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201808041316.w74DGLlt083737@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Sat, 4 Aug 2018 13:16:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r337308 - vendor/lldb/lldb-release_70-r338892 X-SVN-Group: vendor X-SVN-Commit-Author: dim X-SVN-Commit-Paths: vendor/lldb/lldb-release_70-r338892 X-SVN-Commit-Revision: 337308 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Aug 2018 13:16:25 -0000 Author: dim Date: Sat Aug 4 13:16:20 2018 New Revision: 337308 URL: https://svnweb.freebsd.org/changeset/base/337308 Log: Tag lldb release_70 branch r338892. Added: vendor/lldb/lldb-release_70-r338892/ - copied from r337307, vendor/lldb/dist-release_70/