From owner-svn-src-vendor@freebsd.org Mon May 22 19:43:32 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53D19D7895D; Mon, 22 May 2017 19:43:32 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C29621866; Mon, 22 May 2017 19:43:31 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJhUpp076449; Mon, 22 May 2017 19:43:30 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJhTrW076434; Mon, 22 May 2017 19:43:29 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221943.v4MJhTrW076434@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:43:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318663 - in vendor/llvm/dist: docs include/llvm include/llvm/ADT include/llvm/Analysis include/llvm/CodeGen include/llvm/DebugInfo/CodeView include/llvm/DebugInfo/DWARF include/llvm/De... X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:43:32 -0000 Author: dim Date: Mon May 22 19:43:28 2017 New Revision: 318663 URL: https://svnweb.freebsd.org/changeset/base/318663 Log: Vendor import of llvm trunk r303571: https://llvm.org/svn/llvm-project/llvm/trunk@303571 Added: vendor/llvm/dist/include/llvm/DebugInfo/CodeView/LazyRandomTypeCollection.h (contents, props changed) vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeCollection.h (contents, props changed) vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeTableCollection.h (contents, props changed) vendor/llvm/dist/include/llvm/IR/DerivedUser.h (contents, props changed) vendor/llvm/dist/include/llvm/Object/COFFModuleDefinition.h (contents, props changed) vendor/llvm/dist/include/llvm/Object/WindowsResource.h (contents, props changed) vendor/llvm/dist/lib/DebugInfo/CodeView/LazyRandomTypeCollection.cpp (contents, props changed) vendor/llvm/dist/lib/DebugInfo/CodeView/TypeIndex.cpp (contents, props changed) vendor/llvm/dist/lib/DebugInfo/CodeView/TypeTableCollection.cpp (contents, props changed) vendor/llvm/dist/lib/Object/COFFImportFile.cpp (contents, props changed) vendor/llvm/dist/lib/Object/COFFModuleDefinition.cpp (contents, props changed) vendor/llvm/dist/lib/Object/WindowsResource.cpp (contents, props changed) vendor/llvm/dist/lib/Support/BinaryStreamRef.cpp (contents, props changed) vendor/llvm/dist/test/CodeGen/AMDGPU/frame-index-elimination.ll vendor/llvm/dist/test/CodeGen/AMDGPU/function-args.ll vendor/llvm/dist/test/CodeGen/AMDGPU/function-returns.ll vendor/llvm/dist/test/CodeGen/AMDGPU/lshl64-to-32.ll vendor/llvm/dist/test/CodeGen/AMDGPU/regcoalesce-prune.mir vendor/llvm/dist/test/CodeGen/Generic/opt-codegen-no-target-machine.ll vendor/llvm/dist/test/CodeGen/Mips/dins.ll vendor/llvm/dist/test/CodeGen/Mips/micromips-attr.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/add-vec.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/legalize-add-v128.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/legalize-add-v256.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/legalize-add-v512.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/legalize-sub-v128.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/legalize-sub-v256.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/legalize-sub-v512.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-add-v128.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-add-v256.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-add-v512.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-sub-v128.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-sub-v256.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-sub-v512.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/sub-vec.ll vendor/llvm/dist/test/CodeGen/X86/bitcast-setcc-128.ll vendor/llvm/dist/test/CodeGen/X86/bitcast-setcc-256.ll vendor/llvm/dist/test/CodeGen/X86/fmsubadd-combine.ll vendor/llvm/dist/test/CodeGen/X86/leaFixup32.mir vendor/llvm/dist/test/CodeGen/X86/leaFixup64.mir vendor/llvm/dist/test/CodeGen/X86/subcarry.ll vendor/llvm/dist/test/DebugInfo/Inputs/split-dwarf-addr-object-relocation.dwo (contents, props changed) vendor/llvm/dist/test/DebugInfo/Inputs/split-dwarf-addr-object-relocation.o (contents, props changed) vendor/llvm/dist/test/DebugInfo/Inputs/split-dwarf-multiple-cu.dwo (contents, props changed) vendor/llvm/dist/test/DebugInfo/Inputs/split-dwarf-multiple-cu.o (contents, props changed) vendor/llvm/dist/test/DebugInfo/PDB/Inputs/merge1.yaml vendor/llvm/dist/test/DebugInfo/PDB/Inputs/merge2.yaml vendor/llvm/dist/test/DebugInfo/PDB/pdbdump-mergetypes.test vendor/llvm/dist/test/LTO/X86/Inputs/strip-debug-info-bar.ll vendor/llvm/dist/test/MC/Disassembler/AMDGPU/exp_vi.txt (contents, props changed) vendor/llvm/dist/test/ThinLTO/X86/Inputs/merge-triple.ll vendor/llvm/dist/test/ThinLTO/X86/merge-triple.ll vendor/llvm/dist/test/Transforms/InstCombine/AMDGPU/amdgcn-demanded-vector-elts.ll vendor/llvm/dist/test/Transforms/InstCombine/NVPTX/ vendor/llvm/dist/test/Transforms/InstCombine/NVPTX/lit.local.cfg vendor/llvm/dist/test/Transforms/InstCombine/NVPTX/nvvm-intrins.ll vendor/llvm/dist/test/Transforms/InstCombine/wcslen-1.ll vendor/llvm/dist/test/Transforms/InstCombine/wcslen-2.ll vendor/llvm/dist/test/Transforms/InstCombine/wcslen-3.ll vendor/llvm/dist/test/Transforms/InstSimplify/icmp-bool-constant.ll vendor/llvm/dist/test/Transforms/LoopStrengthReduce/X86/canonical-2.ll vendor/llvm/dist/test/Transforms/NewGVN/completeness.ll vendor/llvm/dist/test/Transforms/NewGVN/pr32838.ll vendor/llvm/dist/test/Transforms/NewGVN/pr32845.ll vendor/llvm/dist/test/Transforms/NewGVN/pr32897.ll vendor/llvm/dist/test/Transforms/NewGVN/pr32945.ll vendor/llvm/dist/test/Transforms/NewGVN/pr33014.ll vendor/llvm/dist/test/Transforms/NewGVN/pr33086.ll vendor/llvm/dist/test/Transforms/NewGVN/pr33116.ll vendor/llvm/dist/test/tools/llvm-cvtres/Inputs/ vendor/llvm/dist/test/tools/llvm-cvtres/Inputs/cursor_small.bmp (contents, props changed) vendor/llvm/dist/test/tools/llvm-cvtres/Inputs/okay_small.bmp (contents, props changed) vendor/llvm/dist/test/tools/llvm-cvtres/Inputs/test_resource.rc vendor/llvm/dist/test/tools/llvm-cvtres/Inputs/test_resource.res (contents, props changed) vendor/llvm/dist/test/tools/llvm-cvtres/resource.test Deleted: vendor/llvm/dist/include/llvm/DebugInfo/CodeView/CVTypeDumper.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/RandomAccessTypeVisitor.h vendor/llvm/dist/lib/CodeGen/LiveRangeShrink.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/CVTypeDumper.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/RandomAccessTypeVisitor.cpp vendor/llvm/dist/test/CodeGen/X86/avx512-mask-bugfix.ll vendor/llvm/dist/test/CodeGen/X86/lrshrink.ll vendor/llvm/dist/test/LTO/X86/Inputs/strip-debug-info.bc vendor/llvm/dist/test/Transforms/InstCombine/2008-09-29-FoldingOr.ll vendor/llvm/dist/test/Transforms/InstCombine/amdgcn-demanded-vector-elts.ll vendor/llvm/dist/test/Transforms/InstCombine/nvvm-intrins.ll Modified: vendor/llvm/dist/docs/CMake.rst vendor/llvm/dist/docs/GetElementPtr.rst vendor/llvm/dist/docs/LangRef.rst vendor/llvm/dist/include/llvm/ADT/APInt.h vendor/llvm/dist/include/llvm/ADT/SmallPtrSet.h vendor/llvm/dist/include/llvm/ADT/Statistic.h vendor/llvm/dist/include/llvm/ADT/Triple.h vendor/llvm/dist/include/llvm/Analysis/MemorySSA.h vendor/llvm/dist/include/llvm/Analysis/ScalarEvolution.h vendor/llvm/dist/include/llvm/Analysis/TargetLibraryInfo.h vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h vendor/llvm/dist/include/llvm/CodeGen/LiveIntervalAnalysis.h vendor/llvm/dist/include/llvm/CodeGen/MachineValueType.h vendor/llvm/dist/include/llvm/CodeGen/Passes.h vendor/llvm/dist/include/llvm/CodeGen/StackProtector.h vendor/llvm/dist/include/llvm/CodeGen/ValueTypes.td vendor/llvm/dist/include/llvm/DebugInfo/CodeView/CVRecord.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/CVTypeVisitor.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/SymbolDumper.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeDatabase.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeDumpVisitor.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeIndex.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeSerializer.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeStreamMerger.h vendor/llvm/dist/include/llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFCompileUnit.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFContext.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFRelocMap.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFTypeUnit.h vendor/llvm/dist/include/llvm/DebugInfo/DWARF/DWARFUnit.h vendor/llvm/dist/include/llvm/DebugInfo/PDB/Native/PDBTypeServerHandler.h vendor/llvm/dist/include/llvm/DebugInfo/PDB/Native/TpiStreamBuilder.h vendor/llvm/dist/include/llvm/IR/Argument.h vendor/llvm/dist/include/llvm/IR/BasicBlock.h vendor/llvm/dist/include/llvm/IR/Constant.h vendor/llvm/dist/include/llvm/IR/Constants.h vendor/llvm/dist/include/llvm/IR/Function.h vendor/llvm/dist/include/llvm/IR/GlobalValue.h vendor/llvm/dist/include/llvm/IR/GlobalVariable.h vendor/llvm/dist/include/llvm/IR/InlineAsm.h vendor/llvm/dist/include/llvm/IR/InstrTypes.h vendor/llvm/dist/include/llvm/IR/Instruction.def vendor/llvm/dist/include/llvm/IR/Instruction.h vendor/llvm/dist/include/llvm/IR/Instructions.h vendor/llvm/dist/include/llvm/IR/Metadata.h vendor/llvm/dist/include/llvm/IR/OperandTraits.h vendor/llvm/dist/include/llvm/IR/Operator.h vendor/llvm/dist/include/llvm/IR/PatternMatch.h vendor/llvm/dist/include/llvm/IR/User.h vendor/llvm/dist/include/llvm/IR/Value.def vendor/llvm/dist/include/llvm/IR/Value.h vendor/llvm/dist/include/llvm/InitializePasses.h vendor/llvm/dist/include/llvm/Object/Binary.h vendor/llvm/dist/include/llvm/Object/COFFImportFile.h vendor/llvm/dist/include/llvm/Object/Decompressor.h vendor/llvm/dist/include/llvm/Object/ELF.h vendor/llvm/dist/include/llvm/Object/RelocVisitor.h vendor/llvm/dist/include/llvm/PassInfo.h vendor/llvm/dist/include/llvm/PassSupport.h vendor/llvm/dist/include/llvm/Support/BinaryStreamReader.h vendor/llvm/dist/include/llvm/Support/BinaryStreamRef.h vendor/llvm/dist/include/llvm/Support/BinaryStreamWriter.h vendor/llvm/dist/include/llvm/Support/FileSystem.h vendor/llvm/dist/include/llvm/Target/GlobalISel/SelectionDAGCompat.td vendor/llvm/dist/include/llvm/Transforms/IPO/FunctionImport.h vendor/llvm/dist/include/llvm/Transforms/Scalar/GVNExpression.h vendor/llvm/dist/include/llvm/Transforms/Utils/SimplifyLibCalls.h vendor/llvm/dist/lib/Analysis/BasicAliasAnalysis.cpp vendor/llvm/dist/lib/Analysis/BranchProbabilityInfo.cpp vendor/llvm/dist/lib/Analysis/CallGraphSCCPass.cpp vendor/llvm/dist/lib/Analysis/InstructionSimplify.cpp vendor/llvm/dist/lib/Analysis/MemorySSA.cpp vendor/llvm/dist/lib/Analysis/ScalarEvolution.cpp vendor/llvm/dist/lib/Analysis/TargetLibraryInfo.cpp vendor/llvm/dist/lib/Analysis/ValueTracking.cpp vendor/llvm/dist/lib/AsmParser/LLParser.cpp vendor/llvm/dist/lib/Bitcode/Reader/BitcodeReader.cpp vendor/llvm/dist/lib/Bitcode/Reader/ValueList.cpp vendor/llvm/dist/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp vendor/llvm/dist/lib/CodeGen/AtomicExpandPass.cpp vendor/llvm/dist/lib/CodeGen/CMakeLists.txt vendor/llvm/dist/lib/CodeGen/CodeGen.cpp vendor/llvm/dist/lib/CodeGen/CodeGenPrepare.cpp vendor/llvm/dist/lib/CodeGen/DwarfEHPrepare.cpp vendor/llvm/dist/lib/CodeGen/GlobalISel/IRTranslator.cpp vendor/llvm/dist/lib/CodeGen/GlobalISel/InstructionSelector.cpp vendor/llvm/dist/lib/CodeGen/InterleavedAccessPass.cpp vendor/llvm/dist/lib/CodeGen/LLVMTargetMachine.cpp vendor/llvm/dist/lib/CodeGen/LowerEmuTLS.cpp vendor/llvm/dist/lib/CodeGen/MachineBlockPlacement.cpp vendor/llvm/dist/lib/CodeGen/MachineModuleInfo.cpp vendor/llvm/dist/lib/CodeGen/PrologEpilogInserter.cpp vendor/llvm/dist/lib/CodeGen/RegisterCoalescer.cpp vendor/llvm/dist/lib/CodeGen/SafeStack.cpp vendor/llvm/dist/lib/CodeGen/SafeStackColoring.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/DAGCombiner.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/SelectionDAG.cpp vendor/llvm/dist/lib/CodeGen/SelectionDAG/StatepointLowering.cpp vendor/llvm/dist/lib/CodeGen/StackProtector.cpp vendor/llvm/dist/lib/CodeGen/TargetPassConfig.cpp vendor/llvm/dist/lib/CodeGen/WinEHPrepare.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/CMakeLists.txt vendor/llvm/dist/lib/DebugInfo/CodeView/CVTypeVisitor.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/SymbolDumper.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/TypeDatabase.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/TypeDumpVisitor.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/TypeSerializer.cpp vendor/llvm/dist/lib/DebugInfo/CodeView/TypeStreamMerger.cpp vendor/llvm/dist/lib/DebugInfo/DWARF/DWARFContext.cpp vendor/llvm/dist/lib/DebugInfo/DWARF/DWARFUnit.cpp vendor/llvm/dist/lib/DebugInfo/PDB/Native/DbiStream.cpp vendor/llvm/dist/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp vendor/llvm/dist/lib/DebugInfo/PDB/Native/PDBTypeServerHandler.cpp vendor/llvm/dist/lib/DebugInfo/PDB/Native/TpiStream.cpp vendor/llvm/dist/lib/DebugInfo/PDB/Native/TpiStreamBuilder.cpp vendor/llvm/dist/lib/Demangle/ItaniumDemangle.cpp vendor/llvm/dist/lib/IR/Attributes.cpp vendor/llvm/dist/lib/IR/AutoUpgrade.cpp vendor/llvm/dist/lib/IR/Constants.cpp vendor/llvm/dist/lib/IR/ConstantsContext.h vendor/llvm/dist/lib/IR/DataLayout.cpp vendor/llvm/dist/lib/IR/Function.cpp vendor/llvm/dist/lib/IR/IRBuilder.cpp vendor/llvm/dist/lib/IR/InlineAsm.cpp vendor/llvm/dist/lib/IR/Instruction.cpp vendor/llvm/dist/lib/IR/Instructions.cpp vendor/llvm/dist/lib/IR/LLVMContextImpl.cpp vendor/llvm/dist/lib/IR/PassRegistry.cpp vendor/llvm/dist/lib/IR/User.cpp vendor/llvm/dist/lib/IR/Value.cpp vendor/llvm/dist/lib/IR/ValueTypes.cpp vendor/llvm/dist/lib/IR/Verifier.cpp vendor/llvm/dist/lib/LTO/ThinLTOCodeGenerator.cpp vendor/llvm/dist/lib/Linker/IRMover.cpp vendor/llvm/dist/lib/Object/Binary.cpp vendor/llvm/dist/lib/Object/CMakeLists.txt vendor/llvm/dist/lib/Object/Decompressor.cpp vendor/llvm/dist/lib/Passes/PassBuilder.cpp vendor/llvm/dist/lib/Support/APInt.cpp vendor/llvm/dist/lib/Support/BinaryStreamReader.cpp vendor/llvm/dist/lib/Support/BinaryStreamWriter.cpp vendor/llvm/dist/lib/Support/CMakeLists.txt vendor/llvm/dist/lib/Support/FormattedStream.cpp vendor/llvm/dist/lib/Support/Triple.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64InstrInfo.td vendor/llvm/dist/lib/Target/AArch64/AArch64Subtarget.cpp vendor/llvm/dist/lib/Target/AArch64/AArch64Subtarget.h vendor/llvm/dist/lib/Target/AArch64/AArch64TargetMachine.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPU.h vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUAnnotateKernelFeatures.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUCallLowering.h vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUCallingConv.td vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUISelDAGToDAG.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUISelLowering.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUISelLowering.h vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUInstrInfo.td vendor/llvm/dist/lib/Target/AMDGPU/AMDGPULowerIntrinsics.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUMCInstLower.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUMachineFunction.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUPromoteAlloca.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPURegisterInfo.cpp vendor/llvm/dist/lib/Target/AMDGPU/AMDGPURegisterInfo.h vendor/llvm/dist/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp vendor/llvm/dist/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp vendor/llvm/dist/lib/Target/AMDGPU/BUFInstructions.td vendor/llvm/dist/lib/Target/AMDGPU/Disassembler/AMDGPUDisassembler.cpp vendor/llvm/dist/lib/Target/AMDGPU/Disassembler/AMDGPUDisassembler.h vendor/llvm/dist/lib/Target/AMDGPU/GCNRegPressure.cpp vendor/llvm/dist/lib/Target/AMDGPU/GCNRegPressure.h vendor/llvm/dist/lib/Target/AMDGPU/R600ClauseMergePass.cpp vendor/llvm/dist/lib/Target/AMDGPU/R600ControlFlowFinalizer.cpp vendor/llvm/dist/lib/Target/AMDGPU/R600ExpandSpecialInstrs.cpp vendor/llvm/dist/lib/Target/AMDGPU/R600OptimizeVectorRegisters.cpp vendor/llvm/dist/lib/Target/AMDGPU/R600Packetizer.cpp vendor/llvm/dist/lib/Target/AMDGPU/R600RegisterInfo.cpp vendor/llvm/dist/lib/Target/AMDGPU/R600RegisterInfo.h vendor/llvm/dist/lib/Target/AMDGPU/SIFrameLowering.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIFrameLowering.h vendor/llvm/dist/lib/Target/AMDGPU/SIISelLowering.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIISelLowering.h vendor/llvm/dist/lib/Target/AMDGPU/SIInstrFormats.td vendor/llvm/dist/lib/Target/AMDGPU/SILoadStoreOptimizer.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIMachineFunctionInfo.h vendor/llvm/dist/lib/Target/AMDGPU/SIPeepholeSDWA.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIRegisterInfo.cpp vendor/llvm/dist/lib/Target/AMDGPU/SIRegisterInfo.h vendor/llvm/dist/lib/Target/AMDGPU/SOPInstructions.td vendor/llvm/dist/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp vendor/llvm/dist/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h vendor/llvm/dist/lib/Target/ARM/ARMTargetMachine.cpp vendor/llvm/dist/lib/Target/Hexagon/HexagonLoopIdiomRecognition.cpp vendor/llvm/dist/lib/Target/Hexagon/HexagonTargetMachine.cpp vendor/llvm/dist/lib/Target/Mips/Mips.h vendor/llvm/dist/lib/Target/Mips/Mips16HardFloat.cpp vendor/llvm/dist/lib/Target/Mips/MipsDelaySlotFiller.cpp vendor/llvm/dist/lib/Target/Mips/MipsISelLowering.cpp vendor/llvm/dist/lib/Target/Mips/MipsLongBranch.cpp vendor/llvm/dist/lib/Target/Mips/MipsModuleISelDAGToDAG.cpp vendor/llvm/dist/lib/Target/Mips/MipsOptimizePICCall.cpp vendor/llvm/dist/lib/Target/Mips/MipsOs16.cpp vendor/llvm/dist/lib/Target/Mips/MipsTargetMachine.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCISelLowering.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCInstrAltivec.td vendor/llvm/dist/lib/Target/PowerPC/PPCInstrInfo.cpp vendor/llvm/dist/lib/Target/PowerPC/PPCTargetMachine.cpp vendor/llvm/dist/lib/Target/Sparc/DelaySlotFiller.cpp vendor/llvm/dist/lib/Target/Sparc/LeonPasses.cpp vendor/llvm/dist/lib/Target/Sparc/LeonPasses.h vendor/llvm/dist/lib/Target/Sparc/Sparc.h vendor/llvm/dist/lib/Target/Sparc/SparcTargetMachine.cpp vendor/llvm/dist/lib/Target/WebAssembly/WebAssemblyTargetMachine.cpp vendor/llvm/dist/lib/Target/X86/X86.td vendor/llvm/dist/lib/Target/X86/X86CallingConv.td vendor/llvm/dist/lib/Target/X86/X86FastISel.cpp vendor/llvm/dist/lib/Target/X86/X86FixupLEAs.cpp vendor/llvm/dist/lib/Target/X86/X86ISelLowering.cpp vendor/llvm/dist/lib/Target/X86/X86InstrAVX512.td vendor/llvm/dist/lib/Target/X86/X86InstrFragmentsSIMD.td vendor/llvm/dist/lib/Target/X86/X86InstrInfo.cpp vendor/llvm/dist/lib/Target/X86/X86InstrInfo.td vendor/llvm/dist/lib/Target/X86/X86InstrTSX.td vendor/llvm/dist/lib/Target/X86/X86InstructionSelector.cpp vendor/llvm/dist/lib/Target/X86/X86LegalizerInfo.cpp vendor/llvm/dist/lib/Target/X86/X86RegisterInfo.td vendor/llvm/dist/lib/Target/X86/X86Subtarget.cpp vendor/llvm/dist/lib/Target/X86/X86Subtarget.h vendor/llvm/dist/lib/Target/X86/X86TargetMachine.cpp vendor/llvm/dist/lib/Target/X86/X86TargetTransformInfo.cpp vendor/llvm/dist/lib/Target/XCore/XCoreTargetMachine.cpp vendor/llvm/dist/lib/Transforms/IPO/PassManagerBuilder.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineAddSub.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineAndOrXor.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineCasts.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineCompares.cpp vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineInternal.h vendor/llvm/dist/lib/Transforms/InstCombine/InstCombineMulDivRem.cpp vendor/llvm/dist/lib/Transforms/Scalar/EarlyCSE.cpp vendor/llvm/dist/lib/Transforms/Scalar/GVN.cpp vendor/llvm/dist/lib/Transforms/Scalar/JumpThreading.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoadCombine.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopIdiomRecognize.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopPredication.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopRotation.cpp vendor/llvm/dist/lib/Transforms/Scalar/LoopStrengthReduce.cpp vendor/llvm/dist/lib/Transforms/Scalar/NewGVN.cpp vendor/llvm/dist/lib/Transforms/Scalar/Reassociate.cpp vendor/llvm/dist/lib/Transforms/Scalar/SROA.cpp vendor/llvm/dist/lib/Transforms/Scalar/StraightLineStrengthReduce.cpp vendor/llvm/dist/lib/Transforms/Utils/CloneFunction.cpp vendor/llvm/dist/lib/Transforms/Utils/SimplifyCFG.cpp vendor/llvm/dist/lib/Transforms/Utils/SimplifyLibCalls.cpp vendor/llvm/dist/lib/Transforms/Vectorize/LoopVectorize.cpp vendor/llvm/dist/lib/Transforms/Vectorize/SLPVectorizer.cpp vendor/llvm/dist/test/Analysis/BranchProbabilityInfo/basic.ll vendor/llvm/dist/test/Analysis/BranchProbabilityInfo/deopt-intrinsic.ll vendor/llvm/dist/test/Analysis/BranchProbabilityInfo/noreturn.ll vendor/llvm/dist/test/Analysis/CostModel/X86/ctlz.ll vendor/llvm/dist/test/Analysis/CostModel/X86/ctpop.ll vendor/llvm/dist/test/Analysis/CostModel/X86/cttz.ll vendor/llvm/dist/test/Analysis/ScalarEvolution/nsw.ll vendor/llvm/dist/test/Analysis/ScalarEvolution/trip-count-pow2.ll vendor/llvm/dist/test/CodeGen/AArch64/arm64-ccmp.ll vendor/llvm/dist/test/CodeGen/AArch64/arm64-misched-multimmo.ll vendor/llvm/dist/test/CodeGen/AMDGPU/GlobalISel/amdgpu-irtranslator.ll vendor/llvm/dist/test/CodeGen/AMDGPU/add.i16.ll vendor/llvm/dist/test/CodeGen/AMDGPU/add.v2i16.ll vendor/llvm/dist/test/CodeGen/AMDGPU/bfe-patterns.ll vendor/llvm/dist/test/CodeGen/AMDGPU/coalescer_distribute.ll vendor/llvm/dist/test/CodeGen/AMDGPU/ctlz.ll vendor/llvm/dist/test/CodeGen/AMDGPU/ctlz_zero_undef.ll vendor/llvm/dist/test/CodeGen/AMDGPU/ds_write2.ll vendor/llvm/dist/test/CodeGen/AMDGPU/endcf-loop-header.ll vendor/llvm/dist/test/CodeGen/AMDGPU/fmed3.ll vendor/llvm/dist/test/CodeGen/AMDGPU/hsa-func.ll vendor/llvm/dist/test/CodeGen/AMDGPU/i1-copy-phi.ll vendor/llvm/dist/test/CodeGen/AMDGPU/inline-asm.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.atomic.dec.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.atomic.inc.ll vendor/llvm/dist/test/CodeGen/AMDGPU/llvm.amdgcn.s.barrier.ll vendor/llvm/dist/test/CodeGen/AMDGPU/packed-op-sel.ll vendor/llvm/dist/test/CodeGen/AMDGPU/sdwa-peephole.ll vendor/llvm/dist/test/CodeGen/AMDGPU/shift-and-i64-ubfe.ll vendor/llvm/dist/test/CodeGen/AMDGPU/srl.ll vendor/llvm/dist/test/CodeGen/AMDGPU/sub.i16.ll vendor/llvm/dist/test/CodeGen/AMDGPU/subreg_interference.mir vendor/llvm/dist/test/CodeGen/AMDGPU/vgpr-spill-emergency-stack-slot-compute.ll vendor/llvm/dist/test/CodeGen/ARM/GlobalISel/arm-irtranslator.ll vendor/llvm/dist/test/CodeGen/ARM/ifcvt-branch-weight-bug.ll vendor/llvm/dist/test/CodeGen/Mips/mips64-f128.ll vendor/llvm/dist/test/CodeGen/NVPTX/sched1.ll vendor/llvm/dist/test/CodeGen/NVPTX/sched2.ll vendor/llvm/dist/test/CodeGen/NVPTX/vec8.ll vendor/llvm/dist/test/CodeGen/PowerPC/opt-cmp-inst-cr0-live.ll vendor/llvm/dist/test/CodeGen/PowerPC/shift128.ll vendor/llvm/dist/test/CodeGen/SPARC/LeonItinerariesUT.ll vendor/llvm/dist/test/CodeGen/X86/2007-01-08-InstrSched.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/add-scalar.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/binop.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/br.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/callingconv.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/cmp.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/constant.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/ext-x86-64.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/ext.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/frameIndex.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/gep.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/memop-scalar-x32.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/memop-scalar.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/memop-vec.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/mul-scalar.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/mul-vec.ll vendor/llvm/dist/test/CodeGen/X86/GlobalISel/regbankselect-AVX2.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/regbankselect-AVX512.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-br.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-cmp.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-constant.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-ext-x86-64.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-ext.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-frameIndex.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-gep.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/select-trunc.mir vendor/llvm/dist/test/CodeGen/X86/GlobalISel/trunc.ll vendor/llvm/dist/test/CodeGen/X86/O0-pipeline.ll vendor/llvm/dist/test/CodeGen/X86/addcarry.ll vendor/llvm/dist/test/CodeGen/X86/avg.ll vendor/llvm/dist/test/CodeGen/X86/avx-intrinsics-fast-isel.ll vendor/llvm/dist/test/CodeGen/X86/avx.ll vendor/llvm/dist/test/CodeGen/X86/avx512-cmp-kor-sequence.ll vendor/llvm/dist/test/CodeGen/X86/avx512-cmp.ll vendor/llvm/dist/test/CodeGen/X86/avx512-cvt.ll vendor/llvm/dist/test/CodeGen/X86/avx512-ext.ll vendor/llvm/dist/test/CodeGen/X86/avx512-fsel.ll vendor/llvm/dist/test/CodeGen/X86/avx512-gather-scatter-intrin.ll vendor/llvm/dist/test/CodeGen/X86/avx512-i1test.ll vendor/llvm/dist/test/CodeGen/X86/avx512-insert-extract.ll vendor/llvm/dist/test/CodeGen/X86/avx512-insert-extract_i1.ll vendor/llvm/dist/test/CodeGen/X86/avx512-intrinsics-upgrade.ll vendor/llvm/dist/test/CodeGen/X86/avx512-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512-load-store.ll vendor/llvm/dist/test/CodeGen/X86/avx512-mask-op.ll vendor/llvm/dist/test/CodeGen/X86/avx512-mask-spills.ll vendor/llvm/dist/test/CodeGen/X86/avx512-memfold.ll vendor/llvm/dist/test/CodeGen/X86/avx512-regcall-NoMask.ll vendor/llvm/dist/test/CodeGen/X86/avx512-scalar_mask.ll vendor/llvm/dist/test/CodeGen/X86/avx512-select.ll vendor/llvm/dist/test/CodeGen/X86/avx512bw-intrinsics-upgrade.ll vendor/llvm/dist/test/CodeGen/X86/avx512bw-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512bwvl-intrinsics-upgrade.ll vendor/llvm/dist/test/CodeGen/X86/avx512cdvl-intrinsics-upgrade.ll vendor/llvm/dist/test/CodeGen/X86/avx512cdvl-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512dq-intrinsics-upgrade.ll vendor/llvm/dist/test/CodeGen/X86/avx512dq-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512dqvl-intrinsics-upgrade.ll vendor/llvm/dist/test/CodeGen/X86/avx512dqvl-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512er-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512ifma-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512ifmavl-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/avx512vl-intrinsics-upgrade.ll vendor/llvm/dist/test/CodeGen/X86/avx512vl-intrinsics.ll vendor/llvm/dist/test/CodeGen/X86/bswap_tree2.ll vendor/llvm/dist/test/CodeGen/X86/constant-combines.ll vendor/llvm/dist/test/CodeGen/X86/fast-isel-load-i1.ll vendor/llvm/dist/test/CodeGen/X86/fma-fneg-combine.ll vendor/llvm/dist/test/CodeGen/X86/fold-tied-op.ll vendor/llvm/dist/test/CodeGen/X86/fp128-i128.ll vendor/llvm/dist/test/CodeGen/X86/haddsub-2.ll vendor/llvm/dist/test/CodeGen/X86/madd.ll vendor/llvm/dist/test/CodeGen/X86/masked_gather_scatter.ll vendor/llvm/dist/test/CodeGen/X86/merge-consecutive-loads-128.ll vendor/llvm/dist/test/CodeGen/X86/misched-matrix.ll vendor/llvm/dist/test/CodeGen/X86/mul-i1024.ll vendor/llvm/dist/test/CodeGen/X86/mul-i256.ll vendor/llvm/dist/test/CodeGen/X86/mul-i512.ll vendor/llvm/dist/test/CodeGen/X86/oddshuffles.ll vendor/llvm/dist/test/CodeGen/X86/overflow.ll vendor/llvm/dist/test/CodeGen/X86/pmul.ll vendor/llvm/dist/test/CodeGen/X86/pr27591.ll vendor/llvm/dist/test/CodeGen/X86/pr28173.ll vendor/llvm/dist/test/CodeGen/X86/pr29112.ll vendor/llvm/dist/test/CodeGen/X86/pr31088.ll vendor/llvm/dist/test/CodeGen/X86/pr32241.ll vendor/llvm/dist/test/CodeGen/X86/pr32256.ll vendor/llvm/dist/test/CodeGen/X86/pr32284.ll vendor/llvm/dist/test/CodeGen/X86/pr32451.ll vendor/llvm/dist/test/CodeGen/X86/rotate.ll vendor/llvm/dist/test/CodeGen/X86/rtm.ll vendor/llvm/dist/test/CodeGen/X86/sad.ll vendor/llvm/dist/test/CodeGen/X86/select.ll vendor/llvm/dist/test/CodeGen/X86/setcc-wide-types.ll vendor/llvm/dist/test/CodeGen/X86/shrink_vmul_sse.ll vendor/llvm/dist/test/CodeGen/X86/sse-intrinsics-fast-isel.ll vendor/llvm/dist/test/CodeGen/X86/sse-scalar-fp-arith.ll vendor/llvm/dist/test/CodeGen/X86/sse1.ll vendor/llvm/dist/test/CodeGen/X86/sse3-avx-addsub-2.ll vendor/llvm/dist/test/CodeGen/X86/sse41.ll vendor/llvm/dist/test/CodeGen/X86/vec_int_to_fp.ll vendor/llvm/dist/test/CodeGen/X86/vector-bitreverse.ll vendor/llvm/dist/test/CodeGen/X86/vector-blend.ll vendor/llvm/dist/test/CodeGen/X86/vector-sqrt.ll vendor/llvm/dist/test/CodeGen/X86/x86-interleaved-access.ll vendor/llvm/dist/test/CodeGen/X86/xmulo.ll vendor/llvm/dist/test/CodeGen/X86/xor-select-i1-combine.ll vendor/llvm/dist/test/DebugInfo/PDB/pdbdump-headers.test vendor/llvm/dist/test/DebugInfo/llvm-symbolizer.test vendor/llvm/dist/test/Instrumentation/MemorySanitizer/csr.ll vendor/llvm/dist/test/LTO/X86/strip-debug-info.ll vendor/llvm/dist/test/MC/AMDGPU/exp.s vendor/llvm/dist/test/MC/Disassembler/AMDGPU/sopc_vi.txt vendor/llvm/dist/test/TableGen/GlobalISelEmitter.td vendor/llvm/dist/test/TableGen/intrinsic-varargs.td vendor/llvm/dist/test/Transforms/InstCombine/cast.ll vendor/llvm/dist/test/Transforms/InstCombine/lshr.ll vendor/llvm/dist/test/Transforms/InstCombine/memchr.ll vendor/llvm/dist/test/Transforms/InstCombine/set.ll vendor/llvm/dist/test/Transforms/InstSimplify/AndOrXor.ll vendor/llvm/dist/test/Transforms/JumpThreading/assume.ll vendor/llvm/dist/test/Transforms/JumpThreading/fold-not-thread.ll vendor/llvm/dist/test/Transforms/JumpThreading/guards.ll vendor/llvm/dist/test/Transforms/NewGVN/storeoverstore.ll vendor/llvm/dist/test/Transforms/SafeStack/X86/coloring-ssp.ll vendor/llvm/dist/test/Transforms/SafeStack/X86/coloring.ll vendor/llvm/dist/test/Transforms/SafeStack/X86/coloring2.ll vendor/llvm/dist/test/Transforms/SafeStack/X86/layout-frag.ll vendor/llvm/dist/tools/llc/llc.cpp vendor/llvm/dist/tools/llvm-cvtres/CMakeLists.txt vendor/llvm/dist/tools/llvm-cvtres/llvm-cvtres.cpp vendor/llvm/dist/tools/llvm-cvtres/llvm-cvtres.h vendor/llvm/dist/tools/llvm-dwp/llvm-dwp.cpp vendor/llvm/dist/tools/llvm-lto/llvm-lto.cpp vendor/llvm/dist/tools/llvm-pdbdump/Analyze.cpp vendor/llvm/dist/tools/llvm-pdbdump/CompactTypeDumpVisitor.cpp vendor/llvm/dist/tools/llvm-pdbdump/CompactTypeDumpVisitor.h vendor/llvm/dist/tools/llvm-pdbdump/LLVMOutputStyle.cpp vendor/llvm/dist/tools/llvm-pdbdump/LLVMOutputStyle.h vendor/llvm/dist/tools/llvm-pdbdump/PdbYaml.cpp vendor/llvm/dist/tools/llvm-pdbdump/YAMLOutputStyle.h vendor/llvm/dist/tools/llvm-pdbdump/YamlTypeDumper.cpp vendor/llvm/dist/tools/llvm-pdbdump/llvm-pdbdump.cpp vendor/llvm/dist/tools/llvm-readobj/COFFDumper.cpp vendor/llvm/dist/tools/opt/opt.cpp vendor/llvm/dist/unittests/ADT/APIntTest.cpp vendor/llvm/dist/unittests/DebugInfo/CodeView/RandomAccessVisitorTest.cpp vendor/llvm/dist/unittests/IR/AttributesTest.cpp vendor/llvm/dist/unittests/IR/ConstantsTest.cpp vendor/llvm/dist/unittests/IR/DominatorTreeTest.cpp vendor/llvm/dist/unittests/IR/InstructionsTest.cpp vendor/llvm/dist/unittests/IR/MetadataTest.cpp vendor/llvm/dist/unittests/Support/BinaryStreamTest.cpp vendor/llvm/dist/unittests/Support/CrashRecoveryTest.cpp vendor/llvm/dist/unittests/Support/ScaledNumberTest.cpp vendor/llvm/dist/unittests/Target/AArch64/InstSizes.cpp vendor/llvm/dist/unittests/Transforms/Utils/Cloning.cpp vendor/llvm/dist/utils/TableGen/CodeGenTarget.cpp vendor/llvm/dist/utils/TableGen/GlobalISelEmitter.cpp vendor/llvm/dist/utils/git-svn/git-llvm vendor/llvm/dist/utils/lit/lit/TestRunner.py Modified: vendor/llvm/dist/docs/CMake.rst ============================================================================== --- vendor/llvm/dist/docs/CMake.rst Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/docs/CMake.rst Mon May 22 19:43:28 2017 (r318663) @@ -186,8 +186,8 @@ CMake manual, or execute ``cmake --help- Sets the build type for ``make``-based generators. Possible values are Release, Debug, RelWithDebInfo and MinSizeRel. If you are using an IDE such as Visual Studio, you should use the IDE settings to set the build type. - Be aware that Release and RelWithDebInfo are not using the same optimization - level on most platform. + Be aware that Release and RelWithDebInfo use different optimization levels on + most platforms. **CMAKE_INSTALL_PREFIX**:PATH Path where LLVM will be installed if "make install" is invoked or the Modified: vendor/llvm/dist/docs/GetElementPtr.rst ============================================================================== --- vendor/llvm/dist/docs/GetElementPtr.rst Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/docs/GetElementPtr.rst Mon May 22 19:43:28 2017 (r318663) @@ -9,10 +9,11 @@ Introduction ============ This document seeks to dispel the mystery and confusion surrounding LLVM's -`GetElementPtr `_ (GEP) instruction. Questions -about the wily GEP instruction are probably the most frequently occurring -questions once a developer gets down to coding with LLVM. Here we lay out the -sources of confusion and show that the GEP instruction is really quite simple. +`GetElementPtr `_ (GEP) instruction. +Questions about the wily GEP instruction are probably the most frequently +occurring questions once a developer gets down to coding with LLVM. Here we lay +out the sources of confusion and show that the GEP instruction is really quite +simple. Address Computation =================== @@ -429,7 +430,8 @@ because LLVM has no restrictions on mixi LLVM's type-based alias analysis pass uses metadata to describe a different type system (such as the C type system), and performs type-based aliasing on top of -that. Further details are in the `language reference `_. +that. Further details are in the +`language reference `_. What happens if a GEP computation overflows? -------------------------------------------- Modified: vendor/llvm/dist/docs/LangRef.rst ============================================================================== --- vendor/llvm/dist/docs/LangRef.rst Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/docs/LangRef.rst Mon May 22 19:43:28 2017 (r318663) @@ -161,7 +161,7 @@ symbol table entries. Here is an example ; Definition of main function define i32 @main() { ; i32()* - ; Convert [13 x i8]* to i8 *... + ; Convert [13 x i8]* to i8*... %cast210 = getelementptr [13 x i8], [13 x i8]* @.str, i64 0, i64 0 ; Call puts function to write out the string to stdout. @@ -9548,7 +9548,7 @@ Syntax: :: - declare i8 *@llvm.returnaddress(i32 ) + declare i8* @llvm.returnaddress(i32 ) Overview: """"""""" @@ -9586,7 +9586,7 @@ Syntax: :: - declare i8 *@llvm.addressofreturnaddress() + declare i8* @llvm.addressofreturnaddress() Overview: """"""""" Modified: vendor/llvm/dist/include/llvm/ADT/APInt.h ============================================================================== --- vendor/llvm/dist/include/llvm/ADT/APInt.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/ADT/APInt.h Mon May 22 19:43:28 2017 (r318663) @@ -182,8 +182,9 @@ private: /// provides a more convenient form of divide for internal use since KnuthDiv /// has specific constraints on its inputs. If those constraints are not met /// then it provides a simpler form of divide. - static void divide(const APInt &LHS, unsigned lhsWords, const APInt &RHS, - unsigned rhsWords, APInt *Quotient, APInt *Remainder); + static void divide(const WordType *LHS, unsigned lhsWords, + const WordType *RHS, unsigned rhsWords, WordType *Quotient, + WordType *Remainder); /// out-of-line slow case for inline constructor void initSlowCase(uint64_t val, bool isSigned); @@ -1016,11 +1017,13 @@ public: /// /// \returns a new APInt value containing the division result APInt udiv(const APInt &RHS) const; + APInt udiv(uint64_t RHS) const; /// \brief Signed division function for APInt. /// /// Signed divide this APInt by APInt RHS. APInt sdiv(const APInt &RHS) const; + APInt sdiv(int64_t RHS) const; /// \brief Unsigned remainder operation. /// @@ -1032,11 +1035,13 @@ public: /// /// \returns a new APInt value containing the remainder result APInt urem(const APInt &RHS) const; + uint64_t urem(uint64_t RHS) const; /// \brief Function for signed remainder operation. /// /// Signed remainder operation on APInt. APInt srem(const APInt &RHS) const; + int64_t srem(int64_t RHS) const; /// \brief Dual division/remainder interface. /// @@ -1047,9 +1052,13 @@ public: /// udivrem(X, Y, X, Y), for example. static void udivrem(const APInt &LHS, const APInt &RHS, APInt &Quotient, APInt &Remainder); + static void udivrem(const APInt &LHS, uint64_t RHS, APInt &Quotient, + uint64_t &Remainder); static void sdivrem(const APInt &LHS, const APInt &RHS, APInt &Quotient, APInt &Remainder); + static void sdivrem(const APInt &LHS, int64_t RHS, APInt &Quotient, + int64_t &Remainder); // Operations that return overflow indicators. APInt sadd_ov(const APInt &RHS, bool &Overflow) const; @@ -2015,7 +2024,7 @@ inline APInt operator-(APInt a, const AP } inline APInt operator-(const APInt &a, APInt &&b) { - b = -std::move(b); + b.negate(); b += a; return std::move(b); } @@ -2026,7 +2035,7 @@ inline APInt operator-(APInt a, uint64_t } inline APInt operator-(uint64_t LHS, APInt b) { - b = -std::move(b); + b.negate(); b += LHS; return b; } Modified: vendor/llvm/dist/include/llvm/ADT/SmallPtrSet.h ============================================================================== --- vendor/llvm/dist/include/llvm/ADT/SmallPtrSet.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/ADT/SmallPtrSet.h Mon May 22 19:43:28 2017 (r318663) @@ -365,6 +365,8 @@ protected: public: using iterator = SmallPtrSetIterator; using const_iterator = SmallPtrSetIterator; + using key_type = ConstPtrType; + using value_type = PtrType; SmallPtrSetImpl(const SmallPtrSetImpl &) = delete; Modified: vendor/llvm/dist/include/llvm/ADT/Statistic.h ============================================================================== --- vendor/llvm/dist/include/llvm/ADT/Statistic.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/ADT/Statistic.h Mon May 22 19:43:28 2017 (r318663) @@ -101,6 +101,16 @@ public: return init(); } + void updateMax(unsigned V) { + unsigned PrevMax = Value.load(std::memory_order_relaxed); + // Keep trying to update max until we succeed or another thread produces + // a bigger max than us. + while (V > PrevMax && !Value.compare_exchange_weak( + PrevMax, V, std::memory_order_relaxed)) { + } + init(); + } + #else // Statistics are disabled in release builds. const Statistic &operator=(unsigned Val) { @@ -131,6 +141,8 @@ public: return *this; } + void updateMax(unsigned V) {} + #endif // !defined(NDEBUG) || defined(LLVM_ENABLE_STATS) protected: Modified: vendor/llvm/dist/include/llvm/ADT/Triple.h ============================================================================== --- vendor/llvm/dist/include/llvm/ADT/Triple.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/ADT/Triple.h Mon May 22 19:43:28 2017 (r318663) @@ -252,6 +252,10 @@ public: ObjectFormat == Other.ObjectFormat; } + bool operator!=(const Triple &Other) const { + return !(*this == Other); + } + /// @} /// @name Normalization /// @{ @@ -722,6 +726,12 @@ public: /// \returns true if the triple is little endian, false otherwise. bool isLittleEndian() const; + /// Test whether target triples are compatible. + bool isCompatibleWith(const Triple &Other) const; + + /// Merge target triples. + std::string merge(const Triple &Other) const; + /// @} /// @name Static helpers for IDs. /// @{ Modified: vendor/llvm/dist/include/llvm/Analysis/MemorySSA.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/MemorySSA.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/Analysis/MemorySSA.h Mon May 22 19:43:28 2017 (r318663) @@ -84,6 +84,7 @@ #include "llvm/Analysis/MemoryLocation.h" #include "llvm/Analysis/PHITransAddr.h" #include "llvm/IR/BasicBlock.h" +#include "llvm/IR/DerivedUser.h" #include "llvm/IR/Dominators.h" #include "llvm/IR/Module.h" #include "llvm/IR/OperandTraits.h" @@ -127,7 +128,7 @@ using const_memoryaccess_def_iterator = // \brief The base for all memory accesses. All memory accesses in a block are // linked together using an intrusive list. class MemoryAccess - : public User, + : public DerivedUser, public ilist_node>, public ilist_node> { public: @@ -145,15 +146,14 @@ public: MemoryAccess(const MemoryAccess &) = delete; MemoryAccess &operator=(const MemoryAccess &) = delete; - ~MemoryAccess() override; void *operator new(size_t, unsigned) = delete; void *operator new(size_t) = delete; BasicBlock *getBlock() const { return Block; } - virtual void print(raw_ostream &OS) const = 0; - virtual void dump() const; + void print(raw_ostream &OS) const; + void dump() const; /// \brief The user iterators for a memory access typedef user_iterator iterator; @@ -207,11 +207,12 @@ protected: /// \brief Used for debugging and tracking things about MemoryAccesses. /// Guaranteed unique among MemoryAccesses, no guarantees otherwise. - virtual unsigned getID() const = 0; + inline unsigned getID() const; - MemoryAccess(LLVMContext &C, unsigned Vty, BasicBlock *BB, - unsigned NumOperands) - : User(Type::getVoidTy(C), Vty, nullptr, NumOperands), Block(BB) {} + MemoryAccess(LLVMContext &C, unsigned Vty, DeleteValueTy DeleteValue, + BasicBlock *BB, unsigned NumOperands) + : DerivedUser(Type::getVoidTy(C), Vty, nullptr, NumOperands, DeleteValue), + Block(BB) {} private: BasicBlock *Block; @@ -248,21 +249,21 @@ public: // Sadly, these have to be public because they are needed in some of the // iterators. - virtual bool isOptimized() const = 0; - virtual MemoryAccess *getOptimized() const = 0; - virtual void setOptimized(MemoryAccess *) = 0; + inline bool isOptimized() const; + inline MemoryAccess *getOptimized() const; + inline void setOptimized(MemoryAccess *); /// \brief Reset the ID of what this MemoryUse was optimized to, causing it to /// be rewalked by the walker if necessary. /// This really should only be called by tests. - virtual void resetOptimized() = 0; + inline void resetOptimized(); protected: friend class MemorySSA; friend class MemorySSAUpdater; MemoryUseOrDef(LLVMContext &C, MemoryAccess *DMA, unsigned Vty, - Instruction *MI, BasicBlock *BB) - : MemoryAccess(C, Vty, BB, 1), MemoryInst(MI) { + DeleteValueTy DeleteValue, Instruction *MI, BasicBlock *BB) + : MemoryAccess(C, Vty, DeleteValue, BB, 1), MemoryInst(MI) { setDefiningAccess(DMA); } void setDefiningAccess(MemoryAccess *DMA, bool Optimized = false) { @@ -292,7 +293,8 @@ public: DECLARE_TRANSPARENT_OPERAND_ACCESSORS(MemoryAccess); MemoryUse(LLVMContext &C, MemoryAccess *DMA, Instruction *MI, BasicBlock *BB) - : MemoryUseOrDef(C, DMA, MemoryUseVal, MI, BB), OptimizedID(0) {} + : MemoryUseOrDef(C, DMA, MemoryUseVal, deleteMe, MI, BB), + OptimizedID(0) {} // allocate space for exactly one operand void *operator new(size_t s) { return User::operator new(s, 1); } @@ -302,32 +304,30 @@ public: return MA->getValueID() == MemoryUseVal; } - void print(raw_ostream &OS) const override; + void print(raw_ostream &OS) const; - virtual void setOptimized(MemoryAccess *DMA) override { + void setOptimized(MemoryAccess *DMA) { OptimizedID = DMA->getID(); setOperand(0, DMA); } - virtual bool isOptimized() const override { + bool isOptimized() const { return getDefiningAccess() && OptimizedID == getDefiningAccess()->getID(); } - virtual MemoryAccess *getOptimized() const override { + MemoryAccess *getOptimized() const { return getDefiningAccess(); } - virtual void resetOptimized() override { + void resetOptimized() { OptimizedID = INVALID_MEMORYACCESS_ID; } protected: friend class MemorySSA; - unsigned getID() const override { - llvm_unreachable("MemoryUses do not have IDs"); - } - private: + static void deleteMe(DerivedUser *Self); + unsigned int OptimizedID; }; @@ -350,8 +350,8 @@ public: MemoryDef(LLVMContext &C, MemoryAccess *DMA, Instruction *MI, BasicBlock *BB, unsigned Ver) - : MemoryUseOrDef(C, DMA, MemoryDefVal, MI, BB), ID(Ver), - Optimized(nullptr), OptimizedID(INVALID_MEMORYACCESS_ID) {} + : MemoryUseOrDef(C, DMA, MemoryDefVal, deleteMe, MI, BB), + ID(Ver), Optimized(nullptr), OptimizedID(INVALID_MEMORYACCESS_ID) {} // allocate space for exactly one operand void *operator new(size_t s) { return User::operator new(s, 1); } @@ -361,27 +361,28 @@ public: return MA->getValueID() == MemoryDefVal; } - virtual void setOptimized(MemoryAccess *MA) override { + void setOptimized(MemoryAccess *MA) { Optimized = MA; OptimizedID = getDefiningAccess()->getID(); } - virtual MemoryAccess *getOptimized() const override { return Optimized; } - virtual bool isOptimized() const override { + MemoryAccess *getOptimized() const { return Optimized; } + bool isOptimized() const { return getOptimized() && getDefiningAccess() && OptimizedID == getDefiningAccess()->getID(); } - virtual void resetOptimized() override { + void resetOptimized() { OptimizedID = INVALID_MEMORYACCESS_ID; } - void print(raw_ostream &OS) const override; + void print(raw_ostream &OS) const; -protected: friend class MemorySSA; - unsigned getID() const override { return ID; } + unsigned getID() const { return ID; } private: + static void deleteMe(DerivedUser *Self); + const unsigned ID; MemoryAccess *Optimized; unsigned int OptimizedID; @@ -432,7 +433,8 @@ public: DECLARE_TRANSPARENT_OPERAND_ACCESSORS(MemoryAccess); MemoryPhi(LLVMContext &C, BasicBlock *BB, unsigned Ver, unsigned NumPreds = 0) - : MemoryAccess(C, MemoryPhiVal, BB, 0), ID(Ver), ReservedSpace(NumPreds) { + : MemoryAccess(C, MemoryPhiVal, deleteMe, BB, 0), ID(Ver), + ReservedSpace(NumPreds) { allocHungoffUses(ReservedSpace); } @@ -534,7 +536,9 @@ public: return V->getValueID() == MemoryPhiVal; } - void print(raw_ostream &OS) const override; + void print(raw_ostream &OS) const; + + unsigned getID() const { return ID; } protected: friend class MemorySSA; @@ -546,8 +550,6 @@ protected: User::allocHungoffUses(N, /* IsPhi */ true); } - unsigned getID() const final { return ID; } - private: // For debugging only const unsigned ID; @@ -561,8 +563,45 @@ private: ReservedSpace = std::max(E + E / 2, 2u); growHungoffUses(ReservedSpace, /* IsPhi */ true); } + + static void deleteMe(DerivedUser *Self); }; +inline unsigned MemoryAccess::getID() const { + assert((isa(this) || isa(this)) && + "only memory defs and phis have ids"); + if (const auto *MD = dyn_cast(this)) + return MD->getID(); + return cast(this)->getID(); +} + +inline bool MemoryUseOrDef::isOptimized() const { + if (const auto *MD = dyn_cast(this)) + return MD->isOptimized(); + return cast(this)->isOptimized(); +} + +inline MemoryAccess *MemoryUseOrDef::getOptimized() const { + if (const auto *MD = dyn_cast(this)) + return MD->getOptimized(); + return cast(this)->getOptimized(); +} + +inline void MemoryUseOrDef::setOptimized(MemoryAccess *MA) { + if (auto *MD = dyn_cast(this)) + MD->setOptimized(MA); + else + cast(this)->setOptimized(MA); +} + +inline void MemoryUseOrDef::resetOptimized() { + if (auto *MD = dyn_cast(this)) + MD->resetOptimized(); + else + cast(this)->resetOptimized(); +} + + template <> struct OperandTraits : public HungoffOperandTraits<2> {}; DEFINE_TRANSPARENT_OPERAND_ACCESSORS(MemoryPhi, MemoryAccess) Modified: vendor/llvm/dist/include/llvm/Analysis/ScalarEvolution.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/ScalarEvolution.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/Analysis/ScalarEvolution.h Mon May 22 19:43:28 2017 (r318663) @@ -656,10 +656,12 @@ private: /// Test whether this BackedgeTakenInfo contains complete information. bool hasFullInfo() const { return isComplete(); } - /// Return an expression indicating the exact backedge-taken count of the - /// loop if it is known or SCEVCouldNotCompute otherwise. This is the - /// number of times the loop header can be guaranteed to execute, minus - /// one. + /// Return an expression indicating the exact *backedge-taken* + /// count of the loop if it is known or SCEVCouldNotCompute + /// otherwise. If execution makes it to the backedge on every + /// iteration (i.e. there are no abnormal exists like exception + /// throws and thread exits) then this is the number of times the + /// loop header will execute minus one. /// /// If the SCEV predicate associated with the answer can be different /// from AlwaysTrue, we must add a (non null) Predicates argument. @@ -1398,11 +1400,11 @@ public: const SCEV *getExitCount(const Loop *L, BasicBlock *ExitingBlock); /// If the specified loop has a predictable backedge-taken count, return it, - /// otherwise return a SCEVCouldNotCompute object. The backedge-taken count - /// is the number of times the loop header will be branched to from within - /// the loop. This is one less than the trip count of the loop, since it - /// doesn't count the first iteration, when the header is branched to from - /// outside the loop. + /// otherwise return a SCEVCouldNotCompute object. The backedge-taken count is + /// the number of times the loop header will be branched to from within the + /// loop, assuming there are no abnormal exists like exception throws. This is + /// one less than the trip count of the loop, since it doesn't count the first + /// iteration, when the header is branched to from outside the loop. /// /// Note that it is not valid to call this method on a loop without a /// loop-invariant backedge-taken count (see @@ -1417,8 +1419,10 @@ public: const SCEV *getPredicatedBackedgeTakenCount(const Loop *L, SCEVUnionPredicate &Predicates); - /// Similar to getBackedgeTakenCount, except return the least SCEV value - /// that is known never to be less than the actual backedge taken count. + /// When successful, this returns a SCEVConstant that is greater than or equal + /// to (i.e. a "conservative over-approximation") of the value returend by + /// getBackedgeTakenCount. If such a value cannot be computed, it returns the + /// SCEVCouldNotCompute object. const SCEV *getMaxBackedgeTakenCount(const Loop *L); /// Return true if the backedge taken count is either the value returned by Modified: vendor/llvm/dist/include/llvm/Analysis/TargetLibraryInfo.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/TargetLibraryInfo.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/Analysis/TargetLibraryInfo.h Mon May 22 19:43:28 2017 (r318663) @@ -191,6 +191,14 @@ public: void setShouldSignExtI32Param(bool Val) { ShouldSignExtI32Param = Val; } + + /// Returns the size of the wchar_t type in bytes. + unsigned getWCharSize(const Module &M) const; + + /// Returns size of the default wchar_t type on target \p T. This is mostly + /// intended to verify that the size in the frontend matches LLVM. All other + /// queries should use getWCharSize() instead. + static unsigned getTargetWCharSize(const Triple &T); }; /// Provides information about what library functions are available for @@ -307,6 +315,11 @@ public: return Attribute::None; } + /// \copydoc TargetLibraryInfoImpl::getWCharSize() + unsigned getWCharSize(const Module &M) const { + return Impl->getWCharSize(M); + } + /// Handle invalidation from the pass manager. /// /// If we try to invalidate this info, just return false. It cannot become Modified: vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h ============================================================================== --- vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/Analysis/ValueTracking.h Mon May 22 19:43:28 2017 (r318663) @@ -218,9 +218,38 @@ template class ArrayRef; DL); } - /// Returns true if the GEP is based on a pointer to a string (array of i8), - /// and is indexing into this string. - bool isGEPBasedOnPointerToString(const GEPOperator *GEP); + /// Returns true if the GEP is based on a pointer to a string (array of + // \p CharSize integers) and is indexing into this string. + bool isGEPBasedOnPointerToString(const GEPOperator *GEP, + unsigned CharSize = 8); + + /// Represents offset+length into a ConstantDataArray. + struct ConstantDataArraySlice { + /// ConstantDataArray pointer. nullptr indicates a zeroinitializer (a valid + /// initializer, it just doesn't fit the ConstantDataArray interface). + const ConstantDataArray *Array; + /// Slice starts at this Offset. + uint64_t Offset; + /// Length of the slice. + uint64_t Length; + + /// Moves the Offset and adjusts Length accordingly. + void move(uint64_t Delta) { + assert(Delta < Length); + Offset += Delta; + Length -= Delta; + } + /// Convenience accessor for elements in the slice. + uint64_t operator[](unsigned I) const { + return Array==nullptr ? 0 : Array->getElementAsInteger(I + Offset); + } + }; + + /// Returns true if the value \p V is a pointer into a ContantDataArray. + /// If successfull \p Index will point to a ConstantDataArray info object + /// with an apropriate offset. + bool getConstantDataArrayInfo(const Value *V, ConstantDataArraySlice &Slice, + unsigned ElementSize, uint64_t Offset = 0); /// This function computes the length of a null-terminated C string pointed to /// by V. If successful, it returns true and returns the string in Str. If @@ -233,7 +262,7 @@ template class ArrayRef; /// If we can compute the length of the string pointed to by the specified /// pointer, return 'len+1'. If we can't, return 0. - uint64_t GetStringLength(const Value *V); + uint64_t GetStringLength(const Value *V, unsigned CharSize = 8); /// This method strips off any GEP address adjustments and pointer casts from /// the specified value, returning the original object being addressed. Note Modified: vendor/llvm/dist/include/llvm/CodeGen/LiveIntervalAnalysis.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/LiveIntervalAnalysis.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/CodeGen/LiveIntervalAnalysis.h Mon May 22 19:43:28 2017 (r318663) @@ -189,7 +189,7 @@ extern cl::opt UseSegmentSetForPhy void pruneValue(LiveRange &LR, SlotIndex Kill, SmallVectorImpl *EndPoints); - /// This function should be used. Its intend is to tell you that + /// This function should not be used. Its intend is to tell you that /// you are doing something wrong if you call pruveValue directly on a /// LiveInterval. Indeed, you are supposed to call pruneValue on the main /// LiveRange and all the LiveRange of the subranges if any. Modified: vendor/llvm/dist/include/llvm/CodeGen/MachineValueType.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/MachineValueType.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/CodeGen/MachineValueType.h Mon May 22 19:43:28 2017 (r318663) @@ -56,117 +56,119 @@ class MVT { FIRST_FP_VALUETYPE = f16, LAST_FP_VALUETYPE = ppcf128, - v2i1 = 14, // 2 x i1 - v4i1 = 15, // 4 x i1 - v8i1 = 16, // 8 x i1 - v16i1 = 17, // 16 x i1 - v32i1 = 18, // 32 x i1 - v64i1 = 19, // 64 x i1 - v512i1 = 20, // 512 x i1 - v1024i1 = 21, // 1024 x i1 - - v1i8 = 22, // 1 x i8 - v2i8 = 23, // 2 x i8 - v4i8 = 24, // 4 x i8 - v8i8 = 25, // 8 x i8 - v16i8 = 26, // 16 x i8 - v32i8 = 27, // 32 x i8 - v64i8 = 28, // 64 x i8 - v128i8 = 29, //128 x i8 - v256i8 = 30, //256 x i8 - - v1i16 = 31, // 1 x i16 - v2i16 = 32, // 2 x i16 - v4i16 = 33, // 4 x i16 - v8i16 = 34, // 8 x i16 - v16i16 = 35, // 16 x i16 - v32i16 = 36, // 32 x i16 - v64i16 = 37, // 64 x i16 - v128i16 = 38, //128 x i16 - - v1i32 = 39, // 1 x i32 - v2i32 = 40, // 2 x i32 - v4i32 = 41, // 4 x i32 - v8i32 = 42, // 8 x i32 - v16i32 = 43, // 16 x i32 - v32i32 = 44, // 32 x i32 - v64i32 = 45, // 64 x i32 - - v1i64 = 46, // 1 x i64 - v2i64 = 47, // 2 x i64 - v4i64 = 48, // 4 x i64 - v8i64 = 49, // 8 x i64 - v16i64 = 50, // 16 x i64 - v32i64 = 51, // 32 x i64 + v1i1 = 14, // 1 x i1 + v2i1 = 15, // 2 x i1 + v4i1 = 16, // 4 x i1 + v8i1 = 17, // 8 x i1 + v16i1 = 18, // 16 x i1 + v32i1 = 19, // 32 x i1 + v64i1 = 20, // 64 x i1 + v512i1 = 21, // 512 x i1 + v1024i1 = 22, // 1024 x i1 + + v1i8 = 23, // 1 x i8 + v2i8 = 24, // 2 x i8 + v4i8 = 25, // 4 x i8 + v8i8 = 26, // 8 x i8 + v16i8 = 27, // 16 x i8 + v32i8 = 28, // 32 x i8 + v64i8 = 29, // 64 x i8 + v128i8 = 30, //128 x i8 + v256i8 = 31, //256 x i8 + + v1i16 = 32, // 1 x i16 + v2i16 = 33, // 2 x i16 + v4i16 = 34, // 4 x i16 + v8i16 = 35, // 8 x i16 + v16i16 = 36, // 16 x i16 + v32i16 = 37, // 32 x i16 + v64i16 = 38, // 64 x i16 + v128i16 = 39, //128 x i16 + + v1i32 = 40, // 1 x i32 + v2i32 = 41, // 2 x i32 + v4i32 = 42, // 4 x i32 + v8i32 = 43, // 8 x i32 + v16i32 = 44, // 16 x i32 + v32i32 = 45, // 32 x i32 + v64i32 = 46, // 64 x i32 + + v1i64 = 47, // 1 x i64 + v2i64 = 48, // 2 x i64 + v4i64 = 49, // 4 x i64 + v8i64 = 50, // 8 x i64 + v16i64 = 51, // 16 x i64 + v32i64 = 52, // 32 x i64 - v1i128 = 52, // 1 x i128 + v1i128 = 53, // 1 x i128 // Scalable integer types - nxv2i1 = 53, // n x 2 x i1 - nxv4i1 = 54, // n x 4 x i1 - nxv8i1 = 55, // n x 8 x i1 - nxv16i1 = 56, // n x 16 x i1 - nxv32i1 = 57, // n x 32 x i1 - - nxv1i8 = 58, // n x 1 x i8 - nxv2i8 = 59, // n x 2 x i8 - nxv4i8 = 60, // n x 4 x i8 - nxv8i8 = 61, // n x 8 x i8 - nxv16i8 = 62, // n x 16 x i8 - nxv32i8 = 63, // n x 32 x i8 - - nxv1i16 = 64, // n x 1 x i16 - nxv2i16 = 65, // n x 2 x i16 - nxv4i16 = 66, // n x 4 x i16 - nxv8i16 = 67, // n x 8 x i16 - nxv16i16 = 68, // n x 16 x i16 - nxv32i16 = 69, // n x 32 x i16 - - nxv1i32 = 70, // n x 1 x i32 - nxv2i32 = 71, // n x 2 x i32 - nxv4i32 = 72, // n x 4 x i32 - nxv8i32 = 73, // n x 8 x i32 - nxv16i32 = 74, // n x 16 x i32 - nxv32i32 = 75, // n x 32 x i32 - - nxv1i64 = 76, // n x 1 x i64 - nxv2i64 = 77, // n x 2 x i64 - nxv4i64 = 78, // n x 4 x i64 - nxv8i64 = 79, // n x 8 x i64 - nxv16i64 = 80, // n x 16 x i64 - nxv32i64 = 81, // n x 32 x i64 + nxv1i1 = 54, // n x 1 x i1 + nxv2i1 = 55, // n x 2 x i1 + nxv4i1 = 56, // n x 4 x i1 + nxv8i1 = 57, // n x 8 x i1 + nxv16i1 = 58, // n x 16 x i1 + nxv32i1 = 59, // n x 32 x i1 + + nxv1i8 = 60, // n x 1 x i8 + nxv2i8 = 61, // n x 2 x i8 + nxv4i8 = 62, // n x 4 x i8 + nxv8i8 = 63, // n x 8 x i8 + nxv16i8 = 64, // n x 16 x i8 + nxv32i8 = 65, // n x 32 x i8 + + nxv1i16 = 66, // n x 1 x i16 + nxv2i16 = 67, // n x 2 x i16 + nxv4i16 = 68, // n x 4 x i16 + nxv8i16 = 69, // n x 8 x i16 + nxv16i16 = 70, // n x 16 x i16 + nxv32i16 = 71, // n x 32 x i16 + + nxv1i32 = 72, // n x 1 x i32 + nxv2i32 = 73, // n x 2 x i32 + nxv4i32 = 74, // n x 4 x i32 + nxv8i32 = 75, // n x 8 x i32 + nxv16i32 = 76, // n x 16 x i32 + nxv32i32 = 77, // n x 32 x i32 + + nxv1i64 = 78, // n x 1 x i64 + nxv2i64 = 79, // n x 2 x i64 + nxv4i64 = 80, // n x 4 x i64 + nxv8i64 = 81, // n x 8 x i64 + nxv16i64 = 82, // n x 16 x i64 + nxv32i64 = 83, // n x 32 x i64 - FIRST_INTEGER_VECTOR_VALUETYPE = v2i1, + FIRST_INTEGER_VECTOR_VALUETYPE = v1i1, LAST_INTEGER_VECTOR_VALUETYPE = nxv32i64, - FIRST_INTEGER_SCALABLE_VALUETYPE = nxv2i1, + FIRST_INTEGER_SCALABLE_VALUETYPE = nxv1i1, LAST_INTEGER_SCALABLE_VALUETYPE = nxv32i64, - v2f16 = 82, // 2 x f16 - v4f16 = 83, // 4 x f16 - v8f16 = 84, // 8 x f16 - v1f32 = 85, // 1 x f32 - v2f32 = 86, // 2 x f32 - v4f32 = 87, // 4 x f32 - v8f32 = 88, // 8 x f32 - v16f32 = 89, // 16 x f32 - v1f64 = 90, // 1 x f64 - v2f64 = 91, // 2 x f64 - v4f64 = 92, // 4 x f64 - v8f64 = 93, // 8 x f64 - - nxv2f16 = 94, // n x 2 x f16 - nxv4f16 = 95, // n x 4 x f16 - nxv8f16 = 96, // n x 8 x f16 - nxv1f32 = 97, // n x 1 x f32 - nxv2f32 = 98, // n x 2 x f32 - nxv4f32 = 99, // n x 4 x f32 - nxv8f32 = 100, // n x 8 x f32 - nxv16f32 = 101, // n x 16 x f32 - nxv1f64 = 102, // n x 1 x f64 - nxv2f64 = 103, // n x 2 x f64 - nxv4f64 = 104, // n x 4 x f64 - nxv8f64 = 105, // n x 8 x f64 + v2f16 = 84, // 2 x f16 + v4f16 = 85, // 4 x f16 + v8f16 = 86, // 8 x f16 + v1f32 = 87, // 1 x f32 + v2f32 = 88, // 2 x f32 + v4f32 = 89, // 4 x f32 + v8f32 = 90, // 8 x f32 + v16f32 = 91, // 16 x f32 + v1f64 = 92, // 1 x f64 + v2f64 = 93, // 2 x f64 + v4f64 = 94, // 4 x f64 + v8f64 = 95, // 8 x f64 + + nxv2f16 = 96, // n x 2 x f16 + nxv4f16 = 97, // n x 4 x f16 + nxv8f16 = 98, // n x 8 x f16 + nxv1f32 = 99, // n x 1 x f32 + nxv2f32 = 100, // n x 2 x f32 + nxv4f32 = 101, // n x 4 x f32 + nxv8f32 = 102, // n x 8 x f32 + nxv16f32 = 103, // n x 16 x f32 + nxv1f64 = 104, // n x 1 x f64 + nxv2f64 = 105, // n x 2 x f64 + nxv4f64 = 106, // n x 4 x f64 + nxv8f64 = 107, // n x 8 x f64 FIRST_FP_VECTOR_VALUETYPE = v2f16, LAST_FP_VECTOR_VALUETYPE = nxv8f64, @@ -174,21 +176,21 @@ class MVT { FIRST_FP_SCALABLE_VALUETYPE = nxv2f16, LAST_FP_SCALABLE_VALUETYPE = nxv8f64, - FIRST_VECTOR_VALUETYPE = v2i1, + FIRST_VECTOR_VALUETYPE = v1i1, LAST_VECTOR_VALUETYPE = nxv8f64, - x86mmx = 106, // This is an X86 MMX value + x86mmx = 108, // This is an X86 MMX value - Glue = 107, // This glues nodes together during pre-RA sched + Glue = 109, // This glues nodes together during pre-RA sched - isVoid = 108, // This has no value + isVoid = 110, // This has no value - Untyped = 109, // This value takes a register, but has + Untyped = 111, // This value takes a register, but has // unspecified type. The register class // will be determined by the opcode. FIRST_VALUETYPE = 1, // This is always the beginning of the list. - LAST_VALUETYPE = 110, // This always remains at the end of the list. + LAST_VALUETYPE = 112, // This always remains at the end of the list. // This is the current maximum for LAST_VALUETYPE. // MVT::MAX_ALLOWED_VALUETYPE is used for asserts and to size bit vectors @@ -411,6 +413,7 @@ class MVT { switch (SimpleTy) { default: llvm_unreachable("Not a vector MVT!"); + case v1i1: case v2i1: case v4i1: case v8i1: @@ -419,6 +422,7 @@ class MVT { case v64i1: case v512i1: case v1024i1: + case nxv1i1: case nxv2i1: case nxv4i1: case nxv8i1: @@ -589,6 +593,7 @@ class MVT { case nxv2f16: case nxv2f32: case nxv2f64: return 2; + case v1i1: case v1i8: case v1i16: case v1i32: @@ -596,6 +601,7 @@ class MVT { case v1i128: case v1f32: case v1f64: + case nxv1i1: case nxv1i8: case nxv1i16: case nxv1i32: @@ -628,7 +634,9 @@ class MVT { "in codegen and has no size"); case Metadata: llvm_unreachable("Value type is metadata."); - case i1 : return 1; + case i1: + case v1i1: + case nxv1i1: return 1; case v2i1: case nxv2i1: return 2; case v4i1: @@ -814,6 +822,7 @@ class MVT { default: break; case MVT::i1: + if (NumElements == 1) return MVT::v1i1; if (NumElements == 2) return MVT::v2i1; if (NumElements == 4) return MVT::v4i1; if (NumElements == 8) return MVT::v8i1; @@ -891,6 +900,7 @@ class MVT { default: break; case MVT::i1: + if (NumElements == 1) return MVT::nxv1i1; if (NumElements == 2) return MVT::nxv2i1; if (NumElements == 4) return MVT::nxv4i1; if (NumElements == 8) return MVT::nxv8i1; Modified: vendor/llvm/dist/include/llvm/CodeGen/Passes.h ============================================================================== --- vendor/llvm/dist/include/llvm/CodeGen/Passes.h Mon May 22 19:40:56 2017 (r318662) +++ vendor/llvm/dist/include/llvm/CodeGen/Passes.h Mon May 22 19:43:28 2017 (r318663) @@ -33,7 +33,7 @@ class raw_ostream; /// List of target independent CodeGen pass IDs. namespace llvm { - FunctionPass *createAtomicExpandPass(const TargetMachine *TM); + FunctionPass *createAtomicExpandPass(); /// createUnreachableBlockEliminationPass - The LLVM code generator does not /// work well with unreachable basic blocks (what live ranges make sense for a @@ -66,7 +66,7 @@ namespace llvm { /// createCodeGenPreparePass - Transform the code to expose more pattern /// matching during instruction selection. - FunctionPass *createCodeGenPreparePass(const TargetMachine *TM = nullptr); + FunctionPass *createCodeGenPreparePass(); /// createScalarizeMaskedMemIntrinPass - Replace masked load, store, gather /// and scatter intrinsics with scalar code when target doesn't support them. @@ -133,10 +133,6 @@ namespace llvm { // instruction and update the MachineFunctionInfo with that information. extern char &ShrinkWrapID; - /// LiveRangeShrink pass. Move instruction close to its definition to shrink - /// the definition's live range. - extern char &LiveRangeShrinkID; - /// Greedy register allocator. extern char &RAGreedyID; @@ -177,7 +173,7 @@ namespace llvm { /// PrologEpilogCodeInserter - This pass inserts prolog and epilog code, /// and eliminates abstract frame references. extern char &PrologEpilogCodeInserterID; - MachineFunctionPass *createPrologEpilogInserterPass(const TargetMachine *TM); + MachineFunctionPass *createPrologEpilogInserterPass(); /// ExpandPostRAPseudos - This pass expands pseudo instructions after /// register allocation. @@ -305,7 +301,7 @@ namespace llvm { /// createStackProtectorPass - This pass adds stack protectors to functions. /// - FunctionPass *createStackProtectorPass(const TargetMachine *TM); + FunctionPass *createStackProtectorPass(); /// createMachineVerifierPass - This pass verifies cenerated machine code /// instructions for correctness. @@ -314,11 +310,11 @@ namespace llvm { /// createDwarfEHPass - This pass mulches exception handling code into a form /// adapted to code generation. Required if using dwarf exception handling. - FunctionPass *createDwarfEHPass(const TargetMachine *TM); + FunctionPass *createDwarfEHPass(); /// createWinEHPass - Prepares personality functions used by MSVC on Windows, /// in addition to the Itanium LSDA based personalities. - FunctionPass *createWinEHPass(const TargetMachine *TM); + FunctionPass *createWinEHPass(); /// createSjLjEHPreparePass - This pass adapts exception handling code to use /// the GCC-style builtin setjmp/longjmp (sjlj) to handling EH control flow. @@ -362,12 +358,12 @@ namespace llvm { /// InterleavedAccess Pass - This pass identifies and matches interleaved /// memory accesses to target specific intrinsics. /// - FunctionPass *createInterleavedAccessPass(const TargetMachine *TM); + FunctionPass *createInterleavedAccessPass(); /// LowerEmuTLS - This pass generates __emutls_[vt].xyz variables for all /// TLS variables for the emulated TLS model. /// - ModulePass *createLowerEmuTLSPass(const TargetMachine *TM); + ModulePass *createLowerEmuTLSPass(); /// This pass lowers the @llvm.load.relative intrinsic to instructions. /// This is unsafe to do earlier because a pass may combine the constant @@ -384,7 +380,7 @@ namespace llvm { /// This pass splits the stack into a safe stack and an unsafe stack to /// protect against stack-based overflow vulnerabilities. *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Mon May 22 19:43:37 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 034CFD78962; Mon, 22 May 2017 19:43:37 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE0AF1869; Mon, 22 May 2017 19:43:36 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJhZWP076500; Mon, 22 May 2017 19:43:35 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJhZfM076499; Mon, 22 May 2017 19:43:35 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221943.v4MJhZfM076499@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:43:35 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318664 - vendor/llvm/llvm-trunk-r303571 X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:43:37 -0000 Author: dim Date: Mon May 22 19:43:35 2017 New Revision: 318664 URL: https://svnweb.freebsd.org/changeset/base/318664 Log: Tag llvm trunk r303571. Added: vendor/llvm/llvm-trunk-r303571/ - copied from r318663, vendor/llvm/dist/ From owner-svn-src-vendor@freebsd.org Mon May 22 19:43:55 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B53AD78A20; Mon, 22 May 2017 19:43:55 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C8BC019CE; Mon, 22 May 2017 19:43:54 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJhrr6076640; Mon, 22 May 2017 19:43:53 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJhrCp076639; Mon, 22 May 2017 19:43:53 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221943.v4MJhrCp076639@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:43:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318666 - vendor/clang/clang-trunk-r303571 X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:43:55 -0000 Author: dim Date: Mon May 22 19:43:53 2017 New Revision: 318666 URL: https://svnweb.freebsd.org/changeset/base/318666 Log: Tag clang trunk r303571. Added: vendor/clang/clang-trunk-r303571/ - copied from r318665, vendor/clang/dist/ From owner-svn-src-vendor@freebsd.org Mon May 22 19:43:50 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CB84D789EE; Mon, 22 May 2017 19:43:50 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E8CD21963; Mon, 22 May 2017 19:43:49 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJhmjO076593; Mon, 22 May 2017 19:43:48 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJhjdW076562; Mon, 22 May 2017 19:43:45 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221943.v4MJhjdW076562@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:43:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318665 - in vendor/clang/dist: include/clang/AST include/clang/Basic include/clang/Format include/clang/Lex include/clang/Parse include/clang/Sema include/clang/Serialization lib/AST l... X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:43:50 -0000 Author: dim Date: Mon May 22 19:43:45 2017 New Revision: 318665 URL: https://svnweb.freebsd.org/changeset/base/318665 Log: Vendor import of clang trunk r303571: https://llvm.org/svn/llvm-project/cfe/trunk@303571 Added: vendor/clang/dist/test/CodeGen/micromips-attr.c (contents, props changed) vendor/clang/dist/test/CodeGen/wchar-size.c (contents, props changed) vendor/clang/dist/test/CodeGenCXX/amdgcn-automatic-variable.cpp (contents, props changed) vendor/clang/dist/test/CodeGenOpenCL/amdgcn-automatic-variable.cl vendor/clang/dist/test/Modules/extern_cxx.cpp (contents, props changed) vendor/clang/dist/test/Modules/macro-redefinition.cpp (contents, props changed) vendor/clang/dist/test/Sema/attr-micromips.c (contents, props changed) vendor/clang/dist/test/SemaCXX/Inputs/nullability-completeness.h (contents, props changed) Modified: vendor/clang/dist/include/clang/AST/ASTContext.h vendor/clang/dist/include/clang/AST/CXXInheritance.h vendor/clang/dist/include/clang/AST/DeclBase.h vendor/clang/dist/include/clang/AST/Type.h vendor/clang/dist/include/clang/Basic/AddressSpaces.h vendor/clang/dist/include/clang/Basic/Attr.td vendor/clang/dist/include/clang/Basic/AttrDocs.td vendor/clang/dist/include/clang/Basic/DiagnosticSemaKinds.td vendor/clang/dist/include/clang/Basic/LangOptions.h vendor/clang/dist/include/clang/Format/Format.h vendor/clang/dist/include/clang/Lex/ModuleMap.h vendor/clang/dist/include/clang/Lex/Preprocessor.h vendor/clang/dist/include/clang/Parse/Parser.h vendor/clang/dist/include/clang/Sema/DeclSpec.h vendor/clang/dist/include/clang/Sema/Sema.h vendor/clang/dist/include/clang/Serialization/ASTReader.h vendor/clang/dist/include/clang/Serialization/ASTWriter.h vendor/clang/dist/lib/AST/ASTContext.cpp vendor/clang/dist/lib/AST/CXXInheritance.cpp vendor/clang/dist/lib/AST/DeclBase.cpp vendor/clang/dist/lib/AST/ExprConstant.cpp vendor/clang/dist/lib/AST/Type.cpp vendor/clang/dist/lib/AST/TypePrinter.cpp vendor/clang/dist/lib/Basic/Targets.cpp vendor/clang/dist/lib/CodeGen/CGAtomic.cpp vendor/clang/dist/lib/CodeGen/CGBlocks.cpp vendor/clang/dist/lib/CodeGen/CGBuiltin.cpp vendor/clang/dist/lib/CodeGen/CGClass.cpp vendor/clang/dist/lib/CodeGen/CGDebugInfo.cpp vendor/clang/dist/lib/CodeGen/CGDecl.cpp vendor/clang/dist/lib/CodeGen/CGExpr.cpp vendor/clang/dist/lib/CodeGen/CGExprCXX.cpp vendor/clang/dist/lib/CodeGen/CGExprScalar.cpp vendor/clang/dist/lib/CodeGen/CGObjC.cpp vendor/clang/dist/lib/CodeGen/CGObjCRuntime.cpp vendor/clang/dist/lib/CodeGen/CGOpenMPRuntime.cpp vendor/clang/dist/lib/CodeGen/CGStmtOpenMP.cpp vendor/clang/dist/lib/CodeGen/CGValue.h vendor/clang/dist/lib/CodeGen/CodeGenFunction.cpp vendor/clang/dist/lib/CodeGen/CodeGenFunction.h vendor/clang/dist/lib/CodeGen/CodeGenModule.cpp vendor/clang/dist/lib/CodeGen/CodeGenTypeCache.h vendor/clang/dist/lib/CodeGen/TargetInfo.cpp vendor/clang/dist/lib/CodeGen/TargetInfo.h vendor/clang/dist/lib/Format/BreakableToken.cpp vendor/clang/dist/lib/Format/ContinuationIndenter.cpp vendor/clang/dist/lib/Format/Format.cpp vendor/clang/dist/lib/Format/TokenAnnotator.cpp vendor/clang/dist/lib/Format/UnwrappedLineFormatter.cpp vendor/clang/dist/lib/Format/UnwrappedLineParser.cpp vendor/clang/dist/lib/Index/IndexBody.cpp vendor/clang/dist/lib/Index/IndexDecl.cpp vendor/clang/dist/lib/Index/IndexSymbol.cpp vendor/clang/dist/lib/Index/USRGeneration.cpp vendor/clang/dist/lib/Lex/ModuleMap.cpp vendor/clang/dist/lib/Lex/PPDirectives.cpp vendor/clang/dist/lib/Lex/PPLexerChange.cpp vendor/clang/dist/lib/Parse/ParseCXXInlineMethods.cpp vendor/clang/dist/lib/Parse/ParseDecl.cpp vendor/clang/dist/lib/Parse/ParseDeclCXX.cpp vendor/clang/dist/lib/Parse/ParseExpr.cpp vendor/clang/dist/lib/Parse/ParseExprCXX.cpp vendor/clang/dist/lib/Parse/ParseOpenMP.cpp vendor/clang/dist/lib/Parse/ParsePragma.cpp vendor/clang/dist/lib/Parse/ParseStmt.cpp vendor/clang/dist/lib/Parse/ParseTemplate.cpp vendor/clang/dist/lib/Parse/ParseTentative.cpp vendor/clang/dist/lib/Parse/Parser.cpp vendor/clang/dist/lib/Sema/SemaDecl.cpp vendor/clang/dist/lib/Sema/SemaDeclAttr.cpp vendor/clang/dist/lib/Sema/SemaLookup.cpp vendor/clang/dist/lib/Sema/SemaTemplate.cpp vendor/clang/dist/lib/Sema/SemaTemplateInstantiateDecl.cpp vendor/clang/dist/lib/Sema/SemaType.cpp vendor/clang/dist/lib/Serialization/ASTReader.cpp vendor/clang/dist/lib/Serialization/ASTReaderDecl.cpp vendor/clang/dist/lib/Serialization/ASTWriter.cpp vendor/clang/dist/lib/Serialization/ASTWriterDecl.cpp vendor/clang/dist/lib/StaticAnalyzer/Core/BugReporter.cpp vendor/clang/dist/lib/StaticAnalyzer/Frontend/AnalysisConsumer.cpp vendor/clang/dist/test/CodeGen/address-space.c vendor/clang/dist/test/CodeGenCUDA/flush-denormals.cu vendor/clang/dist/test/CodeGenCXX/strict-vtable-pointers.cpp vendor/clang/dist/test/CodeGenOpenCL/amdgpu-alignment.cl vendor/clang/dist/test/CodeGenOpenCL/amdgpu-debug-info-pointer-address-space.cl vendor/clang/dist/test/CodeGenOpenCL/amdgpu-debug-info-variable-expression.cl vendor/clang/dist/test/CodeGenOpenCL/amdgpu-nullptr.cl vendor/clang/dist/test/CodeGenOpenCL/builtins-amdgcn.cl vendor/clang/dist/test/CodeGenOpenCL/byval.cl vendor/clang/dist/test/CodeGenOpenCL/size_t.cl vendor/clang/dist/test/Index/Core/external-source-symbol-attr.m vendor/clang/dist/test/Index/Core/index-dependent-source.cpp vendor/clang/dist/test/Index/Core/index-source.cpp vendor/clang/dist/test/Index/Core/index-source.m vendor/clang/dist/test/Misc/pragma-attribute-supported-attributes-list.test vendor/clang/dist/test/Modules/preprocess-module.cpp vendor/clang/dist/test/Parser/cxx-template-argument.cpp vendor/clang/dist/test/Parser/cxx0x-ambig.cpp vendor/clang/dist/test/Parser/cxx0x-decl.cpp vendor/clang/dist/test/Sema/block-args.c vendor/clang/dist/test/Sema/integer-overflow.c vendor/clang/dist/test/Sema/sizeof-struct-non-zero-as-member.cl vendor/clang/dist/test/SemaCXX/lambda-expressions.cpp vendor/clang/dist/test/SemaCXX/local-classes.cpp vendor/clang/dist/test/SemaCXX/modules-ts.cppm vendor/clang/dist/test/SemaCXX/nullability.cpp vendor/clang/dist/test/SemaCXX/warn-shadow.cpp vendor/clang/dist/test/SemaObjC/unguarded-availability.m vendor/clang/dist/test/SemaOpenCL/storageclass-cl20.cl vendor/clang/dist/test/SemaOpenCL/storageclass.cl vendor/clang/dist/tools/libclang/CIndex.cpp vendor/clang/dist/unittests/Format/FormatTest.cpp vendor/clang/dist/unittests/Format/FormatTestComments.cpp vendor/clang/dist/unittests/Format/FormatTestJS.cpp vendor/clang/dist/unittests/Format/FormatTestSelective.cpp Modified: vendor/clang/dist/include/clang/AST/ASTContext.h ============================================================================== --- vendor/clang/dist/include/clang/AST/ASTContext.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/AST/ASTContext.h Mon May 22 19:43:45 2017 (r318665) @@ -935,7 +935,7 @@ public: /// \brief Get the additional modules in which the definition \p Def has /// been merged. - ArrayRef getModulesWithMergedDefinition(NamedDecl *Def) { + ArrayRef getModulesWithMergedDefinition(const NamedDecl *Def) { auto MergedIt = MergedDefModules.find(Def); if (MergedIt == MergedDefModules.end()) return None; @@ -2324,8 +2324,7 @@ public: uint64_t getTargetNullPointerValue(QualType QT) const; bool addressSpaceMapManglingFor(unsigned AS) const { - return AddrSpaceMapMangling || - AS >= LangAS::Count; + return AddrSpaceMapMangling || AS >= LangAS::FirstTargetAddressSpace; } private: Modified: vendor/clang/dist/include/clang/AST/CXXInheritance.h ============================================================================== --- vendor/clang/dist/include/clang/AST/CXXInheritance.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/AST/CXXInheritance.h Mon May 22 19:43:45 2017 (r318665) @@ -127,7 +127,11 @@ class CXXBasePaths { /// class subobjects for that class type. The key of the map is /// the cv-unqualified canonical type of the base class subobject. llvm::SmallDenseMap, 8> ClassSubobjects; - + + /// VisitedDependentRecords - Records the dependent records that have been + /// already visited. + llvm::SmallDenseSet VisitedDependentRecords; + /// FindAmbiguities - Whether Sema::IsDerivedFrom should try find /// ambiguous paths while it is looking for a path from a derived /// type to a base type. Modified: vendor/clang/dist/include/clang/AST/DeclBase.h ============================================================================== --- vendor/clang/dist/include/clang/AST/DeclBase.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/AST/DeclBase.h Mon May 22 19:43:45 2017 (r318665) @@ -34,6 +34,7 @@ class DeclarationName; class DependentDiagnostic; class EnumDecl; class ExportDecl; +class ExternalSourceSymbolAttr; class FunctionDecl; class FunctionType; enum Linkage : unsigned char; @@ -332,15 +333,15 @@ private: bool AccessDeclContextSanity() const; protected: - Decl(Kind DK, DeclContext *DC, SourceLocation L) - : NextInContextAndBits(), DeclCtx(DC), - Loc(L), DeclKind(DK), InvalidDecl(0), - HasAttrs(false), Implicit(false), Used(false), Referenced(false), - Access(AS_none), FromASTFile(0), Hidden(DC && cast(DC)->Hidden), - IdentifierNamespace(getIdentifierNamespaceForKind(DK)), - CacheValidAndLinkage(0) - { + : NextInContextAndBits(), DeclCtx(DC), Loc(L), DeclKind(DK), + InvalidDecl(0), HasAttrs(false), Implicit(false), Used(false), + Referenced(false), Access(AS_none), FromASTFile(0), + Hidden(DC && cast(DC)->Hidden && + (!cast(DC)->isFromASTFile() || + hasLocalOwningModuleStorage())), + IdentifierNamespace(getIdentifierNamespaceForKind(DK)), + CacheValidAndLinkage(0) { if (StatisticsEnabled) add(DK); } @@ -562,6 +563,10 @@ public: NextInContextAndBits.setInt(Bits); } + /// \brief Looks on this and related declarations for an applicable + /// external source symbol attribute. + ExternalSourceSymbolAttr *getExternalSourceSymbolAttr() const; + /// \brief Whether this declaration was marked as being private to the /// module in which it was defined. bool isModulePrivate() const { @@ -698,6 +703,9 @@ public: Module *getLocalOwningModule() const { if (isFromASTFile() || !Hidden) return nullptr; + + assert(hasLocalOwningModuleStorage() && + "hidden local decl but no local module storage"); return reinterpret_cast(this)[-1]; } void setLocalOwningModule(Module *M) { Modified: vendor/clang/dist/include/clang/AST/Type.h ============================================================================== --- vendor/clang/dist/include/clang/AST/Type.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/AST/Type.h Mon May 22 19:43:45 2017 (r318665) @@ -333,15 +333,18 @@ public: bool hasAddressSpace() const { return Mask & AddressSpaceMask; } unsigned getAddressSpace() const { return Mask >> AddressSpaceShift; } + bool hasTargetSpecificAddressSpace() const { + return getAddressSpace() >= LangAS::FirstTargetAddressSpace; + } /// Get the address space attribute value to be printed by diagnostics. unsigned getAddressSpaceAttributePrintValue() const { auto Addr = getAddressSpace(); // This function is not supposed to be used with language specific // address spaces. If that happens, the diagnostic message should consider // printing the QualType instead of the address space value. - assert(Addr == 0 || Addr >= LangAS::Count); + assert(Addr == 0 || hasTargetSpecificAddressSpace()); if (Addr) - return Addr - LangAS::Count; + return Addr - LangAS::FirstTargetAddressSpace; // TODO: The diagnostic messages where Addr may be 0 should be fixed // since it cannot differentiate the situation where 0 denotes the default // address space or user specified __attribute__((address_space(0))). @@ -2008,10 +2011,11 @@ public: Optional getNullability(const ASTContext &context) const; /// Determine whether the given type can have a nullability - /// specifier applied to it, i.e., if it is any kind of pointer type - /// or a dependent type that could instantiate to any kind of - /// pointer type. - bool canHaveNullability() const; + /// specifier applied to it, i.e., if it is any kind of pointer type. + /// + /// \param ResultIfUnknown The value to return if we don't yet know whether + /// this type can have nullability because it is dependent. + bool canHaveNullability(bool ResultIfUnknown = true) const; /// Retrieve the set of substitutions required when accessing a member /// of the Objective-C receiver type that is declared in the given context. Modified: vendor/clang/dist/include/clang/Basic/AddressSpaces.h ============================================================================== --- vendor/clang/dist/include/clang/Basic/AddressSpaces.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Basic/AddressSpaces.h Mon May 22 19:43:45 2017 (r318665) @@ -45,13 +45,12 @@ enum ID { // This denotes the count of language-specific address spaces and also // the offset added to the target-specific address spaces, which are usually // specified by address space attributes __attribute__(address_space(n))). - Count + FirstTargetAddressSpace }; /// The type of a lookup table which maps from language-specific address spaces /// to target-specific ones. -typedef unsigned Map[Count]; - +typedef unsigned Map[FirstTargetAddressSpace]; } } Modified: vendor/clang/dist/include/clang/Basic/Attr.td ============================================================================== --- vendor/clang/dist/include/clang/Basic/Attr.td Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Basic/Attr.td Mon May 22 19:43:45 2017 (r318665) @@ -1179,6 +1179,12 @@ def MipsInterrupt : InheritableAttr, Tar let Documentation = [MipsInterruptDocs]; } +def MicroMips : InheritableAttr, TargetSpecificAttr { + let Spellings = [GCC<"micromips">]; + let Subjects = SubjectList<[Function], ErrorDiag>; + let Documentation = [MicroMipsDocs]; +} + def Mode : Attr { let Spellings = [GCC<"mode">]; let Subjects = SubjectList<[Var, Enum, TypedefName, Field], ErrorDiag, @@ -1261,6 +1267,12 @@ def NoMips16 : InheritableAttr, TargetSp let Documentation = [Undocumented]; } +def NoMicroMips : InheritableAttr, TargetSpecificAttr { + let Spellings = [GCC<"nomicromips">]; + let Subjects = SubjectList<[Function], ErrorDiag>; + let Documentation = [MicroMipsDocs]; +} + // This is not a TargetSpecificAttr so that is silently accepted and // ignored on other targets as encouraged by the OpenCL spec. // Modified: vendor/clang/dist/include/clang/Basic/AttrDocs.td ============================================================================== --- vendor/clang/dist/include/clang/Basic/AttrDocs.td Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Basic/AttrDocs.td Mon May 22 19:43:45 2017 (r318665) @@ -1269,6 +1269,19 @@ The semantics are as follows: }]; } +def MicroMipsDocs : Documentation { + let Category = DocCatFunction; + let Content = [{ +Clang supports the GNU style ``__attribute__((micromips))`` and +``__attribute__((nomicromips))`` attributes on MIPS targets. These attributes +may be attached to a function definition and instructs the backend to generate +or not to generate microMIPS code for that function. + +These attributes override the `-mmicromips` and `-mno-micromips` options +on the command line. + }]; +} + def AVRInterruptDocs : Documentation { let Category = DocCatFunction; let Content = [{ Modified: vendor/clang/dist/include/clang/Basic/DiagnosticSemaKinds.td ============================================================================== --- vendor/clang/dist/include/clang/Basic/DiagnosticSemaKinds.td Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Basic/DiagnosticSemaKinds.td Mon May 22 19:43:45 2017 (r318665) @@ -2490,7 +2490,7 @@ def err_attribute_address_multiple_quali def err_attribute_address_function_type : Error< "function type may not be qualified with an address space">; def err_as_qualified_auto_decl : Error< - "automatic variable qualified with an address space">; + "automatic variable qualified with an%select{| invalid}0 address space">; def err_arg_with_address_space : Error< "parameter may not be qualified with an address space">; def err_field_with_address_space : Error< Modified: vendor/clang/dist/include/clang/Basic/LangOptions.h ============================================================================== --- vendor/clang/dist/include/clang/Basic/LangOptions.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Basic/LangOptions.h Mon May 22 19:43:45 2017 (r318665) @@ -168,7 +168,7 @@ public: /// Do we need to track the owning module for a local declaration? bool trackLocalOwningModule() const { - return ModulesLocalVisibility; + return isCompilingModule() || ModulesLocalVisibility || ModulesTS; } bool isSignedOverflowDefined() const { Modified: vendor/clang/dist/include/clang/Format/Format.h ============================================================================== --- vendor/clang/dist/include/clang/Format/Format.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Format/Format.h Mon May 22 19:43:45 2017 (r318665) @@ -1133,6 +1133,9 @@ struct FormatStyle { /// ``Foo `` instead of ``Foo``. bool ObjCSpaceBeforeProtocolList; + /// \brief The penalty for breaking around an assignment operator. + unsigned PenaltyBreakAssignment; + /// \brief The penalty for breaking a function call after ``call(``. unsigned PenaltyBreakBeforeFirstCallParameter; @@ -1420,6 +1423,8 @@ struct FormatStyle { ObjCBlockIndentWidth == R.ObjCBlockIndentWidth && ObjCSpaceAfterProperty == R.ObjCSpaceAfterProperty && ObjCSpaceBeforeProtocolList == R.ObjCSpaceBeforeProtocolList && + PenaltyBreakAssignment == + R.PenaltyBreakAssignment && PenaltyBreakBeforeFirstCallParameter == R.PenaltyBreakBeforeFirstCallParameter && PenaltyBreakComment == R.PenaltyBreakComment && Modified: vendor/clang/dist/include/clang/Lex/ModuleMap.h ============================================================================== --- vendor/clang/dist/include/clang/Lex/ModuleMap.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Lex/ModuleMap.h Mon May 22 19:43:45 2017 (r318665) @@ -507,16 +507,6 @@ public: /// false otherwise. bool resolveConflicts(Module *Mod, bool Complain); - /// \brief Infers the (sub)module based on the given source location and - /// source manager. - /// - /// \param Loc The location within the source that we are querying, along - /// with its source manager. - /// - /// \returns The module that owns this source location, or null if no - /// module owns this source location. - Module *inferModuleFromLocation(FullSourceLoc Loc); - /// \brief Sets the umbrella header of the given module to the given /// header. void setUmbrellaHeader(Module *Mod, const FileEntry *UmbrellaHeader, Modified: vendor/clang/dist/include/clang/Lex/Preprocessor.h ============================================================================== --- vendor/clang/dist/include/clang/Lex/Preprocessor.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Lex/Preprocessor.h Mon May 22 19:43:45 2017 (r318665) @@ -1917,14 +1917,11 @@ public: /// into a module, or is outside any module, returns nullptr. Module *getModuleForLocation(SourceLocation Loc); - /// \brief Find the module that contains the specified location, either - /// directly or indirectly. - Module *getModuleContainingLocation(SourceLocation Loc); - /// \brief We want to produce a diagnostic at location IncLoc concerning a /// missing module import. /// /// \param IncLoc The location at which the missing import was detected. + /// \param M The desired module. /// \param MLoc A location within the desired module at which some desired /// effect occurred (eg, where a desired entity was declared). /// @@ -1932,6 +1929,7 @@ public: /// Null if no such file could be determined or if a #include is not /// appropriate. const FileEntry *getModuleHeaderToIncludeForDiagnostics(SourceLocation IncLoc, + Module *M, SourceLocation MLoc); private: Modified: vendor/clang/dist/include/clang/Parse/Parser.h ============================================================================== --- vendor/clang/dist/include/clang/Parse/Parser.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Parse/Parser.h Mon May 22 19:43:45 2017 (r318665) @@ -304,8 +304,9 @@ public: } /// ConsumeToken - Consume the current 'peek token' and lex the next one. - /// This does not work with special tokens: string literals, code completion - /// and balanced tokens must be handled using the specific consume methods. + /// This does not work with special tokens: string literals, code completion, + /// annotation tokens and balanced tokens must be handled using the specific + /// consume methods. /// Returns the location of the consumed token. SourceLocation ConsumeToken() { assert(!isTokenSpecial() && @@ -366,7 +367,7 @@ private: /// isTokenSpecial - True if this token requires special consumption methods. bool isTokenSpecial() const { return isTokenStringLiteral() || isTokenParen() || isTokenBracket() || - isTokenBrace() || Tok.is(tok::code_completion); + isTokenBrace() || Tok.is(tok::code_completion) || Tok.isAnnotation(); } /// \brief Returns true if the current token is '=' or is a type of '='. @@ -397,9 +398,19 @@ private: if (Tok.is(tok::code_completion)) return ConsumeCodeCompletionTok ? ConsumeCodeCompletionToken() : handleUnexpectedCodeCompletionToken(); + if (Tok.isAnnotation()) + return ConsumeAnnotationToken(); return ConsumeToken(); } + SourceLocation ConsumeAnnotationToken() { + assert(Tok.isAnnotation() && "wrong consume method"); + SourceLocation Loc = Tok.getLocation(); + PrevTokLocation = Tok.getAnnotationEndLoc(); + PP.Lex(Tok); + return Loc; + } + /// ConsumeParen - This consume method keeps the paren count up-to-date. /// SourceLocation ConsumeParen() { @@ -1449,6 +1460,7 @@ public: }; ExprResult ParseExpression(TypeCastState isTypeCast = NotTypeCast); + ExprResult ParseConstantExpressionInExprEvalContext(TypeCastState isTypeCast); ExprResult ParseConstantExpression(TypeCastState isTypeCast = NotTypeCast); ExprResult ParseConstraintExpression(); // Expr that doesn't include commas. Modified: vendor/clang/dist/include/clang/Sema/DeclSpec.h ============================================================================== --- vendor/clang/dist/include/clang/Sema/DeclSpec.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Sema/DeclSpec.h Mon May 22 19:43:45 2017 (r318665) @@ -1999,41 +1999,6 @@ public: llvm_unreachable("unknown context kind!"); } - /// diagnoseIdentifier - Return true if the identifier is prohibited and - /// should be diagnosed (because it cannot be anything else). - bool diagnoseIdentifier() const { - switch (Context) { - case FileContext: - case KNRTypeListContext: - case MemberContext: - case BlockContext: - case ForContext: - case InitStmtContext: - case ConditionContext: - case PrototypeContext: - case LambdaExprParameterContext: - case TemplateParamContext: - case CXXCatchContext: - case ObjCCatchContext: - case TypeNameContext: - case FunctionalCastContext: - case ConversionIdContext: - case ObjCParameterContext: - case ObjCResultContext: - case BlockLiteralContext: - case CXXNewContext: - case LambdaExprContext: - return false; - - case AliasDeclContext: - case AliasTemplateContext: - case TemplateTypeArgContext: - case TrailingReturnContext: - return true; - } - llvm_unreachable("unknown context kind!"); - } - /// Return true if the context permits a C++17 decomposition declarator. bool mayHaveDecompositionDeclarator() const { switch (Context) { Modified: vendor/clang/dist/include/clang/Sema/Sema.h ============================================================================== --- vendor/clang/dist/include/clang/Sema/Sema.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Sema/Sema.h Mon May 22 19:43:45 2017 (r318665) @@ -1507,6 +1507,12 @@ public: hasVisibleDefaultArgument(const NamedDecl *D, llvm::SmallVectorImpl *Modules = nullptr); + /// Determine if there is a visible declaration of \p D that is an explicit + /// specialization declaration for a specialization of a template. (For a + /// member specialization, use hasVisibleMemberSpecialization.) + bool hasVisibleExplicitSpecialization( + const NamedDecl *D, llvm::SmallVectorImpl *Modules = nullptr); + /// Determine if there is a visible declaration of \p D that is a member /// specialization declaration (as opposed to an instantiated declaration). bool hasVisibleMemberSpecialization( @@ -2360,7 +2366,7 @@ public: void MergeVarDeclTypes(VarDecl *New, VarDecl *Old, bool MergeTypeWithOld); void MergeVarDeclExceptionSpecs(VarDecl *New, VarDecl *Old); bool checkVarDeclRedefinition(VarDecl *OldDefn, VarDecl *NewDefn); - void notePreviousDefinition(SourceLocation Old, SourceLocation New); + void notePreviousDefinition(const NamedDecl *Old, SourceLocation New); bool MergeCXXFunctionDecl(FunctionDecl *New, FunctionDecl *Old, Scope *S); // AssignmentAction - This is used by all the assignment diagnostic functions @@ -7377,9 +7383,9 @@ public: /// but have not yet been performed. std::deque PendingInstantiations; - class SavePendingInstantiationsAndVTableUsesRAII { + class GlobalEagerInstantiationScope { public: - SavePendingInstantiationsAndVTableUsesRAII(Sema &S, bool Enabled) + GlobalEagerInstantiationScope(Sema &S, bool Enabled) : S(S), Enabled(Enabled) { if (!Enabled) return; @@ -7387,7 +7393,14 @@ public: SavedVTableUses.swap(S.VTableUses); } - ~SavePendingInstantiationsAndVTableUsesRAII() { + void perform() { + if (Enabled) { + S.DefineUsedVTables(); + S.PerformPendingInstantiations(); + } + } + + ~GlobalEagerInstantiationScope() { if (!Enabled) return; // Restore the set of pending vtables. @@ -7417,14 +7430,16 @@ public: /// types, static variables, enumerators, etc. std::deque PendingLocalImplicitInstantiations; - class SavePendingLocalImplicitInstantiationsRAII { + class LocalEagerInstantiationScope { public: - SavePendingLocalImplicitInstantiationsRAII(Sema &S): S(S) { + LocalEagerInstantiationScope(Sema &S) : S(S) { SavedPendingLocalImplicitInstantiations.swap( S.PendingLocalImplicitInstantiations); } - ~SavePendingLocalImplicitInstantiationsRAII() { + void perform() { S.PerformPendingInstantiations(/*LocalOnly=*/true); } + + ~LocalEagerInstantiationScope() { assert(S.PendingLocalImplicitInstantiations.empty() && "there shouldn't be any pending local implicit instantiations"); SavedPendingLocalImplicitInstantiations.swap( @@ -7434,7 +7449,7 @@ public: private: Sema &S; std::deque - SavedPendingLocalImplicitInstantiations; + SavedPendingLocalImplicitInstantiations; }; /// A helper class for building up ExtParameterInfos. Modified: vendor/clang/dist/include/clang/Serialization/ASTReader.h ============================================================================== --- vendor/clang/dist/include/clang/Serialization/ASTReader.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Serialization/ASTReader.h Mon May 22 19:43:45 2017 (r318665) @@ -478,10 +478,18 @@ private: /// in the chain. DeclUpdateOffsetsMap DeclUpdateOffsets; + struct PendingUpdateRecord { + Decl *D; + serialization::GlobalDeclID ID; + // Whether the declaration was just deserialized. + bool JustLoaded; + PendingUpdateRecord(serialization::GlobalDeclID ID, Decl *D, + bool JustLoaded) + : D(D), ID(ID), JustLoaded(JustLoaded) {} + }; /// \brief Declaration updates for already-loaded declarations that we need /// to apply once we finish processing an import. - llvm::SmallVector, 16> - PendingUpdateRecords; + llvm::SmallVector PendingUpdateRecords; enum class PendingFakeDefinitionKind { NotFake, Fake, FakeLoaded }; @@ -1279,7 +1287,7 @@ private: RecordLocation DeclCursorForID(serialization::DeclID ID, SourceLocation &Location); - void loadDeclUpdateRecords(serialization::DeclID ID, Decl *D); + void loadDeclUpdateRecords(PendingUpdateRecord &Record); void loadPendingDeclChain(Decl *D, uint64_t LocalOffset); void loadObjCCategories(serialization::GlobalDeclID ID, ObjCInterfaceDecl *D, unsigned PreviousGeneration = 0); Modified: vendor/clang/dist/include/clang/Serialization/ASTWriter.h ============================================================================== --- vendor/clang/dist/include/clang/Serialization/ASTWriter.h Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/include/clang/Serialization/ASTWriter.h Mon May 22 19:43:45 2017 (r318665) @@ -627,10 +627,6 @@ public: /// \brief Add a version tuple to the given record void AddVersionTuple(const VersionTuple &Version, RecordDataImpl &Record); - /// \brief Infer the submodule ID that contains an entity at the given - /// source location. - serialization::SubmoduleID inferSubmoduleIDFromLocation(SourceLocation Loc); - /// \brief Retrieve or create a submodule ID for this module, or return 0 if /// the submodule is neither local (a submodle of the currently-written module) /// nor from an imported module. Modified: vendor/clang/dist/lib/AST/ASTContext.cpp ============================================================================== --- vendor/clang/dist/lib/AST/ASTContext.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/AST/ASTContext.cpp Mon May 22 19:43:45 2017 (r318665) @@ -8730,8 +8730,8 @@ static QualType DecodeTypeFromStr(const char *End; unsigned AddrSpace = strtoul(Str, &End, 10); if (End != Str && AddrSpace != 0) { - Type = Context.getAddrSpaceQualType(Type, AddrSpace + - LangAS::Count); + Type = Context.getAddrSpaceQualType( + Type, AddrSpace + LangAS::FirstTargetAddressSpace); Str = End; } if (c == '*') @@ -9546,13 +9546,8 @@ uint64_t ASTContext::getTargetNullPointe } unsigned ASTContext::getTargetAddressSpace(unsigned AS) const { - // For OpenCL, only function local variables are not explicitly marked with - // an address space in the AST, and these need to be the address space of - // alloca. - if (!AS && LangOpts.OpenCL) - return getTargetInfo().getDataLayout().getAllocaAddrSpace(); - if (AS >= LangAS::Count) - return AS - LangAS::Count; + if (AS >= LangAS::FirstTargetAddressSpace) + return AS - LangAS::FirstTargetAddressSpace; else return (*AddrSpaceMap)[AS]; } Modified: vendor/clang/dist/lib/AST/CXXInheritance.cpp ============================================================================== --- vendor/clang/dist/lib/AST/CXXInheritance.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/AST/CXXInheritance.cpp Mon May 22 19:43:45 2017 (r318665) @@ -57,6 +57,7 @@ bool CXXBasePaths::isAmbiguous(CanQualTy void CXXBasePaths::clear() { Paths.clear(); ClassSubobjects.clear(); + VisitedDependentRecords.clear(); ScratchPath.clear(); DetectedVirtual = nullptr; } @@ -67,6 +68,7 @@ void CXXBasePaths::swap(CXXBasePaths &Ot std::swap(Origin, Other.Origin); Paths.swap(Other.Paths); ClassSubobjects.swap(Other.ClassSubobjects); + VisitedDependentRecords.swap(Other.VisitedDependentRecords); std::swap(FindAmbiguities, Other.FindAmbiguities); std::swap(RecordPaths, Other.RecordPaths); std::swap(DetectVirtual, Other.DetectVirtual); @@ -278,8 +280,14 @@ bool CXXBasePaths::lookupInBases(ASTCont dyn_cast_or_null(TN.getAsTemplateDecl())) BaseRecord = TD->getTemplatedDecl(); } - if (BaseRecord && !BaseRecord->hasDefinition()) - BaseRecord = nullptr; + if (BaseRecord) { + if (!BaseRecord->hasDefinition() || + VisitedDependentRecords.count(BaseRecord)) { + BaseRecord = nullptr; + } else { + VisitedDependentRecords.insert(BaseRecord); + } + } } else { BaseRecord = cast( BaseSpec.getType()->castAs()->getDecl()); Modified: vendor/clang/dist/lib/AST/DeclBase.cpp ============================================================================== --- vendor/clang/dist/lib/AST/DeclBase.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/AST/DeclBase.cpp Mon May 22 19:43:45 2017 (r318665) @@ -407,6 +407,27 @@ bool Decl::isExported() const { return false; } +ExternalSourceSymbolAttr *Decl::getExternalSourceSymbolAttr() const { + const Decl *Definition = nullptr; + if (auto ID = dyn_cast(this)) { + Definition = ID->getDefinition(); + } else if (auto PD = dyn_cast(this)) { + Definition = PD->getDefinition(); + } else if (auto TD = dyn_cast(this)) { + Definition = TD->getDefinition(); + } + if (!Definition) + Definition = this; + + if (auto *attr = Definition->getAttr()) + return attr; + if (auto *dcd = dyn_cast(getDeclContext())) { + return dcd->getAttr(); + } + + return nullptr; +} + bool Decl::hasDefiningAttr() const { return hasAttr() || hasAttr(); } Modified: vendor/clang/dist/lib/AST/ExprConstant.cpp ============================================================================== --- vendor/clang/dist/lib/AST/ExprConstant.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/AST/ExprConstant.cpp Mon May 22 19:43:45 2017 (r318665) @@ -4579,7 +4579,7 @@ public: } bool handleCallExpr(const CallExpr *E, APValue &Result, - const LValue *ResultSlot) { + const LValue *ResultSlot) { const Expr *Callee = E->getCallee()->IgnoreParens(); QualType CalleeType = Callee->getType(); @@ -4588,6 +4588,23 @@ public: auto Args = llvm::makeArrayRef(E->getArgs(), E->getNumArgs()); bool HasQualifier = false; + struct EvaluateIgnoredRAII { + public: + EvaluateIgnoredRAII(EvalInfo &Info, llvm::ArrayRef ToEval) + : Info(Info), ToEval(ToEval) {} + ~EvaluateIgnoredRAII() { + if (Info.noteFailure()) { + for (auto E : ToEval) + EvaluateIgnoredValue(Info, E); + } + } + void cancel() { ToEval = {}; } + void drop_front() { ToEval = ToEval.drop_front(); } + private: + EvalInfo &Info; + llvm::ArrayRef ToEval; + } EvalArguments(Info, Args); + // Extract function decl and 'this' pointer from the callee. if (CalleeType->isSpecificBuiltinType(BuiltinType::BoundMember)) { const ValueDecl *Member = nullptr; @@ -4637,10 +4654,12 @@ public: if (Args.empty()) return Error(E); - if (!EvaluateObjectArgument(Info, Args[0], ThisVal)) + const Expr *FirstArg = Args[0]; + Args = Args.drop_front(); + EvalArguments.drop_front(); + if (!EvaluateObjectArgument(Info, FirstArg, ThisVal)) return false; This = &ThisVal; - Args = Args.slice(1); } else if (MD && MD->isLambdaStaticInvoker()) { // Map the static invoker for the lambda back to the call operator. // Conveniently, we don't have to slice out the 'this' argument (as is @@ -4692,8 +4711,12 @@ public: const FunctionDecl *Definition = nullptr; Stmt *Body = FD->getBody(Definition); - if (!CheckConstexprFunction(Info, E->getExprLoc(), FD, Definition, Body) || - !HandleFunctionCall(E->getExprLoc(), Definition, This, Args, Body, Info, + if (!CheckConstexprFunction(Info, E->getExprLoc(), FD, Definition, Body)) + return false; + + EvalArguments.cancel(); + + if (!HandleFunctionCall(E->getExprLoc(), Definition, This, Args, Body, Info, Result, ResultSlot)) return false; Modified: vendor/clang/dist/lib/AST/Type.cpp ============================================================================== --- vendor/clang/dist/lib/AST/Type.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/AST/Type.cpp Mon May 22 19:43:45 2017 (r318665) @@ -3531,7 +3531,7 @@ Optional Type::getNulla } while (true); } -bool Type::canHaveNullability() const { +bool Type::canHaveNullability(bool ResultIfUnknown) const { QualType type = getCanonicalTypeInternal(); switch (type->getTypeClass()) { @@ -3559,7 +3559,8 @@ bool Type::canHaveNullability() const { case Type::SubstTemplateTypeParmPack: case Type::DependentName: case Type::DependentTemplateSpecialization: - return true; + case Type::Auto: + return ResultIfUnknown; // Dependent template specializations can instantiate to pointer // types unless they're known to be specializations of a class @@ -3571,12 +3572,7 @@ bool Type::canHaveNullability() const { if (isa(templateDecl)) return false; } - return true; - - // auto is considered dependent when it isn't deduced. - case Type::Auto: - case Type::DeducedTemplateSpecialization: - return !cast(type.getTypePtr())->isDeduced(); + return ResultIfUnknown; case Type::Builtin: switch (cast(type.getTypePtr())->getKind()) { @@ -3595,7 +3591,7 @@ bool Type::canHaveNullability() const { case BuiltinType::PseudoObject: case BuiltinType::UnknownAny: case BuiltinType::ARCUnbridgedCast: - return true; + return ResultIfUnknown; case BuiltinType::Void: case BuiltinType::ObjCId: @@ -3614,6 +3610,7 @@ bool Type::canHaveNullability() const { case BuiltinType::OMPArraySection: return false; } + llvm_unreachable("unknown builtin type"); // Non-pointer types. case Type::Complex: @@ -3629,6 +3626,7 @@ bool Type::canHaveNullability() const { case Type::FunctionProto: case Type::FunctionNoProto: case Type::Record: + case Type::DeducedTemplateSpecialization: case Type::Enum: case Type::InjectedClassName: case Type::PackExpansion: Modified: vendor/clang/dist/lib/AST/TypePrinter.cpp ============================================================================== --- vendor/clang/dist/lib/AST/TypePrinter.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/AST/TypePrinter.cpp Mon May 22 19:43:45 2017 (r318665) @@ -1668,9 +1668,9 @@ void Qualifiers::print(raw_ostream &OS, OS << "__shared"; break; default: - assert(addrspace >= LangAS::Count); + assert(addrspace >= LangAS::FirstTargetAddressSpace); OS << "__attribute__((address_space("; - OS << addrspace - LangAS::Count; + OS << addrspace - LangAS::FirstTargetAddressSpace; OS << ")))"; } } Modified: vendor/clang/dist/lib/Basic/Targets.cpp ============================================================================== --- vendor/clang/dist/lib/Basic/Targets.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/Basic/Targets.cpp Mon May 22 19:43:45 2017 (r318665) @@ -2034,25 +2034,45 @@ ArrayRef NVPTXTargetInfo:: return llvm::makeArrayRef(GCCRegNames); } -static const LangAS::Map AMDGPUPrivateIsZeroMap = { - 4, // Default - 1, // opencl_global - 3, // opencl_local - 2, // opencl_constant - 4, // opencl_generic - 1, // cuda_device - 2, // cuda_constant - 3 // cuda_shared -}; -static const LangAS::Map AMDGPUGenericIsZeroMap = { - 0, // Default - 1, // opencl_global - 3, // opencl_local - 2, // opencl_constant - 0, // opencl_generic - 1, // cuda_device - 2, // cuda_constant - 3 // cuda_shared +static const LangAS::Map AMDGPUNonOpenCLPrivateIsZeroMap = { + 4, // Default + 1, // opencl_global + 3, // opencl_local + 2, // opencl_constant + 4, // opencl_generic + 1, // cuda_device + 2, // cuda_constant + 3 // cuda_shared +}; +static const LangAS::Map AMDGPUNonOpenCLGenericIsZeroMap = { + 0, // Default + 1, // opencl_global + 3, // opencl_local + 2, // opencl_constant + 0, // opencl_generic + 1, // cuda_device + 2, // cuda_constant + 3 // cuda_shared +}; +static const LangAS::Map AMDGPUOpenCLPrivateIsZeroMap = { + 0, // Default + 1, // opencl_global + 3, // opencl_local + 2, // opencl_constant + 4, // opencl_generic + 1, // cuda_device + 2, // cuda_constant + 3 // cuda_shared +}; +static const LangAS::Map AMDGPUOpenCLGenericIsZeroMap = { + 5, // Default + 1, // opencl_global + 3, // opencl_local + 2, // opencl_constant + 0, // opencl_generic + 1, // cuda_device + 2, // cuda_constant + 3 // cuda_shared }; // If you edit the description strings, make sure you update @@ -2149,8 +2169,12 @@ public: : DataLayoutStringR600); assert(DataLayout->getAllocaAddrSpace() == AS.Private); - AddrSpaceMap = IsGenericZero ? &AMDGPUGenericIsZeroMap : - &AMDGPUPrivateIsZeroMap; + AddrSpaceMap = + llvm::StringSwitch(Triple.getEnvironmentName()) + .Case("opencl", &AMDGPUOpenCLPrivateIsZeroMap) + .Case("amdgiz", &AMDGPUNonOpenCLGenericIsZeroMap) + .Case("amdgizcl", &AMDGPUOpenCLGenericIsZeroMap) + .Default(&AMDGPUNonOpenCLPrivateIsZeroMap); UseAddrSpaceMapMangling = true; } Modified: vendor/clang/dist/lib/CodeGen/CGAtomic.cpp ============================================================================== --- vendor/clang/dist/lib/CodeGen/CGAtomic.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/CodeGen/CGAtomic.cpp Mon May 22 19:43:45 2017 (r318665) @@ -95,7 +95,7 @@ namespace { BFI.StorageOffset += OffsetInChars; LVal = LValue::MakeBitfield(Address(Addr, lvalue.getAlignment()), BFI, lvalue.getType(), - lvalue.getAlignmentSource()); + lvalue.getBaseInfo()); LVal.setTBAAInfo(lvalue.getTBAAInfo()); AtomicTy = C.getIntTypeForBitwidth(AtomicSizeInBits, OrigBFI.IsSigned); if (AtomicTy.isNull()) { @@ -203,7 +203,7 @@ namespace { addr = CGF.Builder.CreateStructGEP(addr, 0, CharUnits()); return LValue::MakeAddr(addr, getValueType(), CGF.getContext(), - LVal.getAlignmentSource(), LVal.getTBAAInfo()); + LVal.getBaseInfo(), LVal.getTBAAInfo()); } /// \brief Emits atomic load. @@ -1181,15 +1181,15 @@ RValue AtomicInfo::convertAtomicTempToRV if (LVal.isBitField()) return CGF.EmitLoadOfBitfieldLValue( LValue::MakeBitfield(addr, LVal.getBitFieldInfo(), LVal.getType(), - LVal.getAlignmentSource()), loc); + LVal.getBaseInfo()), loc); if (LVal.isVectorElt()) return CGF.EmitLoadOfLValue( LValue::MakeVectorElt(addr, LVal.getVectorIdx(), LVal.getType(), - LVal.getAlignmentSource()), loc); + LVal.getBaseInfo()), loc); assert(LVal.isExtVectorElt()); return CGF.EmitLoadOfExtVectorElementLValue(LValue::MakeExtVectorElt( addr, LVal.getExtVectorElts(), LVal.getType(), - LVal.getAlignmentSource())); + LVal.getBaseInfo())); } RValue AtomicInfo::ConvertIntToValueOrAtomic(llvm::Value *IntVal, @@ -1506,26 +1506,26 @@ EmitAtomicUpdateValue(CodeGenFunction &C UpdateLVal = LValue::MakeBitfield(Ptr, AtomicLVal.getBitFieldInfo(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); DesiredLVal = LValue::MakeBitfield(DesiredAddr, AtomicLVal.getBitFieldInfo(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); } else if (AtomicLVal.isVectorElt()) { UpdateLVal = LValue::MakeVectorElt(Ptr, AtomicLVal.getVectorIdx(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); DesiredLVal = LValue::MakeVectorElt( DesiredAddr, AtomicLVal.getVectorIdx(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); } else { assert(AtomicLVal.isExtVectorElt()); UpdateLVal = LValue::MakeExtVectorElt(Ptr, AtomicLVal.getExtVectorElts(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); DesiredLVal = LValue::MakeExtVectorElt( DesiredAddr, AtomicLVal.getExtVectorElts(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); } UpdateLVal.setTBAAInfo(AtomicLVal.getTBAAInfo()); DesiredLVal.setTBAAInfo(AtomicLVal.getTBAAInfo()); @@ -1612,17 +1612,17 @@ static void EmitAtomicUpdateValue(CodeGe DesiredLVal = LValue::MakeBitfield(DesiredAddr, AtomicLVal.getBitFieldInfo(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); } else if (AtomicLVal.isVectorElt()) { DesiredLVal = LValue::MakeVectorElt(DesiredAddr, AtomicLVal.getVectorIdx(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); } else { assert(AtomicLVal.isExtVectorElt()); DesiredLVal = LValue::MakeExtVectorElt( DesiredAddr, AtomicLVal.getExtVectorElts(), AtomicLVal.getType(), - AtomicLVal.getAlignmentSource()); + AtomicLVal.getBaseInfo()); } DesiredLVal.setTBAAInfo(AtomicLVal.getTBAAInfo()); // Store new value in the corresponding memory area Modified: vendor/clang/dist/lib/CodeGen/CGBlocks.cpp ============================================================================== --- vendor/clang/dist/lib/CodeGen/CGBlocks.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/CodeGen/CGBlocks.cpp Mon May 22 19:43:45 2017 (r318665) @@ -918,8 +918,9 @@ llvm::Value *CodeGenFunction::EmitBlockL // FIXME: Pass a specific location for the expr init so that the store is // attributed to a reasonable location - otherwise it may be attributed to // locations of subexpressions in the initialization. + LValueBaseInfo BaseInfo(AlignmentSource::Decl, false); EmitExprAsInit(&l2r, &blockFieldPseudoVar, - MakeAddrLValue(blockField, type, AlignmentSource::Decl), + MakeAddrLValue(blockField, type, BaseInfo), /*captured by init*/ false); } Modified: vendor/clang/dist/lib/CodeGen/CGBuiltin.cpp ============================================================================== --- vendor/clang/dist/lib/CodeGen/CGBuiltin.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/CodeGen/CGBuiltin.cpp Mon May 22 19:43:45 2017 (r318665) @@ -8842,9 +8842,8 @@ Value *CodeGenFunction::EmitNVPTXBuiltin const CallExpr *E) { auto MakeLdg = [&](unsigned IntrinsicID) { Value *Ptr = EmitScalarExpr(E->getArg(0)); - AlignmentSource AlignSource; clang::CharUnits Align = - getNaturalPointeeTypeAlignment(E->getArg(0)->getType(), &AlignSource); + getNaturalPointeeTypeAlignment(E->getArg(0)->getType()); return Builder.CreateCall( CGM.getIntrinsic(IntrinsicID, {Ptr->getType()->getPointerElementType(), Ptr->getType()}), Modified: vendor/clang/dist/lib/CodeGen/CGClass.cpp ============================================================================== --- vendor/clang/dist/lib/CodeGen/CGClass.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/CodeGen/CGClass.cpp Mon May 22 19:43:45 2017 (r318665) @@ -129,14 +129,14 @@ Address CodeGenFunction::EmitCXXMemberDataPointerAddress(const Expr *E, Address base, llvm::Value *memberPtr, const MemberPointerType *memberPtrType, - AlignmentSource *alignSource) { + LValueBaseInfo *BaseInfo) { // Ask the ABI to compute the actual address. llvm::Value *ptr = CGM.getCXXABI().EmitMemberDataPointerAddress(*this, E, base, memberPtr, memberPtrType); QualType memberType = memberPtrType->getPointeeType(); - CharUnits memberAlign = getNaturalTypeAlignment(memberType, alignSource); + CharUnits memberAlign = getNaturalTypeAlignment(memberType, BaseInfo); memberAlign = CGM.getDynamicOffsetAlignment(base.getAlignment(), memberPtrType->getClass()->getAsCXXRecordDecl(), Modified: vendor/clang/dist/lib/CodeGen/CGDebugInfo.cpp ============================================================================== --- vendor/clang/dist/lib/CodeGen/CGDebugInfo.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/CodeGen/CGDebugInfo.cpp Mon May 22 19:43:45 2017 (r318665) @@ -2613,7 +2613,7 @@ llvm::DIModule *CGDebugInfo::getParentMo // best to make this behavior a command line or debugger tuning // option. FullSourceLoc Loc(D->getLocation(), CGM.getContext().getSourceManager()); - if (Module *M = ClangModuleMap->inferModuleFromLocation(Loc)) { + if (Module *M = D->getOwningModule()) { // This is a (sub-)module. auto Info = ExternalASTSource::ASTSourceDescriptor(*M); return getOrCreateModuleRef(Info, /*SkeletonCU=*/false); Modified: vendor/clang/dist/lib/CodeGen/CGDecl.cpp ============================================================================== --- vendor/clang/dist/lib/CodeGen/CGDecl.cpp Mon May 22 19:43:35 2017 (r318664) +++ vendor/clang/dist/lib/CodeGen/CGDecl.cpp Mon May 22 19:43:45 2017 (r318665) @@ -11,14 +11,15 @@ // //===----------------------------------------------------------------------===// -#include "CodeGenFunction.h" #include "CGBlocks.h" #include "CGCXXABI.h" #include "CGCleanup.h" #include "CGDebugInfo.h" #include "CGOpenCLRuntime.h" #include "CGOpenMPRuntime.h" +#include "CodeGenFunction.h" #include "CodeGenModule.h" +#include "TargetInfo.h" #include "clang/AST/ASTContext.h" #include "clang/AST/CharUnits.h" #include "clang/AST/Decl.h" @@ -1105,6 +1106,21 @@ CodeGenFunction::EmitAutoVarAlloca(const address = Address(vla, alignment); } + // Alloca always returns a pointer in alloca address space, which may + // be different from the type defined by the language. For example, + // in C++ the auto variables are in the default address space. Therefore + // cast alloca to the expected address space when necessary. + auto T = D.getType(); + assert(T.getAddressSpace() == LangAS::Default); *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:00 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87B04D78A7E; Mon, 22 May 2017 19:44:00 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 500D91A5D; Mon, 22 May 2017 19:44:00 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJhxtO076715; Mon, 22 May 2017 19:43:59 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJhu8L076688; Mon, 22 May 2017 19:43:56 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221943.v4MJhu8L076688@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:43:56 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318667 - in vendor/compiler-rt/dist: include/xray lib/asan lib/asan/tests lib/lsan lib/msan lib/msan/tests lib/sanitizer_common lib/sanitizer_common/tests lib/scudo lib/xray test/asan/... X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:00 -0000 Author: dim Date: Mon May 22 19:43:56 2017 New Revision: 318667 URL: https://svnweb.freebsd.org/changeset/base/318667 Log: Vendor import of compiler-rt trunk r303571: https://llvm.org/svn/llvm-project/compiler-rt/trunk@303571 Deleted: vendor/compiler-rt/dist/test/asan/TestCases/Posix/strndup_oob_test.cc vendor/compiler-rt/dist/test/msan/strndup.cc Modified: vendor/compiler-rt/dist/include/xray/xray_interface.h vendor/compiler-rt/dist/lib/asan/asan_flags.cc vendor/compiler-rt/dist/lib/asan/tests/asan_str_test.cc vendor/compiler-rt/dist/lib/lsan/lsan_common_mac.cc vendor/compiler-rt/dist/lib/msan/msan_interceptors.cc vendor/compiler-rt/dist/lib/msan/tests/msan_test.cc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.cc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.h vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common_interceptors.inc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flag_parser.h vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.h vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.inc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux.cc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux_libcdep.cc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mac.cc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_platform_interceptors.h vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_common.cc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_mac.cc vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc vendor/compiler-rt/dist/lib/sanitizer_common/tests/CMakeLists.txt vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_flags_test.cc vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_test_utils.h vendor/compiler-rt/dist/lib/scudo/scudo_flags.cpp vendor/compiler-rt/dist/lib/xray/xray_interface.cc vendor/compiler-rt/dist/test/asan/TestCases/Posix/asan-sigbus.cpp vendor/compiler-rt/dist/test/xray/TestCases/Linux/custom-event-logging.cc vendor/compiler-rt/dist/test/xray/TestCases/Linux/func-id-utils.cc Modified: vendor/compiler-rt/dist/include/xray/xray_interface.h ============================================================================== --- vendor/compiler-rt/dist/include/xray/xray_interface.h Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/include/xray/xray_interface.h Mon May 22 19:43:56 2017 (r318667) @@ -69,6 +69,10 @@ extern int __xray_remove_handler_arg1(); /// Provide a function to invoke when XRay encounters a custom event. extern int __xray_set_customevent_handler(void (*entry)(void*, std::size_t)); +/// This removes whatever the currently provided custom event handler is. +/// Returns 1 on success, 0 on error. +extern int __xray_remove_customevent_handler(); + enum XRayPatchingStatus { NOT_INITIALIZED = 0, SUCCESS = 1, Modified: vendor/compiler-rt/dist/lib/asan/asan_flags.cc ============================================================================== --- vendor/compiler-rt/dist/lib/asan/asan_flags.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/asan/asan_flags.cc Mon May 22 19:43:56 2017 (r318667) @@ -194,10 +194,6 @@ void InitializeFlags() { Report("WARNING: strchr* interceptors are enabled even though " "replace_str=0. Use intercept_strchr=0 to disable them."); } - if (!f->replace_str && common_flags()->intercept_strndup) { - Report("WARNING: strndup* interceptors are enabled even though " - "replace_str=0. Use intercept_strndup=0 to disable them."); - } } } // namespace __asan Modified: vendor/compiler-rt/dist/lib/asan/tests/asan_str_test.cc ============================================================================== --- vendor/compiler-rt/dist/lib/asan/tests/asan_str_test.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/asan/tests/asan_str_test.cc Mon May 22 19:43:56 2017 (r318667) @@ -154,27 +154,6 @@ TEST(AddressSanitizer, MAYBE_StrDupOOBTe free(str); } -#if SANITIZER_TEST_HAS_STRNDUP -TEST(AddressSanitizer, MAYBE_StrNDupOOBTest) { - size_t size = Ident(42); - char *str = MallocAndMemsetString(size); - char *new_str; - // Normal strndup calls. - str[size - 1] = '\0'; - new_str = strndup(str, size - 13); - free(new_str); - new_str = strndup(str + size - 1, 13); - free(new_str); - // Argument points to not allocated memory. - EXPECT_DEATH(Ident(strndup(str - 1, 13)), LeftOOBReadMessage(1)); - EXPECT_DEATH(Ident(strndup(str + size, 13)), RightOOBReadMessage(0)); - // Overwrite the terminating '\0' and hit unallocated memory. - str[size - 1] = 'z'; - EXPECT_DEATH(Ident(strndup(str, size + 13)), RightOOBReadMessage(0)); - free(str); -} -#endif // SANITIZER_TEST_HAS_STRNDUP - TEST(AddressSanitizer, StrCpyOOBTest) { size_t to_size = Ident(30); size_t from_size = Ident(6); // less than to_size Modified: vendor/compiler-rt/dist/lib/lsan/lsan_common_mac.cc ============================================================================== --- vendor/compiler-rt/dist/lib/lsan/lsan_common_mac.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/lsan/lsan_common_mac.cc Mon May 22 19:43:56 2017 (r318667) @@ -110,7 +110,8 @@ void ProcessGlobalRegions(Frontier *fron for (const __sanitizer::LoadedModule::AddressRange &range : modules[i].ranges()) { - if (range.executable || !range.readable) continue; + // Sections storing global variables are writable and non-executable + if (range.executable || !range.writable) continue; ScanGlobalRange(range.beg, range.end, frontier); } Modified: vendor/compiler-rt/dist/lib/msan/msan_interceptors.cc ============================================================================== --- vendor/compiler-rt/dist/lib/msan/msan_interceptors.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/msan/msan_interceptors.cc Mon May 22 19:43:56 2017 (r318667) @@ -341,6 +341,33 @@ INTERCEPTOR(char *, __strdup, char *src) #define MSAN_MAYBE_INTERCEPT___STRDUP #endif +INTERCEPTOR(char *, strndup, char *src, SIZE_T n) { + ENSURE_MSAN_INITED(); + GET_STORE_STACK_TRACE; + // On FreeBSD strndup() leverages strnlen(). + InterceptorScope interceptor_scope; + SIZE_T copy_size = REAL(strnlen)(src, n); + char *res = REAL(strndup)(src, n); + CopyShadowAndOrigin(res, src, copy_size, &stack); + __msan_unpoison(res + copy_size, 1); // \0 + return res; +} + +#if !SANITIZER_FREEBSD +INTERCEPTOR(char *, __strndup, char *src, SIZE_T n) { + ENSURE_MSAN_INITED(); + GET_STORE_STACK_TRACE; + SIZE_T copy_size = REAL(strnlen)(src, n); + char *res = REAL(__strndup)(src, n); + CopyShadowAndOrigin(res, src, copy_size, &stack); + __msan_unpoison(res + copy_size, 1); // \0 + return res; +} +#define MSAN_MAYBE_INTERCEPT___STRNDUP INTERCEPT_FUNCTION(__strndup) +#else +#define MSAN_MAYBE_INTERCEPT___STRNDUP +#endif + INTERCEPTOR(char *, gcvt, double number, SIZE_T ndigit, char *buf) { ENSURE_MSAN_INITED(); char *res = REAL(gcvt)(number, ndigit, buf); @@ -1344,13 +1371,6 @@ int OnExit() { return __msan_memcpy(to, from, size); \ } -#define COMMON_INTERCEPTOR_COPY_STRING(ctx, to, from, size) \ - do { \ - GET_STORE_STACK_TRACE; \ - CopyShadowAndOrigin(to, from, size, &stack); \ - __msan_unpoison(to + size, 1); \ - } while (false) - #include "sanitizer_common/sanitizer_platform_interceptors.h" #include "sanitizer_common/sanitizer_common_interceptors.inc" @@ -1518,6 +1538,8 @@ void InitializeInterceptors() { INTERCEPT_FUNCTION(stpcpy); // NOLINT INTERCEPT_FUNCTION(strdup); MSAN_MAYBE_INTERCEPT___STRDUP; + INTERCEPT_FUNCTION(strndup); + MSAN_MAYBE_INTERCEPT___STRNDUP; INTERCEPT_FUNCTION(strncpy); // NOLINT INTERCEPT_FUNCTION(gcvt); INTERCEPT_FUNCTION(strcat); // NOLINT Modified: vendor/compiler-rt/dist/lib/msan/tests/msan_test.cc ============================================================================== --- vendor/compiler-rt/dist/lib/msan/tests/msan_test.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/msan/tests/msan_test.cc Mon May 22 19:43:56 2017 (r318667) @@ -1581,8 +1581,7 @@ TEST(MemorySanitizer, strdup) { TEST(MemorySanitizer, strndup) { char buf[4] = "abc"; __msan_poison(buf + 2, sizeof(*buf)); - char *x; - EXPECT_UMR(x = strndup(buf, 3)); + char *x = strndup(buf, 3); EXPECT_NOT_POISONED(x[0]); EXPECT_NOT_POISONED(x[1]); EXPECT_POISONED(x[2]); @@ -1594,8 +1593,7 @@ TEST(MemorySanitizer, strndup_short) { char buf[4] = "abc"; __msan_poison(buf + 1, sizeof(*buf)); __msan_poison(buf + 2, sizeof(*buf)); - char *x; - EXPECT_UMR(x = strndup(buf, 2)); + char *x = strndup(buf, 2); EXPECT_NOT_POISONED(x[0]); EXPECT_POISONED(x[1]); EXPECT_NOT_POISONED(x[2]); Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.cc Mon May 22 19:43:56 2017 (r318667) @@ -285,9 +285,9 @@ void LoadedModule::clear() { } void LoadedModule::addAddressRange(uptr beg, uptr end, bool executable, - bool readable) { + bool writable) { void *mem = InternalAlloc(sizeof(AddressRange)); - AddressRange *r = new(mem) AddressRange(beg, end, executable, readable); + AddressRange *r = new(mem) AddressRange(beg, end, executable, writable); ranges_.push_back(r); if (executable && end > max_executable_address_) max_executable_address_ = end; Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.h ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.h Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common.h Mon May 22 19:43:56 2017 (r318667) @@ -717,7 +717,7 @@ class LoadedModule { void set(const char *module_name, uptr base_address, ModuleArch arch, u8 uuid[kModuleUUIDSize], bool instrumented); void clear(); - void addAddressRange(uptr beg, uptr end, bool executable, bool readable); + void addAddressRange(uptr beg, uptr end, bool executable, bool writable); bool containsAddress(uptr address) const; const char *full_name() const { return full_name_; } @@ -732,14 +732,14 @@ class LoadedModule { uptr beg; uptr end; bool executable; - bool readable; + bool writable; - AddressRange(uptr beg, uptr end, bool executable, bool readable) + AddressRange(uptr beg, uptr end, bool executable, bool writable) : next(nullptr), beg(beg), end(end), executable(executable), - readable(readable) {} + writable(writable) {} }; const IntrusiveList &ranges() const { return ranges_; } Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common_interceptors.inc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common_interceptors.inc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_common_interceptors.inc Mon May 22 19:43:56 2017 (r318667) @@ -34,8 +34,6 @@ // COMMON_INTERCEPTOR_MEMSET_IMPL // COMMON_INTERCEPTOR_MEMMOVE_IMPL // COMMON_INTERCEPTOR_MEMCPY_IMPL -// COMMON_INTERCEPTOR_COPY_STRING -// COMMON_INTERCEPTOR_STRNDUP_IMPL //===----------------------------------------------------------------------===// #include "interception/interception.h" @@ -219,25 +217,6 @@ bool PlatformHasDifferentMemcpyAndMemmov } #endif -#ifndef COMMON_INTERCEPTOR_COPY_STRING -#define COMMON_INTERCEPTOR_COPY_STRING(ctx, to, from, size) {} -#endif - -#ifndef COMMON_INTERCEPTOR_STRNDUP_IMPL -#define COMMON_INTERCEPTOR_STRNDUP_IMPL(ctx, s, size) \ - COMMON_INTERCEPTOR_ENTER(ctx, strndup, s, size); \ - uptr from_length = internal_strnlen(s, size); \ - uptr copy_length = Min(size, from_length); \ - char *new_mem = (char *)WRAP(malloc)(copy_length + 1); \ - if (common_flags()->intercept_strndup) { \ - COMMON_INTERCEPTOR_READ_RANGE(ctx, s, copy_length + 1); \ - } \ - COMMON_INTERCEPTOR_COPY_STRING(ctx, new_mem, s, copy_length); \ - internal_memcpy(new_mem, s, copy_length); \ - new_mem[copy_length] = '\0'; \ - return new_mem; -#endif - struct FileMetadata { // For open_memstream(). char **addr; @@ -321,26 +300,6 @@ INTERCEPTOR(SIZE_T, strnlen, const char #define INIT_STRNLEN #endif -#if SANITIZER_INTERCEPT_STRNDUP -INTERCEPTOR(char*, strndup, const char *s, uptr size) { - void *ctx; - COMMON_INTERCEPTOR_STRNDUP_IMPL(ctx, s, size); -} -#define INIT_STRNDUP COMMON_INTERCEPT_FUNCTION(strndup) -#else -#define INIT_STRNDUP -#endif // SANITIZER_INTERCEPT_STRNDUP - -#if SANITIZER_INTERCEPT___STRNDUP -INTERCEPTOR(char*, __strndup, const char *s, uptr size) { - void *ctx; - COMMON_INTERCEPTOR_STRNDUP_IMPL(ctx, s, size); -} -#define INIT___STRNDUP COMMON_INTERCEPT_FUNCTION(__strndup) -#else -#define INIT___STRNDUP -#endif // SANITIZER_INTERCEPT___STRNDUP - #if SANITIZER_INTERCEPT_TEXTDOMAIN INTERCEPTOR(char*, textdomain, const char *domainname) { void *ctx; @@ -6204,8 +6163,6 @@ static void InitializeCommonInterceptors INIT_TEXTDOMAIN; INIT_STRLEN; INIT_STRNLEN; - INIT_STRNDUP; - INIT___STRNDUP; INIT_STRCMP; INIT_STRNCMP; INIT_STRCASECMP; Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flag_parser.h ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flag_parser.h Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flag_parser.h Mon May 22 19:43:56 2017 (r318667) @@ -34,25 +34,41 @@ class FlagHandler : public FlagHandlerBa bool Parse(const char *value) final; }; -template <> -inline bool FlagHandler::Parse(const char *value) { +inline bool ParseBool(const char *value, bool *b) { if (internal_strcmp(value, "0") == 0 || internal_strcmp(value, "no") == 0 || internal_strcmp(value, "false") == 0) { - *t_ = false; + *b = false; return true; } if (internal_strcmp(value, "1") == 0 || internal_strcmp(value, "yes") == 0 || internal_strcmp(value, "true") == 0) { - *t_ = true; + *b = true; return true; } + return false; +} + +template <> +inline bool FlagHandler::Parse(const char *value) { + if (ParseBool(value, t_)) return true; Printf("ERROR: Invalid value for bool option: '%s'\n", value); return false; } template <> +inline bool FlagHandler::Parse(const char *value) { + bool b; + if (ParseBool(value, &b)) { + *t_ = b ? kHandleSignalYes : kHandleSignalNo; + return true; + } + Printf("ERROR: Invalid value for signal handler option: '%s'\n", value); + return false; +} + +template <> inline bool FlagHandler::Parse(const char *value) { *t_ = internal_strdup(value); return true; Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.h ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.h Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.h Mon May 22 19:43:56 2017 (r318667) @@ -18,6 +18,11 @@ namespace __sanitizer { +enum HandleSignalMode { + kHandleSignalNo, + kHandleSignalYes, +}; + struct CommonFlags { #define COMMON_FLAG(Type, Name, DefaultValue, Description) Type Name; #include "sanitizer_flags.inc" Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.inc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.inc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_flags.inc Mon May 22 19:43:56 2017 (r318667) @@ -78,16 +78,20 @@ COMMON_FLAG(int, print_module_map, 0, "OS X only. 0 = don't print, 1 = print only once before process " "exits, 2 = print after each report.") COMMON_FLAG(bool, check_printf, true, "Check printf arguments.") -COMMON_FLAG(bool, handle_segv, true, - "If set, registers the tool's custom SIGSEGV handler.") -COMMON_FLAG(bool, handle_sigbus, true, - "If set, registers the tool's custom SIGBUS handler.") -COMMON_FLAG(bool, handle_abort, false, - "If set, registers the tool's custom SIGABRT handler.") -COMMON_FLAG(bool, handle_sigill, false, - "If set, registers the tool's custom SIGILL handler.") -COMMON_FLAG(bool, handle_sigfpe, true, - "If set, registers the tool's custom SIGFPE handler.") +#define COMMON_FLAG_HANDLE_SIGNAL_HELP(signal) \ + "Controls custom tool's " #signal " handler (0 - do not registers the " \ + "handler, 1 - register the handler). " +COMMON_FLAG(HandleSignalMode, handle_segv, kHandleSignalYes, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGSEGV)) +COMMON_FLAG(HandleSignalMode, handle_sigbus, kHandleSignalYes, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGBUS)) +COMMON_FLAG(HandleSignalMode, handle_abort, kHandleSignalNo, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGABRT)) +COMMON_FLAG(HandleSignalMode, handle_sigill, kHandleSignalNo, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGILL)) +COMMON_FLAG(HandleSignalMode, handle_sigfpe, kHandleSignalYes, + COMMON_FLAG_HANDLE_SIGNAL_HELP(SIGFPE)) +#undef COMMON_FLAG_HANDLE_SIGNAL_HELP COMMON_FLAG(bool, allow_user_segv_handler, false, "If set, allows user to register a SEGV handler even if the tool " "registers one.") @@ -195,9 +199,6 @@ COMMON_FLAG(bool, intercept_strpbrk, tru COMMON_FLAG(bool, intercept_strlen, true, "If set, uses custom wrappers for strlen and strnlen functions " "to find more errors.") -COMMON_FLAG(bool, intercept_strndup, true, - "If set, uses custom wrappers for strndup functions " - "to find more errors.") COMMON_FLAG(bool, intercept_strchr, true, "If set, uses custom wrappers for strchr, strchrnul, and strrchr " "functions to find more errors.") Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux.cc Mon May 22 19:43:56 2017 (r318667) @@ -1395,15 +1395,19 @@ AndroidApiLevel AndroidGetApiLevel() { #endif bool IsHandledDeadlySignal(int signum) { - if (common_flags()->handle_abort && signum == SIGABRT) - return true; - if (common_flags()->handle_sigill && signum == SIGILL) - return true; - if (common_flags()->handle_sigfpe && signum == SIGFPE) - return true; - if (common_flags()->handle_segv && signum == SIGSEGV) - return true; - return common_flags()->handle_sigbus && signum == SIGBUS; + switch (signum) { + case SIGABRT: + return common_flags()->handle_abort; + case SIGILL: + return common_flags()->handle_sigill; + case SIGFPE: + return common_flags()->handle_sigfpe; + case SIGSEGV: + return common_flags()->handle_segv; + case SIGBUS: + return common_flags()->handle_sigbus; + } + return false; } #if !SANITIZER_GO Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux_libcdep.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux_libcdep.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_linux_libcdep.cc Mon May 22 19:43:56 2017 (r318667) @@ -447,9 +447,9 @@ static int dl_iterate_phdr_cb(dl_phdr_in uptr cur_beg = info->dlpi_addr + phdr->p_vaddr; uptr cur_end = cur_beg + phdr->p_memsz; bool executable = phdr->p_flags & PF_X; - bool readable = phdr->p_flags & PF_R; + bool writable = phdr->p_flags & PF_W; cur_module.addAddressRange(cur_beg, cur_end, executable, - readable); + writable); } } data->modules->push_back(cur_module); Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mac.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mac.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_mac.cc Mon May 22 19:43:56 2017 (r318667) @@ -394,18 +394,22 @@ void ListOfModules::init() { } bool IsHandledDeadlySignal(int signum) { + // Handling fatal signals on watchOS and tvOS devices is disallowed. if ((SANITIZER_WATCHOS || SANITIZER_TVOS) && !(SANITIZER_IOSSIM)) - // Handling fatal signals on watchOS and tvOS devices is disallowed. return false; - if (common_flags()->handle_abort && signum == SIGABRT) - return true; - if (common_flags()->handle_sigill && signum == SIGILL) - return true; - if (common_flags()->handle_sigfpe && signum == SIGFPE) - return true; - if (common_flags()->handle_segv && signum == SIGSEGV) - return true; - return common_flags()->handle_sigbus && signum == SIGBUS; + switch (signum) { + case SIGABRT: + return common_flags()->handle_abort; + case SIGILL: + return common_flags()->handle_sigill; + case SIGFPE: + return common_flags()->handle_sigfpe; + case SIGSEGV: + return common_flags()->handle_segv; + case SIGBUS: + return common_flags()->handle_sigbus; + } + return false; } MacosVersion cached_macos_version = MACOS_VERSION_UNINITIALIZED; Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_platform_interceptors.h ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_platform_interceptors.h Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_platform_interceptors.h Mon May 22 19:43:56 2017 (r318667) @@ -25,12 +25,6 @@ # define SI_NOT_WINDOWS 0 #endif -#if SANITIZER_POSIX -# define SI_POSIX 1 -#else -# define SI_POSIX 0 -#endif - #if SANITIZER_LINUX && !SANITIZER_ANDROID # define SI_LINUX_NOT_ANDROID 1 #else @@ -75,12 +69,6 @@ # define SI_UNIX_NOT_MAC 0 #endif -#if SANITIZER_LINUX && !SANITIZER_FREEBSD -# define SI_LINUX_NOT_FREEBSD 1 -# else -# define SI_LINUX_NOT_FREEBSD 0 -#endif - #define SANITIZER_INTERCEPT_STRLEN 1 #define SANITIZER_INTERCEPT_STRNLEN SI_NOT_MAC #define SANITIZER_INTERCEPT_STRCMP 1 @@ -98,8 +86,6 @@ #define SANITIZER_INTERCEPT_MEMMOVE 1 #define SANITIZER_INTERCEPT_MEMCPY 1 #define SANITIZER_INTERCEPT_MEMCMP 1 -#define SANITIZER_INTERCEPT_STRNDUP SI_POSIX -#define SANITIZER_INTERCEPT___STRNDUP SI_LINUX_NOT_FREEBSD #if defined(__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__) && \ __ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__ < 1070 # define SI_MAC_DEPLOYMENT_BELOW_10_7 1 Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_common.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_common.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_common.cc Mon May 22 19:43:56 2017 (r318667) @@ -142,7 +142,7 @@ void MemoryMappingLayout::DumpListOfModu LoadedModule cur_module; cur_module.set(cur_name, base_address); cur_module.addAddressRange(cur_beg, cur_end, prot & kProtectionExecute, - prot & kProtectionRead); + prot & kProtectionWrite); modules->push_back(cur_module); } } Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_mac.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_mac.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_procmaps_mac.cc Mon May 22 19:43:56 2017 (r318667) @@ -336,7 +336,7 @@ void MemoryMappingLayout::DumpListOfModu current_instrumented_); } cur_module->addAddressRange(cur_beg, cur_end, prot & kProtectionExecute, - prot & kProtectionRead); + prot & kProtectionWrite); } } Modified: vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/sanitizer_win.cc Mon May 22 19:43:56 2017 (r318667) @@ -554,7 +554,7 @@ void ListOfModules::init() { cur_module.set(module_name, adjusted_base); // We add the whole module as one single address range. cur_module.addAddressRange(base_address, end_address, /*executable*/ true, - /*readable*/ true); + /*writable*/ true); modules_.push_back(cur_module); } UnmapOrDie(hmodules, modules_buffer_size); Modified: vendor/compiler-rt/dist/lib/sanitizer_common/tests/CMakeLists.txt ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/tests/CMakeLists.txt Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/tests/CMakeLists.txt Mon May 22 19:43:56 2017 (r318667) @@ -146,6 +146,15 @@ set_target_properties(SanitizerUnitTests # Adds sanitizer tests for architecture. macro(add_sanitizer_tests_for_arch arch) get_target_flags_for_arch(${arch} TARGET_FLAGS) + + # If the sanitizer library was built with _FILE_OFFSET_BITS=64 we need + # to ensure that the library and tests agree on the layout of certain + # structures such as 'struct stat'. + if( CMAKE_SIZEOF_VOID_P EQUAL 4 ) + list(APPEND TARGET_FLAGS "-D_LARGEFILE_SOURCE") + list(APPEND TARGET_FLAGS "-D_FILE_OFFSET_BITS=64") + endif() + set(SANITIZER_TEST_SOURCES ${SANITIZER_UNITTESTS} ${COMPILER_RT_GTEST_SOURCE}) set(SANITIZER_TEST_COMPILE_DEPS ${SANITIZER_TEST_HEADERS}) Modified: vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_flags_test.cc ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_flags_test.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_flags_test.cc Mon May 22 19:43:56 2017 (r318667) @@ -59,6 +59,36 @@ TEST(SanitizerCommon, BooleanFlags) { TestFlag(true, "flag_name=0", false); TestFlag(true, "flag_name=no", false); TestFlag(true, "flag_name=false", false); + + EXPECT_DEATH(TestFlag(false, "flag_name", true), "expected '='"); + EXPECT_DEATH(TestFlag(false, "flag_name=", true), + "Invalid value for bool option: ''"); + EXPECT_DEATH(TestFlag(false, "flag_name=2", true), + "Invalid value for bool option: '2'"); + EXPECT_DEATH(TestFlag(false, "flag_name=-1", true), + "Invalid value for bool option: '-1'"); + EXPECT_DEATH(TestFlag(false, "flag_name=on", true), + "Invalid value for bool option: 'on'"); +} + +TEST(SanitizerCommon, HandleSignalMode) { + TestFlag(kHandleSignalNo, "flag_name=1", kHandleSignalYes); + TestFlag(kHandleSignalNo, "flag_name=yes", kHandleSignalYes); + TestFlag(kHandleSignalNo, "flag_name=true", kHandleSignalYes); + TestFlag(kHandleSignalYes, "flag_name=0", kHandleSignalNo); + TestFlag(kHandleSignalYes, "flag_name=no", kHandleSignalNo); + TestFlag(kHandleSignalYes, "flag_name=false", kHandleSignalNo); + + EXPECT_DEATH(TestFlag(kHandleSignalNo, "flag_name", kHandleSignalNo), + "expected '='"); + EXPECT_DEATH(TestFlag(kHandleSignalNo, "flag_name=", kHandleSignalNo), + "Invalid value for signal handler option: ''"); + EXPECT_DEATH(TestFlag(kHandleSignalNo, "flag_name=2", kHandleSignalNo), + "Invalid value for signal handler option: '2'"); + EXPECT_DEATH(TestFlag(kHandleSignalNo, "flag_name=-1", kHandleSignalNo), + "Invalid value for signal handler option: '-1'"); + EXPECT_DEATH(TestFlag(kHandleSignalNo, "flag_name=on", kHandleSignalNo), + "Invalid value for signal handler option: 'on'"); } TEST(SanitizerCommon, IntFlags) { Modified: vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_test_utils.h ============================================================================== --- vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_test_utils.h Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/sanitizer_common/tests/sanitizer_test_utils.h Mon May 22 19:43:56 2017 (r318667) @@ -124,10 +124,4 @@ static inline uint32_t my_rand() { # define SANITIZER_TEST_HAS_PRINTF_L 0 #endif -#if !defined(_MSC_VER) -# define SANITIZER_TEST_HAS_STRNDUP 1 -#else -# define SANITIZER_TEST_HAS_STRNDUP 0 -#endif - #endif // SANITIZER_TEST_UTILS_H Modified: vendor/compiler-rt/dist/lib/scudo/scudo_flags.cpp ============================================================================== --- vendor/compiler-rt/dist/lib/scudo/scudo_flags.cpp Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/scudo/scudo_flags.cpp Mon May 22 19:43:56 2017 (r318667) @@ -68,7 +68,7 @@ void initFlags() { // Sanity checks and default settings for the Quarantine parameters. if (f->QuarantineSizeMb < 0) { - const int DefaultQuarantineSizeMb = FIRST_32_SECOND_64(16, 64); + const int DefaultQuarantineSizeMb = FIRST_32_SECOND_64(4, 16); f->QuarantineSizeMb = DefaultQuarantineSizeMb; } // We enforce an upper limit for the quarantine size of 4Gb. @@ -77,7 +77,7 @@ void initFlags() { } if (f->ThreadLocalQuarantineSizeKb < 0) { const int DefaultThreadLocalQuarantineSizeKb = - FIRST_32_SECOND_64(256, 1024); + FIRST_32_SECOND_64(64, 256); f->ThreadLocalQuarantineSizeKb = DefaultThreadLocalQuarantineSizeKb; } // And an upper limit of 128Mb for the thread quarantine cache. Modified: vendor/compiler-rt/dist/lib/xray/xray_interface.cc ============================================================================== --- vendor/compiler-rt/dist/lib/xray/xray_interface.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/lib/xray/xray_interface.cc Mon May 22 19:43:56 2017 (r318667) @@ -119,10 +119,15 @@ int __xray_set_customevent_handler(void return 0; } + int __xray_remove_handler() XRAY_NEVER_INSTRUMENT { return __xray_set_handler(nullptr); } +int __xray_remove_customevent_handler() XRAY_NEVER_INSTRUMENT { + return __xray_set_customevent_handler(nullptr); +} + __sanitizer::atomic_uint8_t XRayPatching{0}; using namespace __xray; @@ -326,7 +331,14 @@ uintptr_t __xray_function_address(int32_ __sanitizer::SpinMutexLock Guard(&XRayInstrMapMutex); if (FuncId <= 0 || static_cast(FuncId) > XRayInstrMap.Functions) return 0; - return XRayInstrMap.SledsIndex[FuncId - 1].Begin->Address; + return XRayInstrMap.SledsIndex[FuncId - 1].Begin->Address +// On PPC, function entries are always aligned to 16 bytes. The beginning of a +// sled might be a local entry, which is always +8 based on the global entry. +// Always return the global entry. +#ifdef __PPC__ + & ~0xf +#endif + ; } size_t __xray_max_function_id() XRAY_NEVER_INSTRUMENT { Modified: vendor/compiler-rt/dist/test/asan/TestCases/Posix/asan-sigbus.cpp ============================================================================== --- vendor/compiler-rt/dist/test/asan/TestCases/Posix/asan-sigbus.cpp Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/test/asan/TestCases/Posix/asan-sigbus.cpp Mon May 22 19:43:56 2017 (r318667) @@ -2,7 +2,7 @@ // Defaults to true // RUN: %clangxx_asan -std=c++11 %s -o %t // RUN: not %run %t %T/file 2>&1 | FileCheck %s -check-prefix=CHECK-BUS -// RUN: %env_asan_opts=handle_sigbus=false not --crash %run %t %T/file 2>&1 | FileCheck %s +// RUN: %env_asan_opts=handle_sigbus=0 not --crash %run %t %T/file 2>&1 | FileCheck %s // UNSUPPORTED: ios Modified: vendor/compiler-rt/dist/test/xray/TestCases/Linux/custom-event-logging.cc ============================================================================== --- vendor/compiler-rt/dist/test/xray/TestCases/Linux/custom-event-logging.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/test/xray/TestCases/Linux/custom-event-logging.cc Mon May 22 19:43:56 2017 (r318667) @@ -33,7 +33,7 @@ int main() { // CHECK-NEXT: after calling the custom logging... printf("removing custom event handler...\n"); // CHECK-NEXT: removing custom event handler... - __xray_set_customevent_handler(nullptr); + __xray_remove_customevent_handler(); foo(); // CHECK-NEXT: before calling the custom logging... // CHECK-NEXT: after calling the custom logging... Modified: vendor/compiler-rt/dist/test/xray/TestCases/Linux/func-id-utils.cc ============================================================================== --- vendor/compiler-rt/dist/test/xray/TestCases/Linux/func-id-utils.cc Mon May 22 19:43:53 2017 (r318666) +++ vendor/compiler-rt/dist/test/xray/TestCases/Linux/func-id-utils.cc Mon May 22 19:43:56 2017 (r318667) @@ -31,18 +31,10 @@ "each function id must be assigned to a unique function"); std::set not_instrumented; - const auto comp = [](void *lhs, void *rhs) { -#ifdef __PPC__ - return reinterpret_cast(lhs) + 8 < - reinterpret_cast(rhs); -#else - return lhs < rhs; -#endif - }; - std::set_difference(must_be_instrumented.begin(), must_be_instrumented.end(), - all_instrumented.begin(), all_instrumented.end(), - std::inserter(not_instrumented, not_instrumented.begin()), - comp); + std::set_difference( + must_be_instrumented.begin(), must_be_instrumented.end(), + all_instrumented.begin(), all_instrumented.end(), + std::inserter(not_instrumented, not_instrumented.begin())); assert( not_instrumented.empty() && "we should see all explicitly instrumented functions with function ids"); From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:04 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60ADBD78AA0; Mon, 22 May 2017 19:44:04 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A3ED1AC9; Mon, 22 May 2017 19:44:04 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJi3gw076786; Mon, 22 May 2017 19:44:03 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJi3uE076785; Mon, 22 May 2017 19:44:03 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221944.v4MJi3uE076785@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:44:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318668 - vendor/compiler-rt/compiler-rt-trunk-r303571 X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:04 -0000 Author: dim Date: Mon May 22 19:44:02 2017 New Revision: 318668 URL: https://svnweb.freebsd.org/changeset/base/318668 Log: Tag compiler-rt trunk r303571. Added: vendor/compiler-rt/compiler-rt-trunk-r303571/ - copied from r318667, vendor/compiler-rt/dist/ From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:07 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD303D78AD7; Mon, 22 May 2017 19:44:07 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F4E61B1C; Mon, 22 May 2017 19:44:07 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJi6AF076839; Mon, 22 May 2017 19:44:06 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJi5Np076832; Mon, 22 May 2017 19:44:05 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221944.v4MJi5Np076832@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:44:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318669 - in vendor/libc++/dist: test/libcxx/selftest utils/libcxx utils/libcxx/test X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:07 -0000 Author: dim Date: Mon May 22 19:44:05 2017 New Revision: 318669 URL: https://svnweb.freebsd.org/changeset/base/318669 Log: Vendor import of libc++ trunk r303571: https://llvm.org/svn/llvm-project/libcxx/trunk@303571 Added: vendor/libc++/dist/test/libcxx/selftest/test.arc.fail.mm vendor/libc++/dist/test/libcxx/selftest/test.arc.pass.mm vendor/libc++/dist/test/libcxx/selftest/test.fail.mm vendor/libc++/dist/test/libcxx/selftest/test.pass.mm Modified: vendor/libc++/dist/utils/libcxx/compiler.py vendor/libc++/dist/utils/libcxx/test/config.py vendor/libc++/dist/utils/libcxx/test/format.py Added: vendor/libc++/dist/test/libcxx/selftest/test.arc.fail.mm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libc++/dist/test/libcxx/selftest/test.arc.fail.mm Mon May 22 19:44:05 2017 (r318669) @@ -0,0 +1,13 @@ +// -*- C++ -*- +//===----------------------------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is dual licensed under the MIT and the University of Illinois Open +// Source Licenses. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// + +#if __has_feature(objc_arc) +#error This test should not compile. +#endif Added: vendor/libc++/dist/test/libcxx/selftest/test.arc.pass.mm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libc++/dist/test/libcxx/selftest/test.arc.pass.mm Mon May 22 19:44:05 2017 (r318669) @@ -0,0 +1,17 @@ +// -*- C++ -*- +//===----------------------------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is dual licensed under the MIT and the University of Illinois Open +// Source Licenses. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// + +#if __has_feature(objc_arc) == 0 +#error "arc should be enabled" +#endif + +int main() +{ +} Added: vendor/libc++/dist/test/libcxx/selftest/test.fail.mm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libc++/dist/test/libcxx/selftest/test.fail.mm Mon May 22 19:44:05 2017 (r318669) @@ -0,0 +1,13 @@ +// -*- C++ -*- +//===----------------------------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is dual licensed under the MIT and the University of Illinois Open +// Source Licenses. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// + +#if __has_feature(objc_arc) == 0 +#error This test should not compile. +#endif Added: vendor/libc++/dist/test/libcxx/selftest/test.pass.mm ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor/libc++/dist/test/libcxx/selftest/test.pass.mm Mon May 22 19:44:05 2017 (r318669) @@ -0,0 +1,17 @@ +// -*- C++ -*- +//===----------------------------------------------------------------------===// +// +// The LLVM Compiler Infrastructure +// +// This file is dual licensed under the MIT and the University of Illinois Open +// Source Licenses. See LICENSE.TXT for details. +// +//===----------------------------------------------------------------------===// + +#if __has_feature(objc_arc) +#error "arc should *not* be enabled" +#endif + +int main() +{ +} Modified: vendor/libc++/dist/utils/libcxx/compiler.py ============================================================================== --- vendor/libc++/dist/utils/libcxx/compiler.py Mon May 22 19:44:02 2017 (r318668) +++ vendor/libc++/dist/utils/libcxx/compiler.py Mon May 22 19:44:05 2017 (r318669) @@ -24,6 +24,7 @@ class CXXCompiler(object): modules_flags=None, use_modules=False, use_ccache=False, use_warnings=False, compile_env=None, cxx_type=None, cxx_version=None): + self.source_lang = 'c++' self.path = path self.flags = list(flags or []) self.compile_flags = list(compile_flags or []) @@ -108,7 +109,7 @@ class CXXCompiler(object): if out is not None: cmd += ['-o', out] if input_is_cxx: - cmd += ['-x', 'c++'] + cmd += ['-x', self.source_lang] if isinstance(source_files, list): cmd += source_files elif isinstance(source_files, str): Modified: vendor/libc++/dist/utils/libcxx/test/config.py ============================================================================== --- vendor/libc++/dist/utils/libcxx/test/config.py Mon May 22 19:44:02 2017 (r318668) +++ vendor/libc++/dist/utils/libcxx/test/config.py Mon May 22 19:44:05 2017 (r318669) @@ -466,6 +466,12 @@ class Configuration(object): self.config.available_features.add('glibc-%s' % maj_v) self.config.available_features.add('glibc-%s.%s' % (maj_v, min_v)) + # Support Objective-C++ only on MacOS and if the compiler supports it. + if self.target_info.platform() == "darwin" and \ + self.target_info.is_host_macosx() and \ + self.cxx.hasCompileFlag(["-x", "objective-c++", "-fobjc-arc"]): + self.config.available_features.add("objective-c++") + def configure_compile_flags(self): no_default_flags = self.get_lit_bool('no_default_flags', False) if not no_default_flags: Modified: vendor/libc++/dist/utils/libcxx/test/format.py ============================================================================== --- vendor/libc++/dist/utils/libcxx/test/format.py Mon May 22 19:44:02 2017 (r318668) +++ vendor/libc++/dist/utils/libcxx/test/format.py Mon May 22 19:44:05 2017 (r318669) @@ -87,14 +87,22 @@ class LibcxxTestFormat(object): name_root, name_ext = os.path.splitext(name) is_libcxx_test = test.path_in_suite[0] == 'libcxx' is_sh_test = name_root.endswith('.sh') - is_pass_test = name.endswith('.pass.cpp') - is_fail_test = name.endswith('.fail.cpp') - assert is_sh_test or name_ext == '.cpp', 'non-cpp file must be sh test' + is_pass_test = name.endswith('.pass.cpp') or name.endswith('.pass.mm') + is_fail_test = name.endswith('.fail.cpp') or name.endswith('.fail.mm') + is_objcxx_test = name.endswith('.mm') + is_objcxx_arc_test = name.endswith('.arc.pass.mm') or \ + name.endswith('.arc.fail.mm') + assert is_sh_test or name_ext == '.cpp' or name_ext == '.mm', \ + 'non-cpp file must be sh test' if test.config.unsupported: return (lit.Test.UNSUPPORTED, "A lit.local.cfg marked this unsupported") + if is_objcxx_test and not \ + 'objective-c++' in test.config.available_features: + return (lit.Test.UNSUPPORTED, "Objective-C++ is not supported") + parsers = self._make_custom_parsers() script = lit.TestRunner.parseIntegratedTestScript( test, additional_parsers=parsers, require_script=is_sh_test) @@ -133,6 +141,14 @@ class LibcxxTestFormat(object): if '#define _LIBCPP_ASSERT' in contents: test_cxx.useModules(False) + if is_objcxx_test: + test_cxx.source_lang = 'objective-c++' + if is_objcxx_arc_test: + test_cxx.compile_flags += ['-fobjc-arc'] + else: + test_cxx.compile_flags += ['-fno-objc-arc'] + test_cxx.link_flags += ['-framework', 'Foundation'] + # Dispatch the test based on its suffix. if is_sh_test: if not isinstance(self.executor, LocalExecutor): From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:11 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93D9AD78B1D; Mon, 22 May 2017 19:44:11 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47D131BB2; Mon, 22 May 2017 19:44:11 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJiAAS076887; Mon, 22 May 2017 19:44:10 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJiA39076886; Mon, 22 May 2017 19:44:10 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221944.v4MJiA39076886@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:44:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318670 - vendor/libc++/libc++-trunk-r303571 X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:11 -0000 Author: dim Date: Mon May 22 19:44:10 2017 New Revision: 318670 URL: https://svnweb.freebsd.org/changeset/base/318670 Log: Tag libc++ trunk r303571. Added: vendor/libc++/libc++-trunk-r303571/ - copied from r318669, vendor/libc++/dist/ From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:19 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E183AD78BC9; Mon, 22 May 2017 19:44:19 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 93E501CBC; Mon, 22 May 2017 19:44:19 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJiIDe077000; Mon, 22 May 2017 19:44:18 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJiIHm076999; Mon, 22 May 2017 19:44:18 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221944.v4MJiIHm076999@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:44:18 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318672 - vendor/lld/lld-trunk-r303571 X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:20 -0000 Author: dim Date: Mon May 22 19:44:18 2017 New Revision: 318672 URL: https://svnweb.freebsd.org/changeset/base/318672 Log: Tag lld trunk r303571. Added: vendor/lld/lld-trunk-r303571/ - copied from r318671, vendor/lld/dist/ From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:22 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5096D78C02; Mon, 22 May 2017 19:44:22 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B237B1D01; Mon, 22 May 2017 19:44:22 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJiLH1077052; Mon, 22 May 2017 19:44:21 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJiLc4077047; Mon, 22 May 2017 19:44:21 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221944.v4MJiLc4077047@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:44:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318673 - in vendor/lldb/dist: include/lldb/Utility packages/Python/lldbsuite/test source/Core source/Utility unittests/Utility X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:22 -0000 Author: dim Date: Mon May 22 19:44:21 2017 New Revision: 318673 URL: https://svnweb.freebsd.org/changeset/base/318673 Log: Vendor import of lldb trunk r303571: https://llvm.org/svn/llvm-project/lldb/trunk@303571 Modified: vendor/lldb/dist/include/lldb/Utility/Status.h vendor/lldb/dist/packages/Python/lldbsuite/test/configuration.py vendor/lldb/dist/source/Core/IOHandler.cpp vendor/lldb/dist/source/Utility/Status.cpp vendor/lldb/dist/unittests/Utility/StatusTest.cpp Modified: vendor/lldb/dist/include/lldb/Utility/Status.h ============================================================================== --- vendor/lldb/dist/include/lldb/Utility/Status.h Mon May 22 19:44:18 2017 (r318672) +++ vendor/lldb/dist/include/lldb/Utility/Status.h Mon May 22 19:44:21 2017 (r318673) @@ -1,5 +1,4 @@ -//===-- Status.h -------------------------------------------------*- C++ -//-*-===// +//===-- Status.h ------------------------------------------------*- C++ -*-===// // // The LLVM Compiler Infrastructure // @@ -8,22 +7,20 @@ // //===----------------------------------------------------------------------===// -#ifndef __DCError_h__ -#define __DCError_h__ -#if defined(__cplusplus) +#ifndef LLDB_UTILITY_STATUS_H +#define LLDB_UTILITY_STATUS_H #include "lldb/lldb-defines.h" #include "lldb/lldb-enumerations.h" // for ErrorType, ErrorType... #include "llvm/ADT/StringRef.h" // for StringRef +#include "llvm/Support/Error.h" #include "llvm/Support/FormatVariadic.h" - #include +#include // for uint32_t #include #include // for error_code #include // for forward -#include // for uint32_t - namespace llvm { class raw_ostream; } @@ -106,6 +103,10 @@ public: ~Status(); + // llvm::Error support + explicit Status(llvm::Error error); + llvm::Error ToError() const; + //------------------------------------------------------------------ /// Get the error string associated with the current error. // @@ -274,5 +275,4 @@ template <> struct format_providerCreateSubWindow("Threads", threads_bounds, false)); WindowSP status_window_sp( - main_window_sp->CreateSubWindow("Error", status_bounds, false)); + main_window_sp->CreateSubWindow("Status", status_bounds, false)); status_window_sp->SetCanBeActive( false); // Don't let the status bar become the active window main_window_sp->SetDelegate( Modified: vendor/lldb/dist/source/Utility/Status.cpp ============================================================================== --- vendor/lldb/dist/source/Utility/Status.cpp Mon May 22 19:44:18 2017 (r318672) +++ vendor/lldb/dist/source/Utility/Status.cpp Mon May 22 19:44:21 2017 (r318673) @@ -56,6 +56,37 @@ Status::Status(const char *format, ...) va_end(args); } +Status::Status(llvm::Error error) + : m_code(0), m_type(ErrorType::eErrorTypeGeneric) { + if (!error) + return; + + // if the error happens to be a errno error, preserve the error code + error = llvm::handleErrors( + std::move(error), [&](std::unique_ptr e) -> llvm::Error { + std::error_code ec = e->convertToErrorCode(); + if (ec.category() == std::generic_category()) { + m_code = ec.value(); + m_type = ErrorType::eErrorTypePOSIX; + return llvm::Error::success(); + } + return llvm::Error(std::move(e)); + }); + + // Otherwise, just preserve the message + if (error) + SetErrorString(llvm::toString(std::move(error))); +} + +llvm::Error Status::ToError() const { + if (Success()) + return llvm::Error::success(); + if (m_type == ErrorType::eErrorTypePOSIX) + return llvm::errorCodeToError(std::error_code(m_code, std::generic_category())); + return llvm::make_error(AsCString(), + llvm::inconvertibleErrorCode()); +} + //---------------------------------------------------------------------- // Assignment operator //---------------------------------------------------------------------- Modified: vendor/lldb/dist/unittests/Utility/StatusTest.cpp ============================================================================== --- vendor/lldb/dist/unittests/Utility/StatusTest.cpp Mon May 22 19:44:18 2017 (r318672) +++ vendor/lldb/dist/unittests/Utility/StatusTest.cpp Mon May 22 19:44:21 2017 (r318673) @@ -11,9 +11,40 @@ #include "gtest/gtest.h" using namespace lldb_private; +using namespace lldb; TEST(StatusTest, Formatv) { EXPECT_EQ("", llvm::formatv("{0}", Status()).str()); EXPECT_EQ("Hello Status", llvm::formatv("{0}", Status("Hello Status")).str()); EXPECT_EQ("Hello", llvm::formatv("{0:5}", Status("Hello Error")).str()); } + +TEST(StatusTest, ErrorConstructor) { + EXPECT_TRUE(Status(llvm::Error::success()).Success()); + + Status eagain( + llvm::errorCodeToError(std::error_code(EAGAIN, std::generic_category()))); + EXPECT_TRUE(eagain.Fail()); + EXPECT_EQ(eErrorTypePOSIX, eagain.GetType()); + EXPECT_EQ(Status::ValueType(EAGAIN), eagain.GetError()); + + Status foo(llvm::make_error( + "foo", llvm::inconvertibleErrorCode())); + EXPECT_TRUE(foo.Fail()); + EXPECT_EQ(eErrorTypeGeneric, foo.GetType()); + EXPECT_STREQ("foo", foo.AsCString()); +} + +TEST(StatusTest, ErrorConversion) { + EXPECT_FALSE(bool(Status().ToError())); + + llvm::Error eagain = Status(EAGAIN, ErrorType::eErrorTypePOSIX).ToError(); + EXPECT_TRUE(bool(eagain)); + std::error_code ec = llvm::errorToErrorCode(std::move(eagain)); + EXPECT_EQ(EAGAIN, ec.value()); + EXPECT_EQ(std::generic_category(), ec.category()); + + llvm::Error foo = Status("foo").ToError(); + EXPECT_TRUE(bool(foo)); + EXPECT_EQ("foo", llvm::toString(std::move(foo))); +} From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:15 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5082D78B5E; Mon, 22 May 2017 19:44:15 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9CFFB1C2C; Mon, 22 May 2017 19:44:15 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJiE01076952; Mon, 22 May 2017 19:44:14 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJiCi8076934; Mon, 22 May 2017 19:44:12 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221944.v4MJiCi8076934@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:44:12 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318671 - in vendor/lld/dist: COFF ELF test/COFF test/ELF X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:16 -0000 Author: dim Date: Mon May 22 19:44:12 2017 New Revision: 318671 URL: https://svnweb.freebsd.org/changeset/base/318671 Log: Vendor import of lld trunk r303571: https://llvm.org/svn/llvm-project/lld/trunk@303571 Added: vendor/lld/dist/test/COFF/pdb-options.test vendor/lld/dist/test/ELF/arm-sbrel32.s (contents, props changed) Deleted: vendor/lld/dist/COFF/Librarian.cpp vendor/lld/dist/COFF/ModuleDef.cpp Modified: vendor/lld/dist/COFF/CMakeLists.txt vendor/lld/dist/COFF/Config.h vendor/lld/dist/COFF/DLL.cpp vendor/lld/dist/COFF/DLL.h vendor/lld/dist/COFF/Driver.cpp vendor/lld/dist/COFF/Driver.h vendor/lld/dist/COFF/DriverUtils.cpp vendor/lld/dist/COFF/InputFiles.cpp vendor/lld/dist/COFF/InputFiles.h vendor/lld/dist/COFF/Options.td vendor/lld/dist/COFF/PDB.cpp vendor/lld/dist/COFF/SymbolTable.h vendor/lld/dist/COFF/Writer.cpp vendor/lld/dist/ELF/InputSection.cpp vendor/lld/dist/ELF/InputSection.h vendor/lld/dist/ELF/LinkerScript.cpp vendor/lld/dist/ELF/MapFile.cpp vendor/lld/dist/ELF/MapFile.h vendor/lld/dist/ELF/Relocations.cpp vendor/lld/dist/ELF/Relocations.h vendor/lld/dist/ELF/SyntheticSections.cpp vendor/lld/dist/ELF/SyntheticSections.h vendor/lld/dist/ELF/Target.cpp vendor/lld/dist/ELF/Writer.cpp vendor/lld/dist/test/COFF/armnt-imports.test vendor/lld/dist/test/COFF/hello32.test vendor/lld/dist/test/COFF/imports.test vendor/lld/dist/test/COFF/invalid-debug-type.test vendor/lld/dist/test/COFF/options.test vendor/lld/dist/test/COFF/pdb-none.test vendor/lld/dist/test/COFF/pdb.test Modified: vendor/lld/dist/COFF/CMakeLists.txt ============================================================================== --- vendor/lld/dist/COFF/CMakeLists.txt Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/CMakeLists.txt Mon May 22 19:44:12 2017 (r318671) @@ -14,11 +14,9 @@ add_lld_library(lldCOFF Error.cpp ICF.cpp InputFiles.cpp - Librarian.cpp LTO.cpp MapFile.cpp MarkLive.cpp - ModuleDef.cpp PDB.cpp Strings.cpp SymbolTable.cpp Modified: vendor/lld/dist/COFF/Config.h ============================================================================== --- vendor/lld/dist/COFF/Config.h Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/Config.h Mon May 22 19:44:12 2017 (r318671) @@ -155,7 +155,6 @@ struct Configuration { uint32_t MajorOSVersion = 6; uint32_t MinorOSVersion = 0; bool DynamicBase = true; - bool AllowBind = true; bool NxCompat = true; bool AllowIsolation = true; bool TerminalServerAware = true; @@ -164,7 +163,6 @@ struct Configuration { bool AppContainer = false; // This is for debugging. - bool DebugPdb = false; bool DumpPdb = false; }; Modified: vendor/lld/dist/COFF/DLL.cpp ============================================================================== --- vendor/lld/dist/COFF/DLL.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/DLL.cpp Mon May 22 19:44:12 2017 (r318671) @@ -100,13 +100,17 @@ public: void writeTo(uint8_t *Buf) const override { auto *E = (coff_import_directory_table_entry *)(Buf + OutputSectionOff); - E->ImportLookupTableRVA = LookupTab->getRVA(); E->NameRVA = DLLName->getRVA(); + + // The import descriptor table contains two pointers to + // the tables describing dllimported symbols. But the + // Windows loader actually uses only one. So we create + // only one table and set both fields to its address. + E->ImportLookupTableRVA = AddressTab->getRVA(); E->ImportAddressTableRVA = AddressTab->getRVA(); } Chunk *DLLName; - Chunk *LookupTab; Chunk *AddressTab; }; @@ -136,9 +140,9 @@ binImports(const std::vectorgetDLLName().lower()].push_back(Sym); std::vector> V; - for (auto &P : M) { + for (auto &KV : M) { // Sort symbols by name for each group. - std::vector &Syms = P.second; + std::vector &Syms = KV.second; std::sort(Syms.begin(), Syms.end(), [](DefinedImportData *A, DefinedImportData *B) { return A->getName() < B->getName(); @@ -383,21 +387,14 @@ uint64_t IdataContents::getIATSize() { // See Microsoft PE/COFF spec 5.4 for details. std::vector IdataContents::getChunks() { create(); - std::vector V; + // The loader assumes a specific order of data. // Add each type in the correct order. - for (std::unique_ptr &C : Dirs) - V.push_back(C.get()); - for (std::unique_ptr &C : Lookups) - V.push_back(C.get()); - for (std::unique_ptr &C : Addresses) - V.push_back(C.get()); - for (std::unique_ptr &C : Hints) - V.push_back(C.get()); - for (auto &P : DLLNames) { - std::unique_ptr &C = P.second; - V.push_back(C.get()); - } + std::vector V; + V.insert(V.end(), Dirs.begin(), Dirs.end()); + V.insert(V.end(), Addresses.begin(), Addresses.end()); + V.insert(V.end(), Hints.begin(), Hints.end()); + V.insert(V.end(), DLLNames.begin(), DLLNames.end()); return V; } @@ -406,65 +403,50 @@ void IdataContents::create() { // Create .idata contents for each DLL. for (std::vector &Syms : V) { - StringRef Name = Syms[0]->getDLLName(); - // Create lookup and address tables. If they have external names, // we need to create HintName chunks to store the names. // If they don't (if they are import-by-ordinals), we store only // ordinal values to the table. - size_t Base = Lookups.size(); + size_t Base = Addresses.size(); for (DefinedImportData *S : Syms) { uint16_t Ord = S->getOrdinal(); if (S->getExternalName().empty()) { - Lookups.push_back(make_unique(Ord)); - Addresses.push_back(make_unique(Ord)); + Addresses.push_back(make(Ord)); continue; } - auto C = make_unique(S->getExternalName(), Ord); - Lookups.push_back(make_unique(C.get())); - Addresses.push_back(make_unique(C.get())); - Hints.push_back(std::move(C)); + auto *C = make(S->getExternalName(), Ord); + Addresses.push_back(make(C)); + Hints.push_back(C); } // Terminate with null values. - Lookups.push_back(make_unique(ptrSize())); - Addresses.push_back(make_unique(ptrSize())); + Addresses.push_back(make(ptrSize())); for (int I = 0, E = Syms.size(); I < E; ++I) - Syms[I]->setLocation(Addresses[Base + I].get()); + Syms[I]->setLocation(Addresses[Base + I]); // Create the import table header. - if (!DLLNames.count(Name)) - DLLNames[Name] = make_unique(Name); - auto Dir = make_unique(DLLNames[Name].get()); - Dir->LookupTab = Lookups[Base].get(); - Dir->AddressTab = Addresses[Base].get(); - Dirs.push_back(std::move(Dir)); + DLLNames.push_back(make(Syms[0]->getDLLName())); + auto *Dir = make(DLLNames.back()); + Dir->AddressTab = Addresses[Base]; + Dirs.push_back(Dir); } // Add null terminator. - Dirs.push_back(make_unique(sizeof(ImportDirectoryTableEntry))); + Dirs.push_back(make(sizeof(ImportDirectoryTableEntry))); } std::vector DelayLoadContents::getChunks() { std::vector V; - for (std::unique_ptr &C : Dirs) - V.push_back(C.get()); - for (std::unique_ptr &C : Names) - V.push_back(C.get()); - for (std::unique_ptr &C : HintNames) - V.push_back(C.get()); - for (auto &P : DLLNames) { - std::unique_ptr &C = P.second; - V.push_back(C.get()); - } + V.insert(V.end(), Dirs.begin(), Dirs.end()); + V.insert(V.end(), Names.begin(), Names.end()); + V.insert(V.end(), HintNames.begin(), HintNames.end()); + V.insert(V.end(), DLLNames.begin(), DLLNames.end()); return V; } std::vector DelayLoadContents::getDataChunks() { std::vector V; - for (std::unique_ptr &C : ModuleHandles) - V.push_back(C.get()); - for (std::unique_ptr &C : Addresses) - V.push_back(C.get()); + V.insert(V.end(), ModuleHandles.begin(), ModuleHandles.end()); + V.insert(V.end(), Addresses.begin(), Addresses.end()); return V; } @@ -478,55 +460,51 @@ void DelayLoadContents::create(Defined * // Create .didat contents for each DLL. for (std::vector &Syms : V) { - StringRef Name = Syms[0]->getDLLName(); - // Create the delay import table header. - if (!DLLNames.count(Name)) - DLLNames[Name] = make_unique(Name); - auto Dir = make_unique(DLLNames[Name].get()); + DLLNames.push_back(make(Syms[0]->getDLLName())); + auto *Dir = make(DLLNames.back()); size_t Base = Addresses.size(); for (DefinedImportData *S : Syms) { - Chunk *T = newThunkChunk(S, Dir.get()); - auto A = make_unique(T); - Addresses.push_back(std::move(A)); - Thunks.push_back(std::unique_ptr(T)); + Chunk *T = newThunkChunk(S, Dir); + auto *A = make(T); + Addresses.push_back(A); + Thunks.push_back(T); StringRef ExtName = S->getExternalName(); if (ExtName.empty()) { - Names.push_back(make_unique(S->getOrdinal())); + Names.push_back(make(S->getOrdinal())); } else { - auto C = make_unique(ExtName, 0); - Names.push_back(make_unique(C.get())); - HintNames.push_back(std::move(C)); + auto *C = make(ExtName, 0); + Names.push_back(make(C)); + HintNames.push_back(C); } } // Terminate with null values. - Addresses.push_back(make_unique(8)); - Names.push_back(make_unique(8)); + Addresses.push_back(make(8)); + Names.push_back(make(8)); for (int I = 0, E = Syms.size(); I < E; ++I) - Syms[I]->setLocation(Addresses[Base + I].get()); - auto *MH = new NullChunk(8); + Syms[I]->setLocation(Addresses[Base + I]); + auto *MH = make(8); MH->setAlign(8); - ModuleHandles.push_back(std::unique_ptr(MH)); + ModuleHandles.push_back(MH); // Fill the delay import table header fields. Dir->ModuleHandle = MH; - Dir->AddressTab = Addresses[Base].get(); - Dir->NameTab = Names[Base].get(); - Dirs.push_back(std::move(Dir)); + Dir->AddressTab = Addresses[Base]; + Dir->NameTab = Names[Base]; + Dirs.push_back(Dir); } // Add null terminator. - Dirs.push_back( - make_unique(sizeof(delay_import_directory_table_entry))); + Dirs.push_back(make(sizeof(delay_import_directory_table_entry))); } Chunk *DelayLoadContents::newThunkChunk(DefinedImportData *S, Chunk *Dir) { switch (Config->Machine) { case AMD64: - return new ThunkChunkX64(S, Dir, Helper); + return make(S, Dir, Helper); case I386: - return new ThunkChunkX86(S, Dir, Helper); + return make(S, Dir, Helper); default: llvm_unreachable("unsupported machine type"); } @@ -537,34 +515,32 @@ EdataContents::EdataContents() { for (Export &E : Config->Exports) MaxOrdinal = std::max(MaxOrdinal, E.Ordinal); - auto *DLLName = new StringChunk(sys::path::filename(Config->OutputFile)); - auto *AddressTab = new AddressTableChunk(MaxOrdinal); + auto *DLLName = make(sys::path::filename(Config->OutputFile)); + auto *AddressTab = make(MaxOrdinal); std::vector Names; for (Export &E : Config->Exports) if (!E.Noname) - Names.push_back(new StringChunk(E.ExportName)); + Names.push_back(make(E.ExportName)); std::vector Forwards; for (Export &E : Config->Exports) { if (E.ForwardTo.empty()) continue; - E.ForwardChunk = new StringChunk(E.ForwardTo); + E.ForwardChunk = make(E.ForwardTo); Forwards.push_back(E.ForwardChunk); } - auto *NameTab = new NamePointersChunk(Names); - auto *OrdinalTab = new ExportOrdinalChunk(Names.size()); - auto *Dir = new ExportDirectoryChunk(MaxOrdinal, Names.size(), DLLName, - AddressTab, NameTab, OrdinalTab); - Chunks.push_back(std::unique_ptr(Dir)); - Chunks.push_back(std::unique_ptr(DLLName)); - Chunks.push_back(std::unique_ptr(AddressTab)); - Chunks.push_back(std::unique_ptr(NameTab)); - Chunks.push_back(std::unique_ptr(OrdinalTab)); - for (Chunk *C : Names) - Chunks.push_back(std::unique_ptr(C)); - for (Chunk *C : Forwards) - Chunks.push_back(std::unique_ptr(C)); + auto *NameTab = make(Names); + auto *OrdinalTab = make(Names.size()); + auto *Dir = make(MaxOrdinal, Names.size(), DLLName, + AddressTab, NameTab, OrdinalTab); + Chunks.push_back(Dir); + Chunks.push_back(DLLName); + Chunks.push_back(AddressTab); + Chunks.push_back(NameTab); + Chunks.push_back(OrdinalTab); + Chunks.insert(Chunks.end(), Names.begin(), Names.end()); + Chunks.insert(Chunks.end(), Forwards.begin(), Forwards.end()); } } // namespace coff Modified: vendor/lld/dist/COFF/DLL.h ============================================================================== --- vendor/lld/dist/COFF/DLL.h Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/DLL.h Mon May 22 19:44:12 2017 (r318671) @@ -35,11 +35,10 @@ private: void create(); std::vector Imports; - std::vector> Dirs; - std::vector> Lookups; - std::vector> Addresses; - std::vector> Hints; - std::map> DLLNames; + std::vector Dirs; + std::vector Addresses; + std::vector Hints; + std::vector DLLNames; }; // Windows-specific. @@ -51,7 +50,7 @@ public: void create(Defined *Helper); std::vector getChunks(); std::vector getDataChunks(); - std::vector> &getCodeChunks() { return Thunks; } + ArrayRef getCodeChunks() { return Thunks; } uint64_t getDirRVA() { return Dirs[0]->getRVA(); } uint64_t getDirSize(); @@ -61,13 +60,13 @@ private: Defined *Helper; std::vector Imports; - std::vector> Dirs; - std::vector> ModuleHandles; - std::vector> Addresses; - std::vector> Names; - std::vector> HintNames; - std::vector> Thunks; - std::map> DLLNames; + std::vector Dirs; + std::vector ModuleHandles; + std::vector Addresses; + std::vector Names; + std::vector HintNames; + std::vector Thunks; + std::vector DLLNames; }; // Windows-specific. @@ -75,7 +74,7 @@ private: class EdataContents { public: EdataContents(); - std::vector> Chunks; + std::vector Chunks; }; } // namespace coff Modified: vendor/lld/dist/COFF/Driver.cpp ============================================================================== --- vendor/lld/dist/COFF/Driver.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/Driver.cpp Mon May 22 19:44:12 2017 (r318671) @@ -19,6 +19,8 @@ #include "llvm/ADT/Optional.h" #include "llvm/ADT/StringSwitch.h" #include "llvm/Object/ArchiveWriter.h" +#include "llvm/Object/COFFImportFile.h" +#include "llvm/Object/COFFModuleDefinition.h" #include "llvm/Option/Arg.h" #include "llvm/Option/ArgList.h" #include "llvm/Option/Option.h" @@ -35,6 +37,7 @@ #include using namespace llvm; +using namespace llvm::object; using namespace llvm::COFF; using llvm::sys::Process; using llvm::sys::fs::file_magic; @@ -97,12 +100,11 @@ static std::future createFutu MemoryBufferRef LinkerDriver::takeBuffer(std::unique_ptr MB) { MemoryBufferRef MBRef = *MB; - OwningMBs.push_back(std::move(MB)); + make>(std::move(MB)); // take ownership if (Driver->Tar) Driver->Tar->append(relativeToRoot(MBRef.getBufferIdentifier()), MBRef.getBuffer()); - return MBRef; } @@ -420,6 +422,84 @@ static std::string getMapFile(const opt: return (OutFile.substr(0, OutFile.rfind('.')) + ".map").str(); } +static std::string getImplibPath() { + if (!Config->Implib.empty()) + return Config->Implib; + SmallString<128> Out = StringRef(Config->OutputFile); + sys::path::replace_extension(Out, ".lib"); + return Out.str(); +} + +std::vector createCOFFShortExportFromConfig() { + std::vector Exports; + for (Export &E1 : Config->Exports) { + COFFShortExport E2; + E2.Name = E1.Name; + E2.ExtName = E1.ExtName; + E2.Ordinal = E1.Ordinal; + E2.Noname = E1.Noname; + E2.Data = E1.Data; + E2.Private = E1.Private; + E2.Constant = E1.Constant; + Exports.push_back(E2); + } + return Exports; +} + +static void createImportLibrary() { + std::vector Exports = createCOFFShortExportFromConfig(); + std::string DLLName = sys::path::filename(Config->OutputFile); + std::string Path = getImplibPath(); + writeImportLibrary(DLLName, Path, Exports, Config->Machine); +} + +static void parseModuleDefs(StringRef Path) { + std::unique_ptr MB = check( + MemoryBuffer::getFile(Path, -1, false, true), "could not open " + Path); + MemoryBufferRef MBRef = MB->getMemBufferRef(); + + Expected Def = + parseCOFFModuleDefinition(MBRef, Config->Machine); + if (!Def) + fatal(errorToErrorCode(Def.takeError()).message()); + + COFFModuleDefinition &M = *Def; + if (Config->OutputFile.empty()) + Config->OutputFile = Saver.save(M.OutputFile); + + if (M.ImageBase) + Config->ImageBase = M.ImageBase; + if (M.StackReserve) + Config->StackReserve = M.StackReserve; + if (M.StackCommit) + Config->StackCommit = M.StackCommit; + if (M.HeapReserve) + Config->HeapReserve = M.HeapReserve; + if (M.HeapCommit) + Config->HeapCommit = M.HeapCommit; + if (M.MajorImageVersion) + Config->MajorImageVersion = M.MajorImageVersion; + if (M.MinorImageVersion) + Config->MinorImageVersion = M.MinorImageVersion; + if (M.MajorOSVersion) + Config->MajorOSVersion = M.MajorOSVersion; + if (M.MinorOSVersion) + Config->MinorOSVersion = M.MinorOSVersion; + + for (COFFShortExport E1 : M.Exports) { + Export E2; + E2.Name = Saver.save(E1.Name); + if (E1.isWeak()) + E2.ExtName = Saver.save(E1.ExtName); + E2.Ordinal = E1.Ordinal; + E2.Noname = E1.Noname; + E2.Data = E1.Data; + E2.Private = E1.Private; + E2.Constant = E1.Constant; + Config->Exports.push_back(E2); + } +} + std::vector getArchiveMembers(Archive *File) { std::vector V; Error Err = Error::success(); @@ -821,8 +901,6 @@ void LinkerDriver::link(ArrayRefManifestInput.push_back(Arg->getValue()); // Handle miscellaneous boolean flags. - if (Args.hasArg(OPT_allowbind_no)) - Config->AllowBind = false; if (Args.hasArg(OPT_allowisolation_no)) Config->AllowIsolation = false; if (Args.hasArg(OPT_dynamicbase_no)) @@ -834,7 +912,6 @@ void LinkerDriver::link(ArrayRefWriteSymtab = false; Config->DumpPdb = Args.hasArg(OPT_dumppdb); - Config->DebugPdb = Args.hasArg(OPT_debugpdb); Config->MapFile = getMapFile(Args); @@ -916,9 +993,7 @@ void LinkerDriver::link(ArrayRefgetValue()), - Twine("could not open ") + Arg->getValue()))); + parseModuleDefs(Arg->getValue()); } // Handle /delayload @@ -1038,7 +1113,7 @@ void LinkerDriver::link(ArrayRefExports.empty() || Config->DLL) { fixupExports(); - writeImportLibrary(); + createImportLibrary(); assignExportOrdinals(); } Modified: vendor/lld/dist/COFF/Driver.h ============================================================================== --- vendor/lld/dist/COFF/Driver.h Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/Driver.h Mon May 22 19:44:12 2017 (r318671) @@ -119,18 +119,11 @@ private: void enqueueTask(std::function Task); bool run(); - // Driver is the owner of all opened files. - // InputFiles have MemoryBufferRefs to them. - std::vector> OwningMBs; - std::list> TaskQueue; std::vector FilePaths; std::vector Resources; }; -void parseModuleDefs(MemoryBufferRef MB); -void writeImportLibrary(); - // Functions below this line are defined in DriverUtils.cpp. void printHelp(const char *Argv0); Modified: vendor/lld/dist/COFF/DriverUtils.cpp ============================================================================== --- vendor/lld/dist/COFF/DriverUtils.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/DriverUtils.cpp Mon May 22 19:44:12 2017 (r318671) @@ -43,7 +43,7 @@ namespace { class Executor { public: - explicit Executor(StringRef S) : Saver(Alloc), Prog(Saver.save(S)) {} + explicit Executor(StringRef S) : Prog(Saver.save(S)) {} void add(StringRef S) { Args.push_back(Saver.save(S)); } void add(std::string &S) { Args.push_back(Saver.save(S)); } void add(Twine S) { Args.push_back(Saver.save(S)); } @@ -67,8 +67,6 @@ public: } private: - BumpPtrAllocator Alloc; - StringSaver Saver; StringRef Prog; std::vector Args; }; Modified: vendor/lld/dist/COFF/InputFiles.cpp ============================================================================== --- vendor/lld/dist/COFF/InputFiles.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/InputFiles.cpp Mon May 22 19:44:12 2017 (r318671) @@ -137,13 +137,13 @@ void ObjectFile::initializeChunks() { // CodeView sections are stored to a different vector because they are // not linked in the regular manner. if (Name == ".debug" || Name.startswith(".debug$")) { - DebugChunks.push_back(new (Alloc) SectionChunk(this, Sec)); + DebugChunks.push_back(make(this, Sec)); continue; } if (Sec->Characteristics & llvm::COFF::IMAGE_SCN_LNK_REMOVE) continue; - auto *C = new (Alloc) SectionChunk(this, Sec); + auto *C = make(this, Sec); Chunks.push_back(C); SparseChunks[I] = C; } @@ -200,7 +200,7 @@ SymbolBody *ObjectFile::createDefined(CO bool IsFirst) { StringRef Name; if (Sym.isCommon()) { - auto *C = new (Alloc) CommonChunk(Sym); + auto *C = make(Sym); Chunks.push_back(C); COFFObj->getSymbolName(Sym, Name); Symbol *S = @@ -221,7 +221,7 @@ SymbolBody *ObjectFile::createDefined(CO if (Sym.isExternal()) return Symtab->addAbsolute(Name, Sym)->body(); else - return new (Alloc) DefinedAbsolute(Name, Sym); + return make(Name, Sym); } int32_t SectionNumber = Sym.getSectionNumber(); if (SectionNumber == llvm::COFF::IMAGE_SYM_DEBUG) @@ -258,8 +258,8 @@ SymbolBody *ObjectFile::createDefined(CO Symtab->addRegular(this, Name, SC->isCOMDAT(), Sym.getGeneric(), SC); B = cast(S->body()); } else - B = new (Alloc) DefinedRegular(this, /*Name*/ "", SC->isCOMDAT(), - /*IsExternal*/ false, Sym.getGeneric(), SC); + B = make(this, /*Name*/ "", SC->isCOMDAT(), + /*IsExternal*/ false, Sym.getGeneric(), SC); if (SC->isCOMDAT() && Sym.getValue() == 0 && !AuxP) SC->setSymbol(B); @@ -301,8 +301,8 @@ void ImportFile::parse() { fatal("broken import library"); // Read names and create an __imp_ symbol. - StringRef Name = StringAlloc.save(StringRef(Buf + sizeof(*Hdr))); - StringRef ImpName = StringAlloc.save("__imp_" + Name); + StringRef Name = Saver.save(StringRef(Buf + sizeof(*Hdr))); + StringRef ImpName = Saver.save("__imp_" + Name); const char *NameStart = Buf + sizeof(coff_import_header) + Name.size() + 1; DLLName = StringRef(NameStart); StringRef ExtName; Modified: vendor/lld/dist/COFF/InputFiles.h ============================================================================== --- vendor/lld/dist/COFF/InputFiles.h Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/InputFiles.h Mon May 22 19:44:12 2017 (r318671) @@ -130,7 +130,6 @@ private: SymbolBody *createUndefined(COFFSymbolRef Sym); std::unique_ptr COFFObj; - llvm::BumpPtrAllocator Alloc; const coff_section *SXData = nullptr; // List of all chunks defined by this file. This includes both section @@ -162,8 +161,7 @@ private: // for details about the format. class ImportFile : public InputFile { public: - explicit ImportFile(MemoryBufferRef M) - : InputFile(ImportKind, M), StringAlloc(StringAllocAux) {} + explicit ImportFile(MemoryBufferRef M) : InputFile(ImportKind, M) {} static bool classof(const InputFile *F) { return F->kind() == ImportKind; } DefinedImportData *ImpSym = nullptr; @@ -174,9 +172,6 @@ public: private: void parse() override; - llvm::BumpPtrAllocator StringAllocAux; - llvm::StringSaver StringAlloc; - public: StringRef ExternalName; const coff_import_header *Hdr; Modified: vendor/lld/dist/COFF/Options.td ============================================================================== --- vendor/lld/dist/COFF/Options.td Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/Options.td Mon May 22 19:44:12 2017 (r318671) @@ -102,7 +102,6 @@ def nosymtab : F<"nosymtab">; def msvclto : F<"msvclto">; // Flags for debugging -def debugpdb : F<"debugpdb">; def dumppdb : Joined<["/", "-"], "dumppdb">; def lldmap : F<"lldmap">; def lldmap_file : Joined<["/", "-"], "lldmap:">; Modified: vendor/lld/dist/COFF/PDB.cpp ============================================================================== --- vendor/lld/dist/COFF/PDB.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/PDB.cpp Mon May 22 19:44:12 2017 (r318671) @@ -14,7 +14,8 @@ #include "SymbolTable.h" #include "Symbols.h" #include "llvm/DebugInfo/CodeView/CVDebugRecord.h" -#include "llvm/DebugInfo/CodeView/CVTypeDumper.h" +#include "llvm/DebugInfo/CodeView/CVTypeVisitor.h" +#include "llvm/DebugInfo/CodeView/LazyRandomTypeCollection.h" #include "llvm/DebugInfo/CodeView/SymbolDumper.h" #include "llvm/DebugInfo/CodeView/TypeDatabase.h" #include "llvm/DebugInfo/CodeView/TypeDumpVisitor.h" @@ -107,6 +108,7 @@ static void mergeDebugT(SymbolTable *Sym BinaryByteStream Stream(Data, support::little); codeview::CVTypeArray Types; BinaryStreamReader Reader(Stream); + SmallVector SourceToDest; // Follow type servers. If the same type server is encountered more than // once for this instance of `PDBTypeServerHandler` (for example if many // object files reference the same TypeServer), the types from the @@ -115,8 +117,8 @@ static void mergeDebugT(SymbolTable *Sym Handler.addSearchPath(llvm::sys::path::parent_path(File->getName())); if (auto EC = Reader.readArray(Types, Reader.getLength())) fatal(EC, "Reader::readArray failed"); - if (auto Err = - codeview::mergeTypeStreams(IDTable, TypeTable, &Handler, Types)) + if (auto Err = codeview::mergeTypeStreams(IDTable, TypeTable, SourceToDest, + &Handler, Types)) fatal(Err, "codeview::mergeTypeStreams failed"); } @@ -133,12 +135,11 @@ static void dumpDebugT(ScopedPrinter &W, if (Data.empty()) return; - TypeDatabase TDB(0); - TypeDumpVisitor TDV(TDB, &W, false); + LazyRandomTypeCollection Types(Data, 100); + TypeDumpVisitor TDV(Types, &W, false); // Use a default implementation that does not follow type servers and instead // just dumps the contents of the TypeServer2 record. - CVTypeDumper TypeDumper(TDB); - if (auto EC = TypeDumper.dump(Data, TDV)) + if (auto EC = codeview::visitTypeStream(Types, TDV)) fatal(EC, "CVTypeDumper::dump failed"); } Modified: vendor/lld/dist/COFF/SymbolTable.h ============================================================================== --- vendor/lld/dist/COFF/SymbolTable.h Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/SymbolTable.h Mon May 22 19:44:12 2017 (r318671) @@ -15,7 +15,6 @@ #include "llvm/ADT/CachedHashString.h" #include "llvm/ADT/DenseMap.h" #include "llvm/ADT/DenseMapInfo.h" -#include "llvm/Support/Allocator.h" #include "llvm/Support/raw_ostream.h" namespace llvm { Modified: vendor/lld/dist/COFF/Writer.cpp ============================================================================== --- vendor/lld/dist/COFF/Writer.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/COFF/Writer.cpp Mon May 22 19:44:12 2017 (r318671) @@ -48,8 +48,7 @@ namespace { class DebugDirectoryChunk : public Chunk { public: - DebugDirectoryChunk(const std::vector> &R) - : Records(R) {} + DebugDirectoryChunk(const std::vector &R) : Records(R) {} size_t getSize() const override { return Records.size() * sizeof(debug_directory); @@ -58,7 +57,7 @@ public: void writeTo(uint8_t *B) const override { auto *D = reinterpret_cast(B + OutputSectionOff); - for (const std::unique_ptr &Record : Records) { + for (const Chunk *Record : Records) { D->Characteristics = 0; D->TimeDateStamp = 0; D->MajorVersion = 0; @@ -74,7 +73,7 @@ public: } private: - const std::vector> &Records; + const std::vector &Records; }; class CVDebugRecordChunk : public Chunk { @@ -142,10 +141,10 @@ private: IdataContents Idata; DelayLoadContents DelayIdata; EdataContents Edata; - std::unique_ptr SEHTable; + SEHTableChunk *SEHTable = nullptr; - std::unique_ptr DebugDirectory; - std::vector> DebugRecords; + Chunk *DebugDirectory = nullptr; + std::vector DebugRecords; CVDebugRecordChunk *BuildId = nullptr; ArrayRef SectionTable; @@ -153,8 +152,6 @@ private: uint32_t PointerToSymbolTable = 0; uint64_t SizeOfImage; uint64_t SizeOfHeaders; - - std::vector> Chunks; }; } // anonymous namespace @@ -258,7 +255,7 @@ void Writer::run() { sortExceptionTable(); writeBuildId(); - if (!Config->PDBPath.empty()) { + if (!Config->PDBPath.empty() && Config->Debug) { const llvm::codeview::DebugInfo *DI = nullptr; if (Config->DebugTypes & static_cast(coff::DebugType::CV)) DI = BuildId->DI; @@ -324,19 +321,19 @@ void Writer::createMiscChunks() { // Create Debug Information Chunks if (Config->Debug) { - DebugDirectory = llvm::make_unique(DebugRecords); + DebugDirectory = make(DebugRecords); // TODO(compnerd) create a coffgrp entry if DebugType::CV is not enabled if (Config->DebugTypes & static_cast(coff::DebugType::CV)) { - auto Chunk = llvm::make_unique(); + auto *Chunk = make(); - BuildId = Chunk.get(); - DebugRecords.push_back(std::move(Chunk)); + BuildId = Chunk; + DebugRecords.push_back(Chunk); } - RData->addChunk(DebugDirectory.get()); - for (const std::unique_ptr &C : DebugRecords) - RData->addChunk(C.get()); + RData->addChunk(DebugDirectory); + for (Chunk *C : DebugRecords) + RData->addChunk(C); } // Create SEH table. x86-only. @@ -352,8 +349,8 @@ void Writer::createMiscChunks() { Handlers.insert(cast(B)); } - SEHTable.reset(new SEHTableChunk(Handlers)); - RData->addChunk(SEHTable.get()); + SEHTable = make(Handlers); + RData->addChunk(SEHTable); } // Create .idata section for the DLL-imported symbol table. @@ -398,8 +395,8 @@ void Writer::createImportTables() { for (Chunk *C : DelayIdata.getDataChunks()) Sec->addChunk(C); Sec = createSection(".text"); - for (std::unique_ptr &C : DelayIdata.getCodeChunks()) - Sec->addChunk(C.get()); + for (Chunk *C : DelayIdata.getCodeChunks()) + Sec->addChunk(C); } } @@ -407,8 +404,8 @@ void Writer::createExportTable() { if (Config->Exports.empty()) return; OutputSection *Sec = createSection(".edata"); - for (std::unique_ptr &C : Edata.Chunks) - Sec->addChunk(C.get()); + for (Chunk *C : Edata.Chunks) + Sec->addChunk(C); } // The Windows loader doesn't seem to like empty sections, @@ -602,14 +599,19 @@ template void Writ PE->SizeOfStackCommit = Config->StackCommit; PE->SizeOfHeapReserve = Config->HeapReserve; PE->SizeOfHeapCommit = Config->HeapCommit; + + // Import Descriptor Tables and Import Address Tables are merged + // in our output. That's not compatible with the Binding feature + // that is sort of prelinking. Setting this flag to make it clear + // that our outputs are not for the Binding. + PE->DLLCharacteristics = IMAGE_DLL_CHARACTERISTICS_NO_BIND; + if (Config->AppContainer) PE->DLLCharacteristics |= IMAGE_DLL_CHARACTERISTICS_APPCONTAINER; if (Config->DynamicBase) PE->DLLCharacteristics |= IMAGE_DLL_CHARACTERISTICS_DYNAMIC_BASE; if (Config->HighEntropyVA) PE->DLLCharacteristics |= IMAGE_DLL_CHARACTERISTICS_HIGH_ENTROPY_VA; - if (!Config->AllowBind) - PE->DLLCharacteristics |= IMAGE_DLL_CHARACTERISTICS_NO_BIND; if (Config->NxCompat) PE->DLLCharacteristics |= IMAGE_DLL_CHARACTERISTICS_NX_COMPAT; if (!Config->AllowIsolation) Modified: vendor/lld/dist/ELF/InputSection.cpp ============================================================================== --- vendor/lld/dist/ELF/InputSection.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/ELF/InputSection.cpp Mon May 22 19:44:12 2017 (r318671) @@ -390,14 +390,28 @@ static uint64_t getAArch64UndefinedRelat } } -template -static typename ELFT::uint -getRelocTargetVA(uint32_t Type, int64_t A, typename ELFT::uint P, - const SymbolBody &Body, RelExpr Expr) { +// ARM SBREL relocations are of the form S + A - B where B is the static base +// The ARM ABI defines base to be "addressing origin of the output segment +// defining the symbol S". We defined the "addressing origin"/static base to be +// the base of the PT_LOAD segment containing the Body. +// The procedure call standard only defines a Read Write Position Independent +// RWPI variant so in practice we should expect the static base to be the base +// of the RW segment. +static uint64_t getARMStaticBase(const SymbolBody &Body) { + OutputSection *OS = Body.getOutputSection(); + if (!OS || !OS->FirstInPtLoad) + fatal("SBREL relocation to " + Body.getName() + " without static base\n"); + return OS->FirstInPtLoad->Addr; +} + +static uint64_t getRelocTargetVA(uint32_t Type, int64_t A, uint64_t P, + const SymbolBody &Body, RelExpr Expr) { switch (Expr) { case R_ABS: case R_RELAX_GOT_PC_NOPIC: return Body.getVA(A); + case R_ARM_SBREL: + return Body.getVA(A) - getARMStaticBase(Body); case R_GOT: case R_RELAX_TLS_GD_TO_IE_ABS: return Body.getGotVA() + A; @@ -518,7 +532,7 @@ getRelocTargetVA(uint32_t Type, int64_t case R_NEG_TLS: return Out::TlsPhdr->p_memsz - Body.getVA(A); case R_SIZE: - return Body.getSize() + A; + return A; // Body.getSize was already folded into the addend. case R_TLSDESC: return InX::Got->getGlobalDynAddr(Body) + A; case R_TLSDESC_PAGE: @@ -566,7 +580,7 @@ void InputSection::relocateNonAlloc(uint uint64_t SymVA = 0; if (!Sym.isTls() || Out::TlsPhdr) SymVA = SignExtend64( - getRelocTargetVA(Type, Addend, AddrLoc, Sym, R_ABS)); + getRelocTargetVA(Type, Addend, AddrLoc, Sym, R_ABS)); Target->relocateOne(BufLoc, Type, SymVA); } } @@ -577,19 +591,28 @@ template elf::ObjectFile void InputSectionBase::relocate(uint8_t *Buf, uint8_t *BufEnd) { + if (Flags & SHF_ALLOC) + relocateAlloc(Buf, BufEnd); + else + relocateNonAlloc(Buf, BufEnd); +} + +template +void InputSectionBase::relocateNonAlloc(uint8_t *Buf, uint8_t *BufEnd) { // scanReloc function in Writer.cpp constructs Relocations // vector only for SHF_ALLOC'ed sections. For other sections, // we handle relocations directly here. - auto *IS = dyn_cast(this); - if (IS && !(IS->Flags & SHF_ALLOC)) { - if (IS->AreRelocsRela) - IS->relocateNonAlloc(Buf, IS->template relas()); - else - IS->relocateNonAlloc(Buf, IS->template rels()); - return; - } + auto *IS = cast(this); + assert(!(IS->Flags & SHF_ALLOC)); + if (IS->AreRelocsRela) + IS->relocateNonAlloc(Buf, IS->template relas()); + else + IS->relocateNonAlloc(Buf, IS->template rels()); +} - const unsigned Bits = sizeof(typename ELFT::uint) * 8; +void InputSectionBase::relocateAlloc(uint8_t *Buf, uint8_t *BufEnd) { + assert(Flags & SHF_ALLOC); + const unsigned Bits = Config->Wordsize * 8; for (const Relocation &Rel : Relocations) { uint64_t Offset = getOffset(Rel.Offset); uint8_t *BufLoc = Buf + Offset; @@ -597,8 +620,8 @@ void InputSectionBase::relocate(uint8_t uint64_t AddrLoc = getOutputSection()->Addr + Offset; RelExpr Expr = Rel.Expr; - uint64_t TargetVA = SignExtend64( - getRelocTargetVA(Type, Rel.Addend, AddrLoc, *Rel.Sym, Expr)); + uint64_t TargetVA = SignExtend64( + getRelocTargetVA(Type, Rel.Addend, AddrLoc, *Rel.Sym, Expr), Bits); switch (Expr) { case R_RELAX_GOT_PC: Modified: vendor/lld/dist/ELF/InputSection.h ============================================================================== --- vendor/lld/dist/ELF/InputSection.h Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/ELF/InputSection.h Mon May 22 19:44:12 2017 (r318671) @@ -167,6 +167,8 @@ public: template std::string getObjMsg(uint64_t Offset); template void relocate(uint8_t *Buf, uint8_t *BufEnd); + void relocateAlloc(uint8_t *Buf, uint8_t *BufEnd); + template void relocateNonAlloc(uint8_t *Buf, uint8_t *BufEnd); std::vector Relocations; Modified: vendor/lld/dist/ELF/LinkerScript.cpp ============================================================================== --- vendor/lld/dist/ELF/LinkerScript.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/ELF/LinkerScript.cpp Mon May 22 19:44:12 2017 (r318671) @@ -440,9 +440,6 @@ void LinkerScript::fabricateDefaultComma // For each OutputSection that needs a VA fabricate an OutputSectionCommand // with an InputSectionDescription describing the InputSections for (OutputSection *Sec : *OutputSections) { - if (!(Sec->Flags & SHF_ALLOC)) - continue; - auto *OSCmd = make(Sec->Name); OSCmd->Sec = Sec; SecToCommand[Sec] = OSCmd; Modified: vendor/lld/dist/ELF/MapFile.cpp ============================================================================== --- vendor/lld/dist/ELF/MapFile.cpp Mon May 22 19:44:10 2017 (r318670) +++ vendor/lld/dist/ELF/MapFile.cpp Mon May 22 19:44:12 2017 (r318671) @@ -21,6 +21,8 @@ #include "MapFile.h" #include "InputFiles.h" +#include "LinkerScript.h" +#include "OutputSections.h" #include "Strings.h" #include "SymbolTable.h" #include "Threads.h" @@ -98,7 +100,7 @@ getSymbolStrings(ArrayRef -void elf::writeMapFile(ArrayRef OutputSections) { +void elf::writeMapFile(llvm::ArrayRef Script) { if (Config->MapFile.empty()) *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Mon May 22 19:44:27 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D238FD78CA2; Mon, 22 May 2017 19:44:27 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 878CD1D8E; Mon, 22 May 2017 19:44:27 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4MJiQJm077100; Mon, 22 May 2017 19:44:26 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4MJiQ14077099; Mon, 22 May 2017 19:44:26 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201705221944.v4MJiQ14077099@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 22 May 2017 19:44:26 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318674 - vendor/lldb/lldb-trunk-r303571 X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 May 2017 19:44:28 -0000 Author: dim Date: Mon May 22 19:44:26 2017 New Revision: 318674 URL: https://svnweb.freebsd.org/changeset/base/318674 Log: Tag lldb trunk r303571. Added: vendor/lldb/lldb-trunk-r303571/ - copied from r318673, vendor/lldb/dist/ From owner-svn-src-vendor@freebsd.org Thu May 25 19:38:44 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6988FD824AD; Thu, 25 May 2017 19:38:44 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 21006138B; Thu, 25 May 2017 19:38:44 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4PJchQo083565; Thu, 25 May 2017 19:38:43 GMT (envelope-from jkim@FreeBSD.org) Received: (from jkim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4PJccxA083522; Thu, 25 May 2017 19:38:38 GMT (envelope-from jkim@FreeBSD.org) Message-Id: <201705251938.v4PJccxA083522@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jkim set sender to jkim@FreeBSD.org using -f From: Jung-uk Kim Date: Thu, 25 May 2017 19:38:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318897 - in vendor-crypto/openssl/dist: . apps crypto crypto/aes crypto/aes/asm crypto/asn1 crypto/bio crypto/bn crypto/bn/asm crypto/comp crypto/conf crypto/des crypto/dh crypto/ec cr... X-SVN-Group: vendor-crypto MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 19:38:44 -0000 Author: jkim Date: Thu May 25 19:38:38 2017 New Revision: 318897 URL: https://svnweb.freebsd.org/changeset/base/318897 Log: Import OpenSSL 1.0.2l. Added: vendor-crypto/openssl/dist/doc/man3/ vendor-crypto/openssl/dist/doc/man3/SSL_CTX_set_tlsext_servername_callback.pod Modified: vendor-crypto/openssl/dist/CHANGES vendor-crypto/openssl/dist/Configure vendor-crypto/openssl/dist/FREEBSD-Xlist vendor-crypto/openssl/dist/FREEBSD-upgrade vendor-crypto/openssl/dist/LICENSE vendor-crypto/openssl/dist/Makefile vendor-crypto/openssl/dist/Makefile.org vendor-crypto/openssl/dist/NEWS vendor-crypto/openssl/dist/README vendor-crypto/openssl/dist/apps/ca.c vendor-crypto/openssl/dist/apps/dhparam.c vendor-crypto/openssl/dist/apps/enc.c vendor-crypto/openssl/dist/apps/engine.c vendor-crypto/openssl/dist/apps/pkeyutl.c vendor-crypto/openssl/dist/apps/prime.c vendor-crypto/openssl/dist/apps/progs.h vendor-crypto/openssl/dist/apps/progs.pl vendor-crypto/openssl/dist/apps/req.c vendor-crypto/openssl/dist/apps/s_client.c vendor-crypto/openssl/dist/apps/s_server.c vendor-crypto/openssl/dist/apps/srp.c vendor-crypto/openssl/dist/appveyor.yml vendor-crypto/openssl/dist/config vendor-crypto/openssl/dist/crypto/aes/Makefile vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha1-x86_64.pl vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha256-x86_64.pl vendor-crypto/openssl/dist/crypto/aes/asm/bsaes-armv7.pl vendor-crypto/openssl/dist/crypto/asn1/a_bitstr.c vendor-crypto/openssl/dist/crypto/asn1/a_digest.c vendor-crypto/openssl/dist/crypto/asn1/a_gentm.c vendor-crypto/openssl/dist/crypto/asn1/a_strnid.c vendor-crypto/openssl/dist/crypto/asn1/a_time.c vendor-crypto/openssl/dist/crypto/asn1/a_utctm.c vendor-crypto/openssl/dist/crypto/asn1/f_enum.c vendor-crypto/openssl/dist/crypto/asn1/f_int.c vendor-crypto/openssl/dist/crypto/asn1/tasn_dec.c vendor-crypto/openssl/dist/crypto/asn1/tasn_new.c vendor-crypto/openssl/dist/crypto/asn1/x_long.c vendor-crypto/openssl/dist/crypto/asn1/x_name.c vendor-crypto/openssl/dist/crypto/bio/b_print.c vendor-crypto/openssl/dist/crypto/bio/bio_cb.c vendor-crypto/openssl/dist/crypto/bio/bss_file.c vendor-crypto/openssl/dist/crypto/bn/Makefile vendor-crypto/openssl/dist/crypto/bn/asm/sparcv9-mont.pl vendor-crypto/openssl/dist/crypto/bn/bn_prime.c vendor-crypto/openssl/dist/crypto/bn/bn_prime.h vendor-crypto/openssl/dist/crypto/bn/bn_prime.pl vendor-crypto/openssl/dist/crypto/bn/bn_print.c vendor-crypto/openssl/dist/crypto/comp/c_rle.c vendor-crypto/openssl/dist/crypto/conf/conf.h vendor-crypto/openssl/dist/crypto/conf/conf_def.c vendor-crypto/openssl/dist/crypto/conf/conf_err.c vendor-crypto/openssl/dist/crypto/des/Makefile vendor-crypto/openssl/dist/crypto/des/set_key.c vendor-crypto/openssl/dist/crypto/dh/dh.h vendor-crypto/openssl/dist/crypto/ec/ec_ameth.c vendor-crypto/openssl/dist/crypto/ec/ec_asn1.c vendor-crypto/openssl/dist/crypto/ec/ec_mult.c vendor-crypto/openssl/dist/crypto/ec/eck_prn.c vendor-crypto/openssl/dist/crypto/engine/eng_cryptodev.c vendor-crypto/openssl/dist/crypto/err/err.c vendor-crypto/openssl/dist/crypto/err/err.h vendor-crypto/openssl/dist/crypto/evp/e_aes.c vendor-crypto/openssl/dist/crypto/evp/e_aes_cbc_hmac_sha1.c vendor-crypto/openssl/dist/crypto/evp/e_aes_cbc_hmac_sha256.c vendor-crypto/openssl/dist/crypto/evp/e_des3.c vendor-crypto/openssl/dist/crypto/evp/evp_enc.c vendor-crypto/openssl/dist/crypto/evp/pmeth_lib.c vendor-crypto/openssl/dist/crypto/ex_data.c vendor-crypto/openssl/dist/crypto/hmac/hm_pmeth.c vendor-crypto/openssl/dist/crypto/md5/Makefile vendor-crypto/openssl/dist/crypto/mem.c vendor-crypto/openssl/dist/crypto/modes/Makefile vendor-crypto/openssl/dist/crypto/o_dir.c vendor-crypto/openssl/dist/crypto/o_time.c vendor-crypto/openssl/dist/crypto/opensslv.h vendor-crypto/openssl/dist/crypto/perlasm/x86_64-xlate.pl vendor-crypto/openssl/dist/crypto/pkcs12/p12_mutl.c vendor-crypto/openssl/dist/crypto/ppccap.c vendor-crypto/openssl/dist/crypto/rand/md_rand.c vendor-crypto/openssl/dist/crypto/rc4/Makefile vendor-crypto/openssl/dist/crypto/rsa/rsa_pmeth.c vendor-crypto/openssl/dist/crypto/rsa/rsa_pss.c vendor-crypto/openssl/dist/crypto/sha/Makefile vendor-crypto/openssl/dist/crypto/srp/srp_vfy.c vendor-crypto/openssl/dist/crypto/txt_db/txt_db.c vendor-crypto/openssl/dist/crypto/ui/ui_lib.c vendor-crypto/openssl/dist/crypto/x509/x509_lu.c vendor-crypto/openssl/dist/crypto/x509v3/v3_alt.c vendor-crypto/openssl/dist/crypto/x509v3/v3_cpols.c vendor-crypto/openssl/dist/crypto/x509v3/v3_info.c vendor-crypto/openssl/dist/crypto/x509v3/v3_purp.c vendor-crypto/openssl/dist/crypto/x86_64cpuid.pl vendor-crypto/openssl/dist/crypto/x86cpuid.pl vendor-crypto/openssl/dist/doc/apps/ciphers.pod vendor-crypto/openssl/dist/doc/apps/config.pod vendor-crypto/openssl/dist/doc/apps/genrsa.pod vendor-crypto/openssl/dist/doc/apps/req.pod vendor-crypto/openssl/dist/doc/apps/s_client.pod vendor-crypto/openssl/dist/doc/apps/s_server.pod vendor-crypto/openssl/dist/doc/crypto/EVP_EncryptInit.pod vendor-crypto/openssl/dist/doc/crypto/RSA_private_encrypt.pod vendor-crypto/openssl/dist/doc/crypto/RSA_public_encrypt.pod vendor-crypto/openssl/dist/doc/crypto/X509_STORE_CTX_new.pod vendor-crypto/openssl/dist/doc/crypto/des.pod vendor-crypto/openssl/dist/ssl/d1_both.c vendor-crypto/openssl/dist/ssl/d1_clnt.c vendor-crypto/openssl/dist/ssl/d1_pkt.c vendor-crypto/openssl/dist/ssl/d1_srvr.c vendor-crypto/openssl/dist/ssl/s23_clnt.c vendor-crypto/openssl/dist/ssl/s23_srvr.c vendor-crypto/openssl/dist/ssl/s3_clnt.c vendor-crypto/openssl/dist/ssl/s3_enc.c vendor-crypto/openssl/dist/ssl/s3_lib.c vendor-crypto/openssl/dist/ssl/s3_pkt.c vendor-crypto/openssl/dist/ssl/s3_srvr.c vendor-crypto/openssl/dist/ssl/ssl_cert.c vendor-crypto/openssl/dist/ssl/ssl_ciph.c vendor-crypto/openssl/dist/ssl/ssl_lib.c vendor-crypto/openssl/dist/ssl/ssl_locl.h vendor-crypto/openssl/dist/ssl/ssl_rsa.c vendor-crypto/openssl/dist/ssl/ssl_sess.c vendor-crypto/openssl/dist/ssl/ssltest.c vendor-crypto/openssl/dist/ssl/t1_ext.c vendor-crypto/openssl/dist/ssl/t1_lib.c vendor-crypto/openssl/dist/util/domd vendor-crypto/openssl/dist/util/mk1mf.pl Modified: vendor-crypto/openssl/dist/CHANGES ============================================================================== --- vendor-crypto/openssl/dist/CHANGES Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/CHANGES Thu May 25 19:38:38 2017 (r318897) @@ -2,6 +2,12 @@ OpenSSL CHANGES _______________ + Changes between 1.0.2k and 1.0.2l [25 May 2017] + + *) Have 'config' recognise 64-bit mingw and choose 'mingw64' as the target + platform rather than 'mingw'. + [Richard Levitte] + Changes between 1.0.2j and 1.0.2k [26 Jan 2017] *) Truncated packet could crash via OOB read Modified: vendor-crypto/openssl/dist/Configure ============================================================================== --- vendor-crypto/openssl/dist/Configure Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/Configure Thu May 25 19:38:38 2017 (r318897) @@ -109,7 +109,7 @@ my $usage="Usage: Configure [no- # Minimum warning options... any contributions to OpenSSL should at least get # past these. -my $gcc_devteam_warn = "-Wall -pedantic -DPEDANTIC -Wno-long-long -Wsign-compare -Wmissing-prototypes -Wshadow -Wformat -Werror -DCRYPTO_MDEBUG_ALL -DCRYPTO_MDEBUG_ABORT -DREF_CHECK -DOPENSSL_NO_DEPRECATED"; +my $gcc_devteam_warn = "-Wall -pedantic -DPEDANTIC -Wno-long-long -Wsign-compare -Wmissing-prototypes -Wshadow -Wformat -Wundef -Werror -DCRYPTO_MDEBUG_ALL -DCRYPTO_MDEBUG_ABORT -DREF_CHECK -DOPENSSL_NO_DEPRECATED"; # TODO(openssl-team): fix problems and investigate if (at least) the following # warnings can also be enabled: @@ -2041,12 +2041,13 @@ EOF close(OUT); } else { my $make_command = "$make PERL=\'$perl\'"; - my $make_targets = ""; - $make_targets .= " links" if $symlink; - $make_targets .= " depend" if $depflags ne $default_depflags && $make_depend; - $make_targets .= " gentests" if $symlink; - (system $make_command.$make_targets) == 0 or exit $? - if $make_targets ne ""; + my @make_targets = (); + push @make_targets, "links" if $symlink; + push @make_targets, "depend" if $depflags ne $default_depflags && $make_depend; + push @make_targets, "gentests" if $symlink; + foreach my $make_target (@make_targets) { + (system "$make_command $make_target") == 0 or exit $?; + } if ( $perl =~ m@^/@) { &dofile("tools/c_rehash",$perl,'^#!/', '#!%s','^my \$dir;$', 'my $dir = "' . $openssldir . '";', '^my \$prefix;$', 'my $prefix = "' . $prefix . '";'); &dofile("apps/CA.pl",$perl,'^#!/', '#!%s'); @@ -2056,8 +2057,8 @@ EOF &dofile("apps/CA.pl",'/usr/local/bin/perl','^#!/', '#!%s'); } if ($depflags ne $default_depflags && !$make_depend) { - $warn_make_depend++; - } + $warn_make_depend++; + } } # create the ms/version32.rc file if needed Modified: vendor-crypto/openssl/dist/FREEBSD-Xlist ============================================================================== --- vendor-crypto/openssl/dist/FREEBSD-Xlist Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/FREEBSD-Xlist Thu May 25 19:38:38 2017 (r318897) @@ -57,9 +57,11 @@ openssl-*/crypto/rand/rand_vms.c openssl-*/crypto/rand/rand_win.c openssl-*/crypto/threads/solaris.sh openssl-*/demos +openssl-*/doc-nits openssl-*/engines/*.ec openssl-*/engines/*.opt openssl-*/engines/ccgost/e_gost_err.proto +openssl-*/fuzz openssl-*/include openssl-*/ms openssl-*/op Modified: vendor-crypto/openssl/dist/FREEBSD-upgrade ============================================================================== --- vendor-crypto/openssl/dist/FREEBSD-upgrade Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/FREEBSD-upgrade Thu May 25 19:38:38 2017 (r318897) @@ -11,8 +11,8 @@ First, read http://wiki.freebsd.org/Subv # Xlist setenv XLIST /FreeBSD/work/openssl/svn-FREEBSD-files/FREEBSD-Xlist setenv FSVN "svn+ssh://repo.freebsd.org/base" -setenv OSSLVER 1.0.2j -# OSSLTAG format: v1_0_2j +setenv OSSLVER 1.0.2l +# OSSLTAG format: v1_0_2l ###setenv OSSLTAG v`echo ${OSSLVER} | tr . _` Modified: vendor-crypto/openssl/dist/LICENSE ============================================================================== --- vendor-crypto/openssl/dist/LICENSE Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/LICENSE Thu May 25 19:38:38 2017 (r318897) @@ -2,7 +2,7 @@ LICENSE ISSUES ============== - The OpenSSL toolkit stays under a dual license, i.e. both the conditions of + The OpenSSL toolkit stays under a double license, i.e. both the conditions of the OpenSSL License and the original SSLeay license apply to the toolkit. See below for the actual license texts. Actually both licenses are BSD-style Open Source licenses. In case of any license issues related to OpenSSL @@ -12,7 +12,7 @@ --------------- /* ==================================================================== - * Copyright (c) 1998-2016 The OpenSSL Project. All rights reserved. + * Copyright (c) 1998-2017 The OpenSSL Project. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions Modified: vendor-crypto/openssl/dist/Makefile ============================================================================== --- vendor-crypto/openssl/dist/Makefile Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/Makefile Thu May 25 19:38:38 2017 (r318897) @@ -4,7 +4,7 @@ ## Makefile for OpenSSL ## -VERSION=1.0.2k +VERSION=1.0.2l MAJOR=1 MINOR=0.2 SHLIB_VERSION_NUMBER=1.0.0 @@ -426,6 +426,14 @@ clean: libclean rm -fr $$i/*; \ done +distclean: clean + -$(RM) `find . -name .git -prune -o -type l -print` + $(RM) apps/CA.pl + $(RM) test/evptests.txt test/newkey.pem test/testkey.pem test/testreq.pem + $(RM) tools/c_rehash + $(RM) crypto/opensslconf.h + $(RM) Makefile Makefile.bak + makefile.one: files $(PERL) util/mk1mf.pl >makefile.one; \ sh util/do_ms.sh Modified: vendor-crypto/openssl/dist/Makefile.org ============================================================================== --- vendor-crypto/openssl/dist/Makefile.org Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/Makefile.org Thu May 25 19:38:38 2017 (r318897) @@ -424,6 +424,14 @@ clean: libclean rm -fr $$i/*; \ done +distclean: clean + -$(RM) `find . -name .git -prune -o -type l -print` + $(RM) apps/CA.pl + $(RM) test/evptests.txt test/newkey.pem test/testkey.pem test/testreq.pem + $(RM) tools/c_rehash + $(RM) crypto/opensslconf.h + $(RM) Makefile Makefile.bak + makefile.one: files $(PERL) util/mk1mf.pl >makefile.one; \ sh util/do_ms.sh Modified: vendor-crypto/openssl/dist/NEWS ============================================================================== --- vendor-crypto/openssl/dist/NEWS Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/NEWS Thu May 25 19:38:38 2017 (r318897) @@ -5,6 +5,10 @@ This file gives a brief overview of the major changes between each OpenSSL release. For more details please read the CHANGES file. + Major changes between OpenSSL 1.0.2k and OpenSSL 1.0.2l [25 May 2017] + + o config now recognises 64-bit mingw and chooses mingw64 instead of mingw + Major changes between OpenSSL 1.0.2j and OpenSSL 1.0.2k [26 Jan 2017] o Truncated packet could crash via OOB read (CVE-2017-3731) Modified: vendor-crypto/openssl/dist/README ============================================================================== --- vendor-crypto/openssl/dist/README Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/README Thu May 25 19:38:38 2017 (r318897) @@ -1,5 +1,5 @@ - OpenSSL 1.0.2k 26 Jan 2017 + OpenSSL 1.0.2l 25 May 2017 Copyright (c) 1998-2015 The OpenSSL Project Copyright (c) 1995-1998 Eric A. Young, Tim J. Hudson Modified: vendor-crypto/openssl/dist/apps/ca.c ============================================================================== --- vendor-crypto/openssl/dist/apps/ca.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/ca.c Thu May 25 19:38:38 2017 (r318897) @@ -2126,10 +2126,8 @@ static int do_body(X509 **xret, EVP_PKEY goto err; } - for (i = 0; i < DB_NUMBER; i++) { + for (i = 0; i < DB_NUMBER; i++) irow[i] = row[i]; - row[i] = NULL; - } irow[DB_NUMBER] = NULL; if (!TXT_DB_insert(db->db, irow)) { @@ -2137,11 +2135,14 @@ static int do_body(X509 **xret, EVP_PKEY BIO_printf(bio_err, "TXT_DB error number %ld\n", db->db->error); goto err; } + irow = NULL; ok = 1; err: - for (i = 0; i < DB_NUMBER; i++) - if (row[i] != NULL) + if (irow != NULL) { + for (i = 0; i < DB_NUMBER; i++) OPENSSL_free(row[i]); + OPENSSL_free(irow); + } if (CAname != NULL) X509_NAME_free(CAname); @@ -2396,18 +2397,20 @@ static int do_revoke(X509 *x509, CA_DB * goto err; } - for (i = 0; i < DB_NUMBER; i++) { + for (i = 0; i < DB_NUMBER; i++) irow[i] = row[i]; - row[i] = NULL; - } irow[DB_NUMBER] = NULL; if (!TXT_DB_insert(db->db, irow)) { BIO_printf(bio_err, "failed to update database\n"); BIO_printf(bio_err, "TXT_DB error number %ld\n", db->db->error); + OPENSSL_free(irow); goto err; } + for (i = 0; i < DB_NUMBER; i++) + row[i] = NULL; + /* Revoke Certificate */ if (type == -1) ok = 1; Modified: vendor-crypto/openssl/dist/apps/dhparam.c ============================================================================== --- vendor-crypto/openssl/dist/apps/dhparam.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/dhparam.c Thu May 25 19:38:38 2017 (r318897) @@ -381,10 +381,19 @@ int MAIN(int argc, char **argv) } else # endif { - if (informat == FORMAT_ASN1) + if (informat == FORMAT_ASN1) { + /* + * We have no PEM header to determine what type of DH params it + * is. We'll just try both. + */ dh = d2i_DHparams_bio(in, NULL); - else /* informat == FORMAT_PEM */ + /* BIO_reset() returns 0 for success for file BIOs only!!! */ + if (dh == NULL && BIO_reset(in) == 0) + dh = d2i_DHxparams_bio(in, NULL); + } else { + /* informat == FORMAT_PEM */ dh = PEM_read_bio_DHparams(in, NULL, NULL, NULL); + } if (dh == NULL) { BIO_printf(bio_err, "unable to load DH parameters\n"); @@ -484,10 +493,13 @@ int MAIN(int argc, char **argv) } if (!noout) { - if (outformat == FORMAT_ASN1) - i = i2d_DHparams_bio(out, dh); - else if (outformat == FORMAT_PEM) { - if (dh->q) + if (outformat == FORMAT_ASN1) { + if (dh->q != NULL) + i = i2d_DHxparams_bio(out, dh); + else + i = i2d_DHparams_bio(out, dh); + } else if (outformat == FORMAT_PEM) { + if (dh->q != NULL) i = PEM_write_bio_DHxparams(out, dh); else i = PEM_write_bio_DHparams(out, dh); Modified: vendor-crypto/openssl/dist/apps/enc.c ============================================================================== --- vendor-crypto/openssl/dist/apps/enc.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/enc.c Thu May 25 19:38:38 2017 (r318897) @@ -81,20 +81,32 @@ int set_hex(char *in, unsigned char *out #define BSIZE (8*1024) #define PROG enc_main -static void show_ciphers(const OBJ_NAME *name, void *bio_) +struct doall_enc_ciphers { + BIO *bio; + int n; +}; + +static void show_ciphers(const OBJ_NAME *name, void *arg) { - BIO *bio = bio_; - static int n; + struct doall_enc_ciphers *dec = (struct doall_enc_ciphers *)arg; + const EVP_CIPHER *cipher; if (!islower((unsigned char)*name->name)) return; - BIO_printf(bio, "-%-25s", name->name); - if (++n == 3) { - BIO_printf(bio, "\n"); - n = 0; + /* Filter out ciphers that we cannot use */ + cipher = EVP_get_cipherbyname(name->name); + if (cipher == NULL || + (EVP_CIPHER_flags(cipher) & EVP_CIPH_FLAG_AEAD_CIPHER) != 0 || + EVP_CIPHER_mode(cipher) == EVP_CIPH_XTS_MODE) + return; + + BIO_printf(dec->bio, "-%-25s", name->name); + if (++dec->n == 3) { + BIO_printf(dec->bio, "\n"); + dec->n = 0; } else - BIO_printf(bio, " "); + BIO_printf(dec->bio, " "); } int MAIN(int, char **); @@ -130,6 +142,7 @@ int MAIN(int argc, char **argv) ENGINE *e = NULL; const EVP_MD *dgst = NULL; int non_fips_allow = 0; + struct doall_enc_ciphers dec; apps_startup(); @@ -311,8 +324,10 @@ int MAIN(int argc, char **argv) #endif BIO_printf(bio_err, "Cipher Types\n"); + dec.n = 0; + dec.bio = bio_err; OBJ_NAME_do_all_sorted(OBJ_NAME_TYPE_CIPHER_METH, - show_ciphers, bio_err); + show_ciphers, &dec); BIO_printf(bio_err, "\n"); goto end; Modified: vendor-crypto/openssl/dist/apps/engine.c ============================================================================== --- vendor-crypto/openssl/dist/apps/engine.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/engine.c Thu May 25 19:38:38 2017 (r318897) @@ -108,13 +108,16 @@ static int append_buf(char **buf, const } if (strlen(*buf) + strlen(s) >= (unsigned int)*size) { + char *p = *buf; + *size += step; *buf = OPENSSL_realloc(*buf, *size); + if (*buf == NULL) { + OPENSSL_free(p); + return 0; + } } - if (*buf == NULL) - return 0; - if (**buf != '\0') BUF_strlcat(*buf, ", ", *size); BUF_strlcat(*buf, s, *size); Modified: vendor-crypto/openssl/dist/apps/pkeyutl.c ============================================================================== --- vendor-crypto/openssl/dist/apps/pkeyutl.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/pkeyutl.c Thu May 25 19:38:38 2017 (r318897) @@ -322,8 +322,10 @@ int MAIN(int argc, char **argv) buf_in, (size_t)buf_inlen); if (rv == 0) BIO_puts(out, "Signature Verification Failure\n"); - else if (rv == 1) + else if (rv == 1) { BIO_puts(out, "Signature Verified Successfully\n"); + ret = 0; + } if (rv >= 0) goto end; } else { Modified: vendor-crypto/openssl/dist/apps/prime.c ============================================================================== --- vendor-crypto/openssl/dist/apps/prime.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/prime.c Thu May 25 19:38:38 2017 (r318897) @@ -155,5 +155,8 @@ int MAIN(int argc, char **argv) BIO_printf(bio_err, "options are\n"); BIO_printf(bio_err, "%-14s hex\n", "-hex"); BIO_printf(bio_err, "%-14s number of checks\n", "-checks "); + BIO_printf(bio_err, "%-14s generate prime\n", "-generate"); + BIO_printf(bio_err, "%-14s number of bits\n", "-bits "); + BIO_printf(bio_err, "%-14s safe prime\n", "-safe"); return 1; } Modified: vendor-crypto/openssl/dist/apps/progs.h ============================================================================== --- vendor-crypto/openssl/dist/apps/progs.h Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/progs.h Thu May 25 19:38:38 2017 (r318897) @@ -58,7 +58,7 @@ extern int srp_main(int argc, char *argv typedef struct { int type; const char *name; - int (*func) (int argc, char *argv[]); + int (*func)(int argc, char *argv[]); } FUNCTION; DECLARE_LHASH_OF(FUNCTION); Modified: vendor-crypto/openssl/dist/apps/progs.pl ============================================================================== --- vendor-crypto/openssl/dist/apps/progs.pl Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/progs.pl Thu May 25 19:38:38 2017 (r318897) @@ -6,22 +6,22 @@ print "/* automatically generated by pro grep(s/^asn1pars$/asn1parse/,@ARGV); foreach (@ARGV) - { printf "extern int %s_main(int argc,char *argv[]);\n",$_; } + { printf "extern int %s_main(int argc, char *argv[]);\n",$_; } print <<'EOF'; -#define FUNC_TYPE_GENERAL 1 -#define FUNC_TYPE_MD 2 -#define FUNC_TYPE_CIPHER 3 -#define FUNC_TYPE_PKEY 4 -#define FUNC_TYPE_MD_ALG 5 -#define FUNC_TYPE_CIPHER_ALG 6 +#define FUNC_TYPE_GENERAL 1 +#define FUNC_TYPE_MD 2 +#define FUNC_TYPE_CIPHER 3 +#define FUNC_TYPE_PKEY 4 +#define FUNC_TYPE_MD_ALG 5 +#define FUNC_TYPE_CIPHER_ALG 6 typedef struct { - int type; - const char *name; - int (*func)(int argc,char *argv[]); - } FUNCTION; + int type; + const char *name; + int (*func)(int argc, char *argv[]); +} FUNCTION; DECLARE_LHASH_OF(FUNCTION); FUNCTION functions[] = { @@ -30,7 +30,7 @@ EOF foreach (@ARGV) { push(@files,$_); - $str="\t{FUNC_TYPE_GENERAL,\"$_\",${_}_main},\n"; + $str=" {FUNC_TYPE_GENERAL, \"$_\", ${_}_main},\n"; if (($_ =~ /^s_/) || ($_ =~ /^ciphers$/)) { print "#if !defined(OPENSSL_NO_SOCK)\n${str}#endif\n"; } elsif ( ($_ =~ /^speed$/)) @@ -60,7 +60,7 @@ foreach (@ARGV) foreach ("md2","md4","md5","sha","sha1","mdc2","rmd160") { push(@files,$_); - printf "#ifndef OPENSSL_NO_".uc($_)."\n\t{FUNC_TYPE_MD,\"".$_."\",dgst_main},\n#endif\n"; + printf "#ifndef OPENSSL_NO_".uc($_)."\n {FUNC_TYPE_MD, \"".$_."\", dgst_main},\n#endif\n"; } foreach ( @@ -86,7 +86,7 @@ foreach ( { push(@files,$_); - $t=sprintf("\t{FUNC_TYPE_CIPHER,\"%s\",enc_main},\n",$_); + $t=sprintf(" {FUNC_TYPE_CIPHER, \"%s\", enc_main},\n",$_); if ($_ =~ /des/) { $t="#ifndef OPENSSL_NO_DES\n${t}#endif\n"; } elsif ($_ =~ /aes/) { $t="#ifndef OPENSSL_NO_AES\n${t}#endif\n"; } elsif ($_ =~ /camellia/) { $t="#ifndef OPENSSL_NO_CAMELLIA\n${t}#endif\n"; } @@ -101,4 +101,4 @@ foreach ( print $t; } -print "\t{0,NULL,NULL}\n\t};\n"; +print " {0, NULL, NULL}\n};\n"; Modified: vendor-crypto/openssl/dist/apps/req.c ============================================================================== --- vendor-crypto/openssl/dist/apps/req.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/req.c Thu May 25 19:38:38 2017 (r318897) @@ -331,7 +331,6 @@ int MAIN(int argc, char **argv) else if (strcmp(*argv, "-text") == 0) text = 1; else if (strcmp(*argv, "-x509") == 0) { - newreq = 1; x509 = 1; } else if (strcmp(*argv, "-asn1-kludge") == 0) kludge = 1; @@ -447,6 +446,9 @@ int MAIN(int argc, char **argv) goto end; } + if (x509 && infile == NULL) + newreq = 1; + ERR_load_crypto_strings(); if (!app_passwd(bio_err, passargin, passargout, &passin, &passout)) { BIO_printf(bio_err, "Error getting passwords\n"); @@ -753,7 +755,7 @@ int MAIN(int argc, char **argv) } } - if (newreq) { + if (newreq || x509) { if (pkey == NULL) { BIO_printf(bio_err, "you need to specify a private key\n"); goto end; Modified: vendor-crypto/openssl/dist/apps/s_client.c ============================================================================== --- vendor-crypto/openssl/dist/apps/s_client.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/s_client.c Thu May 25 19:38:38 2017 (r318897) @@ -2132,6 +2132,7 @@ int MAIN(int argc, char **argv) BIO_free(bio_c_msg); bio_c_msg = NULL; } + SSL_COMP_free_compression_methods(); apps_shutdown(); OPENSSL_EXIT(ret); } Modified: vendor-crypto/openssl/dist/apps/s_server.c ============================================================================== --- vendor-crypto/openssl/dist/apps/s_server.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/s_server.c Thu May 25 19:38:38 2017 (r318897) @@ -2132,6 +2132,7 @@ int MAIN(int argc, char *argv[]) BIO_free(bio_s_msg); bio_s_msg = NULL; } + SSL_COMP_free_compression_methods(); apps_shutdown(); OPENSSL_EXIT(ret); } Modified: vendor-crypto/openssl/dist/apps/srp.c ============================================================================== --- vendor-crypto/openssl/dist/apps/srp.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/apps/srp.c Thu May 25 19:38:38 2017 (r318897) @@ -183,10 +183,8 @@ static int update_index(CA_DB *db, BIO * return 0; } - for (i = 0; i < DB_NUMBER; i++) { + for (i = 0; i < DB_NUMBER; i++) irow[i] = row[i]; - row[i] = NULL; - } irow[DB_NUMBER] = NULL; if (!TXT_DB_insert(db->db, irow)) { Modified: vendor-crypto/openssl/dist/appveyor.yml ============================================================================== --- vendor-crypto/openssl/dist/appveyor.yml Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/appveyor.yml Thu May 25 19:38:38 2017 (r318897) @@ -50,11 +50,3 @@ build_script: test_script: - nmake /f ms\%MAK% test - -notifications: - - provider: Email - to: - - openssl-commits@openssl.org - on_build_success: false - on_build_failure: true - on_build_status_changed: true Modified: vendor-crypto/openssl/dist/config ============================================================================== --- vendor-crypto/openssl/dist/config Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/config Thu May 25 19:38:38 2017 (r318897) @@ -344,6 +344,15 @@ case "${SYSTEM}:${RELEASE}:${VERSION}:${ echo "mips-sony-newsos4"; exit 0; ;; + # The following combinations are supported + # MINGW64* on x86_64 => mingw64 + # MINGW32* on x86_64 => mingw + # MINGW32* on i?86 => mingw + # + # MINGW64* on i?86 isn't expected to work... + MINGW64*:*:*:x86_64) + echo "${MACHINE}-whatever-mingw64"; exit 0; + ;; MINGW*) echo "${MACHINE}-whatever-mingw"; exit 0; ;; Modified: vendor-crypto/openssl/dist/crypto/aes/Makefile ============================================================================== --- vendor-crypto/openssl/dist/crypto/aes/Makefile Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/aes/Makefile Thu May 25 19:38:38 2017 (r318897) @@ -133,7 +133,7 @@ dclean: mv -f Makefile.new $(MAKEFILE) clean: - rm -f *.s *.o *.obj lib tags core .pure .nfs* *.old *.bak fluff + rm -f *.s *.S *.o *.obj lib tags core .pure .nfs* *.old *.bak fluff # DO NOT DELETE THIS LINE -- make depend depends on it. Modified: vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha1-x86_64.pl ============================================================================== --- vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha1-x86_64.pl Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha1-x86_64.pl Thu May 25 19:38:38 2017 (r318897) @@ -1702,6 +1702,7 @@ $code.=<<___; mov 240($key),$rounds sub $in0,$out movups ($key),$rndkey0 # $key[0] + movups ($ivp),$iv # load IV movups 16($key),$rndkey[0] # forward reference lea 112($key),$key # size optimization Modified: vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha256-x86_64.pl ============================================================================== --- vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha256-x86_64.pl Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/aes/asm/aesni-sha256-x86_64.pl Thu May 25 19:38:38 2017 (r318897) @@ -1299,6 +1299,7 @@ $code.=<<___; mov 240($key),$rounds sub $in0,$out movups ($key),$rndkey0 # $key[0] + movups ($ivp),$iv # load IV movups 16($key),$rndkey[0] # forward reference lea 112($key),$key # size optimization Modified: vendor-crypto/openssl/dist/crypto/aes/asm/bsaes-armv7.pl ============================================================================== --- vendor-crypto/openssl/dist/crypto/aes/asm/bsaes-armv7.pl Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/aes/asm/bsaes-armv7.pl Thu May 25 19:38:38 2017 (r318897) @@ -1333,7 +1333,7 @@ bsaes_cbc_encrypt: vmov @XMM[4],@XMM[15] @ just in case ensure that IV vmov @XMM[5],@XMM[0] @ and input are preserved bl AES_decrypt - vld1.8 {@XMM[0]}, [$fp,:64] @ load result + vld1.8 {@XMM[0]}, [$fp] @ load result veor @XMM[0], @XMM[0], @XMM[4] @ ^= IV vmov @XMM[15], @XMM[5] @ @XMM[5] holds input vst1.8 {@XMM[0]}, [$rounds] @ write output Modified: vendor-crypto/openssl/dist/crypto/asn1/a_bitstr.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/a_bitstr.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/a_bitstr.c Thu May 25 19:38:38 2017 (r318897) @@ -114,10 +114,11 @@ int i2c_ASN1_BIT_STRING(ASN1_BIT_STRING *(p++) = (unsigned char)bits; d = a->data; - memcpy(p, d, len); - p += len; - if (len > 0) + if (len > 0) { + memcpy(p, d, len); + p += len; p[-1] &= (0xff << bits); + } *pp = p; return (ret); } Modified: vendor-crypto/openssl/dist/crypto/asn1/a_digest.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/a_digest.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/a_digest.c Thu May 25 19:38:38 2017 (r318897) @@ -86,8 +86,10 @@ int ASN1_digest(i2d_of_void *i2d, const p = str; i2d(data, &p); - if (!EVP_Digest(str, i, md, len, type, NULL)) + if (!EVP_Digest(str, i, md, len, type, NULL)) { + OPENSSL_free(str); return 0; + } OPENSSL_free(str); return (1); } @@ -104,8 +106,10 @@ int ASN1_item_digest(const ASN1_ITEM *it if (!str) return (0); - if (!EVP_Digest(str, i, md, len, type, NULL)) + if (!EVP_Digest(str, i, md, len, type, NULL)) { + OPENSSL_free(str); return 0; + } OPENSSL_free(str); return (1); } Modified: vendor-crypto/openssl/dist/crypto/asn1/a_gentm.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/a_gentm.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/a_gentm.c Thu May 25 19:38:38 2017 (r318897) @@ -202,7 +202,7 @@ int asn1_generalizedtime_to_tm(struct tm if (a[o] == 'Z') o++; else if ((a[o] == '+') || (a[o] == '-')) { - int offsign = a[o] == '-' ? -1 : 1, offset = 0; + int offsign = a[o] == '-' ? 1 : -1, offset = 0; o++; if (o + 4 > l) goto err; Modified: vendor-crypto/openssl/dist/crypto/asn1/a_strnid.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/a_strnid.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/a_strnid.c Thu May 25 19:38:38 2017 (r318897) @@ -192,7 +192,8 @@ static const ASN1_STRING_TABLE tbl_stand {NID_name, 1, ub_name, DIRSTRING_TYPE, 0}, {NID_dnQualifier, -1, -1, B_ASN1_PRINTABLESTRING, STABLE_NO_MASK}, {NID_domainComponent, 1, -1, B_ASN1_IA5STRING, STABLE_NO_MASK}, - {NID_ms_csp_name, -1, -1, B_ASN1_BMPSTRING, STABLE_NO_MASK} + {NID_ms_csp_name, -1, -1, B_ASN1_BMPSTRING, STABLE_NO_MASK}, + {NID_jurisdictionCountryName, 2, 2, B_ASN1_PRINTABLESTRING, STABLE_NO_MASK} }; static int sk_table_cmp(const ASN1_STRING_TABLE *const *a, Modified: vendor-crypto/openssl/dist/crypto/asn1/a_time.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/a_time.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/a_time.c Thu May 25 19:38:38 2017 (r318897) @@ -137,7 +137,7 @@ int ASN1_TIME_check(ASN1_TIME *t) ASN1_GENERALIZEDTIME *ASN1_TIME_to_generalizedtime(ASN1_TIME *t, ASN1_GENERALIZEDTIME **out) { - ASN1_GENERALIZEDTIME *ret; + ASN1_GENERALIZEDTIME *ret = NULL; char *str; int newlen; @@ -146,22 +146,21 @@ ASN1_GENERALIZEDTIME *ASN1_TIME_to_gener if (!out || !*out) { if (!(ret = ASN1_GENERALIZEDTIME_new())) - return NULL; - if (out) - *out = ret; - } else + goto err; + } else { ret = *out; + } /* If already GeneralizedTime just copy across */ if (t->type == V_ASN1_GENERALIZEDTIME) { if (!ASN1_STRING_set(ret, t->data, t->length)) - return NULL; - return ret; + goto err; + goto done; } /* grow the string */ if (!ASN1_STRING_set(ret, NULL, t->length + 2)) - return NULL; + goto err; /* ASN1_STRING_set() allocated 'len + 1' bytes. */ newlen = t->length + 2 + 1; str = (char *)ret->data; @@ -173,9 +172,18 @@ ASN1_GENERALIZEDTIME *ASN1_TIME_to_gener BUF_strlcat(str, (char *)t->data, newlen); - return ret; + done: + if (out != NULL && *out == NULL) + *out = ret; + return ret; + + err: + if (out == NULL || *out != ret) + ASN1_GENERALIZEDTIME_free(ret); + return NULL; } + int ASN1_TIME_set_string(ASN1_TIME *s, const char *str) { ASN1_TIME t; Modified: vendor-crypto/openssl/dist/crypto/asn1/a_utctm.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/a_utctm.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/a_utctm.c Thu May 25 19:38:38 2017 (r318897) @@ -172,7 +172,7 @@ int asn1_utctime_to_tm(struct tm *tm, co if (a[o] == 'Z') o++; else if ((a[o] == '+') || (a[o] == '-')) { - int offsign = a[o] == '-' ? -1 : 1, offset = 0; + int offsign = a[o] == '-' ? 1 : -1, offset = 0; o++; if (o + 4 > l) goto err; Modified: vendor-crypto/openssl/dist/crypto/asn1/f_enum.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/f_enum.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/f_enum.c Thu May 25 19:38:38 2017 (r318897) @@ -138,7 +138,7 @@ int a2i_ASN1_ENUMERATED(BIO *bp, ASN1_EN bufp = (unsigned char *)buf; if (first) { first = 0; - if ((bufp[0] == '0') && (buf[1] == '0')) { + if ((bufp[0] == '0') && (bufp[1] == '0')) { bufp += 2; i -= 2; } Modified: vendor-crypto/openssl/dist/crypto/asn1/f_int.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/f_int.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/f_int.c Thu May 25 19:38:38 2017 (r318897) @@ -152,7 +152,7 @@ int a2i_ASN1_INTEGER(BIO *bp, ASN1_INTEG bufp = (unsigned char *)buf; if (first) { first = 0; - if ((bufp[0] == '0') && (buf[1] == '0')) { + if ((bufp[0] == '0') && (bufp[1] == '0')) { bufp += 2; i -= 2; } Modified: vendor-crypto/openssl/dist/crypto/asn1/tasn_dec.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/tasn_dec.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/tasn_dec.c Thu May 25 19:38:38 2017 (r318897) @@ -673,6 +673,7 @@ static int asn1_template_noexp_d2i(ASN1_ } len -= p - q; if (!sk_ASN1_VALUE_push((STACK_OF(ASN1_VALUE) *)*val, skfield)) { + ASN1_item_ex_free(&skfield, ASN1_ITEM_ptr(tt->item)); ASN1err(ASN1_F_ASN1_TEMPLATE_NOEXP_D2I, ERR_R_MALLOC_FAILURE); goto err; } Modified: vendor-crypto/openssl/dist/crypto/asn1/tasn_new.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/tasn_new.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/tasn_new.c Thu May 25 19:38:38 2017 (r318897) @@ -158,7 +158,7 @@ static int asn1_item_ex_combine_new(ASN1 } asn1_set_choice_selector(pval, -1, it); if (asn1_cb && !asn1_cb(ASN1_OP_NEW_POST, pval, it, NULL)) - goto auxerr; + goto auxerr2; break; case ASN1_ITYPE_NDEF_SEQUENCE: @@ -186,10 +186,10 @@ static int asn1_item_ex_combine_new(ASN1 for (i = 0, tt = it->templates; i < it->tcount; tt++, i++) { pseqval = asn1_get_field_ptr(pval, tt); if (!ASN1_template_new(pseqval, tt)) - goto memerr; + goto memerr2; } if (asn1_cb && !asn1_cb(ASN1_OP_NEW_POST, pval, it, NULL)) - goto auxerr; + goto auxerr2; break; } #ifdef CRYPTO_MDEBUG @@ -198,6 +198,8 @@ static int asn1_item_ex_combine_new(ASN1 #endif return 1; + memerr2: + ASN1_item_ex_free(pval, it); memerr: ASN1err(ASN1_F_ASN1_ITEM_EX_COMBINE_NEW, ERR_R_MALLOC_FAILURE); #ifdef CRYPTO_MDEBUG @@ -206,9 +208,10 @@ static int asn1_item_ex_combine_new(ASN1 #endif return 0; + auxerr2: + ASN1_item_ex_free(pval, it); auxerr: ASN1err(ASN1_F_ASN1_ITEM_EX_COMBINE_NEW, ASN1_R_AUX_ERROR); - ASN1_item_ex_free(pval, it); #ifdef CRYPTO_MDEBUG if (it->sname) CRYPTO_pop_info(); Modified: vendor-crypto/openssl/dist/crypto/asn1/x_long.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/x_long.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/x_long.c Thu May 25 19:38:38 2017 (r318897) @@ -126,7 +126,7 @@ static int long_i2c(ASN1_VALUE **pval, u * set. */ if (ltmp < 0) - utmp = -ltmp - 1; + utmp = 0 - (unsigned long)ltmp - 1; else utmp = ltmp; clen = BN_num_bits_word(utmp); @@ -155,19 +155,41 @@ static int long_i2c(ASN1_VALUE **pval, u static int long_c2i(ASN1_VALUE **pval, const unsigned char *cont, int len, int utype, char *free_cont, const ASN1_ITEM *it) { - int neg, i; + int neg = -1, i; long ltmp; unsigned long utmp = 0; char *cp = (char *)pval; + + if (len) { + /* + * Check possible pad byte. Worst case, we're skipping past actual + * content, but since that's only with 0x00 and 0xff and we set neg + * accordingly, the result will be correct in the end anyway. + */ + switch (cont[0]) { + case 0xff: + cont++; + len--; + neg = 1; + break; + case 0: + cont++; + len--; + neg = 0; + break; + } + } if (len > (int)sizeof(long)) { ASN1err(ASN1_F_LONG_C2I, ASN1_R_INTEGER_TOO_LARGE_FOR_LONG); return 0; } - /* Is it negative? */ - if (len && (cont[0] & 0x80)) - neg = 1; - else - neg = 0; + if (neg == -1) { + /* Is it negative? */ + if (len && (cont[0] & 0x80)) + neg = 1; + else + neg = 0; + } utmp = 0; for (i = 0; i < len; i++) { utmp <<= 8; @@ -178,8 +200,8 @@ static int long_c2i(ASN1_VALUE **pval, c } ltmp = (long)utmp; if (neg) { - ltmp++; ltmp = -ltmp; + ltmp--; } if (ltmp == it->size) { ASN1err(ASN1_F_LONG_C2I, ASN1_R_INTEGER_TOO_LARGE_FOR_LONG); Modified: vendor-crypto/openssl/dist/crypto/asn1/x_name.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/asn1/x_name.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/asn1/x_name.c Thu May 25 19:38:38 2017 (r318897) @@ -178,6 +178,16 @@ static void x509_name_ex_free(ASN1_VALUE *pval = NULL; } +static void local_sk_X509_NAME_ENTRY_free(STACK_OF(X509_NAME_ENTRY) *ne) +{ + sk_X509_NAME_ENTRY_free(ne); +} + +static void local_sk_X509_NAME_ENTRY_pop_free(STACK_OF(X509_NAME_ENTRY) *ne) +{ + sk_X509_NAME_ENTRY_pop_free(ne, X509_NAME_ENTRY_free); +} + static int x509_name_ex_d2i(ASN1_VALUE **val, const unsigned char **in, long len, const ASN1_ITEM *it, int tag, int aclass, @@ -228,13 +238,14 @@ static int x509_name_ex_d2i(ASN1_VALUE * entry->set = i; if (!sk_X509_NAME_ENTRY_push(nm.x->entries, entry)) goto err; + sk_X509_NAME_ENTRY_set(entries, j, NULL); } - sk_X509_NAME_ENTRY_free(entries); } - sk_STACK_OF_X509_NAME_ENTRY_free(intname.s); ret = x509_name_canon(nm.x); if (!ret) goto err; + sk_STACK_OF_X509_NAME_ENTRY_pop_free(intname.s, + local_sk_X509_NAME_ENTRY_free); nm.x->modified = 0; *val = nm.a; *in = p; @@ -242,6 +253,8 @@ static int x509_name_ex_d2i(ASN1_VALUE * err: if (nm.x != NULL) X509_NAME_free(nm.x); + sk_STACK_OF_X509_NAME_ENTRY_pop_free(intname.s, + local_sk_X509_NAME_ENTRY_pop_free); ASN1err(ASN1_F_X509_NAME_EX_D2I, ERR_R_NESTED_ASN1_ERROR); return 0; } @@ -267,16 +280,6 @@ static int x509_name_ex_i2d(ASN1_VALUE * return ret; } -static void local_sk_X509_NAME_ENTRY_free(STACK_OF(X509_NAME_ENTRY) *ne) -{ - sk_X509_NAME_ENTRY_free(ne); -} - -static void local_sk_X509_NAME_ENTRY_pop_free(STACK_OF(X509_NAME_ENTRY) *ne) -{ - sk_X509_NAME_ENTRY_pop_free(ne, X509_NAME_ENTRY_free); -} - static int x509_name_encode(X509_NAME *a) { union { @@ -299,8 +302,10 @@ static int x509_name_encode(X509_NAME *a entries = sk_X509_NAME_ENTRY_new_null(); if (!entries) goto memerr; - if (!sk_STACK_OF_X509_NAME_ENTRY_push(intname.s, entries)) + if (!sk_STACK_OF_X509_NAME_ENTRY_push(intname.s, entries)) { + sk_X509_NAME_ENTRY_free(entries); goto memerr; + } set = entry->set; } if (!sk_X509_NAME_ENTRY_push(entries, entry)) @@ -370,8 +375,10 @@ static int x509_name_canon(X509_NAME *a) entries = sk_X509_NAME_ENTRY_new_null(); if (!entries) goto err; - if (!sk_STACK_OF_X509_NAME_ENTRY_push(intname, entries)) + if (!sk_STACK_OF_X509_NAME_ENTRY_push(intname, entries)) { + sk_X509_NAME_ENTRY_free(entries); goto err; + } set = entry->set; } tmpentry = X509_NAME_ENTRY_new(); Modified: vendor-crypto/openssl/dist/crypto/bio/b_print.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/bio/b_print.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/bio/b_print.c Thu May 25 19:38:38 2017 (r318897) @@ -502,7 +502,7 @@ fmtint(char **sbuffer, if (!(flags & DP_F_UNSIGNED)) { if (value < 0) { signvalue = '-'; - uvalue = -value; + uvalue = -(unsigned LLONG)value; } else if (flags & DP_F_PLUS) signvalue = '+'; else if (flags & DP_F_SPACE) Modified: vendor-crypto/openssl/dist/crypto/bio/bio_cb.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/bio/bio_cb.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/bio/bio_cb.c Thu May 25 19:38:38 2017 (r318897) @@ -78,6 +78,9 @@ long MS_CALLBACK BIO_debug_callback(BIO len = BIO_snprintf(buf,sizeof buf,"BIO[%p]: ",(void *)bio); + /* Ignore errors and continue printing the other information. */ + if (len < 0) + len = 0; p = buf + len; p_maxlen = sizeof(buf) - len; Modified: vendor-crypto/openssl/dist/crypto/bio/bss_file.c ============================================================================== --- vendor-crypto/openssl/dist/crypto/bio/bss_file.c Thu May 25 19:20:06 2017 (r318896) +++ vendor-crypto/openssl/dist/crypto/bio/bss_file.c Thu May 25 19:38:38 2017 (r318897) @@ -251,7 +251,7 @@ static int MS_CALLBACK file_read(BIO *b, ret = fread(out, 1, (int)outl, (FILE *)b->ptr); if (ret == 0 && (b->flags & BIO_FLAGS_UPLINK) ? UP_ferror((FILE *)b->ptr) : - ferror((FILE *)b->ptr)) { + ferror((FILE *)b->ptr)) { SYSerr(SYS_F_FREAD, get_last_sys_error()); BIOerr(BIO_F_FILE_READ, ERR_R_SYS_LIB); ret = -1; @@ -287,6 +287,7 @@ static long MS_CALLBACK file_ctrl(BIO *b FILE *fp = (FILE *)b->ptr; FILE **fpp; char p[4]; *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Thu May 25 19:39:18 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02EF8D824FF; Thu, 25 May 2017 19:39:18 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AF0371537; Thu, 25 May 2017 19:39:17 +0000 (UTC) (envelope-from jkim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4PJdG6G083628; Thu, 25 May 2017 19:39:16 GMT (envelope-from jkim@FreeBSD.org) Received: (from jkim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4PJdG7h083627; Thu, 25 May 2017 19:39:16 GMT (envelope-from jkim@FreeBSD.org) Message-Id: <201705251939.v4PJdG7h083627@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jkim set sender to jkim@FreeBSD.org using -f From: Jung-uk Kim Date: Thu, 25 May 2017 19:39:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318898 - vendor-crypto/openssl/1.0.2l X-SVN-Group: vendor-crypto MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 May 2017 19:39:18 -0000 Author: jkim Date: Thu May 25 19:39:16 2017 New Revision: 318898 URL: https://svnweb.freebsd.org/changeset/base/318898 Log: Tag OpenSSL 1.0.2l. Added: vendor-crypto/openssl/1.0.2l/ - copied from r318897, vendor-crypto/openssl/dist/ From owner-svn-src-vendor@freebsd.org Fri May 26 11:29:32 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3225D823FA; Fri, 26 May 2017 11:29:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5AACB1483; Fri, 26 May 2017 11:29:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBTVng076156; Fri, 26 May 2017 11:29:31 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBTVl9076155; Fri, 26 May 2017 11:29:31 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261129.v4QBTVl9076155@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:29:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318927 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:29:32 -0000 Author: avg Date: Fri May 26 11:29:31 2017 New Revision: 318927 URL: https://svnweb.freebsd.org/changeset/base/318927 Log: 8025 dbuf_read() creates unnecessary zio_root() for bonus buf illumos/illumos-gate@def4fac5882b4ca67bd0f4a53509b6d1fa8ae14e https://github.com/illumos/illumos-gate/commit/def4fac5882b4ca67bd0f4a53509b6d1fa8ae14e https://www.illumos.org/issues/8025 dbuf_read() creates a zio_root() to track and wait for all the zio's that may happen as part of this call. However, if the blkptr_t for this buffer is NULL or a hole, we will not create any more zio's, so this zio_root() is unnecessary. This is always the case when calling dbuf_read() on a bonus buffer, because it has no blkptr (it's part of the containing dnode). For workloads that read a lot of bonus buffers (e.g. file creation and removal), creating and destroying these unnecessary zio's can decrease performance by around 3%. Reviewed by: Dan Kimmel Reviewed by: Pavel Zakharov Reviewed by: Prashanth Sreenivasa Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Fri May 26 11:25:44 2017 (r318926) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Fri May 26 11:29:31 2017 (r318927) @@ -1088,7 +1088,6 @@ int dbuf_read(dmu_buf_impl_t *db, zio_t *zio, uint32_t flags) { int err = 0; - boolean_t havepzio = (zio != NULL); boolean_t prefetch; dnode_t *dn; @@ -1132,9 +1131,13 @@ dbuf_read(dmu_buf_impl_t *db, zio_t *zio DB_DNODE_EXIT(db); } else if (db->db_state == DB_UNCACHED) { spa_t *spa = dn->dn_objset->os_spa; + boolean_t need_wait = B_FALSE; - if (zio == NULL) + if (zio == NULL && + db->db_blkptr != NULL && !BP_IS_HOLE(db->db_blkptr)) { zio = zio_root(spa, NULL, NULL, ZIO_FLAG_CANFAIL); + need_wait = B_TRUE; + } dbuf_read_impl(db, zio, flags); /* dbuf_read_impl has dropped db_mtx for us */ @@ -1146,7 +1149,7 @@ dbuf_read(dmu_buf_impl_t *db, zio_t *zio rw_exit(&dn->dn_struct_rwlock); DB_DNODE_EXIT(db); - if (!havepzio) + if (need_wait) err = zio_wait(zio); } else { /* @@ -1181,7 +1184,6 @@ dbuf_read(dmu_buf_impl_t *db, zio_t *zio mutex_exit(&db->db_mtx); } - ASSERT(err || havepzio || db->db_state == DB_CACHED); return (err); } From owner-svn-src-vendor@freebsd.org Fri May 26 11:32:07 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2DCACD825D5; Fri, 26 May 2017 11:32:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E6BCE1A80; Fri, 26 May 2017 11:32:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBW5oQ079970; Fri, 26 May 2017 11:32:05 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBW5bb079969; Fri, 26 May 2017 11:32:05 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261132.v4QBW5bb079969@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:32:05 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318929 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:32:07 -0000 Author: avg Date: Fri May 26 11:32:05 2017 New Revision: 318929 URL: https://svnweb.freebsd.org/changeset/base/318929 Log: 7786 zfs`vdev_online() needs better notification about state changes illumos/illumos-gate@5f368aef86387d6ef4eda84030ae9b402313ee4c https://github.com/illumos/illumos-gate/commit/5f368aef86387d6ef4eda84030ae9b402313ee4c https://www.illumos.org/issues/7786 Currently, vdev_online() will only post sysevent if previous state was "offline". It should also post the event when the state changes from "removed" or "faulted" to "healthy" or "degraded". This will fix the following scenario: - pull disk from slot A - check that hotspare has taken its place (if available) - insert disk into slot B - check that hotspare moved back to "avail" state (if spare was used) The problem here is that we don't get any ESC_ZFS_VDEV_* notification and fail to update the vdev FRU. Reviewed by: Matthew Ahrens mahrens@delphix.com Reviewed by: George Wilson george.wilson@delphix.com Approved by: Albert Lee Author: Yuri Pankov Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 11:30:55 2017 (r318928) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 11:32:05 2017 (r318929) @@ -22,7 +22,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2011, 2015 by Delphix. All rights reserved. - * Copyright 2015 Nexenta Systems, Inc. All rights reserved. + * Copyright 2017 Nexenta Systems, Inc. * Copyright (c) 2014 Integros [integros.com] * Copyright 2016 Toomas Soome */ @@ -2449,7 +2449,8 @@ int vdev_online(spa_t *spa, uint64_t guid, uint64_t flags, vdev_state_t *newstate) { vdev_t *vd, *tvd, *pvd, *rvd = spa->spa_root_vdev; - boolean_t postevent = B_FALSE; + boolean_t wasoffline; + vdev_state_t oldstate; spa_vdev_state_enter(spa, SCL_NONE); @@ -2459,9 +2460,8 @@ vdev_online(spa_t *spa, uint64_t guid, u if (!vd->vdev_ops->vdev_op_leaf) return (spa_vdev_state_exit(spa, NULL, ENOTSUP)); - postevent = - (vd->vdev_offline == B_TRUE || vd->vdev_tmpoffline == B_TRUE) ? - B_TRUE : B_FALSE; + wasoffline = (vd->vdev_offline || vd->vdev_tmpoffline); + oldstate = vd->vdev_state; tvd = vd->vdev_top; vd->vdev_offline = B_FALSE; @@ -2499,7 +2499,9 @@ vdev_online(spa_t *spa, uint64_t guid, u spa_async_request(spa, SPA_ASYNC_CONFIG_UPDATE); } - if (postevent) + if (wasoffline || + (oldstate < VDEV_STATE_DEGRADED && + vd->vdev_state >= VDEV_STATE_DEGRADED)) spa_event_notify(spa, vd, ESC_ZFS_VDEV_ONLINE); return (spa_vdev_state_exit(spa, vd, 0)); From owner-svn-src-vendor@freebsd.org Fri May 26 11:35:36 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6183ED829CB; Fri, 26 May 2017 11:35:36 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3BEC11DEF; Fri, 26 May 2017 11:35:36 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBZZ4c080230; Fri, 26 May 2017 11:35:35 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBZYRB080222; Fri, 26 May 2017 11:35:34 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261135.v4QBZYRB080222@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:35:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318931 - in vendor-sys/illumos/dist/uts/common/fs/zfs: . sys X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:35:36 -0000 Author: avg Date: Fri May 26 11:35:34 2017 New Revision: 318931 URL: https://svnweb.freebsd.org/changeset/base/318931 Log: 8063 verify that we do not attempt to access inactive txg illumos/illumos-gate@b7b2590dd9f11b12a0b4878db3886068cce176af https://github.com/illumos/illumos-gate/commit/b7b2590dd9f11b12a0b4878db3886068cce176af https://www.illumos.org/issues/8063 A standard practice in ZFS is to keep track of "per-txg" state. Any of the 3 active TXG's (open, quiescing, syncing) can have different values for this state. We should assert that we do not attempt to modify other (inactive) TXG's. Reviewed by: Serapheim Dimitropoulos Reviewed by: Pavel Zakharov Approved by: Robert Mustacchi Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_tx.c Fri May 26 11:35:34 2017 (r318931) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. - * Copyright (c) 2012, 2016 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] */ @@ -72,7 +72,7 @@ dmu_tx_create_assigned(struct dsl_pool * { dmu_tx_t *tx = dmu_tx_create_dd(NULL); - ASSERT3U(txg, <=, dp->dp_tx.tx_open_txg); + txg_verify(dp->dp_spa, txg); tx->tx_pool = dp; tx->tx_txg = txg; tx->tx_anyobj = TRUE; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_pool.c Fri May 26 11:35:34 2017 (r318931) @@ -159,13 +159,13 @@ dsl_pool_open_impl(spa_t *spa, uint64_t rrw_init(&dp->dp_config_rwlock, B_TRUE); txg_init(dp, txg); - txg_list_create(&dp->dp_dirty_datasets, + txg_list_create(&dp->dp_dirty_datasets, spa, offsetof(dsl_dataset_t, ds_dirty_link)); - txg_list_create(&dp->dp_dirty_zilogs, + txg_list_create(&dp->dp_dirty_zilogs, spa, offsetof(zilog_t, zl_dirty_link)); - txg_list_create(&dp->dp_dirty_dirs, + txg_list_create(&dp->dp_dirty_dirs, spa, offsetof(dsl_dir_t, dd_dirty_link)); - txg_list_create(&dp->dp_sync_tasks, + txg_list_create(&dp->dp_sync_tasks, spa, offsetof(dsl_sync_task_t, dst_node)); dp->dp_sync_taskq = taskq_create("dp_sync_taskq", Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Fri May 26 11:35:34 2017 (r318931) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2014 by Delphix. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright (c) 2015, Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2014 Spectra Logic Corporation, All rights reserved. * Copyright 2013 Saso Kiselkov. All rights reserved. @@ -1090,7 +1090,7 @@ spa_activate(spa_t *spa, int mode) list_create(&spa->spa_state_dirty_list, sizeof (vdev_t), offsetof(vdev_t, vdev_state_dirty_node)); - txg_list_create(&spa->spa_vdev_txg_list, + txg_list_create(&spa->spa_vdev_txg_list, spa, offsetof(struct vdev, vdev_txg_node)); avl_create(&spa->spa_errlist_scrub, Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg.h Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/txg.h Fri May 26 11:35:34 2017 (r318931) @@ -23,7 +23,7 @@ * Use is subject to license terms. */ /* - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. */ #ifndef _SYS_TXG_H @@ -60,6 +60,7 @@ typedef struct txg_node { typedef struct txg_list { kmutex_t tl_lock; size_t tl_offset; + spa_t *tl_spa; txg_node_t *tl_head[TXG_SIZE]; } txg_list_t; @@ -103,13 +104,15 @@ extern boolean_t txg_stalled(struct dsl_ /* returns TRUE if someone is waiting for the next txg to sync */ extern boolean_t txg_sync_waiting(struct dsl_pool *dp); +extern void txg_verify(spa_t *spa, uint64_t txg); + /* * Per-txg object lists. */ #define TXG_CLEAN(txg) ((txg) - 1) -extern void txg_list_create(txg_list_t *tl, size_t offset); +extern void txg_list_create(txg_list_t *tl, spa_t *spa, size_t offset); extern void txg_list_destroy(txg_list_t *tl); extern boolean_t txg_list_empty(txg_list_t *tl, uint64_t txg); extern boolean_t txg_all_lists_empty(txg_list_t *tl); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h Fri May 26 11:35:34 2017 (r318931) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] */ @@ -96,6 +96,15 @@ typedef struct zil_chain { #define ZIL_MIN_BLKSZ 4096ULL /* + * ziltest is by and large an ugly hack, but very useful in + * checking replay without tedious work. + * When running ziltest we want to keep all itx's and so maintain + * a single list in the zl_itxg[] that uses a high txg: ZILTEST_TXG + * We subtract TXG_CONCURRENT_STATES to allow for common code. + */ +#define ZILTEST_TXG (UINT64_MAX - TXG_CONCURRENT_STATES) + +/* * The words of a log block checksum. */ #define ZIL_ZC_GUID_0 0 Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/txg.c Fri May 26 11:35:34 2017 (r318931) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Portions Copyright 2011 Martin Matuska - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. */ #include @@ -30,6 +30,7 @@ #include #include #include +#include #include /* @@ -685,16 +686,32 @@ txg_sync_waiting(dsl_pool_t *dp) } /* + * Verify that this txg is active (open, quiescing, syncing). Non-active + * txg's should not be manipulated. + */ +void +txg_verify(spa_t *spa, uint64_t txg) +{ + dsl_pool_t *dp = spa_get_dsl(spa); + if (txg <= TXG_INITIAL || txg == ZILTEST_TXG) + return; + ASSERT3U(txg, <=, dp->dp_tx.tx_open_txg); + ASSERT3U(txg, >=, dp->dp_tx.tx_synced_txg); + ASSERT3U(txg, >=, dp->dp_tx.tx_open_txg - TXG_CONCURRENT_STATES); +} + +/* * Per-txg object lists. */ void -txg_list_create(txg_list_t *tl, size_t offset) +txg_list_create(txg_list_t *tl, spa_t *spa, size_t offset) { int t; mutex_init(&tl->tl_lock, NULL, MUTEX_DEFAULT, NULL); tl->tl_offset = offset; + tl->tl_spa = spa; for (t = 0; t < TXG_SIZE; t++) tl->tl_head[t] = NULL; @@ -714,15 +731,16 @@ txg_list_destroy(txg_list_t *tl) boolean_t txg_list_empty(txg_list_t *tl, uint64_t txg) { + txg_verify(tl->tl_spa, txg); return (tl->tl_head[txg & TXG_MASK] == NULL); } /* * Returns true if all txg lists are empty. * - * Warning: this is inherently racy (an item could be added immediately after this - * function returns). We don't bother with the lock because it wouldn't change the - * semantics. + * Warning: this is inherently racy (an item could be added immediately + * after this function returns). We don't bother with the lock because + * it wouldn't change the semantics. */ boolean_t txg_all_lists_empty(txg_list_t *tl) @@ -746,6 +764,7 @@ txg_list_add(txg_list_t *tl, void *p, ui txg_node_t *tn = (txg_node_t *)((char *)p + tl->tl_offset); boolean_t add; + txg_verify(tl->tl_spa, txg); mutex_enter(&tl->tl_lock); add = (tn->tn_member[t] == 0); if (add) { @@ -770,6 +789,7 @@ txg_list_add_tail(txg_list_t *tl, void * txg_node_t *tn = (txg_node_t *)((char *)p + tl->tl_offset); boolean_t add; + txg_verify(tl->tl_spa, txg); mutex_enter(&tl->tl_lock); add = (tn->tn_member[t] == 0); if (add) { @@ -797,6 +817,7 @@ txg_list_remove(txg_list_t *tl, uint64_t txg_node_t *tn; void *p = NULL; + txg_verify(tl->tl_spa, txg); mutex_enter(&tl->tl_lock); if ((tn = tl->tl_head[t]) != NULL) { p = (char *)tn - tl->tl_offset; @@ -818,6 +839,7 @@ txg_list_remove_this(txg_list_t *tl, voi int t = txg & TXG_MASK; txg_node_t *tn, **tp; + txg_verify(tl->tl_spa, txg); mutex_enter(&tl->tl_lock); for (tp = &tl->tl_head[t]; (tn = *tp) != NULL; tp = &tn->tn_next[t]) { @@ -841,6 +863,7 @@ txg_list_member(txg_list_t *tl, void *p, int t = txg & TXG_MASK; txg_node_t *tn = (txg_node_t *)((char *)p + tl->tl_offset); + txg_verify(tl->tl_spa, txg); return (tn->tn_member[t] != 0); } @@ -853,6 +876,7 @@ txg_list_head(txg_list_t *tl, uint64_t t int t = txg & TXG_MASK; txg_node_t *tn = tl->tl_head[t]; + txg_verify(tl->tl_spa, txg); return (tn == NULL ? NULL : (char *)tn - tl->tl_offset); } @@ -862,6 +886,7 @@ txg_list_next(txg_list_t *tl, void *p, u int t = txg & TXG_MASK; txg_node_t *tn = (txg_node_t *)((char *)p + tl->tl_offset); + txg_verify(tl->tl_spa, txg); tn = tn->tn_next[t]; return (tn == NULL ? NULL : (char *)tn - tl->tl_offset); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 11:35:34 2017 (r318931) @@ -354,9 +354,9 @@ vdev_alloc_common(spa_t *spa, uint_t id, vd->vdev_dtl[t] = range_tree_create(NULL, NULL, &vd->vdev_dtl_lock); } - txg_list_create(&vd->vdev_ms_list, + txg_list_create(&vd->vdev_ms_list, spa, offsetof(struct metaslab, ms_txg_node)); - txg_list_create(&vd->vdev_dtl_list, + txg_list_create(&vd->vdev_dtl_list, spa, offsetof(struct vdev, vdev_dtl_node)); vd->vdev_stat.vs_timestamp = gethrtime(); vdev_queue_init(vd); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Fri May 26 11:33:34 2017 (r318930) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Fri May 26 11:35:34 2017 (r318931) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2011, 2016 by Delphix. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] */ @@ -85,16 +85,6 @@ static void zil_async_to_sync(zilog_t *z #define LWB_EMPTY(lwb) ((BP_GET_LSIZE(&lwb->lwb_blk) - \ sizeof (zil_chain_t)) == (lwb->lwb_sz - lwb->lwb_nused)) - -/* - * ziltest is by and large an ugly hack, but very useful in - * checking replay without tedious work. - * When running ziltest we want to keep all itx's and so maintain - * a single list in the zl_itxg[] that uses a high txg: ZILTEST_TXG - * We subtract TXG_CONCURRENT_STATES to allow for common code. - */ -#define ZILTEST_TXG (UINT64_MAX - TXG_CONCURRENT_STATES) - static int zil_bp_compare(const void *x1, const void *x2) { From owner-svn-src-vendor@freebsd.org Fri May 26 11:39:36 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2954AD82B33; Fri, 26 May 2017 11:39:36 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 00DE310D0; Fri, 26 May 2017 11:39:35 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBdYr4080469; Fri, 26 May 2017 11:39:34 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBdY1w080462; Fri, 26 May 2017 11:39:34 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261139.v4QBdY1w080462@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:39:34 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318933 - in vendor-sys/illumos/dist/uts/common: fs fs/zfs sys X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:39:36 -0000 Author: avg Date: Fri May 26 11:39:34 2017 New Revision: 318933 URL: https://svnweb.freebsd.org/changeset/base/318933 Log: 8064 need a static DTrace probe in VN_HOLD illumos/illumos-gate@ade42b557a6e29c3d17a61b1535d99af10e379be https://github.com/illumos/illumos-gate/commit/ade42b557a6e29c3d17a61b1535d99af10e379be https://www.illumos.org/issues/8064 It's currently nearly impossible to trace what process places a hold on a vnode, as the only ways holds are place is via the `VN_HOLD()` and `VN_HOLD_CALLER()` macros, which inline the bumping of `v_count`. Adding static DTrace probes to these macros would enable tracing of where specific vnode references come from. For completeness and symmetry, a similar static probe should be added to `vn_rele()` and `vn_rele_dnlc()`. Reviewed by: Pavel Zakharov Reviewed by: Prakash Surya Reviewed by: Prashanth Sreenivasa Reviewed by: Matthew Ahrens Approved by: Robert Mustacchi Author: Sebastien Roy Modified: vendor-sys/illumos/dist/uts/common/fs/gfs.c vendor-sys/illumos/dist/uts/common/fs/vnode.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ctldir.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c vendor-sys/illumos/dist/uts/common/sys/vnode.h Modified: vendor-sys/illumos/dist/uts/common/fs/gfs.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/gfs.c Fri May 26 11:37:11 2017 (r318932) +++ vendor-sys/illumos/dist/uts/common/fs/gfs.c Fri May 26 11:39:34 2017 (r318933) @@ -23,8 +23,9 @@ * Copyright 2008 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ - -#pragma ident "%Z%%M% %I% %E% SMI" +/* + * Copyright (c) 2017 by Delphix. All rights reserved. + */ #include #include @@ -671,7 +672,7 @@ found: } vn_free(vp); } else { - vp->v_count--; + VN_RELE_LOCKED(vp); data = NULL; mutex_exit(&vp->v_lock); if (vp->v_flag & V_XATTRDIR) { Modified: vendor-sys/illumos/dist/uts/common/fs/vnode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/vnode.c Fri May 26 11:37:11 2017 (r318932) +++ vendor-sys/illumos/dist/uts/common/fs/vnode.c Fri May 26 11:39:34 2017 (r318933) @@ -22,6 +22,8 @@ /* * Copyright (c) 1988, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright 2016 Nexenta Systems, Inc. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. */ /* Copyright (c) 1983, 1984, 1985, 1986, 1987, 1988, 1989 AT&T */ @@ -836,7 +838,7 @@ vn_rele(vnode_t *vp) VOP_INACTIVE(vp, CRED(), NULL); return; } - vp->v_count--; + VN_RELE_LOCKED(vp); mutex_exit(&vp->v_lock); } @@ -857,15 +859,15 @@ vn_rele_dnlc(vnode_t *vp) VOP_INACTIVE(vp, CRED(), NULL); return; } - vp->v_count--; + VN_RELE_LOCKED(vp); } mutex_exit(&vp->v_lock); } /* * Like vn_rele() except that it clears v_stream under v_lock. - * This is used by sockfs when it dismantels the association between - * the sockfs node and the vnode in the underlaying file system. + * This is used by sockfs when it dismantles the association between + * the sockfs node and the vnode in the underlying file system. * v_lock has to be held to prevent a thread coming through the lookupname * path from accessing a stream head that is going away. */ @@ -880,7 +882,7 @@ vn_rele_stream(vnode_t *vp) VOP_INACTIVE(vp, CRED(), NULL); return; } - vp->v_count--; + VN_RELE_LOCKED(vp); mutex_exit(&vp->v_lock); } @@ -911,7 +913,7 @@ vn_rele_async(vnode_t *vp, taskq_t *task vp, TQ_SLEEP) != NULL); return; } - vp->v_count--; + VN_RELE_LOCKED(vp); mutex_exit(&vp->v_lock); } Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ctldir.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ctldir.c Fri May 26 11:37:11 2017 (r318932) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_ctldir.c Fri May 26 11:39:34 2017 (r318933) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright 2015, OmniTI Computer Consulting, Inc. All rights reserved. */ @@ -1210,7 +1210,7 @@ zfsctl_snapshot_inactive(vnode_t *vp, cr mutex_enter(&vp->v_lock); if (vp->v_count > 1) { - vp->v_count--; + VN_RELE_LOCKED(vp); mutex_exit(&vp->v_lock); mutex_exit(&sdp->sd_lock); VN_RELE(dvp); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Fri May 26 11:37:11 2017 (r318932) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_vnops.c Fri May 26 11:39:34 2017 (r318933) @@ -21,7 +21,7 @@ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2015 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] * Copyright 2015 Joyent, Inc. * Copyright 2017 Nexenta Systems, Inc. @@ -1819,7 +1819,7 @@ top: ASSERT0(error); } mutex_enter(&vp->v_lock); - vp->v_count--; + VN_RELE_LOCKED(vp); ASSERT0(vp->v_count); mutex_exit(&vp->v_lock); mutex_exit(&zp->z_lock); @@ -4411,7 +4411,7 @@ zfs_inactive(vnode_t *vp, cred_t *cr, ca mutex_enter(&zp->z_lock); mutex_enter(&vp->v_lock); ASSERT(vp->v_count == 1); - vp->v_count = 0; + VN_RELE_LOCKED(vp); mutex_exit(&vp->v_lock); mutex_exit(&zp->z_lock); rw_exit(&zfsvfs->z_teardown_inactive_lock); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Fri May 26 11:37:11 2017 (r318932) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_znode.c Fri May 26 11:39:34 2017 (r318933) @@ -20,7 +20,7 @@ */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. - * Copyright (c) 2012, 2014 by Delphix. All rights reserved. + * Copyright (c) 2012, 2017 by Delphix. All rights reserved. * Copyright (c) 2014 Integros [integros.com] */ @@ -1293,7 +1293,7 @@ zfs_zinactive(znode_t *zp) mutex_enter(&zp->z_lock); mutex_enter(&vp->v_lock); - vp->v_count--; + VN_RELE_LOCKED(vp); if (vp->v_count > 0 || vn_has_cached_data(vp)) { /* * If the hold count is greater than zero, somebody has Modified: vendor-sys/illumos/dist/uts/common/sys/vnode.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/vnode.h Fri May 26 11:37:11 2017 (r318932) +++ vendor-sys/illumos/dist/uts/common/sys/vnode.h Fri May 26 11:39:34 2017 (r318933) @@ -22,6 +22,7 @@ /* * Copyright (c) 1988, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. + * Copyright (c) 2011, 2017 by Delphix. All rights reserved. */ /* Copyright (c) 1983, 1984, 1985, 1986, 1987, 1988, 1989 AT&T */ @@ -53,6 +54,7 @@ #include #ifdef _KERNEL #include +#include #endif /* _KERNEL */ #ifdef __cplusplus @@ -1342,10 +1344,32 @@ int vn_vmpss_usepageio(vnode_t *); */ extern uint_t pvn_vmodsort_supported; -#define VN_HOLD(vp) { \ - mutex_enter(&(vp)->v_lock); \ - (vp)->v_count++; \ - mutex_exit(&(vp)->v_lock); \ +/* + * All changes to v_count should be done through VN_HOLD() or VN_RELE(), or + * one of their variants. This makes it possible to ensure proper locking, + * and to guarantee that all modifications are accompanied by a firing of + * the vn-hold or vn-rele SDT DTrace probe. + * + * Example DTrace command for tracing vnode references using these probes: + * + * dtrace -q -n 'sdt:::vn-hold,sdt:::vn-rele + * { + * this->vp = (vnode_t *)arg0; + * printf("%s %s(%p[%s]) %d\n", execname, probename, this->vp, + * this->vp->v_path == NULL ? "NULL" : stringof(this->vp->v_path), + * this->vp->v_count) + * }' + */ +#define VN_HOLD_LOCKED(vp) { \ + ASSERT(mutex_owned(&(vp)->v_lock)); \ + (vp)->v_count++; \ + DTRACE_PROBE1(vn__hold, vnode_t *, vp); \ +} + +#define VN_HOLD(vp) { \ + mutex_enter(&(vp)->v_lock); \ + VN_HOLD_LOCKED(vp); \ + mutex_exit(&(vp)->v_lock); \ } #define VN_RELE(vp) { \ @@ -1356,6 +1380,13 @@ extern uint_t pvn_vmodsort_supported; vn_rele_async(vp, taskq); \ } +#define VN_RELE_LOCKED(vp) { \ + ASSERT(mutex_owned(&(vp)->v_lock)); \ + ASSERT((vp)->v_count >= 1); \ + (vp)->v_count--; \ + DTRACE_PROBE1(vn__rele, vnode_t *, vp); \ +} + #define VN_SET_VFS_TYPE_DEV(vp, vfsp, type, dev) { \ (vp)->v_vfsp = (vfsp); \ (vp)->v_type = (type); \ From owner-svn-src-vendor@freebsd.org Fri May 26 11:48:30 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF24ED82DF8; Fri, 26 May 2017 11:48:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 750AA17A8; Fri, 26 May 2017 11:48:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBmThw085091; Fri, 26 May 2017 11:48:29 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBmTkV085089; Fri, 26 May 2017 11:48:29 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261148.v4QBmTkV085089@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:48:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318934 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:48:30 -0000 Author: avg Date: Fri May 26 11:48:29 2017 New Revision: 318934 URL: https://svnweb.freebsd.org/changeset/base/318934 Log: 8070 Add some ZFS comments illumos/illumos-gate@40713f2b249d289022c715107b3951055a63aef0 https://github.com/illumos/illumos-gate/commit/40713f2b249d289022c715107b3951055a63aef0 https://www.illumos.org/issues/8070 Add some ZFS comments left by various developers at different times Reviewed by: Yuri Pankov Reviewed by: Matthew Ahrens Approved by: Robert Mustacchi Author: Alan Somers Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Fri May 26 11:39:34 2017 (r318933) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c Fri May 26 11:48:29 2017 (r318934) @@ -1219,6 +1219,11 @@ dbuf_unoverride(dbuf_dirty_record_t *dr) uint64_t txg = dr->dr_txg; ASSERT(MUTEX_HELD(&db->db_mtx)); + /* + * This assert is valid because dmu_sync() expects to be called by + * a zilog's get_data while holding a range lock. This call only + * comes from dbuf_dirty() callers who must also hold a range lock. + */ ASSERT(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC); ASSERT(db->db_level == 0); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Fri May 26 11:39:34 2017 (r318933) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c Fri May 26 11:48:29 2017 (r318934) @@ -779,7 +779,7 @@ dsl_scan_visitbp(blkptr_t *bp, const zbo return; /* - * If dsl_scan_ddt() has aready visited this block, it will have + * If dsl_scan_ddt() has already visited this block, it will have * already done any translations or scrubbing, so don't call the * callback again. */ @@ -1446,6 +1446,7 @@ dsl_scan_active(dsl_scan_t *scn) return (used != 0); } +/* Called whenever a txg syncs. */ void dsl_scan_sync(dsl_pool_t *dp, dmu_tx_t *tx) { @@ -1863,6 +1864,7 @@ dsl_scan_scrub_cb(dsl_pool_t *dp, return (0); } +/* Called by the ZFS_IOC_POOL_SCAN ioctl to start a scrub or resilver */ int dsl_scan(dsl_pool_t *dp, pool_scan_func_t func) { From owner-svn-src-vendor@freebsd.org Fri May 26 11:54:44 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06E79D83078; Fri, 26 May 2017 11:54:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B71431D67; Fri, 26 May 2017 11:54:43 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBsgLA089026; Fri, 26 May 2017 11:54:42 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBsguB089025; Fri, 26 May 2017 11:54:42 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261154.v4QBsguB089025@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:54:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318936 - vendor/illumos/dist/man/man1m X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:54:44 -0000 Author: avg Date: Fri May 26 11:54:42 2017 New Revision: 318936 URL: https://svnweb.freebsd.org/changeset/base/318936 Log: 4465 zpool(1M) is able to offline cache vdevs despite what man page says 5659 in the manual page for zpool(1M), one misuse of the word 'zpool' to describe a pool illumos/illumos-gate@c8323d4323a565676ba44883bfeb289d9ed8813e https://github.com/illumos/illumos-gate/commit/c8323d4323a565676ba44883bfeb289d9ed8813e https://www.illumos.org/issues/4465 zpool(1M) is able to offline cache vdevs despite man page saying that it isn't: zpool offline [-t] pool device ... Takes the specified physical device offline. While the device is offline, no attempt is made to read or write to the device. This command is not applicable to spares or cache devices. altair:root:~# zpool create testoff c9t67d0 cache c9t71d0 altair:root:~# zpool status testoff pool: testoff state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM testoff ONLINE 0 0 0 c9t67d0 ONLINE 0 0 0 cache c9t71d0 ONLINE 0 0 0 errors: No known data errors altair:root:~# zpool offline testoff c9t71d0 altair:root:~# zpool status testoff pool: testoff state: ONLINE status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: none requested https://www.illumos.org/issues/5659 At https://github.com/illumos/illumos-gate/blob/master/usr/src/man/man1m/ zpool.1m#L931 Do not add a disk that is currently configured as a quorum device to a zpool. – should be: Do not add a disk that is currently configured as a quorum device to a pool. Reviewed by: Matthew Ahrens Approved by: Hans Rosenfeld Author: Yuri Pankov Modified: vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:49:42 2017 (r318935) +++ vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:54:42 2017 (r318936) @@ -1,747 +1,535 @@ -'\" te +.\" +.\" CDDL HEADER START +.\" +.\" The contents of this file are subject to the terms of the +.\" Common Development and Distribution License (the "License"). +.\" You may not use this file except in compliance with the License. +.\" +.\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE +.\" or http://www.opensolaris.org/os/licensing. +.\" See the License for the specific language governing permissions +.\" and limitations under the License. +.\" +.\" When distributing Covered Code, include this CDDL HEADER in each +.\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. +.\" If applicable, add the following below this CDDL HEADER, with the +.\" fields enclosed by brackets "[]" replaced with your own identifying +.\" information: Portions Copyright [yyyy] [name of copyright owner] +.\" +.\" CDDL HEADER END +.\" +.\" .\" Copyright (c) 2007, Sun Microsystems, Inc. All Rights Reserved. -.\" Copyright 2011, Nexenta Systems, Inc. All Rights Reserved. .\" Copyright (c) 2013 by Delphix. All rights reserved. -.\" The contents of this file are subject to the terms of the Common Development -.\" and Distribution License (the "License"). You may not use this file except -.\" in compliance with the License. You can obtain a copy of the license at -.\" usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. +.\" Copyright 2016 Nexenta Systems, Inc. .\" -.\" See the License for the specific language governing permissions and -.\" limitations under the License. When distributing Covered Code, include this -.\" CDDL HEADER in each file and include the License file at -.\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this -.\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your -.\" own identifying information: -.\" Portions Copyright [yyyy] [name of copyright owner] -.TH ZPOOL 1M "Mar 6, 2014" -.SH NAME -zpool \- configures ZFS storage pools -.SH SYNOPSIS -.LP -.nf -\fBzpool\fR [\fB-?\fR] -.fi - -.LP -.nf -\fBzpool add\fR [\fB-fn\fR] \fIpool\fR \fIvdev\fR ... -.fi - -.LP -.nf -\fBzpool attach\fR [\fB-f\fR] \fIpool\fR \fIdevice\fR \fInew_device\fR -.fi - -.LP -.nf -\fBzpool clear\fR \fIpool\fR [\fIdevice\fR] -.fi - -.LP -.nf -\fBzpool create\fR [\fB-fnd\fR] [\fB-o\fR \fIproperty=value\fR] ... [\fB-O\fR \fIfile-system-property=value\fR] - ... [\fB-m\fR \fImountpoint\fR] [\fB-R\fR \fIroot\fR] \fIpool\fR \fIvdev\fR ... -.fi - -.LP -.nf -\fBzpool destroy\fR [\fB-f\fR] \fIpool\fR -.fi - -.LP -.nf -\fBzpool detach\fR \fIpool\fR \fIdevice\fR -.fi - -.LP -.nf -\fBzpool export\fR [\fB-f\fR] \fIpool\fR ... -.fi - -.LP -.nf -\fBzpool get\fR [\fB-Hp\fR] [\fB-o \fR\fIfield\fR[,...]] "\fIall\fR" | \fIproperty\fR[,...] \fIpool\fR ... -.fi - -.LP -.nf -\fBzpool history\fR [\fB-il\fR] [\fIpool\fR] ... -.fi - -.LP -.nf -\fBzpool import\fR [\fB-d\fR \fIdir\fR] [\fB-D\fR] -.fi - -.LP -.nf -\fBzpool import\fR [\fB-o \fImntopts\fR\fR] [\fB-o\fR \fIproperty=value\fR] ... [\fB-d\fR \fIdir\fR | \fB-c\fR \fIcachefile\fR] - [\fB-D\fR] [\fB-f\fR] [\fB-m\fR] [\fB-N\fR] [\fB-R\fR \fIroot\fR] [\fB-F\fR [\fB-n\fR]] \fB-a\fR -.fi - -.LP -.nf -\fBzpool import\fR [\fB-o \fImntopts\fR\fR] [\fB-o\fR \fIproperty=value\fR] ... [\fB-d\fR \fIdir\fR | \fB-c\fR \fIcachefile\fR] - [\fB-D\fR] [\fB-f\fR] [\fB-m\fR] [\fB-R\fR \fIroot\fR] [\fB-F\fR [\fB-n\fR]] \fIpool\fR |\fIid\fR [\fInewpool\fR] -.fi - -.LP -.nf -\fBzpool iostat\fR [\fB-T\fR \fBu\fR | \fBd\fR ] [\fB-v\fR] [\fIpool\fR] ... [\fIinterval\fR[\fIcount\fR]] -.fi - -.LP -.nf -\fBzpool list\fR [\fB-T\fR \fBu\fR | \fBd\fR ] [\fB-Hpv\fR] [\fB-o\fR \fIproperty\fR[,...]] [\fIpool\fR] ... [\fIinterval\fR[\fIcount\fR]] -.fi - -.LP -.nf -\fBzpool offline\fR [\fB-t\fR] \fIpool\fR \fIdevice\fR ... -.fi - -.LP -.nf -\fBzpool online\fR \fIpool\fR \fIdevice\fR ... -.fi - -.LP -.nf -\fBzpool reguid\fR \fIpool\fR -.fi - -.LP -.nf -\fBzpool reopen\fR \fIpool\fR -.fi - -.LP -.nf -\fBzpool remove\fR \fIpool\fR \fIdevice\fR ... -.fi - -.LP -.nf -\fBzpool replace\fR [\fB-f\fR] \fIpool\fR \fIdevice\fR [\fInew_device\fR] -.fi - -.LP -.nf -\fBzpool scrub\fR [\fB-s\fR] \fIpool\fR ... -.fi - -.LP -.nf -\fBzpool set\fR \fIproperty\fR=\fIvalue\fR \fIpool\fR -.fi - -.LP -.nf -\fBzpool status\fR [\fB-xvD\fR] [\fB-T\fR \fBu\fR | \fBd\fR ] [\fIpool\fR] ... [\fIinterval\fR [\fIcount\fR]] -.fi - -.LP -.nf -\fBzpool upgrade\fR -.fi - -.LP -.nf -\fBzpool upgrade\fR \fB-v\fR -.fi - -.LP -.nf -\fBzpool upgrade\fR [\fB-V\fR \fIversion\fR] \fB-a\fR | \fIpool\fR ... -.fi - -.SH DESCRIPTION -.sp -.LP -The \fBzpool\fR command configures \fBZFS\fR storage pools. A storage pool is a -collection of devices that provides physical storage and data replication for -\fBZFS\fR datasets. -.sp -.LP -All datasets within a storage pool share the same space. See \fBzfs\fR(1M) for -information on managing datasets. -.SS "Virtual Devices (\fBvdev\fRs)" -.sp -.LP +.Dd February 15, 2016 +.Dt ZPOOL 1M +.Os +.Sh NAME +.Nm zpool +.Nd configure ZFS storage pools +.Sh SYNOPSIS +.Nm +.Fl \? +.Nm +.Cm add +.Op Fl fn +.Ar pool vdev Ns ... +.Nm +.Cm attach +.Op Fl f +.Ar pool device new_device +.Nm +.Cm clear +.Ar pool +.Op Ar device +.Nm +.Cm create +.Op Fl dfn +.Op Fl m Ar mountpoint +.Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... +.Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ... +.Op Fl R Ar root +.Ar pool vdev Ns ... +.Nm +.Cm destroy +.Op Fl f +.Ar pool +.Nm +.Cm detach +.Ar pool device +.Nm +.Cm export +.Op Fl f +.Ar pool Ns ... +.Nm +.Cm get +.Op Fl Hp +.Op Fl o Ar field Ns Oo , Ns Ar field Oc Ns ... +.Sy all Ns | Ns Ar property Ns Oo , Ns Ar property Oc Ns ... +.Ar pool Ns ... +.Nm +.Cm history +.Op Fl il +.Oo Ar pool Oc Ns ... +.Nm +.Cm import +.Op Fl D +.Op Fl d Ar dir +.Nm +.Cm import +.Fl a +.Op Fl DfmN +.Op Fl F Op Fl n +.Op Fl c Ar cachefile Ns | Ns Fl d Ar dir +.Op Fl o Ar mntopts +.Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... +.Op Fl R Ar root +.Nm +.Cm import +.Op Fl Dfm +.Op Fl F Op Fl n +.Op Fl c Ar cachefile Ns | Ns Fl d Ar dir +.Op Fl o Ar mntopts +.Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... +.Op Fl R Ar root +.Ar pool Ns | Ns Ar id +.Op Ar newpool +.Nm +.Cm iostat +.Op Fl v +.Op Fl T Sy u Ns | Ns Sy d +.Oo Ar pool Oc Ns ... +.Op Ar interval Op Ar count +.Nm +.Cm list +.Op Fl Hpv +.Op Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... +.Op Fl T Sy u Ns | Ns Sy d +.Oo Ar pool Oc Ns ... +.Op Ar interval Op Ar count +.Nm +.Cm offline +.Op Fl t +.Ar pool Ar device Ns ... +.Nm +.Cm online +.Op Fl e +.Ar pool Ar device Ns ... +.Nm +.Cm reguid +.Ar pool +.Nm +.Cm reopen +.Ar pool +.Nm +.Cm remove +.Ar pool Ar device Ns ... +.Nm +.Cm replace +.Op Fl f +.Ar pool Ar device Op Ar new_device +.Nm +.Cm scrub +.Op Fl s +.Ar pool Ns ... +.Nm +.Cm set +.Ar property Ns = Ns Ar value +.Ar pool +.Nm +.Cm status +.Op Fl Dvx +.Op Fl T Sy u Ns | Ns Sy d +.Oo Ar pool Oc Ns ... +.Op Ar interval Op Ar count +.Nm +.Cm upgrade +.Nm +.Cm upgrade +.Fl v +.Nm +.Cm upgrade +.Op Fl V Ar version +.Fl a Ns | Ns Ar pool Ns ... +.Sh DESCRIPTION +The +.Nm +command configures ZFS storage pools. A storage pool is a collection of devices +that provides physical storage and data replication for ZFS datasets. All +datasets within a storage pool share the same space. See +.Xr zfs 1M +for information on managing datasets. +.Ss Virtual Devices (vdevs) A "virtual device" describes a single device or a collection of devices organized according to certain performance and fault characteristics. The following virtual devices are supported: -.sp -.ne 2 -.na -\fB\fBdisk\fR\fR -.ad -.RS 10n -A block device, typically located under \fB/dev/dsk\fR. \fBZFS\fR can use -individual slices or partitions, though the recommended mode of operation is to -use whole disks. A disk can be specified by a full path, or it can be a -shorthand name (the relative portion of the path under "/dev/dsk"). A whole -disk can be specified by omitting the slice or partition designation. For -example, "c0t0d0" is equivalent to "/dev/dsk/c0t0d0s2". When given a whole -disk, \fBZFS\fR automatically labels the disk, if necessary. -.RE - -.sp -.ne 2 -.na -\fB\fBfile\fR\fR -.ad -.RS 10n +.Bl -tag -width Ds +.It Sy disk +A block device, typically located under +.Pa /dev/dsk . +ZFS can use individual slices or partitions, though the recommended mode of +operation is to use whole disks. A disk can be specified by a full path, or it +can be a shorthand name +.Po the relative portion of the path under +.Pa /dev/dsk +.Pc . +A whole disk can be specified by omitting the slice or partition designation. +For example, +.Pa c0t0d0 +is equivalent to +.Pa /dev/dsk/c0t0d0s2 . +When given a whole disk, ZFS automatically labels the disk, if necessary. +.It Sy file A regular file. The use of files as a backing store is strongly discouraged. It is designed primarily for experimental purposes, as the fault tolerance of a file is only as good as the file system of which it is a part. A file must be specified by a full path. -.RE - -.sp -.ne 2 -.na -\fB\fBmirror\fR\fR -.ad -.RS 10n +.It Sy mirror A mirror of two or more devices. Data is replicated in an identical fashion -across all components of a mirror. A mirror with \fIN\fR disks of size \fIX\fR -can hold \fIX\fR bytes and can withstand (\fIN-1\fR) devices failing before -data integrity is compromised. -.RE - -.sp -.ne 2 -.na -\fB\fBraidz\fR\fR -.ad -.br -.na -\fB\fBraidz1\fR\fR -.ad -.br -.na -\fB\fBraidz2\fR\fR -.ad -.br -.na -\fB\fBraidz3\fR\fR -.ad -.RS 10n -A variation on \fBRAID-5\fR that allows for better distribution of parity and -eliminates the "\fBRAID-5\fR write hole" (in which data and parity become -inconsistent after a power loss). Data and parity is striped across all disks -within a \fBraidz\fR group. -.sp -A \fBraidz\fR group can have single-, double- , or triple parity, meaning that -the \fBraidz\fR group can sustain one, two, or three failures, respectively, -without losing any data. The \fBraidz1\fR \fBvdev\fR type specifies a -single-parity \fBraidz\fR group; the \fBraidz2\fR \fBvdev\fR type specifies a -double-parity \fBraidz\fR group; and the \fBraidz3\fR \fBvdev\fR type specifies -a triple-parity \fBraidz\fR group. The \fBraidz\fR \fBvdev\fR type is an alias -for \fBraidz1\fR. -.sp -A \fBraidz\fR group with \fIN\fR disks of size \fIX\fR with \fIP\fR parity -disks can hold approximately (\fIN-P\fR)*\fIX\fR bytes and can withstand -\fIP\fR device(s) failing before data integrity is compromised. The minimum -number of devices in a \fBraidz\fR group is one more than the number of parity -disks. The recommended number is between 3 and 9 to help increase performance. -.RE - -.sp -.ne 2 -.na -\fB\fBspare\fR\fR -.ad -.RS 10n -A special pseudo-\fBvdev\fR which keeps track of available hot spares for a -pool. For more information, see the "Hot Spares" section. -.RE - -.sp -.ne 2 -.na -\fB\fBlog\fR\fR -.ad -.RS 10n -A separate-intent log device. If more than one log device is specified, then +across all components of a mirror. A mirror with N disks of size X can hold X +bytes and can withstand (N-1) devices failing before data integrity is +compromised. +.It Sy raidz , raidz1 , raidz2 , raidz3 +A variation on RAID-5 that allows for better distribution of parity and +eliminates the RAID-5 +.Qq write hole +.Pq in which data and parity become inconsistent after a power loss . +Data and parity is striped across all disks within a raidz group. +.Pp +A raidz group can have single-, double-, or triple-parity, meaning that the +raidz group can sustain one, two, or three failures, respectively, without +losing any data. The +.Sy raidz1 +vdev type specifies a single-parity raidz group; the +.Sy raidz2 +vdev type specifies a double-parity raidz group; and the +.Sy raidz3 +vdev type specifies a triple-parity raidz group. The +.Sy raidz +vdev type is an alias for +.Sy raidz1 . +.Pp +A raidz group with N disks of size X with P parity disks can hold approximately +(N-P)*X bytes and can withstand P device(s) failing before data integrity is +compromised. The minimum number of devices in a raidz group is one more than +the number of parity disks. The recommended number is between 3 and 9 to help +increase performance. +.It Sy spare +A special pseudo-vdev which keeps track of available hot spares for a pool. For +more information, see the +.Sx Hot Spares +section. +.It Sy log +A separate intent log device. If more than one log device is specified, then writes are load-balanced between devices. Log devices can be mirrored. However, -\fBraidz\fR \fBvdev\fR types are not supported for the intent log. For more -information, see the "Intent Log" section. -.RE - -.sp -.ne 2 -.na -\fB\fBcache\fR\fR -.ad -.RS 10n +raidz vdev types are not supported for the intent log. For more information, +see the +.Sx Intent Log +section. +.It Sy cache A device used to cache storage pool data. A cache device cannot be cannot be -configured as a mirror or \fBraidz\fR group. For more information, see the -"Cache Devices" section. -.RE - -.sp -.LP -Virtual devices cannot be nested, so a mirror or \fBraidz\fR virtual device can -only contain files or disks. Mirrors of mirrors (or other combinations) are not -allowed. -.sp -.LP +configured as a mirror or raidz group. For more information, see the +.Sx Cache Devices +section. +.El +.Pp +Virtual devices cannot be nested, so a mirror or raidz virtual device can only +contain files or disks. Mirrors of mirrors +.Pq or other combinations +are not allowed. +.Pp A pool can have any number of virtual devices at the top of the configuration -(known as "root vdevs"). Data is dynamically distributed across all top-level -devices to balance data among devices. As new virtual devices are added, -\fBZFS\fR automatically places data on the newly available devices. -.sp -.LP +.Po known as +.Qq root vdevs +.Pc . +Data is dynamically distributed across all top-level devices to balance data +among devices. As new virtual devices are added, ZFS automatically places data +on the newly available devices. +.Pp Virtual devices are specified one at a time on the command line, separated by -whitespace. The keywords "mirror" and "raidz" are used to distinguish where a -group ends and another begins. For example, the following creates two root -vdevs, each a mirror of two disks: -.sp -.in +2 -.nf -# \fBzpool create mypool mirror c0t0d0 c0t1d0 mirror c1t0d0 c1t1d0\fR -.fi -.in -2 -.sp - -.SS "Device Failure and Recovery" -.sp -.LP -\fBZFS\fR supports a rich set of mechanisms for handling device failure and -data corruption. All metadata and data is checksummed, and \fBZFS\fR -automatically repairs bad data from a good copy when corruption is detected. -.sp -.LP +whitespace. The keywords +.Sy mirror +and +.Sy raidz +are used to distinguish where a group ends and another begins. For example, +the following creates two root vdevs, each a mirror of two disks: +.Bd -literal +# zpool create mypool mirror c0t0d0 c0t1d0 mirror c1t0d0 c1t1d0 +.Ed +.Ss Device Failure and Recovery +ZFS supports a rich set of mechanisms for handling device failure and data +corruption. All metadata and data is checksummed, and ZFS automatically repairs +bad data from a good copy when corruption is detected. +.Pp In order to take advantage of these features, a pool must make use of some form -of redundancy, using either mirrored or \fBraidz\fR groups. While \fBZFS\fR -supports running in a non-redundant configuration, where each root vdev is -simply a disk or file, this is strongly discouraged. A single case of bit -corruption can render some or all of your data unavailable. -.sp -.LP +of redundancy, using either mirrored or raidz groups. While ZFS supports +running in a non-redundant configuration, where each root vdev is simply a disk +or file, this is strongly discouraged. A single case of bit corruption can +render some or all of your data unavailable. +.Pp A pool's health status is described by one of three states: online, degraded, or faulted. An online pool has all devices operating normally. A degraded pool is one in which one or more devices have failed, but the data is still available due to a redundant configuration. A faulted pool has corrupted metadata, or one or more faulted devices, and insufficient replicas to continue functioning. -.sp -.LP -The health of the top-level vdev, such as mirror or \fBraidz\fR device, is +.Pp +The health of the top-level vdev, such as mirror or raidz device, is potentially impacted by the state of its associated vdevs, or component devices. A top-level vdev or component device is in one of the following states: -.sp -.ne 2 -.na -\fB\fBDEGRADED\fR\fR -.ad -.RS 12n +.Bl -tag -width "DEGRADED" +.It Sy DEGRADED One or more top-level vdevs is in the degraded state because one or more component devices are offline. Sufficient replicas exist to continue functioning. -.sp +.Pp One or more component devices is in the degraded or faulted state, but sufficient replicas exist to continue functioning. The underlying conditions are as follows: -.RS +4 -.TP -.ie t \(bu -.el o +.Bl -bullet +.It The number of checksum errors exceeds acceptable levels and the device is -degraded as an indication that something may be wrong. \fBZFS\fR continues to -use the device as necessary. -.RE -.RS +4 -.TP -.ie t \(bu -.el o +degraded as an indication that something may be wrong. ZFS continues to use the +device as necessary. +.It The number of I/O errors exceeds acceptable levels. The device could not be marked as faulted because there are insufficient replicas to continue functioning. -.RE -.RE - -.sp -.ne 2 -.na -\fB\fBFAULTED\fR\fR -.ad -.RS 12n +.El +.It Sy FAULTED One or more top-level vdevs is in the faulted state because one or more component devices are offline. Insufficient replicas exist to continue functioning. -.sp +.Pp One or more component devices is in the faulted state, and insufficient replicas exist to continue functioning. The underlying conditions are as follows: -.RS +4 -.TP -.ie t \(bu -.el o +.Bl -bullet +.It The device could be opened, but the contents did not match expected values. -.RE -.RS +4 -.TP -.ie t \(bu -.el o +.It The number of I/O errors exceeds acceptable levels and the device is faulted to prevent further use of the device. -.RE -.RE - -.sp -.ne 2 -.na -\fB\fBOFFLINE\fR\fR -.ad -.RS 12n -The device was explicitly taken offline by the "\fBzpool offline\fR" command. -.RE - -.sp -.ne 2 -.na -\fB\fBONLINE\fR\fR -.ad -.RS 12n +.El +.It Sy OFFLINE +The device was explicitly taken offline by the +.Nm zpool Cm offline +command. +.It Sy ONLINE The device is online and functioning. -.RE - -.sp -.ne 2 -.na -\fB\fBREMOVED\fR\fR -.ad -.RS 12n +.It Sy REMOVED The device was physically removed while the system was running. Device removal detection is hardware-dependent and may not be supported on all platforms. -.RE - -.sp -.ne 2 -.na -\fB\fBUNAVAIL\fR\fR -.ad -.RS 12n +.It Sy UNAVAIL The device could not be opened. If a pool is imported when a device was unavailable, then the device will be identified by a unique identifier instead of its path since the path was never correct in the first place. -.RE - -.sp -.LP -If a device is removed and later re-attached to the system, \fBZFS\fR attempts +.El +.Pp +If a device is removed and later re-attached to the system, ZFS attempts to put the device online automatically. Device attach detection is hardware-dependent and might not be supported on all platforms. -.SS "Hot Spares" -.sp -.LP -\fBZFS\fR allows devices to be associated with pools as "hot spares". These -devices are not actively used in the pool, but when an active device fails, it -is automatically replaced by a hot spare. To create a pool with hot spares, -specify a "spare" \fBvdev\fR with any number of devices. For example, -.sp -.in +2 -.nf +.Ss Hot Spares +ZFS allows devices to be associated with pools as +.Qq hot spares . +These devices are not actively used in the pool, but when an active device +fails, it is automatically replaced by a hot spare. To create a pool with hot +spares, specify a +.Sy spare +vdev with any number of devices. For example, +.Bd -literal # zpool create pool mirror c0d0 c1d0 spare c2d0 c3d0 -.fi -.in -2 -.sp - -.sp -.LP -Spares can be shared across multiple pools, and can be added with the "\fBzpool -add\fR" command and removed with the "\fBzpool remove\fR" command. Once a spare -replacement is initiated, a new "spare" \fBvdev\fR is created within the -configuration that will remain there until the original device is replaced. At -this point, the hot spare becomes available again if another device fails. -.sp -.LP +.Ed +.Pp +Spares can be shared across multiple pools, and can be added with the +.Nm zpool Cm add +command and removed with the +.Nm zpool Cm remove +command. Once a spare replacement is initiated, a new +.Sy spare +vdev is created within the configuration that will remain there until the +original device is replaced. At this point, the hot spare becomes available +again if another device fails. +.Pp If a pool has a shared spare that is currently being used, the pool can not be exported since other pools may use this shared spare, which may lead to potential data corruption. -.sp -.LP +.Pp An in-progress spare replacement can be cancelled by detaching the hot spare. If the original faulted device is detached, then the hot spare assumes its place in the configuration, and is removed from the spare list of all active pools. -.sp -.LP +.Pp Spares cannot replace log devices. -.SS "Intent Log" -.sp -.LP -The \fBZFS\fR Intent Log (\fBZIL\fR) satisfies \fBPOSIX\fR requirements for -synchronous transactions. For instance, databases often require their -transactions to be on stable storage devices when returning from a system call. -\fBNFS\fR and other applications can also use \fBfsync\fR() to ensure data -stability. By default, the intent log is allocated from blocks within the main -pool. However, it might be possible to get better performance using separate -intent log devices such as \fBNVRAM\fR or a dedicated disk. For example: -.sp -.in +2 -.nf -\fB# zpool create pool c0d0 c1d0 log c2d0\fR -.fi -.in -2 -.sp - -.sp -.LP +.Ss Intent Log +The ZFS Intent Log (ZIL) satisfies POSIX requirements for synchronous +transactions. For instance, databases often require their transactions to be on +stable storage devices when returning from a system call. NFS and other +applications can also use +.Xr fsync 3C +to ensure data stability. By default, the intent log is allocated from blocks +within the main pool. However, it might be possible to get better performance +using separate intent log devices such as NVRAM or a dedicated disk. For +example: +.Bd -literal +# zpool create pool c0d0 c1d0 log c2d0 +.Ed +.Pp Multiple log devices can also be specified, and they can be mirrored. See the -EXAMPLES section for an example of mirroring multiple log devices. -.sp -.LP +.Sx EXAMPLES +section for an example of mirroring multiple log devices. +.Pp Log devices can be added, replaced, attached, detached, and imported and exported as part of the larger pool. Mirrored log devices can be removed by specifying the top-level mirror for the log. -.SS "Cache Devices" -.sp -.LP -Devices can be added to a storage pool as "cache devices." These devices -provide an additional layer of caching between main memory and disk. For -read-heavy workloads, where the working set size is much larger than what can -be cached in main memory, using cache devices allow much more of this working -set to be served from low latency media. Using cache devices provides the -greatest performance improvement for random read-workloads of mostly static +.Ss Cache Devices +Devices can be added to a storage pool as +.Qq cache devices . +These devices provide an additional layer of caching between main memory and +disk. For read-heavy workloads, where the working set size is much larger than +what can be cached in main memory, using cache devices allow much more of this +working set to be served from low latency media. Using cache devices provides +the greatest performance improvement for random read-workloads of mostly static content. -.sp -.LP -To create a pool with cache devices, specify a "cache" \fBvdev\fR with any -number of devices. For example: -.sp -.in +2 -.nf -\fB# zpool create pool c0d0 c1d0 cache c2d0 c3d0\fR -.fi -.in -2 -.sp - -.sp -.LP -Cache devices cannot be mirrored or part of a \fBraidz\fR configuration. If a -read error is encountered on a cache device, that read \fBI/O\fR is reissued to -the original storage pool device, which might be part of a mirrored or -\fBraidz\fR configuration. -.sp -.LP +.Pp +To create a pool with cache devices, specify a +.Sy cache +vdev with any number of devices. For example: +.Bd -literal +# zpool create pool c0d0 c1d0 cache c2d0 c3d0 +.Ed +.Pp +Cache devices cannot be mirrored or part of a raidz configuration. If a read +error is encountered on a cache device, that read I/O is reissued to the +original storage pool device, which might be part of a mirrored or raidz +configuration. +.Pp The content of the cache devices is considered volatile, as is the case with other system caches. -.SS "Properties" -.sp -.LP +.Ss Properties Each pool has several properties associated with it. Some properties are read-only statistics while others are configurable and change the behavior of -the pool. The following are read-only properties: -.sp -.ne 2 -.na -\fB\fBavailable\fR\fR -.ad -.RS 20n +the pool. +.Pp +The following are read-only properties: +.Bl -tag -width Ds +.It Sy available Amount of storage available within the pool. This property can also be referred -to by its shortened column name, "avail". -.RE - -.sp -.ne 2 -.na -\fB\fBcapacity\fR\fR -.ad -.RS 20n +to by its shortened column name, +.Sy avail . +.It Sy capacity Percentage of pool space used. This property can also be referred to by its -shortened column name, "cap". -.RE - -.sp -.ne 2 -.na -\fB\fBexpandsize\fR\fR -.ad -.RS 20n +shortened column name, +.Sy cap . +.It Sy expandsize Amount of uninitialized space within the pool or device that can be used to increase the total capacity of the pool. Uninitialized space consists of any space on an EFI labeled vdev which has not been brought online -(i.e. zpool online -e). This space occurs when a LUN is dynamically expanded. -.RE - -.sp -.ne 2 -.na -\fB\fBfragmentation\fR\fR -.ad -.RS 20n +.Po e.g, using +.Nm zpool Cm online Fl e +.Pc . +This space occurs when a LUN is dynamically expanded. +.It Sy fragmentation The amount of fragmentation in the pool. -.RE - -.sp -.ne 2 -.na -\fB\fBfree\fR\fR -.ad -.RS 20n +.It Sy free The amount of free space available in the pool. -.RE - -.sp -.ne 2 -.na -\fB\fBfreeing\fR\fR -.ad -.RS 20n +.It Sy freeing After a file system or snapshot is destroyed, the space it was using is -returned to the pool asynchronously. \fB\fBfreeing\fR\fR is the amount of -space remaining to be reclaimed. Over time \fB\fBfreeing\fR\fR will decrease -while \fB\fBfree\fR\fR increases. -.RE - -.sp -.ne 2 -.na -\fB\fBhealth\fR\fR -.ad -.RS 20n -The current health of the pool. Health can be "\fBONLINE\fR", "\fBDEGRADED\fR", -"\fBFAULTED\fR", " \fBOFFLINE\fR", "\fBREMOVED\fR", or "\fBUNAVAIL\fR". -.RE - -.sp -.ne 2 -.na -\fB\fBguid\fR\fR -.ad -.RS 20n +returned to the pool asynchronously. +.Sy freeing +is the amount of space remaining to be reclaimed. Over time +.Sy freeing +will decrease while +.Sy free +increases. +.It Sy health +The current health of the pool. Health can be one of +.Sy ONLINE , DEGRADED , FAULTED , OFFLINE, REMOVED , UNAVAIL . +.It Sy guid A unique identifier for the pool. -.RE - -.sp -.ne 2 -.na *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Fri May 26 11:55:33 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EF71D83118; Fri, 26 May 2017 11:55:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 254AF1F10; Fri, 26 May 2017 11:55:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBtWJw089129; Fri, 26 May 2017 11:55:32 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBtWT0089128; Fri, 26 May 2017 11:55:32 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261155.v4QBtWT0089128@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:55:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318937 - vendor/illumos/dist/man/man1m X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:55:33 -0000 Author: avg Date: Fri May 26 11:55:31 2017 New Revision: 318937 URL: https://svnweb.freebsd.org/changeset/base/318937 Log: 2897 "zpool split" documentation missing from manpage illumos/illumos-gate@879bece34ee134b2be670393fe9ffe33ce268aa0 https://github.com/illumos/illumos-gate/commit/879bece34ee134b2be670393fe9ffe33ce268aa0 https://www.illumos.org/issues/2897 Found this option in some Oracle documentation and wanted to check out the zpool manpage on it in OI. Unfortunately it seems to be missing from the manpage, so I first thought it was unsupported. However, "# zpool split" does print the correct usage. Testing with the "-n" switch makes me believe it is supported (I don't actually need to split my pool). Reviewed by: Dan McDonald Reviewed by: Matthew Ahrens Approved by: Hans Rosenfeld Author: Steven Burgess Modified: vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:54:42 2017 (r318936) +++ vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:55:31 2017 (r318937) @@ -139,6 +139,12 @@ .Ar property Ns = Ns Ar value .Ar pool .Nm +.Cm split +.Op Fl n +.Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... +.Op Fl R Ar root +.Ar pool newpool +.Nm .Cm status .Op Fl Dvx .Op Fl T Sy u Ns | Ns Sy d @@ -1327,6 +1333,44 @@ section for more information on what pro values. .It Xo .Nm +.Cm split +.Op Fl n +.Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... +.Op Fl R Ar root +.Ar pool newpool +.Xc +Splits devices off +.Ar pool +creating +.Ar newpool . +All vdevs in +.Ar pool +must be mirrors. At the time of the split, +.Ar newpool +will be a replica of +.Ar pool . +.Bl -tag -width Ds +.It Fl n +Do dry run, do not actually perform the split. Print out the expected +configuration of +.Ar newpool . +.It Fl o Ar property Ns = Ns Ar value +Sets the specified property for +.Ar newpool . +See the +.Sx Properties +section for more information on the available pool properties. +.It Fl R Ar root +Set +.Sy altroot +for +.Ar newpool +to +.Ar root +and automaticaly import it. +.El +.It Xo +.Nm .Cm status .Op Fl Dvx .Op Fl T Sy u Ns | Ns Sy d From owner-svn-src-vendor@freebsd.org Fri May 26 11:56:30 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C905D83197; Fri, 26 May 2017 11:56:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14B641070; Fri, 26 May 2017 11:56:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBuTMo089209; Fri, 26 May 2017 11:56:29 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBuT6u089208; Fri, 26 May 2017 11:56:29 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261156.v4QBuT6u089208@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:56:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318938 - vendor/illumos/dist/man/man1m X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:56:30 -0000 Author: avg Date: Fri May 26 11:56:28 2017 New Revision: 318938 URL: https://svnweb.freebsd.org/changeset/base/318938 Log: 6781 zpool man page needs updated to remove duplicate entry of "cannot be" where it discusses cache devices illumos/illumos-gate@e4cb59f791c5c6dea7ad4caa6dee845024ce71ef https://github.com/illumos/illumos-gate/commit/e4cb59f791c5c6dea7ad4caa6dee845024ce71ef https://www.illumos.org/issues/6781 cache A device used to cache storage pool data. A cache device cannot be cannot be configured as a mirror or raidz group. For more information, see the "Cache Devices" section. needs changed to cache A device used to cache storage pool data. A cache device cannot be configured as a mirror or raidz group. For more information, see the "Cache Devices" section. Reviewed by: Toomas Soome Reviewed by: Robert Mustacchi Approved by: Matthew Ahrens Author: Alexander Pyhalov Modified: vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:55:31 2017 (r318937) +++ vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:56:28 2017 (r318938) @@ -23,7 +23,7 @@ .\" Copyright (c) 2013 by Delphix. All rights reserved. .\" Copyright 2016 Nexenta Systems, Inc. .\" -.Dd February 15, 2016 +.Dd March 25, 2016 .Dt ZPOOL 1M .Os .Sh NAME @@ -235,8 +235,8 @@ see the .Sx Intent Log section. .It Sy cache -A device used to cache storage pool data. A cache device cannot be cannot be -configured as a mirror or raidz group. For more information, see the +A device used to cache storage pool data. A cache device cannot be configured +as a mirror or raidz group. For more information, see the .Sx Cache Devices section. .El From owner-svn-src-vendor@freebsd.org Fri May 26 11:59:21 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2C7BD83223; Fri, 26 May 2017 11:59:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C4711239; Fri, 26 May 2017 11:59:21 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QBxKMQ089353; Fri, 26 May 2017 11:59:20 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QBxKRO089352; Fri, 26 May 2017 11:59:20 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261159.v4QBxKRO089352@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 11:59:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318939 - vendor/illumos/dist/man/man1m X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 11:59:21 -0000 Author: avg Date: Fri May 26 11:59:20 2017 New Revision: 318939 URL: https://svnweb.freebsd.org/changeset/base/318939 Log: 6418 zpool should have a label clearing command illumos/illumos-gate@6401734d545a04c18f68b448202f9d9a77216bb9 https://github.com/illumos/illumos-gate/commit/6401734d545a04c18f68b448202f9d9a77216bb9 https://www.illumos.org/issues/6418 An easy, direct means of sanitizing pool vdevs can be helpful for management purposes. FreeBSD has had a 'zpool labelclear' for some time, see: https:// svnweb.freebsd.org/base?view=revision&revision=224171 SpectraBSD has a slightly updated version, which I propose for inclusion. Reviewed by: Yuri Pankov Reviewed by: Matthew Ahrens Reviewed by: George Wilson Approved by: Robert Mustacchi Author: Will Andrews Note: the bulk of the change has been already imported, this is a follow up that imports zpool.1m changes. Modified: vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:56:28 2017 (r318938) +++ vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:59:20 2017 (r318939) @@ -23,7 +23,7 @@ .\" Copyright (c) 2013 by Delphix. All rights reserved. .\" Copyright 2016 Nexenta Systems, Inc. .\" -.Dd March 25, 2016 +.Dd March 30, 2016 .Dt ZPOOL 1M .Os .Sh NAME @@ -103,6 +103,10 @@ .Oo Ar pool Oc Ns ... .Op Ar interval Op Ar count .Nm +.Cm labelclear +.Op Fl f +.Ar device +.Nm .Cm list .Op Fl Hpv .Op Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... @@ -1154,6 +1158,21 @@ pool, in addition to the pool-wide stati .El .It Xo .Nm +.Cm labelclear +.Op Fl f +.Ar device +.Xc +Removes ZFS label information from the specified +.Ar device . +The +.Ar device +must not be part of an active pool configuration. +.Bl -tag -width Ds +.It Fl f +Treat exported or foreign devices as inactive. +.El +.It Xo +.Nm .Cm list .Op Fl Hpv .Op Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... From owner-svn-src-vendor@freebsd.org Fri May 26 12:00:32 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3E04D83722; Fri, 26 May 2017 12:00:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 85F94166A; Fri, 26 May 2017 12:00:32 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QC0V4T089488; Fri, 26 May 2017 12:00:31 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QC0VEs089487; Fri, 26 May 2017 12:00:31 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261200.v4QC0VEs089487@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:00:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318940 - vendor/illumos/dist/man/man1m X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:00:32 -0000 Author: avg Date: Fri May 26 12:00:31 2017 New Revision: 318940 URL: https://svnweb.freebsd.org/changeset/base/318940 Log: 7956 "minimum" is misspelled in zpool manpage illumos/illumos-gate@eba8726136851528470dad9bad27892859bc1e3e https://github.com/illumos/illumos-gate/commit/eba8726136851528470dad9bad27892859bc1e3e https://www.illumos.org/issues/7956 Reviewed by: Prashanth Sreenivasa Reviewed by: Matthew Ahrens Reviewed by: Sebastien Roy Reviewed by: Yuri Pankov Approved by: Richard Lowe Author: Brad Lewis Modified: vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 11:59:20 2017 (r318939) +++ vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 12:00:31 2017 (r318940) @@ -569,7 +569,7 @@ Threshold for the number of block ditto deduplicated block increases above this number, a new ditto copy of this block is automatically stored. The default setting is .Sy 0 -which causes no ditto copies to be created for deduplicated blocks. The miniumum +which causes no ditto copies to be created for deduplicated blocks. The minimum legal nonzero setting is .Sy 100 . .It Sy delegation Ns = Ns Sy on Ns | Ns Sy off From owner-svn-src-vendor@freebsd.org Fri May 26 12:02:16 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2AB5ED83A49; Fri, 26 May 2017 12:02:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 07E5A1AA7; Fri, 26 May 2017 12:02:15 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QC2Fb3093085; Fri, 26 May 2017 12:02:15 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QC2Eaa093079; Fri, 26 May 2017 12:02:14 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261202.v4QC2Eaa093079@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:02:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318941 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:02:16 -0000 Author: avg Date: Fri May 26 12:02:14 2017 New Revision: 318941 URL: https://svnweb.freebsd.org/changeset/base/318941 Log: 7446 zpool create should support efi system partition illumos/illumos-gate@7855d95b30fd903e3918bad5a29b777e765db821 https://github.com/illumos/illumos-gate/commit/7855d95b30fd903e3918bad5a29b777e765db821 https://www.illumos.org/issues/7446 Since we support whole-disk configuration for boot pool, we also will need whole disk support with UEFI boot and for this, zpool create should create efi- system partition. I have borrowed the idea from oracle solaris, and introducing zpool create - B switch to provide an way to specify that boot partition should be created. However, there is still an question, how big should the system partition be. For time being, I have set default size 256MB (thats minimum size for FAT32 with 4k blocks). To support custom size, the set on creation "bootsize" property is created and so the custom size can be set as: zpool create B - o bootsize=34MB rpool c0t0d0 After pool is created, the "bootsize" property is read only. When -B switch is not used, the bootsize defaults to 0 and is shown in zpool get output with value ''. Older zfs/zpool implementations are ignoring this property. https://www.illumos.org/rb/r/219/ Reviewed by: Andrew Stormont Reviewed by: Yuri Pankov Approved by: Dan McDonald Author: Toomas Soome Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/zpool/zpool_util.h vendor/illumos/dist/cmd/zpool/zpool_vdev.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/man/man1m/zpool.1m Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zpool_prop.c Fri May 26 12:00:31 2017 (r318940) +++ vendor-sys/illumos/dist/common/zfs/zpool_prop.c Fri May 26 12:02:14 2017 (r318941) @@ -100,6 +100,10 @@ zpool_prop_init(void) PROP_READONLY, ZFS_TYPE_POOL, "<1.00x or higher if deduped>", "DEDUP"); + /* system partition size */ + zprop_register_number(ZPOOL_PROP_BOOTSIZE, "bootsize", 0, PROP_ONETIME, + ZFS_TYPE_POOL, "", "BOOTSIZE"); + /* default number properties */ zprop_register_number(ZPOOL_PROP_VERSION, "version", SPA_VERSION, PROP_DEFAULT, ZFS_TYPE_POOL, "", "VERSION"); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Fri May 26 12:00:31 2017 (r318940) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c Fri May 26 12:02:14 2017 (r318941) @@ -394,6 +394,7 @@ metaslab_class_expandable_space(metaslab spa_config_enter(mc->mc_spa, SCL_VDEV, FTAG, RW_READER); for (int c = 0; c < rvd->vdev_children; c++) { + uint64_t tspace; vdev_t *tvd = rvd->vdev_child[c]; metaslab_group_t *mg = tvd->vdev_mg; @@ -406,9 +407,13 @@ metaslab_class_expandable_space(metaslab * Calculate if we have enough space to add additional * metaslabs. We report the expandable space in terms * of the metaslab size since that's the unit of expansion. + * Adjust by efi system partition size. */ - space += P2ALIGN(tvd->vdev_max_asize - tvd->vdev_asize, - 1ULL << tvd->vdev_ms_shift); + tspace = tvd->vdev_max_asize - tvd->vdev_asize; + if (tspace > mc->mc_spa->spa_bootsize) { + tspace -= mc->mc_spa->spa_bootsize; + } + space += P2ALIGN(tspace, 1ULL << tvd->vdev_ms_shift); } spa_config_exit(mc->mc_spa, SCL_VDEV, FTAG); return (space); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Fri May 26 12:00:31 2017 (r318940) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c Fri May 26 12:02:14 2017 (r318941) @@ -2738,6 +2738,7 @@ spa_load_impl(spa_t *spa, uint64_t pool_ spa_prop_find(spa, ZPOOL_PROP_DELEGATION, &spa->spa_delegation); spa_prop_find(spa, ZPOOL_PROP_FAILUREMODE, &spa->spa_failmode); spa_prop_find(spa, ZPOOL_PROP_AUTOEXPAND, &spa->spa_autoexpand); + spa_prop_find(spa, ZPOOL_PROP_BOOTSIZE, &spa->spa_bootsize); spa_prop_find(spa, ZPOOL_PROP_DEDUPDITTO, &spa->spa_dedup_ditto); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Fri May 26 12:00:31 2017 (r318940) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h Fri May 26 12:02:14 2017 (r318941) @@ -237,6 +237,7 @@ struct spa { int spa_mode; /* FREAD | FWRITE */ spa_log_state_t spa_log_state; /* log state */ uint64_t spa_autoexpand; /* lun expansion on/off */ + uint64_t spa_bootsize; /* efi system partition size */ ddt_t *spa_ddt[ZIO_CHECKSUM_FUNCTIONS]; /* in-core DDTs */ uint64_t spa_ddt_stat_object; /* DDT statistics */ uint64_t spa_dedup_ditto; /* dedup ditto threshold */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 12:00:31 2017 (r318940) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 12:02:14 2017 (r318941) @@ -2765,8 +2765,8 @@ vdev_get_stats(vdev_t *vd, vdev_stat_t * * since that determines how much space the pool can expand. */ if (vd->vdev_aux == NULL && tvd != NULL) { - vs->vs_esize = P2ALIGN(vd->vdev_max_asize - vd->vdev_asize, - 1ULL << tvd->vdev_ms_shift); + vs->vs_esize = P2ALIGN(vd->vdev_max_asize - vd->vdev_asize - + spa->spa_bootsize, 1ULL << tvd->vdev_ms_shift); } if (vd->vdev_aux == NULL && vd == vd->vdev_top && !vd->vdev_ishole) { vs->vs_fragmentation = vd->vdev_mg->mg_fragmentation; Modified: vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Fri May 26 12:00:31 2017 (r318940) +++ vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Fri May 26 12:02:14 2017 (r318941) @@ -203,6 +203,7 @@ typedef enum { ZPOOL_PROP_FRAGMENTATION, ZPOOL_PROP_LEAKED, ZPOOL_PROP_MAXBLOCKSIZE, + ZPOOL_PROP_BOOTSIZE, ZPOOL_NUM_PROPS } zpool_prop_t; From owner-svn-src-vendor@freebsd.org Fri May 26 12:02:16 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB35AD83A4E; Fri, 26 May 2017 12:02:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AB4A21AAB; Fri, 26 May 2017 12:02:16 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QC2FCa093096; Fri, 26 May 2017 12:02:15 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QC2Fq2093090; Fri, 26 May 2017 12:02:15 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261202.v4QC2Fq2093090@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:02:15 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318941 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor-sys/illumos/dist/uts/common/sys/fs vendor/ill... X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:02:17 -0000 Author: avg Date: Fri May 26 12:02:14 2017 New Revision: 318941 URL: https://svnweb.freebsd.org/changeset/base/318941 Log: 7446 zpool create should support efi system partition illumos/illumos-gate@7855d95b30fd903e3918bad5a29b777e765db821 https://github.com/illumos/illumos-gate/commit/7855d95b30fd903e3918bad5a29b777e765db821 https://www.illumos.org/issues/7446 Since we support whole-disk configuration for boot pool, we also will need whole disk support with UEFI boot and for this, zpool create should create efi- system partition. I have borrowed the idea from oracle solaris, and introducing zpool create - B switch to provide an way to specify that boot partition should be created. However, there is still an question, how big should the system partition be. For time being, I have set default size 256MB (thats minimum size for FAT32 with 4k blocks). To support custom size, the set on creation "bootsize" property is created and so the custom size can be set as: zpool create B - o bootsize=34MB rpool c0t0d0 After pool is created, the "bootsize" property is read only. When -B switch is not used, the bootsize defaults to 0 and is shown in zpool get output with value ''. Older zfs/zpool implementations are ignoring this property. https://www.illumos.org/rb/r/219/ Reviewed by: Andrew Stormont Reviewed by: Yuri Pankov Approved by: Dan McDonald Author: Toomas Soome Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c vendor/illumos/dist/cmd/zpool/zpool_util.h vendor/illumos/dist/cmd/zpool/zpool_vdev.c vendor/illumos/dist/lib/libzfs/common/libzfs.h vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c vendor/illumos/dist/man/man1m/zpool.1m Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/common/zfs/zpool_prop.c vendor-sys/illumos/dist/uts/common/fs/zfs/metaslab.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/sys/fs/zfs.h Modified: vendor/illumos/dist/cmd/zpool/zpool_main.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_main.c Fri May 26 12:00:31 2017 (r318940) +++ vendor/illumos/dist/cmd/zpool/zpool_main.c Fri May 26 12:02:14 2017 (r318941) @@ -208,7 +208,8 @@ get_usage(zpool_help_t idx) case HELP_CLEAR: return (gettext("\tclear [-nF] [device]\n")); case HELP_CREATE: - return (gettext("\tcreate [-fnd] [-o property=value] ... \n" + return (gettext("\tcreate [-fnd] [-B] " + "[-o property=value] ... \n" "\t [-O file-system-property=value] ... \n" "\t [-m mountpoint] [-R root] ...\n")); case HELP_DESTROY: @@ -496,6 +497,8 @@ zpool_do_add(int argc, char **argv) int c; nvlist_t *nvroot; char *poolname; + zpool_boot_label_t boot_type; + uint64_t boot_size; int ret; zpool_handle_t *zhp; nvlist_t *config; @@ -544,9 +547,15 @@ zpool_do_add(int argc, char **argv) return (1); } + if (zpool_is_bootable(zhp)) + boot_type = ZPOOL_COPY_BOOT_LABEL; + else + boot_type = ZPOOL_NO_BOOT_LABEL; + /* pass off to get_vdev_spec for processing */ + boot_size = zpool_get_prop_int(zhp, ZPOOL_PROP_BOOTSIZE, NULL); nvroot = make_root_vdev(zhp, force, !force, B_FALSE, dryrun, - argc, argv); + boot_type, boot_size, argc, argv); if (nvroot == NULL) { zpool_close(zhp); return (1); @@ -767,10 +776,11 @@ errout: } /* - * zpool create [-fnd] [-o property=value] ... + * zpool create [-fnd] [-B] [-o property=value] ... * [-O file-system-property=value] ... * [-R root] [-m mountpoint] ... * + * -B Create boot partition. * -f Force creation, even if devices appear in use * -n Do not create the pool, but display the resulting layout if it * were to be created. @@ -787,12 +797,16 @@ errout: * we get the nvlist back from get_vdev_spec(), we either print out the contents * (if '-n' was specified), or pass it to libzfs to do the creation. */ + +#define SYSTEM256 (256 * 1024 * 1024) int zpool_do_create(int argc, char **argv) { boolean_t force = B_FALSE; boolean_t dryrun = B_FALSE; boolean_t enable_all_pool_feat = B_TRUE; + zpool_boot_label_t boot_type = ZPOOL_NO_BOOT_LABEL; + uint64_t boot_size = 0; int c; nvlist_t *nvroot = NULL; char *poolname; @@ -804,7 +818,7 @@ zpool_do_create(int argc, char **argv) char *propval; /* check options */ - while ((c = getopt(argc, argv, ":fndR:m:o:O:")) != -1) { + while ((c = getopt(argc, argv, ":fndBR:m:o:O:")) != -1) { switch (c) { case 'f': force = B_TRUE; @@ -815,6 +829,15 @@ zpool_do_create(int argc, char **argv) case 'd': enable_all_pool_feat = B_FALSE; break; + case 'B': + /* + * We should create the system partition. + * Also make sure the size is set. + */ + boot_type = ZPOOL_CREATE_BOOT_LABEL; + if (boot_size == 0) + boot_size = SYSTEM256; + break; case 'R': altroot = optarg; if (add_prop_list(zpool_prop_to_name( @@ -845,6 +868,20 @@ zpool_do_create(int argc, char **argv) goto errout; /* + * Get bootsize value for make_root_vdev(). + */ + if (zpool_name_to_prop(optarg) == ZPOOL_PROP_BOOTSIZE) { + if (zfs_nicestrtonum(g_zfs, propval, + &boot_size) < 0 || boot_size == 0) { + (void) fprintf(stderr, + gettext("bad boot partition size " + "'%s': %s\n"), propval, + libzfs_error_description(g_zfs)); + goto errout; + } + } + + /* * If the user is creating a pool that doesn't support * feature flags, don't enable any features. */ @@ -921,9 +958,43 @@ zpool_do_create(int argc, char **argv) goto errout; } + /* + * Make sure the bootsize is set when ZPOOL_CREATE_BOOT_LABEL is used, + * and not set otherwise. + */ + if (boot_type == ZPOOL_CREATE_BOOT_LABEL) { + const char *propname; + char *strptr, *buf = NULL; + int rv; + + propname = zpool_prop_to_name(ZPOOL_PROP_BOOTSIZE); + if (nvlist_lookup_string(props, propname, &strptr) != 0) { + (void) asprintf(&buf, "%" PRIu64, boot_size); + if (buf == NULL) { + (void) fprintf(stderr, + gettext("internal error: out of memory\n")); + goto errout; + } + rv = add_prop_list(propname, buf, &props, B_TRUE); + free(buf); + if (rv != 0) + goto errout; + } + } else { + const char *propname; + char *strptr; + + propname = zpool_prop_to_name(ZPOOL_PROP_BOOTSIZE); + if (nvlist_lookup_string(props, propname, &strptr) == 0) { + (void) fprintf(stderr, gettext("error: setting boot " + "partition size requires option '-B'\n")); + goto errout; + } + } + /* pass off to get_vdev_spec for bulk processing */ nvroot = make_root_vdev(NULL, force, !force, B_FALSE, dryrun, - argc - 1, argv + 1); + boot_type, boot_size, argc - 1, argv + 1); if (nvroot == NULL) goto errout; @@ -3183,6 +3254,8 @@ zpool_do_attach_or_replace(int argc, cha nvlist_t *nvroot; char *poolname, *old_disk, *new_disk; zpool_handle_t *zhp; + zpool_boot_label_t boot_type; + uint64_t boot_size; int ret; /* check options */ @@ -3247,8 +3320,14 @@ zpool_do_attach_or_replace(int argc, cha return (1); } + if (zpool_is_bootable(zhp)) + boot_type = ZPOOL_COPY_BOOT_LABEL; + else + boot_type = ZPOOL_NO_BOOT_LABEL; + + boot_size = zpool_get_prop_int(zhp, ZPOOL_PROP_BOOTSIZE, NULL); nvroot = make_root_vdev(zhp, force, B_FALSE, replacing, B_FALSE, - argc, argv); + boot_type, boot_size, argc, argv); if (nvroot == NULL) { zpool_close(zhp); return (1); Modified: vendor/illumos/dist/cmd/zpool/zpool_util.h ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_util.h Fri May 26 12:00:31 2017 (r318940) +++ vendor/illumos/dist/cmd/zpool/zpool_util.h Fri May 26 12:02:14 2017 (r318941) @@ -44,7 +44,8 @@ uint_t num_logs(nvlist_t *nv); */ nvlist_t *make_root_vdev(zpool_handle_t *zhp, int force, int check_rep, - boolean_t replacing, boolean_t dryrun, int argc, char **argv); + boolean_t replacing, boolean_t dryrun, zpool_boot_label_t boot_type, + uint64_t boot_size, int argc, char **argv); nvlist_t *split_mirror_vdev(zpool_handle_t *zhp, char *newname, nvlist_t *props, splitflags_t flags, int argc, char **argv); Modified: vendor/illumos/dist/cmd/zpool/zpool_vdev.c ============================================================================== --- vendor/illumos/dist/cmd/zpool/zpool_vdev.c Fri May 26 12:00:31 2017 (r318940) +++ vendor/illumos/dist/cmd/zpool/zpool_vdev.c Fri May 26 12:02:14 2017 (r318941) @@ -876,14 +876,15 @@ check_replication(nvlist_t *config, nvli * Go through and find any whole disks in the vdev specification, labelling them * as appropriate. When constructing the vdev spec, we were unable to open this * device in order to provide a devid. Now that we have labelled the disk and - * know that slice 0 is valid, we can construct the devid now. + * know the pool slice is valid, we can construct the devid now. * * If the disk was already labeled with an EFI label, we will have gotten the * devid already (because we were able to open the whole disk). Otherwise, we * need to get the devid after we label the disk. */ static int -make_disks(zpool_handle_t *zhp, nvlist_t *nv) +make_disks(zpool_handle_t *zhp, nvlist_t *nv, zpool_boot_label_t boot_type, + uint64_t boot_size) { nvlist_t **child; uint_t c, children; @@ -892,6 +893,7 @@ make_disks(zpool_handle_t *zhp, nvlist_t uint64_t wholedisk; int fd; int ret; + int slice; ddi_devid_t devid; char *minor = NULL, *devid_str = NULL; @@ -909,20 +911,36 @@ make_disks(zpool_handle_t *zhp, nvlist_t * slice and stat()ing the device. */ verify(nvlist_lookup_string(nv, ZPOOL_CONFIG_PATH, &path) == 0); - if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_WHOLE_DISK, - &wholedisk) != 0 || !wholedisk) - return (0); diskname = strrchr(path, '/'); assert(diskname != NULL); diskname++; - if (zpool_label_disk(g_zfs, zhp, diskname) == -1) - return (-1); + + if (nvlist_lookup_uint64(nv, ZPOOL_CONFIG_WHOLE_DISK, + &wholedisk) != 0 || !wholedisk) { + /* + * This is not whole disk, return error if + * boot partition creation was requested + */ + if (boot_type == ZPOOL_CREATE_BOOT_LABEL) { + (void) fprintf(stderr, + gettext("creating boot partition is only " + "supported on whole disk vdevs: %s\n"), + diskname); + return (-1); + } + return (0); + } + + ret = zpool_label_disk(g_zfs, zhp, diskname, boot_type, + boot_size, &slice); + if (ret == -1) + return (ret); /* * Fill in the devid, now that we've labeled the disk. */ - (void) snprintf(buf, sizeof (buf), "%ss0", path); + (void) snprintf(buf, sizeof (buf), "%ss%d", path, slice); if ((fd = open(buf, O_RDONLY)) < 0) { (void) fprintf(stderr, gettext("cannot open '%s': %s\n"), @@ -945,7 +963,7 @@ make_disks(zpool_handle_t *zhp, nvlist_t } /* - * Update the path to refer to the 's0' slice. The presence of + * Update the path to refer to the pool slice. The presence of * the 'whole_disk' field indicates to the CLI that we should * chop off the slice number when displaying the device in * future output. @@ -957,21 +975,36 @@ make_disks(zpool_handle_t *zhp, nvlist_t return (0); } - for (c = 0; c < children; c++) - if ((ret = make_disks(zhp, child[c])) != 0) + /* illumos kernel does not support booting from multi-vdev pools. */ + if ((boot_type == ZPOOL_CREATE_BOOT_LABEL)) { + if ((strcmp(type, VDEV_TYPE_ROOT) == 0) && children > 1) { + (void) fprintf(stderr, gettext("boot pool " + "can not have more than one vdev\n")); + return (-1); + } + } + + for (c = 0; c < children; c++) { + ret = make_disks(zhp, child[c], boot_type, boot_size); + if (ret != 0) return (ret); + } if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_SPARES, &child, &children) == 0) - for (c = 0; c < children; c++) - if ((ret = make_disks(zhp, child[c])) != 0) + for (c = 0; c < children; c++) { + ret = make_disks(zhp, child[c], boot_type, boot_size); + if (ret != 0) return (ret); + } if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_L2CACHE, &child, &children) == 0) - for (c = 0; c < children; c++) - if ((ret = make_disks(zhp, child[c])) != 0) + for (c = 0; c < children; c++) { + ret = make_disks(zhp, child[c], boot_type, boot_size); + if (ret != 0) return (ret); + } return (0); } @@ -1367,6 +1400,7 @@ split_mirror_vdev(zpool_handle_t *zhp, c { nvlist_t *newroot = NULL, **child; uint_t c, children; + zpool_boot_label_t boot_type; if (argc > 0) { if ((newroot = construct_spec(argc, argv)) == NULL) { @@ -1375,7 +1409,13 @@ split_mirror_vdev(zpool_handle_t *zhp, c return (NULL); } - if (!flags.dryrun && make_disks(zhp, newroot) != 0) { + if (zpool_is_bootable(zhp)) + boot_type = ZPOOL_COPY_BOOT_LABEL; + else + boot_type = ZPOOL_NO_BOOT_LABEL; + + if (!flags.dryrun && + make_disks(zhp, newroot, boot_type, 0) != 0) { nvlist_free(newroot); return (NULL); } @@ -1419,7 +1459,8 @@ split_mirror_vdev(zpool_handle_t *zhp, c */ nvlist_t * make_root_vdev(zpool_handle_t *zhp, int force, int check_rep, - boolean_t replacing, boolean_t dryrun, int argc, char **argv) + boolean_t replacing, boolean_t dryrun, zpool_boot_label_t boot_type, + uint64_t boot_size, int argc, char **argv) { nvlist_t *newroot; nvlist_t *poolconfig = NULL; @@ -1460,7 +1501,7 @@ make_root_vdev(zpool_handle_t *zhp, int /* * Run through the vdev specification and label any whole disks found. */ - if (!dryrun && make_disks(zhp, newroot) != 0) { + if (!dryrun && make_disks(zhp, newroot, boot_type, boot_size) != 0) { nvlist_free(newroot); return (NULL); } Modified: vendor/illumos/dist/lib/libzfs/common/libzfs.h ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs.h Fri May 26 12:00:31 2017 (r318940) +++ vendor/illumos/dist/lib/libzfs/common/libzfs.h Fri May 26 12:02:14 2017 (r318941) @@ -132,6 +132,18 @@ typedef enum zfs_error { } zfs_error_t; /* + * UEFI boot support parameters. When creating whole disk boot pool, + * zpool create should allow to create EFI System partition for UEFI boot + * program. In case of BIOS, the EFI System partition is not used + * even if it does exist. + */ +typedef enum zpool_boot_label { + ZPOOL_NO_BOOT_LABEL = 0, + ZPOOL_CREATE_BOOT_LABEL, + ZPOOL_COPY_BOOT_LABEL +} zpool_boot_label_t; + +/* * The following data structures are all part * of the zfs_allow_t data structure which is * used for printing 'allow' permissions. @@ -262,7 +274,8 @@ extern nvlist_t *zpool_find_vdev(zpool_h boolean_t *, boolean_t *); extern nvlist_t *zpool_find_vdev_by_physpath(zpool_handle_t *, const char *, boolean_t *, boolean_t *, boolean_t *); -extern int zpool_label_disk(libzfs_handle_t *, zpool_handle_t *, const char *); +extern int zpool_label_disk(libzfs_handle_t *, zpool_handle_t *, const char *, + zpool_boot_label_t, uint64_t, int *); /* * Functions to manage pool properties @@ -344,6 +357,7 @@ extern nvlist_t *zpool_get_config(zpool_ extern nvlist_t *zpool_get_features(zpool_handle_t *); extern int zpool_refresh_stats(zpool_handle_t *, boolean_t *); extern int zpool_get_errlog(zpool_handle_t *, nvlist_t **); +extern boolean_t zpool_is_bootable(zpool_handle_t *); /* * Import and export functions Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Fri May 26 12:00:31 2017 (r318940) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_pool.c Fri May 26 12:02:14 2017 (r318941) @@ -48,7 +48,7 @@ #include "zfs_comutil.h" #include "zfeature_common.h" -static int read_efi_label(nvlist_t *config, diskaddr_t *sb); +static int read_efi_label(nvlist_t *, diskaddr_t *, boolean_t *); #define BACKUP_SLICE "s2" @@ -313,6 +313,7 @@ zpool_get_prop(zpool_handle_t *zhp, zpoo (void) zfs_nicenum(intval, buf, len); } break; + case ZPOOL_PROP_BOOTSIZE: case ZPOOL_PROP_EXPANDSZ: if (intval == 0) { (void) strlcpy(buf, "-", len); @@ -514,6 +515,16 @@ zpool_valid_proplist(libzfs_handle_t *hd } break; + case ZPOOL_PROP_BOOTSIZE: + if (!flags.create) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "property '%s' can only be set during pool " + "creation"), propname); + (void) zfs_error(hdl, EZFS_BADPROP, errbuf); + goto error; + } + break; + case ZPOOL_PROP_BOOTFS: if (flags.create || flags.import) { zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, @@ -1964,8 +1975,9 @@ vdev_to_nvlist_iter(nvlist_t *nv, nvlist /* * Search for the requested value. Special cases: * - * - ZPOOL_CONFIG_PATH for whole disk entries. These end in - * "s0" or "s0/old". The "s0" part is hidden from the user, + * - ZPOOL_CONFIG_PATH for whole disk entries. To support + * UEFI boot, these end in "s0" or "s0/old" or "s1" or + * "s1/old". The "s0" or "s1" part is hidden from the user, * but included in the string, so this matches around it. * - looking for a top-level vdev name (i.e. ZPOOL_CONFIG_TYPE). * @@ -1995,14 +2007,16 @@ vdev_to_nvlist_iter(nvlist_t *nv, nvlist /* * strings identical except trailing "s0" */ - if (strcmp(&val[vlen - 2], "s0") == 0 && + if ((strcmp(&val[vlen - 2], "s0") == 0 || + strcmp(&val[vlen - 2], "s1") == 0) && strncmp(srchval, val, slen) == 0) return (nv); /* * strings identical except trailing "s0/old" */ - if (strcmp(&val[vlen - 6], "s0/old") == 0 && + if ((strcmp(&val[vlen - 6], "s0/old") == 0 || + strcmp(&val[vlen - 6], "s1/old") == 0) && strcmp(&srchval[slen - 4], "/old") == 0 && strncmp(srchval, val, slen - 4) == 0) return (nv); @@ -3406,15 +3420,17 @@ zpool_vdev_name(libzfs_handle_t *hdl, zp char *tmp = zfs_strdup(hdl, path); /* - * If it starts with c#, and ends with "s0", chop - * the "s0" off, or if it ends with "s0/old", remove - * the "s0" from the middle. + * If it starts with c#, and ends with "s0" or "s1", + * chop the slice off, or if it ends with "s0/old" or + * "s1/old", remove the slice from the middle. */ if (CTD_CHECK(tmp)) { - if (strcmp(&tmp[pathlen - 2], "s0") == 0) { + if (strcmp(&tmp[pathlen - 2], "s0") == 0 || + strcmp(&tmp[pathlen - 2], "s1") == 0) { tmp[pathlen - 2] = '\0'; } else if (pathlen > 6 && - strcmp(&tmp[pathlen - 6], "s0/old") == 0) { + (strcmp(&tmp[pathlen - 6], "s0/old") == 0 || + strcmp(&tmp[pathlen - 6], "s1/old") == 0)) { (void) strcpy(&tmp[pathlen - 6], "/old"); } @@ -3807,15 +3823,18 @@ zpool_obj_to_path(zpool_handle_t *zhp, u * Read the EFI label from the config, if a label does not exist then * pass back the error to the caller. If the caller has passed a non-NULL * diskaddr argument then we set it to the starting address of the EFI - * partition. + * partition. If the caller has passed a non-NULL boolean argument, then + * we set it to indicate if the disk does have efi system partition. */ static int -read_efi_label(nvlist_t *config, diskaddr_t *sb) +read_efi_label(nvlist_t *config, diskaddr_t *sb, boolean_t *system) { char *path; int fd; char diskname[MAXPATHLEN]; + boolean_t boot = B_FALSE; int err = -1; + int slice; if (nvlist_lookup_string(config, ZPOOL_CONFIG_PATH, &path) != 0) return (err); @@ -3826,8 +3845,16 @@ read_efi_label(nvlist_t *config, diskadd struct dk_gpt *vtoc; if ((err = efi_alloc_and_read(fd, &vtoc)) >= 0) { - if (sb != NULL) - *sb = vtoc->efi_parts[0].p_start; + for (slice = 0; slice < vtoc->efi_nparts; slice++) { + if (vtoc->efi_parts[slice].p_tag == V_SYSTEM) + boot = B_TRUE; + if (vtoc->efi_parts[slice].p_tag == V_USR) + break; + } + if (sb != NULL && vtoc->efi_parts[slice].p_tag == V_USR) + *sb = vtoc->efi_parts[slice].p_start; + if (system != NULL) + *system = boot; efi_free(vtoc); } (void) close(fd); @@ -3854,7 +3881,7 @@ find_start_block(nvlist_t *config) &wholedisk) != 0 || !wholedisk) { return (MAXOFFSET_T); } - if (read_efi_label(config, &sb) < 0) + if (read_efi_label(config, &sb, NULL) < 0) sb = MAXOFFSET_T; return (sb); } @@ -3873,7 +3900,8 @@ find_start_block(nvlist_t *config) * stripped of any leading /dev path. */ int -zpool_label_disk(libzfs_handle_t *hdl, zpool_handle_t *zhp, const char *name) +zpool_label_disk(libzfs_handle_t *hdl, zpool_handle_t *zhp, const char *name, + zpool_boot_label_t boot_type, uint64_t boot_size, int *slice) { char path[MAXPATHLEN]; struct dk_gpt *vtoc; @@ -3931,15 +3959,6 @@ zpool_label_disk(libzfs_handle_t *hdl, z return (zfs_error(hdl, EZFS_NOCAP, errbuf)); } - slice_size = vtoc->efi_last_u_lba + 1; - slice_size -= EFI_MIN_RESV_SIZE; - if (start_block == MAXOFFSET_T) - start_block = NEW_START_BLOCK; - slice_size -= start_block; - - vtoc->efi_parts[0].p_start = start_block; - vtoc->efi_parts[0].p_size = slice_size; - /* * Why we use V_USR: V_BACKUP confuses users, and is considered * disposable by some EFI utilities (since EFI doesn't have a backup @@ -3948,12 +3967,103 @@ zpool_label_disk(libzfs_handle_t *hdl, z * etc. were all pretty specific. V_USR is as close to reality as we * can get, in the absence of V_OTHER. */ - vtoc->efi_parts[0].p_tag = V_USR; - (void) strcpy(vtoc->efi_parts[0].p_name, "zfs"); + /* first fix the partition start block */ + if (start_block == MAXOFFSET_T) + start_block = NEW_START_BLOCK; + + /* + * EFI System partition is using slice 0. + * ZFS is on slice 1 and slice 8 is reserved. + * We assume the GPT partition table without system + * partition has zfs p_start == NEW_START_BLOCK. + * If start_block != NEW_START_BLOCK, it means we have + * system partition. Correct solution would be to query/cache vtoc + * from existing vdev member. + */ + if (boot_type == ZPOOL_CREATE_BOOT_LABEL) { + if (boot_size % vtoc->efi_lbasize != 0) { + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "boot partition size must be a multiple of %d"), + vtoc->efi_lbasize); + (void) close(fd); + efi_free(vtoc); + return (zfs_error(hdl, EZFS_LABELFAILED, errbuf)); + } + /* + * System partition size checks. + * Note the 1MB is quite arbitrary value, since we + * are creating dedicated pool, it should be enough + * to hold fat + efi bootloader. May need to be + * adjusted if the bootloader size will grow. + */ + if (boot_size < 1024 * 1024) { + char buf[64]; + zfs_nicenum(boot_size, buf, sizeof (buf)); + zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, + "Specified size %s for EFI System partition is too " + "small, the minimum size is 1MB."), buf); + (void) close(fd); + efi_free(vtoc); + return (zfs_error(hdl, EZFS_LABELFAILED, errbuf)); + } + /* 33MB is tested with mkfs -F pcfs */ + if (hdl->libzfs_printerr && + ((vtoc->efi_lbasize == 512 && + boot_size < 33 * 1024 * 1024) || + (vtoc->efi_lbasize == 4096 && + boot_size < 256 * 1024 * 1024))) { + char buf[64]; + zfs_nicenum(boot_size, buf, sizeof (buf)); + (void) fprintf(stderr, dgettext(TEXT_DOMAIN, + "Warning: EFI System partition size %s is " + "not allowing to create FAT32 file\nsystem, which " + "may result in unbootable system.\n"), buf); + } + /* Adjust zfs partition start by size of system partition. */ + start_block += boot_size / vtoc->efi_lbasize; + } - vtoc->efi_parts[8].p_start = slice_size + start_block; - vtoc->efi_parts[8].p_size = resv; - vtoc->efi_parts[8].p_tag = V_RESERVED; + if (start_block == NEW_START_BLOCK) { + /* + * Use default layout. + * ZFS is on slice 0 and slice 8 is reserved. + */ + slice_size = vtoc->efi_last_u_lba + 1; + slice_size -= EFI_MIN_RESV_SIZE; + slice_size -= start_block; + if (slice != NULL) + *slice = 0; + + vtoc->efi_parts[0].p_start = start_block; + vtoc->efi_parts[0].p_size = slice_size; + + vtoc->efi_parts[0].p_tag = V_USR; + (void) strcpy(vtoc->efi_parts[0].p_name, "zfs"); + + vtoc->efi_parts[8].p_start = slice_size + start_block; + vtoc->efi_parts[8].p_size = resv; + vtoc->efi_parts[8].p_tag = V_RESERVED; + } else { + slice_size = start_block - NEW_START_BLOCK; + vtoc->efi_parts[0].p_start = NEW_START_BLOCK; + vtoc->efi_parts[0].p_size = slice_size; + vtoc->efi_parts[0].p_tag = V_SYSTEM; + (void) strcpy(vtoc->efi_parts[0].p_name, "loader"); + if (slice != NULL) + *slice = 1; + /* prepare slice 1 */ + slice_size = vtoc->efi_last_u_lba + 1 - slice_size; + slice_size -= resv; + slice_size -= NEW_START_BLOCK; + vtoc->efi_parts[1].p_start = start_block; + vtoc->efi_parts[1].p_size = slice_size; + vtoc->efi_parts[1].p_tag = V_USR; + (void) strcpy(vtoc->efi_parts[1].p_name, "zfs"); + + vtoc->efi_parts[8].p_start = slice_size + start_block; + vtoc->efi_parts[8].p_size = resv; + vtoc->efi_parts[8].p_tag = V_RESERVED; + } if (efi_write(fd, vtoc) != 0) { /* Modified: vendor/illumos/dist/man/man1m/zpool.1m ============================================================================== --- vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 12:00:31 2017 (r318940) +++ vendor/illumos/dist/man/man1m/zpool.1m Fri May 26 12:02:14 2017 (r318941) @@ -23,7 +23,7 @@ .\" Copyright (c) 2013 by Delphix. All rights reserved. .\" Copyright 2016 Nexenta Systems, Inc. .\" -.Dd March 30, 2016 +.Dd Oct 2, 2016 .Dt ZPOOL 1M .Os .Sh NAME @@ -47,6 +47,7 @@ .Nm .Cm create .Op Fl dfn +.Op Fl B .Op Fl m Ar mountpoint .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ... @@ -430,6 +431,12 @@ The following are read-only properties: Amount of storage available within the pool. This property can also be referred to by its shortened column name, .Sy avail . +.It Sy bootsize +The size of the system boot partition. This property can only be set at pool +creation time and is read-only once pool is created. Setting this property +implies using the +.Fl B +option. .It Sy capacity Percentage of pool space used. This property can also be referred to by its shortened column name, @@ -708,6 +715,7 @@ those errors associated with the specifi .Nm .Cm create .Op Fl dfn +.Op Fl B .Op Fl m Ar mountpoint .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ... @@ -766,6 +774,16 @@ By default all supported features are en .Fl d option is specified. .Bl -tag -width Ds +.It Fl B +Create whole disk pool with EFI System partition to support booting system +with UEFI firmware. Default size is 256MB. To create boot partition with +custom size, set the +.Sy bootsize +property with the +.Fl o +option. See the +.Sx Properties +section for details. .It Fl d Do not enable any features on the new pool. Individual features can be enabled by setting their corresponding properties to From owner-svn-src-vendor@freebsd.org Fri May 26 12:02:53 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2DDB8D83B05; Fri, 26 May 2017 12:02:53 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D91281CD6; Fri, 26 May 2017 12:02:52 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QC2pmr093160; Fri, 26 May 2017 12:02:51 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QC2pcU093159; Fri, 26 May 2017 12:02:51 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261202.v4QC2pcU093159@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:02:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318942 - vendor-sys/illumos/dist/uts/common/fs/zfs X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:02:53 -0000 Author: avg Date: Fri May 26 12:02:51 2017 New Revision: 318942 URL: https://svnweb.freebsd.org/changeset/base/318942 Log: 8166 zpool scrub thinks it repaired offline device illumos/illumos-gate@2d2f193a21231a58c583466dc23ba71f1a25f424 https://github.com/illumos/illumos-gate/commit/2d2f193a21231a58c583466dc23ba71f1a25f424 https://www.illumos.org/issues/8166 If we do a scrub while a leaf device is offline (via "zpool offline"), we will inadvertently clear the DTL (dirty time log) of the offline device, even though it is still damaged. When the device comes back online, we will incompletely resilver it, thinking that the scrub repaired blocks written before the scrub was started. The incomplete resilver can lead to data loss if there is a subsequent failure of a different leaf device. The fix is to never clear the DTL of offline devices. Note that if a device is onlined while a scrub is in progress, the scrub will be restarted. The problem can be worked around by running "zpool scrub" after "zpool online". See also https://github.com/zfsonlinux/zfs/issues/5806 Reviewed by: George Wilson george.wilson@delphix.com Reviewed by: Brad Lewis Approved by: Richard Lowe Author: Matthew Ahrens Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 12:02:14 2017 (r318941) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c Fri May 26 12:02:51 2017 (r318942) @@ -1788,6 +1788,9 @@ vdev_dtl_should_excise(vdev_t *vd) ASSERT0(scn->scn_phys.scn_errors); ASSERT0(vd->vdev_children); + if (vd->vdev_state < VDEV_STATE_DEGRADED) + return (B_FALSE); + if (vd->vdev_resilver_txg == 0 || range_tree_space(vd->vdev_dtl[DTL_MISSING]) == 0) return (B_TRUE); From owner-svn-src-vendor@freebsd.org Fri May 26 12:07:49 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13505D83C42; Fri, 26 May 2017 12:07:49 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D92231065; Fri, 26 May 2017 12:07:48 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QC7leM093439; Fri, 26 May 2017 12:07:47 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QC7l8k093438; Fri, 26 May 2017 12:07:47 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261207.v4QC7l8k093438@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:07:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318944 - vendor-sys/illumos/dist/uts/common/fs/zfs/sys X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:07:49 -0000 Author: avg Date: Fri May 26 12:07:47 2017 New Revision: 318944 URL: https://svnweb.freebsd.org/changeset/base/318944 Log: 8265 Reserve send stream flag for large dnode feature illumos/illumos-gate@bc83969fdbd1cb0d97ba00218c0a3de5c89fba92 https://github.com/illumos/illumos-gate/commit/bc83969fdbd1cb0d97ba00218c0a3de5c89fba92 https://www.illumos.org/issues/8265 Reserve bit 23 in the zfs send stream flags for the large dnode feature which has been implemented for Linux. Reviewed by: Matthew Ahrens Approved by: Robert Mustacchi Author: Brian Behlendorf Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h Fri May 26 12:04:21 2017 (r318943) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zfs_ioctl.h Fri May 26 12:07:47 2017 (r318944) @@ -93,6 +93,7 @@ typedef enum drr_headertype { #define DMU_BACKUP_FEATURE_RESUMING (1 << 20) /* flag #21 is reserved for a Delphix feature */ #define DMU_BACKUP_FEATURE_COMPRESSED (1 << 22) +/* flag #23 is reserved for the large dnode feature */ /* * Mask of all supported backup features From owner-svn-src-vendor@freebsd.org Fri May 26 12:13:29 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 726D0D83E3C; Fri, 26 May 2017 12:13:29 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3AAC21680; Fri, 26 May 2017 12:13:29 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QCDSQE097271; Fri, 26 May 2017 12:13:28 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QCDSsb097265; Fri, 26 May 2017 12:13:28 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261213.v4QCDSsb097265@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:13:27 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318946 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/di... X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:13:29 -0000 Author: avg Date: Fri May 26 12:13:27 2017 New Revision: 318946 URL: https://svnweb.freebsd.org/changeset/base/318946 Log: 8021 ARC buf data scatter-ization 8100 8021 seems to cause random BAD TRAP: type=d (#gp General protection) illumos/illumos-gate@770499e185d15678ccb0be57ebc626ad18d93383 https://github.com/illumos/illumos-gate/commit/770499e185d15678ccb0be57ebc626ad18d93383 https://www.illumos.org/issues/8021 The ARC buf data project (known simply as "ABD" since its genesis in the ZoL community) changes the way the ARC allocates `b_pdata` memory from using linear `void *` buffers to using scatter/gather lists of fixed-size 1KB chunks. This improves ZFS's performance by helping to defragment the address space occupied by the ARC, in particular for cases where compressed ARC is enabled. It could also ease future work to allocate pages directly from `segkpm` for minimal- overhead memory allocations, bypassing the `kmem` subsystem. This is essentially the same change as the one which recently landed in ZFS on Linux, although they made some platform-specific changes while adapting this work to their codebase: 1. Implemented the equivalent of the `segkpm` suggestion for future work mentioned above to bypass issues that they've had with the Linux kernel memory allocator. 2. Changed the internal representation of the ABD's scatter/gather list so it could be used to pass I/O directly into Linux block device drivers. (This feature is not available in the illumos block device interface yet.) https://www.illumos.org/issues/8100 My supermicro system is getting random BAD TRAP: type=d (#gp General protection) at about the stage where ZFS filesystems are mounted - usually console login prompt is already present but the services are still starting. After backing out 8021, the boot is completed and no panics do occur. Machine does dump, however savecore fails: savecore: bad magic number baddcafe I can get more data out with boot -k, if needed. # psrinfo -vp The physical processor has 4 cores and 8 virtual processors (0-7) The core has 2 virtual processors (0 4) The core has 2 virtual processors (1 5) The core has 2 virtual processors (2 6) The core has 2 virtual processors (3 7) x86 (GenuineIntel 306C3 family 6 model 60 step 3 clock 3500 MHz) Intel(r) Xeon(r) CPU E3-1246 v3 @ 3.50GHz # prtconf -m 32657 $ zpool status pool: rpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c3t0d0 ONLINE 0 0 0 c3t1d0 ONLINE 0 0 0 Reviewed by: Matthew Ahrens mahrens@delphix.com Reviewed by: George Wilson george.wilson@delphix.com Reviewed by: Paul Dagnelie pcd@delphix.com Reviewed by: John Kennedy john.kennedy@delphix.com Reviewed by: Prakash Surya prakash.surya@delphix.com Reviewed by: Prashanth Sreenivasa pks@delphix.com Reviewed by: Pavel Zakharov pavel.zakharov@delphix.com Reviewed by: Chris Williamson chris.williamson@delphix.com Approved by: Richard Lowe Author: Dan Kimmel Added: vendor-sys/illumos/dist/uts/common/fs/zfs/abd.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/abd.h (contents, props changed) Modified: vendor-sys/illumos/dist/common/zfs/zfs_fletcher.c vendor-sys/illumos/dist/common/zfs/zfs_fletcher.h vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/blkptr.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/edonr_zfs.c vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c vendor-sys/illumos/dist/uts/common/fs/zfs/sha256.c vendor-sys/illumos/dist/uts/common/fs/zfs/skein_zfs.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/ddt.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_checksum.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_compress.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_cache.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio_checksum.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio_compress.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zdb/zdb_il.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Modified: vendor-sys/illumos/dist/common/zfs/zfs_fletcher.c ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_fletcher.c Fri May 26 12:08:38 2017 (r318945) +++ vendor-sys/illumos/dist/common/zfs/zfs_fletcher.c Fri May 26 12:13:27 2017 (r318946) @@ -24,6 +24,7 @@ */ /* * Copyright 2013 Saso Kiselkov. All rights reserved. + * Copyright (c) 2016 by Delphix. All rights reserved. */ /* @@ -133,17 +134,29 @@ #include #include #include +#include -/*ARGSUSED*/ void -fletcher_2_native(const void *buf, uint64_t size, - const void *ctx_template, zio_cksum_t *zcp) +fletcher_init(zio_cksum_t *zcp) { + ZIO_SET_CHECKSUM(zcp, 0, 0, 0, 0); +} + +int +fletcher_2_incremental_native(void *buf, size_t size, void *data) +{ + zio_cksum_t *zcp = data; + const uint64_t *ip = buf; const uint64_t *ipend = ip + (size / sizeof (uint64_t)); uint64_t a0, b0, a1, b1; - for (a0 = b0 = a1 = b1 = 0; ip < ipend; ip += 2) { + a0 = zcp->zc_word[0]; + a1 = zcp->zc_word[1]; + b0 = zcp->zc_word[2]; + b1 = zcp->zc_word[3]; + + for (; ip < ipend; ip += 2) { a0 += ip[0]; a1 += ip[1]; b0 += a0; @@ -151,18 +164,33 @@ fletcher_2_native(const void *buf, uint6 } ZIO_SET_CHECKSUM(zcp, a0, a1, b0, b1); + return (0); } /*ARGSUSED*/ void -fletcher_2_byteswap(const void *buf, uint64_t size, +fletcher_2_native(const void *buf, size_t size, const void *ctx_template, zio_cksum_t *zcp) { + fletcher_init(zcp); + (void) fletcher_2_incremental_native((void *) buf, size, zcp); +} + +int +fletcher_2_incremental_byteswap(void *buf, size_t size, void *data) +{ + zio_cksum_t *zcp = data; + const uint64_t *ip = buf; const uint64_t *ipend = ip + (size / sizeof (uint64_t)); uint64_t a0, b0, a1, b1; - for (a0 = b0 = a1 = b1 = 0; ip < ipend; ip += 2) { + a0 = zcp->zc_word[0]; + a1 = zcp->zc_word[1]; + b0 = zcp->zc_word[2]; + b1 = zcp->zc_word[3]; + + for (; ip < ipend; ip += 2) { a0 += BSWAP_64(ip[0]); a1 += BSWAP_64(ip[1]); b0 += a0; @@ -170,50 +198,23 @@ fletcher_2_byteswap(const void *buf, uin } ZIO_SET_CHECKSUM(zcp, a0, a1, b0, b1); + return (0); } /*ARGSUSED*/ void -fletcher_4_native(const void *buf, uint64_t size, +fletcher_2_byteswap(const void *buf, size_t size, const void *ctx_template, zio_cksum_t *zcp) { - const uint32_t *ip = buf; - const uint32_t *ipend = ip + (size / sizeof (uint32_t)); - uint64_t a, b, c, d; - - for (a = b = c = d = 0; ip < ipend; ip++) { - a += ip[0]; - b += a; - c += b; - d += c; - } - - ZIO_SET_CHECKSUM(zcp, a, b, c, d); + fletcher_init(zcp); + (void) fletcher_2_incremental_byteswap((void *) buf, size, zcp); } -/*ARGSUSED*/ -void -fletcher_4_byteswap(const void *buf, uint64_t size, - const void *ctx_template, zio_cksum_t *zcp) +int +fletcher_4_incremental_native(void *buf, size_t size, void *data) { - const uint32_t *ip = buf; - const uint32_t *ipend = ip + (size / sizeof (uint32_t)); - uint64_t a, b, c, d; + zio_cksum_t *zcp = data; - for (a = b = c = d = 0; ip < ipend; ip++) { - a += BSWAP_32(ip[0]); - b += a; - c += b; - d += c; - } - - ZIO_SET_CHECKSUM(zcp, a, b, c, d); -} - -void -fletcher_4_incremental_native(const void *buf, uint64_t size, - zio_cksum_t *zcp) -{ const uint32_t *ip = buf; const uint32_t *ipend = ip + (size / sizeof (uint32_t)); uint64_t a, b, c, d; @@ -231,12 +232,23 @@ fletcher_4_incremental_native(const void } ZIO_SET_CHECKSUM(zcp, a, b, c, d); + return (0); } +/*ARGSUSED*/ void -fletcher_4_incremental_byteswap(const void *buf, uint64_t size, - zio_cksum_t *zcp) +fletcher_4_native(const void *buf, size_t size, + const void *ctx_template, zio_cksum_t *zcp) { + fletcher_init(zcp); + (void) fletcher_4_incremental_native((void *) buf, size, zcp); +} + +int +fletcher_4_incremental_byteswap(void *buf, size_t size, void *data) +{ + zio_cksum_t *zcp = data; + const uint32_t *ip = buf; const uint32_t *ipend = ip + (size / sizeof (uint32_t)); uint64_t a, b, c, d; @@ -254,4 +266,14 @@ fletcher_4_incremental_byteswap(const vo } ZIO_SET_CHECKSUM(zcp, a, b, c, d); + return (0); +} + +/*ARGSUSED*/ +void +fletcher_4_byteswap(const void *buf, size_t size, + const void *ctx_template, zio_cksum_t *zcp) +{ + fletcher_init(zcp); + (void) fletcher_4_incremental_byteswap((void *) buf, size, zcp); } Modified: vendor-sys/illumos/dist/common/zfs/zfs_fletcher.h ============================================================================== --- vendor-sys/illumos/dist/common/zfs/zfs_fletcher.h Fri May 26 12:08:38 2017 (r318945) +++ vendor-sys/illumos/dist/common/zfs/zfs_fletcher.h Fri May 26 12:13:27 2017 (r318946) @@ -24,6 +24,7 @@ */ /* * Copyright 2013 Saso Kiselkov. All rights reserved. + * Copyright (c) 2016 by Delphix. All rights reserved. */ #ifndef _ZFS_FLETCHER_H @@ -40,12 +41,15 @@ extern "C" { * fletcher checksum functions */ -void fletcher_2_native(const void *, uint64_t, const void *, zio_cksum_t *); -void fletcher_2_byteswap(const void *, uint64_t, const void *, zio_cksum_t *); -void fletcher_4_native(const void *, uint64_t, const void *, zio_cksum_t *); -void fletcher_4_byteswap(const void *, uint64_t, const void *, zio_cksum_t *); -void fletcher_4_incremental_native(const void *, uint64_t, zio_cksum_t *); -void fletcher_4_incremental_byteswap(const void *, uint64_t, zio_cksum_t *); +void fletcher_init(zio_cksum_t *); +void fletcher_2_native(const void *, size_t, const void *, zio_cksum_t *); +void fletcher_2_byteswap(const void *, size_t, const void *, zio_cksum_t *); +int fletcher_2_incremental_native(void *, size_t, void *); +int fletcher_2_incremental_byteswap(void *, size_t, void *); +void fletcher_4_native(const void *, size_t, const void *, zio_cksum_t *); +void fletcher_4_byteswap(const void *, size_t, const void *, zio_cksum_t *); +int fletcher_4_incremental_native(void *, size_t, void *); +int fletcher_4_incremental_byteswap(void *, size_t, void *); #ifdef __cplusplus } Modified: vendor-sys/illumos/dist/uts/common/Makefile.files ============================================================================== --- vendor-sys/illumos/dist/uts/common/Makefile.files Fri May 26 12:08:38 2017 (r318945) +++ vendor-sys/illumos/dist/uts/common/Makefile.files Fri May 26 12:13:27 2017 (r318946) @@ -1333,6 +1333,7 @@ NOTIFY_OBJS += md_notify.o TRANS_OBJS += mdtrans.o trans_ioctl.o trans_log.o ZFS_COMMON_OBJS += \ + abd.o \ arc.o \ blkptr.o \ bplist.o \ Added: vendor-sys/illumos/dist/uts/common/fs/zfs/abd.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/abd.c Fri May 26 12:13:27 2017 (r318946) @@ -0,0 +1,940 @@ +/* + * This file and its contents are supplied under the terms of the + * Common Development and Distribution License ("CDDL"), version 1.0. + * You may only use this file in accordance with the terms of version + * 1.0 of the CDDL. + * + * A full copy of the text of the CDDL should have accompanied this + * source. A copy of the CDDL is also available via the Internet at + * http://www.illumos.org/license/CDDL. + */ + +/* + * Copyright (c) 2014 by Chunwei Chen. All rights reserved. + * Copyright (c) 2016 by Delphix. All rights reserved. + */ + +/* + * ARC buffer data (ABD). + * + * ABDs are an abstract data structure for the ARC which can use two + * different ways of storing the underlying data: + * + * (a) Linear buffer. In this case, all the data in the ABD is stored in one + * contiguous buffer in memory (from a zio_[data_]buf_* kmem cache). + * + * +-------------------+ + * | ABD (linear) | + * | abd_flags = ... | + * | abd_size = ... | +--------------------------------+ + * | abd_buf ------------->| raw buffer of size abd_size | + * +-------------------+ +--------------------------------+ + * no abd_chunks + * + * (b) Scattered buffer. In this case, the data in the ABD is split into + * equal-sized chunks (from the abd_chunk_cache kmem_cache), with pointers + * to the chunks recorded in an array at the end of the ABD structure. + * + * +-------------------+ + * | ABD (scattered) | + * | abd_flags = ... | + * | abd_size = ... | + * | abd_offset = 0 | +-----------+ + * | abd_chunks[0] ----------------------------->| chunk 0 | + * | abd_chunks[1] ---------------------+ +-----------+ + * | ... | | +-----------+ + * | abd_chunks[N-1] ---------+ +------->| chunk 1 | + * +-------------------+ | +-----------+ + * | ... + * | +-----------+ + * +----------------->| chunk N-1 | + * +-----------+ + * + * Using a large proportion of scattered ABDs decreases ARC fragmentation since + * when we are at the limit of allocatable space, using equal-size chunks will + * allow us to quickly reclaim enough space for a new large allocation (assuming + * it is also scattered). + * + * In addition to directly allocating a linear or scattered ABD, it is also + * possible to create an ABD by requesting the "sub-ABD" starting at an offset + * within an existing ABD. In linear buffers this is simple (set abd_buf of + * the new ABD to the starting point within the original raw buffer), but + * scattered ABDs are a little more complex. The new ABD makes a copy of the + * relevant abd_chunks pointers (but not the underlying data). However, to + * provide arbitrary rather than only chunk-aligned starting offsets, it also + * tracks an abd_offset field which represents the starting point of the data + * within the first chunk in abd_chunks. For both linear and scattered ABDs, + * creating an offset ABD marks the original ABD as the offset's parent, and the + * original ABD's abd_children refcount is incremented. This data allows us to + * ensure the root ABD isn't deleted before its children. + * + * Most consumers should never need to know what type of ABD they're using -- + * the ABD public API ensures that it's possible to transparently switch from + * using a linear ABD to a scattered one when doing so would be beneficial. + * + * If you need to use the data within an ABD directly, if you know it's linear + * (because you allocated it) you can use abd_to_buf() to access the underlying + * raw buffer. Otherwise, you should use one of the abd_borrow_buf* functions + * which will allocate a raw buffer if necessary. Use the abd_return_buf* + * functions to return any raw buffers that are no longer necessary when you're + * done using them. + * + * There are a variety of ABD APIs that implement basic buffer operations: + * compare, copy, read, write, and fill with zeroes. If you need a custom + * function which progressively accesses the whole ABD, use the abd_iterate_* + * functions. + */ + +#include +#include +#include +#include +#include + +typedef struct abd_stats { + kstat_named_t abdstat_struct_size; + kstat_named_t abdstat_scatter_cnt; + kstat_named_t abdstat_scatter_data_size; + kstat_named_t abdstat_scatter_chunk_waste; + kstat_named_t abdstat_linear_cnt; + kstat_named_t abdstat_linear_data_size; +} abd_stats_t; + +static abd_stats_t abd_stats = { + /* Amount of memory occupied by all of the abd_t struct allocations */ + { "struct_size", KSTAT_DATA_UINT64 }, + /* + * The number of scatter ABDs which are currently allocated, excluding + * ABDs which don't own their data (for instance the ones which were + * allocated through abd_get_offset()). + */ + { "scatter_cnt", KSTAT_DATA_UINT64 }, + /* Amount of data stored in all scatter ABDs tracked by scatter_cnt */ + { "scatter_data_size", KSTAT_DATA_UINT64 }, + /* + * The amount of space wasted at the end of the last chunk across all + * scatter ABDs tracked by scatter_cnt. + */ + { "scatter_chunk_waste", KSTAT_DATA_UINT64 }, + /* + * The number of linear ABDs which are currently allocated, excluding + * ABDs which don't own their data (for instance the ones which were + * allocated through abd_get_offset() and abd_get_from_buf()). If an + * ABD takes ownership of its buf then it will become tracked. + */ + { "linear_cnt", KSTAT_DATA_UINT64 }, + /* Amount of data stored in all linear ABDs tracked by linear_cnt */ + { "linear_data_size", KSTAT_DATA_UINT64 }, +}; + +#define ABDSTAT(stat) (abd_stats.stat.value.ui64) +#define ABDSTAT_INCR(stat, val) \ + atomic_add_64(&abd_stats.stat.value.ui64, (val)) +#define ABDSTAT_BUMP(stat) ABDSTAT_INCR(stat, 1) +#define ABDSTAT_BUMPDOWN(stat) ABDSTAT_INCR(stat, -1) + +/* + * It is possible to make all future ABDs be linear by setting this to B_FALSE. + * Otherwise, ABDs are allocated scattered by default unless the caller uses + * abd_alloc_linear(). + */ +boolean_t zfs_abd_scatter_enabled = B_TRUE; + +/* + * The size of the chunks ABD allocates. Because the sizes allocated from the + * kmem_cache can't change, this tunable can only be modified at boot. Changing + * it at runtime would cause ABD iteration to work incorrectly for ABDs which + * were allocated with the old size, so a safeguard has been put in place which + * will cause the machine to panic if you change it and try to access the data + * within a scattered ABD. + */ +size_t zfs_abd_chunk_size = 4096; + +#ifdef _KERNEL +extern vmem_t *zio_alloc_arena; +#endif + +kmem_cache_t *abd_chunk_cache; +static kstat_t *abd_ksp; + +static void * +abd_alloc_chunk() +{ + void *c = kmem_cache_alloc(abd_chunk_cache, KM_PUSHPAGE); + ASSERT3P(c, !=, NULL); + return (c); +} + +static void +abd_free_chunk(void *c) +{ + kmem_cache_free(abd_chunk_cache, c); +} + +void +abd_init(void) +{ + vmem_t *data_alloc_arena = NULL; + +#ifdef _KERNEL + data_alloc_arena = zio_alloc_arena; +#endif + + /* + * Since ABD chunks do not appear in crash dumps, we pass KMC_NOTOUCH + * so that no allocator metadata is stored with the buffers. + */ + abd_chunk_cache = kmem_cache_create("abd_chunk", zfs_abd_chunk_size, 0, + NULL, NULL, NULL, NULL, data_alloc_arena, KMC_NOTOUCH); + + abd_ksp = kstat_create("zfs", 0, "abdstats", "misc", KSTAT_TYPE_NAMED, + sizeof (abd_stats) / sizeof (kstat_named_t), KSTAT_FLAG_VIRTUAL); + if (abd_ksp != NULL) { + abd_ksp->ks_data = &abd_stats; + kstat_install(abd_ksp); + } +} + +void +abd_fini(void) +{ + if (abd_ksp != NULL) { + kstat_delete(abd_ksp); + abd_ksp = NULL; + } + + kmem_cache_destroy(abd_chunk_cache); + abd_chunk_cache = NULL; +} + +static inline size_t +abd_chunkcnt_for_bytes(size_t size) +{ + return (P2ROUNDUP(size, zfs_abd_chunk_size) / zfs_abd_chunk_size); +} + +static inline size_t +abd_scatter_chunkcnt(abd_t *abd) +{ + ASSERT(!abd_is_linear(abd)); + return (abd_chunkcnt_for_bytes( + abd->abd_u.abd_scatter.abd_offset + abd->abd_size)); +} + +static inline void +abd_verify(abd_t *abd) +{ + ASSERT3U(abd->abd_size, >, 0); + ASSERT3U(abd->abd_size, <=, SPA_MAXBLOCKSIZE); + ASSERT3U(abd->abd_flags, ==, abd->abd_flags & (ABD_FLAG_LINEAR | + ABD_FLAG_OWNER | ABD_FLAG_META)); + IMPLY(abd->abd_parent != NULL, !(abd->abd_flags & ABD_FLAG_OWNER)); + IMPLY(abd->abd_flags & ABD_FLAG_META, abd->abd_flags & ABD_FLAG_OWNER); + if (abd_is_linear(abd)) { + ASSERT3P(abd->abd_u.abd_linear.abd_buf, !=, NULL); + } else { + ASSERT3U(abd->abd_u.abd_scatter.abd_offset, <, + zfs_abd_chunk_size); + size_t n = abd_scatter_chunkcnt(abd); + for (int i = 0; i < n; i++) { + ASSERT3P( + abd->abd_u.abd_scatter.abd_chunks[i], !=, NULL); + } + } +} + +static inline abd_t * +abd_alloc_struct(size_t chunkcnt) +{ + size_t size = offsetof(abd_t, abd_u.abd_scatter.abd_chunks[chunkcnt]); + abd_t *abd = kmem_alloc(size, KM_PUSHPAGE); + ASSERT3P(abd, !=, NULL); + ABDSTAT_INCR(abdstat_struct_size, size); + + return (abd); +} + +static inline void +abd_free_struct(abd_t *abd) +{ + size_t chunkcnt = abd_is_linear(abd) ? 0 : abd_scatter_chunkcnt(abd); + int size = offsetof(abd_t, abd_u.abd_scatter.abd_chunks[chunkcnt]); + kmem_free(abd, size); + ABDSTAT_INCR(abdstat_struct_size, -size); +} + +/* + * Allocate an ABD, along with its own underlying data buffers. Use this if you + * don't care whether the ABD is linear or not. + */ +abd_t * +abd_alloc(size_t size, boolean_t is_metadata) +{ + if (!zfs_abd_scatter_enabled) + return (abd_alloc_linear(size, is_metadata)); + + VERIFY3U(size, <=, SPA_MAXBLOCKSIZE); + + size_t n = abd_chunkcnt_for_bytes(size); + abd_t *abd = abd_alloc_struct(n); + + abd->abd_flags = ABD_FLAG_OWNER; + if (is_metadata) { + abd->abd_flags |= ABD_FLAG_META; + } + abd->abd_size = size; + abd->abd_parent = NULL; + refcount_create(&abd->abd_children); + + abd->abd_u.abd_scatter.abd_offset = 0; + abd->abd_u.abd_scatter.abd_chunk_size = zfs_abd_chunk_size; + + for (int i = 0; i < n; i++) { + void *c = abd_alloc_chunk(); + ASSERT3P(c, !=, NULL); + abd->abd_u.abd_scatter.abd_chunks[i] = c; + } + + ABDSTAT_BUMP(abdstat_scatter_cnt); + ABDSTAT_INCR(abdstat_scatter_data_size, size); + ABDSTAT_INCR(abdstat_scatter_chunk_waste, + n * zfs_abd_chunk_size - size); + + return (abd); +} + +static void +abd_free_scatter(abd_t *abd) +{ + size_t n = abd_scatter_chunkcnt(abd); + for (int i = 0; i < n; i++) { + abd_free_chunk(abd->abd_u.abd_scatter.abd_chunks[i]); + } + + refcount_destroy(&abd->abd_children); + ABDSTAT_BUMPDOWN(abdstat_scatter_cnt); + ABDSTAT_INCR(abdstat_scatter_data_size, -(int)abd->abd_size); + ABDSTAT_INCR(abdstat_scatter_chunk_waste, + abd->abd_size - n * zfs_abd_chunk_size); + + abd_free_struct(abd); +} + +/* + * Allocate an ABD that must be linear, along with its own underlying data + * buffer. Only use this when it would be very annoying to write your ABD + * consumer with a scattered ABD. + */ +abd_t * +abd_alloc_linear(size_t size, boolean_t is_metadata) +{ + abd_t *abd = abd_alloc_struct(0); + + VERIFY3U(size, <=, SPA_MAXBLOCKSIZE); + + abd->abd_flags = ABD_FLAG_LINEAR | ABD_FLAG_OWNER; + if (is_metadata) { + abd->abd_flags |= ABD_FLAG_META; + } + abd->abd_size = size; + abd->abd_parent = NULL; + refcount_create(&abd->abd_children); + + if (is_metadata) { + abd->abd_u.abd_linear.abd_buf = zio_buf_alloc(size); + } else { + abd->abd_u.abd_linear.abd_buf = zio_data_buf_alloc(size); + } + + ABDSTAT_BUMP(abdstat_linear_cnt); + ABDSTAT_INCR(abdstat_linear_data_size, size); + + return (abd); +} + +static void +abd_free_linear(abd_t *abd) +{ + if (abd->abd_flags & ABD_FLAG_META) { + zio_buf_free(abd->abd_u.abd_linear.abd_buf, abd->abd_size); + } else { + zio_data_buf_free(abd->abd_u.abd_linear.abd_buf, abd->abd_size); + } + + refcount_destroy(&abd->abd_children); + ABDSTAT_BUMPDOWN(abdstat_linear_cnt); + ABDSTAT_INCR(abdstat_linear_data_size, -(int)abd->abd_size); + + abd_free_struct(abd); +} + +/* + * Free an ABD. Only use this on ABDs allocated with abd_alloc() or + * abd_alloc_linear(). + */ +void +abd_free(abd_t *abd) +{ + abd_verify(abd); + ASSERT3P(abd->abd_parent, ==, NULL); + ASSERT(abd->abd_flags & ABD_FLAG_OWNER); + if (abd_is_linear(abd)) + abd_free_linear(abd); + else + abd_free_scatter(abd); +} + +/* + * Allocate an ABD of the same format (same metadata flag, same scatterize + * setting) as another ABD. + */ +abd_t * +abd_alloc_sametype(abd_t *sabd, size_t size) +{ + boolean_t is_metadata = (sabd->abd_flags & ABD_FLAG_META) != 0; + if (abd_is_linear(sabd)) { + return (abd_alloc_linear(size, is_metadata)); + } else { + return (abd_alloc(size, is_metadata)); + } +} + +/* + * If we're going to use this ABD for doing I/O using the block layer, the + * consumer of the ABD data doesn't care if it's scattered or not, and we don't + * plan to store this ABD in memory for a long period of time, we should + * allocate the ABD type that requires the least data copying to do the I/O. + * + * Currently this is linear ABDs, however if ldi_strategy() can ever issue I/Os + * using a scatter/gather list we should switch to that and replace this call + * with vanilla abd_alloc(). + */ +abd_t * +abd_alloc_for_io(size_t size, boolean_t is_metadata) +{ + return (abd_alloc_linear(size, is_metadata)); +} + +/* + * Allocate a new ABD to point to offset off of sabd. It shares the underlying + * buffer data with sabd. Use abd_put() to free. sabd must not be freed while + * any derived ABDs exist. + */ +abd_t * +abd_get_offset(abd_t *sabd, size_t off) +{ + abd_t *abd; + + abd_verify(sabd); + ASSERT3U(off, <=, sabd->abd_size); + + if (abd_is_linear(sabd)) { + abd = abd_alloc_struct(0); + + /* + * Even if this buf is filesystem metadata, we only track that + * if we own the underlying data buffer, which is not true in + * this case. Therefore, we don't ever use ABD_FLAG_META here. + */ + abd->abd_flags = ABD_FLAG_LINEAR; + + abd->abd_u.abd_linear.abd_buf = + (char *)sabd->abd_u.abd_linear.abd_buf + off; + } else { + size_t new_offset = sabd->abd_u.abd_scatter.abd_offset + off; + size_t chunkcnt = abd_scatter_chunkcnt(sabd) - + (new_offset / zfs_abd_chunk_size); + + abd = abd_alloc_struct(chunkcnt); + + /* + * Even if this buf is filesystem metadata, we only track that + * if we own the underlying data buffer, which is not true in + * this case. Therefore, we don't ever use ABD_FLAG_META here. + */ + abd->abd_flags = 0; + + abd->abd_u.abd_scatter.abd_offset = + new_offset % zfs_abd_chunk_size; + abd->abd_u.abd_scatter.abd_chunk_size = zfs_abd_chunk_size; + + /* Copy the scatterlist starting at the correct offset */ + (void) memcpy(&abd->abd_u.abd_scatter.abd_chunks, + &sabd->abd_u.abd_scatter.abd_chunks[new_offset / + zfs_abd_chunk_size], + chunkcnt * sizeof (void *)); + } + + abd->abd_size = sabd->abd_size - off; + abd->abd_parent = sabd; + refcount_create(&abd->abd_children); + (void) refcount_add_many(&sabd->abd_children, abd->abd_size, abd); + + return (abd); +} + +/* + * Allocate a linear ABD structure for buf. You must free this with abd_put() + * since the resulting ABD doesn't own its own buffer. + */ +abd_t * +abd_get_from_buf(void *buf, size_t size) +{ + abd_t *abd = abd_alloc_struct(0); + + VERIFY3U(size, <=, SPA_MAXBLOCKSIZE); + + /* + * Even if this buf is filesystem metadata, we only track that if we + * own the underlying data buffer, which is not true in this case. + * Therefore, we don't ever use ABD_FLAG_META here. + */ + abd->abd_flags = ABD_FLAG_LINEAR; + abd->abd_size = size; + abd->abd_parent = NULL; + refcount_create(&abd->abd_children); + + abd->abd_u.abd_linear.abd_buf = buf; + + return (abd); +} + +/* + * Free an ABD allocated from abd_get_offset() or abd_get_from_buf(). Will not + * free the underlying scatterlist or buffer. + */ +void +abd_put(abd_t *abd) +{ + abd_verify(abd); + ASSERT(!(abd->abd_flags & ABD_FLAG_OWNER)); + + if (abd->abd_parent != NULL) { + (void) refcount_remove_many(&abd->abd_parent->abd_children, + abd->abd_size, abd); + } + + refcount_destroy(&abd->abd_children); + abd_free_struct(abd); +} + +/* + * Get the raw buffer associated with a linear ABD. + */ +void * +abd_to_buf(abd_t *abd) +{ + ASSERT(abd_is_linear(abd)); + abd_verify(abd); + return (abd->abd_u.abd_linear.abd_buf); +} + +/* + * Borrow a raw buffer from an ABD without copying the contents of the ABD + * into the buffer. If the ABD is scattered, this will allocate a raw buffer + * whose contents are undefined. To copy over the existing data in the ABD, use + * abd_borrow_buf_copy() instead. + */ +void * +abd_borrow_buf(abd_t *abd, size_t n) +{ + void *buf; + abd_verify(abd); + ASSERT3U(abd->abd_size, >=, n); + if (abd_is_linear(abd)) { + buf = abd_to_buf(abd); + } else { + buf = zio_buf_alloc(n); + } + (void) refcount_add_many(&abd->abd_children, n, buf); + + return (buf); +} + +void * +abd_borrow_buf_copy(abd_t *abd, size_t n) +{ + void *buf = abd_borrow_buf(abd, n); + if (!abd_is_linear(abd)) { + abd_copy_to_buf(buf, abd, n); + } + return (buf); +} + +/* + * Return a borrowed raw buffer to an ABD. If the ABD is scattered, this will + * not change the contents of the ABD and will ASSERT that you didn't modify + * the buffer since it was borrowed. If you want any changes you made to buf to + * be copied back to abd, use abd_return_buf_copy() instead. + */ +void +abd_return_buf(abd_t *abd, void *buf, size_t n) +{ + abd_verify(abd); + ASSERT3U(abd->abd_size, >=, n); + if (abd_is_linear(abd)) { + ASSERT3P(buf, ==, abd_to_buf(abd)); + } else { + ASSERT0(abd_cmp_buf(abd, buf, n)); + zio_buf_free(buf, n); + } + (void) refcount_remove_many(&abd->abd_children, n, buf); +} + +void +abd_return_buf_copy(abd_t *abd, void *buf, size_t n) +{ + if (!abd_is_linear(abd)) { + abd_copy_from_buf(abd, buf, n); + } + abd_return_buf(abd, buf, n); +} + +/* + * Give this ABD ownership of the buffer that it's storing. Can only be used on + * linear ABDs which were allocated via abd_get_from_buf(), or ones allocated + * with abd_alloc_linear() which subsequently released ownership of their buf + * with abd_release_ownership_of_buf(). + */ +void +abd_take_ownership_of_buf(abd_t *abd, boolean_t is_metadata) +{ + ASSERT(abd_is_linear(abd)); + ASSERT(!(abd->abd_flags & ABD_FLAG_OWNER)); + abd_verify(abd); + + abd->abd_flags |= ABD_FLAG_OWNER; + if (is_metadata) { + abd->abd_flags |= ABD_FLAG_META; + } + + ABDSTAT_BUMP(abdstat_linear_cnt); + ABDSTAT_INCR(abdstat_linear_data_size, abd->abd_size); +} + +void +abd_release_ownership_of_buf(abd_t *abd) +{ + ASSERT(abd_is_linear(abd)); + ASSERT(abd->abd_flags & ABD_FLAG_OWNER); + abd_verify(abd); + + abd->abd_flags &= ~ABD_FLAG_OWNER; + /* Disable this flag since we no longer own the data buffer */ + abd->abd_flags &= ~ABD_FLAG_META; + + ABDSTAT_BUMPDOWN(abdstat_linear_cnt); + ABDSTAT_INCR(abdstat_linear_data_size, -(int)abd->abd_size); +} + +struct abd_iter { + abd_t *iter_abd; /* ABD being iterated through */ + size_t iter_pos; /* position (relative to abd_offset) */ + void *iter_mapaddr; /* addr corresponding to iter_pos */ + size_t iter_mapsize; /* length of data valid at mapaddr */ +}; + +static inline size_t +abd_iter_scatter_chunk_offset(struct abd_iter *aiter) +{ + ASSERT(!abd_is_linear(aiter->iter_abd)); + return ((aiter->iter_abd->abd_u.abd_scatter.abd_offset + + aiter->iter_pos) % zfs_abd_chunk_size); +} + +static inline size_t +abd_iter_scatter_chunk_index(struct abd_iter *aiter) +{ + ASSERT(!abd_is_linear(aiter->iter_abd)); + return ((aiter->iter_abd->abd_u.abd_scatter.abd_offset + + aiter->iter_pos) / zfs_abd_chunk_size); +} + +/* + * Initialize the abd_iter. + */ +static void +abd_iter_init(struct abd_iter *aiter, abd_t *abd) +{ + abd_verify(abd); + aiter->iter_abd = abd; + aiter->iter_pos = 0; + aiter->iter_mapaddr = NULL; + aiter->iter_mapsize = 0; +} + +/* + * Advance the iterator by a certain amount. Cannot be called when a chunk is + * in use. This can be safely called when the aiter has already exhausted, in + * which case this does nothing. + */ +static void +abd_iter_advance(struct abd_iter *aiter, size_t amount) +{ + ASSERT3P(aiter->iter_mapaddr, ==, NULL); + ASSERT0(aiter->iter_mapsize); + + /* There's nothing left to advance to, so do nothing */ + if (aiter->iter_pos == aiter->iter_abd->abd_size) + return; + + aiter->iter_pos += amount; +} + +/* + * Map the current chunk into aiter. This can be safely called when the aiter + * has already exhausted, in which case this does nothing. + */ +static void +abd_iter_map(struct abd_iter *aiter) +{ + void *paddr; + size_t offset = 0; + + ASSERT3P(aiter->iter_mapaddr, ==, NULL); + ASSERT0(aiter->iter_mapsize); + + /* Panic if someone has changed zfs_abd_chunk_size */ + IMPLY(!abd_is_linear(aiter->iter_abd), zfs_abd_chunk_size == + aiter->iter_abd->abd_u.abd_scatter.abd_chunk_size); + + /* There's nothing left to iterate over, so do nothing */ + if (aiter->iter_pos == aiter->iter_abd->abd_size) + return; + + if (abd_is_linear(aiter->iter_abd)) { + offset = aiter->iter_pos; + aiter->iter_mapsize = aiter->iter_abd->abd_size - offset; + paddr = aiter->iter_abd->abd_u.abd_linear.abd_buf; + } else { + size_t index = abd_iter_scatter_chunk_index(aiter); + offset = abd_iter_scatter_chunk_offset(aiter); + aiter->iter_mapsize = zfs_abd_chunk_size - offset; + paddr = aiter->iter_abd->abd_u.abd_scatter.abd_chunks[index]; + } + aiter->iter_mapaddr = (char *)paddr + offset; +} + +/* + * Unmap the current chunk from aiter. This can be safely called when the aiter + * has already exhausted, in which case this does nothing. + */ +static void +abd_iter_unmap(struct abd_iter *aiter) +{ + /* There's nothing left to unmap, so do nothing */ + if (aiter->iter_pos == aiter->iter_abd->abd_size) + return; + + ASSERT3P(aiter->iter_mapaddr, !=, NULL); + ASSERT3U(aiter->iter_mapsize, >, 0); + + aiter->iter_mapaddr = NULL; + aiter->iter_mapsize = 0; +} + +int +abd_iterate_func(abd_t *abd, size_t off, size_t size, + abd_iter_func_t *func, void *private) +{ + int ret = 0; + struct abd_iter aiter; + + abd_verify(abd); + ASSERT3U(off + size, <=, abd->abd_size); + + abd_iter_init(&aiter, abd); + abd_iter_advance(&aiter, off); + + while (size > 0) { + abd_iter_map(&aiter); + + size_t len = MIN(aiter.iter_mapsize, size); + ASSERT3U(len, >, 0); + + ret = func(aiter.iter_mapaddr, len, private); + + abd_iter_unmap(&aiter); + + if (ret != 0) + break; + + size -= len; + abd_iter_advance(&aiter, len); + } + + return (ret); +} + +struct buf_arg { + void *arg_buf; +}; + +static int +abd_copy_to_buf_off_cb(void *buf, size_t size, void *private) +{ + struct buf_arg *ba_ptr = private; *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** From owner-svn-src-vendor@freebsd.org Fri May 26 12:13:30 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3E51D83E40; Fri, 26 May 2017 12:13:29 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B8AD61681; Fri, 26 May 2017 12:13:29 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QCDSJt097280; Fri, 26 May 2017 12:13:28 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QCDSir097276; Fri, 26 May 2017 12:13:28 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261213.v4QCDSir097276@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:13:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318946 - vendor-sys/illumos/dist/common/zfs vendor-sys/illumos/dist/uts/common vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/di... X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:13:30 -0000 Author: avg Date: Fri May 26 12:13:27 2017 New Revision: 318946 URL: https://svnweb.freebsd.org/changeset/base/318946 Log: 8021 ARC buf data scatter-ization 8100 8021 seems to cause random BAD TRAP: type=d (#gp General protection) illumos/illumos-gate@770499e185d15678ccb0be57ebc626ad18d93383 https://github.com/illumos/illumos-gate/commit/770499e185d15678ccb0be57ebc626ad18d93383 https://www.illumos.org/issues/8021 The ARC buf data project (known simply as "ABD" since its genesis in the ZoL community) changes the way the ARC allocates `b_pdata` memory from using linear `void *` buffers to using scatter/gather lists of fixed-size 1KB chunks. This improves ZFS's performance by helping to defragment the address space occupied by the ARC, in particular for cases where compressed ARC is enabled. It could also ease future work to allocate pages directly from `segkpm` for minimal- overhead memory allocations, bypassing the `kmem` subsystem. This is essentially the same change as the one which recently landed in ZFS on Linux, although they made some platform-specific changes while adapting this work to their codebase: 1. Implemented the equivalent of the `segkpm` suggestion for future work mentioned above to bypass issues that they've had with the Linux kernel memory allocator. 2. Changed the internal representation of the ABD's scatter/gather list so it could be used to pass I/O directly into Linux block device drivers. (This feature is not available in the illumos block device interface yet.) https://www.illumos.org/issues/8100 My supermicro system is getting random BAD TRAP: type=d (#gp General protection) at about the stage where ZFS filesystems are mounted - usually console login prompt is already present but the services are still starting. After backing out 8021, the boot is completed and no panics do occur. Machine does dump, however savecore fails: savecore: bad magic number baddcafe I can get more data out with boot -k, if needed. # psrinfo -vp The physical processor has 4 cores and 8 virtual processors (0-7) The core has 2 virtual processors (0 4) The core has 2 virtual processors (1 5) The core has 2 virtual processors (2 6) The core has 2 virtual processors (3 7) x86 (GenuineIntel 306C3 family 6 model 60 step 3 clock 3500 MHz) Intel(r) Xeon(r) CPU E3-1246 v3 @ 3.50GHz # prtconf -m 32657 $ zpool status pool: rpool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c3t0d0 ONLINE 0 0 0 c3t1d0 ONLINE 0 0 0 Reviewed by: Matthew Ahrens mahrens@delphix.com Reviewed by: George Wilson george.wilson@delphix.com Reviewed by: Paul Dagnelie pcd@delphix.com Reviewed by: John Kennedy john.kennedy@delphix.com Reviewed by: Prakash Surya prakash.surya@delphix.com Reviewed by: Prashanth Sreenivasa pks@delphix.com Reviewed by: Pavel Zakharov pavel.zakharov@delphix.com Reviewed by: Chris Williamson chris.williamson@delphix.com Approved by: Richard Lowe Author: Dan Kimmel Modified: vendor/illumos/dist/cmd/zdb/zdb.c vendor/illumos/dist/cmd/zdb/zdb_il.c vendor/illumos/dist/cmd/ztest/ztest.c vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Changes in other areas also in this revision: Added: vendor-sys/illumos/dist/uts/common/fs/zfs/abd.c (contents, props changed) vendor-sys/illumos/dist/uts/common/fs/zfs/sys/abd.h (contents, props changed) Modified: vendor-sys/illumos/dist/common/zfs/zfs_fletcher.c vendor-sys/illumos/dist/common/zfs/zfs_fletcher.h vendor-sys/illumos/dist/uts/common/Makefile.files vendor-sys/illumos/dist/uts/common/fs/zfs/arc.c vendor-sys/illumos/dist/uts/common/fs/zfs/blkptr.c vendor-sys/illumos/dist/uts/common/fs/zfs/dbuf.c vendor-sys/illumos/dist/uts/common/fs/zfs/ddt.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu.c vendor-sys/illumos/dist/uts/common/fs/zfs/dmu_send.c vendor-sys/illumos/dist/uts/common/fs/zfs/dsl_scan.c vendor-sys/illumos/dist/uts/common/fs/zfs/edonr_zfs.c vendor-sys/illumos/dist/uts/common/fs/zfs/lz4.c vendor-sys/illumos/dist/uts/common/fs/zfs/sha256.c vendor-sys/illumos/dist/uts/common/fs/zfs/skein_zfs.c vendor-sys/illumos/dist/uts/common/fs/zfs/spa.c vendor-sys/illumos/dist/uts/common/fs/zfs/sys/ddt.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/spa.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/vdev_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_checksum.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio_compress.h vendor-sys/illumos/dist/uts/common/fs/zfs/vdev.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_cache.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_disk.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_file.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_label.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_mirror.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_queue.c vendor-sys/illumos/dist/uts/common/fs/zfs/vdev_raidz.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio_checksum.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio_compress.c Modified: vendor/illumos/dist/cmd/zdb/zdb.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb.c Fri May 26 12:08:38 2017 (r318945) +++ vendor/illumos/dist/cmd/zdb/zdb.c Fri May 26 12:13:27 2017 (r318946) @@ -60,6 +60,7 @@ #include #include #include +#include #include #undef verify #include @@ -2535,7 +2536,7 @@ zdb_blkptr_done(zio_t *zio) zdb_cb_t *zcb = zio->io_private; zbookmark_phys_t *zb = &zio->io_bookmark; - zio_data_buf_free(zio->io_data, zio->io_size); + abd_free(zio->io_abd); mutex_enter(&spa->spa_scrub_lock); spa->spa_scrub_inflight--; @@ -2601,7 +2602,7 @@ zdb_blkptr_cb(spa_t *spa, zilog_t *zilog if (!BP_IS_EMBEDDED(bp) && (dump_opt['c'] > 1 || (dump_opt['c'] && is_metadata))) { size_t size = BP_GET_PSIZE(bp); - void *data = zio_data_buf_alloc(size); + abd_t *abd = abd_alloc(size, B_FALSE); int flags = ZIO_FLAG_CANFAIL | ZIO_FLAG_SCRUB | ZIO_FLAG_RAW; /* If it's an intent log block, failure is expected. */ @@ -2614,7 +2615,7 @@ zdb_blkptr_cb(spa_t *spa, zilog_t *zilog spa->spa_scrub_inflight++; mutex_exit(&spa->spa_scrub_lock); - zio_nowait(zio_read(NULL, spa, bp, data, size, + zio_nowait(zio_read(NULL, spa, bp, abd, size, zdb_blkptr_done, zcb, ZIO_PRIORITY_ASYNC_READ, flags, zb)); } @@ -3394,6 +3395,13 @@ name: return (NULL); } +/* ARGSUSED */ +static int +random_get_pseudo_bytes_cb(void *buf, size_t len, void *unused) +{ + return (random_get_pseudo_bytes(buf, len)); +} + /* * Read a block from a pool and print it out. The syntax of the * block descriptor is: @@ -3425,7 +3433,8 @@ zdb_read_block(char *thing, spa_t *spa) uint64_t offset = 0, size = 0, psize = 0, lsize = 0, blkptr_offset = 0; zio_t *zio; vdev_t *vd; - void *pbuf, *lbuf, *buf; + abd_t *pabd; + void *lbuf, *buf; char *s, *p, *dup, *vdev, *flagstr; int i, error; @@ -3496,7 +3505,7 @@ zdb_read_block(char *thing, spa_t *spa) psize = size; lsize = size; - pbuf = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL); + pabd = abd_alloc_linear(SPA_MAXBLOCKSIZE, B_FALSE); lbuf = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL); BP_ZERO(bp); @@ -3524,15 +3533,15 @@ zdb_read_block(char *thing, spa_t *spa) /* * Treat this as a normal block read. */ - zio_nowait(zio_read(zio, spa, bp, pbuf, psize, NULL, NULL, + zio_nowait(zio_read(zio, spa, bp, pabd, psize, NULL, NULL, ZIO_PRIORITY_SYNC_READ, ZIO_FLAG_CANFAIL | ZIO_FLAG_RAW, NULL)); } else { /* * Treat this as a vdev child I/O. */ - zio_nowait(zio_vdev_child_io(zio, bp, vd, offset, pbuf, psize, - ZIO_TYPE_READ, ZIO_PRIORITY_SYNC_READ, + zio_nowait(zio_vdev_child_io(zio, bp, vd, offset, pabd, + psize, ZIO_TYPE_READ, ZIO_PRIORITY_SYNC_READ, ZIO_FLAG_DONT_CACHE | ZIO_FLAG_DONT_QUEUE | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY | ZIO_FLAG_CANFAIL | ZIO_FLAG_RAW, NULL, NULL)); @@ -3555,21 +3564,21 @@ zdb_read_block(char *thing, spa_t *spa) void *pbuf2 = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL); void *lbuf2 = umem_alloc(SPA_MAXBLOCKSIZE, UMEM_NOFAIL); - bcopy(pbuf, pbuf2, psize); + abd_copy_to_buf(pbuf2, pabd, psize); - VERIFY(random_get_pseudo_bytes((uint8_t *)pbuf + psize, - SPA_MAXBLOCKSIZE - psize) == 0); + VERIFY0(abd_iterate_func(pabd, psize, SPA_MAXBLOCKSIZE - psize, + random_get_pseudo_bytes_cb, NULL)); - VERIFY(random_get_pseudo_bytes((uint8_t *)pbuf2 + psize, - SPA_MAXBLOCKSIZE - psize) == 0); + VERIFY0(random_get_pseudo_bytes((uint8_t *)pbuf2 + psize, + SPA_MAXBLOCKSIZE - psize)); for (lsize = SPA_MAXBLOCKSIZE; lsize > psize; lsize -= SPA_MINBLOCKSIZE) { for (c = 0; c < ZIO_COMPRESS_FUNCTIONS; c++) { - if (zio_decompress_data(c, pbuf, lbuf, - psize, lsize) == 0 && - zio_decompress_data(c, pbuf2, lbuf2, - psize, lsize) == 0 && + if (zio_decompress_data(c, pabd, + lbuf, psize, lsize) == 0 && + zio_decompress_data_buf(c, pbuf2, + lbuf2, psize, lsize) == 0 && bcmp(lbuf, lbuf2, lsize) == 0) break; } @@ -3588,7 +3597,7 @@ zdb_read_block(char *thing, spa_t *spa) buf = lbuf; size = lsize; } else { - buf = pbuf; + buf = abd_to_buf(pabd); size = psize; } @@ -3606,7 +3615,7 @@ zdb_read_block(char *thing, spa_t *spa) zdb_dump_block(thing, buf, size, flags); out: - umem_free(pbuf, SPA_MAXBLOCKSIZE); + abd_free(pabd); umem_free(lbuf, SPA_MAXBLOCKSIZE); free(dup); } Modified: vendor/illumos/dist/cmd/zdb/zdb_il.c ============================================================================== --- vendor/illumos/dist/cmd/zdb/zdb_il.c Fri May 26 12:08:38 2017 (r318945) +++ vendor/illumos/dist/cmd/zdb/zdb_il.c Fri May 26 12:13:27 2017 (r318946) @@ -24,7 +24,7 @@ */ /* - * Copyright (c) 2013, 2014 by Delphix. All rights reserved. + * Copyright (c) 2013, 2016 by Delphix. All rights reserved. */ /* @@ -41,6 +41,7 @@ #include #include #include +#include extern uint8_t dump_opt[256]; @@ -117,13 +118,27 @@ zil_prt_rec_rename(zilog_t *zilog, int t } /* ARGSUSED */ +static int +zil_prt_rec_write_cb(void *data, size_t len, void *unused) +{ + char *cdata = data; + for (int i = 0; i < len; i++) { + if (isprint(*cdata)) + (void) printf("%c ", *cdata); + else + (void) printf("%2X", *cdata); + cdata++; + } + return (0); +} + +/* ARGSUSED */ static void zil_prt_rec_write(zilog_t *zilog, int txtype, lr_write_t *lr) { - char *data, *dlimit; + abd_t *data; blkptr_t *bp = &lr->lr_blkptr; zbookmark_phys_t zb; - char buf[SPA_MAXBLOCKSIZE]; int verbose = MAX(dump_opt['d'], dump_opt['i']); int error; @@ -144,7 +159,6 @@ zil_prt_rec_write(zilog_t *zilog, int tx if (BP_IS_HOLE(bp)) { (void) printf("\t\t\tLSIZE 0x%llx\n", (u_longlong_t)BP_GET_LSIZE(bp)); - bzero(buf, sizeof (buf)); (void) printf("%s\n", prefix); return; } @@ -157,28 +171,26 @@ zil_prt_rec_write(zilog_t *zilog, int tx lr->lr_foid, ZB_ZIL_LEVEL, lr->lr_offset / BP_GET_LSIZE(bp)); + data = abd_alloc(BP_GET_LSIZE(bp), B_FALSE); error = zio_wait(zio_read(NULL, zilog->zl_spa, - bp, buf, BP_GET_LSIZE(bp), NULL, NULL, + bp, data, BP_GET_LSIZE(bp), NULL, NULL, ZIO_PRIORITY_SYNC_READ, ZIO_FLAG_CANFAIL, &zb)); if (error) - return; - data = buf; + goto out; } else { - data = (char *)(lr + 1); + /* data is stored after the end of the lr_write record */ + data = abd_alloc(lr->lr_length, B_FALSE); + abd_copy_from_buf(data, lr + 1, lr->lr_length); } - dlimit = data + MIN(lr->lr_length, - (verbose < 6 ? 20 : SPA_MAXBLOCKSIZE)); - (void) printf("%s", prefix); - while (data < dlimit) { - if (isprint(*data)) - (void) printf("%c ", *data); - else - (void) printf("%2X", *data); - data++; - } + (void) abd_iterate_func(data, + 0, MIN(lr->lr_length, (verbose < 6 ? 20 : SPA_MAXBLOCKSIZE)), + zil_prt_rec_write_cb, NULL); (void) printf("\n"); + +out: + abd_free(data); } /* ARGSUSED */ Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Fri May 26 12:08:38 2017 (r318945) +++ vendor/illumos/dist/cmd/ztest/ztest.c Fri May 26 12:13:27 2017 (r318946) @@ -111,6 +111,7 @@ #include #include #include +#include #include #include #include @@ -188,6 +189,7 @@ extern uint64_t metaslab_df_alloc_thresh extern uint64_t zfs_deadman_synctime_ms; extern int metaslab_preload_limit; extern boolean_t zfs_compressed_arc_enabled; +extern boolean_t zfs_abd_scatter_enabled; static ztest_shared_opts_t *ztest_shared_opts; static ztest_shared_opts_t ztest_opts; @@ -5048,7 +5050,7 @@ ztest_ddt_repair(ztest_ds_t *zd, uint64_ enum zio_checksum checksum = spa_dedup_checksum(spa); dmu_buf_t *db; dmu_tx_t *tx; - void *buf; + abd_t *abd; blkptr_t blk; int copies = 2 * ZIO_DEDUPDITTO_MIN; @@ -5128,14 +5130,14 @@ ztest_ddt_repair(ztest_ds_t *zd, uint64_ * Damage the block. Dedup-ditto will save us when we read it later. */ psize = BP_GET_PSIZE(&blk); - buf = zio_buf_alloc(psize); - ztest_pattern_set(buf, psize, ~pattern); + abd = abd_alloc_linear(psize, B_TRUE); + ztest_pattern_set(abd_to_buf(abd), psize, ~pattern); (void) zio_wait(zio_rewrite(NULL, spa, 0, &blk, - buf, psize, NULL, NULL, ZIO_PRIORITY_SYNC_WRITE, + abd, psize, NULL, NULL, ZIO_PRIORITY_SYNC_WRITE, ZIO_FLAG_CANFAIL | ZIO_FLAG_INDUCE_DAMAGE, NULL)); - zio_buf_free(buf, psize); + abd_free(abd); (void) rw_unlock(&ztest_name_lock); } @@ -5418,6 +5420,12 @@ ztest_resume_thread(void *arg) */ if (ztest_random(10) == 0) zfs_compressed_arc_enabled = ztest_random(2); + + /* + * Periodically change the zfs_abd_scatter_enabled setting. + */ + if (ztest_random(10) == 0) + zfs_abd_scatter_enabled = ztest_random(2); } return (NULL); } Modified: vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c ============================================================================== --- vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Fri May 26 12:08:38 2017 (r318945) +++ vendor/illumos/dist/lib/libzfs/common/libzfs_sendrecv.c Fri May 26 12:13:27 2017 (r318946) @@ -192,19 +192,19 @@ dump_record(dmu_replay_record_t *drr, vo { ASSERT3U(offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), ==, sizeof (dmu_replay_record_t) - sizeof (zio_cksum_t)); - fletcher_4_incremental_native(drr, + (void) fletcher_4_incremental_native(drr, offsetof(dmu_replay_record_t, drr_u.drr_checksum.drr_checksum), zc); if (drr->drr_type != DRR_BEGIN) { ASSERT(ZIO_CHECKSUM_IS_ZERO(&drr->drr_u. drr_checksum.drr_checksum)); drr->drr_u.drr_checksum.drr_checksum = *zc; } - fletcher_4_incremental_native(&drr->drr_u.drr_checksum.drr_checksum, - sizeof (zio_cksum_t), zc); + (void) fletcher_4_incremental_native( + &drr->drr_u.drr_checksum.drr_checksum, sizeof (zio_cksum_t), zc); if (write(outfd, drr, sizeof (*drr)) == -1) return (errno); if (payload_len != 0) { - fletcher_4_incremental_native(payload, payload_len, zc); + (void) fletcher_4_incremental_native(payload, payload_len, zc); if (write(outfd, payload, payload_len) == -1) return (errno); } @@ -2093,9 +2093,9 @@ recv_read(libzfs_handle_t *hdl, int fd, if (zc) { if (byteswap) - fletcher_4_incremental_byteswap(buf, ilen, zc); + (void) fletcher_4_incremental_byteswap(buf, ilen, zc); else - fletcher_4_incremental_native(buf, ilen, zc); + (void) fletcher_4_incremental_native(buf, ilen, zc); } return (0); } @@ -3649,7 +3649,8 @@ zfs_receive_impl(libzfs_handle_t *hdl, c * recv_read() above; do it again correctly. */ bzero(&zcksum, sizeof (zio_cksum_t)); - fletcher_4_incremental_byteswap(&drr, sizeof (drr), &zcksum); + (void) fletcher_4_incremental_byteswap(&drr, + sizeof (drr), &zcksum); flags->byteswap = B_TRUE; drr.drr_type = BSWAP_32(drr.drr_type); From owner-svn-src-vendor@freebsd.org Fri May 26 12:14:00 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06A88D83ECE; Fri, 26 May 2017 12:14:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C8DB418D2; Fri, 26 May 2017 12:13:59 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QCDwbm097344; Fri, 26 May 2017 12:13:58 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QCDw29097336; Fri, 26 May 2017 12:13:58 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261213.v4QCDw29097336@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:13:58 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318947 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor-sys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:14:00 -0000 Author: avg Date: Fri May 26 12:13:58 2017 New Revision: 318947 URL: https://svnweb.freebsd.org/changeset/base/318947 Log: 7578 Fix/improve some aspects of ZIL writing. illumos/illumos-gate@c5ee46810f82e8a53d2cc5a487568a573f449039 https://github.com/illumos/illumos-gate/commit/c5ee46810f82e8a53d2cc5a487568a573f449039 https://www.illumos.org/issues/7578 After some ZIL changes 6 years ago zil_slog_limit got partially broken due to zl_itx_list_sz not updated when async itx'es upgraded to sync. Actually because of other changes about that time zl_itx_list_sz is not really required to implement the functionality, so this patch removes some unneeded broken code and variables. Original idea of zil_slog_limit was to reduce chance of SLOG abuse by single heavy logger, that increased latency for other (more latency critical) loggers, by pushing heavy log out into the main pool instead of SLOG. Beside huge latency increase for heavy writers, this implementation caused double write of all data, since the log records were explicitly prepared for SLOG. Since we now have I/O scheduler, I've found it can be much more efficient to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG. Existing ZIL implementation had problem with space efficiency when it has to write large chunks of data into log blocks of limited size. In some cases efficiency stopped to almost as low as 50%. In case of ZIL stored on spinning rust, that also reduced log write speed in half, since head had to uselessly fly over allocated but not written areas. This change improves the situation by offloading problematic operations from z*_log_write() to zil_lwb_commit(), which knows real situation of log blocks allocation and can split large requests into pieces much more efficiently. Also as side effect it removes one of two data copy operations done by ZIL code WR_COPIED case. While there, untangle and unify code of z*_log_write() functions. Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing block boundary, that may also improve efficiency if ZPL is made to do that. Reviewed by: Matthew Ahrens Reviewed by: Prakash Surya Reviewed by: Andriy Gapon Reviewed by: Steven Hartland Reviewed by: Brad Lewis Reviewed by: Richard Elling Approved by: Robert Mustacchi Author: Alexander Motin Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_log.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Changes in other areas also in this revision: Modified: vendor/illumos/dist/cmd/ztest/ztest.c Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h Fri May 26 12:13:27 2017 (r318946) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h Fri May 26 12:13:58 2017 (r318947) @@ -378,7 +378,6 @@ typedef struct itx { void *itx_private; /* type-specific opaque data */ itx_wr_state_t itx_wr_state; /* write state */ uint8_t itx_sync; /* synchronous transaction */ - uint64_t itx_sod; /* record size on disk */ uint64_t itx_oid; /* object id */ lr_t itx_lr; /* common part of log record */ /* followed by type-specific part of lr_xx_t and its immediate data */ Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h Fri May 26 12:13:27 2017 (r318946) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h Fri May 26 12:13:58 2017 (r318947) @@ -42,6 +42,7 @@ extern "C" { typedef struct lwb { zilog_t *lwb_zilog; /* back pointer to log struct */ blkptr_t lwb_blk; /* on disk address of this log blk */ + boolean_t lwb_slog; /* lwb_blk is on SLOG device */ int lwb_nused; /* # used bytes in buffer */ int lwb_sz; /* size of block and buffer */ char *lwb_buf; /* log write buffer */ @@ -62,7 +63,6 @@ typedef struct itxs { typedef struct itxg { kmutex_t itxg_lock; /* lock for this structure */ uint64_t itxg_txg; /* txg for this chain */ - uint64_t itxg_sod; /* total size on disk for this txg */ itxs_t *itxg_itxs; /* sync and async itxs */ } itxg_t; @@ -120,7 +120,6 @@ struct zilog { kcondvar_t zl_cv_batch[2]; /* batch condition variables */ itxg_t zl_itxg[TXG_SIZE]; /* intent log txg chains */ list_t zl_itx_commit_list; /* itx list to be committed */ - uint64_t zl_itx_list_sz; /* total size of records on list */ uint64_t zl_cur_used; /* current commit log size used */ list_t zl_lwb_list; /* in-flight log write list */ kmutex_t zl_vdev_lock; /* protects zl_vdev_tree */ @@ -140,9 +139,26 @@ typedef struct zil_bp_node { avl_node_t zn_node; } zil_bp_node_t; +/* + * Maximum amount of write data that can be put into single log block. + */ #define ZIL_MAX_LOG_DATA (SPA_OLD_MAXBLOCKSIZE - sizeof (zil_chain_t) - \ sizeof (lr_write_t)) +/* + * Maximum amount of log space we agree to waste to reduce number of + * WR_NEED_COPY chunks to reduce zl_get_data() overhead (~12%). + */ +#define ZIL_MAX_WASTE_SPACE (ZIL_MAX_LOG_DATA / 8) + +/* + * Maximum amount of write data for WR_COPIED. Fall back to WR_NEED_COPY + * as more space efficient if we can't fit at least two log records into + * maximum sized log block. + */ +#define ZIL_MAX_COPIED_DATA ((SPA_OLD_MAXBLOCKSIZE - \ + sizeof (zil_chain_t)) / 2 - sizeof (lr_write_t)) + #ifdef __cplusplus } #endif Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h Fri May 26 12:13:27 2017 (r318946) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h Fri May 26 12:13:58 2017 (r318947) @@ -505,7 +505,7 @@ extern zio_t *zio_free_sync(zio_t *pio, const blkptr_t *bp, enum zio_flag flags); extern int zio_alloc_zil(spa_t *spa, uint64_t txg, blkptr_t *new_bp, - blkptr_t *old_bp, uint64_t size, boolean_t use_slog); + blkptr_t *old_bp, uint64_t size, boolean_t *slog); extern void zio_free_zil(spa_t *spa, uint64_t txg, blkptr_t *bp); extern void zio_flush(zio_t *zio, vdev_t *vd); extern void zio_shrink(zio_t *zio, uint64_t size); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_log.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_log.c Fri May 26 12:13:27 2017 (r318946) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_log.c Fri May 26 12:13:58 2017 (r318947) @@ -454,20 +454,17 @@ void zfs_log_write(zilog_t *zilog, dmu_tx_t *tx, int txtype, znode_t *zp, offset_t off, ssize_t resid, int ioflag) { + uint32_t blocksize = zp->z_blksz; itx_wr_state_t write_state; - boolean_t slogging; uintptr_t fsync_cnt; - ssize_t immediate_write_sz; if (zil_replaying(zilog, tx) || zp->z_unlinked) return; - immediate_write_sz = (zilog->zl_logbias == ZFS_LOGBIAS_THROUGHPUT) - ? 0 : zfs_immediate_write_sz; - - slogging = spa_has_slogs(zilog->zl_spa) && - (zilog->zl_logbias == ZFS_LOGBIAS_LATENCY); - if (resid > immediate_write_sz && !slogging && resid <= zp->z_blksz) + if (zilog->zl_logbias == ZFS_LOGBIAS_THROUGHPUT) + write_state = WR_INDIRECT; + else if (!spa_has_slogs(zilog->zl_spa) && + resid >= zfs_immediate_write_sz) write_state = WR_INDIRECT; else if (ioflag & (FSYNC | FDSYNC)) write_state = WR_COPIED; @@ -481,30 +478,26 @@ zfs_log_write(zilog_t *zilog, dmu_tx_t * while (resid) { itx_t *itx; lr_write_t *lr; - ssize_t len; + itx_wr_state_t wr_state = write_state; + ssize_t len = resid; - /* - * If the write would overflow the largest block then split it. - */ - if (write_state != WR_INDIRECT && resid > ZIL_MAX_LOG_DATA) - len = SPA_OLD_MAXBLOCKSIZE >> 1; - else - len = resid; + if (wr_state == WR_COPIED && resid > ZIL_MAX_COPIED_DATA) + wr_state = WR_NEED_COPY; + else if (wr_state == WR_INDIRECT) + len = MIN(blocksize - P2PHASE(off, blocksize), resid); itx = zil_itx_create(txtype, sizeof (*lr) + - (write_state == WR_COPIED ? len : 0)); + (wr_state == WR_COPIED ? len : 0)); lr = (lr_write_t *)&itx->itx_lr; - if (write_state == WR_COPIED && dmu_read(zp->z_zfsvfs->z_os, + if (wr_state == WR_COPIED && dmu_read(zp->z_zfsvfs->z_os, zp->z_id, off, len, lr + 1, DMU_READ_NO_PREFETCH) != 0) { zil_itx_destroy(itx); itx = zil_itx_create(txtype, sizeof (*lr)); lr = (lr_write_t *)&itx->itx_lr; - write_state = WR_NEED_COPY; + wr_state = WR_NEED_COPY; } - itx->itx_wr_state = write_state; - if (write_state == WR_NEED_COPY) - itx->itx_sod += len; + itx->itx_wr_state = wr_state; lr->lr_foid = zp->z_id; lr->lr_offset = off; lr->lr_length = len; Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Fri May 26 12:13:27 2017 (r318946) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c Fri May 26 12:13:58 2017 (r318947) @@ -79,6 +79,13 @@ int zil_replay_disable = 0; */ boolean_t zfs_nocacheflush = B_FALSE; +/* + * Limit SLOG write size per commit executed with synchronous priority. + * Any writes above that will be executed with lower (asynchronous) priority + * to limit potential SLOG device abuse by single active ZIL writer. + */ +uint64_t zil_slog_bulk = 768 * 1024; + static kmem_cache_t *zil_lwb_cache; static void zil_async_to_sync(zilog_t *zilog, uint64_t foid); @@ -430,13 +437,14 @@ zil_free_log_record(zilog_t *zilog, lr_t } static lwb_t * -zil_alloc_lwb(zilog_t *zilog, blkptr_t *bp, uint64_t txg) +zil_alloc_lwb(zilog_t *zilog, blkptr_t *bp, boolean_t slog, uint64_t txg) { lwb_t *lwb; lwb = kmem_cache_alloc(zil_lwb_cache, KM_SLEEP); lwb->lwb_zilog = zilog; lwb->lwb_blk = *bp; + lwb->lwb_slog = slog; lwb->lwb_buf = zio_buf_alloc(BP_GET_LSIZE(bp)); lwb->lwb_max_txg = txg; lwb->lwb_zio = NULL; @@ -520,6 +528,7 @@ zil_create(zilog_t *zilog) dmu_tx_t *tx = NULL; blkptr_t blk; int error = 0; + boolean_t slog = FALSE; /* * Wait for any previous destroy to complete. @@ -548,7 +557,7 @@ zil_create(zilog_t *zilog) } error = zio_alloc_zil(zilog->zl_spa, txg, &blk, NULL, - ZIL_MIN_BLKSZ, zilog->zl_logbias == ZFS_LOGBIAS_LATENCY); + ZIL_MIN_BLKSZ, &slog); if (error == 0) zil_init_log_chain(zilog, &blk); @@ -558,7 +567,7 @@ zil_create(zilog_t *zilog) * Allocate a log write buffer (lwb) for the first log block. */ if (error == 0) - lwb = zil_alloc_lwb(zilog, &blk, txg); + lwb = zil_alloc_lwb(zilog, &blk, slog, txg); /* * If we just allocated the first log block, commit our transaction @@ -890,6 +899,7 @@ static void zil_lwb_write_init(zilog_t *zilog, lwb_t *lwb) { zbookmark_phys_t zb; + zio_priority_t prio; SET_BOOKMARK(&zb, lwb->lwb_blk.blk_cksum.zc_word[ZIL_ZC_OBJSET], ZB_ZIL_OBJECT, ZB_ZIL_LEVEL, @@ -902,9 +912,13 @@ zil_lwb_write_init(zilog_t *zilog, lwb_t if (lwb->lwb_zio == NULL) { abd_t *lwb_abd = abd_get_from_buf(lwb->lwb_buf, BP_GET_LSIZE(&lwb->lwb_blk)); + if (!lwb->lwb_slog || zilog->zl_cur_used <= zil_slog_bulk) + prio = ZIO_PRIORITY_SYNC_WRITE; + else + prio = ZIO_PRIORITY_ASYNC_WRITE; lwb->lwb_zio = zio_rewrite(zilog->zl_root_zio, zilog->zl_spa, 0, &lwb->lwb_blk, lwb_abd, BP_GET_LSIZE(&lwb->lwb_blk), - zil_lwb_write_done, lwb, ZIO_PRIORITY_SYNC_WRITE, + zil_lwb_write_done, lwb, prio, ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE, &zb); } } @@ -924,16 +938,6 @@ uint64_t zil_block_buckets[] = { }; /* - * Use the slog as long as the logbias is 'latency' and the current commit size - * is less than the limit or the total list size is less than 2X the limit. - * Limit checking is disabled by setting zil_slog_limit to UINT64_MAX. - */ -uint64_t zil_slog_limit = 1024 * 1024; -#define USE_SLOG(zilog) (((zilog)->zl_logbias == ZFS_LOGBIAS_LATENCY) && \ - (((zilog)->zl_cur_used < zil_slog_limit) || \ - ((zilog)->zl_itx_list_sz < (zil_slog_limit << 1)))) - -/* * Start a log block write and advance to the next log block. * Calls are serialized. */ @@ -948,6 +952,7 @@ zil_lwb_write_start(zilog_t *zilog, lwb_ uint64_t txg; uint64_t zil_blksz, wsz; int i, error; + boolean_t slog; if (BP_GET_CHECKSUM(&lwb->lwb_blk) == ZIO_CHECKSUM_ZILOG2) { zilc = (zil_chain_t *)lwb->lwb_buf; @@ -1004,8 +1009,7 @@ zil_lwb_write_start(zilog_t *zilog, lwb_ BP_ZERO(bp); /* pass the old blkptr in order to spread log blocks across devs */ - error = zio_alloc_zil(spa, txg, bp, &lwb->lwb_blk, zil_blksz, - USE_SLOG(zilog)); + error = zio_alloc_zil(spa, txg, bp, &lwb->lwb_blk, zil_blksz, &slog); if (error == 0) { ASSERT3U(bp->blk_birth, ==, txg); bp->blk_cksum = lwb->lwb_blk.blk_cksum; @@ -1014,7 +1018,7 @@ zil_lwb_write_start(zilog_t *zilog, lwb_ /* * Allocate a new log write buffer (lwb). */ - nlwb = zil_alloc_lwb(zilog, bp, txg); + nlwb = zil_alloc_lwb(zilog, bp, slog, txg); /* Record the block for later vdev flushing */ zil_add_block(zilog, &lwb->lwb_blk); @@ -1051,45 +1055,53 @@ zil_lwb_write_start(zilog_t *zilog, lwb_ static lwb_t * zil_lwb_commit(zilog_t *zilog, itx_t *itx, lwb_t *lwb) { - lr_t *lrc = &itx->itx_lr; /* common log record */ - lr_write_t *lrw = (lr_write_t *)lrc; + lr_t *lrcb, *lrc; + lr_write_t *lrwb, *lrw; char *lr_buf; - uint64_t txg = lrc->lrc_txg; - uint64_t reclen = lrc->lrc_reclen; - uint64_t dlen = 0; + uint64_t dlen, dnow, lwb_sp, reclen, txg; if (lwb == NULL) return (NULL); ASSERT(lwb->lwb_buf != NULL); - if (lrc->lrc_txtype == TX_WRITE && itx->itx_wr_state == WR_NEED_COPY) + lrc = &itx->itx_lr; /* Common log record inside itx. */ + lrw = (lr_write_t *)lrc; /* Write log record inside itx. */ + if (lrc->lrc_txtype == TX_WRITE && itx->itx_wr_state == WR_NEED_COPY) { dlen = P2ROUNDUP_TYPED( lrw->lr_length, sizeof (uint64_t), uint64_t); - + } else { + dlen = 0; + } + reclen = lrc->lrc_reclen; zilog->zl_cur_used += (reclen + dlen); + txg = lrc->lrc_txg; zil_lwb_write_init(zilog, lwb); +cont: /* * If this record won't fit in the current log block, start a new one. + * For WR_NEED_COPY optimize layout for minimal number of chunks. */ - if (lwb->lwb_nused + reclen + dlen > lwb->lwb_sz) { + lwb_sp = lwb->lwb_sz - lwb->lwb_nused; + if (reclen > lwb_sp || (reclen + dlen > lwb_sp && + lwb_sp < ZIL_MAX_WASTE_SPACE && (dlen % ZIL_MAX_LOG_DATA == 0 || + lwb_sp < reclen + dlen % ZIL_MAX_LOG_DATA))) { lwb = zil_lwb_write_start(zilog, lwb); if (lwb == NULL) return (NULL); zil_lwb_write_init(zilog, lwb); ASSERT(LWB_EMPTY(lwb)); - if (lwb->lwb_nused + reclen + dlen > lwb->lwb_sz) { - txg_wait_synced(zilog->zl_dmu_pool, txg); - return (lwb); - } + lwb_sp = lwb->lwb_sz - lwb->lwb_nused; + ASSERT3U(reclen + MIN(dlen, sizeof (uint64_t)), <=, lwb_sp); } + dnow = MIN(dlen, lwb_sp - reclen); lr_buf = lwb->lwb_buf + lwb->lwb_nused; bcopy(lrc, lr_buf, reclen); - lrc = (lr_t *)lr_buf; - lrw = (lr_write_t *)lrc; + lrcb = (lr_t *)lr_buf; /* Like lrc, but inside lwb. */ + lrwb = (lr_write_t *)lrcb; /* Like lrw, but inside lwb. */ /* * If it's a write, fetch the data or get its blkptr as appropriate. @@ -1101,16 +1113,19 @@ zil_lwb_commit(zilog_t *zilog, itx_t *it char *dbuf; int error; - if (dlen) { - ASSERT(itx->itx_wr_state == WR_NEED_COPY); + if (itx->itx_wr_state == WR_NEED_COPY) { dbuf = lr_buf + reclen; - lrw->lr_common.lrc_reclen += dlen; + lrcb->lrc_reclen += dnow; + if (lrwb->lr_length > dnow) + lrwb->lr_length = dnow; + lrw->lr_offset += dnow; + lrw->lr_length -= dnow; } else { ASSERT(itx->itx_wr_state == WR_INDIRECT); dbuf = NULL; } error = zilog->zl_get_data( - itx->itx_private, lrw, dbuf, lwb->lwb_zio); + itx->itx_private, lrwb, dbuf, lwb->lwb_zio); if (error == EIO) { txg_wait_synced(zilog->zl_dmu_pool, txg); return (lwb); @@ -1129,12 +1144,18 @@ zil_lwb_commit(zilog_t *zilog, itx_t *it * equal to the itx sequence number because not all transactions * are synchronous, and sometimes spa_sync() gets there first. */ - lrc->lrc_seq = ++zilog->zl_lr_seq; /* we are single threaded */ - lwb->lwb_nused += reclen + dlen; + lrcb->lrc_seq = ++zilog->zl_lr_seq; /* we are single threaded */ + lwb->lwb_nused += reclen + dnow; lwb->lwb_max_txg = MAX(lwb->lwb_max_txg, txg); ASSERT3U(lwb->lwb_nused, <=, lwb->lwb_sz); ASSERT0(P2PHASE(lwb->lwb_nused, sizeof (uint64_t))); + dlen -= dnow; + if (dlen > 0) { + zilog->zl_cur_used += reclen; + goto cont; + } + return (lwb); } @@ -1148,7 +1169,6 @@ zil_itx_create(uint64_t txtype, size_t l itx = kmem_alloc(offsetof(itx_t, itx_lr) + lrsize, KM_SLEEP); itx->itx_lr.lrc_txtype = txtype; itx->itx_lr.lrc_reclen = lrsize; - itx->itx_sod = lrsize; /* if write & WR_NEED_COPY will be increased */ itx->itx_lr.lrc_seq = 0; /* defensive */ itx->itx_sync = B_TRUE; /* default is synchronous */ @@ -1299,11 +1319,8 @@ zil_itx_assign(zilog_t *zilog, itx_t *it */ zfs_dbgmsg("zil_itx_assign: missed itx cleanup for " "txg %llu", itxg->itxg_txg); - atomic_add_64(&zilog->zl_itx_list_sz, -itxg->itxg_sod); - itxg->itxg_sod = 0; clean = itxg->itxg_itxs; } - ASSERT(itxg->itxg_sod == 0); itxg->itxg_txg = txg; itxs = itxg->itxg_itxs = kmem_zalloc(sizeof (itxs_t), KM_SLEEP); @@ -1315,8 +1332,6 @@ zil_itx_assign(zilog_t *zilog, itx_t *it } if (itx->itx_sync) { list_insert_tail(&itxs->i_sync_list, itx); - atomic_add_64(&zilog->zl_itx_list_sz, itx->itx_sod); - itxg->itxg_sod += itx->itx_sod; } else { avl_tree_t *t = &itxs->i_async_tree; uint64_t foid = ((lr_ooo_t *)&itx->itx_lr)->lr_foid; @@ -1364,8 +1379,6 @@ zil_clean(zilog_t *zilog, uint64_t synce ASSERT3U(itxg->itxg_txg, <=, synced_txg); ASSERT(itxg->itxg_txg != 0); ASSERT(zilog->zl_clean_taskq != NULL); - atomic_add_64(&zilog->zl_itx_list_sz, -itxg->itxg_sod); - itxg->itxg_sod = 0; clean_me = itxg->itxg_itxs; itxg->itxg_itxs = NULL; itxg->itxg_txg = 0; @@ -1389,7 +1402,6 @@ zil_get_commit_list(zilog_t *zilog) { uint64_t otxg, txg; list_t *commit_list = &zilog->zl_itx_commit_list; - uint64_t push_sod = 0; if (spa_freeze_txg(zilog->zl_spa) != UINT64_MAX) /* ziltest support */ otxg = ZILTEST_TXG; @@ -1421,12 +1433,9 @@ zil_get_commit_list(zilog_t *zilog) ASSERT(zilog_is_dirty_in_txg(zilog, txg) || spa_freeze_txg(zilog->zl_spa) != UINT64_MAX); list_move_tail(commit_list, &itxg->itxg_itxs->i_sync_list); - push_sod += itxg->itxg_sod; - itxg->itxg_sod = 0; mutex_exit(&itxg->itxg_lock); } - atomic_add_64(&zilog->zl_itx_list_sz, -push_sod); } /* Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Fri May 26 12:13:27 2017 (r318946) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c Fri May 26 12:13:58 2017 (r318947) @@ -2915,7 +2915,7 @@ zio_dva_unallocate(zio_t *zio, zio_gang_ */ int zio_alloc_zil(spa_t *spa, uint64_t txg, blkptr_t *new_bp, blkptr_t *old_bp, - uint64_t size, boolean_t use_slog) + uint64_t size, boolean_t *slog) { int error = 1; zio_alloc_list_t io_alloc_list; @@ -2923,17 +2923,16 @@ zio_alloc_zil(spa_t *spa, uint64_t txg, ASSERT(txg > spa_syncing_txg(spa)); metaslab_trace_init(&io_alloc_list); - - if (use_slog) { - error = metaslab_alloc(spa, spa_log_class(spa), size, - new_bp, 1, txg, old_bp, METASLAB_HINTBP_AVOID, - &io_alloc_list, NULL); - } - - if (error) { + error = metaslab_alloc(spa, spa_log_class(spa), size, new_bp, 1, + txg, old_bp, METASLAB_HINTBP_AVOID, &io_alloc_list, NULL); + if (error == 0) { + *slog = TRUE; + } else { error = metaslab_alloc(spa, spa_normal_class(spa), size, new_bp, 1, txg, old_bp, METASLAB_HINTBP_AVOID, &io_alloc_list, NULL); + if (error == 0) + *slog = FALSE; } metaslab_trace_fini(&io_alloc_list); Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c ============================================================================== --- vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Fri May 26 12:13:27 2017 (r318946) +++ vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Fri May 26 12:13:58 2017 (r318947) @@ -1058,54 +1058,44 @@ zvol_log_write(zvol_state_t *zv, dmu_tx_ { uint32_t blocksize = zv->zv_volblocksize; zilog_t *zilog = zv->zv_zilog; - boolean_t slogging; - ssize_t immediate_write_sz; + itx_wr_state_t write_state; if (zil_replaying(zilog, tx)) return; - immediate_write_sz = (zilog->zl_logbias == ZFS_LOGBIAS_THROUGHPUT) - ? 0 : zvol_immediate_write_sz; - - slogging = spa_has_slogs(zilog->zl_spa) && - (zilog->zl_logbias == ZFS_LOGBIAS_LATENCY); + if (zilog->zl_logbias == ZFS_LOGBIAS_THROUGHPUT) + write_state = WR_INDIRECT; + else if (!spa_has_slogs(zilog->zl_spa) && + resid >= blocksize && blocksize > zvol_immediate_write_sz) + write_state = WR_INDIRECT; + else if (sync) + write_state = WR_COPIED; + else + write_state = WR_NEED_COPY; while (resid) { itx_t *itx; lr_write_t *lr; - ssize_t len; - itx_wr_state_t write_state; + itx_wr_state_t wr_state = write_state; + ssize_t len = resid; - /* - * Unlike zfs_log_write() we can be called with - * upto DMU_MAX_ACCESS/2 (5MB) writes. - */ - if (blocksize > immediate_write_sz && !slogging && - resid >= blocksize && off % blocksize == 0) { - write_state = WR_INDIRECT; /* uses dmu_sync */ - len = blocksize; - } else if (sync) { - write_state = WR_COPIED; - len = MIN(ZIL_MAX_LOG_DATA, resid); - } else { - write_state = WR_NEED_COPY; - len = MIN(ZIL_MAX_LOG_DATA, resid); - } + if (wr_state == WR_COPIED && resid > ZIL_MAX_COPIED_DATA) + wr_state = WR_NEED_COPY; + else if (wr_state == WR_INDIRECT) + len = MIN(blocksize - P2PHASE(off, blocksize), resid); itx = zil_itx_create(TX_WRITE, sizeof (*lr) + - (write_state == WR_COPIED ? len : 0)); + (wr_state == WR_COPIED ? len : 0)); lr = (lr_write_t *)&itx->itx_lr; - if (write_state == WR_COPIED && dmu_read(zv->zv_objset, + if (wr_state == WR_COPIED && dmu_read(zv->zv_objset, ZVOL_OBJ, off, len, lr + 1, DMU_READ_NO_PREFETCH) != 0) { zil_itx_destroy(itx); itx = zil_itx_create(TX_WRITE, sizeof (*lr)); lr = (lr_write_t *)&itx->itx_lr; - write_state = WR_NEED_COPY; + wr_state = WR_NEED_COPY; } - itx->itx_wr_state = write_state; - if (write_state == WR_NEED_COPY) - itx->itx_sod += len; + itx->itx_wr_state = wr_state; lr->lr_foid = ZVOL_OBJ; lr->lr_offset = off; lr->lr_length = len; From owner-svn-src-vendor@freebsd.org Fri May 26 12:14:00 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21C3BD83ECF; Fri, 26 May 2017 12:14:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F336718D4; Fri, 26 May 2017 12:13:59 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v4QCDxHj097350; Fri, 26 May 2017 12:13:59 GMT (envelope-from avg@FreeBSD.org) Received: (from avg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v4QCDxT5097349; Fri, 26 May 2017 12:13:59 GMT (envelope-from avg@FreeBSD.org) Message-Id: <201705261213.v4QCDxT5097349@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: avg set sender to avg@FreeBSD.org using -f From: Andriy Gapon Date: Fri, 26 May 2017 12:13:59 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r318947 - vendor-sys/illumos/dist/uts/common/fs/zfs vendor-sys/illumos/dist/uts/common/fs/zfs/sys vendor/illumos/dist/cmd/ztest X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 May 2017 12:14:00 -0000 Author: avg Date: Fri May 26 12:13:58 2017 New Revision: 318947 URL: https://svnweb.freebsd.org/changeset/base/318947 Log: 7578 Fix/improve some aspects of ZIL writing. illumos/illumos-gate@c5ee46810f82e8a53d2cc5a487568a573f449039 https://github.com/illumos/illumos-gate/commit/c5ee46810f82e8a53d2cc5a487568a573f449039 https://www.illumos.org/issues/7578 After some ZIL changes 6 years ago zil_slog_limit got partially broken due to zl_itx_list_sz not updated when async itx'es upgraded to sync. Actually because of other changes about that time zl_itx_list_sz is not really required to implement the functionality, so this patch removes some unneeded broken code and variables. Original idea of zil_slog_limit was to reduce chance of SLOG abuse by single heavy logger, that increased latency for other (more latency critical) loggers, by pushing heavy log out into the main pool instead of SLOG. Beside huge latency increase for heavy writers, this implementation caused double write of all data, since the log records were explicitly prepared for SLOG. Since we now have I/O scheduler, I've found it can be much more efficient to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG. Existing ZIL implementation had problem with space efficiency when it has to write large chunks of data into log blocks of limited size. In some cases efficiency stopped to almost as low as 50%. In case of ZIL stored on spinning rust, that also reduced log write speed in half, since head had to uselessly fly over allocated but not written areas. This change improves the situation by offloading problematic operations from z*_log_write() to zil_lwb_commit(), which knows real situation of log blocks allocation and can split large requests into pieces much more efficiently. Also as side effect it removes one of two data copy operations done by ZIL code WR_COPIED case. While there, untangle and unify code of z*_log_write() functions. Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing block boundary, that may also improve efficiency if ZPL is made to do that. Reviewed by: Matthew Ahrens Reviewed by: Prakash Surya Reviewed by: Andriy Gapon Reviewed by: Steven Hartland Reviewed by: Brad Lewis Reviewed by: Richard Elling Approved by: Robert Mustacchi Author: Alexander Motin Modified: vendor/illumos/dist/cmd/ztest/ztest.c Changes in other areas also in this revision: Modified: vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zil_impl.h vendor-sys/illumos/dist/uts/common/fs/zfs/sys/zio.h vendor-sys/illumos/dist/uts/common/fs/zfs/zfs_log.c vendor-sys/illumos/dist/uts/common/fs/zfs/zil.c vendor-sys/illumos/dist/uts/common/fs/zfs/zio.c vendor-sys/illumos/dist/uts/common/fs/zfs/zvol.c Modified: vendor/illumos/dist/cmd/ztest/ztest.c ============================================================================== --- vendor/illumos/dist/cmd/ztest/ztest.c Fri May 26 12:13:27 2017 (r318946) +++ vendor/illumos/dist/cmd/ztest/ztest.c Fri May 26 12:13:58 2017 (r318947) @@ -1377,7 +1377,6 @@ ztest_log_write(ztest_ds_t *zd, dmu_tx_t itx->itx_private = zd; itx->itx_wr_state = write_state; itx->itx_sync = (ztest_random(8) == 0); - itx->itx_sod += (write_state == WR_NEED_COPY ? lr->lr_length : 0); bcopy(&lr->lr_common + 1, &itx->itx_lr + 1, sizeof (*lr) - sizeof (lr_t));