From owner-svn-src-vendor@freebsd.org Mon Jun 26 20:33:48 2017 Return-Path: Delivered-To: svn-src-vendor@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 589FFD914F1; Mon, 26 Jun 2017 20:33:48 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D830365032; Mon, 26 Jun 2017 20:33:47 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v5QKXlO2099152; Mon, 26 Jun 2017 20:33:47 GMT (envelope-from dim@FreeBSD.org) Received: (from dim@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v5QKXj8a099136; Mon, 26 Jun 2017 20:33:45 GMT (envelope-from dim@FreeBSD.org) Message-Id: <201706262033.v5QKXj8a099136@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: dim set sender to dim@FreeBSD.org using -f From: Dimitry Andric Date: Mon, 26 Jun 2017 20:33:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-vendor@freebsd.org Subject: svn commit: r320382 - in vendor/lld/dist: . COFF ELF ELF/Arch docs lib/Driver lib/ReaderWriter/MachO test test/COFF test/COFF/Inputs test/ELF test/ELF/Inputs test/ELF/linkerscript test/ELF/lto test... X-SVN-Group: vendor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-vendor@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the vendor work area tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jun 2017 20:33:48 -0000 Author: dim Date: Mon Jun 26 20:33:45 2017 New Revision: 320382 URL: https://svnweb.freebsd.org/changeset/base/320382 Log: Vendor import of lld trunk r306325: https://llvm.org/svn/llvm-project/lld/trunk@306325 Added: vendor/lld/dist/ELF/Arch/MipsArchTree.cpp (contents, props changed) vendor/lld/dist/test/COFF/Inputs/associative-comdat-2.s (contents, props changed) vendor/lld/dist/test/COFF/Inputs/pdb_comdat_bar.yaml vendor/lld/dist/test/COFF/Inputs/pdb_comdat_main.yaml vendor/lld/dist/test/COFF/Inputs/pdb_lines_1.yaml vendor/lld/dist/test/COFF/Inputs/pdb_lines_2.yaml vendor/lld/dist/test/COFF/associative-comdat.s (contents, props changed) vendor/lld/dist/test/COFF/guardcf.test vendor/lld/dist/test/COFF/pdb-comdat.test vendor/lld/dist/test/COFF/pdb-safeseh.yaml vendor/lld/dist/test/COFF/pdb-secrel-absolute.yaml vendor/lld/dist/test/COFF/pdb-source-lines.test vendor/lld/dist/test/COFF/pdb-symbol-types.yaml vendor/lld/dist/test/COFF/safeseh-diag-feat.test vendor/lld/dist/test/COFF/safeseh.s (contents, props changed) vendor/lld/dist/test/COFF/secidx-absolute.s (contents, props changed) vendor/lld/dist/test/COFF/secrel-absolute.s (contents, props changed) vendor/lld/dist/test/COFF/secrel-common.s (contents, props changed) vendor/lld/dist/test/ELF/Inputs/exclude-libs.s (contents, props changed) vendor/lld/dist/test/ELF/exclude-libs.s (contents, props changed) vendor/lld/dist/test/ELF/global-offset-table-position-aarch64.s (contents, props changed) vendor/lld/dist/test/ELF/global-offset-table-position-arm.s (contents, props changed) vendor/lld/dist/test/ELF/global-offset-table-position-i386.s (contents, props changed) vendor/lld/dist/test/ELF/global-offset-table-position-mips.s (contents, props changed) vendor/lld/dist/test/ELF/global-offset-table-position.s (contents, props changed) vendor/lld/dist/test/ELF/invalid-z.s (contents, props changed) vendor/lld/dist/test/ELF/linkerscript/discard-section-err.s (contents, props changed) vendor/lld/dist/test/ELF/linkerscript/segment-none.s (contents, props changed) vendor/lld/dist/test/ELF/linkerscript/ttext-script.s (contents, props changed) Deleted: vendor/lld/dist/ELF/Mips.cpp vendor/lld/dist/test/COFF/safeseh.test Modified: vendor/lld/dist/COFF/Chunks.cpp vendor/lld/dist/COFF/Chunks.h vendor/lld/dist/COFF/Config.h vendor/lld/dist/COFF/Driver.cpp vendor/lld/dist/COFF/Driver.h vendor/lld/dist/COFF/DriverUtils.cpp vendor/lld/dist/COFF/InputFiles.cpp vendor/lld/dist/COFF/Options.td vendor/lld/dist/COFF/PDB.cpp vendor/lld/dist/COFF/Strings.cpp vendor/lld/dist/COFF/SymbolTable.cpp vendor/lld/dist/COFF/SymbolTable.h vendor/lld/dist/COFF/Symbols.cpp vendor/lld/dist/COFF/Symbols.h vendor/lld/dist/COFF/Writer.cpp vendor/lld/dist/ELF/Arch/AArch64.cpp vendor/lld/dist/ELF/Arch/AMDGPU.cpp vendor/lld/dist/ELF/Arch/ARM.cpp vendor/lld/dist/ELF/Arch/AVR.cpp vendor/lld/dist/ELF/Arch/Mips.cpp vendor/lld/dist/ELF/Arch/PPC.cpp vendor/lld/dist/ELF/Arch/PPC64.cpp vendor/lld/dist/ELF/Arch/X86.cpp vendor/lld/dist/ELF/Arch/X86_64.cpp vendor/lld/dist/ELF/CMakeLists.txt vendor/lld/dist/ELF/Config.h vendor/lld/dist/ELF/Driver.cpp vendor/lld/dist/ELF/Driver.h vendor/lld/dist/ELF/DriverUtils.cpp vendor/lld/dist/ELF/InputFiles.cpp vendor/lld/dist/ELF/InputFiles.h vendor/lld/dist/ELF/InputSection.cpp vendor/lld/dist/ELF/LinkerScript.cpp vendor/lld/dist/ELF/MarkLive.cpp vendor/lld/dist/ELF/Options.td vendor/lld/dist/ELF/OutputSections.cpp vendor/lld/dist/ELF/Relocations.cpp vendor/lld/dist/ELF/SymbolTable.cpp vendor/lld/dist/ELF/SymbolTable.h vendor/lld/dist/ELF/Symbols.cpp vendor/lld/dist/ELF/Symbols.h vendor/lld/dist/ELF/Target.cpp vendor/lld/dist/ELF/Target.h vendor/lld/dist/ELF/Writer.cpp vendor/lld/dist/README.md vendor/lld/dist/docs/windows_support.rst vendor/lld/dist/lib/Driver/DarwinLdDriver.cpp vendor/lld/dist/lib/ReaderWriter/MachO/MachONormalizedFileToAtoms.cpp vendor/lld/dist/test/COFF/Inputs/pdb1.yaml vendor/lld/dist/test/COFF/def-name.test vendor/lld/dist/test/COFF/dll.test vendor/lld/dist/test/COFF/dllimport-gc.test vendor/lld/dist/test/COFF/hello32.test vendor/lld/dist/test/COFF/icf-associative.test vendor/lld/dist/test/COFF/manifestinput.test vendor/lld/dist/test/COFF/noentry.test vendor/lld/dist/test/COFF/out.test vendor/lld/dist/test/COFF/pdb-lib.s vendor/lld/dist/test/COFF/pdb.test vendor/lld/dist/test/COFF/resource.test vendor/lld/dist/test/ELF/arm-gnu-ifunc-plt.s vendor/lld/dist/test/ELF/arm-got-relative.s vendor/lld/dist/test/ELF/arm-thumb-branch.s vendor/lld/dist/test/ELF/arm-thumb-plt-reloc.s vendor/lld/dist/test/ELF/defsym.s vendor/lld/dist/test/ELF/global_offset_table_shared.s vendor/lld/dist/test/ELF/linkerscript/sections.s vendor/lld/dist/test/ELF/lto/cache.ll vendor/lld/dist/test/ELF/lto/wrap-2.ll vendor/lld/dist/test/ELF/mips-got16-relocatable.s vendor/lld/dist/test/ELF/mips-npic-call-pic-os.s vendor/lld/dist/test/ELF/mips-npic-call-pic-script.s vendor/lld/dist/test/ELF/mips-npic-call-pic.s vendor/lld/dist/test/ELF/mips-plt-r6.s vendor/lld/dist/test/ELF/wrap.s vendor/lld/dist/test/lit.cfg vendor/lld/dist/test/mach-o/executable-exports.yaml Modified: vendor/lld/dist/COFF/Chunks.cpp ============================================================================== --- vendor/lld/dist/COFF/Chunks.cpp Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Chunks.cpp Mon Jun 26 20:33:45 2017 (r320382) @@ -37,8 +37,14 @@ SectionChunk::SectionChunk(ObjectFile *F, const coff_s Align = Header->getAlignment(); - // Only COMDAT sections are subject of dead-stripping. - Live = !isCOMDAT(); + // Chunks may be discarded during comdat merging. + Discarded = false; + + // If linker GC is disabled, every chunk starts out alive. If linker GC is + // enabled, treat non-comdat sections as roots. Generally optimized object + // files will be built with -ffunction-sections or /Gy, so most things worth + // stripping will be in a comdat. + Live = !Config->DoGC || !isCOMDAT(); } static void add16(uint8_t *P, int16_t V) { write16le(P, read16le(P) + V); } @@ -46,6 +52,15 @@ static void add32(uint8_t *P, int32_t V) { write32le(P static void add64(uint8_t *P, int64_t V) { write64le(P, read64le(P) + V); } static void or16(uint8_t *P, uint16_t V) { write16le(P, read16le(P) | V); } +static void applySecRel(const SectionChunk *Sec, uint8_t *Off, Defined *Sym) { + // Don't apply section relative relocations to absolute symbols in codeview + // debug info sections. MSVC does not treat such relocations as fatal errors, + // and they can be found in the standard library for linker-provided symbols + // like __guard_fids_table and __safe_se_handler_table. + if (!(isa(Sym) && Sec->isCodeView())) + add32(Off, Sym->getSecrel()); +} + void SectionChunk::applyRelX64(uint8_t *Off, uint16_t Type, Defined *Sym, uint64_t P) const { uint64_t S = Sym->getRVA(); @@ -60,7 +75,7 @@ void SectionChunk::applyRelX64(uint8_t *Off, uint16_t case IMAGE_REL_AMD64_REL32_4: add32(Off, S - P - 8); break; case IMAGE_REL_AMD64_REL32_5: add32(Off, S - P - 9); break; case IMAGE_REL_AMD64_SECTION: add16(Off, Sym->getSectionIndex()); break; - case IMAGE_REL_AMD64_SECREL: add32(Off, Sym->getSecrel()); break; + case IMAGE_REL_AMD64_SECREL: applySecRel(this, Off, Sym); break; default: fatal("unsupported relocation type 0x" + Twine::utohexstr(Type)); } @@ -75,7 +90,7 @@ void SectionChunk::applyRelX86(uint8_t *Off, uint16_t case IMAGE_REL_I386_DIR32NB: add32(Off, S); break; case IMAGE_REL_I386_REL32: add32(Off, S - P - 4); break; case IMAGE_REL_I386_SECTION: add16(Off, Sym->getSectionIndex()); break; - case IMAGE_REL_I386_SECREL: add32(Off, Sym->getSecrel()); break; + case IMAGE_REL_I386_SECREL: applySecRel(this, Off, Sym); break; default: fatal("unsupported relocation type 0x" + Twine::utohexstr(Type)); } @@ -135,7 +150,7 @@ void SectionChunk::applyRelARM(uint8_t *Off, uint16_t case IMAGE_REL_ARM_BRANCH20T: applyBranch20T(Off, S - P - 4); break; case IMAGE_REL_ARM_BRANCH24T: applyBranch24T(Off, S - P - 4); break; case IMAGE_REL_ARM_BLX23T: applyBranch24T(Off, S - P - 4); break; - case IMAGE_REL_ARM_SECREL: add32(Off, Sym->getSecrel()); break; + case IMAGE_REL_ARM_SECREL: applySecRel(this, Off, Sym); break; default: fatal("unsupported relocation type 0x" + Twine::utohexstr(Type)); } @@ -226,8 +241,12 @@ bool SectionChunk::isCOMDAT() const { void SectionChunk::printDiscardedMessage() const { // Removed by dead-stripping. If it's removed by ICF, ICF already // printed out the name, so don't repeat that here. - if (Sym && this == Repl) - message("Discarded " + Sym->getName()); + if (Sym && this == Repl) { + if (Discarded) + message("Discarded comdat symbol " + Sym->getName()); + else if (!Live) + message("Discarded " + Sym->getName()); + } } StringRef SectionChunk::getDebugName() { Modified: vendor/lld/dist/COFF/Chunks.h ============================================================================== --- vendor/lld/dist/COFF/Chunks.h Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Chunks.h Mon Jun 26 20:33:45 2017 (r320382) @@ -64,7 +64,6 @@ class Chunk { (public) uint64_t getRVA() const { return RVA; } uint32_t getAlign() const { return Align; } void setRVA(uint64_t V) { RVA = V; } - void setOutputSectionOff(uint64_t V) { OutputSectionOff = V; } // Returns true if this has non-zero data. BSS chunks return // false. If false is returned, the space occupied by this chunk @@ -97,17 +96,19 @@ class Chunk { (public) Chunk(Kind K = OtherKind) : ChunkKind(K) {} const Kind ChunkKind; + // The alignment of this chunk. The writer uses the value. + uint32_t Align = 1; + // The RVA of this chunk in the output. The writer sets a value. uint64_t RVA = 0; +public: // The offset from beginning of the output section. The writer sets a value. uint64_t OutputSectionOff = 0; +protected: // The output section for this chunk. OutputSection *Out = nullptr; - - // The alignment of this chunk. The writer uses the value. - uint32_t Align = 1; }; // A chunk corresponding a section of an input file. @@ -159,13 +160,29 @@ class SectionChunk : public Chunk { (public) StringRef getDebugName() override; void setSymbol(DefinedRegular *S) { if (!Sym) Sym = S; } + // Returns true if the chunk was not dropped by GC or COMDAT deduplication. + bool isLive() { return Live && !Discarded; } + // Used by the garbage collector. - bool isLive() { return !Config->DoGC || Live; } void markLive() { + assert(Config->DoGC && "should only mark things live from GC"); assert(!isLive() && "Cannot mark an already live section!"); Live = true; } + // Returns true if this chunk was dropped by COMDAT deduplication. + bool isDiscarded() const { return Discarded; } + + // Used by the SymbolTable when discarding unused comdat sections. This is + // redundant when GC is enabled, as all comdat sections will start out dead. + void markDiscarded() { Discarded = true; } + + // True if this is a codeview debug info chunk. These will not be laid out in + // the image. Instead they will end up in the PDB, if one is requested. + bool isCodeView() const { + return SectionName == ".debug" || SectionName.startswith(".debug$"); + } + // Allow iteration over the bodies of this chunk's relocated symbols. llvm::iterator_range symbols() const { return llvm::make_range(symbol_iterator(File, Relocs.begin()), @@ -195,6 +212,9 @@ class SectionChunk : public Chunk { (public) std::vector AssocChildren; llvm::iterator_range Relocs; size_t NumRelocs; + + // True if this chunk was discarded because it was a duplicate comdat section. + bool Discarded; // Used by the garbage collector. bool Live; Modified: vendor/lld/dist/COFF/Config.h ============================================================================== --- vendor/lld/dist/COFF/Config.h Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Config.h Mon Jun 26 20:33:45 2017 (r320382) @@ -161,9 +161,6 @@ struct Configuration { bool LargeAddressAware = false; bool HighEntropyVA = false; bool AppContainer = false; - - // This is for debugging. - bool DumpPdb = false; }; extern Configuration *Config; Modified: vendor/lld/dist/COFF/Driver.cpp ============================================================================== --- vendor/lld/dist/COFF/Driver.cpp Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Driver.cpp Mon Jun 26 20:33:45 2017 (r320382) @@ -905,7 +905,6 @@ void LinkerDriver::link(ArrayRef ArgsArr Config->TerminalServerAware = false; if (Args.hasArg(OPT_nosymtab)) Config->WriteSymtab = false; - Config->DumpPdb = Args.hasArg(OPT_dumppdb); Config->MapFile = getMapFile(Args); @@ -936,9 +935,9 @@ void LinkerDriver::link(ArrayRef ArgsArr Config->Machine = AMD64; } - // Windows specific -- Input files can be Windows resource files (.res files). - // We invoke cvtres.exe to convert resource files to a regular COFF file - // then link the result file normally. + // Input files can be Windows resource files (.res files). We use + // WindowsResource to convert resource files to a regular COFF file, + // then link the resulting file normally. if (!Resources.empty()) addBuffer(convertResToCOFF(Resources)); @@ -1027,17 +1026,21 @@ void LinkerDriver::link(ArrayRef ArgsArr if (Config->ImageBase == uint64_t(-1)) Config->ImageBase = getDefaultImageBase(); - Symtab.addRelative(mangle("__ImageBase"), 0); + Symtab.addSynthetic(mangle("__ImageBase"), nullptr); if (Config->Machine == I386) { - Config->SEHTable = Symtab.addRelative("___safe_se_handler_table", 0); - Config->SEHCount = Symtab.addAbsolute("___safe_se_handler_count", 0); + Symtab.addAbsolute("___safe_se_handler_table", 0); + Symtab.addAbsolute("___safe_se_handler_count", 0); } // We do not support /guard:cf (control flow protection) yet. // Define CFG symbols anyway so that we can link MSVC 2015 CRT. - Symtab.addAbsolute(mangle("__guard_fids_table"), 0); Symtab.addAbsolute(mangle("__guard_fids_count"), 0); + Symtab.addAbsolute(mangle("__guard_fids_table"), 0); Symtab.addAbsolute(mangle("__guard_flags"), 0x100); + Symtab.addAbsolute(mangle("__guard_iat_count"), 0); + Symtab.addAbsolute(mangle("__guard_iat_table"), 0); + Symtab.addAbsolute(mangle("__guard_longjmp_count"), 0); + Symtab.addAbsolute(mangle("__guard_longjmp_table"), 0); // This code may add new undefined symbols to the link, which may enqueue more // symbol resolution tasks, so we need to continue executing tasks until we Modified: vendor/lld/dist/COFF/Driver.h ============================================================================== --- vendor/lld/dist/COFF/Driver.h Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Driver.h Mon Jun 26 20:33:45 2017 (r320382) @@ -178,7 +178,7 @@ void runMSVCLinker(std::string Rsp, ArrayRef +createMemoryBufferForManifestRes(size_t ManifestSize) { + size_t ResSize = alignTo(object::WIN_RES_MAGIC_SIZE + + object::WIN_RES_NULL_ENTRY_SIZE + + sizeof(object::WinResHeaderPrefix) + + sizeof(object::WinResIDs) + + sizeof(object::WinResHeaderSuffix) + + ManifestSize, + object::WIN_RES_DATA_ALIGNMENT); + return MemoryBuffer::getNewMemBuffer(ResSize); +} + +static void writeResFileHeader(char *&Buf) { + memcpy(Buf, COFF::WinResMagic, sizeof(COFF::WinResMagic)); + Buf += sizeof(COFF::WinResMagic); + memset(Buf, 0, object::WIN_RES_NULL_ENTRY_SIZE); + Buf += object::WIN_RES_NULL_ENTRY_SIZE; +} + +static void writeResEntryHeader(char *&Buf, size_t ManifestSize) { + // Write the prefix. + auto *Prefix = reinterpret_cast(Buf); + Prefix->DataSize = ManifestSize; + Prefix->HeaderSize = sizeof(object::WinResHeaderPrefix) + + sizeof(object::WinResIDs) + + sizeof(object::WinResHeaderSuffix); + Buf += sizeof(object::WinResHeaderPrefix); + + // Write the Type/Name IDs. + auto *IDs = reinterpret_cast(Buf); + IDs->setType(RT_MANIFEST); + IDs->setName(Config->ManifestID); + Buf += sizeof(object::WinResIDs); + + // Write the suffix. + auto *Suffix = reinterpret_cast(Buf); + Suffix->DataVersion = 0; + Suffix->MemoryFlags = object::WIN_RES_PURE_MOVEABLE; + Suffix->Language = SUBLANG_ENGLISH_US; + Suffix->Version = 0; + Suffix->Characteristics = 0; + Buf += sizeof(object::WinResHeaderSuffix); +} + // Create a resource file containing a manifest XML. std::unique_ptr createManifestRes() { - // Create a temporary file for the resource script file. - TemporaryFile RCFile("manifest", "rc"); + std::string Manifest = createManifestXml(); - // Open the temporary file for writing. - std::error_code EC; - raw_fd_ostream Out(RCFile.Path, EC, sys::fs::F_Text); - if (EC) - fatal(EC, "failed to open " + RCFile.Path); + std::unique_ptr Res = + createMemoryBufferForManifestRes(Manifest.size()); - // Write resource script to the RC file. - Out << "#define LANG_ENGLISH 9\n" - << "#define SUBLANG_DEFAULT 1\n" - << "#define APP_MANIFEST " << Config->ManifestID << "\n" - << "#define RT_MANIFEST 24\n" - << "LANGUAGE LANG_ENGLISH, SUBLANG_DEFAULT\n" - << "APP_MANIFEST RT_MANIFEST {\n"; - quoteAndPrint(Out, createManifestXml()); - Out << "}\n"; - Out.close(); + char *Buf = const_cast(Res->getBufferStart()); + writeResFileHeader(Buf); + writeResEntryHeader(Buf, Manifest.size()); - // Create output resource file. - TemporaryFile ResFile("output-resource", "res"); - - Executor E("rc.exe"); - E.add("/fo"); - E.add(ResFile.Path); - E.add("/nologo"); - E.add(RCFile.Path); - E.run(); - return ResFile.getMemoryBuffer(); + // Copy the manifest data into the .res file. + std::copy(Manifest.begin(), Manifest.end(), Buf); + return Res; } void createSideBySideManifest() { @@ -592,40 +604,22 @@ void checkFailIfMismatch(StringRef Arg) { // using cvtres.exe. std::unique_ptr convertResToCOFF(const std::vector &MBs) { - // Create an output file path. - TemporaryFile File("resource-file", "obj"); + object::WindowsResourceParser Parser; - // Execute cvtres.exe. - Executor E("cvtres.exe"); - E.add("/machine:" + machineToStr(Config->Machine)); - E.add("/readonly"); - E.add("/nologo"); - E.add("/out:" + Twine(File.Path)); - - // We must create new files because the memory buffers we have may have no - // underlying file still existing on the disk. - // It happens if it was created from a TemporaryFile, which usually delete - // the file just after creating the MemoryBuffer. - std::vector ResFiles; - ResFiles.reserve(MBs.size()); for (MemoryBufferRef MB : MBs) { - // We store the temporary file in a vector to avoid deletion - // before running cvtres - ResFiles.emplace_back("resource-file", "res"); - TemporaryFile& ResFile = ResFiles.back(); - // Write the content of the resource in a temporary file - std::error_code EC; - raw_fd_ostream OS(ResFile.Path, EC, sys::fs::F_None); - if (EC) - fatal(EC, "failed to open " + ResFile.Path); - OS << MB.getBuffer(); - OS.close(); - - E.add(ResFile.Path); + std::unique_ptr Bin = check(object::createBinary(MB)); + object::WindowsResource *RF = dyn_cast(Bin.get()); + if (!RF) + fatal("cannot compile non-resource file as resource"); + if (auto EC = Parser.parse(RF)) + fatal(EC, "failed to parse .res file"); } - E.run(); - return File.getMemoryBuffer(); + Expected> E = + llvm::object::writeWindowsResourceCOFF(Config->Machine, Parser); + if (!E) + fatal(errorToErrorCode(E.takeError()), "failed to write .res to COFF"); + return std::move(E.get()); } // Run MSVC link.exe for given in-memory object files. @@ -657,11 +651,9 @@ void runMSVCLinker(std::string Rsp, ArrayRefDebug && Name.startswith(".debug")) continue; - // CodeView sections are stored to a different vector because they are - // not linked in the regular manner. - if (Name == ".debug" || Name.startswith(".debug$")) { - DebugChunks.push_back(make(this, Sec)); - continue; - } - if (Sec->Characteristics & llvm::COFF::IMAGE_SCN_LNK_REMOVE) continue; auto *C = make(this, Sec); - Chunks.push_back(C); + + // CodeView sections are stored to a different vector because they are not + // linked in the regular manner. + if (C->isCodeView()) + DebugChunks.push_back(C); + else + Chunks.push_back(C); + SparseChunks[I] = C; } } @@ -249,8 +249,12 @@ SymbolBody *ObjectFile::createDefined(COFFSymbolRef Sy auto *Aux = reinterpret_cast(AuxP); if (Aux->Selection == IMAGE_COMDAT_SELECT_ASSOCIATIVE) if (auto *ParentSC = cast_or_null( - SparseChunks[Aux->getNumber(Sym.isBigObj())])) + SparseChunks[Aux->getNumber(Sym.isBigObj())])) { ParentSC->addAssociative(SC); + // If we already discarded the parent, discard the child. + if (ParentSC->isDiscarded()) + SC->markDiscarded(); + } SC->Checksum = Aux->CheckSum; } Modified: vendor/lld/dist/COFF/Options.td ============================================================================== --- vendor/lld/dist/COFF/Options.td Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Options.td Mon Jun 26 20:33:45 2017 (r320382) @@ -102,7 +102,6 @@ def nosymtab : F<"nosymtab">; def msvclto : F<"msvclto">; // Flags for debugging -def dumppdb : Joined<["/", "-"], "dumppdb">; def lldmap : F<"lldmap">; def lldmap_file : Joined<["/", "-"], "lldmap:">; Modified: vendor/lld/dist/COFF/PDB.cpp ============================================================================== --- vendor/lld/dist/COFF/PDB.cpp Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/PDB.cpp Mon Jun 26 20:33:45 2017 (r320382) @@ -15,10 +15,11 @@ #include "Symbols.h" #include "llvm/DebugInfo/CodeView/CVDebugRecord.h" #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h" +#include "llvm/DebugInfo/CodeView/DebugSubsectionRecord.h" +#include "llvm/DebugInfo/CodeView/DebugSubsectionVisitor.h" #include "llvm/DebugInfo/CodeView/LazyRandomTypeCollection.h" -#include "llvm/DebugInfo/CodeView/SymbolDumper.h" -#include "llvm/DebugInfo/CodeView/TypeDatabase.h" #include "llvm/DebugInfo/CodeView/TypeDumpVisitor.h" +#include "llvm/DebugInfo/CodeView/TypeIndexDiscovery.h" #include "llvm/DebugInfo/CodeView/TypeStreamMerger.h" #include "llvm/DebugInfo/CodeView/TypeTableBuilder.h" #include "llvm/DebugInfo/MSF/MSFBuilder.h" @@ -46,8 +47,6 @@ using namespace lld; using namespace lld::coff; using namespace llvm; using namespace llvm::codeview; -using namespace llvm::support; -using namespace llvm::support::endian; using llvm::object::coff_section; @@ -68,22 +67,24 @@ static SectionChunk *findByName(std::vector getDebugSection(ObjectFile *File, StringRef SecName) { - SectionChunk *Sec = findByName(File->getDebugChunks(), SecName); - if (!Sec) - return {}; - +static ArrayRef consumeDebugMagic(ArrayRef Data, + StringRef SecName) { // First 4 bytes are section magic. - ArrayRef Data = Sec->getContents(); if (Data.size() < 4) fatal(SecName + " too short"); - if (read32le(Data.data()) != COFF::DEBUG_SECTION_MAGIC) + if (support::endian::read32le(Data.data()) != COFF::DEBUG_SECTION_MAGIC) fatal(SecName + " has an invalid magic"); return Data.slice(4); } +static ArrayRef getDebugSection(ObjectFile *File, StringRef SecName) { + if (SectionChunk *Sec = findByName(File->getDebugChunks(), SecName)) + return consumeDebugMagic(Sec->getContents(), SecName); + return {}; +} + static void addTypeInfo(pdb::TpiStreamBuilder &TpiBuilder, - codeview::TypeTableBuilder &TypeTable) { + TypeTableBuilder &TypeTable) { // Start the TPI or IPI stream header. TpiBuilder.setVersionHeader(pdb::PdbTpiV80); @@ -94,17 +95,148 @@ static void addTypeInfo(pdb::TpiStreamBuilder &TpiBuil }); } +static void mergeDebugT(ObjectFile *File, + TypeTableBuilder &IDTable, + TypeTableBuilder &TypeTable, + SmallVectorImpl &TypeIndexMap, + pdb::PDBTypeServerHandler &Handler) { + ArrayRef Data = getDebugSection(File, ".debug$T"); + if (Data.empty()) + return; + + BinaryByteStream Stream(Data, support::little); + CVTypeArray Types; + BinaryStreamReader Reader(Stream); + Handler.addSearchPath(sys::path::parent_path(File->getName())); + if (auto EC = Reader.readArray(Types, Reader.getLength())) + fatal(EC, "Reader::readArray failed"); + if (auto Err = mergeTypeAndIdRecords(IDTable, TypeTable, + TypeIndexMap, &Handler, Types)) + fatal(Err, "codeview::mergeTypeStreams failed"); +} + +static bool remapTypeIndex(TypeIndex &TI, ArrayRef TypeIndexMap) { + if (TI.isSimple()) + return true; + if (TI.toArrayIndex() >= TypeIndexMap.size()) + return false; + TI = TypeIndexMap[TI.toArrayIndex()]; + return true; +} + +static bool remapTypesInSymbolRecord(ObjectFile *File, + MutableArrayRef Contents, + ArrayRef TypeIndexMap, + ArrayRef TypeRefs) { + for (const TiReference &Ref : TypeRefs) { + unsigned ByteSize = Ref.Count * sizeof(TypeIndex); + if (Contents.size() < Ref.Offset + ByteSize) { + log("ignoring short symbol record"); + return false; + } + MutableArrayRef TIs( + reinterpret_cast(Contents.data() + Ref.Offset), Ref.Count); + for (TypeIndex &TI : TIs) + if (!remapTypeIndex(TI, TypeIndexMap)) { + log("ignoring symbol record in " + File->getName() + + " with bad type index 0x" + utohexstr(TI.getIndex())); + return false; + } + } + return true; +} + +/// MSVC translates S_PROC_ID_END to S_END. +uint16_t canonicalizeSymbolKind(SymbolKind Kind) { + if (Kind == SymbolKind::S_PROC_ID_END) + return SymbolKind::S_END; + return Kind; +} + +/// Copy the symbol record. In a PDB, symbol records must be 4 byte aligned. +/// The object file may not be aligned. +static MutableArrayRef copySymbolForPdb(const CVSymbol &Sym, + BumpPtrAllocator &Alloc) { + size_t Size = alignTo(Sym.length(), alignOf(CodeViewContainer::Pdb)); + assert(Size >= 4 && "record too short"); + assert(Size <= MaxRecordLength && "record too long"); + void *Mem = Alloc.Allocate(Size, 4); + + // Copy the symbol record and zero out any padding bytes. + MutableArrayRef NewData(reinterpret_cast(Mem), Size); + memcpy(NewData.data(), Sym.data().data(), Sym.length()); + memset(NewData.data() + Sym.length(), 0, Size - Sym.length()); + + // Update the record prefix length. It should point to the beginning of the + // next record. MSVC does some canonicalization of the record kind, so we do + // that as well. + auto *Prefix = reinterpret_cast(Mem); + Prefix->RecordKind = canonicalizeSymbolKind(Sym.kind()); + Prefix->RecordLen = Size - 2; + return NewData; +} + +static void mergeSymbolRecords(BumpPtrAllocator &Alloc, ObjectFile *File, + ArrayRef TypeIndexMap, + BinaryStreamRef SymData) { + // FIXME: Improve error recovery by warning and skipping records when + // possible. + CVSymbolArray Syms; + BinaryStreamReader Reader(SymData); + ExitOnErr(Reader.readArray(Syms, Reader.getLength())); + for (const CVSymbol &Sym : Syms) { + // Discover type index references in the record. Skip it if we don't know + // where they are. + SmallVector TypeRefs; + if (!discoverTypeIndices(Sym, TypeRefs)) { + log("ignoring unknown symbol record with kind 0x" + utohexstr(Sym.kind())); + continue; + } + + // Copy the symbol record so we can mutate it. + MutableArrayRef NewData = copySymbolForPdb(Sym, Alloc); + + // Re-map all the type index references. + MutableArrayRef Contents = + NewData.drop_front(sizeof(RecordPrefix)); + if (!remapTypesInSymbolRecord(File, Contents, TypeIndexMap, TypeRefs)) + continue; + + // FIXME: Fill in "Parent" and "End" fields by maintaining a stack of + // scopes. + + // Add the symbol to the module. + File->ModuleDBI->addSymbol(CVSymbol(Sym.kind(), NewData)); + } +} + +// Allocate memory for a .debug$S section and relocate it. +static ArrayRef relocateDebugChunk(BumpPtrAllocator &Alloc, + SectionChunk *DebugChunk) { + uint8_t *Buffer = Alloc.Allocate(DebugChunk->getSize()); + assert(DebugChunk->OutputSectionOff == 0 && + "debug sections should not be in output sections"); + DebugChunk->writeTo(Buffer); + return consumeDebugMagic(makeArrayRef(Buffer, DebugChunk->getSize()), + ".debug$S"); +} + // Add all object files to the PDB. Merge .debug$T sections into IpiData and // TpiData. -static void addObjectsToPDB(SymbolTable *Symtab, pdb::PDBFileBuilder &Builder, - codeview::TypeTableBuilder &TypeTable, - codeview::TypeTableBuilder &IDTable) { +static void addObjectsToPDB(BumpPtrAllocator &Alloc, SymbolTable *Symtab, + pdb::PDBFileBuilder &Builder, + TypeTableBuilder &TypeTable, + TypeTableBuilder &IDTable) { // Follow type servers. If the same type server is encountered more than // once for this instance of `PDBTypeServerHandler` (for example if many // object files reference the same TypeServer), the types from the // TypeServer will only be visited once. pdb::PDBTypeServerHandler Handler; + // PDBs use a single global string table for filenames in the file checksum + // table. + auto PDBStrTab = std::make_shared(); + // Visit all .debug$T sections to add them to Builder. for (ObjectFile *File : Symtab->ObjectFiles) { // Add a module descriptor for every object file. We need to put an absolute @@ -118,25 +250,74 @@ static void addObjectsToPDB(SymbolTable *Symtab, pdb:: File->ModuleDBI = &ExitOnErr(Builder.getDbiBuilder().addModuleInfo(Name)); File->ModuleDBI->setObjFileName(Path); - // FIXME: Walk the .debug$S sections and add them. Do things like recording - // source files. + // Before we can process symbol substreams from .debug$S, we need to process + // type information, file checksums, and the string table. Add type info to + // the PDB first, so that we can get the map from object file type and item + // indices to PDB type and item indices. + SmallVector TypeIndexMap; + mergeDebugT(File, IDTable, TypeTable, TypeIndexMap, Handler); - ArrayRef Data = getDebugSection(File, ".debug$T"); - if (Data.empty()) - continue; + // Now do all line info. + for (SectionChunk *DebugChunk : File->getDebugChunks()) { + if (!DebugChunk->isLive() || DebugChunk->getSectionName() != ".debug$S") + continue; - BinaryByteStream Stream(Data, support::little); - codeview::CVTypeArray Types; - BinaryStreamReader Reader(Stream); - SmallVector SourceToDest; - Handler.addSearchPath(llvm::sys::path::parent_path(File->getName())); - if (auto EC = Reader.readArray(Types, Reader.getLength())) - fatal(EC, "Reader::readArray failed"); - if (auto Err = codeview::mergeTypeAndIdRecords( - IDTable, TypeTable, SourceToDest, &Handler, Types)) - fatal(Err, "codeview::mergeTypeStreams failed"); + ArrayRef RelocatedDebugContents = + relocateDebugChunk(Alloc, DebugChunk); + if (RelocatedDebugContents.empty()) + continue; + + DebugSubsectionArray Subsections; + BinaryStreamReader Reader(RelocatedDebugContents, support::little); + ExitOnErr(Reader.readArray(Subsections, RelocatedDebugContents.size())); + + DebugStringTableSubsectionRef CVStrTab; + DebugChecksumsSubsectionRef Checksums; + for (const DebugSubsectionRecord &SS : Subsections) { + switch (SS.kind()) { + case DebugSubsectionKind::StringTable: + ExitOnErr(CVStrTab.initialize(SS.getRecordData())); + break; + case DebugSubsectionKind::FileChecksums: + ExitOnErr(Checksums.initialize(SS.getRecordData())); + break; + case DebugSubsectionKind::Lines: + // We can add the relocated line table directly to the PDB without + // modification because the file checksum offsets will stay the same. + File->ModuleDBI->addDebugSubsection(SS); + break; + case DebugSubsectionKind::Symbols: + mergeSymbolRecords(Alloc, File, TypeIndexMap, SS.getRecordData()); + break; + default: + // FIXME: Process the rest of the subsections. + break; + } + } + + if (Checksums.valid()) { + // Make a new file checksum table that refers to offsets in the PDB-wide + // string table. Generally the string table subsection appears after the + // checksum table, so we have to do this after looping over all the + // subsections. + if (!CVStrTab.valid()) + fatal(".debug$S sections must have both a string table subsection " + "and a checksum subsection table or neither"); + auto NewChecksums = + make_unique(*PDBStrTab); + for (FileChecksumEntry &FC : Checksums) { + StringRef FileName = ExitOnErr(CVStrTab.getString(FC.FileNameOffset)); + ExitOnErr(Builder.getDbiBuilder().addModuleSourceFile( + *File->ModuleDBI, FileName)); + NewChecksums->addChecksum(FileName, FC.Kind, FC.Checksum); + } + File->ModuleDBI->addDebugSubsection(std::move(NewChecksums)); + } + } } + Builder.getStringTableBuilder().setStrings(*PDBStrTab); + // Construct TPI stream contents. addTypeInfo(Builder.getTpiBuilder(), TypeTable); @@ -144,56 +325,10 @@ static void addObjectsToPDB(SymbolTable *Symtab, pdb:: addTypeInfo(Builder.getIpiBuilder(), IDTable); } -static void dumpDebugT(ScopedPrinter &W, ObjectFile *File) { - ListScope LS(W, "DebugT"); - ArrayRef Data = getDebugSection(File, ".debug$T"); - if (Data.empty()) - return; - - LazyRandomTypeCollection Types(Data, 100); - TypeDumpVisitor TDV(Types, &W, false); - // Use a default implementation that does not follow type servers and instead - // just dumps the contents of the TypeServer2 record. - if (auto EC = codeview::visitTypeStream(Types, TDV)) - fatal(EC, "CVTypeDumper::dump failed"); -} - -static void dumpDebugS(ScopedPrinter &W, ObjectFile *File) { - ListScope LS(W, "DebugS"); - ArrayRef Data = getDebugSection(File, ".debug$S"); - if (Data.empty()) - return; - - BinaryByteStream Stream(Data, llvm::support::little); - CVSymbolArray Symbols; - BinaryStreamReader Reader(Stream); - if (auto EC = Reader.readArray(Symbols, Reader.getLength())) - fatal(EC, "StreamReader.readArray failed"); - - TypeDatabase TDB(0); - CVSymbolDumper SymbolDumper(W, TDB, CodeViewContainer::ObjectFile, nullptr, - false); - if (auto EC = SymbolDumper.dump(Symbols)) - fatal(EC, "CVSymbolDumper::dump failed"); -} - -// Dump CodeView debug info. This is for debugging. -static void dumpCodeView(SymbolTable *Symtab) { - ScopedPrinter W(outs()); - - for (ObjectFile *File : Symtab->ObjectFiles) { - dumpDebugT(W, File); - dumpDebugS(W, File); - } -} - // Creates a PDB file. void coff::createPDB(StringRef Path, SymbolTable *Symtab, ArrayRef SectionTable, const llvm::codeview::DebugInfo *DI) { - if (Config->DumpPdb) - dumpCodeView(Symtab); - BumpPtrAllocator Alloc; pdb::PDBFileBuilder Builder(Alloc); ExitOnErr(Builder.initialize(4096)); // 4096 is blocksize @@ -219,9 +354,9 @@ void coff::createPDB(StringRef Path, SymbolTable *Symt pdb::DbiStreamBuilder &DbiBuilder = Builder.getDbiBuilder(); DbiBuilder.setVersionHeader(pdb::PdbDbiV110); - codeview::TypeTableBuilder TypeTable(BAlloc); - codeview::TypeTableBuilder IDTable(BAlloc); - addObjectsToPDB(Symtab, Builder, TypeTable, IDTable); + TypeTableBuilder TypeTable(BAlloc); + TypeTableBuilder IDTable(BAlloc); + addObjectsToPDB(Alloc, Symtab, Builder, TypeTable, IDTable); // Add Section Contributions. addSectionContribs(Symtab, DbiBuilder); Modified: vendor/lld/dist/COFF/Strings.cpp ============================================================================== --- vendor/lld/dist/COFF/Strings.cpp Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Strings.cpp Mon Jun 26 20:33:45 2017 (r320382) @@ -8,6 +8,7 @@ //===----------------------------------------------------------------------===// #include "Strings.h" +#include #if defined(_MSC_VER) #include @@ -21,6 +22,10 @@ using namespace llvm; Optional coff::demangle(StringRef S) { #if defined(_MSC_VER) + // UnDecorateSymbolName is not thread-safe, so we need a mutex. + static std::mutex Mu; + std::lock_guard Lock(Mu); + char Buf[4096]; if (S.startswith("?")) if (size_t Len = UnDecorateSymbolName(S.str().c_str(), Buf, sizeof(Buf), 0)) Modified: vendor/lld/dist/COFF/SymbolTable.cpp ============================================================================== --- vendor/lld/dist/COFF/SymbolTable.cpp Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/SymbolTable.cpp Mon Jun 26 20:33:45 2017 (r320382) @@ -219,13 +219,13 @@ Symbol *SymbolTable::addAbsolute(StringRef N, uint64_t return S; } -Symbol *SymbolTable::addRelative(StringRef N, uint64_t VA) { +Symbol *SymbolTable::addSynthetic(StringRef N, Chunk *C) { Symbol *S; bool WasInserted; std::tie(S, WasInserted) = insert(N); S->IsUsedInRegularObj = true; if (WasInserted || isa(S->body()) || isa(S->body())) - replaceBody(S, N, VA); + replaceBody(S, N, C); else if (!isa(S->body())) reportDuplicate(S, nullptr); return S; @@ -244,6 +244,12 @@ Symbol *SymbolTable::addRegular(InputFile *F, StringRe reportDuplicate(S, F); } else if (SP == SP_NEW) { replaceBody(S, F, N, IsCOMDAT, /*IsExternal*/ true, Sym, C); + } else if (SP == SP_EXISTING && IsCOMDAT && C) { + C->markDiscarded(); + // Discard associative chunks that we've parsed so far. No need to recurse + // because an associative section cannot have children. + for (SectionChunk *Child : C->children()) + Child->markDiscarded(); } return S; } Modified: vendor/lld/dist/COFF/SymbolTable.h ============================================================================== --- vendor/lld/dist/COFF/SymbolTable.h Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/SymbolTable.h Mon Jun 26 20:33:45 2017 (r320382) @@ -85,7 +85,7 @@ class SymbolTable { (public) // Creates an Undefined symbol for a given name. SymbolBody *addUndefined(StringRef Name); - Symbol *addRelative(StringRef N, uint64_t VA); + Symbol *addSynthetic(StringRef N, Chunk *C); Symbol *addAbsolute(StringRef N, uint64_t VA); Symbol *addUndefined(StringRef Name, InputFile *F, bool IsWeakAlias); Modified: vendor/lld/dist/COFF/Symbols.cpp ============================================================================== --- vendor/lld/dist/COFF/Symbols.cpp Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Symbols.cpp Mon Jun 26 20:33:45 2017 (r320382) @@ -61,6 +61,8 @@ COFFSymbolRef DefinedCOFF::getCOFFSymbol() { return COFFSymbolRef(reinterpret_cast(Sym)); } +uint16_t DefinedAbsolute::OutputSectionIndex = 0; + static Chunk *makeImportThunk(DefinedImportData *S, uint16_t Machine) { if (Machine == AMD64) return make(S); Modified: vendor/lld/dist/COFF/Symbols.h ============================================================================== --- vendor/lld/dist/COFF/Symbols.h Mon Jun 26 20:33:41 2017 (r320381) +++ vendor/lld/dist/COFF/Symbols.h Mon Jun 26 20:33:45 2017 (r320382) @@ -50,13 +50,13 @@ class SymbolBody { (public) DefinedImportThunkKind, DefinedImportDataKind, DefinedAbsoluteKind, - DefinedRelativeKind, + DefinedSyntheticKind, UndefinedKind, LazyKind, LastDefinedCOFFKind = DefinedCommonKind, - LastDefinedKind = DefinedRelativeKind, + LastDefinedKind = DefinedSyntheticKind, }; Kind kind() const { return static_cast(SymbolKind); } @@ -112,11 +112,11 @@ class Defined : public SymbolBody { (public) // Returns the RVA relative to the beginning of the output section. // Used to implement SECREL relocation type. - uint64_t getSecrel(); + uint32_t getSecrel(); // Returns the output section index. // Used to implement SECTION relocation type. - uint64_t getSectionIndex(); + uint16_t getSectionIndex(); // Returns true if this symbol points to an executable (e.g. .text) section. // Used to implement ARM relocations. @@ -167,6 +167,7 @@ class DefinedRegular : public DefinedCOFF { (public) bool isCOMDAT() { return IsCOMDAT; } SectionChunk *getChunk() { return *Data; } uint32_t getValue() { return Sym->Value; } + uint32_t getSecrel(); private: SectionChunk **Data; @@ -186,6 +187,8 @@ class DefinedCommon : public DefinedCOFF { (public) } uint64_t getRVA() { return Data->getRVA(); } + uint32_t getSecrel() { return Data->OutputSectionOff; } + uint16_t getSectionIndex(); private: friend SymbolTable; @@ -212,28 +215,34 @@ class DefinedAbsolute : public Defined { (public) uint64_t getRVA() { return VA - Config->ImageBase; } void setVA(uint64_t V) { VA = V; } + // The sentinel absolute symbol section index. Section index relocations + // against absolute symbols resolve to this 16 bit number, and it is the + // largest valid section index plus one. This is written by the Writer. + static uint16_t OutputSectionIndex; + private: uint64_t VA; }; -// This is a kind of absolute symbol but relative to the image base. -// Unlike absolute symbols, relocations referring this kind of symbols -// are subject of the base relocation. This type is used rarely -- -// mainly for __ImageBase. *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***