From owner-svn-src-head@freebsd.org Wed Aug 22 20:44:33 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 800941095A0C; Wed, 22 Aug 2018 20:44:33 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 298488B6EC; Wed, 22 Aug 2018 20:44:33 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0775E18242; Wed, 22 Aug 2018 20:44:33 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w7MKiX3I000920; Wed, 22 Aug 2018 20:44:33 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w7MKiUBZ000908; Wed, 22 Aug 2018 20:44:30 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201808222044.w7MKiUBZ000908@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Wed, 22 Aug 2018 20:44:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r338211 - in head/sys: amd64/amd64 arm/arm arm64/arm64 i386/i386 kern mips/mips powerpc/powerpc riscv/riscv sparc64/sparc64 sys X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: in head/sys: amd64/amd64 arm/arm arm64/arm64 i386/i386 kern mips/mips powerpc/powerpc riscv/riscv sparc64/sparc64 sys X-SVN-Commit-Revision: 338211 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2018 20:44:33 -0000 Author: markj Date: Wed Aug 22 20:44:30 2018 New Revision: 338211 URL: https://svnweb.freebsd.org/changeset/base/338211 Log: Prepare the kernel linker to handle PC-relative ifunc relocations. The boot-time ifunc resolver assumes that it only needs to apply IRELATIVE relocations to PLT entries. With an upcoming optimization, this assumption no longer holds, so add the support required to handle PC-relative relocations targeting GNU_IFUNC symbols. - Provide a custom symbol lookup routine that can be used in early boot. The default lookup routine uses kobj, which is not functional at that point. - Apply all existing relocations during boot rather than filtering IRELATIVE relocations. - Ensure that we continue to apply ifunc relocations in a second pass when loading a kernel module. Reviewed by: kib MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D16749 Modified: head/sys/amd64/amd64/elf_machdep.c head/sys/arm/arm/elf_machdep.c head/sys/arm64/arm64/elf_machdep.c head/sys/i386/i386/elf_machdep.c head/sys/kern/link_elf.c head/sys/kern/link_elf_obj.c head/sys/mips/mips/elf_machdep.c head/sys/powerpc/powerpc/elf32_machdep.c head/sys/powerpc/powerpc/elf64_machdep.c head/sys/riscv/riscv/elf_machdep.c head/sys/sparc64/sparc64/elf_machdep.c head/sys/sys/linker.h Modified: head/sys/amd64/amd64/elf_machdep.c ============================================================================== --- head/sys/amd64/amd64/elf_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/amd64/amd64/elf_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -175,13 +175,17 @@ elf64_dump_thread(struct thread *td, void *dst, size_t *off = len; } -#define ERI_LOCAL 0x0001 -#define ERI_ONLYIFUNC 0x0002 +bool +elf_is_ifunc_reloc(Elf_Size r_info) +{ + return (ELF_R_TYPE(r_info) == R_X86_64_IRELATIVE); +} + /* Process one elf relocation with addend. */ static int elf_reloc_internal(linker_file_t lf, Elf_Addr relocbase, const void *data, - int type, elf_lookup_fn lookup, int flags) + int type, elf_lookup_fn lookup) { Elf64_Addr *where, val; Elf32_Addr *where32, val32; @@ -221,9 +225,6 @@ elf_reloc_internal(linker_file_t lf, Elf_Addr relocbas panic("unknown reloc type %d\n", type); } - if (((flags & ERI_ONLYIFUNC) == 0) ^ (rtype != R_X86_64_IRELATIVE)) - return (0); - switch (rtype) { case R_X86_64_NONE: /* none */ break; @@ -300,20 +301,11 @@ elf_reloc_internal(linker_file_t lf, Elf_Addr relocbas } int -elf_reloc_ifunc(linker_file_t lf, Elf_Addr relocbase, const void *data, - int type, elf_lookup_fn lookup) -{ - - return (elf_reloc_internal(lf, relocbase, data, type, lookup, - ERI_ONLYIFUNC)); -} - -int elf_reloc(linker_file_t lf, Elf_Addr relocbase, const void *data, int type, elf_lookup_fn lookup) { - return (elf_reloc_internal(lf, relocbase, data, type, lookup, 0)); + return (elf_reloc_internal(lf, relocbase, data, type, lookup)); } int @@ -321,8 +313,7 @@ elf_reloc_local(linker_file_t lf, Elf_Addr relocbase, int type, elf_lookup_fn lookup) { - return (elf_reloc_internal(lf, relocbase, data, type, lookup, - ERI_LOCAL)); + return (elf_reloc_internal(lf, relocbase, data, type, lookup)); } int Modified: head/sys/arm/arm/elf_machdep.c ============================================================================== --- head/sys/arm/arm/elf_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/arm/arm/elf_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -149,6 +149,13 @@ elf32_dump_thread(struct thread *td, void *dst, size_t #endif } +bool +elf_is_ifunc_reloc(Elf_Size r_info __unused) +{ + + return (false); +} + /* * It is possible for the compiler to emit relocations for unaligned data. * We handle this situation with these inlines. Modified: head/sys/arm64/arm64/elf_machdep.c ============================================================================== --- head/sys/arm64/arm64/elf_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/arm64/arm64/elf_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -129,6 +129,13 @@ elf64_dump_thread(struct thread *td __unused, void *ds } +bool +elf_is_ifunc_reloc(Elf_Size r_info __unused) +{ + + return (false); +} + static int elf_reloc_internal(linker_file_t lf, Elf_Addr relocbase, const void *data, int type, int local, elf_lookup_fn lookup) Modified: head/sys/i386/i386/elf_machdep.c ============================================================================== --- head/sys/i386/i386/elf_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/i386/i386/elf_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -159,8 +159,14 @@ elf32_dump_thread(struct thread *td, void *dst, size_t *off = len; } +bool +elf_is_ifunc_reloc(Elf_Size r_info) +{ + + return (ELF_R_TYPE(r_info) == R_386_IRELATIVE); +} + #define ERI_LOCAL 0x0001 -#define ERI_ONLYIFUNC 0x0002 /* Process one elf relocation with addend. */ static int @@ -194,9 +200,6 @@ elf_reloc_internal(linker_file_t lf, Elf_Addr relocbas panic("unknown reloc type %d\n", type); } - if (((flags & ERI_ONLYIFUNC) == 0) ^ (rtype != R_386_IRELATIVE)) - return (0); - if ((flags & ERI_LOCAL) != 0) { if (rtype == R_386_RELATIVE) { /* A + B */ addr = elf_relocaddr(lf, relocbase + addend); @@ -261,15 +264,6 @@ elf_reloc_internal(linker_file_t lf, Elf_Addr relocbas return -1; } return(0); -} - -int -elf_reloc_ifunc(linker_file_t lf, Elf_Addr relocbase, const void *data, - int type, elf_lookup_fn lookup) -{ - - return (elf_reloc_internal(lf, relocbase, data, type, lookup, - ERI_ONLYIFUNC)); } int Modified: head/sys/kern/link_elf.c ============================================================================== --- head/sys/kern/link_elf.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/kern/link_elf.c Wed Aug 22 20:44:30 2018 (r338211) @@ -188,11 +188,13 @@ static struct linker_class link_elf_class = { link_elf_methods, sizeof(struct elf_file) }; +typedef int (*elf_reloc_fn)(linker_file_t lf, Elf_Addr relocbase, + const void *data, int type, elf_lookup_fn lookup); + static int parse_dynamic(elf_file_t); static int relocate_file(elf_file_t); -static int relocate_file1(elf_file_t ef, int (*elf_reloc_func)( - linker_file_t lf, Elf_Addr relocbase, const void *data, - int type, elf_lookup_fn lookup)); +static int relocate_file1(elf_file_t ef, elf_lookup_fn lookup, + elf_reloc_fn reloc, bool ifuncs); static int link_elf_preload_parse_symbols(elf_file_t); static struct elf_set_head set_pcpu_list; @@ -1185,96 +1187,61 @@ symbol_name(elf_file_t ef, Elf_Size r_info) } static int -relocate_file1(elf_file_t ef, int (*elf_reloc_func)(linker_file_t lf, - Elf_Addr relocbase, const void *data, int type, elf_lookup_fn lookup)) +symbol_type(elf_file_t ef, Elf_Size r_info) { - const Elf_Rel *rellim; + const Elf_Sym *ref; + + if (ELF_R_SYM(r_info)) { + ref = ef->symtab + ELF_R_SYM(r_info); + return (ELF_ST_TYPE(ref->st_info)); + } + return (STT_NOTYPE); +} + +static int +relocate_file1(elf_file_t ef, elf_lookup_fn lookup, elf_reloc_fn reloc, + bool ifuncs) +{ const Elf_Rel *rel; - const Elf_Rela *relalim; const Elf_Rela *rela; const char *symname; - /* Perform relocations without addend if there are any: */ - rel = ef->rel; - if (rel != NULL) { - rellim = (const Elf_Rel *) - ((const char *)ef->rel + ef->relsize); - while (rel < rellim) { - if (elf_reloc_func(&ef->lf, (Elf_Addr)ef->address, rel, - ELF_RELOC_REL, elf_lookup)) { - symname = symbol_name(ef, rel->r_info); - printf("link_elf: symbol %s undefined\n", symname); - return (ENOENT); - } - rel++; - } - } +#define APPLY_RELOCS(iter, tbl, tblsize, type) do { \ + for ((iter) = (tbl); (iter) != NULL && \ + (iter) < (tbl) + (tblsize) / sizeof(*(iter)); (iter)++) { \ + if ((symbol_type(ef, (iter)->r_info) == \ + STT_GNU_IFUNC || \ + elf_is_ifunc_reloc((iter)->r_info)) != ifuncs) \ + continue; \ + if (reloc(&ef->lf, (Elf_Addr)ef->address, \ + (iter), (type), lookup)) { \ + symname = symbol_name(ef, (iter)->r_info); \ + printf("link_elf: symbol %s undefined\n", \ + symname); \ + return (ENOENT); \ + } \ + } \ +} while (0) - /* Perform relocations with addend if there are any: */ - rela = ef->rela; - if (rela != NULL) { - relalim = (const Elf_Rela *) - ((const char *)ef->rela + ef->relasize); - while (rela < relalim) { - if (elf_reloc_func(&ef->lf, (Elf_Addr)ef->address, rela, - ELF_RELOC_RELA, elf_lookup)) { - symname = symbol_name(ef, rela->r_info); - printf("link_elf: symbol %s undefined\n", - symname); - return (ENOENT); - } - rela++; - } - } + APPLY_RELOCS(rel, ef->rel, ef->relsize, ELF_RELOC_REL); + APPLY_RELOCS(rela, ef->rela, ef->relasize, ELF_RELOC_RELA); + APPLY_RELOCS(rel, ef->pltrel, ef->pltrelsize, ELF_RELOC_REL); + APPLY_RELOCS(rela, ef->pltrela, ef->pltrelasize, ELF_RELOC_RELA); - /* Perform PLT relocations without addend if there are any: */ - rel = ef->pltrel; - if (rel != NULL) { - rellim = (const Elf_Rel *) - ((const char *)ef->pltrel + ef->pltrelsize); - while (rel < rellim) { - if (elf_reloc_func(&ef->lf, (Elf_Addr)ef->address, rel, - ELF_RELOC_REL, elf_lookup)) { - symname = symbol_name(ef, rel->r_info); - printf("link_elf: symbol %s undefined\n", - symname); - return (ENOENT); - } - rel++; - } - } +#undef APPLY_RELOCS - /* Perform relocations with addend if there are any: */ - rela = ef->pltrela; - if (rela != NULL) { - relalim = (const Elf_Rela *) - ((const char *)ef->pltrela + ef->pltrelasize); - while (rela < relalim) { - if (elf_reloc_func(&ef->lf, (Elf_Addr)ef->address, rela, - ELF_RELOC_RELA, elf_lookup)) { - symname = symbol_name(ef, rela->r_info); - printf("link_elf: symbol %s undefined\n", - symname); - return (ENOENT); - } - rela++; - } - } - return (0); } static int relocate_file(elf_file_t ef) { - int e; + int error; - e = relocate_file1(ef, elf_reloc); -#if defined(__i386__) || defined(__amd64__) - if (e == 0) - e = relocate_file1(ef, elf_reloc_ifunc); -#endif - return (e); + error = relocate_file1(ef, elf_lookup, elf_reloc, false); + if (error == 0) + error = relocate_file1(ef, elf_lookup, elf_reloc, true); + return (error); } /* @@ -1298,7 +1265,7 @@ elf_hash(const char *name) } static int -link_elf_lookup_symbol(linker_file_t lf, const char* name, c_linker_sym_t* sym) +link_elf_lookup_symbol(linker_file_t lf, const char *name, c_linker_sym_t *sym) { elf_file_t ef = (elf_file_t) lf; unsigned long symnum; @@ -1687,6 +1654,29 @@ link_elf_strtab_get(linker_file_t lf, caddr_t *strtab) } #if defined(__i386__) || defined(__amd64__) +/* + * Use this lookup routine when performing relocations early during boot. + * The generic lookup routine depends on kobj, which is not initialized + * at that point. + */ +static int +elf_lookup_ifunc(linker_file_t lf, Elf_Size symidx, int deps __unused, + Elf_Addr *res) +{ + elf_file_t ef; + const Elf_Sym *symp; + caddr_t val; + + ef = (elf_file_t)lf; + symp = ef->symtab + symidx; + if (ELF_ST_TYPE(symp->st_info) == STT_GNU_IFUNC) { + val = (caddr_t)ef->address + symp->st_value; + *res = ((Elf_Addr (*)(void))val)(); + return (0); + } + return (ENOENT); +} + void link_elf_ireloc(caddr_t kmdp) { @@ -1695,7 +1685,7 @@ link_elf_ireloc(caddr_t kmdp) volatile char *c; size_t i; - ef = &eff; + ef = &eff; /* Do not use bzero/memset before ireloc is done. */ for (c = (char *)ef, i = 0; i < sizeof(*ef); i++) @@ -1706,6 +1696,6 @@ link_elf_ireloc(caddr_t kmdp) parse_dynamic(ef); ef->address = 0; link_elf_preload_parse_symbols(ef); - relocate_file1(ef, elf_reloc_ifunc); + relocate_file1(ef, elf_lookup_ifunc, elf_reloc, true); } #endif Modified: head/sys/kern/link_elf_obj.c ============================================================================== --- head/sys/kern/link_elf_obj.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/kern/link_elf_obj.c Wed Aug 22 20:44:30 2018 (r338211) @@ -1521,15 +1521,10 @@ link_elf_reloc_local(linker_file_t lf, bool ifuncs) /* Only do local relocs */ if (ELF_ST_BIND(sym->st_info) != STB_LOCAL) continue; - if ((ELF_ST_TYPE(sym->st_info) == STT_GNU_IFUNC) == - ifuncs) + if ((ELF_ST_TYPE(sym->st_info) == STT_GNU_IFUNC || + elf_is_ifunc_reloc(rel->r_info)) == ifuncs) elf_reloc_local(lf, base, rel, ELF_RELOC_REL, elf_obj_lookup); -#if defined(__i386__) || defined(__amd64__) - else if (ifuncs) - elf_reloc_ifunc(lf, base, rel, ELF_RELOC_REL, - elf_obj_lookup); -#endif } } @@ -1554,15 +1549,10 @@ link_elf_reloc_local(linker_file_t lf, bool ifuncs) /* Only do local relocs */ if (ELF_ST_BIND(sym->st_info) != STB_LOCAL) continue; - if ((ELF_ST_TYPE(sym->st_info) == STT_GNU_IFUNC) == - ifuncs) + if ((ELF_ST_TYPE(sym->st_info) == STT_GNU_IFUNC || + elf_is_ifunc_reloc(rela->r_info)) == ifuncs) elf_reloc_local(lf, base, rela, ELF_RELOC_RELA, elf_obj_lookup); -#if defined(__i386__) || defined(__amd64__) - else if (ifuncs) - elf_reloc_ifunc(lf, base, rela, ELF_RELOC_RELA, - elf_obj_lookup); -#endif } } return (0); Modified: head/sys/mips/mips/elf_machdep.c ============================================================================== --- head/sys/mips/mips/elf_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/mips/mips/elf_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -295,6 +295,13 @@ mips_tmp_reloc_free(struct mips_tmp_reloc *r) free(r, M_TEMP); } +bool +elf_is_ifunc_reloc(Elf_Size r_info __unused) +{ + + return (false); +} + /* Process one elf relocation with addend. */ static int elf_reloc_internal(linker_file_t lf, Elf_Addr relocbase, const void *data, Modified: head/sys/powerpc/powerpc/elf32_machdep.c ============================================================================== --- head/sys/powerpc/powerpc/elf32_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/powerpc/powerpc/elf32_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -214,6 +214,13 @@ elf32_dump_thread(struct thread *td, void *dst, size_t } #ifndef __powerpc64__ +bool +elf_is_ifunc_reloc(Elf_Size r_info __unused) +{ + + return (false); +} + /* Process one elf relocation with addend. */ static int elf_reloc_internal(linker_file_t lf, Elf_Addr relocbase, const void *data, Modified: head/sys/powerpc/powerpc/elf64_machdep.c ============================================================================== --- head/sys/powerpc/powerpc/elf64_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/powerpc/powerpc/elf64_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -276,6 +276,12 @@ elf64_dump_thread(struct thread *td, void *dst, size_t *off = len; } +bool +elf_is_ifunc_reloc(Elf_Size r_info __unused) +{ + + return (false); +} /* Process one elf relocation with addend. */ static int Modified: head/sys/riscv/riscv/elf_machdep.c ============================================================================== --- head/sys/riscv/riscv/elf_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/riscv/riscv/elf_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -259,6 +259,13 @@ reloctype_to_str(int type) return "*unknown*"; } +bool +elf_is_ifunc_reloc(Elf_Size r_info __unused) +{ + + return (false); +} + /* * Currently kernel loadable module for RISCV is compiled with -fPIC option. * (see also additional CFLAGS definition for RISCV in sys/conf/kmod.mk) Modified: head/sys/sparc64/sparc64/elf_machdep.c ============================================================================== --- head/sys/sparc64/sparc64/elf_machdep.c Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/sparc64/sparc64/elf_machdep.c Wed Aug 22 20:44:30 2018 (r338211) @@ -312,6 +312,13 @@ static const long reloc_target_bitmask[] = { }; #define RELOC_VALUE_BITMASK(t) (reloc_target_bitmask[t]) +bool +elf_is_ifunc_reloc(Elf_Size r_info __unused) +{ + + return (false); +} + int elf_reloc_local(linker_file_t lf, Elf_Addr relocbase, const void *data, int type, elf_lookup_fn lookup __unused) Modified: head/sys/sys/linker.h ============================================================================== --- head/sys/sys/linker.h Wed Aug 22 20:23:08 2018 (r338210) +++ head/sys/sys/linker.h Wed Aug 22 20:44:30 2018 (r338211) @@ -272,9 +272,8 @@ extern int kld_debug; typedef int elf_lookup_fn(linker_file_t, Elf_Size, int, Elf_Addr *); /* Support functions */ +bool elf_is_ifunc_reloc(Elf_Size r_info); int elf_reloc(linker_file_t _lf, Elf_Addr base, const void *_rel, - int _type, elf_lookup_fn _lu); -int elf_reloc_ifunc(linker_file_t _lf, Elf_Addr base, const void *_rel, int _type, elf_lookup_fn _lu); int elf_reloc_local(linker_file_t _lf, Elf_Addr base, const void *_rel, int _type, elf_lookup_fn _lu);