From owner-svn-src-all@freebsd.org Tue Aug 7 18:29:11 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A024710673BE; Tue, 7 Aug 2018 18:29:11 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 54F3E7B5DE; Tue, 7 Aug 2018 18:29:11 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 35EFD1F724; Tue, 7 Aug 2018 18:29:11 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w77ITBPB082614; Tue, 7 Aug 2018 18:29:11 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w77ITAV8082610; Tue, 7 Aug 2018 18:29:10 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201808071829.w77ITAV8082610@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Tue, 7 Aug 2018 18:29:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r337431 - in head/sys/amd64: linux linux32 X-SVN-Group: head X-SVN-Commit-Author: kib X-SVN-Commit-Paths: in head/sys/amd64: linux linux32 X-SVN-Commit-Revision: 337431 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Aug 2018 18:29:11 -0000 Author: kib Date: Tue Aug 7 18:29:10 2018 New Revision: 337431 URL: https://svnweb.freebsd.org/changeset/base/337431 Log: Futex support functions in linux.ko and linux32.ko on amd64 should be aware of SMAP. Reported and tested by: Johannes Lundberg , wulf Sponsored by: The FreeBSD Foundation Modified: head/sys/amd64/linux/linux_machdep.c head/sys/amd64/linux/linux_support.s head/sys/amd64/linux32/linux32_machdep.c head/sys/amd64/linux32/linux32_support.s Modified: head/sys/amd64/linux/linux_machdep.c ============================================================================== --- head/sys/amd64/linux/linux_machdep.c Tue Aug 7 18:26:46 2018 (r337430) +++ head/sys/amd64/linux/linux_machdep.c Tue Aug 7 18:29:10 2018 (r337431) @@ -78,6 +78,9 @@ __FBSDID("$FreeBSD$"); #include #include +#include +#include + #include #include #include @@ -88,8 +91,6 @@ __FBSDID("$FreeBSD$"); #include #include -#include - int linux_execve(struct thread *td, struct linux_execve_args *args) { @@ -275,4 +276,49 @@ linux_set_cloned_tls(struct thread *td, void *desc) td->td_frame->tf_fs = _ufssel; return (0); +} + +int futex_xchgl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_xchgl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_xchgl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_xchgl_smap : futex_xchgl_nosmap); +} + +int futex_addl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_addl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_addl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_addl_smap : futex_addl_nosmap); +} + +int futex_orl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_orl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_orl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_orl_smap : futex_orl_nosmap); +} + +int futex_andl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_andl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_andl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_andl_smap : futex_andl_nosmap); +} + +int futex_xorl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_xorl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_xorl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_xorl_smap : futex_xorl_nosmap); } Modified: head/sys/amd64/linux/linux_support.s ============================================================================== --- head/sys/amd64/linux/linux_support.s Tue Aug 7 18:26:46 2018 (r337430) +++ head/sys/amd64/linux/linux_support.s Tue Aug 7 18:29:10 2018 (r337431) @@ -38,7 +38,7 @@ futex_fault: movl $-EFAULT,%eax ret -ENTRY(futex_xchgl) +ENTRY(futex_xchgl_nosmap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax @@ -49,14 +49,29 @@ ENTRY(futex_xchgl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_xchgl) +END(futex_xchgl_nosmap) -ENTRY(futex_addl) +ENTRY(futex_xchgl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax cmpq %rax,%rsi ja futex_fault + stac + xchgl %edi,(%rsi) + clac + movl %edi,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_xchgl_smap) + +ENTRY(futex_addl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault #ifdef SMP lock #endif @@ -65,14 +80,32 @@ ENTRY(futex_addl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_addl) +END(futex_addl_nosmap) -ENTRY(futex_orl) +ENTRY(futex_addl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax cmpq %rax,%rsi ja futex_fault + stac +#ifdef SMP + lock +#endif + xaddl %edi,(%rsi) + clac + movl %edi,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_addl_smap) + +ENTRY(futex_orl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault movl (%rsi),%eax 1: movl %eax,%ecx orl %edi,%ecx @@ -85,9 +118,9 @@ ENTRY(futex_orl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_orl) +END(futex_orl_nosmap) -ENTRY(futex_andl) +ENTRY(futex_orl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax @@ -95,6 +128,28 @@ ENTRY(futex_andl) ja futex_fault movl (%rsi),%eax 1: movl %eax,%ecx + orl %edi,%ecx + stac +#ifdef SMP + lock +#endif + cmpxchgl %ecx,(%rsi) + clac + jnz 1b + movl %eax,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_orl_smap) + +ENTRY(futex_andl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault + movl (%rsi),%eax +1: movl %eax,%ecx andl %edi,%ecx #ifdef SMP lock @@ -105,9 +160,9 @@ ENTRY(futex_andl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_andl) +END(futex_andl_nosmap) -ENTRY(futex_xorl) +ENTRY(futex_andl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax @@ -115,6 +170,28 @@ ENTRY(futex_xorl) ja futex_fault movl (%rsi),%eax 1: movl %eax,%ecx + andl %edi,%ecx + stac +#ifdef SMP + lock +#endif + cmpxchgl %ecx,(%rsi) + clac + jnz 1b + movl %eax,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_andl_smap) + +ENTRY(futex_xorl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault + movl (%rsi),%eax +1: movl %eax,%ecx xorl %edi,%ecx #ifdef SMP lock @@ -125,4 +202,26 @@ ENTRY(futex_xorl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_xorl) +END(futex_xorl_nosmap) + +ENTRY(futex_xorl_smap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault + movl (%rsi),%eax +1: movl %eax,%ecx + xorl %edi,%ecx + stac +#ifdef SMP + lock +#endif + cmpxchgl %ecx,(%rsi) + clac + jnz 1b + movl %eax,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_xorl_smap) Modified: head/sys/amd64/linux32/linux32_machdep.c ============================================================================== --- head/sys/amd64/linux32/linux32_machdep.c Tue Aug 7 18:26:46 2018 (r337430) +++ head/sys/amd64/linux32/linux32_machdep.c Tue Aug 7 18:29:10 2018 (r337431) @@ -58,10 +58,12 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include #include #include +#include #include #include @@ -821,4 +823,49 @@ linux_set_thread_area(struct thread *td, update_gdt_gsbase(td, info.base_addr); return (0); +} + +int futex_xchgl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_xchgl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_xchgl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_xchgl_smap : futex_xchgl_nosmap); +} + +int futex_addl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_addl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_addl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_addl_smap : futex_addl_nosmap); +} + +int futex_orl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_orl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_orl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_orl_smap : futex_orl_nosmap); +} + +int futex_andl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_andl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_andl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_andl_smap : futex_andl_nosmap); +} + +int futex_xorl_nosmap(int oparg, uint32_t *uaddr, int *oldval); +int futex_xorl_smap(int oparg, uint32_t *uaddr, int *oldval); +DEFINE_IFUNC(, int, futex_xorl, (int, uint32_t *, int *), static) +{ + + return ((cpu_stdext_feature & CPUID_STDEXT_SMAP) != 0 ? + futex_xorl_smap : futex_xorl_nosmap); } Modified: head/sys/amd64/linux32/linux32_support.s ============================================================================== --- head/sys/amd64/linux32/linux32_support.s Tue Aug 7 18:26:46 2018 (r337430) +++ head/sys/amd64/linux32/linux32_support.s Tue Aug 7 18:29:10 2018 (r337431) @@ -38,7 +38,7 @@ futex_fault: movl $-EFAULT,%eax ret -ENTRY(futex_xchgl) +ENTRY(futex_xchgl_nosmap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax @@ -49,14 +49,29 @@ ENTRY(futex_xchgl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_xchgl) +END(futex_xchgl_nosmap) -ENTRY(futex_addl) +ENTRY(futex_xchgl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax cmpq %rax,%rsi ja futex_fault + stac + xchgl %edi,(%rsi) + clac + movl %edi,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_xchgl_smap) + +ENTRY(futex_addl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault #ifdef SMP lock #endif @@ -65,14 +80,32 @@ ENTRY(futex_addl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_addl) +END(futex_addl_nosmap) -ENTRY(futex_orl) +ENTRY(futex_addl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax cmpq %rax,%rsi ja futex_fault + stac +#ifdef SMP + lock +#endif + xaddl %edi,(%rsi) + clac + movl %edi,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_addl_smap) + +ENTRY(futex_orl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault movl (%rsi),%eax 1: movl %eax,%ecx orl %edi,%ecx @@ -85,9 +118,9 @@ ENTRY(futex_orl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_orl) +END(futex_orl_nosmap) -ENTRY(futex_andl) +ENTRY(futex_orl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax @@ -95,6 +128,28 @@ ENTRY(futex_andl) ja futex_fault movl (%rsi),%eax 1: movl %eax,%ecx + orl %edi,%ecx + stac +#ifdef SMP + lock +#endif + cmpxchgl %ecx,(%rsi) + clac + jnz 1b + movl %eax,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_orl_smap) + +ENTRY(futex_andl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault + movl (%rsi),%eax +1: movl %eax,%ecx andl %edi,%ecx #ifdef SMP lock @@ -105,9 +160,9 @@ ENTRY(futex_andl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_andl) +END(futex_andl_nosmap) -ENTRY(futex_xorl) +ENTRY(futex_andl_smap) movq PCPU(CURPCB),%r8 movq $futex_fault,PCB_ONFAULT(%r8) movq $VM_MAXUSER_ADDRESS-4,%rax @@ -115,6 +170,28 @@ ENTRY(futex_xorl) ja futex_fault movl (%rsi),%eax 1: movl %eax,%ecx + andl %edi,%ecx + stac +#ifdef SMP + lock +#endif + cmpxchgl %ecx,(%rsi) + clac + jnz 1b + movl %eax,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_andl_smap) + +ENTRY(futex_xorl_nosmap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault + movl (%rsi),%eax +1: movl %eax,%ecx xorl %edi,%ecx #ifdef SMP lock @@ -125,4 +202,26 @@ ENTRY(futex_xorl) xorl %eax,%eax movq %rax,PCB_ONFAULT(%r8) ret -END(futex_xorl) +END(futex_xorl_nosmap) + +ENTRY(futex_xorl_smap) + movq PCPU(CURPCB),%r8 + movq $futex_fault,PCB_ONFAULT(%r8) + movq $VM_MAXUSER_ADDRESS-4,%rax + cmpq %rax,%rsi + ja futex_fault + movl (%rsi),%eax +1: movl %eax,%ecx + xorl %edi,%ecx + stac +#ifdef SMP + lock +#endif + cmpxchgl %ecx,(%rsi) + clac + jnz 1b + movl %eax,(%rdx) + xorl %eax,%eax + movq %rax,PCB_ONFAULT(%r8) + ret +END(futex_xorl_smap)