From owner-svn-src-head@freebsd.org Wed Feb 12 11:12:14 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7A7A234B38; Wed, 12 Feb 2020 11:12:14 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48HcRf3yBxz4FYV; Wed, 12 Feb 2020 11:12:14 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 831C920348; Wed, 12 Feb 2020 11:12:14 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 01CBCEEd085632; Wed, 12 Feb 2020 11:12:14 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 01CBCExs085630; Wed, 12 Feb 2020 11:12:14 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <202002121112.01CBCExs085630@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Wed, 12 Feb 2020 11:12:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r357805 - head/sys/amd64/include X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: head/sys/amd64/include X-SVN-Commit-Revision: 357805 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Feb 2020 11:12:14 -0000 Author: mjg Date: Wed Feb 12 11:12:13 2020 New Revision: 357805 URL: https://svnweb.freebsd.org/changeset/base/357805 Log: amd64: store per-cpu allocations subtracted by __pcpu This eliminates a runtime subtraction from counter_u64_add. before: mov 0x4f00ed(%rip),%rax # 0xffffffff80c01788 sub 0x808ff6(%rip),%rax # 0xffffffff80f1a698 <__pcpu> addq $0x1,%gs:(%rax) after: mov 0x4f02fd(%rip),%rax # 0xffffffff80c01788 addq $0x1,%gs:(%rax) Reviewed by: jeff Differential Revision: https://reviews.freebsd.org/D23570 Modified: head/sys/amd64/include/counter.h head/sys/amd64/include/pcpu.h Modified: head/sys/amd64/include/counter.h ============================================================================== --- head/sys/amd64/include/counter.h Wed Feb 12 11:11:22 2020 (r357804) +++ head/sys/amd64/include/counter.h Wed Feb 12 11:12:13 2020 (r357805) @@ -33,7 +33,7 @@ #include -#define EARLY_COUNTER &temp_bsp_pcpu.pc_early_dummy_counter +#define EARLY_COUNTER (void *)__offsetof(struct pcpu, pc_early_dummy_counter) #define counter_enter() do {} while (0) #define counter_exit() do {} while (0) @@ -43,6 +43,7 @@ static inline uint64_t counter_u64_read_one(counter_u64_t c, int cpu) { + MPASS(c != EARLY_COUNTER); return (*zpcpu_get_cpu(c, cpu)); } @@ -65,6 +66,7 @@ counter_u64_zero_one_cpu(void *arg) counter_u64_t c; c = arg; + MPASS(c != EARLY_COUNTER); *(zpcpu_get(c)) = 0; } @@ -86,7 +88,7 @@ counter_u64_add(counter_u64_t c, int64_t inc) KASSERT(IS_BSP() || c != EARLY_COUNTER, ("EARLY_COUNTER used on AP")); __asm __volatile("addq\t%1,%%gs:(%0)" : - : "r" ((char *)c - (char *)&__pcpu[0]), "ri" (inc) + : "r" (c), "ri" (inc) : "memory", "cc"); } Modified: head/sys/amd64/include/pcpu.h ============================================================================== --- head/sys/amd64/include/pcpu.h Wed Feb 12 11:11:22 2020 (r357804) +++ head/sys/amd64/include/pcpu.h Wed Feb 12 11:12:13 2020 (r357805) @@ -240,6 +240,10 @@ _Static_assert(sizeof(struct monitorbuf) == 128, "2x c #define IS_BSP() (PCPU_GET(cpuid) == 0) +#define zpcpu_offset_cpu(cpu) ((uintptr_t)&__pcpu[0] + UMA_PCPU_ALLOC_SIZE * cpu) +#define zpcpu_base_to_offset(base) (void *)((uintptr_t)(base) - (uintptr_t)&__pcpu[0]) +#define zpcpu_offset_to_base(base) (void *)((uintptr_t)(base) + (uintptr_t)&__pcpu[0]) + #else /* !__GNUCLIKE_ASM || !__GNUCLIKE___TYPEOF */ #error "this file needs to be ported to your compiler"