From owner-svn-src-head@freebsd.org Tue Jul 10 00:18:14 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC9F91044E42; Tue, 10 Jul 2018 00:18:13 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A9928F4D4; Tue, 10 Jul 2018 00:18:13 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6B8B81C2DE; Tue, 10 Jul 2018 00:18:13 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6A0ID5j058178; Tue, 10 Jul 2018 00:18:13 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6A0ID7V058177; Tue, 10 Jul 2018 00:18:13 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201807100018.w6A0ID7V058177@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Tue, 10 Jul 2018 00:18:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r336149 - in head/sys: kern vm X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: in head/sys: kern vm X-SVN-Commit-Revision: 336149 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jul 2018 00:18:14 -0000 Author: markj Date: Tue Jul 10 00:18:12 2018 New Revision: 336149 URL: https://svnweb.freebsd.org/changeset/base/336149 Log: Fix pre-SI_SUB_CPU initialization of per-CPU counters. r336020 introduced pcpu_page_alloc(), replacing page_alloc() as the backend allocator for PCPU UMA zones. Unlike page_alloc(), it does not honour malloc(9) flags such as M_ZERO or M_NODUMP, so fix that. r336020 also changed counter(9) to initialize each counter using a CPU_FOREACH() loop instead of an SMP rendezvous. Before SI_SUB_CPU, smp_rendezvous() will only execute the callback on the current CPU (i.e., CPU 0), so only one counter gets zeroed. The rest are zeroed by virtue of the fact that UMA gratuitously zeroes slabs when importing them into a zone. Prior to SI_SUB_CPU, all_cpus is clear, so with r336020 we weren't zeroing vm_cnt counters during boot: the CPU_FOREACH() loop had no effect, and pcpu_page_alloc() didn't honour M_ZERO. Fix this by iterating over the full range of CPU IDs when zeroing counters, ignoring whether the corresponding bits in all_cpus are set. Reported and tested by: pho (previous version) Reviewed by: kib (previous version) Differential Revision: https://reviews.freebsd.org/D16190 Modified: head/sys/kern/subr_counter.c head/sys/vm/uma_core.c Modified: head/sys/kern/subr_counter.c ============================================================================== --- head/sys/kern/subr_counter.c Tue Jul 10 00:01:14 2018 (r336148) +++ head/sys/kern/subr_counter.c Tue Jul 10 00:18:12 2018 (r336149) @@ -43,15 +43,6 @@ __FBSDID("$FreeBSD$"); #define IN_SUBR_COUNTER_C #include -static void -counter_u64_zero_sync(counter_u64_t c) -{ - int cpu; - - CPU_FOREACH(cpu) - *(uint64_t*)zpcpu_get_cpu(c, cpu) = 0; -} - void counter_u64_zero(counter_u64_t c) { @@ -69,13 +60,8 @@ counter_u64_fetch(counter_u64_t c) counter_u64_t counter_u64_alloc(int flags) { - counter_u64_t r; - r = uma_zalloc_pcpu(pcpu_zone_64, flags); - if (r != NULL) - counter_u64_zero_sync(r); - - return (r); + return (uma_zalloc_pcpu(pcpu_zone_64, flags | M_ZERO)); } void Modified: head/sys/vm/uma_core.c ============================================================================== --- head/sys/vm/uma_core.c Tue Jul 10 00:01:14 2018 (r336148) +++ head/sys/vm/uma_core.c Tue Jul 10 00:18:12 2018 (r336149) @@ -1186,13 +1186,12 @@ pcpu_page_alloc(uma_zone_t zone, vm_size_t bytes, int struct pcpu *pc; #endif - TAILQ_INIT(&alloctail); - MPASS(bytes == (mp_maxid+1)*PAGE_SIZE); - *pflag = UMA_SLAB_KERNEL; + MPASS(bytes == (mp_maxid + 1) * PAGE_SIZE); + TAILQ_INIT(&alloctail); flags = VM_ALLOC_SYSTEM | VM_ALLOC_WIRED | VM_ALLOC_NOOBJ | - ((wait & M_WAITOK) != 0 ? VM_ALLOC_WAITOK : - VM_ALLOC_NOWAIT); + malloc2vm_flags(wait); + *pflag = UMA_SLAB_KERNEL; for (cpu = 0; cpu <= mp_maxid; cpu++) { if (CPU_ABSENT(cpu)) { p = vm_page_alloc(NULL, 0, flags); @@ -2328,10 +2327,10 @@ uma_zalloc_pcpu_arg(uma_zone_t zone, void *udata, int MPASS(zone->uz_flags & UMA_ZONE_PCPU); #endif - item = uma_zalloc_arg(zone, udata, flags &~ M_ZERO); + item = uma_zalloc_arg(zone, udata, flags & ~M_ZERO); if (item != NULL && (flags & M_ZERO)) { #ifdef SMP - CPU_FOREACH(i) + for (i = 0; i <= mp_maxid; i++) bzero(zpcpu_get_cpu(item, i), zone->uz_size); #else bzero(item, zone->uz_size);