From owner-svn-src-head@freebsd.org Mon Apr 17 17:34:54 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21621D42C2E; Mon, 17 Apr 2017 17:34:54 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B309A1468; Mon, 17 Apr 2017 17:34:53 +0000 (UTC) (envelope-from glebius@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v3HHYq5n022998; Mon, 17 Apr 2017 17:34:52 GMT (envelope-from glebius@FreeBSD.org) Received: (from glebius@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v3HHYlf5022945; Mon, 17 Apr 2017 17:34:47 GMT (envelope-from glebius@FreeBSD.org) Message-Id: <201704171734.v3HHYlf5022945@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: glebius set sender to glebius@FreeBSD.org using -f From: Gleb Smirnoff Date: Mon, 17 Apr 2017 17:34:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r317061 - in head: libexec/rpc.rstatd sys/amd64/amd64 sys/amd64/include sys/arm/arm sys/arm/include sys/arm64/include sys/cddl/contrib/opensolaris/uts/common/fs/zfs sys/compat/linprocfs... X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Apr 2017 17:34:54 -0000 Author: glebius Date: Mon Apr 17 17:34:47 2017 New Revision: 317061 URL: https://svnweb.freebsd.org/changeset/base/317061 Log: - Remove 'struct vmmeter' from 'struct pcpu', leaving only global vmmeter in place. To do per-cpu stats, convert all fields that previously were maintained in the vmmeters that sit in pcpus to counter(9). - Since some vmmeter stats may be touched at very early stages of boot, before we have set up UMA and we can do counter_u64_alloc(), provide an early counter mechanism: o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter. o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter, so that at early stages of boot, before counters are allocated we already point to a counter that can be safely written to. o For sparc64 that required a whole dummy pcpu[MAXCPU] array. Further related changes: - Don't include vmmeter.h into pcpu.h. - vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit, to match kernel representation. - struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion. This is based on benno@'s 4-year old patch: https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html Reviewed by: kib, gallatin, marius, lidl Differential Revision: https://reviews.freebsd.org/D10156 Modified: head/libexec/rpc.rstatd/rstat_proc.c head/sys/amd64/amd64/trap.c head/sys/amd64/include/atomic.h head/sys/amd64/include/counter.h head/sys/amd64/include/pcpu.h head/sys/arm/arm/intr.c head/sys/arm/arm/trap-v4.c head/sys/arm/arm/trap-v6.c head/sys/arm/arm/undefined.c head/sys/arm/include/counter.h head/sys/arm/include/pcpu.h head/sys/arm64/include/counter.h head/sys/arm64/include/pcpu.h head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c head/sys/compat/linprocfs/linprocfs.c head/sys/fs/fuse/fuse_vnops.c head/sys/fs/nfsclient/nfs_clbio.c head/sys/fs/smbfs/smbfs_io.c head/sys/i386/i386/trap.c head/sys/i386/include/atomic.h head/sys/i386/include/counter.h head/sys/i386/include/pcpu.h head/sys/kern/kern_fork.c head/sys/kern/kern_intr.c head/sys/kern/kern_synch.c head/sys/kern/kern_thread.c head/sys/kern/subr_intr.c head/sys/kern/subr_syscall.c head/sys/kern/subr_trap.c head/sys/kern/vfs_bio.c head/sys/mips/include/counter.h head/sys/mips/include/intr_machdep.h head/sys/mips/include/pcpu.h head/sys/powerpc/include/counter.h head/sys/powerpc/include/pcpu.h head/sys/powerpc/powerpc/trap.c head/sys/sparc64/include/counter.h head/sys/sparc64/include/pcpu.h head/sys/sparc64/sparc64/exception.S head/sys/sparc64/sparc64/genassym.c head/sys/sparc64/sparc64/intr_machdep.c head/sys/sparc64/sparc64/machdep.c head/sys/sparc64/sparc64/trap.c head/sys/sys/pcpu.h head/sys/sys/vmmeter.h head/sys/vm/swap_pager.c head/sys/vm/vm_fault.c head/sys/vm/vm_meter.c head/sys/vm/vm_object.c head/sys/vm/vm_page.c head/sys/vm/vm_pageout.c head/sys/vm/vnode_pager.c head/sys/x86/acpica/srat.c head/sys/x86/x86/intr_machdep.c head/usr.bin/top/machine.c head/usr.bin/vmstat/vmstat.c Modified: head/libexec/rpc.rstatd/rstat_proc.c ============================================================================== --- head/libexec/rpc.rstatd/rstat_proc.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/libexec/rpc.rstatd/rstat_proc.c Mon Apr 17 17:34:47 2017 (r317061) @@ -170,6 +170,7 @@ updatestat(void) struct timeval tm, btm; int mib[6]; size_t len; + uint64_t val; int ifcount; #ifdef DEBUG @@ -229,11 +230,12 @@ updatestat(void) #endif #define FETCH_CNT(stat, cnt) do { \ - len = sizeof((stat)); \ - if (sysctlbyname("vm.stats." #cnt , &(stat), &len, 0, 0) < 0) { \ - syslog(LOG_ERR, "sysctl(vm.stats." #cnt "): %m"); \ + len = sizeof(uint64_t); \ + if (sysctlbyname("vm.stats." #cnt , &val, &len, NULL, 0) < 0) { \ + syslog(LOG_ERR, "sysctl(vm.stats." #cnt "): %m"); \ exit(1); \ } \ + stat = val; \ } while (0) FETCH_CNT(stats_all.s1.v_pgpgin, vm.v_vnodepgsin); Modified: head/sys/amd64/amd64/trap.c ============================================================================== --- head/sys/amd64/amd64/trap.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/amd64/amd64/trap.c Mon Apr 17 17:34:47 2017 (r317061) @@ -176,7 +176,7 @@ trap(struct trapframe *frame) register_t addr = 0; ksiginfo_t ksi; - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); type = frame->tf_trapno; #ifdef SMP Modified: head/sys/amd64/include/atomic.h ============================================================================== --- head/sys/amd64/include/atomic.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/amd64/include/atomic.h Mon Apr 17 17:34:47 2017 (r317061) @@ -348,7 +348,7 @@ atomic_testandclear_long(volatile u_long * avoid a dependency on sys/pcpu.h in machine/atomic.h consumers. * An assertion in amd64/vm_machdep.c ensures that the value is correct. */ -#define OFFSETOF_MONITORBUF 0x180 +#define OFFSETOF_MONITORBUF 0x100 #if defined(SMP) static __inline void Modified: head/sys/amd64/include/counter.h ============================================================================== --- head/sys/amd64/include/counter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/amd64/include/counter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -31,7 +31,9 @@ #include -extern struct pcpu __pcpu[1]; +extern struct pcpu __pcpu[]; + +#define EARLY_COUNTER &__pcpu[0].pc_early_dummy_counter #define counter_enter() do {} while (0) #define counter_exit() do {} while (0) Modified: head/sys/amd64/include/pcpu.h ============================================================================== --- head/sys/amd64/include/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/amd64/include/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -66,7 +66,7 @@ uint32_t pc_pcid_next; \ uint32_t pc_pcid_gen; \ uint32_t pc_smp_tlb_done; /* TLB op acknowledgement */ \ - char __pad[145] /* be divisor of PAGE_SIZE \ + char __pad[384] /* be divisor of PAGE_SIZE \ after cache alignment */ #define PC_DBREG_CMD_NONE 0 Modified: head/sys/arm/arm/intr.c ============================================================================== --- head/sys/arm/arm/intr.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm/arm/intr.c Mon Apr 17 17:34:47 2017 (r317061) @@ -185,7 +185,7 @@ intr_irq_handler(struct trapframe *frame struct intr_event *event; int i; - PCPU_INC(cnt.v_intr); + VM_CNT_INC(v_intr); i = -1; while ((i = arm_get_next_irq(i)) != -1) { intrcnt[i]++; Modified: head/sys/arm/arm/trap-v4.c ============================================================================== --- head/sys/arm/arm/trap-v4.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm/arm/trap-v4.c Mon Apr 17 17:34:47 2017 (r317061) @@ -204,7 +204,7 @@ abort_handler(struct trapframe *tf, int td = curthread; p = td->td_proc; - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); /* Data abort came from user mode? */ user = TRAP_USERMODE(tf); @@ -615,7 +615,7 @@ prefetch_abort_handler(struct trapframe td = curthread; p = td->td_proc; - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); if (TRAP_USERMODE(tf)) { td->td_frame = tf; Modified: head/sys/arm/arm/trap-v6.c ============================================================================== --- head/sys/arm/arm/trap-v6.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm/arm/trap-v6.c Mon Apr 17 17:34:47 2017 (r317061) @@ -291,7 +291,7 @@ abort_handler(struct trapframe *tf, int void *onfault; #endif - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); td = curthread; fsr = (prefetch) ? cp15_ifsr_get(): cp15_dfsr_get(); Modified: head/sys/arm/arm/undefined.c ============================================================================== --- head/sys/arm/arm/undefined.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm/arm/undefined.c Mon Apr 17 17:34:47 2017 (r317061) @@ -201,7 +201,7 @@ undefinedinstruction(struct trapframe *f if (__predict_true(frame->tf_spsr & PSR_F) == 0) enable_interrupts(PSR_F); - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); fault_pc = frame->tf_pc; Modified: head/sys/arm/include/counter.h ============================================================================== --- head/sys/arm/include/counter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm/include/counter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -32,6 +32,10 @@ #include #include +extern struct pcpu __pcpu[]; + +#define EARLY_COUNTER &__pcpu[0].pc_early_dummy_counter + #define counter_enter() do {} while (0) #define counter_exit() do {} while (0) Modified: head/sys/arm/include/pcpu.h ============================================================================== --- head/sys/arm/include/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm/include/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -57,10 +57,10 @@ struct vmspace; void *pc_qmap_pte2p; \ unsigned int pc_dbreg[32]; \ int pc_dbreg_cmd; \ - char __pad[27] + char __pad[155] #else #define PCPU_MD_FIELDS \ - char __pad[157] + char __pad[93] #endif #ifdef _KERNEL Modified: head/sys/arm64/include/counter.h ============================================================================== --- head/sys/arm64/include/counter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm64/include/counter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -32,6 +32,10 @@ #include #include +extern struct pcpu __pcpu[]; + +#define EARLY_COUNTER &__pcpu[0].pc_early_dummy_counter + #define counter_enter() do {} while (0) #define counter_exit() do {} while (0) Modified: head/sys/arm64/include/pcpu.h ============================================================================== --- head/sys/arm64/include/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/arm64/include/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -39,7 +39,7 @@ u_int pc_acpi_id; /* ACPI CPU id */ \ u_int pc_midr; /* stored MIDR value */ \ uint64_t pc_clock; \ - char __pad[113] + char __pad[241] #ifdef _KERNEL Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c ============================================================================== --- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c Mon Apr 17 17:34:47 2017 (r317061) @@ -4575,8 +4575,8 @@ zfs_getpages(struct vnode *vp, vm_page_t return (zfs_vm_pagerret_bad); } - PCPU_INC(cnt.v_vnodein); - PCPU_ADD(cnt.v_vnodepgsin, count); + VM_CNT_INC(v_vnodein); + VM_CNT_ADD(v_vnodepgsin, count); lsize = PAGE_SIZE; if (IDX_TO_OFF(mlast->pindex) + lsize > object->un_pager.vnp.vnp_size) @@ -4758,8 +4758,8 @@ zfs_putpages(struct vnode *vp, vm_page_t vm_page_undirty(ma[i]); } zfs_vmobject_wunlock(object); - PCPU_INC(cnt.v_vnodeout); - PCPU_ADD(cnt.v_vnodepgsout, ncount); + VM_CNT_INC(v_vnodeout); + VM_CNT_ADD(v_vnodepgsout, ncount); } dmu_tx_commit(tx); Modified: head/sys/compat/linprocfs/linprocfs.c ============================================================================== --- head/sys/compat/linprocfs/linprocfs.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/compat/linprocfs/linprocfs.c Mon Apr 17 17:34:47 2017 (r317061) @@ -425,17 +425,17 @@ linprocfs_dostat(PFS_FILL_ARGS) } sbuf_printf(sb, "disk 0 0 0 0\n" - "page %u %u\n" - "swap %u %u\n" - "intr %u\n" - "ctxt %u\n" + "page %ju %ju\n" + "swap %ju %ju\n" + "intr %ju\n" + "ctxt %ju\n" "btime %lld\n", - vm_cnt.v_vnodepgsin, - vm_cnt.v_vnodepgsout, - vm_cnt.v_swappgsin, - vm_cnt.v_swappgsout, - vm_cnt.v_intr, - vm_cnt.v_swtch, + (uintmax_t)VM_CNT_FETCH(v_vnodepgsin), + (uintmax_t)VM_CNT_FETCH(v_vnodepgsout), + (uintmax_t)VM_CNT_FETCH(v_swappgsin), + (uintmax_t)VM_CNT_FETCH(v_swappgsout), + (uintmax_t)VM_CNT_FETCH(v_intr), + (uintmax_t)VM_CNT_FETCH(v_swtch), (long long)boottime.tv_sec); return (0); } Modified: head/sys/fs/fuse/fuse_vnops.c ============================================================================== --- head/sys/fs/fuse/fuse_vnops.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/fs/fuse/fuse_vnops.c Mon Apr 17 17:34:47 2017 (r317061) @@ -1793,8 +1793,8 @@ fuse_vnop_getpages(struct vop_getpages_a kva = (vm_offset_t)bp->b_data; pmap_qenter(kva, pages, npages); - PCPU_INC(cnt.v_vnodein); - PCPU_ADD(cnt.v_vnodepgsin, npages); + VM_CNT_INC(v_vnodein); + VM_CNT_ADD(v_vnodepgsin, npages); count = npages << PAGE_SHIFT; iov.iov_base = (caddr_t)kva; @@ -1927,8 +1927,8 @@ fuse_vnop_putpages(struct vop_putpages_a kva = (vm_offset_t)bp->b_data; pmap_qenter(kva, pages, npages); - PCPU_INC(cnt.v_vnodeout); - PCPU_ADD(cnt.v_vnodepgsout, count); + VM_CNT_INC(v_vnodeout); + VM_CNT_ADD(v_vnodepgsout, count); iov.iov_base = (caddr_t)kva; iov.iov_len = count; Modified: head/sys/fs/nfsclient/nfs_clbio.c ============================================================================== --- head/sys/fs/nfsclient/nfs_clbio.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/fs/nfsclient/nfs_clbio.c Mon Apr 17 17:34:47 2017 (r317061) @@ -184,8 +184,8 @@ ncl_getpages(struct vop_getpages_args *a kva = (vm_offset_t) bp->b_data; pmap_qenter(kva, pages, npages); - PCPU_INC(cnt.v_vnodein); - PCPU_ADD(cnt.v_vnodepgsin, npages); + VM_CNT_INC(v_vnodein); + VM_CNT_ADD(v_vnodepgsin, npages); count = npages << PAGE_SHIFT; iov.iov_base = (caddr_t) kva; @@ -320,8 +320,8 @@ ncl_putpages(struct vop_putpages_args *a } mtx_unlock(&np->n_mtx); - PCPU_INC(cnt.v_vnodeout); - PCPU_ADD(cnt.v_vnodepgsout, count); + VM_CNT_INC(v_vnodeout); + VM_CNT_ADD(v_vnodepgsout, count); iov.iov_base = unmapped_buf; iov.iov_len = count; Modified: head/sys/fs/smbfs/smbfs_io.c ============================================================================== --- head/sys/fs/smbfs/smbfs_io.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/fs/smbfs/smbfs_io.c Mon Apr 17 17:34:47 2017 (r317061) @@ -470,8 +470,8 @@ smbfs_getpages(ap) kva = (vm_offset_t) bp->b_data; pmap_qenter(kva, pages, npages); - PCPU_INC(cnt.v_vnodein); - PCPU_ADD(cnt.v_vnodepgsin, npages); + VM_CNT_INC(v_vnodein); + VM_CNT_ADD(v_vnodepgsin, npages); count = npages << PAGE_SHIFT; iov.iov_base = (caddr_t) kva; @@ -595,8 +595,8 @@ smbfs_putpages(ap) kva = (vm_offset_t) bp->b_data; pmap_qenter(kva, pages, npages); - PCPU_INC(cnt.v_vnodeout); - PCPU_ADD(cnt.v_vnodepgsout, count); + VM_CNT_INC(v_vnodeout); + VM_CNT_ADD(v_vnodepgsout, count); iov.iov_base = (caddr_t) kva; iov.iov_len = count; Modified: head/sys/i386/i386/trap.c ============================================================================== --- head/sys/i386/i386/trap.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/i386/i386/trap.c Mon Apr 17 17:34:47 2017 (r317061) @@ -192,7 +192,7 @@ trap(struct trapframe *frame) static int lastalert = 0; #endif - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); type = frame->tf_trapno; #ifdef SMP Modified: head/sys/i386/include/atomic.h ============================================================================== --- head/sys/i386/include/atomic.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/i386/include/atomic.h Mon Apr 17 17:34:47 2017 (r317061) @@ -45,7 +45,7 @@ * avoid a dependency on sys/pcpu.h in machine/atomic.h consumers. * An assertion in i386/vm_machdep.c ensures that the value is correct. */ -#define __OFFSETOF_MONITORBUF 0x180 +#define __OFFSETOF_MONITORBUF 0x80 static __inline void __mbk(void) Modified: head/sys/i386/include/counter.h ============================================================================== --- head/sys/i386/include/counter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/i386/include/counter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -36,6 +36,10 @@ #include #include +extern struct pcpu __pcpu[]; + +#define EARLY_COUNTER &__pcpu[0].pc_early_dummy_counter + #define counter_enter() do { \ if ((cpu_feature & CPUID_CX8) == 0) \ critical_enter(); \ @@ -46,8 +50,6 @@ critical_exit(); \ } while (0) -extern struct pcpu __pcpu[MAXCPU]; - static inline void counter_64_inc_8b(uint64_t *p, int64_t inc) { Modified: head/sys/i386/include/pcpu.h ============================================================================== --- head/sys/i386/include/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/i386/include/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -68,7 +68,7 @@ caddr_t pc_cmap_addr2; \ vm_offset_t pc_qmap_addr; /* KVA for temporary mappings */\ uint32_t pc_smp_tlb_done; /* TLB op acknowledgement */ \ - char __pad[189] + char __pad[445] #ifdef _KERNEL Modified: head/sys/kern/kern_fork.c ============================================================================== --- head/sys/kern/kern_fork.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/kern_fork.c Mon Apr 17 17:34:47 2017 (r317061) @@ -664,20 +664,20 @@ do_fork(struct thread *td, struct fork_r vm_forkproc(td, p2, td2, vm2, fr->fr_flags); if (fr->fr_flags == (RFFDG | RFPROC)) { - PCPU_INC(cnt.v_forks); - PCPU_ADD(cnt.v_forkpages, p2->p_vmspace->vm_dsize + + VM_CNT_INC(v_forks); + VM_CNT_ADD(v_forkpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } else if (fr->fr_flags == (RFFDG | RFPROC | RFPPWAIT | RFMEM)) { - PCPU_INC(cnt.v_vforks); - PCPU_ADD(cnt.v_vforkpages, p2->p_vmspace->vm_dsize + + VM_CNT_INC(v_vforks); + VM_CNT_ADD(v_vforkpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } else if (p1 == &proc0) { - PCPU_INC(cnt.v_kthreads); - PCPU_ADD(cnt.v_kthreadpages, p2->p_vmspace->vm_dsize + + VM_CNT_INC(v_kthreads); + VM_CNT_ADD(v_kthreadpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } else { - PCPU_INC(cnt.v_rforks); - PCPU_ADD(cnt.v_rforkpages, p2->p_vmspace->vm_dsize + + VM_CNT_INC(v_rforks); + VM_CNT_ADD(v_rforkpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } Modified: head/sys/kern/kern_intr.c ============================================================================== --- head/sys/kern/kern_intr.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/kern_intr.c Mon Apr 17 17:34:47 2017 (r317061) @@ -1156,7 +1156,7 @@ swi_sched(void *cookie, int flags) ih->ih_need = 1; if (!(flags & SWI_DELAY)) { - PCPU_INC(cnt.v_soft); + VM_CNT_INC(v_soft); #ifdef INTR_FILTER error = intr_event_schedule_thread(ie, ie->ie_thread); #else Modified: head/sys/kern/kern_synch.c ============================================================================== --- head/sys/kern/kern_synch.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/kern_synch.c Mon Apr 17 17:34:47 2017 (r317061) @@ -426,7 +426,7 @@ mi_switch(int flags, struct thread *newt td->td_incruntime += runtime; PCPU_SET(switchtime, new_switchtime); td->td_generation++; /* bump preempt-detect counter */ - PCPU_INC(cnt.v_swtch); + VM_CNT_INC(v_swtch); PCPU_SET(switchticks, ticks); CTR4(KTR_PROC, "mi_switch: old thread %ld (td_sched %p, pid %ld, %s)", td->td_tid, td_get_sched(td), td->td_proc->p_pid, td->td_name); Modified: head/sys/kern/kern_thread.c ============================================================================== --- head/sys/kern/kern_thread.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/kern_thread.c Mon Apr 17 17:34:47 2017 (r317061) @@ -546,7 +546,7 @@ thread_exit(void) td->td_incruntime += runtime; PCPU_SET(switchtime, new_switchtime); PCPU_SET(switchticks, ticks); - PCPU_INC(cnt.v_swtch); + VM_CNT_INC(v_swtch); /* Save our resource usage in our process. */ td->td_ru.ru_nvcsw++; Modified: head/sys/kern/subr_intr.c ============================================================================== --- head/sys/kern/subr_intr.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/subr_intr.c Mon Apr 17 17:34:47 2017 (r317061) @@ -292,7 +292,7 @@ intr_irq_handler(struct trapframe *tf) KASSERT(irq_root_filter != NULL, ("%s: no filter", __func__)); - PCPU_INC(cnt.v_intr); + VM_CNT_INC(v_intr); critical_enter(); td = curthread; oldframe = td->td_intr_frame; Modified: head/sys/kern/subr_syscall.c ============================================================================== --- head/sys/kern/subr_syscall.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/subr_syscall.c Mon Apr 17 17:34:47 2017 (r317061) @@ -58,7 +58,7 @@ syscallenter(struct thread *td, struct s struct proc *p; int error, traced; - PCPU_INC(cnt.v_syscall); + VM_CNT_INC(v_syscall); p = td->td_proc; td->td_pticks = 0; Modified: head/sys/kern/subr_trap.c ============================================================================== --- head/sys/kern/subr_trap.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/subr_trap.c Mon Apr 17 17:34:47 2017 (r317061) @@ -234,7 +234,7 @@ ast(struct trapframe *framep) td->td_flags &= ~(TDF_ASTPENDING | TDF_NEEDSIGCHK | TDF_NEEDSUSPCHK | TDF_NEEDRESCHED | TDF_ALRMPEND | TDF_PROFPEND | TDF_MACPEND); thread_unlock(td); - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); if (td->td_cowgen != p->p_cowgen) thread_cow_update(td); Modified: head/sys/kern/vfs_bio.c ============================================================================== --- head/sys/kern/vfs_bio.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/kern/vfs_bio.c Mon Apr 17 17:34:47 2017 (r317061) @@ -4769,8 +4769,8 @@ vfs_bio_getpages(struct vnode *vp, vm_pa pgsin += pgsin_a; if (rahead != NULL) *rahead = pgsin_a; - PCPU_INC(cnt.v_vnodein); - PCPU_ADD(cnt.v_vnodepgsin, pgsin); + VM_CNT_INC(v_vnodein); + VM_CNT_ADD(v_vnodepgsin, pgsin); br_flags = (mp != NULL && (mp->mnt_kern_flag & MNTK_UNMAPPED_BUFS) != 0) ? GB_UNMAPPED : 0; Modified: head/sys/mips/include/counter.h ============================================================================== --- head/sys/mips/include/counter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/mips/include/counter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -34,6 +34,8 @@ #include #endif +#define EARLY_COUNTER &((struct pcpu *)pcpu_space)->pc_early_dummy_counter + #define counter_enter() critical_enter() #define counter_exit() critical_exit() Modified: head/sys/mips/include/intr_machdep.h ============================================================================== --- head/sys/mips/include/intr_machdep.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/mips/include/intr_machdep.h Mon Apr 17 17:34:47 2017 (r317061) @@ -71,6 +71,6 @@ mips_intrcnt_inc(mips_intrcnt_t counter) { if (counter) atomic_add_long(counter, 1); - PCPU_INC(cnt.v_intr); + VM_CNT_INC(v_intr); } #endif /* !_MACHINE_INTR_MACHDEP_H_ */ Modified: head/sys/mips/include/pcpu.h ============================================================================== --- head/sys/mips/include/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/mips/include/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -45,11 +45,11 @@ #ifdef __mips_n64 #define PCPU_MD_MIPS64_FIELDS \ PCPU_MD_COMMON_FIELDS \ - char __pad[53] + char __pad[245] #else #define PCPU_MD_MIPS32_FIELDS \ PCPU_MD_COMMON_FIELDS \ - char __pad[189] + char __pad[125] #endif #ifdef __mips_n64 Modified: head/sys/powerpc/include/counter.h ============================================================================== --- head/sys/powerpc/include/counter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/powerpc/include/counter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -34,6 +34,10 @@ #include #endif +extern struct pcpu __pcpu[]; + +#define EARLY_COUNTER &__pcpu[0].pc_early_dummy_counter + #ifdef __powerpc64__ #define counter_enter() do {} while (0) @@ -79,8 +83,6 @@ counter_u64_zero_inline(counter_u64_t c) #define counter_u64_add_protected(c, i) counter_u64_add(c, i) -extern struct pcpu __pcpu[MAXCPU]; - static inline void counter_u64_add(counter_u64_t c, int64_t inc) { Modified: head/sys/powerpc/include/pcpu.h ============================================================================== --- head/sys/powerpc/include/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/powerpc/include/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -57,7 +57,7 @@ struct pvo_entry; vm_offset_t pc_qmap_addr; \ struct pvo_entry *pc_qmap_pvo; \ struct mtx pc_qmap_lock; \ - /* char __pad[0] */ + char __pad[128] #define PCPU_MD_AIM64_FIELDS \ struct slb pc_slb[64]; \ @@ -67,7 +67,7 @@ struct pvo_entry; vm_offset_t pc_qmap_addr; \ struct pvo_entry *pc_qmap_pvo; \ struct mtx pc_qmap_lock; \ - char __pad[1121 - sizeof(struct mtx)] + char __pad[1345] #ifdef __powerpc64__ #define PCPU_MD_AIM_FIELDS PCPU_MD_AIM64_FIELDS @@ -81,9 +81,9 @@ struct pvo_entry; #define BOOKE_TLBSAVE_LEN (BOOKE_TLB_SAVELEN * BOOKE_TLB_MAXNEST) #ifdef __powerpc64__ -#define BOOKE_PCPU_PAD 773 +#define BOOKE_PCPU_PAD 901 #else -#define BOOKE_PCPU_PAD 173 +#define BOOKE_PCPU_PAD 429 #endif #define PCPU_MD_BOOKE_FIELDS \ register_t pc_booke_critsave[BOOKE_CRITSAVE_LEN]; \ Modified: head/sys/powerpc/powerpc/trap.c ============================================================================== --- head/sys/powerpc/powerpc/trap.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/powerpc/powerpc/trap.c Mon Apr 17 17:34:47 2017 (r317061) @@ -171,7 +171,7 @@ trap(struct trapframe *frame) u_int ucode; ksiginfo_t ksi; - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); td = curthread; p = td->td_proc; Modified: head/sys/sparc64/include/counter.h ============================================================================== --- head/sys/sparc64/include/counter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sparc64/include/counter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -34,6 +34,9 @@ #include #endif +extern struct pcpu dummy_pcpu[]; +#define EARLY_COUNTER &dummy_pcpu[0].pc_early_dummy_counter + #define counter_enter() critical_enter() #define counter_exit() critical_exit() Modified: head/sys/sparc64/include/pcpu.h ============================================================================== --- head/sys/sparc64/include/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sparc64/include/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -62,7 +62,7 @@ struct pmap; u_int pc_tlb_ctx; \ u_int pc_tlb_ctx_max; \ u_int pc_tlb_ctx_min; \ - char __pad[397] + char __pad[653] #ifdef _KERNEL Modified: head/sys/sparc64/sparc64/exception.S ============================================================================== --- head/sys/sparc64/sparc64/exception.S Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sparc64/sparc64/exception.S Mon Apr 17 17:34:47 2017 (r317061) @@ -2476,9 +2476,8 @@ ENTRY(tl0_intr) inc %l1 stx %l1, [%l0] - lduw [PCPU(CNT) + V_INTR], %l0 - inc %l0 - stw %l0, [PCPU(CNT) + V_INTR] + call counter_intr_inc + nop ba,a %xcc, tl0_ret nop @@ -2989,11 +2988,8 @@ ENTRY(tl1_intr) add %l5, %l4, %l4 ldx [%l4], %l5 inc %l5 - stx %l5, [%l4] - - lduw [PCPU(CNT) + V_INTR], %l4 - inc %l4 - stw %l4, [PCPU(CNT) + V_INTR] + call counter_intr_inc + stx %l5, [%l4] ldx [%sp + SPOFF + CCFSZ + TF_Y], %l4 Modified: head/sys/sparc64/sparc64/genassym.c ============================================================================== --- head/sys/sparc64/sparc64/genassym.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sparc64/sparc64/genassym.c Mon Apr 17 17:34:47 2017 (r317061) @@ -134,7 +134,6 @@ ASSYM(PC_CPUID, offsetof(struct pcpu, pc ASSYM(PC_IRHEAD, offsetof(struct pcpu, pc_irhead)); ASSYM(PC_IRTAIL, offsetof(struct pcpu, pc_irtail)); ASSYM(PC_IRFREE, offsetof(struct pcpu, pc_irfree)); -ASSYM(PC_CNT, offsetof(struct pcpu, pc_cnt)); ASSYM(PC_SIZEOF, sizeof(struct pcpu)); ASSYM(PC_CACHE, offsetof(struct pcpu, pc_cache)); Modified: head/sys/sparc64/sparc64/intr_machdep.c ============================================================================== --- head/sys/sparc64/sparc64/intr_machdep.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sparc64/sparc64/intr_machdep.c Mon Apr 17 17:34:47 2017 (r317061) @@ -123,6 +123,7 @@ static void intr_stray_level(struct trap static void intr_stray_vector(void *); static int intrcnt_setname(const char *, int); static void intrcnt_updatename(int, const char *, int); +void counter_intr_inc(void); static void intrcnt_updatename(int vec, const char *name, int ispil) @@ -451,6 +452,19 @@ intr_describe(int vec, void *ih, const c return (error); } +/* + * Do VM_CNT_INC(intr), being in the interrupt context already. This is + * called from assembly. + * To avoid counter_enter() and appropriate assertion, unwrap VM_CNT_INC() + * and hardcode the actual increment. + */ +void +counter_intr_inc(void) +{ + + *(uint64_t *)zpcpu_get(vm_cnt.v_intr) += 1; +} + #ifdef SMP /* * Support for balancing interrupt sources across CPUs. For now we just Modified: head/sys/sparc64/sparc64/machdep.c ============================================================================== --- head/sys/sparc64/sparc64/machdep.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sparc64/sparc64/machdep.c Mon Apr 17 17:34:47 2017 (r317061) @@ -126,6 +126,7 @@ long realmem; void *dpcpu0; char pcpu0[PCPU_PAGES * PAGE_SIZE]; +struct pcpu dummy_pcpu[MAXCPU]; struct trapframe frame0; vm_offset_t kstack0; Modified: head/sys/sparc64/sparc64/trap.c ============================================================================== --- head/sys/sparc64/sparc64/trap.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sparc64/sparc64/trap.c Mon Apr 17 17:34:47 2017 (r317061) @@ -267,7 +267,7 @@ trap(struct trapframe *tf) trap_msg[tf->tf_type & ~T_KERNEL], (TRAPF_USERMODE(tf) ? "user" : "kernel"), rdpr(pil)); - PCPU_INC(cnt.v_trap); + VM_CNT_INC(v_trap); if ((tf->tf_tstate & TSTATE_PRIV) == 0) { KASSERT(td != NULL, ("trap: curthread NULL")); Modified: head/sys/sys/pcpu.h ============================================================================== --- head/sys/sys/pcpu.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sys/pcpu.h Mon Apr 17 17:34:47 2017 (r317061) @@ -43,7 +43,6 @@ #include #include #include -#include #include #include @@ -158,7 +157,6 @@ struct pcpu { u_int pc_cpuid; /* This cpu number */ STAILQ_ENTRY(pcpu) pc_allcpu; struct lock_list_entry *pc_spinlocks; - struct vmmeter pc_cnt; /* VM stats counters */ long pc_cp_time[CPUSTATES]; /* statclock ticks */ struct device *pc_device; void *pc_netisr; /* netisr SWI cookie */ @@ -166,6 +164,7 @@ struct pcpu { int pc_domain; /* Memory domain. */ struct rm_queue pc_rm_queue; /* rmlock list of trackers */ uintptr_t pc_dynamic; /* Dynamic per-cpu data area */ + uint64_t pc_early_dummy_counter; /* Startup time counter(9) */ /* * Keep MD fields last, so that CPU-specific variations on a Modified: head/sys/sys/vmmeter.h ============================================================================== --- head/sys/sys/vmmeter.h Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/sys/vmmeter.h Mon Apr 17 17:34:47 2017 (r317061) @@ -39,50 +39,84 @@ */ #define MAXSLP 20 +/* Systemwide totals computed every five seconds. */ +struct vmtotal { + int16_t t_rq; /* length of the run queue */ + int16_t t_dw; /* jobs in ``disk wait'' (neg priority) */ + int16_t t_pw; /* jobs in page wait */ + int16_t t_sl; /* jobs sleeping in core */ + int16_t t_sw; /* swapped out runnable/short block jobs */ + int32_t t_vm; /* total virtual memory */ + int32_t t_avm; /* active virtual memory */ + int32_t t_rm; /* total real memory in use */ + int32_t t_arm; /* active real memory */ + int32_t t_vmshr; /* shared virtual memory */ + int32_t t_avmshr; /* active shared virtual memory */ + int32_t t_rmshr; /* shared real memory */ + int32_t t_armshr; /* active shared real memory */ + int32_t t_free; /* free memory pages */ +}; + +#if defined(_KERNEL) || defined(_WANT_VMMETER) +#include + /* * System wide statistics counters. * Locking: * a - locked by atomic operations * c - constant after initialization * f - locked by vm_page_queue_free_mtx - * p - locked by being in the PCPU and atomicity respect to interrupts + * p - uses counter(9) * q - changes are synchronized by the corresponding vm_pagequeue lock */ struct vmmeter { /* * General system activity. */ - u_int v_swtch; /* (p) context switches */ - u_int v_trap; /* (p) calls to trap */ - u_int v_syscall; /* (p) calls to syscall() */ - u_int v_intr; /* (p) device interrupts */ - u_int v_soft; /* (p) software interrupts */ + counter_u64_t v_swtch; /* (p) context switches */ + counter_u64_t v_trap; /* (p) calls to trap */ + counter_u64_t v_syscall; /* (p) calls to syscall() */ + counter_u64_t v_intr; /* (p) device interrupts */ + counter_u64_t v_soft; /* (p) software interrupts */ /* * Virtual memory activity. */ - u_int v_vm_faults; /* (p) address memory faults */ - u_int v_io_faults; /* (p) page faults requiring I/O */ - u_int v_cow_faults; /* (p) copy-on-writes faults */ - u_int v_cow_optim; /* (p) optimized copy-on-writes faults */ - u_int v_zfod; /* (p) pages zero filled on demand */ - u_int v_ozfod; /* (p) optimized zero fill pages */ - u_int v_swapin; /* (p) swap pager pageins */ - u_int v_swapout; /* (p) swap pager pageouts */ - u_int v_swappgsin; /* (p) swap pager pages paged in */ - u_int v_swappgsout; /* (p) swap pager pages paged out */ - u_int v_vnodein; /* (p) vnode pager pageins */ - u_int v_vnodeout; /* (p) vnode pager pageouts */ - u_int v_vnodepgsin; /* (p) vnode_pager pages paged in */ - u_int v_vnodepgsout; /* (p) vnode pager pages paged out */ - u_int v_intrans; /* (p) intransit blocking page faults */ - u_int v_reactivated; /* (p) pages reactivated by the pagedaemon */ - u_int v_pdwakeups; /* (p) times daemon has awaken from sleep */ - u_int v_pdpages; /* (p) pages analyzed by daemon */ - u_int v_pdshortfalls; /* (p) page reclamation shortfalls */ - - u_int v_dfree; /* (p) pages freed by daemon */ - u_int v_pfree; /* (p) pages freed by exiting processes */ - u_int v_tfree; /* (p) total pages freed */ + counter_u64_t v_vm_faults; /* (p) address memory faults */ + counter_u64_t v_io_faults; /* (p) page faults requiring I/O */ + counter_u64_t v_cow_faults; /* (p) copy-on-writes faults */ + counter_u64_t v_cow_optim; /* (p) optimized COW faults */ + counter_u64_t v_zfod; /* (p) pages zero filled on demand */ + counter_u64_t v_ozfod; /* (p) optimized zero fill pages */ + counter_u64_t v_swapin; /* (p) swap pager pageins */ + counter_u64_t v_swapout; /* (p) swap pager pageouts */ + counter_u64_t v_swappgsin; /* (p) swap pager pages paged in */ + counter_u64_t v_swappgsout; /* (p) swap pager pages paged out */ + counter_u64_t v_vnodein; /* (p) vnode pager pageins */ + counter_u64_t v_vnodeout; /* (p) vnode pager pageouts */ + counter_u64_t v_vnodepgsin; /* (p) vnode_pager pages paged in */ + counter_u64_t v_vnodepgsout; /* (p) vnode pager pages paged out */ + counter_u64_t v_intrans; /* (p) intransit blocking page faults */ + counter_u64_t v_reactivated; /* (p) reactivated by the pagedaemon */ + counter_u64_t v_pdwakeups; /* (p) times daemon has awaken */ + counter_u64_t v_pdpages; /* (p) pages analyzed by daemon */ + counter_u64_t v_pdshortfalls; /* (p) page reclamation shortfalls */ + + counter_u64_t v_dfree; /* (p) pages freed by daemon */ + counter_u64_t v_pfree; /* (p) pages freed by processes */ + counter_u64_t v_tfree; /* (p) total pages freed */ + /* + * Fork/vfork/rfork activity. + */ + counter_u64_t v_forks; /* (p) fork() calls */ + counter_u64_t v_vforks; /* (p) vfork() calls */ + counter_u64_t v_rforks; /* (p) rfork() calls */ + counter_u64_t v_kthreads; /* (p) fork() calls by kernel */ + counter_u64_t v_forkpages; /* (p) pages affected by fork() */ + counter_u64_t v_vforkpages; /* (p) pages affected by vfork() */ + counter_u64_t v_rforkpages; /* (p) pages affected by rfork() */ + counter_u64_t v_kthreadpages; /* (p) ... and by kernel fork() */ +#define VM_METER_NCOUNTERS \ + (offsetof(struct vmmeter, v_page_size) / sizeof(counter_u64_t)) /* * Distribution of page usages. */ @@ -100,24 +134,18 @@ struct vmmeter { u_int v_pageout_free_min; /* (c) min pages reserved for kernel */ u_int v_interrupt_free_min; /* (c) reserved pages for int code */ u_int v_free_severe; /* (c) severe page depletion point */ - /* - * Fork/vfork/rfork activity. - */ - u_int v_forks; /* (p) fork() calls */ - u_int v_vforks; /* (p) vfork() calls */ - u_int v_rforks; /* (p) rfork() calls */ - u_int v_kthreads; /* (p) fork() calls by kernel */ - u_int v_forkpages; /* (p) VM pages affected by fork() */ - u_int v_vforkpages; /* (p) VM pages affected by vfork() */ - u_int v_rforkpages; /* (p) VM pages affected by rfork() */ - u_int v_kthreadpages; /* (p) VM pages affected by fork() by kernel */ }; +#endif /* _KERNEL || _WANT_VMMETER */ + #ifdef _KERNEL extern struct vmmeter vm_cnt; - extern u_int vm_pageout_wakeup_thresh; +#define VM_CNT_ADD(var, x) counter_u64_add(vm_cnt.var, x) +#define VM_CNT_INC(var) VM_CNT_ADD(var, 1) +#define VM_CNT_FETCH(var) counter_u64_fetch(vm_cnt.var) + /* * Return TRUE if we are under our severe low-free-pages threshold * @@ -189,33 +217,5 @@ vm_laundry_target(void) return (vm_paging_target()); } - -/* - * Obtain the value of a per-CPU counter. - */ -#define VM_METER_PCPU_CNT(member) \ - vm_meter_cnt(__offsetof(struct vmmeter, member)) - -u_int vm_meter_cnt(size_t); - -#endif - -/* systemwide totals computed every five seconds */ -struct vmtotal { - int16_t t_rq; /* length of the run queue */ - int16_t t_dw; /* jobs in ``disk wait'' (neg priority) */ - int16_t t_pw; /* jobs in page wait */ - int16_t t_sl; /* jobs sleeping in core */ - int16_t t_sw; /* swapped out runnable/short block jobs */ - int32_t t_vm; /* total virtual memory */ - int32_t t_avm; /* active virtual memory */ - int32_t t_rm; /* total real memory in use */ - int32_t t_arm; /* active real memory */ - int32_t t_vmshr; /* shared virtual memory */ - int32_t t_avmshr; /* active shared virtual memory */ - int32_t t_rmshr; /* shared real memory */ - int32_t t_armshr; /* active shared real memory */ - int32_t t_free; /* free memory pages */ -}; - -#endif +#endif /* _KERNEL */ +#endif /* _SYS_VMMETER_H_ */ Modified: head/sys/vm/swap_pager.c ============================================================================== --- head/sys/vm/swap_pager.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/vm/swap_pager.c Mon Apr 17 17:34:47 2017 (r317061) @@ -1180,8 +1180,8 @@ swap_pager_getpages(vm_object_t object, bp->b_pgbefore = rbehind != NULL ? *rbehind : 0; bp->b_pgafter = rahead != NULL ? *rahead : 0; - PCPU_INC(cnt.v_swapin); - PCPU_ADD(cnt.v_swappgsin, count); + VM_CNT_INC(v_swapin); + VM_CNT_ADD(v_swappgsin, count); /* * perform the I/O. NOTE!!! bp cannot be considered valid after @@ -1205,7 +1205,7 @@ swap_pager_getpages(vm_object_t object, VM_OBJECT_WLOCK(object); while ((m[0]->oflags & VPO_SWAPINPROG) != 0) { m[0]->oflags |= VPO_SWAPSLEEP; - PCPU_INC(cnt.v_intrans); + VM_CNT_INC(v_intrans); if (VM_OBJECT_SLEEP(object, &object->paging_in_progress, PSWP, "swread", hz * 20)) { printf( @@ -1393,8 +1393,8 @@ swap_pager_putpages(vm_object_t object, bp->b_dirtyoff = 0; bp->b_dirtyend = bp->b_bcount; - PCPU_INC(cnt.v_swapout); - PCPU_ADD(cnt.v_swappgsout, bp->b_npages); + VM_CNT_INC(v_swapout); + VM_CNT_ADD(v_swappgsout, bp->b_npages); /* * We unconditionally set rtvals[] to VM_PAGER_PEND so that we Modified: head/sys/vm/vm_fault.c ============================================================================== --- head/sys/vm/vm_fault.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/vm/vm_fault.c Mon Apr 17 17:34:47 2017 (r317061) @@ -497,7 +497,7 @@ vm_fault_hold(vm_map_t map, vm_offset_t boolean_t wired; /* Passed by reference. */ bool dead, growstack, hardfault, is_first_object_locked; - PCPU_INC(cnt.v_vm_faults); + VM_CNT_INC(v_vm_faults); fs.vp = NULL; faultcount = 0; nera = -1; @@ -673,7 +673,7 @@ RetryFault:; } vm_object_pip_wakeup(fs.object); VM_OBJECT_WUNLOCK(fs.object); - PCPU_INC(cnt.v_intrans); + VM_CNT_INC(v_intrans); vm_object_deallocate(fs.first_object); goto RetryFault; } @@ -999,9 +999,9 @@ readrest: if ((fs.m->flags & PG_ZERO) == 0) { pmap_zero_page(fs.m); } else { - PCPU_INC(cnt.v_ozfod); + VM_CNT_INC(v_ozfod); } - PCPU_INC(cnt.v_zfod); + VM_CNT_INC(v_zfod); fs.m->valid = VM_PAGE_BITS_ALL; /* Don't try to prefault neighboring pages. */ faultcount = 1; @@ -1095,7 +1095,7 @@ readrest: vm_page_xbusy(fs.m); fs.first_m = fs.m; fs.m = NULL; - PCPU_INC(cnt.v_cow_optim); + VM_CNT_INC(v_cow_optim); } else { /* * Oh, well, lets copy it. @@ -1131,7 +1131,7 @@ readrest: fs.m = fs.first_m; if (!is_first_object_locked) VM_OBJECT_WLOCK(fs.object); - PCPU_INC(cnt.v_cow_faults); + VM_CNT_INC(v_cow_faults); curthread->td_cow++; } else { prot &= ~VM_PROT_WRITE; @@ -1246,7 +1246,7 @@ readrest: */ unlock_and_deallocate(&fs); if (hardfault) { - PCPU_INC(cnt.v_io_faults); + VM_CNT_INC(v_io_faults); curthread->td_ru.ru_majflt++; #ifdef RACCT if (racct_enable && fs.object->type == OBJT_VNODE) { Modified: head/sys/vm/vm_meter.c ============================================================================== --- head/sys/vm/vm_meter.c Mon Apr 17 17:23:19 2017 (r317060) +++ head/sys/vm/vm_meter.c Mon Apr 17 17:34:47 2017 (r317061) @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include #include @@ -55,7 +56,52 @@ __FBSDID("$FreeBSD$"); #include #include -struct vmmeter vm_cnt; +struct vmmeter vm_cnt = { + .v_swtch = EARLY_COUNTER, *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***