From owner-svn-src-all@freebsd.org Wed Aug 28 16:16:15 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7CF8FE07B3; Wed, 28 Aug 2019 16:16:15 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46JW7z2knJz4NlR; Wed, 28 Aug 2019 16:16:15 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3B59C9E47; Wed, 28 Aug 2019 16:16:15 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x7SGGE2P020595; Wed, 28 Aug 2019 16:16:14 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x7SGGEmB020592; Wed, 28 Aug 2019 16:16:14 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201908281616.x7SGGEmB020592@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Wed, 28 Aug 2019 16:16:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r351570 - stable/12/sys/vm X-SVN-Group: stable-12 X-SVN-Commit-Author: markj X-SVN-Commit-Paths: stable/12/sys/vm X-SVN-Commit-Revision: 351570 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Aug 2019 16:16:15 -0000 Author: markj Date: Wed Aug 28 16:16:14 2019 New Revision: 351570 URL: https://svnweb.freebsd.org/changeset/base/351570 Log: MFC r351333: Simplify vm_page_dequeue() and fix an assertion. Modified: stable/12/sys/vm/vm_page.c stable/12/sys/vm/vm_page.h stable/12/sys/vm/vm_pagequeue.h Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/vm/vm_page.c ============================================================================== --- stable/12/sys/vm/vm_page.c Wed Aug 28 16:08:06 2019 (r351569) +++ stable/12/sys/vm/vm_page.c Wed Aug 28 16:16:14 2019 (r351570) @@ -3079,21 +3079,15 @@ vm_waitpfault(struct domainset *dset) mtx_unlock(&vm_domainset_lock); } -struct vm_pagequeue * +static struct vm_pagequeue * vm_page_pagequeue(vm_page_t m) { - return (&vm_pagequeue_domain(m)->vmd_pagequeues[m->queue]); -} - -static struct mtx * -vm_page_pagequeue_lockptr(vm_page_t m) -{ uint8_t queue; if ((queue = atomic_load_8(&m->queue)) == PQ_NONE) return (NULL); - return (&vm_pagequeue_domain(m)->vmd_pagequeues[queue].pq_mutex); + return (&vm_pagequeue_domain(m)->vmd_pagequeues[queue]); } static inline void @@ -3116,10 +3110,8 @@ vm_pqbatch_process_page(struct vm_pagequeue *pq, vm_pa m, pq, qflags)); if ((qflags & PGA_DEQUEUE) != 0) { - if (__predict_true((qflags & PGA_ENQUEUED) != 0)) { - TAILQ_REMOVE(&pq->pq_pl, m, plinks.q); - vm_pagequeue_cnt_dec(pq); - } + if (__predict_true((qflags & PGA_ENQUEUED) != 0)) + vm_pagequeue_remove(pq, m); vm_page_dequeue_complete(m); } else if ((qflags & (PGA_REQUEUE | PGA_REQUEUE_HEAD)) != 0) { if ((qflags & PGA_ENQUEUED) != 0) @@ -3293,16 +3285,14 @@ vm_page_dequeue_deferred(vm_page_t m) void vm_page_dequeue(vm_page_t m) { - struct mtx *lock, *lock1; - struct vm_pagequeue *pq; + struct vm_pagequeue *pq, *pq1; uint8_t aflags; - KASSERT(mtx_owned(vm_page_lockptr(m)) || m->order == VM_NFREEORDER, + KASSERT(mtx_owned(vm_page_lockptr(m)) || m->object == NULL, ("page %p is allocated and unlocked", m)); - for (;;) { - lock = vm_page_pagequeue_lockptr(m); - if (lock == NULL) { + for (pq = vm_page_pagequeue(m);; pq = pq1) { + if (pq == NULL) { /* * A thread may be concurrently executing * vm_page_dequeue_complete(). Ensure that all queue @@ -3321,27 +3311,24 @@ vm_page_dequeue(vm_page_t m) * critical section. */ cpu_spinwait(); + pq1 = vm_page_pagequeue(m); continue; } - mtx_lock(lock); - if ((lock1 = vm_page_pagequeue_lockptr(m)) == lock) + vm_pagequeue_lock(pq); + if ((pq1 = vm_page_pagequeue(m)) == pq) break; - mtx_unlock(lock); - lock = lock1; + vm_pagequeue_unlock(pq); } - KASSERT(lock == vm_page_pagequeue_lockptr(m), + KASSERT(pq == vm_page_pagequeue(m), ("%s: page %p migrated directly between queues", __func__, m)); KASSERT((m->aflags & PGA_DEQUEUE) != 0 || mtx_owned(vm_page_lockptr(m)), ("%s: queued unlocked page %p", __func__, m)); - if ((m->aflags & PGA_ENQUEUED) != 0) { - pq = vm_page_pagequeue(m); - TAILQ_REMOVE(&pq->pq_pl, m, plinks.q); - vm_pagequeue_cnt_dec(pq); - } + if ((m->aflags & PGA_ENQUEUED) != 0) + vm_pagequeue_remove(pq, m); vm_page_dequeue_complete(m); - mtx_unlock(lock); + vm_pagequeue_unlock(pq); } /* Modified: stable/12/sys/vm/vm_page.h ============================================================================== --- stable/12/sys/vm/vm_page.h Wed Aug 28 16:08:06 2019 (r351569) +++ stable/12/sys/vm/vm_page.h Wed Aug 28 16:16:14 2019 (r351570) @@ -554,7 +554,6 @@ void vm_page_launder(vm_page_t m); vm_page_t vm_page_lookup (vm_object_t, vm_pindex_t); vm_page_t vm_page_next(vm_page_t m); int vm_page_pa_tryrelock(pmap_t, vm_paddr_t, vm_paddr_t *); -struct vm_pagequeue *vm_page_pagequeue(vm_page_t m); vm_page_t vm_page_prev(vm_page_t m); bool vm_page_ps_test(vm_page_t m, int flags, vm_page_t skip_m); void vm_page_putfake(vm_page_t m); Modified: stable/12/sys/vm/vm_pagequeue.h ============================================================================== --- stable/12/sys/vm/vm_pagequeue.h Wed Aug 28 16:08:06 2019 (r351569) +++ stable/12/sys/vm/vm_pagequeue.h Wed Aug 28 16:16:14 2019 (r351570) @@ -199,6 +199,14 @@ vm_pagequeue_cnt_add(struct vm_pagequeue *pq, int adde #define vm_pagequeue_cnt_dec(pq) vm_pagequeue_cnt_add((pq), -1) static inline void +vm_pagequeue_remove(struct vm_pagequeue *pq, vm_page_t m) +{ + + TAILQ_REMOVE(&pq->pq_pl, m, plinks.q); + vm_pagequeue_cnt_dec(pq); +} + +static inline void vm_batchqueue_init(struct vm_batchqueue *bq) {