From owner-svn-src-all@freebsd.org Sun Feb 3 18:43:21 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80CBD14AE940; Sun, 3 Feb 2019 18:43:21 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2009B818D4; Sun, 3 Feb 2019 18:43:21 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 14011229B1; Sun, 3 Feb 2019 18:43:21 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x13IhKo4040608; Sun, 3 Feb 2019 18:43:20 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x13IhK0Y040606; Sun, 3 Feb 2019 18:43:20 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201902031843.x13IhK0Y040606@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Sun, 3 Feb 2019 18:43:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r343720 - head/sys/vm X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: head/sys/vm X-SVN-Commit-Revision: 343720 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 2009B818D4 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.96)[-0.961,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Feb 2019 18:43:21 -0000 Author: markj Date: Sun Feb 3 18:43:20 2019 New Revision: 343720 URL: https://svnweb.freebsd.org/changeset/base/343720 Log: Allow vm_page_free_prep() to dequeue pages without the page lock. This is a step towards being able to free pages without the page lock held. The approach is simply to add an implementation of vm_page_dequeue_deferred() which does not assert that the page lock is held. Formally, the page lock is required to set PGA_DEQUEUE, but in the case of vm_page_free_prep() we get the same mutual exclusion for free by virtue of the fact that no other references to the page may exist. No functional change intended. Reviewed by: kib (previous version) MFC after: 2 weeks Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D19065 Modified: head/sys/vm/vm_page.c head/sys/vm/vm_page.h Modified: head/sys/vm/vm_page.c ============================================================================== --- head/sys/vm/vm_page.c Sun Feb 3 18:38:58 2019 (r343719) +++ head/sys/vm/vm_page.c Sun Feb 3 18:43:20 2019 (r343720) @@ -3175,7 +3175,11 @@ vm_pqbatch_submit_page(vm_page_t m, uint8_t queue) struct vm_pagequeue *pq; int domain; - vm_page_assert_locked(m); + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("page %p is unmanaged", m)); + KASSERT(mtx_owned(vm_page_lockptr(m)) || + (m->object == NULL && (m->aflags & PGA_DEQUEUE) != 0), + ("missing synchronization for page %p", m)); KASSERT(queue < PQ_COUNT, ("invalid queue %d", queue)); domain = vm_phys_domain(m); @@ -3197,8 +3201,9 @@ vm_pqbatch_submit_page(vm_page_t m, uint8_t queue) /* * The page may have been logically dequeued before we acquired the - * page queue lock. In this case, the page lock prevents the page - * from being logically enqueued elsewhere. + * page queue lock. In this case, since we either hold the page lock + * or the page is being freed, a different thread cannot be concurrently + * enqueuing the page. */ if (__predict_true(m->queue == queue)) vm_pqbatch_process_page(pq, m); @@ -3290,6 +3295,30 @@ vm_page_dequeue_deferred(vm_page_t m) } /* + * A variant of vm_page_dequeue_deferred() that does not assert the page + * lock and is only to be called from vm_page_free_prep(). It is just an + * open-coded implementation of vm_page_dequeue_deferred(). Because the + * page is being freed, we can assume that nothing else is scheduling queue + * operations on this page, so we get for free the mutual exclusion that + * is otherwise provided by the page lock. + */ +static void +vm_page_dequeue_deferred_free(vm_page_t m) +{ + uint8_t queue; + + KASSERT(m->object == NULL, ("page %p has an object reference", m)); + + if ((m->aflags & PGA_DEQUEUE) != 0) + return; + atomic_thread_fence_acq(); + if ((queue = m->queue) == PQ_NONE) + return; + vm_page_aflag_set(m, PGA_DEQUEUE); + vm_pqbatch_submit_page(m, queue); +} + +/* * vm_page_dequeue: * * Remove the page from whichever page queue it's in, if any. @@ -3474,7 +3503,7 @@ vm_page_free_prep(vm_page_t m) * dequeue. */ if ((m->oflags & VPO_UNMANAGED) == 0) - vm_page_dequeue_deferred(m); + vm_page_dequeue_deferred_free(m); m->valid = 0; vm_page_undirty(m); Modified: head/sys/vm/vm_page.h ============================================================================== --- head/sys/vm/vm_page.h Sun Feb 3 18:38:58 2019 (r343719) +++ head/sys/vm/vm_page.h Sun Feb 3 18:43:20 2019 (r343720) @@ -351,8 +351,10 @@ extern struct mtx_padalign pa_lock[]; * queue, and cleared when the dequeue request is processed. A page may * have PGA_DEQUEUE set and PGA_ENQUEUED cleared, for instance if a dequeue * is requested after the page is scheduled to be enqueued but before it is - * actually inserted into the page queue. The page lock must be held to set - * this flag, and the queue lock for the page must be held to clear it. + * actually inserted into the page queue. For allocated pages, the page lock + * must be held to set this flag, but it may be set by vm_page_free_prep() + * without the page lock held. The page queue lock must be held to clear the + * PGA_DEQUEUE flag. * * PGA_REQUEUE is set when the page is scheduled to be enqueued or requeued * in its page queue. The page lock must be held to set this flag, and the