From owner-svn-src-head@freebsd.org Wed Sep 13 19:22:08 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 755C5E0B863; Wed, 13 Sep 2017 19:22:08 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FB9B73669; Wed, 13 Sep 2017 19:22:08 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v8DJM7iJ078962; Wed, 13 Sep 2017 19:22:07 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v8DJM7bi078961; Wed, 13 Sep 2017 19:22:07 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201709131922.v8DJM7bi078961@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Wed, 13 Sep 2017 19:22:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r323561 - head/sys/vm X-SVN-Group: head X-SVN-Commit-Author: kib X-SVN-Commit-Paths: head/sys/vm X-SVN-Commit-Revision: 323561 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 19:22:08 -0000 Author: kib Date: Wed Sep 13 19:22:07 2017 New Revision: 323561 URL: https://svnweb.freebsd.org/changeset/base/323561 Log: Do not relock free queue mutex for each page, free whole terminating object' page queue under the single mutex lock. First, all pages on the queue are prepared for free by calls to vm_page_free_prep(), and pages which should not be returned to the physical allocator (e.g. wired or fictitious) are simply removed from the queue. On the second pass, vm_page_free_phys_pglist() inserts all pages from the queue without relocking the mutex. The change improves the object termination, e.g. on the process exit where large anonymous memory objects otherwise cause relocks the free queue mutex for each page. More, if several such processes are exiting or execing in parallel, the mutex was highly contended on the address space demolition. Diagnosed and tested by: mjg (previous version) Reviewed by: alc, markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Modified: head/sys/vm/vm_object.c Modified: head/sys/vm/vm_object.c ============================================================================== --- head/sys/vm/vm_object.c Wed Sep 13 19:12:28 2017 (r323560) +++ head/sys/vm/vm_object.c Wed Sep 13 19:22:07 2017 (r323561) @@ -713,9 +713,14 @@ static void vm_object_terminate_pages(vm_object_t object) { vm_page_t p, p_next; + struct mtx *mtx, *mtx1; + struct vm_pagequeue *pq, *pq1; VM_OBJECT_ASSERT_WLOCKED(object); + mtx = NULL; + pq = NULL; + /* * Free any remaining pageable pages. This also removes them from the * paging queues. However, don't free wired pages, just remove them @@ -724,21 +729,51 @@ vm_object_terminate_pages(vm_object_t object) */ TAILQ_FOREACH_SAFE(p, &object->memq, listq, p_next) { vm_page_assert_unbusied(p); - vm_page_lock(p); - /* - * Optimize the page's removal from the object by resetting - * its "object" field. Specifically, if the page is not - * wired, then the effect of this assignment is that - * vm_page_free()'s call to vm_page_remove() will return - * immediately without modifying the page or the object. - */ + if ((object->flags & OBJ_UNMANAGED) == 0) { + /* + * vm_page_free_prep() only needs the page + * lock for managed pages. + */ + mtx1 = vm_page_lockptr(p); + if (mtx1 != mtx) { + if (mtx != NULL) + mtx_unlock(mtx); + if (pq != NULL) { + vm_pagequeue_unlock(pq); + pq = NULL; + } + mtx = mtx1; + mtx_lock(mtx); + } + } p->object = NULL; - if (p->wire_count == 0) { - vm_page_free(p); - VM_CNT_INC(v_pfree); + if (p->wire_count != 0) + goto unlist; + VM_CNT_INC(v_pfree); + p->flags &= ~PG_ZERO; + if (p->queue != PQ_NONE) { + KASSERT(p->queue < PQ_COUNT, ("vm_object_terminate: " + "page %p is not queued", p)); + pq1 = vm_page_pagequeue(p); + if (pq != pq1) { + if (pq != NULL) + vm_pagequeue_unlock(pq); + pq = pq1; + vm_pagequeue_lock(pq); + } } - vm_page_unlock(p); + if (vm_page_free_prep(p, true)) + continue; +unlist: + TAILQ_REMOVE(&object->memq, p, listq); } + if (pq != NULL) + vm_pagequeue_unlock(pq); + if (mtx != NULL) + mtx_unlock(mtx); + + vm_page_free_phys_pglist(&object->memq); + /* * If the object contained any pages, then reset it to an empty state. * None of the object's fields, including "resident_page_count", were