From owner-svn-src-head@freebsd.org Sun Jan 15 03:50:09 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6662BCB0BEA; Sun, 15 Jan 2017 03:50:09 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AE0D1007; Sun, 15 Jan 2017 03:50:09 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v0F3o80b030076; Sun, 15 Jan 2017 03:50:08 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v0F3o8oK030074; Sun, 15 Jan 2017 03:50:08 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201701150350.v0F3o8oK030074@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Sun, 15 Jan 2017 03:50:08 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r312208 - head/sys/vm X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Jan 2017 03:50:09 -0000 Author: markj Date: Sun Jan 15 03:50:08 2017 New Revision: 312208 URL: https://svnweb.freebsd.org/changeset/base/312208 Log: Avoid unnecessary page lookups in vm_object_madvise(). vm_object_madvise() is frequently used to apply advice to a contiguous set of pages in an object with no backing object. Optimize this case by skipping non-resident subranges in constant time, and by iterating over resident pages using the object memq, thus avoiding radix tree lookups on each page index in the specified range. While here, move MADV_WILLNEED handling to vm_page_advise(), and rename the "advise" parameter to vm_object_madvise() to "advice." Reviewed by: alc, kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D9098 Modified: head/sys/vm/vm_object.c head/sys/vm/vm_page.c Modified: head/sys/vm/vm_object.c ============================================================================== --- head/sys/vm/vm_object.c Sun Jan 15 01:36:45 2017 (r312207) +++ head/sys/vm/vm_object.c Sun Jan 15 03:50:08 2017 (r312208) @@ -1097,7 +1097,7 @@ vm_object_sync(vm_object_t object, vm_oo */ void vm_object_madvise(vm_object_t object, vm_pindex_t pindex, vm_pindex_t end, - int advise) + int advice) { vm_pindex_t tpindex; vm_object_t backing_object, tobject; @@ -1105,11 +1105,9 @@ vm_object_madvise(vm_object_t object, vm if (object == NULL) return; + VM_OBJECT_WLOCK(object); - /* - * Locate and adjust resident pages - */ - for (; pindex < end; pindex += 1) { + for (m = NULL; pindex < end; pindex++) { relookup: tobject = object; tpindex = pindex; @@ -1118,7 +1116,7 @@ shadowlookup: * MADV_FREE only operates on OBJT_DEFAULT or OBJT_SWAP pages * and those pages must be OBJ_ONEMAPPING. */ - if (advise == MADV_FREE) { + if (advice == MADV_FREE) { if ((tobject->type != OBJT_DEFAULT && tobject->type != OBJT_SWAP) || (tobject->flags & OBJ_ONEMAPPING) == 0) { @@ -1126,15 +1124,29 @@ shadowlookup: } } else if ((tobject->flags & OBJ_UNMANAGED) != 0) goto unlock_tobject; - m = vm_page_lookup(tobject, tpindex); - if (m == NULL) { - /* - * There may be swap even if there is no backing page - */ - if (advise == MADV_FREE && tobject->type == OBJT_SWAP) + + /* + * In the common case where the object has no backing object, we + * can avoid performing lookups at each pindex. In either case, + * when applying MADV_FREE we take care to release any swap + * space used to store non-resident pages. + */ + if (object->backing_object == NULL) { + m = (m != NULL) ? TAILQ_NEXT(m, listq) : + vm_page_find_least(object, pindex); + tpindex = (m != NULL && m->pindex < end) ? + m->pindex : end; + if (advice == MADV_FREE && object->type == OBJT_SWAP && + tpindex > pindex) + swap_pager_freespace(object, pindex, + tpindex - pindex); + if ((pindex = tpindex) == end) + break; + } else if ((m = vm_page_lookup(tobject, tpindex)) == NULL) { + if (advice == MADV_FREE && tobject->type == OBJT_SWAP) swap_pager_freespace(tobject, tpindex, 1); /* - * next object + * Prepare to search the next object in the chain. */ backing_object = tobject->backing_object; if (backing_object == NULL) @@ -1145,11 +1157,13 @@ shadowlookup: VM_OBJECT_WUNLOCK(tobject); tobject = backing_object; goto shadowlookup; - } else if (m->valid != VM_PAGE_BITS_ALL) - goto unlock_tobject; + } + /* * If the page is not in a normal state, skip it. */ + if (m->valid != VM_PAGE_BITS_ALL) + goto unlock_tobject; vm_page_lock(m); if (m->hold_count != 0 || m->wire_count != 0) { vm_page_unlock(m); @@ -1160,7 +1174,7 @@ shadowlookup: KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("vm_object_madvise: page %p is not managed", m)); if (vm_page_busied(m)) { - if (advise == MADV_WILLNEED) { + if (advice == MADV_WILLNEED) { /* * Reference the page before unlocking and * sleeping so that the page daemon is less @@ -1172,21 +1186,18 @@ shadowlookup: VM_OBJECT_WUNLOCK(object); VM_OBJECT_WUNLOCK(tobject); vm_page_busy_sleep(m, "madvpo", false); + m = NULL; VM_OBJECT_WLOCK(object); goto relookup; } - if (advise == MADV_WILLNEED) { - vm_page_activate(m); - } else { - vm_page_advise(m, advise); - } + vm_page_advise(m, advice); vm_page_unlock(m); - if (advise == MADV_FREE && tobject->type == OBJT_SWAP) + if (advice == MADV_FREE && tobject->type == OBJT_SWAP) swap_pager_freespace(tobject, tpindex, 1); unlock_tobject: if (tobject != object) VM_OBJECT_WUNLOCK(tobject); - } + } VM_OBJECT_WUNLOCK(object); } Modified: head/sys/vm/vm_page.c ============================================================================== --- head/sys/vm/vm_page.c Sun Jan 15 01:36:45 2017 (r312207) +++ head/sys/vm/vm_page.c Sun Jan 15 03:50:08 2017 (r312208) @@ -3002,7 +3002,7 @@ vm_page_try_to_free(vm_page_t m) /* * vm_page_advise * - * Deactivate or do nothing, as appropriate. + * Apply the specified advice to the given page. * * The object and page must be locked. */ @@ -3020,8 +3020,11 @@ vm_page_advise(vm_page_t m, int advice) * would result in a page fault on a later access. */ vm_page_undirty(m); - else if (advice != MADV_DONTNEED) + else if (advice != MADV_DONTNEED) { + if (advice == MADV_WILLNEED) + vm_page_activate(m); return; + } /* * Clear any references to the page. Otherwise, the page daemon will