From nobody Sun Sep 14 20:13:23 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cPzr35sGKz67Pfd; Sun, 14 Sep 2025 20:13:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cPzr34jcgz3wrg; Sun, 14 Sep 2025 20:13:23 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1757880803; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=u+VexrqUpskmLxmHuUYrEiaIQl/H4PrH7UF4cLqFiFQ=; b=bzLjo2s7FDWfe1jakQHXcDqnE7gbsPTf5zKVBl5htC+7XzTC7rT6J3aRqTdOdmw611waC9 14H3J9S4bkfz72R/oUdLxElc2Q+lBf9CQVtTrV+tFEqPRUmJxxvGBN7K7I+x7EPrv6bf5T ZrWyuw9R/tlIm2Wl37T+YRrPD414PaFFhnBJYXjD3DYFgoU3MzWta6p7iM7T8rDdDuenRN 0OtT7UDnxWLmW7OHIbP+WBG9kkvUKHuYwGMHqA5k83vQ3rni5Cao1SvtuKMGMNZzEmOxvC 4gLvN5r9ZeloKFJiAg5h4VWTJy1Ij+HgtmlsH5ap7ntsDkoQm5ryRSznKg11cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1757880803; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=u+VexrqUpskmLxmHuUYrEiaIQl/H4PrH7UF4cLqFiFQ=; b=suZoH7voMCnWeZNmDAssr4myQvE+V6NMPX6p3AHtSQuP9LbFumFClphN8qDt6m4Th9mSpd B8z/amtUaDXb2C1rr2qwFoD1QqlELCSyXvjLksWZX2c+NSyrzBQmQzp0Uckzy1pCBMhu/+ QAY0TlXO4rvzoZlVyRvq3kPNsBnxJXDyg/Qhi4CtvWX/kxi1cqLEEKHeaY3mlxl/RuFAZb gH6tqoZ1V+6Q+9flrTgygTxH3xDTtFFtzerg56kevIRIWUAySpAOWU3LDlsJo1EYsAPJcH tbfx3z1oI7ZlTQcbECfZ1wGlLjprr8XZJvAia8uaaht49tePi0D44EI3y742oQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1757880803; a=rsa-sha256; cv=none; b=h4ZQ/7lQkeOzMA242c/R9S7HrH4ydrJK3A5buKimR/26S6w1Ar9cUJBMEnYpQSrsd4f1x+ Y39/B2HBsEzWoEOYKZm5r7CbvRvmjgvFMAE4wVESxyxG3s7bjb3q+nhdk74l4TgK66msWT CS1QDbvP4nZo72T5tAc5YERqvninex7QLveGlkEnNJh0Bvn6nt+UctYX8jCSiYvQaAdxl+ xGeit8X6PBTQpTSP4+H7kSGXUSReJDApnyrNzM6I5Rxm863/zMJpuT/6WDNvfmhup65y2+ uYnFL7scUf8XRk4g/GmSsBoWCn69E/+dhU4hk4gIpOo/TzVgT4Mvvnpf1gQ0Cw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cPzr345f3z849; Sun, 14 Sep 2025 20:13:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 58EKDNNr066133; Sun, 14 Sep 2025 20:13:23 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 58EKDNvA066130; Sun, 14 Sep 2025 20:13:23 GMT (envelope-from git) Date: Sun, 14 Sep 2025 20:13:23 GMT Message-Id: <202509142013.58EKDNvA066130@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Konstantin Belousov Subject: git: 149674bbac58 - main - vm_fault: try to only share-busy page for soft faults List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 149674bbac5842ac883414a6c1e75d829c70d42b Auto-Submitted: auto-generated The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=149674bbac5842ac883414a6c1e75d829c70d42b commit 149674bbac5842ac883414a6c1e75d829c70d42b Author: Konstantin Belousov AuthorDate: 2025-07-23 10:44:29 +0000 Commit: Konstantin Belousov CommitDate: 2025-09-14 20:00:36 +0000 vm_fault: try to only share-busy page for soft faults If the fault handler found a vaild page that is definitely not going to be renamed for COW, try to only sbusy the page. We do not need to validate the page, and parallel faults on the same address are excluded by the xbusy state of the page from the top object. Reviewed by: alc, markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D51474 --- sys/vm/vm_fault.c | 108 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 93 insertions(+), 15 deletions(-) diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c index 27b5f72ec945..427c18c63eb7 100644 --- a/sys/vm/vm_fault.c +++ b/sys/vm/vm_fault.c @@ -202,7 +202,10 @@ vm_fault_page_release(vm_page_t *mp) * pageout while optimizing fault restarts. */ vm_page_deactivate(m); - vm_page_xunbusy(m); + if (vm_page_xbusied(m)) + vm_page_xunbusy(m); + else + vm_page_sunbusy(m); *mp = NULL; } } @@ -333,6 +336,13 @@ vm_fault_dirty(struct faultstate *fs, vm_page_t m) } +static bool +vm_fault_is_read(const struct faultstate *fs) +{ + return ((fs->prot & VM_PROT_WRITE) == 0 && + (fs->fault_type & (VM_PROT_COPY | VM_PROT_WRITE)) == 0); +} + /* * Unlocks fs.first_object and fs.map on success. */ @@ -1019,7 +1029,7 @@ vm_fault_can_cow_rename(struct faultstate *fs) static void vm_fault_cow(struct faultstate *fs) { - bool is_first_object_locked; + bool is_first_object_locked, rename_cow; KASSERT(vm_fault_might_be_cow(fs), ("source and target COW objects are identical")); @@ -1033,13 +1043,29 @@ vm_fault_cow(struct faultstate *fs) * object so that it will go out to swap when needed. */ is_first_object_locked = false; - if (vm_fault_can_cow_rename(fs) && - /* - * We don't chase down the shadow chain and we can acquire locks. - */ - (is_first_object_locked = VM_OBJECT_TRYWLOCK(fs->first_object)) && - fs->object == fs->first_object->backing_object && - VM_OBJECT_TRYWLOCK(fs->object)) { + rename_cow = false; + + if (vm_fault_can_cow_rename(fs) && vm_page_xbusied(fs->m)) { + /* + * Check that we don't chase down the shadow chain and + * we can acquire locks. Recheck the conditions for + * rename after the shadow chain is stable after the + * object locking. + */ + is_first_object_locked = VM_OBJECT_TRYWLOCK(fs->first_object); + if (is_first_object_locked && + fs->object == fs->first_object->backing_object) { + if (VM_OBJECT_TRYWLOCK(fs->object)) { + rename_cow = vm_fault_can_cow_rename(fs); + if (!rename_cow) + VM_OBJECT_WUNLOCK(fs->object); + } + } + } + + if (rename_cow) { + vm_page_assert_xbusied(fs->m); + /* * Remove but keep xbusy for replace. fs->m is moved into * fs->first_object and left busy while fs->first_m is @@ -1096,8 +1122,12 @@ vm_fault_cow(struct faultstate *fs) * address space. If OBJ_ONEMAPPING is set after the check, * removing mappings will at worse trigger some unnecessary page * faults. + * + * In the fs->m shared busy case, the xbusy state of + * fs->first_m prevents new mappings of fs->m from + * being created because a parallel fault on this + * shadow chain should wait for xbusy on fs->first_m. */ - vm_page_assert_xbusied(fs->m_cow); if ((fs->first_object->flags & OBJ_ONEMAPPING) == 0) pmap_remove_all(fs->m_cow); } @@ -1493,6 +1523,51 @@ vm_fault_object(struct faultstate *fs, int *behindp, int *aheadp) vm_page_iter_init(&pages, fs->object); fs->m = vm_radix_iter_lookup(&pages, fs->pindex); if (fs->m != NULL) { + /* + * If the found page is valid, will be either shadowed + * or mapped read-only, and will not be renamed for + * COW, then busy it in shared mode. This allows + * other faults needing this page to proceed in + * parallel. + * + * Unlocked check for validity, rechecked after busy + * is obtained. + */ + if (vm_page_all_valid(fs->m) && + /* + * No write permissions for the new fs->m mapping, + * or the first object has only one mapping, so + * other writeable COW mappings of fs->m cannot + * appear under us. + */ + (vm_fault_is_read(fs) || vm_fault_might_be_cow(fs)) && + /* + * fs->m cannot be renamed from object to + * first_object. These conditions will be + * re-checked with proper synchronization in + * vm_fault_cow(). + */ + (!vm_fault_can_cow_rename(fs) || + fs->object != fs->first_object->backing_object)) { + if (!vm_page_trysbusy(fs->m)) { + vm_fault_busy_sleep(fs, VM_ALLOC_SBUSY); + return (FAULT_RESTART); + } + + /* + * Now make sure that racily checked + * conditions are still valid. + */ + if (__predict_true(vm_page_all_valid(fs->m) && + (vm_fault_is_read(fs) || + vm_fault_might_be_cow(fs)))) { + VM_OBJECT_UNLOCK(fs->object); + return (FAULT_SOFT); + } + + vm_page_sunbusy(fs->m); + } + if (!vm_page_tryxbusy(fs->m)) { vm_fault_busy_sleep(fs, 0); return (FAULT_RESTART); @@ -1707,10 +1782,10 @@ RetryFault: found: /* - * A valid page has been found and exclusively busied. The - * object lock must no longer be held. + * A valid page has been found and busied. The object lock + * must no longer be held if the page was busied. */ - vm_page_assert_xbusied(fs.m); + vm_page_assert_busied(fs.m); VM_OBJECT_ASSERT_UNLOCKED(fs.object); /* @@ -1779,7 +1854,7 @@ found: * Page must be completely valid or it is not fit to * map into user space. vm_pager_get_pages() ensures this. */ - vm_page_assert_xbusied(fs.m); + vm_page_assert_busied(fs.m); KASSERT(vm_page_all_valid(fs.m), ("vm_fault: page %p partially invalid", fs.m)); @@ -1811,7 +1886,10 @@ found: (*fs.m_hold) = fs.m; vm_page_wire(fs.m); } - vm_page_xunbusy(fs.m); + if (vm_page_xbusied(fs.m)) + vm_page_xunbusy(fs.m); + else + vm_page_sunbusy(fs.m); fs.m = NULL; /*