From nobody Sat Oct 11 05:08:37 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ckBTf753fz6Cd4r; Sat, 11 Oct 2025 05:08:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ckBTf0m0Fz4Fck; Sat, 11 Oct 2025 05:08:38 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760159318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cdTulDSDuuDwohs0yVyHolbySousZH59Qv6DhBSreZo=; b=jzpjY0MdhltOz23Dz9EMt+LBHhN3wXDbiG8h0kZZ2DlXxJmTBCXf1iLqj3knGKos2KtrXM TzY/SQBtgkedTdvNTcwk1xp7iErFgs/aTLXWVdwtO5sk1BUR7vu7TWwmWGR8zvrkVTUbYs Orx/9y9AJnmEZpwaT8S40dhu7HmL/J6SlUNmglmY8MdwMV85I2XGxP+WH2qzV1iOXu0lTR QZZ3TV2+us5/yyT+L6tqGz1AxJ6/ZFkdz5rZWBRPoRy8vYwpm0U4+ILtDR7/v3lva+v7Zd tWVfnrVEXGB7tuf+0zZl1lGbI6BlEQ5aKRH8xL7WkehDhXhzw5+lbRXT7me+4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760159318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cdTulDSDuuDwohs0yVyHolbySousZH59Qv6DhBSreZo=; b=sQwvmdVh0F8yeb36ejD8sZ9gmrm/elme3cjC8cA6M+Annqehh4ennT1GgxfoYMLxwrW2QW zHMUlfXVmgrfNzZDKHYDg0VNwEC5G+8JLpvKM1u9SW9m2TQ3/ZkFqrXmHOfacKr2p94gFx 87fT7lUeAq49aRdqt/77YPGrPq+R/KvXwK/NDtBfhdfpjUwhQhygUonk3siM2yi65coaAZ vBBXCjRmZ4C1XL2YuIE08oUR8Wstx1pf4qlPAp3j/cG3WrnfxG5bq7r5a5KFq/5TNBC8gL JiUPJqHybuionqiIHWiamZ57APRVCXBmOO23LctPES5s8tk4Y7ZYrxtcs10cjg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1760159318; a=rsa-sha256; cv=none; b=qtSPROtPK4KyyV1sPnU2yrl92xwUxj3ykqudogFsnNclMGAJTubxjpL7xS6TnKtQZ1DBbl eTyBoII1eQIFLjG7Lar+mcvuomAgPG5Dafyc3MQVg5XdhwMR3BlMaNwYKKTll9KRgJmy3E JyTvT9RqK7+NLTWbogqqESWZBKzAggqUjLfCP6AxGhHQMYEfBlcHqUV+rnvzWw9ERqZhRk NPN/rees3qMH+jK9dIP6tI8kQN52uj6C6HJtbtOwVKgK1wknMDeVwPCqNUflww7yHJfE9V ZdIysFFnOfiq/CIw+vdNLFev7FfUdi2TftHl5x8mnM9IG1CtCvPS8hvsYZOc2g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ckBTf0DFMz4jL; Sat, 11 Oct 2025 05:08:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 59B58bKC051421; Sat, 11 Oct 2025 05:08:37 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 59B58bAU051418; Sat, 11 Oct 2025 05:08:37 GMT (envelope-from git) Date: Sat, 11 Oct 2025 05:08:37 GMT Message-Id: <202510110508.59B58bAU051418@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Konstantin Belousov Subject: git: 650a9a12c061 - stable/15 - vm_fault: try to only share-busy page for soft faults List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 650a9a12c06178dd8623b57579f250ec26db4bcc Auto-Submitted: auto-generated The branch stable/15 has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=650a9a12c06178dd8623b57579f250ec26db4bcc commit 650a9a12c06178dd8623b57579f250ec26db4bcc Author: Konstantin Belousov AuthorDate: 2025-07-23 10:44:29 +0000 Commit: Konstantin Belousov CommitDate: 2025-10-11 05:08:13 +0000 vm_fault: try to only share-busy page for soft faults (cherry picked from commit 149674bbac5842ac883414a6c1e75d829c70d42b) --- sys/vm/vm_fault.c | 108 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 93 insertions(+), 15 deletions(-) diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c index fff6558fba7e..859b1f9e1a94 100644 --- a/sys/vm/vm_fault.c +++ b/sys/vm/vm_fault.c @@ -202,7 +202,10 @@ vm_fault_page_release(vm_page_t *mp) * pageout while optimizing fault restarts. */ vm_page_deactivate(m); - vm_page_xunbusy(m); + if (vm_page_xbusied(m)) + vm_page_xunbusy(m); + else + vm_page_sunbusy(m); *mp = NULL; } } @@ -333,6 +336,13 @@ vm_fault_dirty(struct faultstate *fs, vm_page_t m) } +static bool +vm_fault_is_read(const struct faultstate *fs) +{ + return ((fs->prot & VM_PROT_WRITE) == 0 && + (fs->fault_type & (VM_PROT_COPY | VM_PROT_WRITE)) == 0); +} + /* * Unlocks fs.first_object and fs.map on success. */ @@ -1016,7 +1026,7 @@ vm_fault_can_cow_rename(struct faultstate *fs) static void vm_fault_cow(struct faultstate *fs) { - bool is_first_object_locked; + bool is_first_object_locked, rename_cow; KASSERT(vm_fault_might_be_cow(fs), ("source and target COW objects are identical")); @@ -1030,13 +1040,29 @@ vm_fault_cow(struct faultstate *fs) * object so that it will go out to swap when needed. */ is_first_object_locked = false; - if (vm_fault_can_cow_rename(fs) && - /* - * We don't chase down the shadow chain and we can acquire locks. - */ - (is_first_object_locked = VM_OBJECT_TRYWLOCK(fs->first_object)) && - fs->object == fs->first_object->backing_object && - VM_OBJECT_TRYWLOCK(fs->object)) { + rename_cow = false; + + if (vm_fault_can_cow_rename(fs) && vm_page_xbusied(fs->m)) { + /* + * Check that we don't chase down the shadow chain and + * we can acquire locks. Recheck the conditions for + * rename after the shadow chain is stable after the + * object locking. + */ + is_first_object_locked = VM_OBJECT_TRYWLOCK(fs->first_object); + if (is_first_object_locked && + fs->object == fs->first_object->backing_object) { + if (VM_OBJECT_TRYWLOCK(fs->object)) { + rename_cow = vm_fault_can_cow_rename(fs); + if (!rename_cow) + VM_OBJECT_WUNLOCK(fs->object); + } + } + } + + if (rename_cow) { + vm_page_assert_xbusied(fs->m); + /* * Remove but keep xbusy for replace. fs->m is moved into * fs->first_object and left busy while fs->first_m is @@ -1093,8 +1119,12 @@ vm_fault_cow(struct faultstate *fs) * address space. If OBJ_ONEMAPPING is set after the check, * removing mappings will at worse trigger some unnecessary page * faults. + * + * In the fs->m shared busy case, the xbusy state of + * fs->first_m prevents new mappings of fs->m from + * being created because a parallel fault on this + * shadow chain should wait for xbusy on fs->first_m. */ - vm_page_assert_xbusied(fs->m_cow); if ((fs->first_object->flags & OBJ_ONEMAPPING) == 0) pmap_remove_all(fs->m_cow); } @@ -1490,6 +1520,51 @@ vm_fault_object(struct faultstate *fs, int *behindp, int *aheadp) vm_page_iter_init(&pages, fs->object); fs->m = vm_radix_iter_lookup(&pages, fs->pindex); if (fs->m != NULL) { + /* + * If the found page is valid, will be either shadowed + * or mapped read-only, and will not be renamed for + * COW, then busy it in shared mode. This allows + * other faults needing this page to proceed in + * parallel. + * + * Unlocked check for validity, rechecked after busy + * is obtained. + */ + if (vm_page_all_valid(fs->m) && + /* + * No write permissions for the new fs->m mapping, + * or the first object has only one mapping, so + * other writeable COW mappings of fs->m cannot + * appear under us. + */ + (vm_fault_is_read(fs) || vm_fault_might_be_cow(fs)) && + /* + * fs->m cannot be renamed from object to + * first_object. These conditions will be + * re-checked with proper synchronization in + * vm_fault_cow(). + */ + (!vm_fault_can_cow_rename(fs) || + fs->object != fs->first_object->backing_object)) { + if (!vm_page_trysbusy(fs->m)) { + vm_fault_busy_sleep(fs, VM_ALLOC_SBUSY); + return (FAULT_RESTART); + } + + /* + * Now make sure that racily checked + * conditions are still valid. + */ + if (__predict_true(vm_page_all_valid(fs->m) && + (vm_fault_is_read(fs) || + vm_fault_might_be_cow(fs)))) { + VM_OBJECT_UNLOCK(fs->object); + return (FAULT_SOFT); + } + + vm_page_sunbusy(fs->m); + } + if (!vm_page_tryxbusy(fs->m)) { vm_fault_busy_sleep(fs, 0); return (FAULT_RESTART); @@ -1725,10 +1800,10 @@ RetryFault: found: /* - * A valid page has been found and exclusively busied. The - * object lock must no longer be held. + * A valid page has been found and busied. The object lock + * must no longer be held if the page was busied. */ - vm_page_assert_xbusied(fs.m); + vm_page_assert_busied(fs.m); VM_OBJECT_ASSERT_UNLOCKED(fs.object); /* @@ -1797,7 +1872,7 @@ found: * Page must be completely valid or it is not fit to * map into user space. vm_pager_get_pages() ensures this. */ - vm_page_assert_xbusied(fs.m); + vm_page_assert_busied(fs.m); KASSERT(vm_page_all_valid(fs.m), ("vm_fault: page %p partially invalid", fs.m)); @@ -1829,7 +1904,10 @@ found: (*fs.m_hold) = fs.m; vm_page_wire(fs.m); } - vm_page_xunbusy(fs.m); + if (vm_page_xbusied(fs.m)) + vm_page_xunbusy(fs.m); + else + vm_page_sunbusy(fs.m); fs.m = NULL; /*