From owner-svn-src-all@freebsd.org Thu Jan 23 05:11:02 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6E50422C861; Thu, 23 Jan 2020 05:11:02 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4839N62Ncrz45yS; Thu, 23 Jan 2020 05:11:02 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4D25C96B8; Thu, 23 Jan 2020 05:11:02 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 00N5B2Il067330; Thu, 23 Jan 2020 05:11:02 GMT (envelope-from jeff@FreeBSD.org) Received: (from jeff@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 00N5B26k067329; Thu, 23 Jan 2020 05:11:02 GMT (envelope-from jeff@FreeBSD.org) Message-Id: <202001230511.00N5B26k067329@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jeff set sender to jeff@FreeBSD.org using -f From: Jeff Roberson Date: Thu, 23 Jan 2020 05:11:02 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r357023 - head/sys/vm X-SVN-Group: head X-SVN-Commit-Author: jeff X-SVN-Commit-Paths: head/sys/vm X-SVN-Commit-Revision: 357023 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jan 2020 05:11:02 -0000 Author: jeff Date: Thu Jan 23 05:11:01 2020 New Revision: 357023 URL: https://svnweb.freebsd.org/changeset/base/357023 Log: (fault 4/9) Move copy-on-write into a dedicated function. Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D23304 Modified: head/sys/vm/vm_fault.c Modified: head/sys/vm/vm_fault.c ============================================================================== --- head/sys/vm/vm_fault.c Thu Jan 23 05:07:01 2020 (r357022) +++ head/sys/vm/vm_fault.c Thu Jan 23 05:11:01 2020 (r357023) @@ -846,6 +846,92 @@ vm_fault_relookup(struct faultstate *fs) return (KERN_SUCCESS); } +static void +vm_fault_cow(struct faultstate *fs) +{ + bool is_first_object_locked; + + /* + * This allows pages to be virtually copied from a backing_object + * into the first_object, where the backing object has no other + * refs to it, and cannot gain any more refs. Instead of a bcopy, + * we just move the page from the backing object to the first + * object. Note that we must mark the page dirty in the first + * object so that it will go out to swap when needed. + */ + is_first_object_locked = false; + if ( + /* + * Only one shadow object and no other refs. + */ + fs->object->shadow_count == 1 && fs->object->ref_count == 1 && + /* + * No other ways to look the object up + */ + fs->object->handle == NULL && (fs->object->flags & OBJ_ANON) != 0 && + /* + * We don't chase down the shadow chain and we can acquire locks. + */ + (is_first_object_locked = VM_OBJECT_TRYWLOCK(fs->first_object)) && + fs->object == fs->first_object->backing_object && + VM_OBJECT_TRYWLOCK(fs->object)) { + + /* + * Remove but keep xbusy for replace. fs->m is moved into + * fs->first_object and left busy while fs->first_m is + * conditionally freed. + */ + vm_page_remove_xbusy(fs->m); + vm_page_replace(fs->m, fs->first_object, fs->first_pindex, + fs->first_m); + vm_page_dirty(fs->m); +#if VM_NRESERVLEVEL > 0 + /* + * Rename the reservation. + */ + vm_reserv_rename(fs->m, fs->first_object, fs->object, + OFF_TO_IDX(fs->first_object->backing_object_offset)); +#endif + VM_OBJECT_WUNLOCK(fs->object); + VM_OBJECT_WUNLOCK(fs->first_object); + fs->first_m = fs->m; + fs->m = NULL; + VM_CNT_INC(v_cow_optim); + } else { + if (is_first_object_locked) + VM_OBJECT_WUNLOCK(fs->first_object); + /* + * Oh, well, lets copy it. + */ + pmap_copy_page(fs->m, fs->first_m); + vm_page_valid(fs->first_m); + if (fs->wired && (fs->fault_flags & VM_FAULT_WIRE) == 0) { + vm_page_wire(fs->first_m); + vm_page_unwire(fs->m, PQ_INACTIVE); + } + /* + * Save the cow page to be released after + * pmap_enter is complete. + */ + fs->m_cow = fs->m; + fs->m = NULL; + } + /* + * fs->object != fs->first_object due to above + * conditional + */ + vm_object_pip_wakeup(fs->object); + + /* + * Only use the new page below... + */ + fs->object = fs->first_object; + fs->pindex = fs->first_pindex; + fs->m = fs->first_m; + VM_CNT_INC(v_cow_faults); + curthread->td_cow++; +} + /* * Wait/Retry if the page is busy. We have to do this if the page is * either exclusive or shared busy because the vm_pager may be using @@ -893,7 +979,7 @@ vm_fault(vm_map_t map, vm_offset_t vaddr, vm_prot_t fa int ahead, alloc_req, behind, cluster_offset, faultcount; int nera, oom, result, rv; u_char behavior; - bool dead, hardfault, is_first_object_locked; + bool dead, hardfault; VM_CNT_INC(v_vm_faults); @@ -1302,90 +1388,8 @@ next: * We only really need to copy if we want to write it. */ if ((fs.fault_type & (VM_PROT_COPY | VM_PROT_WRITE)) != 0) { + vm_fault_cow(&fs); /* - * This allows pages to be virtually copied from a - * backing_object into the first_object, where the - * backing object has no other refs to it, and cannot - * gain any more refs. Instead of a bcopy, we just - * move the page from the backing object to the - * first object. Note that we must mark the page - * dirty in the first object so that it will go out - * to swap when needed. - */ - is_first_object_locked = false; - if ( - /* - * Only one shadow object - */ - fs.object->shadow_count == 1 && - /* - * No COW refs, except us - */ - fs.object->ref_count == 1 && - /* - * No one else can look this object up - */ - fs.object->handle == NULL && - /* - * No other ways to look the object up - */ - (fs.object->flags & OBJ_ANON) != 0 && - (is_first_object_locked = VM_OBJECT_TRYWLOCK(fs.first_object)) && - /* - * We don't chase down the shadow chain - */ - fs.object == fs.first_object->backing_object && - VM_OBJECT_TRYWLOCK(fs.object)) { - - /* - * Remove but keep xbusy for replace. fs.m is - * moved into fs.first_object and left busy - * while fs.first_m is conditionally freed. - */ - vm_page_remove_xbusy(fs.m); - vm_page_replace(fs.m, fs.first_object, - fs.first_pindex, fs.first_m); - vm_page_dirty(fs.m); -#if VM_NRESERVLEVEL > 0 - /* - * Rename the reservation. - */ - vm_reserv_rename(fs.m, fs.first_object, - fs.object, OFF_TO_IDX( - fs.first_object->backing_object_offset)); -#endif - VM_OBJECT_WUNLOCK(fs.object); - VM_OBJECT_WUNLOCK(fs.first_object); - fs.first_m = fs.m; - fs.m = NULL; - VM_CNT_INC(v_cow_optim); - } else { - if (is_first_object_locked) - VM_OBJECT_WUNLOCK(fs.first_object); - /* - * Oh, well, lets copy it. - */ - pmap_copy_page(fs.m, fs.first_m); - vm_page_valid(fs.first_m); - if (fs.wired && (fs.fault_flags & - VM_FAULT_WIRE) == 0) { - vm_page_wire(fs.first_m); - vm_page_unwire(fs.m, PQ_INACTIVE); - } - /* - * Save the cow page to be released after - * pmap_enter is complete. - */ - fs.m_cow = fs.m; - fs.m = NULL; - } - /* - * fs.object != fs.first_object due to above - * conditional - */ - vm_object_pip_wakeup(fs.object); - - /* * We only try to prefault read-only mappings to the * neighboring pages when this copy-on-write fault is * a hard fault. In other cases, trying to prefault @@ -1394,14 +1398,6 @@ next: if (faultcount == 0) faultcount = 1; - /* - * Only use the new page below... - */ - fs.object = fs.first_object; - fs.pindex = fs.first_pindex; - fs.m = fs.first_m; - VM_CNT_INC(v_cow_faults); - curthread->td_cow++; } else { fs.prot &= ~VM_PROT_WRITE; }