From owner-freebsd-amd64@freebsd.org Sun Jul 1 08:29:24 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A2E8FE31A3; Sun, 1 Jul 2018 08:29:24 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB51B85B35; Sun, 1 Jul 2018 08:29:23 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-wm0-x234.google.com with SMTP id u18-v6so5808102wmc.1; Sun, 01 Jul 2018 01:29:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=K0MZjqff1qkliAawhXMFIp6tdjAjg/+ctyBy/ArE+WQ=; b=Jpi8kYLkyP4BlhHWqWebpbfxfhixET08ttbWiX6Rwp+CiHQQtCRjYqLyie+pAgO3O8 nMHRTAz7NeY214IRPChT2mqTf3P1LAL5Gk7Yglaxut1XfW5jpDuakYxxPHE3TiM0duSr GZvYkgbPv6ygbQNd75yxNm+EHb674Vu1KMqlz3OO9fGdPv82k68tYbRP15mje9VdqDAT /weDIHIPVNyFhUV+8jY38Pr6n+3nLqbgcTiZ1v9v7OM4qBeIXyDFVwhVP9jBq1HkKZ6M LRm5RHxWAuooa4rCAYvin9Vow0ReP9Qs1IX1bwt+FdDi6ICrwo9fKmNCHS6YJsC4V61S ha8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=K0MZjqff1qkliAawhXMFIp6tdjAjg/+ctyBy/ArE+WQ=; b=eLWUE2zLhGdGXw4K28J7OHRt3vh2AEhdrYp74XHckjoWwOFCJtUSIJo0iWAKOZBpE/ 8jT072rxtiGPdnLiC6EyfYt+Cc8nhDLiWnY9VyyIFdscHjgjN52ZN1C0JU0IRd4mxFbq urZgsxyG+1qbsyawzauc16jRaqYA5bgx+sFBahmBHysniFRH0DXXIhCBWTuWf5M1hfsI iyCncnCN9q1v8tpamuJwsrZl+Cltdvmw/rFNWa09JeiDEb+9zQki6pXrB5OB3Z8mGfQD C6NIjKqBzfTkexKS1zGZqD7BP7knr/OfekwxQjwtn+fEBOFFztgi0Xroq5cnSlc3hEJT tJ4Q== X-Gm-Message-State: APt69E2IGeG5kDqrXtCk9H0p0w0BOlhivgM0iVtMdYXuPN6RDqkFw/Ih dxc71lG46U3iocSfSdV41w8= X-Google-Smtp-Source: AAOMgpdSojmlJx6AbgpmCqb/qoXHlgAN/FjazD9uXnuPEzZp8Qmt5zv7H1ff4NVDHm19kx89uAsRwg== X-Received: by 2002:a1c:3f56:: with SMTP id m83-v6mr948847wma.37.1530433762673; Sun, 01 Jul 2018 01:29:22 -0700 (PDT) Received: from pesky.lan (93-34-93-211.ip49.fastwebnet.it. [93.34.93.211]) by smtp.gmail.com with ESMTPSA id s184-v6sm5471626wmf.5.2018.07.01.01.29.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 01 Jul 2018 01:29:21 -0700 (PDT) Sender: Mark Johnston Date: Sun, 1 Jul 2018 04:29:19 -0400 From: Mark Johnston To: Konstantin Belousov Cc: Mihai Carabas , freebsd-amd64@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: Inspect pages created after a vm_object is marked as copy-on-write Message-ID: <20180701082919.GB3926@pesky.lan> References: <20180629225209.GA4238@pesky.lan> <20180630215956.GA1282@pesky.lan> <20180630223401.GW2430@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180630223401.GW2430@kib.kiev.ua> User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Jul 2018 08:29:24 -0000 On Sun, Jul 01, 2018 at 01:34:01AM +0300, Konstantin Belousov wrote: > On Sat, Jun 30, 2018 at 05:59:56PM -0400, Mark Johnston wrote: > > On Sat, Jun 30, 2018 at 10:38:21AM +0300, Mihai Carabas wrote: > > > On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston wrote: > > > > On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote: > > > >> Is there anything I am doing wrong? Maybe I misunderstood something about > > > >> the way the virtual memory works in FreeBSD. > > > > > > > > I'll note that inspecting and manipulating vm_map_entry and vm_object > > > > structures in the bhyve code constitutes something of an abstraction > > > > violation, though it's reasonable to proceed this way while working on a > > > > prototype of the feature. That is, I think you should keep trying your > > > > current approach, but just be aware that you are using the copy-on-write > > > > mechanism in a way that the VM system isn't really expecting. > > > > > > > > > > Can you point out the right approach in our case? > > > > I am merely suggesting that once the required VM interactions are fully > > understood, the mechanism implemented for bhyve should be generalized > > and lifted into the VM code. It's hard to say what the "right" approach > > is, since I don't fully understand the proposed algorithm. It sounds > > like you might be attempting something like: > > > > 1. mark the mappings of to-be-migrated objects as NEEDS_COW, so that a > > subsequent write fault triggers creation of a shadow object > It is actually MAP_ENTRY_COW | MAP_ENTRY_NEEDS_COPY. Indeed. > Note that setting an entry to COW changes the behaviour of mprotect(2), > at least. > > > 2. invalidate all physical mappings of pages in the object to be copied, > > so that subsequent writes trigger a fault > I do not think this is needed to detect writes after the COW is set. > It is enough to remove the write permissions. Same as fork() does, > see the vm_map_copy_entry() code for the handling of MAP_ENTRY_NEEDS_COPY > case. Ah, right. > > 3. copy pages from the backing object to the destination > As I understand, this is done right after the entry is marked > as COW. > > > 4. copy any pages from the shadow object to the desination > And this is done after all backing data is copied and the process is > suspended. Right, I see. Some reading suggests that in general we might perform multiple iterations of this procedure before suspending the process and performing any remaining copies. > > 5. collapse the backing object into the shadow > > 6. if the shadow object exists and was non-empty before the collapse, > > goto 1 > Are you trying to describe how to undo the COW marking ? Marking an > entry as COW really changes its semantic, and we do not need the undo > operation in the base so far. Collapsing the objects would lesser > the pressure on the system pollution with objects, but it does not > change back the meaning of mappings, e.g. their behaviour on inheritance > on fork. I was just thinking about how to avoid creating a long chain of objects if multiple iterations are in fact needed.