From owner-freebsd-amd64@freebsd.org Sat Jun 30 22:00:03 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39402FDCEE5; Sat, 30 Jun 2018 22:00:03 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9CC3C8E6C8; Sat, 30 Jun 2018 22:00:02 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id z13-v6so5213706wma.5; Sat, 30 Jun 2018 15:00:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=uv0iM8A3Q5bi8kbUBZ8u4JhxiCCULZru6pX2TLh/qss=; b=dqu00l/JKA0d0MYRsiJzXWV+csykYRRSAfLjW8oN6lxv7s88Lc8BkQCF5kSrVp4kgp 9JEAnD2fLrhuo3yZXhNgoKx+an2YYmS3GScJuv5FQE73VTq+YqSU4cFgmE1gMq1mVVD0 EDKDDofrR9PgpYhR3dVGn8NhewRJE+nghFmosC/MpWGWsXBKuFXfWFXsfTptzP0bOBDQ Tus/MUXRchJXHCWCyr0fMzLkMl8FYlYsD+XS6/pIf5fOEeaC/yTTdvdESLxbjWT5qHMS TIOpbVRaDEH1u60x0DphxyZp2tknF75p4TiKHD2NxDcFbYYA52OaSdDB9LaX9nJFQZJc JxSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=uv0iM8A3Q5bi8kbUBZ8u4JhxiCCULZru6pX2TLh/qss=; b=MEP7SIZkbq1/tsGiuI5RT9ItreBKp7/S0E3EYowhReXyVsed4DGvwduon5r7GgxHa1 pJpKu2cyzp0SojDFGK8TJ5kpb6dU6VP9KfgkPLCZ6yC5I19OpKko3csjulMRMeEqzWRA /bTsLfXjGR6kOePITPQzhStB9OeARqf1t+nx3g2fJ72PU/8zJUbbaqCzdCHxpjM2Q0q7 uvmqQmGrTzT6fyl1+74dVyRdrwNz6C5GBWJRZ49U5WQqXvZQuPpFwYaVCM8msm3MHaxU TNhSmTxoQTyuwrJ5wXv3mnAwLKGmChWpSW1cNadSUg1qQST52BbHJ9Sq7cP8txr8Incy BNHA== X-Gm-Message-State: APt69E3HPB8a/qc4/kltEpTTrh0VDsqCyd85Xmj3TY1EFKcMW9qVE0GL ky/Nm+U03DaDb87f6t7mA37M/g== X-Google-Smtp-Source: AAOMgpeWb7tOWpceP3apEqviLOtZ1BZhvohSEqttxcFDgUMCDvAv7/xdowIQJlfkmgmv1n/Jr/Y4yw== X-Received: by 2002:a1c:a103:: with SMTP id k3-v6mr4955409wme.161.1530396001452; Sat, 30 Jun 2018 15:00:01 -0700 (PDT) Received: from pesky.lan (93-34-93-211.ip49.fastwebnet.it. [93.34.93.211]) by smtp.gmail.com with ESMTPSA id c17-v6sm5814944wrp.54.2018.06.30.14.59.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 30 Jun 2018 15:00:00 -0700 (PDT) Sender: Mark Johnston Date: Sat, 30 Jun 2018 17:59:56 -0400 From: Mark Johnston To: Mihai Carabas Cc: Elena Mihailescu , freebsd-virtualization@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: Inspect pages created after a vm_object is marked as copy-on-write Message-ID: <20180630215956.GA1282@pesky.lan> References: <20180629225209.GA4238@pesky.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 22:00:03 -0000 On Sat, Jun 30, 2018 at 10:38:21AM +0300, Mihai Carabas wrote: > On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston wrote: > > On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote: > >> Is there anything I am doing wrong? Maybe I misunderstood something about > >> the way the virtual memory works in FreeBSD. > > > > I'll note that inspecting and manipulating vm_map_entry and vm_object > > structures in the bhyve code constitutes something of an abstraction > > violation, though it's reasonable to proceed this way while working on a > > prototype of the feature. That is, I think you should keep trying your > > current approach, but just be aware that you are using the copy-on-write > > mechanism in a way that the VM system isn't really expecting. > > > > Can you point out the right approach in our case? I am merely suggesting that once the required VM interactions are fully understood, the mechanism implemented for bhyve should be generalized and lifted into the VM code. It's hard to say what the "right" approach is, since I don't fully understand the proposed algorithm. It sounds like you might be attempting something like: 1. mark the mappings of to-be-migrated objects as NEEDS_COW, so that a subsequent write fault triggers creation of a shadow object 2. invalidate all physical mappings of pages in the object to be copied, so that subsequent writes trigger a fault 3. copy pages from the backing object to the destination 4. copy any pages from the shadow object to the desination 5. collapse the backing object into the shadow 6. if the shadow object exists and was non-empty before the collapse, goto 1 Is that at all accurate? I'm not familiar with the mechanisms used to implement live migration in other hypervisors.