From owner-freebsd-virtualization@freebsd.org Sat Jun 30 07:38:25 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 211441021F6C; Sat, 30 Jun 2018 07:38:25 +0000 (UTC) (envelope-from mihai.carabas@gmail.com) Received: from mail-wr0-x241.google.com (mail-wr0-x241.google.com [IPv6:2a00:1450:400c:c0c::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CE67729CE; Sat, 30 Jun 2018 07:38:24 +0000 (UTC) (envelope-from mihai.carabas@gmail.com) Received: by mail-wr0-x241.google.com with SMTP id k7-v6so7277335wrq.0; Sat, 30 Jun 2018 00:38:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=f7RuluXkpAnQRk9uDHIvnrFH3Y52S1duSgcGn8WSKvY=; b=QzAU0leOPgqojohnP1pnNmUJImprwTvSYdYoHXmxOfrC/aKjZ/afdiciT4NnbpUgrZ FC9extIPxh3QmX/m+E2lPJGXfhYZ1JV5A9y/iqByLSjIjco39UwosPuiodmhxyH90Ee2 ZzudXiNIsc0Rf00E8aGLdQJHnhB/Uzt5vTH/fPrFTjnfN+jhAULopniFq5ZrwUhFNhx6 i22xdp5hf7+B7MbNqdd6Ls78odxuAJ/5B9ZwpTYd3xbTQnA1fuOsGZfV6wFGrlqMRvqj uVjIzhFC0ZpNhtvdZhMP56uZwKpNFeZ8xufWEch1dKGdchr+9Ektkl2O56fhu6XOZxp/ q67w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=f7RuluXkpAnQRk9uDHIvnrFH3Y52S1duSgcGn8WSKvY=; b=WmEN+KTLg9sBJPzniOYXDlfbf/cy8l6fuawoRDqJjCSgBe5NZ43O7388Z+XJX6xKeT 2Hh0lBzhgXt6cclHqgeHCS3Q2hWINOIKTS9ayygL/XeRiB0Z/EZdbcZJwqvikbyYLGEY wUPY4vsjnHAV9MB6GHzfyZmCgmY/nlor2yWQF/PZJ0Y3N5dRbY2e2uF25xOH2FCe1ApE Be59iWOLw0IRA4/09aVIJMWryhLXRZvRkCYcoPBInHRNAw0jUr3bkZUTxwbX0SJ4OzcA EcHtufbNt1zygNiokXQCK+ggOlmE2SrhL/VQf5n+1pWwi5hPPqrhGmGocenYMDnIlCLg pN+w== X-Gm-Message-State: APt69E3KmrhT3b7K6JSokNSkN3Rjmc3njeztfTRz1Tx7twpcdg2+DgKT PefgJ8+ISIkc/tRq6gXK5wCZWQNi3c481diILi0= X-Google-Smtp-Source: AAOMgpfOf6ZJajqhM11CtCaDP2nUDfB1lWXOp+VU6x/jXOcrXCiDauEsLFBLR9odbGtJh/K+uBZKW9BC0hofd9QzrGI= X-Received: by 2002:a5d:4a09:: with SMTP id m9-v6mr9071047wrq.91.1530344302913; Sat, 30 Jun 2018 00:38:22 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a1c:749:0:0:0:0:0 with HTTP; Sat, 30 Jun 2018 00:38:21 -0700 (PDT) In-Reply-To: <20180629225209.GA4238@pesky.lan> References: <20180629225209.GA4238@pesky.lan> From: Mihai Carabas Date: Sat, 30 Jun 2018 10:38:21 +0300 Message-ID: Subject: Re: Inspect pages created after a vm_object is marked as copy-on-write To: Mark Johnston Cc: Elena Mihailescu , freebsd-virtualization@freebsd.org, freebsd-amd64@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 07:38:25 -0000 On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston wrote: > On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote: >> Hello, >> >> I am interested if there is a method to inspect what pages/objects were >> created after a vm_object (the vm_map_entry associated with the object) is >> marked as copy-on-write. More specifically, I'm interested only in the >> pages that were copied when a write operation was proceed on a page that >> belongs to the object marked copy-on-write. >> >> I need this for a live migration feature for bhyve in order to send the >> pages that were modified between the iterations in which I migrate the >> guest's memory(the guest's memory will be migrated in rounds - firstly, all >> memory will be sent remote, then, only the pages that were modified and so >> on). >> >> What I want to implement is the following: >> Step 1: Given a vm_object *obj, mark its associated vm_map_entry *entry as >> copy-on-write. >> Step 2: After a while (a non-deterministic amount of time), >> inspect/retrieve the pages that were created based on information existent >> in the object. >> >> What I tried until now: >> >> I implemented a function in kernel that: >> - gets the vmspace structure pointer for the current process >> - gets the vm_map structure pointer for the vmspace >> - iterates through each vm_map_entry and based on the vm_offset_start and >> vm_offset_end determines vm_map_entry that contains the object I am >> interested in. >> - for this object, it prints some debug information such as: shadow_count, >> ref_count, whether if it has a backing_object or not. >> The code written is similar with the code from here (the way in which I get >> vmspace for the current process and the way I am iterating through >> vm_map_entry and objects): >> [0] >> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L979 >> >> I have read the following documentation about FreeBSD's implementation for >> virtual memory: >> [1] https://www.freebsd.org/doc/en/books/arch-handbook/vm.html >> [2] >> https://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html >> [3] https://people.freebsd.org/~neel/bhyve/bhyve_nested_paging.pdf >> [4] http://www.cse.chalmers.se/edu/year/2011/course/EDA203/unix4.pdf >> >> As far as I could tell after reading the documentation presented above, I >> should look for the object that the object I am interested in is a shadow >> of or an object that my object is shadow for. > > Right. When a copy-on-write fault results in the creation of a new > object, the new object is said to shadow the original object, which > becomes the backing object for the shadow object. When faults in the > corresponding map entry occur, the fault handler first searches for a > page in the map entry's object, and then falls back to the backing > object if necessary. > >> To do that, I should inspect the following fields from the vm_object >> structure (among others)( >> https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_object.h#L98) : >> >> - int shadow_count; /* how many objects that this is a shadow for */ >> - struct vm_object *backing_object; /* object that I'm a shadow of */ >> >> But in all my tests, for the object I am interested in, the shadow_count is >> 0 and the backing_object is NULL. >> >> The code I use to mark the vm_map_entry for the object I am interested in >> copy-on-write is here: >> [5] >> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L949 > > MAP_ENTRY_NEEDS_COPY needs to be set in order for the copy-on-write > machinery to work the way you expect. Take a look at vm_map_lookup(): > when it sees that flag and the caller is attempting a write fault, it > creates a shadow object, updates the entry and clears > MAP_ENTRY_NEEDS_COPY, leaving MAP_ENTRY_COW set. > >> Is there anything I am doing wrong? Maybe I misunderstood something about >> the way the virtual memory works in FreeBSD. > > I'll note that inspecting and manipulating vm_map_entry and vm_object > structures in the bhyve code constitutes something of an abstraction > violation, though it's reasonable to proceed this way while working on a > prototype of the feature. That is, I think you should keep trying your > current approach, but just be aware that you are using the copy-on-write > mechanism in a way that the VM system isn't really expecting. > Can you point out the right approach in our case? Thanks, Mihai >> There is another way I could inspect what pages were created between the >> moment I mark an object (its vm_map_entry) as copy-on-write and a later >> moment? > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org"