Date: Thu, 7 Mar 2019 14:41:39 -0500 From: Mark Johnston <markj@freebsd.org> To: Elena Mihailescu <elenamihailescu22@gmail.com> Cc: John Baldwin <jhb@freebsd.org>, Matthew Grooms <mgrooms@shrew.net>, Mihai Carabas <mihai.carabas@gmail.com>, alc@rice.edu, Tycho Nightingale <tychon@freebsd.org>, araujo@freebsd.org, Mihai Carabas <mihai@freebsd.org>, Darius Mihai <dariusmihaim@gmail.com>, Sergiu Weisz <sergiu121@gmail.com>, Michael Dexter <editor@callfortesting.org>, alc@freebsd.org, Patrick Mooney <patrick.mooney@joyent.com>, freebsd-virtualization@freebsd.org Subject: Re: bhyve guest's memory representation & live migration using COW Message-ID: <20190307194139.GA56002@raichu> In-Reply-To: <CAGOCPLgcQo6990Trj=oUzwG1jf-KWTvCg9dOrmCrp2cZb_h9sw@mail.gmail.com> References: <20181024181331.GH45118@raichu> <d31fa567-2fd0-cbdf-680d-55b2f87f2bab@shrew.net> <CAGOCPLhNKA2ormuVf5x6EpBykFgyEOsGkugy9BfoONHr=vcixw@mail.gmail.com> <20181026215929.GC33894@raichu> <CAGOCPLgni2coXSZk9DE6gd=_dEzyVGKki2ndXeQy1B4DAQh1WA@mail.gmail.com> <a7caad31-cee8-3fba-4668-7f6a37a69fa6@FreeBSD.org> <245f33cf-1c71-1c15-80e5-0686e992df9a@FreeBSD.org> <CAGOCPLifRxKJBHdT0VAv_ZLCtBF4tTkFb0Gd6LQ9rPuraitaxg@mail.gmail.com> <CAGOCPLjQ_saU-ddLrr2=KCbxScufY=E5NF9eXi=wqro-5AcGhQ@mail.gmail.com> <CAGOCPLgcQo6990Trj=oUzwG1jf-KWTvCg9dOrmCrp2cZb_h9sw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Mar 04, 2019 at 10:52:44AM +0200, Elena Mihailescu wrote: > I'm writing this e-mail to keep you in the loop with the live > migration progress and to ask for advice regarding the process of > cleaning the dirty bit. At Rod Grimes's suggestion, I added the > virtualization list at CC. > > I had some issues with the vmm_dirty bit (a bit from oflags) and > initially I thought that there some issues related to the fact that in > some cases oflags is directly updated (oflags = <smth>) and with the > fact that the dirty bit is set otherwise that using vm_page_dirty(), > but it seems that the issues I had were from other sources and now, > after cleaning the code, and refactored a bit, it seems to work fine. > Some of the code snippets that I thought that could affect the > migration are at [1]. > > However, if I dump the guest memory after the live migration (the > guest is not yet started after migration) and compare it with the > source guest's ctx->baseaddr, there are some differences and I don't > know from where they may come from. > > One of the things I must implement is related to the dirty bit > mechanism. For now, I do not clear the dirty bit after I migrate a > page. Note that there is code that only calls vm_page_dirty(m) if m->dirty != VM_PAGE_BITS_ALL. One example is in vm_page_advise(). So if the page is dirty from the kernel's point of view and you clear the VMM dirty bit, subsequent modifications may not be propagated to the vm_page state. > This is not a wrong implementation because in every round, I > migrate all the pages that have been modified ever (since the guest > was started), but it is not an optimal implementation. To clear the > dirty bit, I've tried different mechanism such as: > - pmap_remove_write - kernel panic: pde entry not found for a wired mapped page > - vm_object_page_clean > - vm_map_sync -> kernel panic > - pmap_protect - remove write permission, copy page and add write > permission -> kernel panic; pmap_demote_pde: page table page for a > wired mapping is missing > > Any idea about a way of forcing a page to be cleared (so the physical > dirty bit to be cleared)? I'm not sure exactly what you mean, but pmap_clear_modify(m) will clear the modification bit for mappings of m. It seems like you want something roughly like: vm_page_xbusy(m); if (pmap_is_modified(m)) m->dirty = m->vmm_dirty = VM_PAGE_BITS_ALL; pmap_clear_modify(m); vm_page_xunbusy(m); Or do we need to restrict the operation to a specific mapping of m?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190307194139.GA56002>