From owner-freebsd-virtualization@freebsd.org Thu Mar 7 19:48:24 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2674D152356E for ; Thu, 7 Mar 2019 19:48:24 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-it1-x136.google.com (mail-it1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC3016FFEF; Thu, 7 Mar 2019 19:48:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-it1-x136.google.com with SMTP id 188so17768455itb.0; Thu, 07 Mar 2019 11:48:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=jQNZyoUF66ID+S/N7W6HqXHmyKdyONlpJGXNlfqjeTQ=; b=S4cGbFGP2NzsBX3kXc7bNcrqf4L5nTabaDOoXqNDQua8Pl8fUtNdaOqrzuEeySdWe2 u68czIf8yEuK81PJ8AcN0XcnjUo21W0fJ6ZcMHlLZwGk7AY+lbglJVagA/jXDKNK1haS oL74q8+L8g5UNkCon1KvLsQc6TT9G4IheytJldmCIAdHxVcItsCXnXOIMWCmVvdaL0EF WZcRQGFyKr0hBiRVJFL4Ooduka3pK8Z3B0qi6gyl8hUPhLLie6VhhvL15xnLOnzt4D6N 453Smj/cRac9n1f3sTODEf7N7JS0IbHBE1iFvkQNY4saqRNyMqGDrelXvoFF/SGKGsFf pE3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=jQNZyoUF66ID+S/N7W6HqXHmyKdyONlpJGXNlfqjeTQ=; b=C4Blua5tIzMK47ipRKDafJEG4/b+9NRYtBzJPzS2huBWwXRvRzlHFub2qiOk7QKs/k IQdgoDHiVpfI1vBmIcykcL3cvHKVNabRIJNyNLhDQc7bocBAaTSH4KF0nuZINubzCTQW VaScGKicJitDr/YUBVpqd4ojGhvfWkYhCO2TaXrK2+3sWaZtPzSRgsi7BUjWLV/RKkgT 1ogHA18BXBaUqhcCau42jWvzlz8VDysHvxcms5DVXAathEd2r3b2+QWSVLT+AtP+vEPR +OsJdf0PHzP5jqNNaBvzIJf7g6aEty7+jE/r6mbaOMxqoEgQN0VqYQpfrhhvewNtdgFt y/+A== X-Gm-Message-State: APjAAAXrnkZvE79kPJpRr0kiWhPpbgzdmr9EUxGlTBqG0DpFOEOHRL5x +2MaCsLQ96pkq5yRFrQNGx0= X-Google-Smtp-Source: APXvYqwKw/RzyHBAvMniRXUGOVr+SBBHcsn3HjbNbrFVH06HN4jsL/13SmP0o8HejIHucHmuz+y4DQ== X-Received: by 2002:a24:1694:: with SMTP id a142mr6244344ita.22.1551988102050; Thu, 07 Mar 2019 11:48:22 -0800 (PST) Received: from raichu (toroon0560w-lp140-01-69-159-36-102.dsl.bell.ca. [69.159.36.102]) by smtp.gmail.com with ESMTPSA id t68sm2877390ita.4.2019.03.07.11.48.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Mar 2019 11:48:20 -0800 (PST) Sender: Mark Johnston Date: Thu, 7 Mar 2019 14:41:39 -0500 From: Mark Johnston To: Elena Mihailescu Cc: John Baldwin , Matthew Grooms , Mihai Carabas , alc@rice.edu, Tycho Nightingale , araujo@freebsd.org, Mihai Carabas , Darius Mihai , Sergiu Weisz , Michael Dexter , alc@freebsd.org, Patrick Mooney , freebsd-virtualization@freebsd.org Subject: Re: bhyve guest's memory representation & live migration using COW Message-ID: <20190307194139.GA56002@raichu> References: <20181024181331.GH45118@raichu> <20181026215929.GC33894@raichu> <245f33cf-1c71-1c15-80e5-0686e992df9a@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.2 (2019-01-07) X-Rspamd-Queue-Id: DC3016FFEF X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=S4cGbFGP; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::136 as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-5.03 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.90)[-0.897,0]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; RCVD_IN_DNSWL_NONE(0.00)[6.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.42)[ip: (-7.28), ipnet: 2607:f8b0::/32(-2.72), asn: 15169(-2.05), country: US(-0.07)]; MID_RHS_NOT_FQDN(0.50)[] X-Mailman-Approved-At: Thu, 07 Mar 2019 23:22:30 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2019 19:48:24 -0000 On Mon, Mar 04, 2019 at 10:52:44AM +0200, Elena Mihailescu wrote: > I'm writing this e-mail to keep you in the loop with the live > migration progress and to ask for advice regarding the process of > cleaning the dirty bit. At Rod Grimes's suggestion, I added the > virtualization list at CC. > > I had some issues with the vmm_dirty bit (a bit from oflags) and > initially I thought that there some issues related to the fact that in > some cases oflags is directly updated (oflags = ) and with the > fact that the dirty bit is set otherwise that using vm_page_dirty(), > but it seems that the issues I had were from other sources and now, > after cleaning the code, and refactored a bit, it seems to work fine. > Some of the code snippets that I thought that could affect the > migration are at [1]. > > However, if I dump the guest memory after the live migration (the > guest is not yet started after migration) and compare it with the > source guest's ctx->baseaddr, there are some differences and I don't > know from where they may come from. > > One of the things I must implement is related to the dirty bit > mechanism. For now, I do not clear the dirty bit after I migrate a > page. Note that there is code that only calls vm_page_dirty(m) if m->dirty != VM_PAGE_BITS_ALL. One example is in vm_page_advise(). So if the page is dirty from the kernel's point of view and you clear the VMM dirty bit, subsequent modifications may not be propagated to the vm_page state. > This is not a wrong implementation because in every round, I > migrate all the pages that have been modified ever (since the guest > was started), but it is not an optimal implementation. To clear the > dirty bit, I've tried different mechanism such as: > - pmap_remove_write - kernel panic: pde entry not found for a wired mapped page > - vm_object_page_clean > - vm_map_sync -> kernel panic > - pmap_protect - remove write permission, copy page and add write > permission -> kernel panic; pmap_demote_pde: page table page for a > wired mapping is missing > > Any idea about a way of forcing a page to be cleared (so the physical > dirty bit to be cleared)? I'm not sure exactly what you mean, but pmap_clear_modify(m) will clear the modification bit for mappings of m. It seems like you want something roughly like: vm_page_xbusy(m); if (pmap_is_modified(m)) m->dirty = m->vmm_dirty = VM_PAGE_BITS_ALL; pmap_clear_modify(m); vm_page_xunbusy(m); Or do we need to restrict the operation to a specific mapping of m?