From owner-soc-status@FreeBSD.ORG Sun Jul 6 11:59:43 2014 Return-Path: Delivered-To: soc-status@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1BC34D4 for ; Sun, 6 Jul 2014 11:59:42 +0000 (UTC) Received: from mail-qg0-x230.google.com (mail-qg0-x230.google.com [IPv6:2607:f8b0:400d:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A3BD62B54 for ; Sun, 6 Jul 2014 11:59:42 +0000 (UTC) Received: by mail-qg0-f48.google.com with SMTP id q108so2678558qgd.21 for ; Sun, 06 Jul 2014 04:59:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=vHbeLIQt9Zy4AO96ZfH0DlE+y42SrYTpEreTLV8iOI4=; b=HKyiPYSKJ0/ahQWoJMIE4NLnYPC7G0MnrwK78GrR5ish2p1TkU8KS7XRusqPmn4TMJ rCrwLAibDsDzCGQMP4u3+5q+rIUgEE7PWOsUqicupHtgzy5VVa41F4OojZ1ot44LNDmn 84Nw5pPQMZAvZd6LJgnbDnIkkLmDmOM04eBIrDnk/cxv6zQpWp765Q46pCvR2wnzKfC8 5ND4ailLKnGc1ef4eTVyHU61OzipXNAYx53wCG7+JpFwC5DZp6lhqmIAzGLjJWHV1oBt 7xg10h50yfLrIZEdkno+GZeKAWh7z7IqLgJMoKRYlGxo6Jtp2D55cBafL3LzBmizqmmO qpDQ== MIME-Version: 1.0 X-Received: by 10.224.45.202 with SMTP id g10mr5731558qaf.30.1404647981799; Sun, 06 Jul 2014 04:59:41 -0700 (PDT) Received: by 10.140.84.85 with HTTP; Sun, 6 Jul 2014 04:59:41 -0700 (PDT) In-Reply-To: References: Date: Sun, 6 Jul 2014 14:59:41 +0300 Message-ID: Subject: Re: [GSOC] bhyve instruction caching From: Mihai Carabas To: soc-status@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jul 2014 11:59:43 -0000 On Sun, Jun 15, 2014 at 8:58 PM, Mihai Carabas wrote: > On Sun, Jun 8, 2014 at 12:38 AM, Mihai Carabas wrote: >>> >>> These days I've started a discussion with Neel about some >>> microbenchmarking mechanisms. I will come with some more details next >>> week. >> I've built a microbenchmarking kernel module which is accessing the >> lapic->id for 1000000 and than I calculate the average of an access >> (each access needs to be emulated by the hypervisor). >> >> I've also implemented the instuction caching mechanism. At each emulation: >> - I check to see if I have that particular instruction cached >> - if not I will cache it in a particula structure named "struct vie_cached" [1] >> - if it's cached I just use that instruction >> >> Right now I am working on write-protecting the pages where the >> instruction reside. I will come with some more details/results when I >> finished this part too (there are some SMP issues I'm still debating >> with Neel). > > I added the write protection for the pages where instruction resides > (I mark them with PROT_READ|PROT_EXECUTE using vm_map_protect). When a > page fault is raised on that page I delete the instruction from the > cache. I've tested the instruction caching feature and it works ok (no > bugs reveal). The results on the microbenchmarking are looking OK, but > they aren't conclusive because I'm not write protecting the pages of > the pagetable (which will eat more time). Right now for 1000000 > accesses to the lapic-id we have an average of 6200 ticks per VM_EXIT > for an instruction emulation. Without instruction caching the average > is at about 10000ticks. Like I've said before, this results aren't > very conclusive until I introduce all protections needed. I will come > in the next weeks with more results when I have the caching feature > fully implemented. > > I've also introduce a new systl to enable/disable instruction caching > "hw.vmm.instruction_cache". In the last two weeks I developed the mechanism of write-protecting the VM pagetable: every page of the pagetable that was pointing to the memory page where the instruction resides was made read-only. I had some trouble with the locking mechanism when having a virtual machine with more than 1 vCPU (one vCPU was write protecting and immediatly another VCPU was remove the protection resulting in a corruption of the write-protection and cache logic). Neel helped me solve this issue with some best effort checks and are working OK. Also I migrated to sx_* locks due to the fact the rm_* locks didn' let me sleep in them (at some point when raising the protection in the VM subsystem a sleep may be encountered). At the end of the week I will come with some new results with the microbenchmarking. Also I'm looking for new real-world benchmarks to see where we can benefict a lot from the instruction caching feature. Thanks, Mihai