From owner-soc-status@freebsd.org Thu Aug 18 19:20:54 2016 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E956BBEC21 for ; Thu, 18 Aug 2016 19:20:54 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 26ADD12F7; Thu, 18 Aug 2016 19:20:54 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: by mail-io0-x235.google.com with SMTP id q83so28402042iod.1; Thu, 18 Aug 2016 12:20:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=UY0ci7mNpYPJTqWJRy1ic/1AEHtM6+uVCrv/3aLGz5s=; b=YFSV494kq5tfZA5+qhZ4YPXuAJ6WqD3+4RK9M7QQiSjiX4GPbqnxI5lS0+dZohwn3t sfJYKs0fnDd2KpQwN70wTY1QU/VmIzlw5ECtAPM3p36UPoHVf5qCGZvhgm6hH5tHsnEs aVGdhfibHJ2U1mkGEGXUD9dKV/BNwy9S74+PPn3za44TmUCzY1gvN9/chA7sOQ5sJ3Rk II8B+W15pfZpDDLNFZu6lJVcYVry7+rxhTTbI3uZCiVUdSPQXSppSEhoRxWEyAExHh06 HPaQEWNSB+nj6sETFnZDkMjvwdrCDNYfoocisMGeYe65HImCHpulQKxwbAsXHRDQ1Apk Sjaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=UY0ci7mNpYPJTqWJRy1ic/1AEHtM6+uVCrv/3aLGz5s=; b=YTV70QG5dT8Tcf5UPx96My6VnCDguluqHieM9eBCY8LaEsWfg+Rqokmsg4IL/b6SyJ pKPbHwYkWDGh+Sv5NnE+Iq8tSC+TVwtpQlGJ1t6ZR36/R/Xv7cH1xGdPJf70RGUg9h5h 9Si1EfPcqf8Hz7oobMmUwlSdZt33HUfTrRnylHC1zoXLY49y4q4qXYcO6XIY0HkDhbuA uz9EaQUn8y6XdwG3I+vwioh6ACJMFphtR+yWPVudKykjtmv1KlxamR2xFxyzQbvM7y1a uSNjYqV72o/ETE6dhDnjCCUTGsk3t+PnPvIMa5HJd76ZFOyyX31pIzeiQ+kaWjvI5n2N 7Usw== X-Gm-Message-State: AEkoouukRDOo8vr1jSacIHq5yRUiqsn5Cc7TWkUtevjO1SHDkh5zNBxZCC0Qms7ly7uMQRXdd+Jy35IrfM+usA== X-Received: by 10.107.32.206 with SMTP id g197mr5199725iog.155.1471548053366; Thu, 18 Aug 2016 12:20:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.20.85 with HTTP; Thu, 18 Aug 2016 12:20:32 -0700 (PDT) In-Reply-To: References: From: Akshay Jaggi Date: Fri, 19 Aug 2016 00:50:32 +0530 Message-ID: Subject: Re: Grant Table Userspace Device - Status Update To: soc-status@freebsd.org Cc: Pedro Giffuni , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2016 19:20:54 -0000 Carrying over discussion from IRC. 20:11 royger: ghost_rider: hello! I've been doing some testing with the > device today, and it seems there's a memory leak somewhere, after shutting > down all my domains I still see 1KB of memory used by the device, which > AFAICT is not expected (you can check with `vmstat -m |grep gntdev`) > Nope. That's not a leak. I ran `vmstat -m | grep gntdev` just after booting up Dom0, without any of the DomU's running, and I still saw 1KB of memory being used by the device. root@freebsd:~ # vmstat -m | grep gntdev gntdev 2 1K - 2 64 That is, 2 requests have been made, out of which both are currently active, without any DomU's active. After this I fired up a DomU with qdisk backends, and vmstat returned: root@freebsd:~/xen_test # vmstat -m | grep gntdev gntdev 2129 134K - 2137 32,64,128 Well in line with expectations. Now, powering off the DomU and running vmstat again, we get: root@freebsd:~/xen_test # vmstat -m | grep gntdev gntdev 2 1K - 2845 32,64,128 The initial 2 requests are still active, and this has nothing to do with the DomU's. The first malloc() that happens in the device is in the device open function at [1]. That means that someone has the device open. `fstat` confirmed my suspicions. `fstat` with DomU active: root@freebsd:~/xen_test # fstat | grep xen/gntdev root qemu-system-i386 1266 29 /dev 62 crw------- xen/gntdev rw root qemu-system-i386 1266 32 /dev 62 crw------- xen/gntdev rw root qemu-system-i386 1266 34 /dev 62 crw------- xen/gntdev rw root xenconsoled 751 6 /dev 62 crw------- xen/gntdev rw root xenstored 746 11 /dev 62 crw------- xen/gntdev rw `fstat` with DomU powered off: root@freebsd:~ # fstat | grep xen/gntdev root xenconsoled 751 6 /dev 62 crw------- xen/gntdev rw root xenstored 746 11 /dev 62 crw------- xen/gntdev rw So yep! It's no leak. Just that xenconsoled and xenstored keep the gntdev device open. I guess this would be expected behaviour. Let me know if it is not. 20:14 royger: ghost_rider: and I've also seen a "Can't find requested > grant-map." after attaching 4 Qdisk to a domain and done heavy IO to to > them. > 20:16 royger: although this last one I haven't been able to reproduce > That's pretty strange. I have never noticed this in any of my manual or stress tests. At this point I would also like to mention, that the xen-gnttab code is kind of buggy (putting it mildly, no offence). Like I pointed out in the xen-devel patch thread, there is a place in code where "-1" is being used to specify there is no CLEAR_BYTE notify. But this is not being checked for inside the function, which would have caused a clear-byte notification on a different page, causing data corruption. The only reason this bug is not doing so, is because of another bug, where this -1 is being passed on to an unsigned int32, which would keep it out of bounds for most requests. I don't think this has to do anything with our device. If we lost some unmap request (which is where this message is generated) we would have surely leaked the memory for the gmap structure associated with that request (because, 1. ref-counting, 2. transferred to global clean list only on an unmap request), and that would have been visible in `vmstat`. Let me know if this repeats. > 20:40 royger: and I'm not sure if you tested it, but if you attach a > ramdisk to a VM (one created with `mdconfig -t malloc ...`) and try to run > newfs against it, it doesn't work, a bunch of read errors appear on both > the DomU console and Qemu log. Although it works with a plain file, so I > guess this is probably some bad interation between Qemu and FreeBSD block > devices... Mhm. Sounds like that. I'll try it out on my setup and post the results. Regards, Akshay [1] https://github.com/akshay1994/freebsd/blob/grant_table/sys/ dev/xen/gntdev/gntdev.c#L896 On 18 August 2016 at 23:21, Akshay Jaggi wrote: > Hello Everyone, > > Hope everyone is doing great. > > This is the last week of GSoC-2016, and the grant table userspace device > is complete. We are actively working to get the code reviewed and pushed. > > Here is the list of patches in the pipeline: > > - https://reviews.freebsd.org/D7429 > - https://reviews.freebsd.org/D7293 > - https://lists.xenproject.org/archives/html/xen-devel/2016- > 08/msg02125.html > > I'll be using this thread to discuss the overall progress of code-review > for all the three patches, and I hope that would be fine. > > Looking forward to getting this stuff to mainline. :) > > Regards, > Akshay > > On 20 July 2016 at 20:33, Akshay Jaggi wrote: > >> Hello Everyone, >> >> >> I hope everyone is doing great. >> >> >> First things first. I want to apologise for not updating my blogpost >> after June 15. Some personal stuff happened on June 17, and since then all >> my work routines are completely disturbed. Soc-admins know about this, and >> I hope the community can pardon my irregularity this once. >> >> >> Now, time for some status update on the project. We have a working Grant >> Table User-space Device (with some caveats, of course). Yay!! >> >> Check out the code on the pull request: https://github.com/fr >> eebsd/freebsd/compare/master...akshay1994:grant_table?expand=1 >> >> Test the device by compiling a custom kernel; from my working branch: >> https://github.com/akshay1994/freebsd/tree/grant_table >> >> >> Caveats and work yet to be done: >> >> - A kernel panic happens if you terminate the program without >> un-mmapping all the mmaps. >> - The interface for a grant map is not completely in sync with the >> Linux interface yet. Notably, we still map and subsequently mmap partially >> working grants. This is a bug since the pager will fail on a page fault for >> the non-working-grant. >> - I'm yet to add event channel code for the notify ioctls. >> - I'm yet to get in touch with the vm-guys. I spent a lot of time >> understanding the vm structure, but I surely do not have a complete >> picture. I'm using a MGTDEVICE cdev pager to handle managed, fictitious >> pages from mapped grants, and a custom PHYS vm object to manage wired >> physical memory for allocated grants. I have to >> - Confirm if the approach I am using is the best one. >> - The interface differs compared to Linux with regards to multiple >> mmaps, which I need a little help with. >> >> >> Looking forward to some community testing of the working parts. >> Comments are welcome and highly appreciated. :) >> >> >> Regards, >> Akshay Jaggi >> > >