From owner-soc-status@freebsd.org Mon Aug 15 13:43:33 2016 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BD4EBBAE27 for ; Mon, 15 Aug 2016 13:43:33 +0000 (UTC) (envelope-from starak.adam@gmail.com) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 37C4F1F67 for ; Mon, 15 Aug 2016 13:43:33 +0000 (UTC) (envelope-from starak.adam@gmail.com) Received: by mail-wm0-x22f.google.com with SMTP id q128so90483569wma.1 for ; Mon, 15 Aug 2016 06:43:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=vW0NkmOMlxp7SMFxPLco8dHVK+s41dkA79KTRacpB6k=; b=CsGqIiXSiqG5eGz+G10IkaFxghMrFDikowr98Y8y2y1AY0QkS5dXcqrOycw7fUIirG 0zufYlHRzflCdQ8k40iblxYCOEaSUXo/jt5cCLUrS6ZskX7C9UHAWDEmklaFRSNT3Yfk xgdmANIRfWmO6cC3FBTLIC9NTnRT/dEKNH1NWVXbj2f1F+/pNf4GFPHNY5hlE3se/tFK pxlQUQIirqcWx4K63P3jq/xWxxwpsrJ7ZcqAVttCX1X773ckKs3K7rDvnjWR6goLBj4a 3xy4lrb5Kfx10sHh80tQghLRtOaIaKX80qJjBbJ0XZGqoDseZ0dsdV57a4y0rVLrZbDu 1vXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=vW0NkmOMlxp7SMFxPLco8dHVK+s41dkA79KTRacpB6k=; b=V3S8WiSIsQMa6E2KCpWhDoUwqMS37tLvIPnBNKDXDJk/FPSIKFnzymeiqy5cr+WEC4 +eM0XJhT2XUDVP5KrGgs8/BFJrSnVcmIxs4P9+ouhDLqB4TrCOIJnCImSVm28KL3wKFZ PKcIgIrPLvQPArIKNWL9IC4/cSyviJvhwljs3073bXMUeuOoa8pitoNzkiRcLZUDKBE4 Nb2uyHc/C+iA7TVRsGc9vp7WIDXDnIIQaTVR54aa0RkoP2e+QuUKPaLcT3z0YjZjH3nl D+RDFpkHUWkBJV3NBAO4hvuj/weAWzpzyxdZwqNvLuB5kRfe/GWnevqEGYB2dn4Q+m7y 4noQ== X-Gm-Message-State: AEkoout4tZo3VEL/tYlOcslv44E/DulWpFgfE1KGlrBTdqqvoZYPJDoTTljv/c09e6Q+jKKzGGO3GnRhsa9k3w== X-Received: by 10.25.211.15 with SMTP id k15mr2917963lfg.178.1471268610497; Mon, 15 Aug 2016 06:43:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.29.193 with HTTP; Mon, 15 Aug 2016 06:43:30 -0700 (PDT) From: Adam Starak Date: Mon, 15 Aug 2016 15:43:30 +0200 Message-ID: Subject: libnv improvements 12 To: soc-status@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Aug 2016 13:43:33 -0000 Hi, Sorry for the delay, but I have some internet problems over my place. Last week, I made some changes to my manpages, finished work witth sysctl, made a lecture about my work during GSOC for my mentor's company. Also, I started to work on one of my previous milestone. I'm trying to port libnv to other OS. I chose Ubuntu and NetBSD. Best regards. Adam Starak From owner-soc-status@freebsd.org Mon Aug 15 14:18:05 2016 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F3CBBBB714 for ; Mon, 15 Aug 2016 14:18:05 +0000 (UTC) (envelope-from y.iozzelli@gmail.com) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 380A2185C for ; Mon, 15 Aug 2016 14:18:05 +0000 (UTC) (envelope-from y.iozzelli@gmail.com) Received: by mail-wm0-x22b.google.com with SMTP id f65so91919900wmi.0 for ; Mon, 15 Aug 2016 07:18:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=xZbkGe6CALlDwVlr0XR6KD7U8SKAQ+fvBMETc333fXI=; b=FlgbL+34THyw36ngM28vBJjNREWGRw0tjpx0CCg5/TXCkE3pmrom6VR/SRhZqb6xnU qegyigcUR+mFdMnLrwXF42BEu+mXeYdopZFxvPaJspUItM2eRPTtEemDryg9HTYrZcg2 iSaZB/07lBDfjtjqN+JJH/jsKqcD1FJJpiMFveiAlWqoX2A50hV5N01G9lItGXZ72iX5 SeUhfothCa5BoqmKEFpDfaJkgYx5fH3ilQLzXGLjJLqjbj3ywYSknAigD8NIJxf5eFHV NANBmudX+08XZBZjdxKsPuN7sP9FtRQ2aNg9xjxDbCXJqyTDiQuE5bVb9wIZMSUvKem8 pGFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=xZbkGe6CALlDwVlr0XR6KD7U8SKAQ+fvBMETc333fXI=; b=Gjw8j/FmGwY3pIvkJfMn8abm2QNZBkSVPfibJjWtpfZx0Wa5+msnqswIB1F0z7LC1h EbyFt2j8qutTCnFTf3NMhaaRwBEIN31xHW8gHhNIGBGYLx5F2As1UvlNFefnJ2nDFH4K +TrYVvATng+A/jWprpHWSAaK614hLGwvXiq7w+AcEXosSPNxAchL6uvcnCmggPdMrFgS RlRXa2rgPbbMAGvDhklwZLpXkOWcFVst8cl502hcer5WQhvZs3up0tzvUME9SAFHWkSg sUW89IYDeDUSElguhDSbgyXvgcHez4pYYxK69Q3F00Xy6ecRDmccu2WoAh0Ddy7OJ0cH jUyA== X-Gm-Message-State: AEkoousypd4wVjBFCIWL9XXuNz2JUdKL99S79w+bEGTuT6MaVPRsJ68VH1DXoZeS8WfopUT3lrrAWrnolRe+6A== X-Received: by 10.46.32.203 with SMTP id g72mr4826665lji.30.1471270683260; Mon, 15 Aug 2016 07:18:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.29.16 with HTTP; Mon, 15 Aug 2016 07:18:02 -0700 (PDT) From: Yuri Iozzelli Date: Mon, 15 Aug 2016 16:18:02 +0200 Message-ID: Subject: High Performance P4 Software Switch - Week 12 To: soc-status@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Aug 2016 14:18:05 -0000 Hi Project summary ======================== In this project I would like to implement a modified version of the reference p4 (http://p4.org) switch on FreeBSD, which uses netmap for faster packet I/O: the goal would be reaching the 1 Mpps order-of-magnitude speed (current reference implementation is limited to 150 Kpps for a simple l2 switch with 2 hosts). Week 12 ========================= Since we are near the end, I listed all the features added and decided what to keep. I sent some fixes upstream and divided the improvements in 3 independent branches. I am also working on the port of the project to freebsd in two flavors: one with the minimal set of patches to make the original code compile on freebsd, and the other one with all the features developed during GSoC. I am still working on this. Links: Wiki page: https://wiki.freebsd.org/SummerOfCode2016/HighPerformanceP4S oftwareSwitch#preview Code (gsoc-* branches): https://github.com/zarghul/behavioral-model Regards, Yuri Iozzelli From owner-soc-status@freebsd.org Mon Aug 15 22:28:18 2016 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F7BBBB93AF for ; Mon, 15 Aug 2016 22:28:18 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D946E11C8; Mon, 15 Aug 2016 22:28:17 +0000 (UTC) (envelope-from mpp302@gmail.com) Received: by mail-wm0-f68.google.com with SMTP id q128so13340013wma.1; Mon, 15 Aug 2016 15:28:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-transfer-encoding:subject:date :message-id:cc:to:mime-version; bh=GZjoOwKmS6c+DmXh9y627edO7LHeUbLsmgtsYUSMP28=; b=DNp5FL39DFGmqyep3vKlfcwVJJGysilOVX8gOOMZI8Lr8Qz+aWWdEdPEiGZo2p/A5+ 3ZSzSAGSA6R7Nc1i2HL+HDY3OmjCrnhhN9sdBKlWjlaLCWUpZUXv/ueZC7XkUNywekSY S7lY4SjsqgdEZFosMIBW+vTyTVjnyAo+kyac4uWLc8LT9J6LGz3CobLu9ZYYRWDxzH5k EX/wak6olLh3w0TXNkFfhkz1ZFpsw+llCkczSGMyIGeVWQkx9MYjwDAvwBMz3EU9vwEf 4wOEbc/9ac1pxcr9kYVa6t9OlHwF+SrhsDuRilmP5dmcldRpaE1s+5N571LhhwaJAdnv RcHA== X-Gm-Message-State: AEkoousQwfA3I4x9VBzf8HqrwTezL2NZhX80ErFNBVXX82zTr4a3cBaBkJvxFmmOEKXpoQ== X-Received: by 10.194.136.196 with SMTP id qc4mr38296002wjb.136.1471300089874; Mon, 15 Aug 2016 15:28:09 -0700 (PDT) Received: from maka.fritz.box (dslb-178-008-181-169.178.008.pools.vodafone-ip.de. [178.8.181.169]) by smtp.gmail.com with ESMTPSA id q4sm23567959wjk.24.2016.08.15.15.28.08 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 15 Aug 2016 15:28:08 -0700 (PDT) From: Mateusz Piotrowski <0mp@FreeBSD.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Week 12 / Non-BSM to BSM Conversion Tools Date: Tue, 16 Aug 2016 00:28:07 +0200 Message-Id: <6734CD7D-FD67-4724-8B30-42453F871809@FreeBSD.org> To: soc-status@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Aug 2016 22:28:18 -0000 Hello, During the last week I finished implementing the minimal conversion=20 of the SYSCALL and EXECVE type Linux Audit records.=20 Moreover, I added a very basic version of au_to_attr(3) to the userland,=20= so that it is possible to create an attribute token outside of the = kernel. My current implementation might not be satisfying in the long run so=20 I emailed the FreeBSD and TrustedBSD mailing lists for advice and = opinion. During the last week I'll try to extend auditdistd with the ability to = receive Linux Audit log trails over the network. The lastest code is available here: = https://github.com/0mp/freebsd/pull/9 Cheers! -Mateusz [wiki]: = https://wiki.freebsd.org/SummerOfCode2016/NonBSMtoBSMConversionTools From owner-soc-status@freebsd.org Thu Aug 18 17:51:24 2016 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E655BBE8F7 for ; Thu, 18 Aug 2016 17:51:24 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: from mail-io0-x22f.google.com (mail-io0-x22f.google.com [IPv6:2607:f8b0:4001:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE40016F4; Thu, 18 Aug 2016 17:51:23 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: by mail-io0-x22f.google.com with SMTP id m101so25981504ioi.2; Thu, 18 Aug 2016 10:51:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=YK2JVQ6q4gbMP/X48Mimp/8nv7bopLd6gbJ+RzB36W0=; b=KthNEH1c887PY9uBqkOFQXcr7UEpJlb162U4SVgvK6p1ACZICL9XGVu5clhZHG6NUQ kiS7S7oyg4ddQRMpdK4O+ndXQEMesOlXKnMwlDZ81xupfK3EJvt2hUJOVqPgGb9HAgPu aBSQ76x7pRcF14IlA9Mx9iHUM51sOmo++AykADQnaxYnwTqaow3RT1QYVzlkA6CmQpWR UXAKX/nqn0oEfydzQImb6F+ZxoSuRFAoRHFy2j2H1Y8e4uPBOCOPP6BRBSt/w6x9i0uK WzywCdRmc+lNjx66IQVkXpwpBoKUacEYW+j3bLpcuwlvwmO5SaUUF5PN9wJnOneMut50 +WPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=YK2JVQ6q4gbMP/X48Mimp/8nv7bopLd6gbJ+RzB36W0=; b=MHR1b439oIoRAs0/3imFXCvfhVjJ6/kHBRn5a2AcIxbSrZZkcumE5xPzOadAWfVVo2 HpwD4aEkvolgafe4WwieA1tb6Vz9pp+/JMhJmrOBx8tZw8E7eXOsw2bybycnd086En8V 9GZI36JKs/ixSVGRaVSo5c4UGcveKeb09E8lD+LYZPT2PSD4nUD547Lhp2qn3w+77uFw 26CXjecFcuLFOH+WqF4+OFUgDiZbF923qIcW333IsKML+8lo+d1a9yVhm9OvULPZemBp 2jaJjG+Bjr772+ms2TIn2i+sINvdGs/7tFtHvg9/OUL4cQOmCFGyOkI630Mq4+5NMGs8 O/Lw== X-Gm-Message-State: AEkoousa1BnTeNL+3QBEVTacV+O7QJZ/xtglxlomALv7Gxl/qsU7+VjPqxRWKgDB2d5iJ98hMgZmhFbjH1H5rQ== X-Received: by 10.107.50.19 with SMTP id y19mr4521211ioy.174.1471542682780; Thu, 18 Aug 2016 10:51:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.20.85 with HTTP; Thu, 18 Aug 2016 10:51:02 -0700 (PDT) In-Reply-To: References: From: Akshay Jaggi Date: Thu, 18 Aug 2016 23:21:02 +0530 Message-ID: Subject: Re: Grant Table Userspace Device - Status Update To: soc-status@freebsd.org Cc: Pedro Giffuni , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2016 17:51:24 -0000 Hello Everyone, Hope everyone is doing great. This is the last week of GSoC-2016, and the grant table userspace device is complete. We are actively working to get the code reviewed and pushed. Here is the list of patches in the pipeline: - https://reviews.freebsd.org/D7429 - https://reviews.freebsd.org/D7293 - https://lists.xenproject.org/archives/html/xen-devel/2016-08/msg02125.html I'll be using this thread to discuss the overall progress of code-review for all the three patches, and I hope that would be fine. Looking forward to getting this stuff to mainline. :) Regards, Akshay On 20 July 2016 at 20:33, Akshay Jaggi wrote: > Hello Everyone, > > > I hope everyone is doing great. > > > First things first. I want to apologise for not updating my blogpost after > June 15. Some personal stuff happened on June 17, and since then all my > work routines are completely disturbed. Soc-admins know about this, and I > hope the community can pardon my irregularity this once. > > > Now, time for some status update on the project. We have a working Grant > Table User-space Device (with some caveats, of course). Yay!! > > Check out the code on the pull request: https://github.com/ > freebsd/freebsd/compare/master...akshay1994:grant_table?expand=1 > > Test the device by compiling a custom kernel; from my working branch: > https://github.com/akshay1994/freebsd/tree/grant_table > > > Caveats and work yet to be done: > > - A kernel panic happens if you terminate the program without > un-mmapping all the mmaps. > - The interface for a grant map is not completely in sync with the > Linux interface yet. Notably, we still map and subsequently mmap partially > working grants. This is a bug since the pager will fail on a page fault for > the non-working-grant. > - I'm yet to add event channel code for the notify ioctls. > - I'm yet to get in touch with the vm-guys. I spent a lot of time > understanding the vm structure, but I surely do not have a complete > picture. I'm using a MGTDEVICE cdev pager to handle managed, fictitious > pages from mapped grants, and a custom PHYS vm object to manage wired > physical memory for allocated grants. I have to > - Confirm if the approach I am using is the best one. > - The interface differs compared to Linux with regards to multiple > mmaps, which I need a little help with. > > > Looking forward to some community testing of the working parts. > Comments are welcome and highly appreciated. :) > > > Regards, > Akshay Jaggi > From owner-soc-status@freebsd.org Thu Aug 18 19:20:54 2016 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E956BBEC21 for ; Thu, 18 Aug 2016 19:20:54 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 26ADD12F7; Thu, 18 Aug 2016 19:20:54 +0000 (UTC) (envelope-from akshay1994.leo@gmail.com) Received: by mail-io0-x235.google.com with SMTP id q83so28402042iod.1; Thu, 18 Aug 2016 12:20:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=UY0ci7mNpYPJTqWJRy1ic/1AEHtM6+uVCrv/3aLGz5s=; b=YFSV494kq5tfZA5+qhZ4YPXuAJ6WqD3+4RK9M7QQiSjiX4GPbqnxI5lS0+dZohwn3t sfJYKs0fnDd2KpQwN70wTY1QU/VmIzlw5ECtAPM3p36UPoHVf5qCGZvhgm6hH5tHsnEs aVGdhfibHJ2U1mkGEGXUD9dKV/BNwy9S74+PPn3za44TmUCzY1gvN9/chA7sOQ5sJ3Rk II8B+W15pfZpDDLNFZu6lJVcYVry7+rxhTTbI3uZCiVUdSPQXSppSEhoRxWEyAExHh06 HPaQEWNSB+nj6sETFnZDkMjvwdrCDNYfoocisMGeYe65HImCHpulQKxwbAsXHRDQ1Apk Sjaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=UY0ci7mNpYPJTqWJRy1ic/1AEHtM6+uVCrv/3aLGz5s=; b=YTV70QG5dT8Tcf5UPx96My6VnCDguluqHieM9eBCY8LaEsWfg+Rqokmsg4IL/b6SyJ pKPbHwYkWDGh+Sv5NnE+Iq8tSC+TVwtpQlGJ1t6ZR36/R/Xv7cH1xGdPJf70RGUg9h5h 9Si1EfPcqf8Hz7oobMmUwlSdZt33HUfTrRnylHC1zoXLY49y4q4qXYcO6XIY0HkDhbuA uz9EaQUn8y6XdwG3I+vwioh6ACJMFphtR+yWPVudKykjtmv1KlxamR2xFxyzQbvM7y1a uSNjYqV72o/ETE6dhDnjCCUTGsk3t+PnPvIMa5HJd76ZFOyyX31pIzeiQ+kaWjvI5n2N 7Usw== X-Gm-Message-State: AEkoouukRDOo8vr1jSacIHq5yRUiqsn5Cc7TWkUtevjO1SHDkh5zNBxZCC0Qms7ly7uMQRXdd+Jy35IrfM+usA== X-Received: by 10.107.32.206 with SMTP id g197mr5199725iog.155.1471548053366; Thu, 18 Aug 2016 12:20:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.20.85 with HTTP; Thu, 18 Aug 2016 12:20:32 -0700 (PDT) In-Reply-To: References: From: Akshay Jaggi Date: Fri, 19 Aug 2016 00:50:32 +0530 Message-ID: Subject: Re: Grant Table Userspace Device - Status Update To: soc-status@freebsd.org Cc: Pedro Giffuni , =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2016 19:20:54 -0000 Carrying over discussion from IRC. 20:11 royger: ghost_rider: hello! I've been doing some testing with the > device today, and it seems there's a memory leak somewhere, after shutting > down all my domains I still see 1KB of memory used by the device, which > AFAICT is not expected (you can check with `vmstat -m |grep gntdev`) > Nope. That's not a leak. I ran `vmstat -m | grep gntdev` just after booting up Dom0, without any of the DomU's running, and I still saw 1KB of memory being used by the device. root@freebsd:~ # vmstat -m | grep gntdev gntdev 2 1K - 2 64 That is, 2 requests have been made, out of which both are currently active, without any DomU's active. After this I fired up a DomU with qdisk backends, and vmstat returned: root@freebsd:~/xen_test # vmstat -m | grep gntdev gntdev 2129 134K - 2137 32,64,128 Well in line with expectations. Now, powering off the DomU and running vmstat again, we get: root@freebsd:~/xen_test # vmstat -m | grep gntdev gntdev 2 1K - 2845 32,64,128 The initial 2 requests are still active, and this has nothing to do with the DomU's. The first malloc() that happens in the device is in the device open function at [1]. That means that someone has the device open. `fstat` confirmed my suspicions. `fstat` with DomU active: root@freebsd:~/xen_test # fstat | grep xen/gntdev root qemu-system-i386 1266 29 /dev 62 crw------- xen/gntdev rw root qemu-system-i386 1266 32 /dev 62 crw------- xen/gntdev rw root qemu-system-i386 1266 34 /dev 62 crw------- xen/gntdev rw root xenconsoled 751 6 /dev 62 crw------- xen/gntdev rw root xenstored 746 11 /dev 62 crw------- xen/gntdev rw `fstat` with DomU powered off: root@freebsd:~ # fstat | grep xen/gntdev root xenconsoled 751 6 /dev 62 crw------- xen/gntdev rw root xenstored 746 11 /dev 62 crw------- xen/gntdev rw So yep! It's no leak. Just that xenconsoled and xenstored keep the gntdev device open. I guess this would be expected behaviour. Let me know if it is not. 20:14 royger: ghost_rider: and I've also seen a "Can't find requested > grant-map." after attaching 4 Qdisk to a domain and done heavy IO to to > them. > 20:16 royger: although this last one I haven't been able to reproduce > That's pretty strange. I have never noticed this in any of my manual or stress tests. At this point I would also like to mention, that the xen-gnttab code is kind of buggy (putting it mildly, no offence). Like I pointed out in the xen-devel patch thread, there is a place in code where "-1" is being used to specify there is no CLEAR_BYTE notify. But this is not being checked for inside the function, which would have caused a clear-byte notification on a different page, causing data corruption. The only reason this bug is not doing so, is because of another bug, where this -1 is being passed on to an unsigned int32, which would keep it out of bounds for most requests. I don't think this has to do anything with our device. If we lost some unmap request (which is where this message is generated) we would have surely leaked the memory for the gmap structure associated with that request (because, 1. ref-counting, 2. transferred to global clean list only on an unmap request), and that would have been visible in `vmstat`. Let me know if this repeats. > 20:40 royger: and I'm not sure if you tested it, but if you attach a > ramdisk to a VM (one created with `mdconfig -t malloc ...`) and try to run > newfs against it, it doesn't work, a bunch of read errors appear on both > the DomU console and Qemu log. Although it works with a plain file, so I > guess this is probably some bad interation between Qemu and FreeBSD block > devices... Mhm. Sounds like that. I'll try it out on my setup and post the results. Regards, Akshay [1] https://github.com/akshay1994/freebsd/blob/grant_table/sys/ dev/xen/gntdev/gntdev.c#L896 On 18 August 2016 at 23:21, Akshay Jaggi wrote: > Hello Everyone, > > Hope everyone is doing great. > > This is the last week of GSoC-2016, and the grant table userspace device > is complete. We are actively working to get the code reviewed and pushed. > > Here is the list of patches in the pipeline: > > - https://reviews.freebsd.org/D7429 > - https://reviews.freebsd.org/D7293 > - https://lists.xenproject.org/archives/html/xen-devel/2016- > 08/msg02125.html > > I'll be using this thread to discuss the overall progress of code-review > for all the three patches, and I hope that would be fine. > > Looking forward to getting this stuff to mainline. :) > > Regards, > Akshay > > On 20 July 2016 at 20:33, Akshay Jaggi wrote: > >> Hello Everyone, >> >> >> I hope everyone is doing great. >> >> >> First things first. I want to apologise for not updating my blogpost >> after June 15. Some personal stuff happened on June 17, and since then all >> my work routines are completely disturbed. Soc-admins know about this, and >> I hope the community can pardon my irregularity this once. >> >> >> Now, time for some status update on the project. We have a working Grant >> Table User-space Device (with some caveats, of course). Yay!! >> >> Check out the code on the pull request: https://github.com/fr >> eebsd/freebsd/compare/master...akshay1994:grant_table?expand=1 >> >> Test the device by compiling a custom kernel; from my working branch: >> https://github.com/akshay1994/freebsd/tree/grant_table >> >> >> Caveats and work yet to be done: >> >> - A kernel panic happens if you terminate the program without >> un-mmapping all the mmaps. >> - The interface for a grant map is not completely in sync with the >> Linux interface yet. Notably, we still map and subsequently mmap partially >> working grants. This is a bug since the pager will fail on a page fault for >> the non-working-grant. >> - I'm yet to add event channel code for the notify ioctls. >> - I'm yet to get in touch with the vm-guys. I spent a lot of time >> understanding the vm structure, but I surely do not have a complete >> picture. I'm using a MGTDEVICE cdev pager to handle managed, fictitious >> pages from mapped grants, and a custom PHYS vm object to manage wired >> physical memory for allocated grants. I have to >> - Confirm if the approach I am using is the best one. >> - The interface differs compared to Linux with regards to multiple >> mmaps, which I need a little help with. >> >> >> Looking forward to some community testing of the working parts. >> Comments are welcome and highly appreciated. :) >> >> >> Regards, >> Akshay Jaggi >> > > From owner-soc-status@freebsd.org Sat Aug 20 19:12:22 2016 Return-Path: Delivered-To: soc-status@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F63FBC0521 for ; Sat, 20 Aug 2016 19:12:22 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 56BE417AA for ; Sat, 20 Aug 2016 19:12:22 +0000 (UTC) (envelope-from v.maffione@gmail.com) Received: by mail-oi0-x22f.google.com with SMTP id 4so104705011oih.2 for ; Sat, 20 Aug 2016 12:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=0DNlHyxX1VUyqhNVLI8MN9xxM93NqALSL9/qkv0Egd8=; b=TF5sSz13/DqP5kDUFVwnsod1MIEQCuobjFvHnKz22Gdia9/DALsZoe4jH4kXMR7mKV Uk7wduXDhijRZTuP/7ChKEoKsJeWBiBgq30OdKHi3NEJyHD8sujMOaauhOBZJ81aNLjU IkDP0sZAuF0xoRsMDOMmMvkE5VtXNKrAt0IKRRJiDwb3bb+V2dqkois8MXaag8rWn75c z4ERdQY9snyELhTOiY7sUh9dxUEC71lg4yw82LBa3lgr7zhzshxOepieA489YjafBAuY izLQXJgoFSVSw/YrfVOQVntTJ90KEMu7CQ4m6tgO1nXQaAIpM46IxRaUIxrgNq2XdVnk /p1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=0DNlHyxX1VUyqhNVLI8MN9xxM93NqALSL9/qkv0Egd8=; b=QnUvq8AszBmw3tVyBLASnknSY0k3YkPoh8Gtu2AsCOV6a1RtSTfbhJIUzAdQSLJB9t X6cXniJ53J2ertZtSCRTr9rYxwK7Tdbb0QfSfqMFlzmhc6fhxZ7AmcG3v9rdQlipsuOS QgYOxBjIX3lC4FEYu60JbIVyMZChHERY2O8cHrR1SV8Z7WY9MKcSNVvFCE5atYJ+xXIE eQfw/vvj2rc2fzeYe40IUewXpHEVst3iDIcrp1NUE7uj4ZKEYJwWNFP3KYJAMLnh89g0 nPRm8Czj4mfOllU82Vc+a5Fl2+mVvPw4Z9CB/V0NBo8EQLv0oopaWWjfwkTddW0kvHSN telw== X-Gm-Message-State: AEkoouuvwpfoIwfSZr7a35EbXxaix5aDyAVep2BHn0JNxHSvOgiGDs60F2Dre4b0i4sT35l5nrrCcDPlPoUyTQ== X-Received: by 10.157.11.176 with SMTP id 45mr8364761oth.82.1471720341420; Sat, 20 Aug 2016 12:12:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.40.215 with HTTP; Sat, 20 Aug 2016 12:12:21 -0700 (PDT) From: Vincenzo Maffione Date: Sat, 20 Aug 2016 21:12:21 +0200 Message-ID: Subject: High-performance TCP/IP networking for bhyve VMs using netmap passthrough To: soc-status@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: soc-status@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Summer of Code Status Reports and Discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Aug 2016 19:12:22 -0000 Hello Sorry for the delayed update, I've been offline for a week. Project summary ================= ptnet is a new multi-queue paravirtualized device that builds on ptnetmap to allow VMs to exchange TCP traffic at 20 Gbps, while still offering the same ptnetmap performance (10-100 Mpps) to native netmap applications. The purpose of this SOC project is to write a ptnet driver for FreeBSD and a corresponding device model for the bhyve hypervisor. Status update for week 12 and 13 =================== During these weeks I've completed the ptnet device model emulation and performed throughput tests. Results are good (~15 Gbps throughput among two VMs), and 20 Mpps with netmap (pkt-gen). I've also slightly optimized the vmm.ko support to speed up the lookup of the callback to be invoked on guest register write, using a RW lock in place of a mutex. Finally, I did bhyve code cleanup, and ported netmap and bhyve to the freebsd head, since my developments were based on 10.3-release. I prepared the documentation for the final submission in the wiki page of my project (see link below). Links: Project wiki page: https://wiki.freebsd.org/SummerOfCode2016/PtnetDriverA ndDeviceModel Code: https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/sys/ dev/netmap/if_ptnet.c?view=markup https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/ usr.sbin/bhyve/pci_ptnetmap_netif.c?view=markup https://svnweb.freebsd.org/socsvn/soc2016/vincenzo/head/ usr.sbin/bhyve/net_backends.c?view=markup -- Vincenzo Maffione