From owner-freebsd-virtualization@freebsd.org Sun Jun 24 10:13:17 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8EE35102291E for ; Sun, 24 Jun 2018 10:13:17 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-lf0-x232.google.com (mail-lf0-x232.google.com [IPv6:2a00:1450:4010:c07::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E563B83039 for ; Sun, 24 Jun 2018 10:13:16 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: by mail-lf0-x232.google.com with SMTP id n24-v6so12266788lfh.3 for ; Sun, 24 Jun 2018 03:13:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=FJ3/oxosEPW6d2ya72MihzIgiEeB6LNcSKbxoal1XFk=; b=QFwhYZo7qPt7oIps3N38utOd2wSnlgco467oFxe56xdqU75/VBsjz02hyTeVlAIm9j J5CN31ueKK+HI+QsxHsKP+ryt1jNMiC0yHP+b8QPgbiqVTJtlwDdXeqQ9BDe70hxrHt4 6BtJb4yqdlgVxrASJ47Q+/hYk9HrOsEKy9VuglwJ/E1LGW9dV8zGE8q9GvEe8k/RHjbj Tpdb2jAkt2k6dMIk5dRx458p0C5a5gf01MApQd+Zode5lN9iNpMrxWDN2rGU5H4mFLiD ugQH1YbK091e/WESEXJJB90APmC42uY3uCVdrz8DNtDZX8BzN3rzCUd+ZsNEZjcFSEFi rW/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:subject:message-id :mime-version:content-disposition:user-agent; bh=FJ3/oxosEPW6d2ya72MihzIgiEeB6LNcSKbxoal1XFk=; b=bTEuqkMxj7DRjUQRbK4A0Bcd3aRrjeTIrTtz/wLUsWkpvicY/ZPqpHSwtgfEqqS/vM RAAwowV5IcbKC3m4OF0B1IraiUtYsT2gCupzACzVFj6gAAYFahB1rvL6ULBF/4194e89 j12+pMoaz4N19cTVq0HpCUjKN7YqIHCqrwGFk2K61Ils7qYiV5v6Ezi1FHqGaiOHorip nmYRun32WcVyfNMSlcYCOXJYuBVbmoIXXxnnxwkFDcB0Mt7lZW8lpZMljPW74rJ0FX9f bJOm6hiCd1B1wbUaIUQlYr3hKm3Kbtk38nlAE1SKUPzkT11nDmwW0kiC0RA2Q3iayhDh qeww== X-Gm-Message-State: APt69E0xUa5tw1qCNehXlLeTO7TeGF3a20YbJPgW5av7edcixxuXPIzL 49TFd5ycYkVSWs1okyWELbB6qA== X-Google-Smtp-Source: ADUXVKKlFPZxZD6P8gSIx0xQIu9wj1EMfhtWkFGF6vgMoW56EYUKM/eTAMk2bf7lNu852yF1fXR/oA== X-Received: by 2002:a19:a78a:: with SMTP id q132-v6mr3527631lfe.126.1529835194776; Sun, 24 Jun 2018 03:13:14 -0700 (PDT) Received: from kloomba ([213.147.213.47]) by smtp.gmail.com with ESMTPSA id i67-v6sm1898346lji.49.2018.06.24.03.13.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 24 Jun 2018 03:13:14 -0700 (PDT) Sender: Roman Bogorodskiy Date: Sun, 24 Jun 2018 14:13:02 +0400 From: Roman Bogorodskiy To: freebsd-virtualization@freebsd.org Subject: RFC: bhyve supported features reporting Message-ID: <20180624101301.GB2252@kloomba> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="xgyAXRrhYN0wYx8y" Content-Disposition: inline User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jun 2018 10:13:17 -0000 --xgyAXRrhYN0wYx8y Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Bhyve evolves over time, and new features get added, such as new devices support, new guest configurations and so on. However, it's not really straight-forward to figure out what features a given bhyve binary supports. This makes it harder to develop tools on top of bhyve, specifically error reporting. For example, libvirt's bhyve driver [1] probes bhyve capabilities [2] using: * Running 'bhyve -h' and parsing output, * For detecting devices, it runs 'bhyve -s 0,dev' and parses error output to see if the device is supported. It's not very effective because 'bhyve' binary has to be executed multiple times just to check things. Additionally, it's harder to check things on a deeper level, e.g. specific device parameters. Also, this is error-prone because help output generally is not designed for machine-parsing, and this could easily break on slight formatting change. I would like to discuss introducing a general way of reporting features bhyve supports. To start a discussion, I've created a simple draft/PoC which adds '-f' (as for features) command line switch to bhyve. This switch makes bhyve print its supported features. I'll use JSON as example, however, as it's done using libxo, XML is also supported. Example JSON output with inline comments below: "bhyve": { // 'features' schema version using the common versioning pattern: // major version increments on incompatible changes, // minor version increments on compatible changes (extensions). "schema_version": "1.0", // there could also go some general properties like max_cpus etc // list of specific features, mostly should be self-descriptive "features": { "rtc_utc": { "description": "RTC keeps UTC time", "cmd": { "switch": "-u" } }, "wire_guest_memory": { "description": "Wire guest memory", "cmd": { "switch": "-S" } }, "devices": { "description": "Devices support", "cmd": { "switch": "-s", "arguments": [ =20 // Probably, it'd be better to make this more organized, // e.g. instead of a string with all arguments and ar= g/value paris, // use a list of objects which would include possible= values, // required/optional, etc {"options": "virtio-net,tapN,mac=3Dxx:xx:xx:xx:xx:xx", "description": "Virtio network device" }, {"options": "virtio-blk,path,nocache,direct,ro,sectorsize=3Dlogical= /physical", "description": "Virtio block device" }, {"options": "fbuf,rfb,rfb=3DIP:port,w=3Dwidth,h=3Dheigh,vga=3Dvgaco= nf,wait,password=3Dpassword", "description": "Framebuffer device" } ] } } } } Sample code is here: https://reviews.freebsd.org/D15992. At this point it's just an excuse to start a discussion; it needs some macros to make items creation easier, and it needs to have all the other features/devices populated. 1: https://libvirt.org/drvbhyve.html 2: https://github.com/libvirt/libvirt/blob/master/src/bhyve/bhyve_capabilities= =2Ec Roman Bogorodskiy --xgyAXRrhYN0wYx8y Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJbL26tAAoJEMltX/4IwiJqFP0H/Ar75ephp349XKXEK3ltpkF+ vsO3KLENOH73EkXHixwLN571rq754fyYzUpeB4NT/a3ao8af8BuinZuIoEVEAqBK +4gi/DUXJd2Rabpxc6yy4OYv2o8eHRkUQfhv0J/LBnlGRGj6mbBoBbBNpnyjEwHg rvDvbyHUiCka60XULNZDf2mUOaOmntmok5Ns7kFZfDenin+GlecVaAGKrUwsUfZ4 mHrdD2xJwaumWQV9tmPdscTUeMEu+7/7HSk7ujH96w/upzGsiYQp60tNrspROmNn tt6ei6CB7PUEvKjzMltz1XGAZ+gW3smWWZerfqeWTufGQ6fVXQNYLREKKRf+Kfk= =OXji -----END PGP SIGNATURE----- --xgyAXRrhYN0wYx8y-- From owner-freebsd-virtualization@freebsd.org Sun Jun 24 18:06:16 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 33069100747D for ; Sun, 24 Jun 2018 18:06:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BDADB7188D for ; Sun, 24 Jun 2018 18:06:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 7E1DE100747C; Sun, 24 Jun 2018 18:06:15 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B4F6100747A for ; Sun, 24 Jun 2018 18:06:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 057ED7188B for ; Sun, 24 Jun 2018 18:06:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4853527FC6 for ; Sun, 24 Jun 2018 18:06:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5OI6EkW081781 for ; Sun, 24 Jun 2018 18:06:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5OI6E1x081778 for virtualization@FreeBSD.org; Sun, 24 Jun 2018 18:06:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 216760] poor performance on QEMU-KVM host Date: Sun, 24 Jun 2018 18:06:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rebecca@bluestop.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jun 2018 18:06:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216760 --- Comment #5 from Rebecca Cran --- This appears to be the same bug report as https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216759 ? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Jun 25 03:41:13 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54BB71021ECD for ; Mon, 25 Jun 2018 03:41:13 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A57E788C15; Mon, 25 Jun 2018 03:41:12 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lj1-x22a.google.com with SMTP id l12-v6so532031lja.1; Sun, 24 Jun 2018 20:41:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc; bh=qfNWgxdpF5InScDXkMrZHLT8PD2GzLi8tZleuq0PiaI=; b=L7kqIuX6/obXXCF+8KQ3gHZVcErpSMABGtbYj1xE6Y7i5lM1Cc/Nbe7CIkVelBDb56 sFSgkELonOHwQvLIVaOlcatnI+UyNviRyfKmyotjvNtbNGfAg7FjeeFTXZPA9hFvidzG 0tgRO3F6kHp9HUZxLVBJ2LbYgpnxFPOiywDjOuu/gnVQyZgKzds2xOK8b3fvPKgBbWF+ 6sXyvar9ZCiJoq3VZocmn+Eye8k/z02VeNEK9K/3M1OIjcZLBL6Ummcy/hvZeita0+AI jy4wHQTeptCBcR7KW/RcB9dbgMisOZfTxZzXFDCrs5Dc4JT51rNjnpopqPId2qBpyzgC 7PJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=qfNWgxdpF5InScDXkMrZHLT8PD2GzLi8tZleuq0PiaI=; b=MeXA1on7zqHmk0RwOYgMzQWst9tIKYa5d3rfKgU2/u9MJYercARXzgpIkYHDelwYXT 80Kr8sYaI0cd6clsVoZ8Bzv9HBdn0Rxg6VdSqDeZmhYSRAbMmoiI09nMmSNcSGYgMJkT /mJeaZ7vJqQviuPi78hORUftCdEqU8FwRLKnFGdEjXJEeYYdcfBF9L103jMEWUUg7rMa rSBeMCnn8I+66Vm2GiGHGdbNHycHhpQHvVTTGys3U9/vys1lfNQPzB/4HOgybWKVEh2e VGvDqGVqSIymBuYb2ReEWzrPf6EbWd6Xwbp+IPKXEQVaCKohuPU63eqbWVe1ybdTBNwF 2F0A== X-Gm-Message-State: APt69E3/na0Nd89+JVI6hY0V9lAlfjGyr8d+amUBUgRDGVA5uzMrcjJO YmHYSO7PGh9TlL0XCFn1NzomPVo49THTUrT7L0HhZg== X-Google-Smtp-Source: ADUXVKIhoYWuEDAxiZDtqSs7LIA6Nz/8/JbZUl5AGrdEEUpjEU3nivfvVOcXK/SXINQDn2832mGHPyYpje6VzSLJ1bY= X-Received: by 2002:a2e:9cd8:: with SMTP id g24-v6mr6647528ljj.141.1529898070287; Sun, 24 Jun 2018 20:41:10 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a19:1f94:0:0:0:0:0 with HTTP; Sun, 24 Jun 2018 20:41:08 -0700 (PDT) Reply-To: araujo@freebsd.org In-Reply-To: <20180624101301.GB2252@kloomba> References: <20180624101301.GB2252@kloomba> From: Marcelo Araujo Date: Mon, 25 Jun 2018 11:41:08 +0800 Message-ID: Subject: Re: RFC: bhyve supported features reporting To: Roman Bogorodskiy Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jun 2018 03:41:13 -0000 Hi Roman, I think it is doable, my only concern is the complexity and maintainability, as along the way we will add more devices and would be nice to have something transparent for those additions or at least something easy to be extended. As an example this patch that you made: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210111 It has much less complexity, and seemed for me to be much easier to maintain, however doesn't give us a detailed information like the latest one you are proposing, I'm not sure if we really need so extensive details, I didn't see any bhyve device changing its options along the past years, but that doesn't means it would not in the future. I do believe if we remove the complexity or at least makes it easy to be extend, IMHO it is a good idea. Best, 2018-06-24 18:13 GMT+08:00 Roman Bogorodskiy : > Bhyve evolves over time, and new features get added, such as new > devices support, new guest configurations and so on. However, it's not > really straight-forward to figure out what features a given bhyve binary > supports. This makes it harder to develop tools on top of bhyve, > specifically error reporting. > > For example, libvirt's bhyve driver [1] probes bhyve capabilities [2] > using: > > * Running 'bhyve -h' and parsing output, > * For detecting devices, it runs 'bhyve -s 0,dev' and parses error > output to see if the device is supported. > > It's not very effective because 'bhyve' binary has to be executed > multiple times just to check things. Additionally, it's harder to check > things on a deeper level, e.g. specific device parameters. Also, this is > error-prone because help output generally is not designed for > machine-parsing, and this could easily break on slight formatting > change. > > I would like to discuss introducing a general way of reporting features > bhyve supports. To start a discussion, I've created a simple draft/PoC > which adds '-f' (as for features) command line switch to bhyve. This > switch makes bhyve print its supported features. I'll use JSON as > example, however, as it's done using libxo, XML is also supported. > Example JSON output with inline comments below: > > "bhyve": { > > // 'features' schema version using the common versioning pattern: > // major version increments on incompatible changes, > // minor version increments on compatible changes (extensions). > "schema_version": "1.0", > > // there could also go some general properties like max_cpus etc > > // list of specific features, mostly should be self-descriptive > "features": { > "rtc_utc": { > "description": "RTC keeps UTC time", > "cmd": { > "switch": "-u" > } > }, > "wire_guest_memory": { > "description": "Wire guest memory", > "cmd": { > "switch": "-S" > } > }, > "devices": { > "description": "Devices support", > "cmd": { > "switch": "-s", > "arguments": [ > > // Probably, it'd be better to make this more > organized, > // e.g. instead of a string with all arguments and > arg/value paris, > // use a list of objects which would include > possible values, > // required/optional, etc > > {"options": "virtio-net,tapN,mac=xx:xx:xx:xx:xx:xx", > "description": "Virtio network device" > }, > {"options": "virtio-blk,path,nocache, > direct,ro,sectorsize=logical/physical", > "description": "Virtio block device" > }, > {"options": "fbuf,rfb,rfb=IP:port,w=width, > h=heigh,vga=vgaconf,wait,password=password", > "description": "Framebuffer device" > } > ] > } > } > } > } > > Sample code is here: https://reviews.freebsd.org/D15992. At this point > it's just an excuse to start a discussion; it needs some macros to make > items creation easier, and it needs to have all the other > features/devices populated. > > 1: https://libvirt.org/drvbhyve.html > 2: > https://github.com/libvirt/libvirt/blob/master/src/bhyve/ > bhyve_capabilities.c > > Roman Bogorodskiy > -- -- Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-virtualization@freebsd.org Mon Jun 25 15:21:27 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28F501013441 for ; Mon, 25 Jun 2018 15:21:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B609D81369 for ; Mon, 25 Jun 2018 15:21:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6FCFB1013440; Mon, 25 Jun 2018 15:21:26 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CC0D101343F for ; Mon, 25 Jun 2018 15:21:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC35D81366 for ; Mon, 25 Jun 2018 15:21:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 3B1EC135E7 for ; Mon, 25 Jun 2018 15:21:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5PFLPdu089871 for ; Mon, 25 Jun 2018 15:21:25 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5PFLPbU089868 for virtualization@FreeBSD.org; Mon, 25 Jun 2018 15:21:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 225791] ena driver causing kernel panics on AWS EC2 Date: Mon, 25 Jun 2018 15:21:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: richard@primarysite.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jun 2018 15:21:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D225791 Richard Paul changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |richard@primarysite.net --- Comment #2 from Richard Paul --- We're seeing this now since we migrated out instance between instance types= .=20 Before we were using a c5.9xlarge instance and we recently scaled that back down (we'd rebuilt and scaled up earlier in the year to deal with seasonal load). This instance is our primary db instance and the kernel panics seem to happ= en either during the db dump process (which occur when we have the least amoun= t of DB traffic) or as has happened once so far, during the peak of the daily lo= ad. Our log output is almost the same except as that already submitted, although out current process line reads: ```current process =3D 12 (irq269: ena0)``` The two previous reports had the same IRQ number and are on the same class = of instance types whereas we're on an M5 class instance type and get a slightly different IRQ number. Also, this was happening on 11.1p4 but we since upgr= aded to p10 and the issue still is occurring. I'm planning on upgrading a clone= to 11.2p0 later this week to check out whether there's a new version of the ENA (from here https://github.com/amzn/amzn-drivers/tree/master/kernel/fbsd/ena= ) in this build. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Jun 26 07:07:18 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 478011015516 for ; Tue, 26 Jun 2018 07:07:18 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [IPv6:2a00:1450:4864:20::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A59083261; Tue, 26 Jun 2018 07:07:17 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: by mail-lj1-x234.google.com with SMTP id u6-v6so4774057lju.13; Tue, 26 Jun 2018 00:07:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=LoToIw59l2/kc6cHIrV/2eTR3r76PNq0H52Sx++hHkM=; b=S0Sfpz3d0yl2jiVSw3LYYbH0CNuS1EttrcRqopCYLD75tlPao3ghaQ4sXlEZnOPNOj 5w6cbfhQhkxmiJjhZB9lu+htK+MuaqM3igoUEn+J/OgnBp5j3Kj18IPO7Kli/QB2cTbD 7nBcRUCC/irb0XlMJDR7W8k0XoP5PI8Nw5pbk94DuwFaOHMDjhfYWS9++dicKW0iDnSm GX8K2gxgMm7Q2jEdI7hiMuOi0VcXujFhab2+4jX9a+GUzE4cLWxaE6zgF9hl1hRmnwHT Xg3EG9zpMmDtCpX9ooM5jz3u1r2xAPiJm6HIuxdiD+pEXXl/+j0xuqB9wMoa9qk6oG6y 8a7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=LoToIw59l2/kc6cHIrV/2eTR3r76PNq0H52Sx++hHkM=; b=VgvzfDKFk9qFHHm1o5hQibNs+c1OA4tg2xhLhftgMf0wpnLv55laJOBqsul2Edkqbh aKUvy6YU5114rh51ng+z54lJDQJO9dEnVb0ZInoo9/L45+c8Jqzhb2AuM7Zhyg9Fkt9N YEVsfkdgUciRIflKgn++SVnxuOolVpZ0Q1GGLbmWHpbbi6MGm6cinl6im2tRemwjS+vs xV7+yWq6X/AESA/1o5MogbsNCFCSZI4UyHv4m12uuve2wKZ8PgGgbIWMB9NKK2yXCn8p bsiNT2dNtfR0wXNWtxGEppE08AKAkTdqCTQbp24dY/gz2Pno6en2JqDjCfouaJbuDbAO Gx+w== X-Gm-Message-State: APt69E2bjv3tlWFs8QtXPKZHTtYe782rh48shaEKvwIICVc9gnwS4CFx UY/Mwi/Lu7Pyk3HXTtGe0M7+Xw== X-Google-Smtp-Source: AAOMgpfSwtgR35P5VesPxfRotAUi3ZTh3bdl6vWKL9pYfvUE9ZQ+btYnvTaDUD0m/GwHX5eunDH+iQ== X-Received: by 2002:a2e:8455:: with SMTP id u21-v6mr260267ljh.66.1529996835672; Tue, 26 Jun 2018 00:07:15 -0700 (PDT) Received: from kloomba ([213.147.213.47]) by smtp.gmail.com with ESMTPSA id h197-v6sm187008lfh.13.2018.06.26.00.07.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Jun 2018 00:07:14 -0700 (PDT) Sender: Roman Bogorodskiy Date: Tue, 26 Jun 2018 11:07:08 +0400 From: Roman Bogorodskiy To: araujo@freebsd.org Cc: freebsd-virtualization@freebsd.org Subject: Re: RFC: bhyve supported features reporting Message-ID: <20180626070704.GA1886@kloomba> References: <20180624101301.GB2252@kloomba> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="M9NhX3UHpAaciwkO" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2018 07:07:18 -0000 --M9NhX3UHpAaciwkO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Marcelo Araujo wrote: > Hi Roman, >=20 > I think it is doable, my only concern is the complexity and > maintainability, as along the way we will add more devices and would be > nice to have something transparent for those additions or at least > something easy to be extended. Yeah, it's true, I think it would be good to make it more compact somehow. I'll play around with macros to see if I can make it easier to add new items. > As an example this patch that you made: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210111 >=20 > It has much less complexity, and seemed for me to be much easier to > maintain, however doesn't give us a detailed information like the latest > one you are proposing, I'm not sure if we really need so extensive detail= s, > I didn't see any bhyve device changing its options along the past years, > but that doesn't means it would not in the future. Yeah, this patch makes things easier, but doesn't cover all the cases. For example, at some point VNC authentication was added [1] through adding a 'password' keyword, but with an approach like that it's not possible to know if the given bhyve version supports that or not. 1: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D319487 > I do believe if we remove the complexity or at least makes it easy to be > extend, IMHO it is a good idea. >=20 > Best, >=20 >=20 > 2018-06-24 18:13 GMT+08:00 Roman Bogorodskiy : >=20 > > Bhyve evolves over time, and new features get added, such as new > > devices support, new guest configurations and so on. However, it's not > > really straight-forward to figure out what features a given bhyve binary > > supports. This makes it harder to develop tools on top of bhyve, > > specifically error reporting. > > > > For example, libvirt's bhyve driver [1] probes bhyve capabilities [2] > > using: > > > > * Running 'bhyve -h' and parsing output, > > * For detecting devices, it runs 'bhyve -s 0,dev' and parses error > > output to see if the device is supported. > > > > It's not very effective because 'bhyve' binary has to be executed > > multiple times just to check things. Additionally, it's harder to check > > things on a deeper level, e.g. specific device parameters. Also, this is > > error-prone because help output generally is not designed for > > machine-parsing, and this could easily break on slight formatting > > change. > > > > I would like to discuss introducing a general way of reporting features > > bhyve supports. To start a discussion, I've created a simple draft/PoC > > which adds '-f' (as for features) command line switch to bhyve. This > > switch makes bhyve print its supported features. I'll use JSON as > > example, however, as it's done using libxo, XML is also supported. > > Example JSON output with inline comments below: > > > > "bhyve": { > > > > // 'features' schema version using the common versioning pattern: > > // major version increments on incompatible changes, > > // minor version increments on compatible changes (extensions). > > "schema_version": "1.0", > > > > // there could also go some general properties like max_cpus etc > > > > // list of specific features, mostly should be self-descriptive > > "features": { > > "rtc_utc": { > > "description": "RTC keeps UTC time", > > "cmd": { > > "switch": "-u" > > } > > }, > > "wire_guest_memory": { > > "description": "Wire guest memory", > > "cmd": { > > "switch": "-S" > > } > > }, > > "devices": { > > "description": "Devices support", > > "cmd": { > > "switch": "-s", > > "arguments": [ > > > > // Probably, it'd be better to make this more > > organized, > > // e.g. instead of a string with all arguments and > > arg/value paris, > > // use a list of objects which would include > > possible values, > > // required/optional, etc > > > > {"options": "virtio-net,tapN,mac=3Dxx:xx:xx:xx:xx= :xx", > > "description": "Virtio network device" > > }, > > {"options": "virtio-blk,path,nocache, > > direct,ro,sectorsize=3Dlogical/physical", > > "description": "Virtio block device" > > }, > > {"options": "fbuf,rfb,rfb=3DIP:port,w=3Dwidth, > > h=3Dheigh,vga=3Dvgaconf,wait,password=3Dpassword", > > "description": "Framebuffer device" > > } > > ] > > } > > } > > } > > } > > > > Sample code is here: https://reviews.freebsd.org/D15992. At this point > > it's just an excuse to start a discussion; it needs some macros to make > > items creation easier, and it needs to have all the other > > features/devices populated. > > > > 1: https://libvirt.org/drvbhyve.html > > 2: > > https://github.com/libvirt/libvirt/blob/master/src/bhyve/ > > bhyve_capabilities.c > > > > Roman Bogorodskiy > > >=20 >=20 >=20 > --=20 >=20 > --=20 > Marcelo Araujo (__)araujo@FreeBSD.org > \\\'',)http://www.FreeBSD.org \/ \ ^ > Power To Server. .\. /_) Roman Bogorodskiy --M9NhX3UHpAaciwkO Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJbMeYYAAoJEMltX/4IwiJqyp8H/iRk2CYJE+eVFlIlOoKM9Y0J eRQlx/Bt3+TA/xFKzdX9lKKo/pTsJH9yxgv7oaEW4dTQ2z9Ry0oDUD4S6+u5S10a HPP1UthQAx6prHaqDccTwxDaXRJY0CyaYx5eOwNJYvesQ+MQ7ANtsCDetagLj2D6 rHvWsbJhv30K+YYe7sYcQx+xFT3HmI4qp7FjMZgyUatWZ4oUGtOWzWE/wp5d3e2E /v7ca+oqEPbqRfLlNY2upCg85HTtbI5epTaE2DzmSc9zV7kfEyh+1whnmhHO0FgN iD+d2DbO34kEjWdpvJ+qAuoFpYh5p5o9AD+SGrzipihLRTVcEqJroDGTv89U42U= =+LDG -----END PGP SIGNATURE----- --M9NhX3UHpAaciwkO-- From owner-freebsd-virtualization@freebsd.org Tue Jun 26 10:49:52 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 906E610208F6 for ; Tue, 26 Jun 2018 10:49:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 25C038B834 for ; Tue, 26 Jun 2018 10:49:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id D385710208F5; Tue, 26 Jun 2018 10:49:51 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C10FE10208F4 for ; Tue, 26 Jun 2018 10:49:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C2A08B832 for ; Tue, 26 Jun 2018 10:49:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 9B6C91D83D for ; Tue, 26 Jun 2018 10:49:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5QAnoa9055794 for ; Tue, 26 Jun 2018 10:49:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5QAnodP055793 for virtualization@FreeBSD.org; Tue, 26 Jun 2018 10:49:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 225791] ena driver causing kernel panics on AWS EC2 Date: Tue, 26 Jun 2018 10:49:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: richard@primarysite.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2018 10:49:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D225791 --- Comment #3 from Richard Paul --- (In reply to Richard Paul from comment #2) Just a quick grab of our DB server's current throughput with `systat -ifstat -pps`: ena0 in 4.498 Kp/s 4.498 Kp/s 279.347 Mp out 12.956 Kp/s 12.956 Kp/s 674.161 Mp We have a couple of Varnish servers in front of this platform (r4.2xlarge) which are rock solid and which don't see anything like this kind of through= put which is possibly why it's only this server we're seeing this issue with. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jun 27 19:06:25 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D02DF100257C for ; Wed, 27 Jun 2018 19:06:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 688618D3F2 for ; Wed, 27 Jun 2018 19:06:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 25E101002579; Wed, 27 Jun 2018 19:06:25 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 145461002577 for ; Wed, 27 Jun 2018 19:06:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A47C78D3EE for ; Wed, 27 Jun 2018 19:06:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 0BD52E5E1 for ; Wed, 27 Jun 2018 19:06:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5RJ6NBx014069 for ; Wed, 27 Jun 2018 19:06:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5RJ6NnE014068 for virtualization@FreeBSD.org; Wed, 27 Jun 2018 19:06:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 225791] ena driver causing kernel panics on AWS EC2 Date: Wed, 27 Jun 2018 19:06:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cperciva@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2018 19:06:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D225791 --- Comment #4 from Colin Percival --- Sorry, I'm coming to this late -- somehow I never saw this PR earlier. It's possible that this is an ENA driver bug, but C5 also has the added complication of using an entirely different virtualization platform, and I'= m a bit suspicious of the backtrace here. Can one of you try to reproduce this= on a different instance type -- m4.16xlarge or r4.* would probably be best -- = so we can see if it's specifically an *ENA* problem or a *C5* problem? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jun 28 10:46:45 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4FD7102B0D7 for ; Thu, 28 Jun 2018 10:46:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3D58E75DED for ; Thu, 28 Jun 2018 10:46:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id F1FC9102B0D5; Thu, 28 Jun 2018 10:46:44 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF916102B0D3 for ; Thu, 28 Jun 2018 10:46:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7D5CF75DE8 for ; Thu, 28 Jun 2018 10:46:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id D2E851691A for ; Thu, 28 Jun 2018 10:46:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5SAkhHP018739 for ; Thu, 28 Jun 2018 10:46:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5SAkh48018738 for virtualization@FreeBSD.org; Thu, 28 Jun 2018 10:46:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 225791] ena driver causing kernel panics on AWS EC2 Date: Thu, 28 Jun 2018 10:46:43 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: richard@primarysite.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 10:46:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D225791 --- Comment #5 from Richard Paul --- Hi Colin, Thanks for responding to this issue. You're right to point out that this m= ay be down to the difference in hypervisors. M5 seem to be based on HVM too so for our purposes maybe moving over to an r4.2xlarge would be our nearest alternative in the r4 range of instances. We're waiting for the inevitable failure of our current instance as it seems to be falling over after 6-7 da= ys. What we need is some kind of reproducible test case for this to better be a= ble to diagnose the issue. As the other reporters in this ticket say, this can take hours to a day to reproduce and in our case, multiple days, which makes finding such a test case so time consuming and difficult. This is what I'm currently attempting to do with a cloned M5 instance type = of our DB server running the database dump and then trying to load the server heavily but it's not producing much in the way of results at the moment. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jun 28 13:23:32 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B26761033F49 for ; Thu, 28 Jun 2018 13:23:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 436DC7BB35 for ; Thu, 28 Jun 2018 13:23:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id F14711033F47; Thu, 28 Jun 2018 13:23:31 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DEDDE1033F46 for ; Thu, 28 Jun 2018 13:23:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 788FC7BB31 for ; Thu, 28 Jun 2018 13:23:31 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id C9EE817F40 for ; Thu, 28 Jun 2018 13:23:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5SDNUxb076841 for ; Thu, 28 Jun 2018 13:23:30 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5SDNU0B076840 for virtualization@FreeBSD.org; Thu, 28 Jun 2018 13:23:30 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 222996] FreeBSD 11.1 on Hyper-V with PCI Express Pass Through Date: Thu, 28 Jun 2018 13:23:31 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dmitry_kuleshov@ukr.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 13:23:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222996 --- Comment #8 from Dmitry --- (In reply to Dexuan Cui from comment #7) here is. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jun 28 13:25:03 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94594103405B for ; Thu, 28 Jun 2018 13:25:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2C5E07BC0A for ; Thu, 28 Jun 2018 13:25:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id E000D1034059; Thu, 28 Jun 2018 13:25:02 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CDBD01034058 for ; Thu, 28 Jun 2018 13:25:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 661F17BC07 for ; Thu, 28 Jun 2018 13:25:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 990AA17F46 for ; Thu, 28 Jun 2018 13:25:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5SDP17Y078418 for ; Thu, 28 Jun 2018 13:25:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5SDP120078417 for virtualization@FreeBSD.org; Thu, 28 Jun 2018 13:25:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 222996] FreeBSD 11.1 on Hyper-V with PCI Express Pass Through Date: Thu, 28 Jun 2018 13:25:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dmitry_kuleshov@ukr.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 13:25:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D222996 --- Comment #9 from Dmitry --- Created attachment 194707 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D194707&action= =3Dedit pciconf -lbv --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jun 28 13:37:01 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8A021034AB6 for ; Thu, 28 Jun 2018 13:37:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4C5DA7C330 for ; Thu, 28 Jun 2018 13:37:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0D12D1034AB4; Thu, 28 Jun 2018 13:37:00 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC3D21034AB3 for ; Thu, 28 Jun 2018 13:36:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 756CE7C32B for ; Thu, 28 Jun 2018 13:36:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B420E180A9 for ; Thu, 28 Jun 2018 13:36:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5SDawnD000351 for ; Thu, 28 Jun 2018 13:36:58 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5SDawrX000350 for virtualization@FreeBSD.org; Thu, 28 Jun 2018 13:36:58 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 225791] ena driver causing kernel panics on AWS EC2 Date: Thu, 28 Jun 2018 13:36:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: richard@primarysite.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 13:37:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D225791 --- Comment #6 from Richard Paul --- Okay, with a bit of effort and this is testing against the v0.7.0 ENA drive= r on FreeBSD 11.1p11 I got a reboot. But this was not easy to get to reproduce. Here's what I did to get this to die.....eventually (it took about 3 hours) * Downsize the instance to an m5.large * I installed nginx on the DB server clone and started it (I'll detail con= fig below) * I installed the phoronix-test-suite and pts/blogbench * I kicked off a backup of our 115GB database to a local (800GB EBS vol.) = ZFS partion (the DB is held on a mirrored ZFS set on another pair of 250GB EBS vol's) * I kicked off a stress run of the phoronix blogbench * With two t2.medium instances in the same VPC, I ran wrk -d 12h -c 2k -t2 http://10.0.0.10/ Additional configuration etc.: ---------------------------------------------------------------------------= ------- root@os-upgrade-test-db:~ # setenv PTS_CONCURRENT_TEST_RUNS 8=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 root@os-upgrade-test-db:~ # setenv TOTAL_LOOP_TIME 30 root@os-upgrade-test-db:~ # phoronix-test-suite stress-run pts/blogbench Choose Option 3 for Test All Options ---------------------------------------------------------------------------= ------- ---------------------------------------------------------------------------= ------- pkg info nginx-full=20=20=20=20=20 nginx-full-1.12.2_11,2=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20 Name : nginx-full=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20 Version : 1.12.2_11,2=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20 Installed on : Thu Jun 28 09:37:29 2018 UTC ___nginx.conf___ worker_processes auto;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 events {=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 worker_connections 2048;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20 }=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20 http {=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 include mime.types;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20 default_type application/octet-stream;=20=20=20=20=20=20=20=20=20 sendfile on;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20 tcp_nopush on;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20 keepalive_timeout 65;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20 gzip on;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 server {=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 listen 80;=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 server_name localhost;=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20 location / {=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20 root /usr/local/www/nginx;=20=20=20=20=20=20=20=20=20=20=20=20 index index.html index.htm;=20=20=20=20=20=20=20=20=20=20=20=20 }=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 location =3D /50x.html {=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20 root /usr/local/www/nginx-dist;=20=20=20=20=20=20=20 }=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 }=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 } ---------------------------------------------------------------------------= ------- ---------------------------------------------------------------------------= ------- This is just to give you an idea of what we're doing, the backup is actuall= y a whole backups script file for doing this for each database in the RDBMS and they rsyncing to to the offsite server __Postgres dump__ sudo -u pgsql pg_dump -j 16 -Fd dbname -f /var/backups/outfile ---------------------------------------------------------------------------= ------- --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jun 28 13:42:05 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFF991035117 for ; Thu, 28 Jun 2018 13:42:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 355BE7C7B9 for ; Thu, 28 Jun 2018 13:42:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id EA4A11035116; Thu, 28 Jun 2018 13:42:04 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ACB881035115 for ; Thu, 28 Jun 2018 13:42:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48FEA7C7AF for ; Thu, 28 Jun 2018 13:42:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8490C181F0 for ; Thu, 28 Jun 2018 13:42:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5SDg32W013053 for ; Thu, 28 Jun 2018 13:42:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5SDg3lh013049 for virtualization@FreeBSD.org; Thu, 28 Jun 2018 13:42:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 225791] ena driver causing kernel panics on AWS EC2 Date: Thu, 28 Jun 2018 13:42:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: richard@primarysite.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 13:42:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D225791 --- Comment #7 from Richard Paul --- I missed a couple of configuration options we'd set: ---------------------------------------------------------------------------= --- sysctl: kern.ipc.soacceptqueue: 8192 ---------------------------------------------------------------------------= --- ---------------------------------------------------------------------------= --- postgresql.conf max_connections =3D 120=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20 shared_buffers =3D 2GB=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 effective_cache_size =3D 6GB=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20 checkpoint_completion_target =3D 0.9=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20 checkpoint_timeout =3D 1h=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20 work_mem =3D 2MB=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 maintenance_work_mem =3D 256MB=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20 max_locks_per_transaction =3D 128=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20 random_page_cost =3D 1.1=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20 max_worker_processes =3D 2=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20 ---------------------------------------------------------------------------= --- I've changed the instance type to an r4.large and I'll tweak some of the postgresql settings for the additional memory on the r4 instance and I'll s= et it off again to attempt getting it to fall over. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jun 28 14:04:24 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0B4610006B9 for ; Thu, 28 Jun 2018 14:04:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 687347D658 for ; Thu, 28 Jun 2018 14:04:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 2612A10006B6; Thu, 28 Jun 2018 14:04:24 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1333B10006B4 for ; Thu, 28 Jun 2018 14:04:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 946EE7D654 for ; Thu, 28 Jun 2018 14:04:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id AB233184DE for ; Thu, 28 Jun 2018 14:04:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w5SE4M8J078359 for ; Thu, 28 Jun 2018 14:04:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w5SE4M6k078358 for virtualization@FreeBSD.org; Thu, 28 Jun 2018 14:04:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 225791] ena driver causing kernel panics on AWS EC2 Date: Thu, 28 Jun 2018 14:04:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: richard@primarysite.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 14:04:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D225791 --- Comment #8 from Richard Paul --- Just to help you out with pts ----------------------------- To run the benchmark in pts you need to install it using the phoronix-test-suite package: #~ pkg install phoronix-test-suite #~ phoronix-test-suite install pts/blogbench --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jun 29 08:59:03 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47C5B1035237; Fri, 29 Jun 2018 08:59:03 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C4E3782BC8; Fri, 29 Jun 2018 08:59:02 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: by mail-oi0-x22c.google.com with SMTP id w126-v6so517819oie.7; Fri, 29 Jun 2018 01:59:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=btaGFOIewTKa9yTmziEpM/4891mrOI3TGYGhUw7YPEs=; b=WHPN8I0fDlxS+IrbEMfuTdylWED+yGUkqXSH9Q8veuJyEG4go7sO6YrCWmslhnEkqA Sg6P3s+3GiokpDLwVUxoZtNjgfL9Ci6hWTTvAERjbH4LBiMNzj0kkK7+1tqyd6GNMW/J niNNdCicublQ68y11VVRvRwPP3QvwBHiTMIdUfUM+ZNz5VThyof/9ZGijkTPNXJLJFch gf6QlhHtYgwXpoykqhn1fjMOu/T0TDvde2JOKdWVSnSB4PqK55vd3i2YSQO0g7gMP3NZ GBZr+9fXUxrdG6ZV30tIgcqqEcUY1+57pD1KOV5oQ8Gw7yAvRbU3SknmbWquNqJKOu6N Xerw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=btaGFOIewTKa9yTmziEpM/4891mrOI3TGYGhUw7YPEs=; b=oqgXXdLKDcb+0X1srQCgpltpTOgm/l738cC5b5Fdx3xt1Q8khFHmciKZYFigtsH0D3 VdNhEYzH2ZvE7capCO4DaxLS1j6OKTuuSSvPcjCcFgxU5YMShdZMUqWSflFZrfXY+P9p p0pXTdgYSzBt3TjVSU7edb5ryYys0bHZANqgnLDWsn3WHoBwyDJBq9Bw10uRcriujO5P rPCDvVvs4a8n0QB3A43Xh/MhKP8EhmevNyATlKSrv3jbcsoKofwMaH4bgH6JGACPQkqF 9NFr60plEXBVigu5/09fPt4xhmioY06Qh30VJHMAC8wWHflKGk3mxPPp266uMgn47ixZ RfEQ== X-Gm-Message-State: APt69E0ZvFRZxJ8zqQdlGUiqCI0CBl1FsYkGfzV7/bHGCiW7h4Sj1nK0 FKcH4weaLz6uvgxH/STX/zFppYt5ABLhuc38qgVgyQ== X-Google-Smtp-Source: AAOMgpeNkLg+B1jKTyyHZE6S0vKqsuyf606meQWvDo/5xH6FyrFf3wefAfpAIyyTnTf8UUjAnCiVAM3nUPu1xJkVbjw= X-Received: by 2002:aca:f409:: with SMTP id s9-v6mr8162984oih.102.1530262741744; Fri, 29 Jun 2018 01:59:01 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a9d:3c4e:0:0:0:0:0 with HTTP; Fri, 29 Jun 2018 01:58:31 -0700 (PDT) From: Elena Mihailescu Date: Fri, 29 Jun 2018 11:58:31 +0300 Message-ID: Subject: Inspect pages created after a vm_object is marked as copy-on-write To: freebsd-amd64@freebsd.org, freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jun 2018 08:59:03 -0000 Hello, I am interested if there is a method to inspect what pages/objects were created after a vm_object (the vm_map_entry associated with the object) is marked as copy-on-write. More specifically, I'm interested only in the pages that were copied when a write operation was proceed on a page that belongs to the object marked copy-on-write. I need this for a live migration feature for bhyve in order to send the pages that were modified between the iterations in which I migrate the guest's memory(the guest's memory will be migrated in rounds - firstly, all memory will be sent remote, then, only the pages that were modified and so on). What I want to implement is the following: Step 1: Given a vm_object *obj, mark its associated vm_map_entry *entry as copy-on-write. Step 2: After a while (a non-deterministic amount of time), inspect/retrieve the pages that were created based on information existent in the object. What I tried until now: I implemented a function in kernel that: - gets the vmspace structure pointer for the current process - gets the vm_map structure pointer for the vmspace - iterates through each vm_map_entry and based on the vm_offset_start and vm_offset_end determines vm_map_entry that contains the object I am interested in. - for this object, it prints some debug information such as: shadow_count, ref_count, whether if it has a backing_object or not. The code written is similar with the code from here (the way in which I get vmspace for the current process and the way I am iterating through vm_map_entry and objects): [0] https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L979 I have read the following documentation about FreeBSD's implementation for virtual memory: [1] https://www.freebsd.org/doc/en/books/arch-handbook/vm.html [2] https://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html [3] https://people.freebsd.org/~neel/bhyve/bhyve_nested_paging.pdf [4] http://www.cse.chalmers.se/edu/year/2011/course/EDA203/unix4.pdf As far as I could tell after reading the documentation presented above, I should look for the object that the object I am interested in is a shadow of or an object that my object is shadow for. To do that, I should inspect the following fields from the vm_object structure (among others)( https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_object.h#L98) : - int shadow_count; /* how many objects that this is a shadow for */ - struct vm_object *backing_object; /* object that I'm a shadow of */ But in all my tests, for the object I am interested in, the shadow_count is 0 and the backing_object is NULL. The code I use to mark the vm_map_entry for the object I am interested in copy-on-write is here: [5] https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L949 Is there anything I am doing wrong? Maybe I misunderstood something about the way the virtual memory works in FreeBSD. There is another way I could inspect what pages were created between the moment I mark an object (its vm_map_entry) as copy-on-write and a later moment? Thank you, Elena From owner-freebsd-virtualization@freebsd.org Fri Jun 29 22:52:23 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23023FD0671; Fri, 29 Jun 2018 22:52:23 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-wr0-x242.google.com (mail-wr0-x242.google.com [IPv6:2a00:1450:400c:c0c::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 93BA27FFCA; Fri, 29 Jun 2018 22:52:22 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-wr0-x242.google.com with SMTP id c13-v6so10169903wrq.2; Fri, 29 Jun 2018 15:52:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=IWBYpSa/0j+dyi3tR8YyCQOoR9qXzbtjfWSijgB7Muw=; b=WyhXBhwRhpdR0WB5YCOR+2hIFklOMJVJCpj6L/gtinjX6ler5ZRYM+RtYyAJfcaKvm w8BZMcDc5aVexGt0Wznwq9dy4C87Gc7SaDuFKnN5/xiQROPsBmHetixPwF9wlBfOkiD5 1IoJxmiPFI8x7GlaNjinlnZjXRioB9X8Wcij5ZF2Nq6cOJ2Pn6ryN0+YoKR+qNGgfKDJ aX0vX3Lgr8ofPQFzv9i0h6ePt8ia2zOjkg9ZFYZ2qTpR7BXIk0RgFasv5y3rSSEmRCuR rtHOtx9547sxs4mQlcw0Snv173d8H+Gu76dve/z2y4cqWVOft/LbLq3C1vuqqst3rkDs ZPRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=IWBYpSa/0j+dyi3tR8YyCQOoR9qXzbtjfWSijgB7Muw=; b=oguf3b5OhgV90XJ+hclvj2WnHuC22s3VdxkTLZVxfaw9dKTe5gBfJrAcevjjxIs8++ WmeFjmLLi/fF6ee+lJ7DIyN58vonTSLzdSt2kZa6UTC1pF26H5B2Mq0Xt99RLddD5pev jYd5QUhxxG+ELjfSmrPRqmZuNMT+30cAPsaItfvPJKXJ2LVinvgt4+6Rn2jF+l6/GBPM JMiQYjL2GsmKIszl4UgLOQqauOWxmV5YC2Y4pU45ysRbUvx/inYjHuxz5ktTdb7ozxps wlSThtdPArRe1fI6dVriwluvV6LeWuUlSyt3ZUQF6OG+y0DCvtmm82kiHhLr6S6c5q7b 9Fxw== X-Gm-Message-State: APt69E31POEQPICQlr6fp/QJZy1At7Wvz2/bnUJbr4c9c6oC7v6SdG2b 5K0awE3HSPsjelgu7YCgMrA= X-Google-Smtp-Source: AAOMgpco/ZUS2rNZaM1Dyc1WEAw+4l9XkpGVaHzLIuI4hDqMWWmzNMbBkcDPpWL0Nmzi4bdaw1T+xA== X-Received: by 2002:adf:a792:: with SMTP id j18-v6mr13826696wrc.187.1530312741513; Fri, 29 Jun 2018 15:52:21 -0700 (PDT) Received: from pesky.lan (93-34-93-211.ip49.fastwebnet.it. [93.34.93.211]) by smtp.gmail.com with ESMTPSA id v15-v6sm1738168wmc.16.2018.06.29.15.52.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 29 Jun 2018 15:52:20 -0700 (PDT) Sender: Mark Johnston Date: Fri, 29 Jun 2018 18:52:15 -0400 From: Mark Johnston To: Elena Mihailescu Cc: freebsd-amd64@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: Inspect pages created after a vm_object is marked as copy-on-write Message-ID: <20180629225209.GA4238@pesky.lan> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jun 2018 22:52:23 -0000 On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote: > Hello, > > I am interested if there is a method to inspect what pages/objects were > created after a vm_object (the vm_map_entry associated with the object) is > marked as copy-on-write. More specifically, I'm interested only in the > pages that were copied when a write operation was proceed on a page that > belongs to the object marked copy-on-write. > > I need this for a live migration feature for bhyve in order to send the > pages that were modified between the iterations in which I migrate the > guest's memory(the guest's memory will be migrated in rounds - firstly, all > memory will be sent remote, then, only the pages that were modified and so > on). > > What I want to implement is the following: > Step 1: Given a vm_object *obj, mark its associated vm_map_entry *entry as > copy-on-write. > Step 2: After a while (a non-deterministic amount of time), > inspect/retrieve the pages that were created based on information existent > in the object. > > What I tried until now: > > I implemented a function in kernel that: > - gets the vmspace structure pointer for the current process > - gets the vm_map structure pointer for the vmspace > - iterates through each vm_map_entry and based on the vm_offset_start and > vm_offset_end determines vm_map_entry that contains the object I am > interested in. > - for this object, it prints some debug information such as: shadow_count, > ref_count, whether if it has a backing_object or not. > The code written is similar with the code from here (the way in which I get > vmspace for the current process and the way I am iterating through > vm_map_entry and objects): > [0] > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L979 > > I have read the following documentation about FreeBSD's implementation for > virtual memory: > [1] https://www.freebsd.org/doc/en/books/arch-handbook/vm.html > [2] > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html > [3] https://people.freebsd.org/~neel/bhyve/bhyve_nested_paging.pdf > [4] http://www.cse.chalmers.se/edu/year/2011/course/EDA203/unix4.pdf > > As far as I could tell after reading the documentation presented above, I > should look for the object that the object I am interested in is a shadow > of or an object that my object is shadow for. Right. When a copy-on-write fault results in the creation of a new object, the new object is said to shadow the original object, which becomes the backing object for the shadow object. When faults in the corresponding map entry occur, the fault handler first searches for a page in the map entry's object, and then falls back to the backing object if necessary. > To do that, I should inspect the following fields from the vm_object > structure (among others)( > https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_object.h#L98) : > > - int shadow_count; /* how many objects that this is a shadow for */ > - struct vm_object *backing_object; /* object that I'm a shadow of */ > > But in all my tests, for the object I am interested in, the shadow_count is > 0 and the backing_object is NULL. > > The code I use to mark the vm_map_entry for the object I am interested in > copy-on-write is here: > [5] > https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L949 MAP_ENTRY_NEEDS_COPY needs to be set in order for the copy-on-write machinery to work the way you expect. Take a look at vm_map_lookup(): when it sees that flag and the caller is attempting a write fault, it creates a shadow object, updates the entry and clears MAP_ENTRY_NEEDS_COPY, leaving MAP_ENTRY_COW set. > Is there anything I am doing wrong? Maybe I misunderstood something about > the way the virtual memory works in FreeBSD. I'll note that inspecting and manipulating vm_map_entry and vm_object structures in the bhyve code constitutes something of an abstraction violation, though it's reasonable to proceed this way while working on a prototype of the feature. That is, I think you should keep trying your current approach, but just be aware that you are using the copy-on-write mechanism in a way that the VM system isn't really expecting. > There is another way I could inspect what pages were created between the > moment I mark an object (its vm_map_entry) as copy-on-write and a later > moment? From owner-freebsd-virtualization@freebsd.org Sat Jun 30 07:38:25 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 211441021F6C; Sat, 30 Jun 2018 07:38:25 +0000 (UTC) (envelope-from mihai.carabas@gmail.com) Received: from mail-wr0-x241.google.com (mail-wr0-x241.google.com [IPv6:2a00:1450:400c:c0c::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8CE67729CE; Sat, 30 Jun 2018 07:38:24 +0000 (UTC) (envelope-from mihai.carabas@gmail.com) Received: by mail-wr0-x241.google.com with SMTP id k7-v6so7277335wrq.0; Sat, 30 Jun 2018 00:38:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=f7RuluXkpAnQRk9uDHIvnrFH3Y52S1duSgcGn8WSKvY=; b=QzAU0leOPgqojohnP1pnNmUJImprwTvSYdYoHXmxOfrC/aKjZ/afdiciT4NnbpUgrZ FC9extIPxh3QmX/m+E2lPJGXfhYZ1JV5A9y/iqByLSjIjco39UwosPuiodmhxyH90Ee2 ZzudXiNIsc0Rf00E8aGLdQJHnhB/Uzt5vTH/fPrFTjnfN+jhAULopniFq5ZrwUhFNhx6 i22xdp5hf7+B7MbNqdd6Ls78odxuAJ/5B9ZwpTYd3xbTQnA1fuOsGZfV6wFGrlqMRvqj uVjIzhFC0ZpNhtvdZhMP56uZwKpNFeZ8xufWEch1dKGdchr+9Ektkl2O56fhu6XOZxp/ q67w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=f7RuluXkpAnQRk9uDHIvnrFH3Y52S1duSgcGn8WSKvY=; b=WmEN+KTLg9sBJPzniOYXDlfbf/cy8l6fuawoRDqJjCSgBe5NZ43O7388Z+XJX6xKeT 2Hh0lBzhgXt6cclHqgeHCS3Q2hWINOIKTS9ayygL/XeRiB0Z/EZdbcZJwqvikbyYLGEY wUPY4vsjnHAV9MB6GHzfyZmCgmY/nlor2yWQF/PZJ0Y3N5dRbY2e2uF25xOH2FCe1ApE Be59iWOLw0IRA4/09aVIJMWryhLXRZvRkCYcoPBInHRNAw0jUr3bkZUTxwbX0SJ4OzcA EcHtufbNt1zygNiokXQCK+ggOlmE2SrhL/VQf5n+1pWwi5hPPqrhGmGocenYMDnIlCLg pN+w== X-Gm-Message-State: APt69E3KmrhT3b7K6JSokNSkN3Rjmc3njeztfTRz1Tx7twpcdg2+DgKT PefgJ8+ISIkc/tRq6gXK5wCZWQNi3c481diILi0= X-Google-Smtp-Source: AAOMgpfOf6ZJajqhM11CtCaDP2nUDfB1lWXOp+VU6x/jXOcrXCiDauEsLFBLR9odbGtJh/K+uBZKW9BC0hofd9QzrGI= X-Received: by 2002:a5d:4a09:: with SMTP id m9-v6mr9071047wrq.91.1530344302913; Sat, 30 Jun 2018 00:38:22 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a1c:749:0:0:0:0:0 with HTTP; Sat, 30 Jun 2018 00:38:21 -0700 (PDT) In-Reply-To: <20180629225209.GA4238@pesky.lan> References: <20180629225209.GA4238@pesky.lan> From: Mihai Carabas Date: Sat, 30 Jun 2018 10:38:21 +0300 Message-ID: Subject: Re: Inspect pages created after a vm_object is marked as copy-on-write To: Mark Johnston Cc: Elena Mihailescu , freebsd-virtualization@freebsd.org, freebsd-amd64@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 07:38:25 -0000 On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston wrote: > On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote: >> Hello, >> >> I am interested if there is a method to inspect what pages/objects were >> created after a vm_object (the vm_map_entry associated with the object) is >> marked as copy-on-write. More specifically, I'm interested only in the >> pages that were copied when a write operation was proceed on a page that >> belongs to the object marked copy-on-write. >> >> I need this for a live migration feature for bhyve in order to send the >> pages that were modified between the iterations in which I migrate the >> guest's memory(the guest's memory will be migrated in rounds - firstly, all >> memory will be sent remote, then, only the pages that were modified and so >> on). >> >> What I want to implement is the following: >> Step 1: Given a vm_object *obj, mark its associated vm_map_entry *entry as >> copy-on-write. >> Step 2: After a while (a non-deterministic amount of time), >> inspect/retrieve the pages that were created based on information existent >> in the object. >> >> What I tried until now: >> >> I implemented a function in kernel that: >> - gets the vmspace structure pointer for the current process >> - gets the vm_map structure pointer for the vmspace >> - iterates through each vm_map_entry and based on the vm_offset_start and >> vm_offset_end determines vm_map_entry that contains the object I am >> interested in. >> - for this object, it prints some debug information such as: shadow_count, >> ref_count, whether if it has a backing_object or not. >> The code written is similar with the code from here (the way in which I get >> vmspace for the current process and the way I am iterating through >> vm_map_entry and objects): >> [0] >> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L979 >> >> I have read the following documentation about FreeBSD's implementation for >> virtual memory: >> [1] https://www.freebsd.org/doc/en/books/arch-handbook/vm.html >> [2] >> https://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/article.html >> [3] https://people.freebsd.org/~neel/bhyve/bhyve_nested_paging.pdf >> [4] http://www.cse.chalmers.se/edu/year/2011/course/EDA203/unix4.pdf >> >> As far as I could tell after reading the documentation presented above, I >> should look for the object that the object I am interested in is a shadow >> of or an object that my object is shadow for. > > Right. When a copy-on-write fault results in the creation of a new > object, the new object is said to shadow the original object, which > becomes the backing object for the shadow object. When faults in the > corresponding map entry occur, the fault handler first searches for a > page in the map entry's object, and then falls back to the backing > object if necessary. > >> To do that, I should inspect the following fields from the vm_object >> structure (among others)( >> https://github.com/freebsd/freebsd/blob/master/sys/vm/vm_object.h#L98) : >> >> - int shadow_count; /* how many objects that this is a shadow for */ >> - struct vm_object *backing_object; /* object that I'm a shadow of */ >> >> But in all my tests, for the object I am interested in, the shadow_count is >> 0 and the backing_object is NULL. >> >> The code I use to mark the vm_map_entry for the object I am interested in >> copy-on-write is here: >> [5] >> https://github.com/FreeBSD-UPB/freebsd/blob/projects/bhyve_migration/sys/amd64/vmm/vmm_dev.c#L949 > > MAP_ENTRY_NEEDS_COPY needs to be set in order for the copy-on-write > machinery to work the way you expect. Take a look at vm_map_lookup(): > when it sees that flag and the caller is attempting a write fault, it > creates a shadow object, updates the entry and clears > MAP_ENTRY_NEEDS_COPY, leaving MAP_ENTRY_COW set. > >> Is there anything I am doing wrong? Maybe I misunderstood something about >> the way the virtual memory works in FreeBSD. > > I'll note that inspecting and manipulating vm_map_entry and vm_object > structures in the bhyve code constitutes something of an abstraction > violation, though it's reasonable to proceed this way while working on a > prototype of the feature. That is, I think you should keep trying your > current approach, but just be aware that you are using the copy-on-write > mechanism in a way that the VM system isn't really expecting. > Can you point out the right approach in our case? Thanks, Mihai >> There is another way I could inspect what pages were created between the >> moment I mark an object (its vm_map_entry) as copy-on-write and a later >> moment? > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@freebsd.org Sat Jun 30 11:41:06 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8DB25102BC74 for ; Sat, 30 Jun 2018 11:41:06 +0000 (UTC) (envelope-from pestalent1@yahoo.com) Received: from sonic313-19.consmr.mail.gq1.yahoo.com (sonic313-19.consmr.mail.gq1.yahoo.com [98.137.65.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 024A77A3A7 for ; Sat, 30 Jun 2018 11:41:05 +0000 (UTC) (envelope-from pestalent1@yahoo.com) X-YMail-OSG: 6Tbh7m4VM1msuTVa6lnld2yxEwz4.Pv_HKh9Umlp89F9SssKGD31lt_djeixlwv jsa.OXei07jk3GiggkS4CpKVgbaE92ba9Ij2Ky80BCYVm9sTcoOVtNi0bjK0f6ut0czLzghU.ZNj vFaPJFwhvmYrvvXYMXqRAjfMSJ.2acsxpo_DsZtE_6_w_lq0e0rFhmaTLOIJjY2IrSI_4FWwijYr pW04D5srX2UQ_Bg_XBOAeLfsilgjBngW70meVkRubT5n2Nafv_QwOWhrOBihQmjzO.DAvJ9UxaWh 8yUvsofYjU0YiwHDvyaTpN0ZyYBskudb5azF2bV1f2vu5NsnJJPA5HY.8SQPC4SZbeI6Dx8A.2a_ xNYBcm1A2jPR976kbUk9K.Pl.bkjxA5OlfSxlOv5sY8uiKAc8BMMu6_tUlFyCKzuMVc1yqpkEgnp NhE1kYT.n1tkZf5oqw1SCl8d8rcF.6J.B9H2KiZNJQoeHUZAR7TkdknTJpHUm7_Seheq4BuHo_Cy Yf3wsj5rQkkpC.dJUMhD5xP5PK._H2bIW3QEDrZGvrGEU7wYwuZwmcMcCNbrGvuXQjgGD3oFJ3lk VxbfDeTLbvuwPA3n_nCEKzv2soOGj8yG2Sd3oki8cRy2rfEkRNV5hMoaM4YWQfn4L8p0VBb8hIes V8Yen4ewIZw_V6qSpmVDHO5_Kpec5dkmRFwF778rn60dlhVQpD7oNsJHTu28qM7veAJ3IuLz_mE9 cmE86d1bJIGnUnw_7JzjHzfr58NHIcDUF1QwngWRj5EdAql1cuenBuQn32hvRJ7Ifc21ZOkGTqxw oFhmVA7TmwQroaj4WR5PQYGhris3zjOcWpg5fZZxxl.WeyCXEedYxL_b7ClrOh7_weOEuYKbcSeG DZ1CRADsMPzoqvxYCbh9JkRBg50wxaRFC_pZoyiXXglrydyoqGtUjkYOpdg-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sat, 30 Jun 2018 11:40:59 +0000 Received: from c-24-91-1-212.hsd1.vt.comcast.net (EHLO [192.168.1.25]) ([24.91.1.212]) by smtp414.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 76b0142a03ef42aaee1510acd13b52af for ; Sat, 30 Jun 2018 11:40:53 +0000 (UTC) Subject: AMD Ryzen Byhyve "sysctl: unknown oid "hw.vmm.amdvi"" To: freebsd-virtualization@freebsd.org References: From: pestalent1 Message-ID: Date: Sat, 30 Jun 2018 07:40:56 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 11:41:06 -0000 Hello, I have bhyve up and running on my "FreeBSD 11.2/bhyve 1.1.8_2  running kde5 desktop with: "ryzen 7 1700 (overclocked to 3700Mhz, if that matters) 32 GB of ram on a MSI x370 Gaming Plus motherboard with the latest bios". I have a windows 2012r2 vm (config below) up and running, it has 4 of my physical hard-drives passed through, all is working. I am trying to passthrough the below USB 3 controller. Not sure if it matters, the IOMMU was set to "auto" in the bios/uefi when I installed the OS. It is now set to "enable" It does not work the Vm fails to boot. Do I have something mis-configured or I am missing something. I hope I have provided all the necessary info for assistance. |ppt0@pci0:40:0:0: class=0x0c0330 card=0x7a331462 chip=0x21421b21 rev=0x00 hdr=0x00 vendor = 'ASMedia Technology Inc.' class = serial bus subclass = USB cap 05[50] = MSI supports 8 messages, 64 bit cap 01[78] = powerspec 3 supports D0 D3 current D0 cap 10[80] = PCI-Express 2 legacy endpoint max data 128(512) RO NS link x2(x2) speed 8.0(8.0) ASPM disabled(L0s/L1) ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected ecap 0019[200] = PCIe Sec 1 lane errors 0 ecap 0018[300] = LTR 1 |  My loader.conf |kern.vty=vt zfs_load="YES" fuse_load="YES" kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" vmm_load="YES" nmdm_load="YES" if_bridge_load="YES" if_tap_load="YES" hw.syscons.disable="1" hw.vmm.amdvi.enable="1" hw.vmm.topology.cores_per_package=4 #hw.vmm.topology.threads_per_core=2 pptdevs="40/0/0"| My VM config file: |uefi="yes" cpu=4 memory=6G graphics="yes" xhci_mouse="yes" graphics_res="1600x900" network0_type="virtio-net" network0_switch="public" #ahci_device_limit="6" debug="yes" disk0_opts="sectorsize=512" disk0_type="ahci-hd" disk0_name="BackupR7.img" disk1_dev="custom" disk1_opts="sectorsize=512" disk1_type="ahci-hd" disk1_name="/dev/ada0,hd:/dev/ada1" disk2_dev="custom" disk2_opts="sectorsize=512" disk2_type="ahci-hd" disk2_name="/dev/ada2,hd:/dev/ada3" uuid="762f78e9-270b-11e8-87a4-309c230c9abc" network0_mac="02:15:5d:45:32:01" passthru0="40/0/0"| |acpidump -t | grep DMAR|shows nothing, I have IOMMU enabled in bios or uefi. sysctl hw.vmm.amdvi  gives me "sysctl: unknown oid "hw.vmm.amdvi"" last line in my vm-bhyve.log : fatal;  pci passthrough not supported on this system (no VT-d)  dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' pci0: at device 0.2 (no driver attached) pci0: at device 0.2 (no driver attached) pci0: at device 0.2 (no driver attached) amd_iommu_init: not implemented I would appreciate any and all help Thanks, Pete K From owner-freebsd-virtualization@freebsd.org Sat Jun 30 11:48:54 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28FB6102C046 for ; Sat, 30 Jun 2018 11:48:54 +0000 (UTC) (envelope-from dariusmihaim@gmail.com) Received: from mail-qt0-x235.google.com (mail-qt0-x235.google.com [IPv6:2607:f8b0:400d:c0d::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AB36A7A6C4 for ; Sat, 30 Jun 2018 11:48:53 +0000 (UTC) (envelope-from dariusmihaim@gmail.com) Received: by mail-qt0-x235.google.com with SMTP id b15-v6so212700qtp.11 for ; Sat, 30 Jun 2018 04:48:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sof/TJZ02y0zKohdrcGnhlvI92LE94Z2l2+dSF6xtdw=; b=koP5136W/VqGeqDTVgN8OoevLFTjgVCm0LTfiSEQ5EWaNC5f8Pqdszdvi5P0m6acV2 GPCihjbHnm5+wG5VUfVSnN420eioPibQub+i07KxdAEKuha8i9YNWWD3NzKMme4QEpDg 2Z8bMRrUDpV558sUP0v7VxDBpdwmhvBvClprgumPXa0eX8TzfyoJ8BGxeuc1QV/ENM5c bswj29QVAVpax24qoNKW58e/Z4dJpQaE1vQOo3nUBqn9MIqcweAje4MUA9HmYo0i328C wF/l6/v81q6iFzGLOT0ksO1Q7UZr3y5/LCsfqNeXPMmjhO74sib+LCFYVjaI2R32s8IO QDEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sof/TJZ02y0zKohdrcGnhlvI92LE94Z2l2+dSF6xtdw=; b=kTNxtyV8Jnkwche0cwCCTBLg3z71iRFaUte1s/3KWVnq3gKuyoKMJbUxBXQjTH2pG/ lCHShE0vjgD4QIlRVUyT8XpEqBVPxRVby6Lxdffsjip/6C5gFteEkK3fV3Yrgi9hQWKh 1sDjo7rKww4toNY7TW3wPtVwoVDYnjb4m1JaQ3bWIO4aEkd9rijSEIzhaGbNPktmLLiO hHBIORYNd7nQu1Nox4xeOqsdj2nodPAb3PuFPje079K2p9/uaekefWm4gjW6C+ehOOP/ bBVoXZoFS025Vd6GZ/d3KcBOj+efzom3LG6vicGwx79aotPk9P1pQmwvhlvceTJLx1/7 DN+A== X-Gm-Message-State: APt69E298sZmA+9LrenNslu/dAqwmvrcVxjP5xuy2KZxzCnrVqEtVkfy isq+CVFX/6GRGnGSAueg3q6NR4CEIihcKkKrVXo= X-Google-Smtp-Source: AAOMgpeHkSRoHsUINGZ2KvLEzA3q3So+Pdt0uUWfWG4MPUex7kgA4hB1O3pp6fkQndkbsFqdY8lbJ7CSh6nh6EZWPB4= X-Received: by 2002:aed:220a:: with SMTP id n10-v6mr4495293qtc.18.1530359333223; Sat, 30 Jun 2018 04:48:53 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Darius Mihai Date: Sat, 30 Jun 2018 14:48:15 +0300 Message-ID: Subject: Re: AMD Ryzen Byhyve "sysctl: unknown oid "hw.vmm.amdvi"" To: pestalent1@yahoo.com Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 11:48:54 -0000 On Sat, Jun 30, 2018 at 2:42 PM pestalent1 via freebsd-virtualization wrote: > > > Hello, > I have bhyve up and running on my "FreeBSD 11.2/bhyve 1.1.8_2 running > kde5 desktop with: "ryzen 7 1700 (overclocked to 3700Mhz, if that > matters) 32 GB of ram on a MSI x370 Gaming Plus motherboard with the > latest bios". I have a windows 2012r2 vm (config below) up and running, > it has 4 of my physical hard-drives passed through, all is working. I am > trying to passthrough the below USB 3 controller. Not sure if it > matters, the IOMMU was set to "auto" in the bios/uefi when I installed > the OS. It is now set to "enable" > > It does not work the Vm fails to boot. Do I have something > mis-configured or I am missing something. I hope I have provided all the > necessary info for assistance. > > |ppt0@pci0:40:0:0: class=0x0c0330 card=0x7a331462 chip=0x21421b21 > rev=0x00 hdr=0x00 vendor = 'ASMedia Technology Inc.' class = serial bus > subclass = USB cap 05[50] = MSI supports 8 messages, 64 bit cap 01[78] = > powerspec 3 supports D0 D3 current D0 cap 10[80] = PCI-Express 2 legacy > endpoint max data 128(512) RO NS link x2(x2) speed 8.0(8.0) ASPM > disabled(L0s/L1) ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected > ecap 0019[200] = PCIe Sec 1 lane errors 0 ecap 0018[300] = LTR 1 | > > My loader.conf > > |kern.vty=vt zfs_load="YES" fuse_load="YES" > kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" > vmm_load="YES" nmdm_load="YES" if_bridge_load="YES" if_tap_load="YES" > hw.syscons.disable="1" hw.vmm.amdvi.enable="1" > hw.vmm.topology.cores_per_package=4 #hw.vmm.topology.threads_per_core=2 > pptdevs="40/0/0"| > > > > My VM config file: > > |uefi="yes" cpu=4 memory=6G graphics="yes" xhci_mouse="yes" > graphics_res="1600x900" network0_type="virtio-net" > network0_switch="public" #ahci_device_limit="6" debug="yes" > disk0_opts="sectorsize=512" disk0_type="ahci-hd" > disk0_name="BackupR7.img" disk1_dev="custom" disk1_opts="sectorsize=512" > disk1_type="ahci-hd" disk1_name="/dev/ada0,hd:/dev/ada1" > disk2_dev="custom" disk2_opts="sectorsize=512" disk2_type="ahci-hd" > disk2_name="/dev/ada2,hd:/dev/ada3" > uuid="762f78e9-270b-11e8-87a4-309c230c9abc" > network0_mac="02:15:5d:45:32:01" passthru0="40/0/0"| > > > |acpidump -t | grep DMAR|shows nothing, I have IOMMU enabled in bios or > uefi. > > sysctl hw.vmm.amdvi gives me "sysctl: unknown oid "hw.vmm.amdvi"" > > last line in my vm-bhyve.log : fatal; pci passthrough not supported on > this system (no VT-d) > > dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' > pci0: at device 0.2 (no driver attached) > pci0: at device 0.2 (no driver attached) > pci0: at device 0.2 (no driver attached) > amd_iommu_init: not implemented > > I would appreciate any and all help > Thanks, > Pete K > > Hi, The first thing that came to mind was that Ryzen processors have virtualization disabled by default. I remember having to go and enable the virtualization capabilities in BIOS when I first created a virtual machine (admittedly, it was using VMWare so I'm not very sure on bhyve error messages). Darius > > > > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@freebsd.org Sat Jun 30 15:07:27 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 037ED1033B03 for ; Sat, 30 Jun 2018 15:07:27 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (hel.cgi.cz [178.238.36.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C03481136 for ; Sat, 30 Jun 2018 15:07:26 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (localhost [127.0.0.1]) by hel.cgi.cz (Postfix) with ESMTP id C099A112D0A for ; Sat, 30 Jun 2018 17:07:16 +0200 (CEST) X-Virus-Scanned: amavisd-new at cgi.cz Received: from hel.cgi.cz ([127.0.0.1]) by hel.cgi.cz (hel.cgi.cz [127.0.0.1]) (amavisd-new, port 10024) with LMTP id gVX17kXWh9Ii for ; Sat, 30 Jun 2018 17:07:14 +0200 (CEST) Received: from mail2.cgi.cz (hermes [172.17.174.1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by hel.cgi.cz (Postfix) with ESMTPS id 4BD0A112CDA for ; Sat, 30 Jun 2018 17:07:14 +0200 (CEST) Received: from [192.168.8.152] (unknown [82.100.31.11]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail2.cgi.cz (Postfix) with ESMTPSA id D7DE236F2D for ; Sat, 30 Jun 2018 17:07:13 +0200 (CEST) To: freebsd-virtualization@freebsd.org From: Jakub Chromy Subject: Possible bug: 11.2-RELEASE guest with vtnet and PF Message-ID: <848b6851-89fb-b6c8-b412-d5ed897f63d2@cgi.cz> Date: Sat, 30 Jun 2018 17:07:09 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Language: cs Content-Type: text/plain; charset=iso-8859-2; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 15:07:27 -0000 Hello guys, I've just installed a 11.2-RELEASE guest under bhyve (hypervisor is 11.1-RELEASE)... and I cant use Virtio network interface with PF: odine:/boot/kernel# /sbin/pfctl -n -f ~/local/tmp/pf.work *pfctl: pfi_get_ifaces: Bad file descriptor* the file contains the following single line only: pass out quick on vtnet0 proto tcp from any to any keep state these are my interfaces: vtnet0: flags=8943 metric 0 mtu 1500         options=80028         ether 58:9c:fc:04:aa:91         hwaddr 58:9c:fc:04:aa:91         inet *.*.206.214 netmask 0xffffffe0 broadcast *.*.206.223         nd6 options=29         media: Ethernet 10Gbase-T         status: active vtnet1: flags=8902 metric 0 mtu 1500         options=80028         ether 58:9c:fc:08:e0:c7         hwaddr 58:9c:fc:08:e0:c7         nd6 options=29         media: Ethernet 10Gbase-T         status: active lo0: flags=8049 metric 0 mtu 16384         options=600003         inet6 ::1 prefixlen 128         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3         inet 127.0.0.1 netmask 0xff000000         nd6 options=21         groups: lo I haven't noticed any other problems with vtnet0.... it is routing properly, everything works... minus Packet Filter. -- regards Jakub Chromy CGI Systems div. ---------------- CGI CZ s.r.o. sales@cgi.cz 775 144 257 234 697 102 www.cgi.cz From owner-freebsd-virtualization@freebsd.org Sat Jun 30 15:32:10 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E86B610352A0 for ; Sat, 30 Jun 2018 15:32:09 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (hel.cgi.cz [178.238.36.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 545D8821F8 for ; Sat, 30 Jun 2018 15:32:09 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (localhost [127.0.0.1]) by hel.cgi.cz (Postfix) with ESMTP id CC2F311283F for ; Sat, 30 Jun 2018 17:32:07 +0200 (CEST) X-Virus-Scanned: amavisd-new at cgi.cz Received: from hel.cgi.cz ([127.0.0.1]) by hel.cgi.cz (hel.cgi.cz [127.0.0.1]) (amavisd-new, port 10024) with LMTP id xTsPvjnMfu-Z for ; Sat, 30 Jun 2018 17:32:05 +0200 (CEST) Received: from mail2.cgi.cz (hermes [172.17.174.1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by hel.cgi.cz (Postfix) with ESMTPS id A269511281B for ; Sat, 30 Jun 2018 17:32:05 +0200 (CEST) Received: from [192.168.8.152] (unknown [82.100.31.11]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail2.cgi.cz (Postfix) with ESMTPSA id 759F640007 for ; Sat, 30 Jun 2018 17:32:04 +0200 (CEST) Subject: Re: Possible bug: 11.2-RELEASE guest with vtnet and PF From: Jakub Chromy To: freebsd-virtualization@freebsd.org References: <848b6851-89fb-b6c8-b412-d5ed897f63d2@cgi.cz> Message-ID: <305ba086-8a06-3967-87cd-dfbce41deafe@cgi.cz> Date: Sat, 30 Jun 2018 17:31:59 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <848b6851-89fb-b6c8-b412-d5ed897f63d2@cgi.cz> Content-Language: cs Content-Type: text/plain; charset=iso-8859-2; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 15:32:10 -0000 ... problem replicated with e1000 / em driver: pass out quick on *em0* proto tcp from any to any keep state -> pfctl: pfi_get_ifaces: Bad file descriptor vs. odine:~# ifconfig -a em0: flags=8843 metric 0 mtu 1500 options=9b         ether 58:9c:fc:04:aa:91         hwaddr 58:9c:fc:04:aa:91         inet ...netmask 0xffffffe0 broadcast....206.223         nd6 options=29         media: Ethernet autoselect (1000baseT )         status: active vtnet0: flags=8902 metric 0 mtu 1500         options=80028         ether 58:9c:fc:08:e0:c7         hwaddr 58:9c:fc:08:e0:c7         nd6 options=29         media: Ethernet 10Gbase-T         status: active lo0: flags=8049 metric 0 mtu 16384 options=600003         inet6 ::1 prefixlen 128         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3         inet 127.0.0.1 netmask 0xff000000         nd6 options=21         groups: lo -- s pozdravem Jakub Chromy CGI Systems div. ---------------- CGI CZ s.r.o. sales@cgi.cz 775 144 257 234 697 102 www.cgi.cz On 30.06.2018 v 17:07 Jakub Chromy wrote: > Hello guys, > > I've just installed a 11.2-RELEASE guest under bhyve (hypervisor is > 11.1-RELEASE)... and I cant use Virtio network interface with PF: > > odine:/boot/kernel# /sbin/pfctl -n -f ~/local/tmp/pf.work > *pfctl: pfi_get_ifaces: Bad file descriptor* > > the file contains the following single line only: > > pass out quick on vtnet0 proto tcp from any to any keep state > > these are my interfaces: > > vtnet0: flags=8943 > metric 0 mtu 1500 >         options=80028 >         ether 58:9c:fc:04:aa:91 >         hwaddr 58:9c:fc:04:aa:91 >         inet *.*.206.214 netmask 0xffffffe0 broadcast *.*.206.223 >         nd6 options=29 >         media: Ethernet 10Gbase-T >         status: active > vtnet1: flags=8902 metric 0 mtu 1500 >         options=80028 >         ether 58:9c:fc:08:e0:c7 >         hwaddr 58:9c:fc:08:e0:c7 >         nd6 options=29 >         media: Ethernet 10Gbase-T >         status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 >         inet6 ::1 prefixlen 128 >         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 >         inet 127.0.0.1 netmask 0xff000000 >         nd6 options=21 >         groups: lo > > I haven't noticed any other problems with vtnet0.... it is routing > properly, everything works... minus Packet Filter. > > From owner-freebsd-virtualization@freebsd.org Sat Jun 30 22:00:03 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39402FDCEE5; Sat, 30 Jun 2018 22:00:03 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9CC3C8E6C8; Sat, 30 Jun 2018 22:00:02 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id z13-v6so5213706wma.5; Sat, 30 Jun 2018 15:00:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=uv0iM8A3Q5bi8kbUBZ8u4JhxiCCULZru6pX2TLh/qss=; b=dqu00l/JKA0d0MYRsiJzXWV+csykYRRSAfLjW8oN6lxv7s88Lc8BkQCF5kSrVp4kgp 9JEAnD2fLrhuo3yZXhNgoKx+an2YYmS3GScJuv5FQE73VTq+YqSU4cFgmE1gMq1mVVD0 EDKDDofrR9PgpYhR3dVGn8NhewRJE+nghFmosC/MpWGWsXBKuFXfWFXsfTptzP0bOBDQ Tus/MUXRchJXHCWCyr0fMzLkMl8FYlYsD+XS6/pIf5fOEeaC/yTTdvdESLxbjWT5qHMS TIOpbVRaDEH1u60x0DphxyZp2tknF75p4TiKHD2NxDcFbYYA52OaSdDB9LaX9nJFQZJc JxSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=uv0iM8A3Q5bi8kbUBZ8u4JhxiCCULZru6pX2TLh/qss=; b=MEP7SIZkbq1/tsGiuI5RT9ItreBKp7/S0E3EYowhReXyVsed4DGvwduon5r7GgxHa1 pJpKu2cyzp0SojDFGK8TJ5kpb6dU6VP9KfgkPLCZ6yC5I19OpKko3csjulMRMeEqzWRA /bTsLfXjGR6kOePITPQzhStB9OeARqf1t+nx3g2fJ72PU/8zJUbbaqCzdCHxpjM2Q0q7 uvmqQmGrTzT6fyl1+74dVyRdrwNz6C5GBWJRZ49U5WQqXvZQuPpFwYaVCM8msm3MHaxU TNhSmTxoQTyuwrJ5wXv3mnAwLKGmChWpSW1cNadSUg1qQST52BbHJ9Sq7cP8txr8Incy BNHA== X-Gm-Message-State: APt69E3HPB8a/qc4/kltEpTTrh0VDsqCyd85Xmj3TY1EFKcMW9qVE0GL ky/Nm+U03DaDb87f6t7mA37M/g== X-Google-Smtp-Source: AAOMgpeWb7tOWpceP3apEqviLOtZ1BZhvohSEqttxcFDgUMCDvAv7/xdowIQJlfkmgmv1n/Jr/Y4yw== X-Received: by 2002:a1c:a103:: with SMTP id k3-v6mr4955409wme.161.1530396001452; Sat, 30 Jun 2018 15:00:01 -0700 (PDT) Received: from pesky.lan (93-34-93-211.ip49.fastwebnet.it. [93.34.93.211]) by smtp.gmail.com with ESMTPSA id c17-v6sm5814944wrp.54.2018.06.30.14.59.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 30 Jun 2018 15:00:00 -0700 (PDT) Sender: Mark Johnston Date: Sat, 30 Jun 2018 17:59:56 -0400 From: Mark Johnston To: Mihai Carabas Cc: Elena Mihailescu , freebsd-virtualization@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: Inspect pages created after a vm_object is marked as copy-on-write Message-ID: <20180630215956.GA1282@pesky.lan> References: <20180629225209.GA4238@pesky.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.0 (2018-05-17) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 22:00:03 -0000 On Sat, Jun 30, 2018 at 10:38:21AM +0300, Mihai Carabas wrote: > On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston wrote: > > On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote: > >> Is there anything I am doing wrong? Maybe I misunderstood something about > >> the way the virtual memory works in FreeBSD. > > > > I'll note that inspecting and manipulating vm_map_entry and vm_object > > structures in the bhyve code constitutes something of an abstraction > > violation, though it's reasonable to proceed this way while working on a > > prototype of the feature. That is, I think you should keep trying your > > current approach, but just be aware that you are using the copy-on-write > > mechanism in a way that the VM system isn't really expecting. > > > > Can you point out the right approach in our case? I am merely suggesting that once the required VM interactions are fully understood, the mechanism implemented for bhyve should be generalized and lifted into the VM code. It's hard to say what the "right" approach is, since I don't fully understand the proposed algorithm. It sounds like you might be attempting something like: 1. mark the mappings of to-be-migrated objects as NEEDS_COW, so that a subsequent write fault triggers creation of a shadow object 2. invalidate all physical mappings of pages in the object to be copied, so that subsequent writes trigger a fault 3. copy pages from the backing object to the destination 4. copy any pages from the shadow object to the desination 5. collapse the backing object into the shadow 6. if the shadow object exists and was non-empty before the collapse, goto 1 Is that at all accurate? I'm not familiar with the mechanisms used to implement live migration in other hypervisors. From owner-freebsd-virtualization@freebsd.org Sat Jun 30 22:34:13 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F046BFDDD18; Sat, 30 Jun 2018 22:34:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 564428F607; Sat, 30 Jun 2018 22:34:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w5UMY1dx068036 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 1 Jul 2018 01:34:04 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w5UMY1dx068036 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w5UMY1Ph068035; Sun, 1 Jul 2018 01:34:01 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 1 Jul 2018 01:34:01 +0300 From: Konstantin Belousov To: Mark Johnston Cc: Mihai Carabas , freebsd-amd64@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: Inspect pages created after a vm_object is marked as copy-on-write Message-ID: <20180630223401.GW2430@kib.kiev.ua> References: <20180629225209.GA4238@pesky.lan> <20180630215956.GA1282@pesky.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180630215956.GA1282@pesky.lan> User-Agent: Mutt/1.10.0 (2018-05-17) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jun 2018 22:34:13 -0000 On Sat, Jun 30, 2018 at 05:59:56PM -0400, Mark Johnston wrote: > On Sat, Jun 30, 2018 at 10:38:21AM +0300, Mihai Carabas wrote: > > On Sat, Jun 30, 2018 at 1:52 AM, Mark Johnston wrote: > > > On Fri, Jun 29, 2018 at 11:58:31AM +0300, Elena Mihailescu wrote: > > >> Is there anything I am doing wrong? Maybe I misunderstood something about > > >> the way the virtual memory works in FreeBSD. > > > > > > I'll note that inspecting and manipulating vm_map_entry and vm_object > > > structures in the bhyve code constitutes something of an abstraction > > > violation, though it's reasonable to proceed this way while working on a > > > prototype of the feature. That is, I think you should keep trying your > > > current approach, but just be aware that you are using the copy-on-write > > > mechanism in a way that the VM system isn't really expecting. > > > > > > > Can you point out the right approach in our case? > > I am merely suggesting that once the required VM interactions are fully > understood, the mechanism implemented for bhyve should be generalized > and lifted into the VM code. It's hard to say what the "right" approach > is, since I don't fully understand the proposed algorithm. It sounds > like you might be attempting something like: > > 1. mark the mappings of to-be-migrated objects as NEEDS_COW, so that a > subsequent write fault triggers creation of a shadow object It is actually MAP_ENTRY_COW | MAP_ENTRY_NEEDS_COPY. Note that setting an entry to COW changes the behaviour of mprotect(2), at least. > 2. invalidate all physical mappings of pages in the object to be copied, > so that subsequent writes trigger a fault I do not think this is needed to detect writes after the COW is set. It is enough to remove the write permissions. Same as fork() does, see the vm_map_copy_entry() code for the handling of MAP_ENTRY_NEEDS_COPY case. > 3. copy pages from the backing object to the destination As I understand, this is done right after the entry is marked as COW. > 4. copy any pages from the shadow object to the desination And this is done after all backing data is copied and the process is suspended. > 5. collapse the backing object into the shadow > 6. if the shadow object exists and was non-empty before the collapse, > goto 1 Are you trying to describe how to undo the COW marking ? Marking an entry as COW really changes its semantic, and we do not need the undo operation in the base so far. Collapsing the objects would lesser the pressure on the system pollution with objects, but it does not change back the meaning of mappings, e.g. their behaviour on inheritance on fork. > > Is that at all accurate? I'm not familiar with the mechanisms used to > implement live migration in other hypervisors. > _______________________________________________ > freebsd-amd64@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-amd64 > To unsubscribe, send any mail to "freebsd-amd64-unsubscribe@freebsd.org"