From owner-freebsd-virtualization@freebsd.org Sun Mar 8 21:01:10 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B09D82759D5 for ; Sun, 8 Mar 2020 21:01:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48bDKf3twTz4J1g for ; Sun, 8 Mar 2020 21:01:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 841A62759A0; Sun, 8 Mar 2020 21:01:10 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 83A1B27599F for ; Sun, 8 Mar 2020 21:01:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48bDKf1l6yz4J1Q for ; Sun, 8 Mar 2020 21:01:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 19A4627EA6 for ; Sun, 8 Mar 2020 21:01:10 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 028L1935000459 for ; Sun, 8 Mar 2020 21:01:09 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 028L19UP000458 for virtualization@FreeBSD.org; Sun, 8 Mar 2020 21:01:09 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202003082101.028L19UP000458@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 8 Mar 2020 21:01:09 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Mar 2020 21:01:10 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 240945 | [hyper-v] [netvsc] hn network driver incorrectly 1 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Mon Mar 9 08:24:42 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADFF625BE0F for ; Mon, 9 Mar 2020 08:24:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48bWVL3sSYz47Bg for ; Mon, 9 Mar 2020 08:24:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 4813B25BE0E; Mon, 9 Mar 2020 08:24:42 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 45EF925BE0C for ; Mon, 9 Mar 2020 08:24:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48bWVK14xPz4786 for ; Mon, 9 Mar 2020 08:24:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9A6F17FF9 for ; Mon, 9 Mar 2020 08:24:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0298OeGj051676 for ; Mon, 9 Mar 2020 08:24:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0298OeQ5051675 for virtualization@FreeBSD.org; Mon, 9 Mar 2020 08:24:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Mon, 09 Mar 2020 08:24:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2020 08:24:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #36 from mail@rubenvos.com --- Hi, This weekend the issue manifested itself again on one of our 12.1 instances (with an EBS volume attached): Mar 7 03:05:47 zfs01 kernel: nvme1: cpl does not map to outstanding cmd Mar 7 03:05:47 zfs01 kernel: cdw0:00000000 sqhd:001b sqid:0001 cid:001b p:0 sc:00 sct:0 m:0 dnr:0 Mar 7 03:05:47 zfs01 kernel: nvme1: Resetting controller due to a timeout. Mar 7 03:05:47 zfs01 kernel: nvme1: resetting controller Mar 7 03:05:47 zfs01 kernel: nvme1: temperature threshold not supported Mar 7 03:05:47 zfs01 kernel: nvme1: aborting outstanding i/o Mar 7 03:06:18 zfs01 kernel: nvme1: Missing interrupt Mar 7 03:06:48 zfs01 kernel: nvme1: Resetting controller due to a timeout. Mar 7 03:06:48 zfs01 kernel: nvme1: resetting controller Mar 7 03:06:48 zfs01 kernel: nvme1: temperature threshold not supported Mar 7 03:06:48 zfs01 kernel: nvme1: aborting outstanding i/o Mar 7 03:06:48 zfs01 syslogd: last message repeated 5 times Mar 7 03:07:20 zfs01 kernel: nvme1: VERIFY sqid:1 cid:27 nsid:0 lba:0 len:1 Mar 7 03:07:20 zfs01 kernel: nvme1: INVALID OPCODE (00/01) sqid:1 cid:27 cdw0:0 Mar 7 03:07:20 zfs01 kernel: nvme1: Missing interrupt Mar 7 03:07:20 zfs01 kernel: nvme1: VERIFY sqid:1 cid:27 nsid:0 lba:0 len:1 Mar 7 03:07:20 zfs01 kernel: nvme1: INVALID OPCODE (00/01) sqid:1 cid:27 cdw0:0 Mar 7 03:08:23 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Mar 7 03:08:23 zfs01 kernel: ena0: Trigger reset is on Mar 7 03:08:23 zfs01 kernel: ena0: device is going DOWN Mar 7 03:08:23 zfs01 kernel: nvme1: Resetting controller due to a timeout. Mar 7 03:08:23 zfs01 kernel: nvme1: resetting controller Mar 7 03:08:23 zfs01 dhclient[40936]: send_packet6: Network is down Mar 7 03:08:23 zfs01 dhclient[40936]: dhc6: send_packet6() sent -1 of 52 b= ytes Mar 7 03:08:23 zfs01 kernel: nvme1: aborting outstanding admin command Mar 7 03:08:23 zfs01 kernel: nvme1: CREATE IO SQ (01) sqid:0 cid:24 nsid:1 cdw10:1cb72e58 cdw11:00000000 Mar 7 03:08:23 zfs01 kernel: nvme1: ABORTED - BY REQUEST (00/07) sqid:0 ci= d:15 cdw0:0 Mar 7 03:08:23 zfs01 kernel: nvme1: temperature threshold not supported Mar 7 03:08:23 zfs01 kernel: nvme1: aborting outstanding i/o Mar 7 03:08:23 zfs01 syslogd: last message repeated 2 times Mar 7 03:08:53 zfs01 kernel: nvme1: WRITE sqid:1 cid:18 nsid:1 lba:3856318 len:64 Mar 7 03:08:53 zfs01 kernel: nvme1: INVALID OPCODE (00/01) sqid:1 cid:27 cdw0:0 Mar 7 03:08:53 zfs01 kernel: nvme1: Missing interrupt Mar 7 03:09:15 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x58 Mar 7 03:09:16 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Mar 7 03:09:16 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Mar 7 03:09:16 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Mar 7 03:09:16 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Mar 7 03:09:16 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Mar 7 03:09:16 zfs01 kernel: ena0: device is going UP Mar 7 03:09:16 zfs01 kernel: ena0: link is UP Mar 7 03:10:30 zfs01 dhclient[40936]: send_packet6: Network is down Mar 7 03:10:30 zfs01 dhclient[40936]: dhc6: send_packet6() sent -1 of 52 b= ytes Mar 7 03:10:32 zfs01 dhclient[69248]: send_packet: Network is down Mar 7 03:11:16 zfs01 syslogd: last message repeated 4 times Mar 7 03:11:33 zfs01 syslogd: last message repeated 1 times Mar 7 03:13:31 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Mar 7 03:13:31 zfs01 kernel: ena0: Trigger reset is on Mar 7 03:13:31 zfs01 kernel: ena0: device is going DOWN Mar 7 03:14:25 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x134 Mar 7 03:14:26 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Mar 7 03:14:26 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 root@zfs01:/usr/home/ruben # ls -lahtuT /etc/periodic/daily/ total 128 -rwxr-xr-x 1 root wheel 1.0K Mar 7 03:01:00 2020 450.status-security -rwxr-xr-x 1 root wheel 1.4K Mar 7 03:01:00 2020 440.status-mailq -rwxr-xr-x 1 root wheel 705B Mar 7 03:01:00 2020 430.status-uptime -rwxr-xr-x 1 root wheel 611B Mar 7 03:01:00 2020 420.status-network -rwxr-xr-x 1 root wheel 684B Mar 7 03:01:00 2020 410.status-mfi -rwxr-xr-x 1 root wheel 590B Mar 7 03:01:00 2020 409.status-gconcat -rwxr-xr-x 1 root wheel 590B Mar 7 03:01:00 2020 408.status-gstripe -rwxr-xr-x 1 root wheel 591B Mar 7 03:01:00 2020 407.status-graid3 -rwxr-xr-x 1 root wheel 596B Mar 7 03:01:00 2020 406.status-gmirror -rwxr-xr-x 1 root wheel 807B Mar 7 03:01:00 2020 404.status-zfs -rwxr-xr-x 1 root wheel 583B Mar 7 03:01:00 2020 401.status-graid -rwxr-xr-x 1 root wheel 773B Mar 7 03:01:00 2020 400.status-disks -rwxr-xr-x 1 root wheel 724B Mar 7 03:01:00 2020 330.news -r-xr-xr-x 1 root wheel 1.4K Mar 7 03:01:00 2020 310.accounting -rwxr-xr-x 1 root wheel 693B Mar 7 03:01:00 2020 300.calendar -rwxr-xr-x 1 root wheel 1.0K Mar 7 03:01:00 2020 210.backup-aliases -rwxr-xr-x 1 root wheel 1.7K Mar 7 03:01:00 2020 200.backup-passwd -rwxr-xr-x 1 root wheel 603B Mar 7 03:01:00 2020 150.clean-hoststat -rwxr-xr-x 1 root wheel 1.0K Mar 7 03:01:00 2020 140.clean-rwho -rwxr-xr-x 1 root wheel 709B Mar 7 03:01:00 2020 130.clean-msgs -rwxr-xr-x 1 root wheel 1.1K Mar 7 03:01:00 2020 120.clean-preserve -rwxr-xr-x 1 root wheel 1.5K Mar 7 03:01:00 2020 110.clean-tmps -rwxr-xr-x 1 root wheel 1.3K Mar 7 03:01:00 2020 100.clean-disks -rwxr-xr-x 1 root wheel 811B Mar 5 03:21:29 2020 999.local -rwxr-xr-x 1 root wheel 2.8K Mar 5 03:21:29 2020 800.scrub-zfs -rwxr-xr-x 1 root wheel 845B Mar 5 03:21:29 2020 510.status-world-kern= el -rwxr-xr-x 1 root wheel 737B Mar 5 03:21:29 2020 500.queuerun -rwxr-xr-x 1 root wheel 498B Mar 5 03:21:29 2020 480.status-ntpd -rwxr-xr-x 1 root wheel 451B Mar 5 03:03:36 2020 480.leapfile-ntpd -rwxr-xr-x 1 root wheel 2.0K Mar 5 03:03:18 2020 460.status-mail-rejec= ts drwxr-xr-x 2 root wheel 1.0K Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:/usr/home/ruben #=20 root@zfs01:/usr/home/ruben # ls -lahtuT /etc/periodic/security/ total 68 -rwxr-xr-x 1 root wheel 2.3K Mar 7 03:01:48 2020 900.tcpwrap -rwxr-xr-x 1 root wheel 2.3K Mar 7 03:01:48 2020 800.loginfail -rwxr-xr-x 1 root wheel 1.9K Mar 7 03:01:48 2020 700.kernelmsg -r--r--r-- 1 root wheel 2.8K Mar 7 03:01:48 2020 security.functions -rwxr-xr-x 1 root wheel 2.0K Mar 7 03:01:48 2020 610.ipf6denied -rwxr-xr-x 1 root wheel 2.2K Mar 7 03:01:48 2020 550.ipfwlimit -rwxr-xr-x 1 root wheel 2.1K Mar 7 03:01:48 2020 520.pfdenied -rwxr-xr-x 1 root wheel 1.9K Mar 7 03:01:48 2020 510.ipfdenied -rwxr-xr-x 1 root wheel 2.0K Mar 7 03:01:48 2020 500.ipfwdenied -rwxr-xr-x 1 root wheel 1.9K Mar 7 03:01:48 2020 410.logincheck -rwxr-xr-x 1 root wheel 1.9K Mar 7 03:01:48 2020 400.passwdless -rwxr-xr-x 1 root wheel 1.9K Mar 7 03:01:48 2020 300.chkuid0 -rwxr-xr-x 1 root wheel 2.3K Mar 7 03:01:48 2020 200.chkmounts -rwxr-xr-x 1 root wheel 2.2K Mar 7 03:01:25 2020 110.neggrpperm -rwxr-xr-x 1 root wheel 2.2K Mar 7 03:01:00 2020 100.chksetuid drwxr-xr-x 2 root wheel 512B Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:/usr/home/ruben #=20 the NIC had been going up/down ever since (for 2 days) until a coworker rebooted it this morning. There does seem to be a relationship with the periodic framework, with the issues occuring at 03:05 while the last timestamp updates around 03:01 ... Will attach the verbose boot log. Feel free to request any additional detai= ls! Kind regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Mar 9 08:27:47 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7B80625BFC8 for ; Mon, 9 Mar 2020 08:27:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48bWYv1k4dz4GFq for ; Mon, 9 Mar 2020 08:27:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 1D73525BFC7; Mon, 9 Mar 2020 08:27:47 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1C7DF25BFC6 for ; Mon, 9 Mar 2020 08:27:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48bWYt5vhKz4GDq for ; Mon, 9 Mar 2020 08:27:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ACCD88003 for ; Mon, 9 Mar 2020 08:27:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0298Rkjx055258 for ; Mon, 9 Mar 2020 08:27:46 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0298Rk78055257 for virtualization@FreeBSD.org; Mon, 9 Mar 2020 08:27:46 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Mon, 09 Mar 2020 08:27:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2020 08:27:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #37 from mail@rubenvos.com --- Created attachment 212278 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D212278&action= =3Dedit verbose_boot_zfs01_25_02 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Mar 9 10:32:00 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8E7EE25E4DE for ; Mon, 9 Mar 2020 10:32:00 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48bZKD02fXz4PWx for ; Mon, 9 Mar 2020 10:32:00 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.nyi.freebsd.org (Postfix) id DD63925E4DA; Mon, 9 Mar 2020 10:31:59 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DC41625E4D7; Mon, 9 Mar 2020 10:31:59 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48bZKB10vyz4PRw; Mon, 9 Mar 2020 10:31:57 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 420086EE67; Mon, 9 Mar 2020 11:31:47 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7t3jq27jDR-D; Mon, 9 Mar 2020 11:31:46 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 336CA6EE66; Mon, 9 Mar 2020 11:31:46 +0100 (CET) To: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers , "ports@freebsd.org" From: Willem Jan Withagen Subject: [RFC] Adding a Rados block driver to bhyve Message-ID: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> Date: Mon, 9 Mar 2020 11:31:43 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Content-Language: nl X-Rspamd-Queue-Id: 48bZKB10vyz4PRw X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 2001:4cb8:90:ffff::3 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-5.38 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[digiware.nl]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; IP_SCORE(-3.10)[ip: (-9.51), ipnet: 2001:4cb8::/29(-4.68), asn: 28878(-1.32), country: NL(0.03)]; NEURAL_HAM_MEDIUM(-0.99)[-0.986,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:2001:4cb8::/29, country:NL]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2020 10:32:00 -0000 Hi all, And sorry for crosspoing three groups, but the answer can/could be a mix of things to do in these three areas. I have a prototype of bhyve running on Rados/Ceph working:     https://github.com/freebsd/freebsd/pull/426 But there are a few catches on how to get it in the FreeBSd sources... 1) Easiest would be to just compile it in with the code of the current bhyve.     That will require librados/librbd libraries...     Ceph of this purpose is LGPL2/3 and could go into contrib.     In this case bhyve will hold the rbd-driver by default and a user does not     need to do anything by himself     But I have the feeling that this is the most unwanted scenario 2) User first installs a Ceph package and FreeBSD sources, and then recompiles     bhyve with the option BHYVE_RBD.     And then reinstalls this new version as bhyve or bhyve-rbd in /usr/sbin 3) Create a bhyve-rbd port.     Problem with that is that it will require the FreeBSD source tree for the     bhyve sources, but there is no Ports option for that?     Or bhyve sources are manually copied into the port. And then     try to keep these sources up to date.     Then compile and install a bhyve-rbd into /usr/local/sbin 4) Create a bhyve-blockrbd port.     This is much like 3) but instead of building a bhyve-rbd executable,     it delivers a libblockrbd.so that is dynamically loadable by the     standaard bhyve that comes with base.     For this bhyve needs to be extended with dynamic loadable driver modules.     This is reasonably doable, but is this acceptable for the bhyve maintainers?     For building the port, the bhyve-blockrbd code will only need a limited set     of files from /usr/src/usr.bin/bhyve thus limiting the chance of running out     sequence with the bhyve from base. Looking over these 4 options, I think that 4 is the most desirable one? But 2 would parhaps be workable for users as well, but the project might think otherwise. Are there other options? And/or is 4 the best way to go, with 2 as a nice intermediate? Thanx, --WjW From owner-freebsd-virtualization@freebsd.org Mon Mar 9 13:46:19 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CB3AC262EE7 for ; Mon, 9 Mar 2020 13:46:19 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48bfdQ2JQ8z44p5 for ; Mon, 9 Mar 2020 13:46:18 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 126A3262EE1; Mon, 9 Mar 2020 13:46:18 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 103AA262EDF; Mon, 9 Mar 2020 13:46:18 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48bfdM0W9Xz44k6; Mon, 9 Mar 2020 13:46:14 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f65.google.com with SMTP id i14so9582708otp.5; Mon, 09 Mar 2020 06:46:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dm9mCCHHaGSgYroZ1H/GGnoTLUEgtPx5U61kNf019Ms=; b=t0XLEiGNh+uuB9T83BsRSmhBRFCPMJK4Awa79sPTuPe40WUCMQHFNbSgc9UKgl0LPo zT38Na41egcavQUsbRguDiTaK06fvw/ajhVsasaCAR0zCzuca4DLceofNCOEcZ5UFcxL hrOtm6A/hUoYGg5srmKUT62vj5Tiyz0QOpUmfR3DqJjSjz89yuBEsbA8EqOPp6/XITZQ UKha3FIrdovT6LwWIPBCYIJFvKpGjTz44X1rbQ/J6Hh3teVlQXM20tuYfbI3dRtPFpQ7 TnFpFSfs7Vhwu/B0s/1ytlkKJz/s4V/QI0rf+/W8JauxOfft8tnh2wg6+UeXAoMfRXdw 24Yg== X-Gm-Message-State: ANhLgQ1eTAD39FbxB3153q1OuFANkDvSYTJalNeqGm6zilTCapdc8szw lZQ/2AzossWp8Vsens7W8tfB/B1e9Xh6xazWolWiDbn8Xb0= X-Google-Smtp-Source: ADFU+vuDOwFW13eFluwQKALgOYOuWAS2TfuIb0MaJR2cxng+ep0qOpFuLfGxAM8iR59eBCAv0FhSt3ym/3ndWuPUHfs= X-Received: by 2002:a05:6830:1182:: with SMTP id u2mr1353949otq.222.1583761573197; Mon, 09 Mar 2020 06:46:13 -0700 (PDT) MIME-Version: 1.0 References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> In-Reply-To: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> From: Alan Somers Date: Mon, 9 Mar 2020 07:46:02 -0600 Message-ID: Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Willem Jan Withagen Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers , "ports@freebsd.org" X-Rspamd-Queue-Id: 48bfdM0W9Xz44k6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.65 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-2.99 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[65.210.85.209.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[65.210.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-0.99)[ip: (-0.28), ipnet: 209.85.128.0/17(-2.98), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2020 13:46:19 -0000 On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen wrote: > Hi all, > > And sorry for crosspoing three groups, but the answer can/could be a mix > of things to do in these three areas. > > I have a prototype of bhyve running on Rados/Ceph working: > https://github.com/freebsd/freebsd/pull/426 > > But there are a few catches on how to get it in the FreeBSd sources... > > 1) Easiest would be to just compile it in with the code of the current > bhyve. > That will require librados/librbd libraries... > Ceph of this purpose is LGPL2/3 and could go into contrib. > In this case bhyve will hold the rbd-driver by default and a user > does not > need to do anything by himself > But I have the feeling that this is the most unwanted scenario > > 2) User first installs a Ceph package and FreeBSD sources, and then > recompiles > bhyve with the option BHYVE_RBD. > And then reinstalls this new version as bhyve or bhyve-rbd in > /usr/sbin > > 3) Create a bhyve-rbd port. > Problem with that is that it will require the FreeBSD source tree > for the > bhyve sources, but there is no Ports option for that? > Or bhyve sources are manually copied into the port. And then > try to keep these sources up to date. > Then compile and install a bhyve-rbd into /usr/local/sbin > > 4) Create a bhyve-blockrbd port. > This is much like 3) but instead of building a bhyve-rbd executable, > it delivers a libblockrbd.so that is dynamically loadable by the > standaard bhyve that comes with base. > > For this bhyve needs to be extended with dynamic loadable driver > modules. > This is reasonably doable, but is this acceptable for the bhyve > maintainers? > > For building the port, the bhyve-blockrbd code will only need a > limited set > of files from /usr/src/usr.bin/bhyve thus limiting the chance of > running out > sequence with the bhyve from base. > > Looking over these 4 options, I think that 4 is the most desirable one? > But 2 would parhaps be workable for users as well, but the project might > think > otherwise. > > Are there other options? > And/or is 4 the best way to go, with 2 as a nice intermediate? > > Thanx, > --WjW > Great work! I also agree that option 4 sounds like the best. There's precedent for ports that require the FreeBSD Sources. For example, see devel/py-libzfs or emulators/virtualbox-ose. You just need to define the SRC_BASE variable. From owner-freebsd-virtualization@freebsd.org Mon Mar 9 19:23:05 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4C64026ADB5 for ; Mon, 9 Mar 2020 19:23:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48bp610LHjz4KNR for ; Mon, 9 Mar 2020 19:23:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 0ABDD26ADB4; Mon, 9 Mar 2020 19:23:05 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A72126ADB3 for ; Mon, 9 Mar 2020 19:23:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48bp606Qlcz4KN6 for ; Mon, 9 Mar 2020 19:23:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C63BEFA89 for ; Mon, 9 Mar 2020 19:23:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 029JN4IU091345 for ; Mon, 9 Mar 2020 19:23:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 029JN4qJ091324 for virtualization@FreeBSD.org; Mon, 9 Mar 2020 19:23:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Mon, 09 Mar 2020 19:23:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cperciva@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2020 19:23:05 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #38 from Colin Percival --- Hi Ruben, Can you confirm that "cpl does not map to outstanding cmd" was the first unusual line to show up? (i.e., there weren't more lines before that which= got lost?) We had a hypothetical explanation for this which should have also involved having a "missing interrupt" message at the same time... Colin Percival --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Mar 10 08:42:44 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EFAEF25C834 for ; Tue, 10 Mar 2020 08:42:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48c7rh562tz4Lcm for ; Tue, 10 Mar 2020 08:42:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 805F325C833; Tue, 10 Mar 2020 08:42:44 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8007125C832 for ; Tue, 10 Mar 2020 08:42:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48c7rg2FtDz4LZc for ; Tue, 10 Mar 2020 08:42:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2B34320E01 for ; Tue, 10 Mar 2020 08:42:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02A8ghOI062830 for ; Tue, 10 Mar 2020 08:42:43 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02A8ghuZ062829 for virtualization@FreeBSD.org; Tue, 10 Mar 2020 08:42:43 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Tue, 10 Mar 2020 08:42:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 08:42:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #39 from mail@rubenvos.com --- (In reply to Colin Percival from comment #38) Hi Colin, There were issues with the nic (perhaps intterupt related?) prior to the "c= pl does not map to outstanding cmd" log entry. I will upload the full /var/log/messages contents between a couple of incidents. Please let me know if you would like me to provide further context/configure additional logging etc. Kind regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Mar 10 08:43:29 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 23A7025C90E for ; Tue, 10 Mar 2020 08:43:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48c7sX5TbKz4NDs for ; Tue, 10 Mar 2020 08:43:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 9B65925C90D; Tue, 10 Mar 2020 08:43:28 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9AFF925C90B for ; Tue, 10 Mar 2020 08:43:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48c7sX2h9Tz4NCt for ; Tue, 10 Mar 2020 08:43:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3072E20E08 for ; Tue, 10 Mar 2020 08:43:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02A8hRag063699 for ; Tue, 10 Mar 2020 08:43:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02A8hRhj063698 for virtualization@FreeBSD.org; Tue, 10 Mar 2020 08:43:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Tue, 10 Mar 2020 08:43:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 08:43:29 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #40 from mail@rubenvos.com --- Created attachment 212295 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D212295&action= =3Dedit messages --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Mar 10 09:59:21 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6E99925E9A9 for ; Tue, 10 Mar 2020 09:59:21 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48c9Y42Mz8z4LxX for ; Tue, 10 Mar 2020 09:59:20 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.nyi.freebsd.org (Postfix) id 18F6825E9A4; Tue, 10 Mar 2020 09:59:20 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0880425E9A3; Tue, 10 Mar 2020 09:59:20 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48c9Y33NhVz4Lvl; Tue, 10 Mar 2020 09:59:19 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 13112969; Tue, 10 Mar 2020 10:59:17 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HD4-c7XDSnoN; Tue, 10 Mar 2020 10:59:16 +0100 (CET) Received: from [192.168.10.9] (vaio [192.168.10.9]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 97DFB968; Tue, 10 Mar 2020 10:59:16 +0100 (CET) Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers , "ports@freebsd.org" References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> From: Willem Jan Withagen Message-ID: Date: Tue, 10 Mar 2020 10:59:16 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-GB X-Rspamd-Queue-Id: 48c9Y33NhVz4Lvl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.10 / 15.00]; REPLY(-4.00)[]; NEURAL_SPAM_MEDIUM(0.89)[0.891,0]; NEURAL_HAM_LONG(-0.99)[-0.994,0] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 09:59:21 -0000 On 9-3-2020 14:46, Alan Somers wrote: > On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen > wrote: > > Hi all, > > And sorry for crosspoing three groups, but the answer can/could be > a mix > of things to do in these three areas. > > I have a prototype of bhyve running on Rados/Ceph working: > https://github.com/freebsd/freebsd/pull/426 > .......... > > 4) Create a bhyve-blockrbd port. >      This is much like 3) but instead of building a bhyve-rbd > executable, >      it delivers a libblockrbd.so that is dynamically loadable by the >      standaard bhyve that comes with base. > ............ > Great work!  I also agree that option 4 sounds like the best. There's precedent for ports that > require the FreeBSD Sources.  For example, see devel/py-libzfs or emulators/virtualbox-ose. > You just need to define the SRC_BASE variable. Hi Alan, Thanx for the hint, and it made me check what is actually available within the poudriere jail And that does have full source, so the Makefile code is mainly for those that build in a different way. I've got a proto version working when compiling stuff with `make buildworld`, but run in the problem that libblock_rbd.so is stripped in such a way that the symbol I need is removed. Using the unstripped version does work. Is there an incantation for the SRC Makefiles that builds a dynamical loadable lib?? And I'm still looking for a PORTS example of building a dynamical loadable lib. Or is there no generic code for that in the PORTS Mk files? --WjW BTW: Still haven't worked in your AIO code :( From owner-freebsd-virtualization@freebsd.org Tue Mar 10 15:16:01 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D160C266E18 for ; Tue, 10 Mar 2020 15:16:01 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48cJZS2GDrz4RF1 for ; Tue, 10 Mar 2020 15:16:00 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 3EEAD266E10; Tue, 10 Mar 2020 15:16:00 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E28C266E0E; Tue, 10 Mar 2020 15:16:00 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cJZQ3vdBz4R8h; Tue, 10 Mar 2020 15:15:58 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f65.google.com with SMTP id a6so5571404otb.10; Tue, 10 Mar 2020 08:15:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=R67LPPOFLjUiCs4J3knDk69qRXAA0Owhpznhxfrb2sY=; b=rwbo719H2PfQRi3Pt+aPSyaj+u4zsCNdea6vXll2Mwc0WK1YTzQ3qenAHVVBiCW16d tkp62Vs/DFlP78TvdEPAHO8RJwF/mcDZLsKK+XidOVk5fNE6FFzSBDaNnPjkCjdRP5W1 ogZd0iONMsNsgxEE3dgFFI667E/UDftlIzi6HvUXRUIWz2XVVLejgMcaPCxj2M9GC4mQ 1cSyLRi0HyXs0O7mO3RDyUbEQ6xyqRtSeHOau1X1T905EZZissNDiXtTcWpMLQSeH2Xz 4Qdie2MMhPBjFSfPCZvJy3MRo0DZItSpydwk0gcZsEvaM8vlcxTO36BrCTtdJDERgB52 VziA== X-Gm-Message-State: ANhLgQ0z57Q6hLsZEfqyYY1Rd6Y2vo5ik+FlGNN1tvRzMFki6K1TrsTe XwF9aKDypqxvMZ3ra1BzxeMujYmoU/lCx+wgNGiXf7bewnA= X-Google-Smtp-Source: ADFU+vuw7TzSwMNFLCPAx9RrQFauFycjTVTY/dtznZTTIZ27JdO11GK6uncNQqUTYqvqGPnEXuoW0utCfihg1VevFDQ= X-Received: by 2002:a9d:7359:: with SMTP id l25mr7861793otk.18.1583853357006; Tue, 10 Mar 2020 08:15:57 -0700 (PDT) MIME-Version: 1.0 References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> In-Reply-To: From: Alan Somers Date: Tue, 10 Mar 2020 09:15:45 -0600 Message-ID: Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Willem Jan Withagen Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers , "ports@freebsd.org" X-Rspamd-Queue-Id: 48cJZQ3vdBz4R8h X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.65 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.99 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; URI_COUNT_ODD(1.00)[11]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[65.210.85.209.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_GOOD(0.00)[65.210.85.209.rep.mailspike.net : 127.0.0.18]; IP_SCORE(-0.99)[ip: (-0.26), ipnet: 209.85.128.0/17(-2.98), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 15:16:01 -0000 On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen wrote: > On 9-3-2020 14:46, Alan Somers wrote: > > On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen > wrote: > >> Hi all, >> >> And sorry for crosspoing three groups, but the answer can/could be a mix >> of things to do in these three areas. >> >> I have a prototype of bhyve running on Rados/Ceph working: >> https://github.com/freebsd/freebsd/pull/426 >> >> .......... > > 4) Create a bhyve-blockrbd port. >> This is much like 3) but instead of building a bhyve-rbd executable, >> it delivers a libblockrbd.so that is dynamically loadable by the >> standaard bhyve that comes with base. >> > ............ > > > Great work! I also agree that option 4 sounds like the best. There's > precedent for ports that > > require the FreeBSD Sources. For example, see devel/py-libzfs or > emulators/virtualbox-ose. > > You just need to define the SRC_BASE variable. > > Hi Alan, > > Thanx for the hint, and it made me check what is actually available within > the poudriere jail > And that does have full source, so the Makefile code is mainly for those > that build in a different way. > > I've got a proto version working when compiling stuff with `make > buildworld`, but run in the > problem that libblock_rbd.so is stripped in such a way that the symbol I > need is removed. > Using the unstripped version does work. > > Is there an incantation for the SRC Makefiles that builds a dynamical > loadable lib?? > And I'm still looking for a PORTS example of building a dynamical loadable > lib. > Or is there no generic code for that in the PORTS Mk files? > > --WjW > > BTW: Still haven't worked in your AIO code :( > There are plenty of dynamic libraries built with the SRC makefiles. For example, https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup . And there are plenty of ports that build shared libraries too, just look at /usr/local/lib/*.so. However, the ports framework doesn't have much special code just to support building libraries. Instead the hard work is always done by the ports themselves. Some use autotools, some cmake, etc etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . From owner-freebsd-virtualization@freebsd.org Tue Mar 10 15:18:41 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 07222266FFD for ; Tue, 10 Mar 2020 15:18:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48cJdX0GtWz4Y0b for ; Tue, 10 Mar 2020 15:18:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id A9EBE266FFC; Tue, 10 Mar 2020 15:18:39 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A9388266FFB for ; Tue, 10 Mar 2020 15:18:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cJdW2mF3z4Xy6 for ; Tue, 10 Mar 2020 15:18:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0B8AC255C6 for ; Tue, 10 Mar 2020 15:18:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02AFIcVL067597 for ; Tue, 10 Mar 2020 15:18:38 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02AFIclW067596 for virtualization@FreeBSD.org; Tue, 10 Mar 2020 15:18:38 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 244721] bhyve: Silence ACPI RTC error/warning in Linux guests Date: Tue, 10 Mar 2020 15:18:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jlduran@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 15:18:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D244721 Bug ID: 244721 Summary: bhyve: Silence ACPI RTC error/warning in Linux guests Product: Base System Version: CURRENT Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: jlduran@gmail.com Created attachment 212308 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D212308&action= =3Dedit bhyve: Silence ACPI RTC error/warning in Linux guests >From dmesg: ACPI Error: Could not enable RealTimeClock event (20180810/evxfevnt-184) ACPI Warning: Could not enable fixed event - RealTimeClock (4) (20180810/evxface-620) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Mar 10 15:41:48 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 319232678BC for ; Tue, 10 Mar 2020 15:41:48 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48cK8C2jb3z4bhj for ; Tue, 10 Mar 2020 15:41:47 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.nyi.freebsd.org (Postfix) id 2A7912678B9; Tue, 10 Mar 2020 15:41:47 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 29C022678B7; Tue, 10 Mar 2020 15:41:47 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48cK8B470Qz4bgt; Tue, 10 Mar 2020 15:41:46 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id D02A713C81; Tue, 10 Mar 2020 16:41:43 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vYtHBxsbLW_x; Tue, 10 Mar 2020 16:41:43 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id CA01113C75; Tue, 10 Mar 2020 16:41:42 +0100 (CET) Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers , "ports@freebsd.org" References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> From: Willem Jan Withagen Message-ID: <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> Date: Tue, 10 Mar 2020 16:41:41 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: nl X-Rspamd-Queue-Id: 48cK8B470Qz4bgt X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.10 / 15.00]; NEURAL_SPAM_MEDIUM(0.90)[0.897,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-0.99)[-0.994,0] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 15:41:48 -0000 On 10-3-2020 16:15, Alan Somers wrote: > On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen > wrote: > > On 9-3-2020 14:46, Alan Somers wrote: >> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >> > wrote: >> >> Hi all, >> >> And sorry for crosspoing three groups, but the answer >> can/could be a mix >> of things to do in these three areas. >> >> I have a prototype of bhyve running on Rados/Ceph working: >> https://github.com/freebsd/freebsd/pull/426 >> > .......... >> >> 4) Create a bhyve-blockrbd port. >>      This is much like 3) but instead of building a bhyve-rbd >> executable, >>      it delivers a libblockrbd.so that is dynamically >> loadable by the >>      standaard bhyve that comes with base. >> > ............ > > > Great work!  I also agree that option 4 sounds like the best.  > There's precedent for ports that > > require the FreeBSD Sources.  For example, see devel/py-libzfs > or emulators/virtualbox-ose. > > You just need to define the SRC_BASE variable. > Hi Alan, > > Thanx for the hint, and it made me check what is actually > available within the poudriere jail > And that does have full source, so the Makefile code is mainly for > those that build in a different way. > > I've got a proto version working when compiling stuff with `make > buildworld`, but run in the > problem that libblock_rbd.so is stripped in such a way that the > symbol I need is removed. > Using the unstripped version does work. > > Is there an incantation for the SRC Makefiles that builds a > dynamical loadable lib?? > And I'm still looking for a PORTS example of building a dynamical > loadable lib. > Or is there no generic code for that in the PORTS Mk files? > > --WjW > > BTW: Still haven't worked in your AIO code :( > > > There are plenty of dynamic libraries built with the SRC makefiles.  > For example, > https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup > . That looks dangerously close to what I have for libblock_rbd. === > cat Makefile-librbd # # $FreeBSD$ # PACKAGE=lib${LIB} .include LIB=            block_rbd SHLIB_MAJOR=    1 SRCS=   block_rbd.c CFLAGS+=-I${SRCTOP}/sys CFLAGS+=-g -O0 -fPIC -rdynamic LDFLAGS+=-Wl,-export-dynamic,-Bdynamic CFLAGS+=-DWITHOUT_CAPSICUM LOCALBASE?=     /usr/local CFLAGS+=        -I${LOCALBASE}/include LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd WARNS?= 2 === This is the code that mk.lib.bsd runs: objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug libblock_rbd.so.1.full libblock_rbd.so.1 So still I get a stripped lib in /usr/lib. And then the one and only symbol I need to load is not found. Copying libblock_rbd.so.1.full actually works for me. So either I'm doing it the wrong way, like special options on the symbols oid. Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? And there are plenty of ports that build shared libraries too, just look at /usr/local/lib/*.so.  However, the ports framework doesn't have much special code just to support building libraries.  Instead the hard work is always done by the ports themselves.  Some use autotools, some cmake, etc etc.  The simplest port I can find that uses both SRC_BASE and INSTALL_LIB is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . Oke thanx, I'll have a look at it, and given that I can see most of the compile build stuff in the SRC_BASE version I'll get it to work. --WjW From owner-freebsd-virtualization@freebsd.org Tue Mar 10 16:21:38 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8E4C12690E8 for ; Tue, 10 Mar 2020 16:21:38 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48cL293NFtz3Dyw for ; Tue, 10 Mar 2020 16:21:37 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 009952690D7; Tue, 10 Mar 2020 16:21:37 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EB1922690D6; Tue, 10 Mar 2020 16:21:36 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ot1-f54.google.com (mail-ot1-f54.google.com [209.85.210.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cL2664VHz3Dtn; Tue, 10 Mar 2020 16:21:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-ot1-f54.google.com with SMTP id g15so7613862otr.0; Tue, 10 Mar 2020 09:21:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FxExWztqzhA7Okl24KY/uH5VFrSEi9pTSZV2EDfdEuM=; b=f0FQcpo8ue7pNlJQu2sy+Xzw+xRT9QGIe/AVGAYa6NO97igSQGWMNgmgZm/gQnlxSE sToDW5+4DN97h3HH+HLLQRXz9kZd6+tCY3jt+ho1HC/EMCbduZzJ7uIEQFJu/MUjQDt1 lwcEGxOu/MdGikLcnLDO3XctuRxrvF/IxDWtnlWu/RGlw004paIzYaR1v52zlYZojvzy E6MZv2C+LKQZccomxNV2uLXr/327nUj7ifAFb4p82/QwA0OD9z8/MshhAE36dw1qzYRH rHyBgBT7L54fK/LbhN2I3/EfMYsEDHQ2nybLMcPfrk0OxDX0Qbw4Ha5KXxRnDDbFM8Zm 4+Gg== X-Gm-Message-State: ANhLgQ133mGcBLWkamUw04IQteMnj+RqXi8EdKuVVy8B6P04DekLXqfd 8z0NvIq5Wwm9kZ31KljFw14xPgb23518bKO+DRhPWV3k X-Google-Smtp-Source: ADFU+vst+kjLRXZAi1FlxUbWxEzB0D06jyjBAZHlLiyV3Ats4t4KLg6B54e1E+NwBBIljEbE/hHDPIUmbh0SQSeZStk= X-Received: by 2002:a9d:b8f:: with SMTP id 15mr15417282oth.256.1583857293687; Tue, 10 Mar 2020 09:21:33 -0700 (PDT) MIME-Version: 1.0 References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> In-Reply-To: <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> From: Alan Somers Date: Tue, 10 Mar 2020 10:21:21 -0600 Message-ID: Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Willem Jan Withagen Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers X-Rspamd-Queue-Id: 48cL2664VHz3Dtn X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.210.54 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.99 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; RWL_MAILSPIKE_GOOD(0.00)[54.210.85.209.rep.mailspike.net : 127.0.0.18]; URI_COUNT_ODD(1.00)[13]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[54.210.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.99)[ip: (-0.27), ipnet: 209.85.128.0/17(-2.98), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 16:21:38 -0000 On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen wrote: > On 10-3-2020 16:15, Alan Somers wrote: > > On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen > wrote: > >> On 9-3-2020 14:46, Alan Somers wrote: >> >> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >> wrote: >> >>> Hi all, >>> >>> And sorry for crosspoing three groups, but the answer can/could be a mix >>> of things to do in these three areas. >>> >>> I have a prototype of bhyve running on Rados/Ceph working: >>> https://github.com/freebsd/freebsd/pull/426 >>> >>> .......... >> >> 4) Create a bhyve-blockrbd port. >>> This is much like 3) but instead of building a bhyve-rbd executable, >>> it delivers a libblockrbd.so that is dynamically loadable by the >>> standaard bhyve that comes with base. >>> >> ............ >> >> > Great work! I also agree that option 4 sounds like the best. There's >> precedent for ports that >> > require the FreeBSD Sources. For example, see devel/py-libzfs or >> emulators/virtualbox-ose. >> > You just need to define the SRC_BASE variable. >> Hi Alan, >> >> Thanx for the hint, and it made me check what is actually available >> within the poudriere jail >> And that does have full source, so the Makefile code is mainly for those >> that build in a different way. >> >> I've got a proto version working when compiling stuff with `make >> buildworld`, but run in the >> problem that libblock_rbd.so is stripped in such a way that the symbol I >> need is removed. >> Using the unstripped version does work. >> >> Is there an incantation for the SRC Makefiles that builds a dynamical >> loadable lib?? >> And I'm still looking for a PORTS example of building a dynamical >> loadable lib. >> Or is there no generic code for that in the PORTS Mk files? >> >> --WjW >> >> BTW: Still haven't worked in your AIO code :( >> > > There are plenty of dynamic libraries built with the SRC makefiles. For > example, > https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup > . > > > That looks dangerously close to what I have for libblock_rbd. > === > > cat Makefile-librbd > # > # $FreeBSD$ > # > > PACKAGE=lib${LIB} > > .include > > LIB= block_rbd > SHLIB_MAJOR= 1 > > SRCS= block_rbd.c > > CFLAGS+=-I${SRCTOP}/sys > CFLAGS+=-g -O0 -fPIC -rdynamic > LDFLAGS+=-Wl,-export-dynamic,-Bdynamic > CFLAGS+=-DWITHOUT_CAPSICUM > > LOCALBASE?= /usr/local > CFLAGS+= -I${LOCALBASE}/include > LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd > > WARNS?= 2 > > === > > This is the code that mk.lib.bsd runs: > objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug > objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug > libblock_rbd.so.1.full libblock_rbd.so.1 > > So still I get a stripped lib in /usr/lib. And then the one and only > symbol I need to load > is not found. Copying libblock_rbd.so.1.full actually works for me. > > So either I'm doing it the wrong way, like special options on the symbols > oid. > Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? > > And there are plenty of ports that build shared libraries too, just look > at /usr/local/lib/*.so. However, the ports framework doesn't have much > special code just to support building libraries. Instead the hard work is > always done by the ports themselves. Some use autotools, some cmake, etc > etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB > is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . > > > Oke thanx, I'll have a look at it, and given that I can see most of the > compile build stuff > in the SRC_BASE version I'll get it to work. > > --WjW > Try setting "STRIP= " in your makefile. That should prevent the stripping. However, I think there's something wrong with your library, too. The library should be usable even if it's stripped. From owner-freebsd-virtualization@freebsd.org Tue Mar 10 16:28:16 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DFCDF2696A7 for ; Tue, 10 Mar 2020 16:28:16 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48cL9q3fBgz41gD for ; Tue, 10 Mar 2020 16:28:15 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: by mailman.nyi.freebsd.org (Postfix) id 6D5852696A3; Tue, 10 Mar 2020 16:28:15 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 66FFA2696A2; Tue, 10 Mar 2020 16:28:15 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48cL9q0hljz41fQ; Tue, 10 Mar 2020 16:28:14 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 9660D1530F; Tue, 10 Mar 2020 17:28:12 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7U7cJZp7et5f; Tue, 10 Mar 2020 17:28:11 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 9A8381530D; Tue, 10 Mar 2020 17:28:11 +0100 (CET) Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> From: Willem Jan Withagen Message-ID: Date: Tue, 10 Mar 2020 17:28:10 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: Content-Language: nl X-Rspamd-Queue-Id: 48cL9q0hljz41fQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.06 / 15.00]; NEURAL_SPAM_MEDIUM(0.93)[0.934,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-0.99)[-0.995,0] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 16:28:17 -0000 On 10-3-2020 17:21, Alan Somers wrote: > On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen > wrote: > > On 10-3-2020 16:15, Alan Somers wrote: >> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen >> > wrote: >> >> On 9-3-2020 14:46, Alan Somers wrote: >>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >>> > wrote: >>> >>> Hi all, >>> >>> And sorry for crosspoing three groups, but the answer >>> can/could be a mix >>> of things to do in these three areas. >>> >>> I have a prototype of bhyve running on Rados/Ceph working: >>> https://github.com/freebsd/freebsd/pull/426 >>> >> .......... >>> >>> 4) Create a bhyve-blockrbd port. >>>      This is much like 3) but instead of building a >>> bhyve-rbd executable, >>>      it delivers a libblockrbd.so that is dynamically >>> loadable by the >>>      standaard bhyve that comes with base. >>> >> ............ >> >> > Great work!  I also agree that option 4 sounds like the >> best.  There's precedent for ports that >> > require the FreeBSD Sources.  For example, see >> devel/py-libzfs or emulators/virtualbox-ose. >> > You just need to define the SRC_BASE variable. >> Hi Alan, >> >> Thanx for the hint, and it made me check what is actually >> available within the poudriere jail >> And that does have full source, so the Makefile code is >> mainly for those that build in a different way. >> >> I've got a proto version working when compiling stuff with >> `make buildworld`, but run in the >> problem that libblock_rbd.so is stripped in such a way that >> the symbol I need is removed. >> Using the unstripped version does work. >> >> Is there an incantation for the SRC Makefiles that builds a >> dynamical loadable lib?? >> And I'm still looking for a PORTS example of building a >> dynamical loadable lib. >> Or is there no generic code for that in the PORTS Mk files? >> >> --WjW >> >> BTW: Still haven't worked in your AIO code :( >> >> >> There are plenty of dynamic libraries built with the SRC >> makefiles.  For example, >> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup >> . > > That looks dangerously close to what I have for libblock_rbd. > === > > cat Makefile-librbd > # > # $FreeBSD$ > # > > PACKAGE=lib${LIB} > > .include > > > LIB=            block_rbd > SHLIB_MAJOR=    1 > > SRCS=   block_rbd.c > > CFLAGS+=-I${SRCTOP}/sys > CFLAGS+=-g -O0 -fPIC -rdynamic > LDFLAGS+=-Wl,-export-dynamic,-Bdynamic > CFLAGS+=-DWITHOUT_CAPSICUM > > LOCALBASE?=     /usr/local > CFLAGS+=        -I${LOCALBASE}/include > LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd > > WARNS?= 2 > > === > > This is the code that mk.lib.bsd runs: > objcopy --only-keep-debug libblock_rbd.so.1.full > libblock_rbd.so.1.debug > objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug > libblock_rbd.so.1.full libblock_rbd.so.1 > > So still I get a stripped lib in /usr/lib. And then the one and > only symbol I need to load > is not found. Copying libblock_rbd.so.1.full actually works for me. > > So either I'm doing it the wrong way, like special options on the > symbols oid. > Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? > > And there are plenty of ports that build shared libraries too, > just look at /usr/local/lib/*.so. However, the ports framework > doesn't have much special code just to support building > libraries.  Instead the hard work is always done by the ports > themselves. Some use autotools, some cmake, etc etc.  The simplest > port I can find that uses both SRC_BASE and INSTALL_LIB is this > one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . > > Oke thanx, I'll have a look at it, and given that I can see most > of the compile build stuff > in the SRC_BASE version I'll get it to work. > > --WjW > > > Try setting "STRIP=    " in your makefile.  That should prevent the > stripping.  However, I think there's something wrong with your > library, too.  The library should be usable even if it's stripped. I checked with objdump, and the symbol that I need is definitly not present in the stripped version. And it does not really matter if I declare it static or not. But I'll give it a few more itterations to try it out. Including 'STRIP= ' Thanx, --WjW From owner-freebsd-virtualization@freebsd.org Tue Mar 10 16:49:08 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EA76F269EC8 for ; Tue, 10 Mar 2020 16:49:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48cLdw1MsVz3xLR for ; Tue, 10 Mar 2020 16:49:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id DA404269EC5; Tue, 10 Mar 2020 16:49:07 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D9872269EC4; Tue, 10 Mar 2020 16:49:07 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cLdt3qgGz3xGk; Tue, 10 Mar 2020 16:49:06 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi1-f177.google.com with SMTP id c1so14561020oiy.2; Tue, 10 Mar 2020 09:49:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gUmYw482+yc1LN/ofvq/PgpZURXsZ+DQY0p98X7Nvmo=; b=MmRpxEAHaL9mvbzWU/dNrUFefFFeC21fojPUXFEtVE0BtmrdXCbOnrjJhCXrfljSuy QYQZin0NrM5lp7eLPBFJL2HE9QGH3dNwlp07z72hi+h7lFUTlPVYrf4dTM2a/M+oo9H0 NBFhpYZBkdT7UdC5Ie/tpp6FuOM/edLwfIjS2bseuUiTEmFcJ0VrPGD84srMzXkO3+Dy FaSd5m8F2Sak92aKjgs0mgghBgPzgTZwAdB+LFjmj6sqWQ9dq8GNgA2saF3PciVsV9Jz DYuehAFdIvx51fbHm0AHPsd5ZXm4hLL3IrQ8iNASWygkirPgEWm8t0peTW40NYnC4LKR YCjw== X-Gm-Message-State: ANhLgQ37/exH7nqT+0rFw6x541TL6sqvFSFKi8SgO0PqtfoJz03Tv+mv MnTEhXfjKalkoFoswd3trpUJ6REowmKFKMbODG1TtI/VJQs= X-Google-Smtp-Source: ADFU+vvswGKrYZRywpblAPI0740yoJPo8JA0imSIhoF+nfEMWd2E0JJoyJv3BJ02C1w6xzseU8OM906D2oNugda2WOA= X-Received: by 2002:aca:3f09:: with SMTP id m9mr1778196oia.55.1583858944935; Tue, 10 Mar 2020 09:49:04 -0700 (PDT) MIME-Version: 1.0 References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> In-Reply-To: From: Alan Somers Date: Tue, 10 Mar 2020 10:48:53 -0600 Message-ID: Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Willem Jan Withagen Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers X-Rspamd-Queue-Id: 48cLdt3qgGz3xGk X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-1.97 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[freebsd.org]; URI_COUNT_ODD(1.00)[13]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[177.167.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.97)[ip: (-0.19), ipnet: 209.85.128.0/17(-2.98), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[177.167.85.209.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 16:49:09 -0000 On Tue, Mar 10, 2020 at 10:28 AM Willem Jan Withagen wrote: > On 10-3-2020 17:21, Alan Somers wrote: > > On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen > wrote: > >> On 10-3-2020 16:15, Alan Somers wrote: >> >> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen >> wrote: >> >>> On 9-3-2020 14:46, Alan Somers wrote: >>> >>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen >>> wrote: >>> >>>> Hi all, >>>> >>>> And sorry for crosspoing three groups, but the answer can/could be a mix >>>> of things to do in these three areas. >>>> >>>> I have a prototype of bhyve running on Rados/Ceph working: >>>> https://github.com/freebsd/freebsd/pull/426 >>>> >>>> .......... >>> >>> 4) Create a bhyve-blockrbd port. >>>> This is much like 3) but instead of building a bhyve-rbd >>>> executable, >>>> it delivers a libblockrbd.so that is dynamically loadable by the >>>> standaard bhyve that comes with base. >>>> >>> ............ >>> >>> > Great work! I also agree that option 4 sounds like the best. There's >>> precedent for ports that >>> > require the FreeBSD Sources. For example, see devel/py-libzfs or >>> emulators/virtualbox-ose. >>> > You just need to define the SRC_BASE variable. >>> Hi Alan, >>> >>> Thanx for the hint, and it made me check what is actually available >>> within the poudriere jail >>> And that does have full source, so the Makefile code is mainly for those >>> that build in a different way. >>> >>> I've got a proto version working when compiling stuff with `make >>> buildworld`, but run in the >>> problem that libblock_rbd.so is stripped in such a way that the symbol I >>> need is removed. >>> Using the unstripped version does work. >>> >>> Is there an incantation for the SRC Makefiles that builds a dynamical >>> loadable lib?? >>> And I'm still looking for a PORTS example of building a dynamical >>> loadable lib. >>> Or is there no generic code for that in the PORTS Mk files? >>> >>> --WjW >>> >>> BTW: Still haven't worked in your AIO code :( >>> >> >> There are plenty of dynamic libraries built with the SRC makefiles. For >> example, >> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup >> . >> >> >> That looks dangerously close to what I have for libblock_rbd. >> === >> > cat Makefile-librbd >> # >> # $FreeBSD$ >> # >> >> PACKAGE=lib${LIB} >> >> .include >> >> LIB= block_rbd >> SHLIB_MAJOR= 1 >> >> SRCS= block_rbd.c >> >> CFLAGS+=-I${SRCTOP}/sys >> CFLAGS+=-g -O0 -fPIC -rdynamic >> LDFLAGS+=-Wl,-export-dynamic,-Bdynamic >> CFLAGS+=-DWITHOUT_CAPSICUM >> >> LOCALBASE?= /usr/local >> CFLAGS+= -I${LOCALBASE}/include >> LDFLAGS+= -L${LOCALBASE}/lib -lrados -lrbd >> >> WARNS?= 2 >> >> === >> >> This is the code that mk.lib.bsd runs: >> objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug >> objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug >> libblock_rbd.so.1.full libblock_rbd.so.1 >> >> So still I get a stripped lib in /usr/lib. And then the one and only >> symbol I need to load >> is not found. Copying libblock_rbd.so.1.full actually works for me. >> >> So either I'm doing it the wrong way, like special options on the symbols >> oid. >> Or mk.lib.bsd cannot deliver dlopen/dlsym-able files? >> >> And there are plenty of ports that build shared libraries too, just look >> at /usr/local/lib/*.so. However, the ports framework doesn't have much >> special code just to support building libraries. Instead the hard work is >> always done by the ports themselves. Some use autotools, some cmake, etc >> etc. The simplest port I can find that uses both SRC_BASE and INSTALL_LIB >> is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ . >> >> Oke thanx, I'll have a look at it, and given that I can see most of the >> compile build stuff >> in the SRC_BASE version I'll get it to work. >> >> --WjW >> > > Try setting "STRIP= " in your makefile. That should prevent the > stripping. However, I think there's something wrong with your library, > too. The library should be usable even if it's stripped. > > I checked with objdump, and the symbol that I need is definitly not > present in the stripped version. > And it does not really matter if I declare it static or not. > But I'll give it a few more itterations to try it out. > Including 'STRIP= ' > > Thanx, > --WjW > What does "nm --dynamic libblock_rbd.so.1" show? From owner-freebsd-virtualization@freebsd.org Tue Mar 10 18:08:39 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 22BD226BDEA for ; Tue, 10 Mar 2020 18:08:39 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48cNPf2cd0z3Cyh for ; Tue, 10 Mar 2020 18:08:38 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 1B06F26BDE4; Tue, 10 Mar 2020 18:08:38 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 18E7826BDE3; Tue, 10 Mar 2020 18:08:38 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cNPc6Y7cz3Cs5; Tue, 10 Mar 2020 18:08:36 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-ot1-f48.google.com with SMTP id a9so7900218otl.6; Tue, 10 Mar 2020 11:08:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=OqcXSDstDOq+pXsLWGS0/j7exXAXCZzQoYRVgeTtPEY=; b=k17NmW0uMZtlo+ZNSL6vV+n/lpPKyidGjmfDKWXL7vLuA5sxZ6K1sOQdcJaUlCxDNV vSKe4/fLQZG7vKfR4Flec2qx00Wq4k08gvbwvxsSL5a0v4IYk8tzvA4+fcN5iYlhCDG4 7Sngt7Gg2HuHxp2iuPksOD41nqUicQx7j8R52uADondmuzluRQwfNZuvDy5zZLrBvlDo iWQ3Ic3gdrxu0EICGOy9HiP+L8eH4mhZ3UmqKbTv2uU7A5H0ohEY3OWnYHplcGt6VD01 PxCBX9IYGkPDpFH35HXAGKHI/ADU5Pgia6V8FJK7m09aFvvV2U5QIHxjVTzL7mga02Ap hj8g== X-Gm-Message-State: ANhLgQ257krOA4IGIDk1aJ9NOK5KRQxanNEG4JffdDydjCYie2Mg23X8 CQXSNINMhmHniDEe2/HIQUdcDSuG X-Google-Smtp-Source: ADFU+vvgrgVnbr/4BokaA86qoBWTSwIVJTPJbsjodhGBQGu9aJKWyp8CgUZVmZO46Bv5t76OXJdVHg== X-Received: by 2002:a9d:20c1:: with SMTP id x59mr18751197ota.286.1583863714666; Tue, 10 Mar 2020 11:08:34 -0700 (PDT) Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com. [209.85.167.171]) by smtp.gmail.com with ESMTPSA id o1sm16131799otl.49.2020.03.10.11.08.33 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Mar 2020 11:08:33 -0700 (PDT) Received: by mail-oi1-f171.google.com with SMTP id v19so14822495oic.12; Tue, 10 Mar 2020 11:08:33 -0700 (PDT) X-Received: by 2002:aca:3542:: with SMTP id c63mr2110985oia.135.1583863713744; Tue, 10 Mar 2020 11:08:33 -0700 (PDT) MIME-Version: 1.0 References: <9c7a8dea-ac8a-4d17-ed33-b6c4e882add8@digiware.nl> <936ed7c2-99d2-5df8-de3f-f64f28d2ba6f@digiware.nl> In-Reply-To: Reply-To: cem@freebsd.org From: Conrad Meyer Date: Tue, 10 Mar 2020 11:08:22 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC] Adding a Rados block driver to bhyve To: Willem Jan Withagen Cc: "freebsd-virtualization@freebsd.org" , FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 48cNPc6Y7cz3Cs5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of csecem@gmail.com designates 209.85.210.48 as permitted sender) smtp.mailfrom=csecem@gmail.com X-Spamd-Result: default: False [-3.15 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[48.210.85.209.rep.mailspike.net : 127.0.0.18]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; REPLYTO_ADDR_EQ_FROM(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[48.210.85.209.list.dnswl.org : 127.0.5.0]; HAS_REPLYTO(0.00)[cem@freebsd.org]; IP_SCORE(-1.15)[ip: (-1.05), ipnet: 209.85.128.0/17(-2.98), asn: 15169(-1.65), country: US(-0.05)]; FORGED_SENDER(0.30)[cem@freebsd.org,csecem@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[cem@freebsd.org,csecem@gmail.com] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Mar 2020 18:08:39 -0000 On Tue, Mar 10, 2020 at 9:28 AM Willem Jan Withagen wrote: > >> problem that libblock_rbd.so is stripped in such a way that > >> the symbol I need is removed. > > > So either I'm doing it the wrong way, like special options on the > > symbols oid. > > > However, I think there's something wrong with your > > library, too. The library should be usable even if it's stripped. Yes, strip only removes local symbols (.symtab), not exported ones (.dynsym). > I checked with objdump, and the symbol that I need is definitly not > present in the stripped version. How are you defining the symbol intended for export? Is the symbol a function or data? Does the compiler flag -fvisibility=hidden get used? Which symbol is missing and what are the symptoms? > And it does not really matter if I declare it static or not. Not declaring it "static" is necessary, if not sufficient. Looking at your code on github, here are some issues: * In block_if.h, you define an object blocklocal_backend. This is a header, and every compilation unit that pulls in the header will get its own copy of blocklocal_backend. You probably want 'extern block_backend_t blocklocal_backend;' instead. * You SET_DECLARE block_backend_set in block_if.c, but I think it needs to be in block_if.h. * There is some weirdness around linker sets being removed by the linker if they are empty, so you may want to add blockbackend_local to the linker set in the main program. (It's unclear to me why blockbackend_local is treated specially regardless.) However, I'm not quite sure why DATA_SET() in block_rbd.c is not creating __start_set_block_backend_set / __stop_set_block_backend_set exported symbols. As Alan asked, please provide 'nm -D foo.so'. Best, Conrad From owner-freebsd-virtualization@freebsd.org Wed Mar 11 09:11:56 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7818425A100 for ; Wed, 11 Mar 2020 09:11:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48cmRw1kmPz4fVK for ; Wed, 11 Mar 2020 09:11:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id F339D25A0FF; Wed, 11 Mar 2020 09:11:55 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F2E7725A0FE for ; Wed, 11 Mar 2020 09:11:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cmRv4czHz4fT6 for ; Wed, 11 Mar 2020 09:11:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6B8F1A388 for ; Wed, 11 Mar 2020 09:11:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02B9BtDw029802 for ; Wed, 11 Mar 2020 09:11:55 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02B9BtRk029793 for virtualization@FreeBSD.org; Wed, 11 Mar 2020 09:11:55 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 244733] pci passthrough doesn't work upgrading from 12.0 to 12.1 Date: Wed, 11 Mar 2020 09:11:55 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 12.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: nospam@ofloo.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Mar 2020 09:11:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D244733 Bug ID: 244733 Summary: pci passthrough doesn't work upgrading from 12.0 to 12.1 Product: Base System Version: 12.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: nospam@ofloo.net Created attachment 212325 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D212325&action= =3Dedit dmesg When I first install kernel and reboot pci passthrough still works. However after I've done installworld and mergemaster -iU reboot and I get this erro= r. In vm-bhyve log Mar 07 10:16:51: initialising Mar 07 10:16:51: [loader: bhyveload] Mar 07 10:16:51: [cpu: 4] Mar 07 10:16:51: [memory: 2048M] Mar 07 10:16:51: [hostbridge: standard] Mar 07 10:16:51: [com ports: com1] Mar 07 10:16:51: [uuid: ae337462-f766-11e7-8028-ac1f6b45bb3c] Mar 07 10:16:51: [utctime: yes] Mar 07 10:16:51: [debug mode: no] Mar 07 10:16:51: [primary disk: disk0] Mar 07 10:16:51: [primary disk dev: sparse-zvol] Mar 07 10:16:53: fatal; pci passthrough not supported on this system (no VT= -d or amdvi) acpidump shows #acpidump -t DMAR /* RSD PTR: OEM=3DSUPERM, ACPI_Rev=3D2.0x (2) XSDT=3D0x000000007e172098, length=3D36, cksum=3D143 */ /* XSDT: Length=3D180, Revision=3D1, Checksum=3D178, OEMID=3DSUPERM, OEM Table ID=3DSMCI--MB, OEM Revision=3D0x1072009, Creator ID=3DAMI, Creator Revision=3D0x10013 Entries=3D{ 0x000000007e176b58, 0x000000007e176c70, 0x000000007e176cb8, 0x000000007e176d58, 0x000000007e176da0, 0x000000007e176de0, 0x000000007e176= f90, 0x000000007e177008, 0x000000007e177038, 0x000000007e177070, 0x000000007e177= 0b8, 0x000000007e1789c0, 0x000000007e178a30, 0x000000007e178a68, 0x000000007e178= b10, 0x000000007e178b40, 0x000000007e178d70, 0x000000007e178ec0 } */ /* FACP: Length=3D276, Revision=3D6, Checksum=3D154, OEMID=3DSUPERM, OEM Table ID=3DSMCI--MB, OEM Revision=3D0x1072009, Creator ID=3DAMI, Creator Revision=3D0x10013 FACS=3D0x7e1be080, DSDT=3D0x0 INT_MODEL=3DAPIC Preferred_PM_Profile=3DEnterprise Server (4) SCI_INT=3D9 SMI_CMD=3D0xb2, ACPI_ENABLE=3D0xa0, ACPI_DISABLE=3D0xa1, S4BIOS_REQ=3D0= x0 PSTATE_CNT=3D0x0 PM1a_EVT_BLK=3D0x0-0x3 PM1a_CNT_BLK=3D0x0-0x1 PM_TMR_BLK=3D0x0-0x3 P_LVL2_LAT=3D101 us, P_LVL3_LAT=3D1001 us FLUSH_SIZE=3D1024, FLUSH_STRIDE=3D16 DUTY_OFFSET=3D1, DUTY_WIDTH=3D3 DAY_ALRM=3D13, MON_ALRM=3D0, CENTURY=3D0 IAPC_BOOT_ARCH=3D{LEGACY_DEVICES} =20=20=20 Flags=3D{WBINVD,C1_SUPPORTED,SLEEP_BUTTON,S4_RTC_WAKE,RESET_REGISTER,PLATFO= RM_CLOCK,REMOTE_POWER_ON} RESET_REG=3D0xcf9:0[8] (IO), RESET_VALUE=3D0xe */ /* FACS: Length=3D64, HwSig=3D0x00000060, Firm_Wake_Vec=3D0x00000000 Global_Lock=3D Flags=3D Version=3D2 */ acpidump: DSDT is corrupt When I rollback to 12.0 everything works again? So clearly it's not a bios setting. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Mar 11 09:26:57 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 63DD025A785 for ; Wed, 11 Mar 2020 09:26:57 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48cmnD61f9z49ZW for ; Wed, 11 Mar 2020 09:26:56 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id B5B5025A784; Wed, 11 Mar 2020 09:26:56 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B549525A783 for ; Wed, 11 Mar 2020 09:26:56 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48cmnC1Vycz49X6 for ; Wed, 11 Mar 2020 09:26:54 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: by mail-qk1-x731.google.com with SMTP id m2so1371734qka.7 for ; Wed, 11 Mar 2020 02:26:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=Ov8Ox0EQpFD3jFt8WbyboTicEImw7Y2yhT3WjXRIJa8=; b=nVJFiVuyLty9UsFmzcdgAkyVp0AXN2bjNgQsauHzHxojZkMywiy/hhDkbZjniohERd tGwP/Un02m2W2WJDJtcxzRSrNKcjlMI5Ol1D0W+fpduktRTCKiajfYDs1Xa2ZYUH5UE0 zz8i7Rnx4Xbf9d6gkQsvOZjkVpOp5jxXRSTiAhjSuy7/dUgsk7IJoumJicBBlpDOAASf j982CRrvLqJO2oKHNv/vh2PJFtGQiTQ63dSvddzCo49WeR0WYusOqqW/nECTr/hHvQx4 dkSxmfScOtltFljbvDxxTWP77RDcY/l7Pv+fFx/agnQ5i9+wgNnz7MTuKr4fN3acCJa5 VXPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Ov8Ox0EQpFD3jFt8WbyboTicEImw7Y2yhT3WjXRIJa8=; b=Xty1Ym6D0HChOFG4bTtysYtPUObb7hNnWWQ42G22toiLyBjpFt0KfpoPAD6xqr6XZi EjoS+X0oSCQ9UgLDvy+E07PgdAJFcTUkYzP3ZS+jFd3h8Z5lAQjgVIann+An0dh9DmMU wB+LRez1vl8eyG3H0TpDKD5ct/Z3zDycvQGTCoYCtaMeiuRvgVugbwZNSl2P9FL7aCtj v8rmAhfuebl5aIfCq/tL5WXpW2HKbSc5boLxVzc3f5tw4XS1EUetv5QPQejwc4I4j5HK 6aOOebuRmepjxpKygFAKPMMjOYZik9vmVSXXhGenu0qBK3ANz4P5OzVnlAeAICpXi8tu 3gcQ== X-Gm-Message-State: ANhLgQ0LS7yY9VXdyhPdQMNCqscQE8w0V2kDX4mdoj+zUK7Xy2IlvBVT KMF8NF1LwMD9R/k6d1pzNOsdjaNsXx2tT98I484+On3NPzs= X-Google-Smtp-Source: ADFU+vtg1xcL2L5jo4k/VovyyH5yDpvBY3x5NTgHNLmYOocS4ycBRfZ5Txjaj29VLODDpAiUL/0bnlhP6uM84HTVQ2s= X-Received: by 2002:ae9:f303:: with SMTP id p3mr1870974qkg.454.1583918813467; Wed, 11 Mar 2020 02:26:53 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?B?6Yqt5LiH5qOa?= Date: Wed, 11 Mar 2020 18:26:23 +0900 Message-ID: Subject: Patch for Nominal Media Rotation Rate field of AHCI disk. To: virtualization@freebsd.org Content-Type: multipart/mixed; boundary="0000000000003afb9705a090d69d" X-Rspamd-Queue-Id: 48cmnC1Vycz49X6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=nVJFiVuy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of wanpengqian@gmail.com designates 2607:f8b0:4864:20::731 as permitted sender) smtp.mailfrom=wanpengqian@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_ATTACHMENT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-9.16), ipnet: 2607:f8b0::/32(-1.85), asn: 15169(-1.65), country: US(-0.05)]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[1.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Mar 2020 09:26:57 -0000 --0000000000003afb9705a090d69d Content-Type: text/plain; charset="UTF-8" Hi all I have a patch for Nominal Media Rotation Rate field. currently bhyve will not report this field. setting nmrr=1 will let ahci controller report disk as Solid State Disk. other valid value will report as RPM of disk. ref document: http://www.t13.org/documents/UploadedDocuments/docs2016/di529r14-ATAATAPI_Command_Set_-_4.pdf 9.11.5.4 NOMINAL MEDIA ROTATION RATE field Best Regards Qian --0000000000003afb9705a090d69d Content-Type: application/octet-stream; name="nmrr.diff" Content-Disposition: attachment; filename="nmrr.diff" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_k7n4bqf40 ZGlmZiAtdSAtciAtTiB1c3Iuc2Jpbi9iaHl2ZS9ibG9ja19pZi5jIHVzci5zYmluL2JoeXZlbm1y ci9ibG9ja19pZi5jCi0tLSB1c3Iuc2Jpbi9iaHl2ZS9ibG9ja19pZi5jCTIwMjAtMDMtMTEgMTg6 MDk6MzEuNTU2MDgwMDAwICswOTAwCisrKyB1c3Iuc2Jpbi9iaHl2ZW5tcnIvYmxvY2tfaWYuYwky MDIwLTAzLTExIDE4OjA2OjE4LjY0MTE2NTAwMCArMDkwMApAQCAtMTAyLDYgKzEwMiw3IEBACiAJ aW50CQkJYmNfc2VjdHN6OwogCWludAkJCWJjX3BzZWN0c3o7CiAJaW50CQkJYmNfcHNlY3RvZmY7 CisJaW50CQkJYmNfbm1ycjsgIC8vIE5vbWluYWwgTWVkaWEgUm90YXRpb24gUmF0ZQogCWludAkJ CWJjX2Nsb3Npbmc7CiAJcHRocmVhZF90CQliY19idGlkW0JMT0NLSUZfTlVNVEhSXTsKIAlwdGhy ZWFkX211dGV4X3QJCWJjX210eDsKQEAgLTQwNyw3ICs0MDgsNyBAQAogCXN0cnVjdCBzdGF0IHNi dWY7CiAJc3RydWN0IGRpb2NnYXR0cl9hcmcgYXJnOwogCW9mZl90IHNpemUsIHBzZWN0c3osIHBz ZWN0b2ZmOwotCWludCBleHRyYSwgZmQsIGksIHNlY3RzejsKKwlpbnQgZXh0cmEsIGZkLCBpLCBz ZWN0c3osIG5tcnI7CiAJaW50IG5vY2FjaGUsIHN5bmMsIHJvLCBjYW5kZWxldGUsIGdlb20sIHNz b3B0LCBwc3NvcHQ7CiAjaWZuZGVmIFdJVEhPVVRfQ0FQU0lDVU0KIAljYXBfcmlnaHRzX3Qgcmln aHRzOwpAQCAtNDIxLDYgKzQyMiw3IEBACiAJbm9jYWNoZSA9IDA7CiAJc3luYyA9IDA7CiAJcm8g PSAwOworICAgICAgICBubXJyID0gMDsKIAogCS8qCiAJICogVGhlIGZpcnN0IGVsZW1lbnQgaW4g dGhlIG9wdHN0cmluZyBpcyBhbHdheXMgYSBwYXRobmFtZS4KQEAgLTQ0MSw2ICs0NDMsMTIgQEAK IAkJCTsKIAkJZWxzZSBpZiAoc3NjYW5mKGNwLCAic2VjdG9yc2l6ZT0lZCIsICZzc29wdCkgPT0g MSkKIAkJCXBzc29wdCA9IHNzb3B0OworCQllbHNlIGlmIChzc2NhbmYoY3AsICJubXJyPSVkIiwg Jm5tcnIpID09IDEpIHsKKwkJCWlmKG5tcnIgPCAweDAKKwkJCSAgfHwgKG5tcnIgPj0gMHgyICYm IG5tcnIgPD0gMHg0MDApCisJCQkgIHx8IG5tcnIgPj0gMHhGRkZGKQorCQkJCW5tcnIgPSAwOwor CQl9CiAJCWVsc2UgewogCQkJZnByaW50ZihzdGRlcnIsICJJbnZhbGlkIGRldmljZSBvcHRpb24g XCIlc1wiXG4iLCBjcCk7CiAJCQlnb3RvIGVycjsKQEAgLTU1Niw2ICs1NjQsNyBAQAogCWJjLT5i Y19zZWN0c3ogPSBzZWN0c3o7CiAJYmMtPmJjX3BzZWN0c3ogPSBwc2VjdHN6OwogCWJjLT5iY19w c2VjdG9mZiA9IHBzZWN0b2ZmOworCWJjLT5iY19ubXJyID0gbm1ycjsKIAlwdGhyZWFkX211dGV4 X2luaXQoJmJjLT5iY19tdHgsIE5VTEwpOwogCXB0aHJlYWRfY29uZF9pbml0KCZiYy0+YmNfY29u ZCwgTlVMTCk7CiAJVEFJTFFfSU5JVCgmYmMtPmJjX2ZyZWVxKTsKQEAgLTg0Niw0ICs4NTUsMTAg QEAKIAogCWFzc2VydChiYy0+YmNfbWFnaWMgPT0gQkxPQ0tJRl9TSUcpOwogCXJldHVybiAoYmMt PmJjX2NhbmRlbGV0ZSk7Cit9CisKK2ludAorYmxvY2tpZl9ubXJyKHN0cnVjdCBibG9ja2lmX2N0 eHQgKmJjKQoreworCXJldHVybiAoYmMtPmJjX25tcnIpOwogfQpkaWZmIC11IC1yIC1OIHVzci5z YmluL2JoeXZlL2Jsb2NrX2lmLmggdXNyLnNiaW4vYmh5dmVubXJyL2Jsb2NrX2lmLmgKLS0tIHVz ci5zYmluL2JoeXZlL2Jsb2NrX2lmLmgJMjAyMC0wMy0xMSAxODowOTozMS41NTk1NjMwMDAgKzA5 MDAKKysrIHVzci5zYmluL2JoeXZlbm1yci9ibG9ja19pZi5oCTIwMjAtMDMtMTEgMTc6NDQ6NDAu ODQzNjMxMDAwICswOTAwCkBAIC02OCw2ICs2OCw3IEBACiBpbnQJYmxvY2tpZl9xdWV1ZXN6KHN0 cnVjdCBibG9ja2lmX2N0eHQgKmJjKTsKIGludAlibG9ja2lmX2lzX3JvKHN0cnVjdCBibG9ja2lm X2N0eHQgKmJjKTsKIGludAlibG9ja2lmX2NhbmRlbGV0ZShzdHJ1Y3QgYmxvY2tpZl9jdHh0ICpi Yyk7CitpbnQJYmxvY2tpZl9ubXJyKHN0cnVjdCBibG9ja2lmX2N0eHQgKmJjKTsKIGludAlibG9j a2lmX3JlYWQoc3RydWN0IGJsb2NraWZfY3R4dCAqYmMsIHN0cnVjdCBibG9ja2lmX3JlcSAqYnJl cSk7CiBpbnQJYmxvY2tpZl93cml0ZShzdHJ1Y3QgYmxvY2tpZl9jdHh0ICpiYywgc3RydWN0IGJs b2NraWZfcmVxICpicmVxKTsKIGludAlibG9ja2lmX2ZsdXNoKHN0cnVjdCBibG9ja2lmX2N0eHQg KmJjLCBzdHJ1Y3QgYmxvY2tpZl9yZXEgKmJyZXEpOwpkaWZmIC11IC1yIC1OIHVzci5zYmluL2Jo eXZlL3BjaV9haGNpLmMgdXNyLnNiaW4vYmh5dmVubXJyL3BjaV9haGNpLmMKLS0tIHVzci5zYmlu L2JoeXZlL3BjaV9haGNpLmMJMjAyMC0wMy0xMSAxODowOTozMS41NTkzNTAwMDAgKzA5MDAKKysr IHVzci5zYmluL2JoeXZlbm1yci9wY2lfYWhjaS5jCTIwMjAtMDMtMTEgMTg6MTY6NTUuOTUyODg4 MDAwICswOTAwCkBAIC0xMDY5LDYgKzEwNjksNyBAQAogCQl9CiAJCWJ1ZlsxMTldID0gKEFUQV9T VVBQT1JUX1JXTE9HRE1BRVhUIHwgMSA8PCAxNCk7CiAJCWJ1ZlsxMjBdID0gKEFUQV9TVVBQT1JU X1JXTE9HRE1BRVhUIHwgMSA8PCAxNCk7CisJCWJ1ZlsyMTddID0gYmxvY2tpZl9ubXJyKHAtPmJj dHgpOwogCQlidWZbMjIyXSA9IDB4MTAyMDsKIAkJYnVmWzI1NV0gPSAweDAwYTU7CiAJCWFoY2lf Y2hlY2tzdW0oKHVpbnQ4X3QgKilidWYsIHNpemVvZihidWYpKTsKZGlmZiAtdSAtciAtTiB1c3Iu c2Jpbi9iaHl2ZS9wY2lfbnZtZS5jIHVzci5zYmluL2JoeXZlbm1yci9wY2lfbnZtZS5jCi0tLSB1 c3Iuc2Jpbi9iaHl2ZS9wY2lfbnZtZS5jCTIwMjAtMDMtMTEgMTg6MDk6MzEuNTU5MDMyMDAwICsw OTAwCisrKyB1c3Iuc2Jpbi9iaHl2ZW5tcnIvcGNpX252bWUuYwkyMDIwLTAzLTA5IDE1OjQyOjM5 LjQ5MTUxNDAwMCArMDkwMApAQCAtNDUxLDYgKzQ1MSw5IEBACiAKIAltZW1zZXQoJnNjLT5lcnJf bG9nLCAwLCBzaXplb2Yoc2MtPmVycl9sb2cpKTsKIAltZW1zZXQoJnNjLT5oZWFsdGhfbG9nLCAw LCBzaXplb2Yoc2MtPmhlYWx0aF9sb2cpKTsKK3NjLT5oZWFsdGhfbG9nLnRlbXBlcmF0dXJlPTMx MTsKK3NjLT5oZWFsdGhfbG9nLmF2YWlsYWJsZV9zcGFyZT0xMDA7CitzYy0+aGVhbHRoX2xvZy5h dmFpbGFibGVfc3BhcmVfdGhyZXNob2xkPTEwOwogCW1lbXNldCgmc2MtPmZ3X2xvZywgMCwgc2l6 ZW9mKHNjLT5md19sb2cpKTsKIH0KIAo= --0000000000003afb9705a090d69d-- From owner-freebsd-virtualization@freebsd.org Wed Mar 11 11:20:55 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 44FEA25DADA for ; Wed, 11 Mar 2020 11:20:55 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48cqJl0WvFz4cW0 for ; Wed, 11 Mar 2020 11:20:55 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id F12C625DAD9; Wed, 11 Mar 2020 11:20:54 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F0E0A25DAD8 for ; Wed, 11 Mar 2020 11:20:54 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48cqJj6SYZz4cSj for ; Wed, 11 Mar 2020 11:20:53 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito.onthenet.com.au (Postfix) with ESMTPS id D8DF620B8DA8 for ; Wed, 11 Mar 2020 21:20:48 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id BEBC620B49B1 for ; Wed, 11 Mar 2020 21:20:48 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id B949720E467D for ; Wed, 11 Mar 2020 21:20:48 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id nMOF7exjNVMa for ; Wed, 11 Mar 2020 21:20:48 +1000 (AEST) Received: from MacBook-Air-4.local (otn-120-29-62-248.broadband.onthenet.net [120.29.62.248]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 232FA20E467A; Wed, 11 Mar 2020 21:20:47 +1000 (AEST) Subject: Re: Patch for Nominal Media Rotation Rate field of AHCI disk. To: =?UTF-8?B?6Yqt5LiH5qOa?= References: Cc: virtualization@freebsd.org From: Peter Grehan Message-ID: <8d8c2855-d8ed-73d7-81cd-0bc1a85a7695@freebsd.org> Date: Wed, 11 Mar 2020 21:20:46 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=UIBp4xXy c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=__bySXl29Ds91t_vsvQA:9 a=QEXdDO2ut3YA:10 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=d81BNirE c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=__bySXl29Ds91t_vsvQA:9 a=QEXdDO2ut3YA:10 X-Rspamd-Queue-Id: 48cqJj6SYZz4cSj X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_LONG(-0.99)[-0.986,0]; NEURAL_SPAM_MEDIUM(0.99)[0.987,0]; ASN(0.00)[asn:9313, ipnet:203.22.124.0/23, country:AU] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Mar 2020 11:20:55 -0000 Hi Qian, > I have a patch for Nominal Media Rotation Rate field. > currently bhyve will not report this field. Nice ! I was thinking another way to do this would be to change the device emulation name for this i.e. "ata-ssd" instead of "ata-hd" with an additional parameter. That way some additional parameters and fields could be changed, such as the name presented in the identify device data. later, Peter. From owner-freebsd-virtualization@freebsd.org Thu Mar 12 00:18:06 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1656226ED47 for ; Thu, 12 Mar 2020 00:18:06 +0000 (UTC) (envelope-from erleya@gmail.com) Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48d8YS4shBz3KJJ for ; Thu, 12 Mar 2020 00:18:03 +0000 (UTC) (envelope-from erleya@gmail.com) Received: by mail-wr1-x443.google.com with SMTP id v9so5112445wrf.10 for ; Wed, 11 Mar 2020 17:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=iG1i2MHmP7F+ydPtabDxjHouoMtbmtZjo5EZu0SZ+1I=; b=uq5hfvbH0HWgYrpu/4zcvo5TbFk4WVgNNAQH5m/zAA3mGPHMTkIYt8FRzgvRrgIzoB 4yX/z/NusGyODeZcRKx4pSqfAUvrBF6tfCpaYf/TQz7by44l3YAMwOHk8lx9CNTBaYKR f9Wdo4KtKG6ERznO0T8ndjPD+5pwOIWnN+wOOK51Kc16A7ZmTX51jMdmM1WsZ8ab5dHb X25ndhnT7zH3q4cKEzPm45pmniVk/jz0kaFttF2sB8womLPItej3lzyC0WM+N9OtW4Li 2jjFFABrQopC0H1EtjmCZ69V3LlXVMzitFr4n3Yztt8mOipIr1Z1qda5J5cPfKBUn/S7 pjNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=iG1i2MHmP7F+ydPtabDxjHouoMtbmtZjo5EZu0SZ+1I=; b=fItldMqPms1Gd4QM9lTIx8NUPWJMuZE6gNrimnq7XKcSpDh00HXlNH1jhju/CmwTWx 8pQcbnGzfHzofteHM/y12z3nAhRWto0E59QkDPUzHmnYnypx0OqIoAv4llnIHRIVW3Bq Y+8P8IwAjT6JErdyz88bFlbBwsAXhxBWPuLyGVx2s3sA6rYBMAKlUIYwgQQjzqhc7+1d GMyKhAZ8vZaQZVWKfE/y3JdZV6Rp7pW+lweHgZgVsr6zZp7hdBb3N5KEOJKyXG+r8XLx hcr+3fXU3J3KNzKBKk7HCXYLsfJ0unf1EhDQdjP1i37ZIwYxeyYNoXCTPgQwL76WXc5j L0sg== X-Gm-Message-State: ANhLgQ2AVwUQfLzNs+5CgsDyMx/EtOLrLjlK5PE3SOZ7V99He4CO67Je mFqaSfQi1JzA8DCtr3YVh3IXivR2OBAuh/mKSQKRo18/ X-Google-Smtp-Source: ADFU+vvgtscBgELtAum6CLtaeTmiwEqjQl79LtOe24YU6CT6yXQ55ykBRU1OgxNk6s0xeszsAdin8LOvpmSH36lgs/k= X-Received: by 2002:adf:9071:: with SMTP id h104mr7169540wrh.359.1583972280954; Wed, 11 Mar 2020 17:18:00 -0700 (PDT) MIME-Version: 1.0 From: Alex Troussov Date: Thu, 12 Mar 2020 01:18:18 +0100 Message-ID: Subject: [GPU pass-through] no compatible bridge window for claimed BAR To: freebsd-virtualization@freebsd.org X-Rspamd-Queue-Id: 48d8YS4shBz3KJJ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=uq5hfvbH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of erleya@gmail.com designates 2a00:1450:4864:20::443 as permitted sender) smtp.mailfrom=erleya@gmail.com X-Spamd-Result: default: False [-1.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (2.39), ipnet: 2a00:1450::/32(-2.40), asn: 15169(-1.65), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[3.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; HTTP_TO_IP(1.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 00:18:06 -0000 Hello all, While trying to pass-through GPU card from Bhyve host (FreeBSD 12.1) to Debian Buster VM, I get error during VM boot like: > dmesg | grep "no compatible bridge window" pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff 64bit pref]: no compatible bridge window but GPU appears in VM: > lspci -v 00:01.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] Flags: fast devsel, IRQ 17 Memory at c1000000 (32-bit, non-prefetchable) [disabled] [size=16M] Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M] Memory at c2000000 (64-bit, prefetchable) [disabled] [size=32M] I/O ports at 2080 [disabled] [size=128] [virtual] Expansion ROM at 000c0000 [disabled] [size=128K] Kernel driver in use: nvidia Kernel modules: nvidia 00:01.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) Subsystem: NVIDIA Corporation GP106 High Definition Audio Controller Flags: fast devsel, IRQ 18 Memory at c4000000 (32-bit, non-prefetchable) [disabled] [size=16K] Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel and its nvidia driver is partially loaded ("nvidia-smi -L" doesn't detect it). On VM complete device tree is: > lspci -tv -[0000:00]-+-00.0 Network Appliance Corporation Device 1275 +-01.0 NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] +-01.1 NVIDIA Corporation GP106 High Definition Audio Controller +-02.0 Red Hat, Inc Virtio network device +-03.0 Red Hat, Inc Virtio block device +-1d.0 Device fb5d:40fb +-1e.0 Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller \-1f.0 Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] And my BHyve command-line parameters are: > bhyve -AHPS -c 4 -m 16G -u \ -s 0,hostbridge \ -s 1:0,passthru,1/0/0 -s 1:1,passthru,1/0/1 \ -s 2,virtio-net,tap0,mac=00:a0:98:fa:5c:8e \ -s 3,virtio-blk,/dev/zvol/attic/tuxdisk0 \ -s 29,fbuf,tcp=192.168.0.2:5900,w=640,h=480,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -l com1,stdio \ tux I tried to remove/rescan GPU card on VM with no success. On host system card is properly masked: > pciconv -lv ppt0@pci0:1:0:0: class=0x030000 card=0x11d710de chip=0x1c0310de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GP106 [GeForce GTX 1060 6GB]' class = display subclass = VGA ppt1@pci0:1:0:1: class=0x040300 card=0x11d710de chip=0x10f110de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GP106 High Definition Audio Controller' class = multimedia subclass = HDA What would be a good strategy to find a solution? >From what I can read from all the info above, somehow the requested memory space for BAR1 for that device is 0xd000000000-0xd00fffffff which is out of addressable space on the system. In the same time device reports prefetchable memory slot of 256 Mb starting from address 0xd0000000. Why it is different? Bhyve or UEFI bug? VM Linux config? Please help... -- Regards, Alex From owner-freebsd-virtualization@freebsd.org Thu Mar 12 00:31:47 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A05326F469 for ; Thu, 12 Mar 2020 00:31:47 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48d8sH0CdMz4BbX for ; Thu, 12 Mar 2020 00:31:46 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito.onthenet.com.au (Postfix) with ESMTPS id AF5DF20CA9C7 for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 95DCE20B49B0 for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 8FA4920E467D for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Dp1MNvduvkgR for ; Thu, 12 Mar 2020 10:31:41 +1000 (AEST) Received: from MacBook-Air-4.local (otn-120-29-62-248.broadband.onthenet.net [120.29.62.248]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id C328720E467A; Thu, 12 Mar 2020 10:31:40 +1000 (AEST) Subject: Re: [GPU pass-through] no compatible bridge window for claimed BAR To: Alex Troussov References: From: Peter Grehan Cc: freebsd-virtualization@freebsd.org Message-ID: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> Date: Thu, 12 Mar 2020 10:31:39 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=UIBp4xXy c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=c3PU01mjbo3xNLXCTk0A:9 a=D_dgQrR0NpD0Az2v:21 a=854ja8IJPeEXZL8-:21 a=QEXdDO2ut3YA:10 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=d81BNirE c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=c3PU01mjbo3xNLXCTk0A:9 a=D_dgQrR0NpD0Az2v:21 a=854ja8IJPeEXZL8-:21 a=QEXdDO2ut3YA:10 X-Rspamd-Queue-Id: 48d8sH0CdMz4BbX X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(0.99)[0.987,0]; ASN(0.00)[asn:9313, ipnet:203.22.124.0/23, country:AU]; NEURAL_HAM_LONG(-0.99)[-0.986,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 00:31:47 -0000 Hi Alex, >> dmesg | grep "no compatible bridge window" > pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff 64bit > pref]: no compatible bridge window ...> From what I can read from all the info above, > somehow the requested memory space for BAR1 for that device is > 0xd000000000-0xd00fffffff which is out of addressable space on the system. Yep, that's the issue, and it's a bhyve bug - there is no check to see if the 64-bit window is within the addressable range of the processor. A quick fix is to change the constant for that range in pci_emul.c #define PCI_EMUL_MEMBASE64 0xD000000000UL #define PCI_EMUL_MEMLIMIT64 0xFD00000000UL .. to a value that is within the address bits of the CPU, but also above guest DRAM. later, Peter. From owner-freebsd-virtualization@freebsd.org Thu Mar 12 11:18:24 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7C3C125D1B9 for ; Thu, 12 Mar 2020 11:18:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48dRCM0ByTz41Xm for ; Thu, 12 Mar 2020 11:18:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 8210125D1B8; Thu, 12 Mar 2020 11:18:22 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8106725D1B7 for ; Thu, 12 Mar 2020 11:18:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48dRCK04CFz41Wx for ; Thu, 12 Mar 2020 11:18:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AAB412460F for ; Thu, 12 Mar 2020 11:18:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02CBIK6V059198 for ; Thu, 12 Mar 2020 11:18:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02CBIKZD059197 for virtualization@FreeBSD.org; Thu, 12 Mar 2020 11:18:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Thu, 12 Mar 2020 11:18:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 11:18:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #41 from mail@rubenvos.com --- Another occurance this night: Mar 11 07:14:28 zfs01 ntpd[6009]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): expired less than 75 days ago Mar 12 03:05:45 zfs01 kernel: nvme1: cpl does not map to outstanding cmd Mar 12 03:05:45 zfs01 kernel: cdw0:00000000 sqhd:001e sqid:0001 cid:0016 p:0 sc:00 sct:0 m:0 dnr:0 Mar 12 03:05:45 zfs01 kernel: nvme1: Missing interrupt Mar 12 03:05:45 zfs01 kernel: nvme1: Resetting controller due to a timeout. Mar 12 03:05:45 zfs01 kernel: nvme1: resetting controller Mar 12 03:05:45 zfs01 kernel: nvme1: temperature threshold not supported Mar 12 03:05:45 zfs01 kernel: nvme1: aborting outstanding i/o Mar 12 03:06:15 zfs01 kernel: nvme1: Missing interrupt Mar 12 03:07:08 zfs01 kernel: nvme1: Resetting controller due to a timeout. Mar 12 03:07:08 zfs01 kernel: nvme1: resetting controller Mar 12 03:07:08 zfs01 kernel: nvme1: temperature threshold not supported Mar 12 03:07:08 zfs01 kernel: nvme1: aborting outstanding i/o Mar 12 03:07:08 zfs01 syslogd: last message repeated 9 times Mar 12 03:07:13 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Mar 12 03:07:13 zfs01 kernel: ena0: Trigger reset is on Mar 12 03:07:13 zfs01 kernel: ena0: device is going DOWN Mar 12 03:07:38 zfs01 kernel: nvme1: Missing interrupt Mar 12 03:08:05 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x344 Mar 12 03:08:05 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Mar 12 03:08:05 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Mar 12 03:08:05 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Mar 12 03:08:05 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Mar 12 03:08:05 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Mar 12 03:08:05 zfs01 kernel: ena0: device is going UP Mar 12 03:08:05 zfs01 kernel: ena0: link is UP Mar 12 03:09:15 zfs01 dhclient[816]: send_packet6: Network is down Mar 12 03:09:15 zfs01 dhclient[816]: dhc6: send_packet6() sent -1 of 52 byt= es Mar 12 03:09:50 zfs01 dhclient[67871]: send_packet: Network is down Mar 12 03:10:12 zfs01 syslogd: last message repeated 3 times Mar 12 03:14:23 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Mar 12 03:14:23 zfs01 kernel: ena0: Trigger reset is on Mar 12 03:14:23 zfs01 kernel: ena0: device is going DOWN Mar 12 03:15:18 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x176 Mar 12 03:15:19 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Mar 12 03:15:19 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Mar 12 03:15:19 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Mar 12 03:15:19 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Mar 12 03:15:19 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Mar 12 03:15:19 zfs01 kernel: ena0: device is going UP Mar 12 03:15:19 zfs01 kernel: ena0: link is UP Mar 12 03:15:23 zfs01 dhclient[816]: send_packet6: Network is down Mar 12 03:15:23 zfs01 dhclient[816]: dhc6: send_packet6() sent -1 of 52 byt= es root@zfs01:~ # ls -lahtuT /etc/periodic/security/ total 68 -r--r--r-- 1 root wheel 2.8K Mar 12 03:01:48 2020 security.functions -rwxr-xr-x 1 root wheel 2.3K Mar 12 03:01:48 2020 900.tcpwrap -rwxr-xr-x 1 root wheel 2.3K Mar 12 03:01:48 2020 800.loginfail -rwxr-xr-x 1 root wheel 1.9K Mar 12 03:01:48 2020 700.kernelmsg -rwxr-xr-x 1 root wheel 2.0K Mar 12 03:01:48 2020 610.ipf6denied -rwxr-xr-x 1 root wheel 2.2K Mar 12 03:01:48 2020 550.ipfwlimit -rwxr-xr-x 1 root wheel 2.1K Mar 12 03:01:48 2020 520.pfdenied -rwxr-xr-x 1 root wheel 1.9K Mar 12 03:01:48 2020 510.ipfdenied -rwxr-xr-x 1 root wheel 2.0K Mar 12 03:01:48 2020 500.ipfwdenied -rwxr-xr-x 1 root wheel 1.9K Mar 12 03:01:48 2020 410.logincheck -rwxr-xr-x 1 root wheel 1.9K Mar 12 03:01:48 2020 400.passwdless -rwxr-xr-x 1 root wheel 1.9K Mar 12 03:01:48 2020 300.chkuid0 -rwxr-xr-x 1 root wheel 2.3K Mar 12 03:01:48 2020 200.chkmounts -rwxr-xr-x 1 root wheel 2.2K Mar 12 03:01:24 2020 110.neggrpperm -rwxr-xr-x 1 root wheel 2.2K Mar 12 03:01:00 2020 100.chksetuid drwxr-xr-x 2 root wheel 512B Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:~ # ls -lahtuT /etc/periodic/daily/ total 128 -rwxr-xr-x 1 root wheel 811B Mar 12 03:12:55 2020 999.local -rwxr-xr-x 1 root wheel 2.8K Mar 12 03:12:55 2020 800.scrub-zfs -rwxr-xr-x 1 root wheel 845B Mar 12 03:12:55 2020 510.status-world-kern= el -rwxr-xr-x 1 root wheel 737B Mar 12 03:12:55 2020 500.queuerun -rwxr-xr-x 1 root wheel 498B Mar 12 03:12:55 2020 480.status-ntpd -rwxr-xr-x 1 root wheel 451B Mar 12 03:02:07 2020 480.leapfile-ntpd -rwxr-xr-x 1 root wheel 2.0K Mar 12 03:01:57 2020 460.status-mail-rejec= ts -rwxr-xr-x 1 root wheel 1.0K Mar 12 03:01:00 2020 450.status-security -rwxr-xr-x 1 root wheel 1.4K Mar 12 03:01:00 2020 440.status-mailq -rwxr-xr-x 1 root wheel 705B Mar 12 03:01:00 2020 430.status-uptime -rwxr-xr-x 1 root wheel 611B Mar 12 03:01:00 2020 420.status-network -rwxr-xr-x 1 root wheel 684B Mar 12 03:01:00 2020 410.status-mfi -rwxr-xr-x 1 root wheel 590B Mar 12 03:01:00 2020 409.status-gconcat -rwxr-xr-x 1 root wheel 590B Mar 12 03:01:00 2020 408.status-gstripe -rwxr-xr-x 1 root wheel 591B Mar 12 03:01:00 2020 407.status-graid3 -rwxr-xr-x 1 root wheel 596B Mar 12 03:01:00 2020 406.status-gmirror -rwxr-xr-x 1 root wheel 807B Mar 12 03:01:00 2020 404.status-zfs -rwxr-xr-x 1 root wheel 583B Mar 12 03:01:00 2020 401.status-graid -rwxr-xr-x 1 root wheel 773B Mar 12 03:01:00 2020 400.status-disks -rwxr-xr-x 1 root wheel 724B Mar 12 03:01:00 2020 330.news -r-xr-xr-x 1 root wheel 1.4K Mar 12 03:01:00 2020 310.accounting -rwxr-xr-x 1 root wheel 693B Mar 12 03:01:00 2020 300.calendar -rwxr-xr-x 1 root wheel 1.0K Mar 12 03:01:00 2020 210.backup-aliases -rwxr-xr-x 1 root wheel 1.7K Mar 12 03:01:00 2020 200.backup-passwd -rwxr-xr-x 1 root wheel 603B Mar 12 03:01:00 2020 150.clean-hoststat -rwxr-xr-x 1 root wheel 1.0K Mar 12 03:01:00 2020 140.clean-rwho -rwxr-xr-x 1 root wheel 709B Mar 12 03:01:00 2020 130.clean-msgs -rwxr-xr-x 1 root wheel 1.1K Mar 12 03:01:00 2020 120.clean-preserve -rwxr-xr-x 1 root wheel 1.5K Mar 12 03:01:00 2020 110.clean-tmps -rwxr-xr-x 1 root wheel 1.3K Mar 12 03:01:00 2020 100.clean-disks drwxr-xr-x 2 root wheel 1.0K Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:~ # --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Mar 12 13:11:22 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0CA72260B3F for ; Thu, 12 Mar 2020 13:11:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48dTjj3nz4z45VQ for ; Thu, 12 Mar 2020 13:11:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 7D254260B3E; Thu, 12 Mar 2020 13:11:21 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7C663260B3D for ; Thu, 12 Mar 2020 13:11:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48dTjh5rflz45TC for ; Thu, 12 Mar 2020 13:11:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9C9DA25AC7 for ; Thu, 12 Mar 2020 13:11:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02CDBKlb040423 for ; Thu, 12 Mar 2020 13:11:20 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02CDBKG3040420 for virtualization@FreeBSD.org; Thu, 12 Mar 2020 13:11:20 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Thu, 12 Mar 2020 13:11:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cperciva@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 13:11:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #42 from Colin Percival --- Can you try setting hw.nvme.timeout_period in /boot/loader.conf? In particular: 1. Setting it to "1" may result in more errors being logged but less hangin= g, and 2. Setting it to "3600" (or something similarly big) may result in fewer er= rors but longer hangs. I don't think either of these is a solution but it would be very enlighteni= ng to know if either of these has the anticipated effect. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Mar 12 13:19:19 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 519E4260D6A for ; Thu, 12 Mar 2020 13:19:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48dTtv0538z4S7k for ; Thu, 12 Mar 2020 13:19:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id E0766260D69; Thu, 12 Mar 2020 13:19:18 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DFF3C260D68 for ; Thu, 12 Mar 2020 13:19:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48dTtt2jWWz4S6k for ; Thu, 12 Mar 2020 13:19:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4E81C25C5F for ; Thu, 12 Mar 2020 13:19:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02CDJIG8054063 for ; Thu, 12 Mar 2020 13:19:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02CDJIu2054059 for virtualization@FreeBSD.org; Thu, 12 Mar 2020 13:19:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Thu, 12 Mar 2020 13:19:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 13:19:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #43 from mail@rubenvos.com --- (In reply to Colin Percival from comment #42) Hi Colin, I've just added this entry to /boot/loader.conf: hw.nvme.timeout_period=3D1 and rebooted the instance. Will provide feedback on perceived effects. Kind Regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Mar 12 14:25:17 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 05A9D26224F for ; Thu, 12 Mar 2020 14:25:17 +0000 (UTC) (envelope-from erleya@gmail.com) Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48dWM05c2Dz4gJ8; Thu, 12 Mar 2020 14:25:16 +0000 (UTC) (envelope-from erleya@gmail.com) Received: by mail-wr1-x436.google.com with SMTP id m9so7699302wro.12; Thu, 12 Mar 2020 07:25:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=VBcLRiPH2QBZYvCoGmfvom4ON2Uv+P3pLfdBwBTlp8A=; b=A4Jqv5f4GTiGK+IgXCJhQNq6BMq0JPYBSckGF5u3SROvE3J/Tb3PfyRPOvjJQwyrtP L8VMSiPXEp46qoaPValHVdt10Mo9cf6x9zalmoMp8guJu/u1wUlAAuzVtITyAfsD5wwM xLK/pLlGxJ/PpbKreussK10S/63hQjV9Jf1WEA50cA7SO3ZvWmQ09lXFA/9qxI1ktCHB 68qCHhcYGiicJ1e19wDZ/BMV1mVa5bk3fwOtob0ww02JXoeG/MHEar/McohT4oENNLYc 0a69naZt7MZeeizQMM/4RlRfFQQ/Ou/CcqCBSvPK+Kh7ck05EWia7Tk7tFFj3mqmgFQb DZug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=VBcLRiPH2QBZYvCoGmfvom4ON2Uv+P3pLfdBwBTlp8A=; b=CJZHKQOEOL9+kVHbdhNFzejYGdFIAqtaT/TY2fJxopmKB+tOOjP/pQODgfbWHGzoK4 CZVVSmLGCCgC0bJDI4T+3tK0eiY47w++I/4jUNyw7B7CRx5rk8FGYR43Nr1xj/OXPcFQ H7c0OwTPj1vsV1YX2TwTTqCSiA8CFty3GFEcPvXenlp8RvrP3qXN11ptXnQ2cDgB6EIP lg06ZkTVDdWU24miz3becRATy5FOhxS2CI1NCxADv6pnyoByIQaZmJXPUDnlhRfEjNk9 NbybEQf9ST9moUalRmGiM3zdx2W/OB6AMO206WDEE4rf1CRHfBN5t1j/2oFKG86HY/Cr PwBg== X-Gm-Message-State: ANhLgQ13Ae2PXoYMW2YOlWvYBuNBloBQ54cX+VQUq2HgwomYa2fkMyc0 bdCoHOmVIstxPWFHDMdOGUSruNt8sz4= X-Google-Smtp-Source: ADFU+vvOZeJDk+h1o0qA9qs93L5Q7kH8f4gsz9DWA8CDVZ7I71XoRO550Ap2h/xi5bG+zew5GOSEyg== X-Received: by 2002:a5d:4711:: with SMTP id y17mr11767554wrq.358.1584023114950; Thu, 12 Mar 2020 07:25:14 -0700 (PDT) Received: from erley.ru (erley.ru. [83.153.157.67]) by smtp.gmail.com with ESMTPSA id i1sm57975796wrs.18.2020.03.12.07.25.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 12 Mar 2020 07:25:14 -0700 (PDT) Received: by erley.ru (OpenSMTPD) with ESMTPSA id 2df0d96a (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Thu, 12 Mar 2020 15:25:13 +0100 (CET) Subject: Re: [GPU pass-through] no compatible bridge window for claimed BAR To: Peter Grehan Cc: freebsd-virtualization@freebsd.org References: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> From: Alex Erley Message-ID: <9b26dee7-8656-b5ba-5d72-9a01638ee438@gmail.com> Date: Thu, 12 Mar 2020 15:25:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48dWM05c2Dz4gJ8 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.74 / 15.00]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; REPLY(-4.00)[]; NEURAL_SPAM_MEDIUM(0.26)[0.257,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 14:25:17 -0000 Hello Peter, Many thanks for your insight! Changing PCI_EMUL_MEMBASE64 from 0xD000000000 to 0x0440000000 (16Gb guest memory limit) and PCI_EMUL_MEMLIMIT64 from 0xFD00000000 to 0x07ffffffff (32Gb host memory limit) still fails. From guest dmesg: ... [mem 0xc0000000-0xffffffff] available for PCI devices ... pci 0000:00:01.0: [10de:1c03] type 00 class 0x030000 pci 0000:00:01.0: reg 0x10: [mem 0xc0000000-0xc0ffffff] pci 0000:00:01.0: reg 0x14: [mem 0x440000000-0x44fffffff 64bit pref] pci 0000:00:01.0: reg 0x1c: [mem 0xc2000000-0xc3ffffff 64bit pref] pci 0000:00:01.0: reg 0x24: [io 0x2000-0x207f] pci 0000:00:01.0: reg 0x30: [mem 0xd3000000-0xd307ffff pref] pci 0000:00:01.1: [10de:10f1] type 00 class 0x040300 pci 0000:00:01.1: reg 0x10: [mem 0xc4000000-0xc4003fff] ... pci 0000:00:01.0: can't claim BAR 1 [mem 0x440000000-0x44fffffff 64bit pref]: no compatible bridge window ... pci 0000:00:01.0: BAR 1: no space for [mem size 0x10000000 64bit pref] pci 0000:00:01.0: BAR 1: trying firmware assignment [mem 0x440000000-0x44fffffff 64bit pref] pci 0000:00:01.0: BAR 1: assigned [mem 0x440000000-0x44fffffff 64bit pref] It seems guest requires BAR to be within its addressable space. Changing PCI_EMUL_MEMBASE64 to 0x00c0000000 (as guest suggests in its dmesg) didn't work either: bhyve: failed to initialize BARs for PCI 1/0/0 device emulation initialization error: Cannot allocate memory From the BHyve sources it becomes evident that when claimed BAR size is <= 32Mb, it is mandatory allocated as 32-bit BAR. The only case BAR goes for 64-bit allocation is when its size > 32Mb. In my case it is 256 Mb and it fails only for this BAR. I think the 64-bit BAR allocation still has to be fixed in BHyve code. Anyway, we are getting closer :) Any ideas are really appreciated. Have a nice day, Alex On 3/12/20 1:31 AM, Peter Grehan wrote: > Hi Alex, > >>> dmesg | grep "no compatible bridge window" >> pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff >> 64bit pref]: no compatible bridge window > ...> From what I can read from all the info above, >> somehow the requested memory space for BAR1 for that device is >> 0xd000000000-0xd00fffffff which is out of addressable space on the >> system. > > Yep, that's the issue, and it's a bhyve bug - there is no check to > see if the 64-bit window is within the addressable range of the > processor. > > A quick fix is to change the constant for that range in pci_emul.c > > #define PCI_EMUL_MEMBASE64 0xD000000000UL #define > PCI_EMUL_MEMLIMIT64 0xFD00000000UL > > .. to a value that is within the address bits of the CPU, but also > above guest DRAM. > > later, > > Peter. From owner-freebsd-virtualization@freebsd.org Thu Mar 12 19:03:53 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CB454268BDB for ; Thu, 12 Mar 2020 19:03:53 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48ddXS6vbwz4ZL6 for ; Thu, 12 Mar 2020 19:03:52 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id E067D22402 for ; Thu, 12 Mar 2020 15:03:51 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Thu, 12 Mar 2020 15:03:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:mime-version:content-type; s= fm2; bh=7f4KtC9SNt2pUGm7Hz2ugiUG5syfSUelIOn6lAZJ+cQ=; b=KH2RFDsP Ol/WfGpvQdb9vhi7zeTL0kw5QOJEFWBY1SDnNaNCmChldtC8LlaxsiIaaFyUTAsY CFpwFNRGWSqVChD6wp3b9an4QdWBUb+iNJmv7UP7BCBvll2P23j5ZhfxLqsmNInC t5436EYzjraSlYIadhMjaN9xgkkLD2nuYkYAZm1ymej/xj9a/o5bUdhcDWEKdYoS JK9nNoi5ZAetBoYyS2iGwtYvfzmmp7RmveJRTCMoIVfZWv7D+ePVE69SL+KxrbSQ eZRk5xo+2RUyIpxvIFBGIc/sLuBjao1jyz3RZXyd3D16FLrMs8MZQ3d3LAbc72lE aJjDNtNq2LyeHw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=7f4KtC9SNt2pUGm7Hz2ugiUG5syfS UelIOn6lAZJ+cQ=; b=x2XFI9imyeZRQ07K+FM94b+mM45DK//NhtdxmjSXTXCUo YpyWUlhkX9++CgWUFxKP/XhLzOhQRWAp835eUG3OQ4fcALLOGhYXQM8NF1zs7qbE g1svAkzn1P3Dbagwk7KJ51IAvmC7dIRxmF/kmQFKGNyDmsPihSUIKivqv6aenc90 MojquVhb8zmASTjsfHed+12XpUezYsh0+2Qie3dL1oi0Yx4AF5g5LZZkp55iK7Fs ou423ZP2nrtgtUYK90RCgJUjaFxjhos2HjFxhsDF5whLGQwT7XZNw2oz74cKjvqj 8ciqlpYRm26WDRN127Yo7BQk1FVGKz4fXXniSwPhQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedugedruddvhedguddvvdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehgtderre dttddvnecuhfhrohhmpehtvggthhdqlhhishhtshcuoehtvggthhdqlhhishhtshesiiih gihsthdrnhgvtheqnecukfhppeekvddrjedtrdeluddrleelnecuvehluhhsthgvrhfuih iivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepthgvtghhqdhlihhsthhsseiihiig shhtrdhnvght X-ME-Proxy: Received: from bastion.zyxst.net (bastion.zyxst.net [82.70.91.99]) by mail.messagingengine.com (Postfix) with ESMTPA id 6A94F306130A for ; Thu, 12 Mar 2020 15:03:51 -0400 (EDT) Date: Thu, 12 Mar 2020 19:03:22 +0000 From: tech-lists To: freebsd-virtualization@freebsd.org Subject: freebsd vm on azure - how to backup the vm as an image Message-ID: <20200312190322.GG58645@bastion.zyxst.net> Mail-Followup-To: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="NY6JkbSqL3W9mApi" Content-Disposition: inline X-Rspamd-Queue-Id: 48ddXS6vbwz4ZL6 X-Spamd-Bar: -------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=KH2RFDsP; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=x2XFI9im; dmarc=none; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 66.111.4.27 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-8.11 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-0.92)[-0.921,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; SIGNED_PGP(-2.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-3.49)[ip: (-9.84), ipnet: 66.111.4.0/24(-4.89), asn: 11403(-2.69), country: US(-0.05)] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2020 19:03:53 -0000 --NY6JkbSqL3W9mApi Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, Is there a way (yet) of backing up a freebsd-12 vm hosted on azure? Of cour= se on bhyve it's simple, turn off the vm and download it. But on Azure? thanks, --=20 J. --NY6JkbSqL3W9mApi Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAl5qh40ACgkQs8o7QhFz NAXMZRAAmfSxUVOW6lI1QHUbeuYkYXrD5gWupDJyVVV+OsYqXC/aOxzs7THDLiIB Zia6LD3SLa1fFhz9Yd+W9pCbnZHy9t0VU9iqU74WbMqd2kNWbzxw1QP/A114QV70 rpaOuejUsGQlwfWiDIUEPYPrPwqYtQG3a2MYV6FnmRiN0ILI2EJgQwzOkHMPj/+B xmXhmVzpQ9xuZ8r7++x8sZB8j2xDskQrcOGDFz6MJMcIYWeCuaY+6seNMhbCnths pQR9oIbvLCoyq0NhJsE6SjmaHMrQsIIidSAiEeC3/WB5z6V5lmAKkld7UgBsQwUI QZGWHP/lSCd5h37/DQoTPH89jZxA8EMYFa9PRHL6oX1tFvnnOGe9g/OQO6ODMedD zs1l5KNuCof7lbMTB/DbyE7bfxIi93lT5CpRHQ7d3VG7QTTJIhziDkjhZbD8OtnQ qdvXy9Mq0uNVO1+FCmosLcpZy5VcNkAhasGchJ+JQlKg2/uV6z2uhDjg08ZZM3Kf 4XmCRRiIsjLEVZmiqFlkZnA8DmIRZDU4mPZJNjyBTd9m7VZ32icUmnvI/5WTfTkL oc76BsXfOjuI0zfLtawXJxxaaHHK7cg2OP9uAs1PXuOJ1rslbGzVb7XyqQbYWdVO gNxsfmONqiCVtorBivzppc25aHNCa58foaXr3Yq/8bo7IBd9PDc= =ule3 -----END PGP SIGNATURE----- --NY6JkbSqL3W9mApi-- From owner-freebsd-virtualization@freebsd.org Fri Mar 13 01:55:12 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38850271546 for ; Fri, 13 Mar 2020 01:55:12 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: from mail-qt1-x844.google.com (mail-qt1-x844.google.com [IPv6:2607:f8b0:4864:20::844]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48dpg12Y9hz4Nxl for ; Fri, 13 Mar 2020 01:55:09 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: by mail-qt1-x844.google.com with SMTP id 59so6284142qtb.1 for ; Thu, 12 Mar 2020 18:55:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=yJfIexU29GZIN6dUMCNY1agtqnkkr9Oo8SnJwzz1zhg=; b=sKDknldLHXIfchxOClC+5bT5rSB4xMS+7D6oiBh8LDQhcCY6loyqRFD4Ewqirjr3cJ Tnb31jR2DiXe2F6cGVP6sFGM7R7qTineMAfQjE14ZHijOSIsZeQZWqObtgT83bZRBOn3 x70Aw0908t4wtkJR4IA9UYM7+x+KVaETv4cN65P+kZERvz/W4flYjSKVan6ZQsmz9hrw wSLhF+htne8R43bokhLBFl0EHzhZ9hZ/HrcuLpEr88+/Wytr5SKs3YUznBsRNeVHOvRA w0xOeUK8BisAb+LO2QRQEON0UZhEtOfWiyalGSNaGg3KgK6lXazIfecmQCR2aWEkFIXB i49Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=yJfIexU29GZIN6dUMCNY1agtqnkkr9Oo8SnJwzz1zhg=; b=hl8g0Q+KweNPG6blE0SM6Yi3Up2QE1oYYftloUX8SFB5qMFaPPxpfRG+V0Rf/XEY1x nvkTsJinqdH3zVtneDyAHcR7349kDuGT6GhxqTKQBZsbyBjKspBzDZ4GDuWTwg7HmQg0 ndW2RKU+jNSp9DVwjDju/G4uzUTjPOrdnwjWd1f6M7q/bENVTBOzfSOyaPkk0EL9oDNO T0YxdzaaTvd0ceWUdygA6adCU62VN6ZAGR5xAPpmlOMw5stedbx8S5y6xGtRfHCWyS6R KDq6N1388jTZpU+0ZAOo8yFXQHCMauH7DwjZ2ErMmU34CNrdjBuFFPkYi+MPa3Pm05gH Pw9w== X-Gm-Message-State: ANhLgQ0u+Mf9QVGzHg1SVXo+vzfmXYX8ObDGyLvhZhX9OEyI6A0in7Ya R2tv/Uvxqg05GWguoAIYMQNJ9ZXJuA5Zr8GJ3cbx6G8zIa0= X-Google-Smtp-Source: ADFU+vs9LpCjoiSGguZBjpwzJSYoVeHLdKK4isL2DLh5rEKBNq+NKFpki8G9eTraXGozOPlr9UmQi86QEfcdSOkdYpY= X-Received: by 2002:ac8:3056:: with SMTP id g22mr10210829qte.209.1584064507659; Thu, 12 Mar 2020 18:55:07 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?B?6Yqt5LiH5qOa?= Date: Fri, 13 Mar 2020 10:54:47 +0900 Message-ID: Subject: Passthru a VF to Windows guest cause BSOD, chelsio T520BT, SR-IOV To: FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 48dpg12Y9hz4Nxl X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=sKDknldL; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of wanpengqian@gmail.com designates 2607:f8b0:4864:20::844 as permitted sender) smtp.mailfrom=wanpengqian@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(0.00)[ip: (1.94), ipnet: 2607:f8b0::/32(-1.85), asn: 15169(-1.65), country: US(-0.05)]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[4.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Mar 2020 01:55:12 -0000 Hi all. I am trying to passthru a VF device to bhyve guest. Host configuration: SuperMicro X10SRM-F XEON E5 1680v3, 128G RAM root@fbsd13:~ # uname -a FreeBSD fbsd13 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r358902: Thu Mar 12 04:24:24 UTC 2020 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 I can create 2 VFs from host, one for host, one for passthru. root@fbsd13:~ # cat /etc/cxl0.conf PF {device:"cxl0";num_vfs:2;} VF-0 {passthrough:false;} VF-1 {passthrough:true;} Both VFs are fine, VF0 on host can communicate outside and I can pass the VF1 to a FreeBSD 12.1 guest, also works. When I pass the same VF1 to Windows 10 guest, ver 1909, the guest got BSOD during installation. I also try to add VF1 post installation. also got BSOD. the message is IRQL_NOT_LESS_OR_EQUAL. I also try Windows server 2012, during installation of Chelsio Driver, BSOD too. Any suggestions? driver issue or bhyve issue? Thanks. From owner-freebsd-virtualization@freebsd.org Fri Mar 13 02:06:24 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 47630271730 for ; Fri, 13 Mar 2020 02:06:24 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48dpvy5kTDz4f1p for ; Fri, 13 Mar 2020 02:06:22 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito.onthenet.com.au (Postfix) with ESMTPS id 82AE820D1599 for ; Fri, 13 Mar 2020 12:06:16 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 6771C20B49B2 for ; Fri, 13 Mar 2020 12:06:16 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 5B1D1208D487 for ; Fri, 13 Mar 2020 12:06:16 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 65n1VSKg65-b for ; Fri, 13 Mar 2020 12:06:16 +1000 (AEST) Received: from MacBook-Air-4.local (otn-120-29-62-248.broadband.onthenet.net [120.29.62.248]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id B1D6320C92F4; Fri, 13 Mar 2020 12:06:15 +1000 (AEST) Subject: Re: Passthru a VF to Windows guest cause BSOD, chelsio T520BT, SR-IOV To: =?UTF-8?B?6Yqt5LiH5qOa?= References: From: Peter Grehan Cc: FreeBSD virtualization Message-ID: <8357fd6d-d02d-76bb-712a-e5c330ecf434@freebsd.org> Date: Fri, 13 Mar 2020 12:06:14 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=UIBp4xXy c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=p1A8HRnZwKVOAvT7yiMA:9 a=QEXdDO2ut3YA:10 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=d81BNirE c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=SS2py6AdgQ4A:10 a=p1A8HRnZwKVOAvT7yiMA:9 a=QEXdDO2ut3YA:10 X-Rspamd-Queue-Id: 48dpvy5kTDz4f1p X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-0.84 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_SPAM_MEDIUM(0.10)[0.101,0]; ASN(0.00)[asn:9313, ipnet:203.22.124.0/23, country:AU]; NEURAL_HAM_LONG(-0.94)[-0.940,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Mar 2020 02:06:24 -0000 > When I pass the same VF1 to Windows 10 guest, ver 1909, the guest got > BSOD during installation. I also try to add VF1 post installation. > also got BSOD. > the message is IRQL_NOT_LESS_OR_EQUAL. > I also try Windows server 2012, during installation of Chelsio Driver, BSOD too. > > Any suggestions? driver issue or bhyve issue? Can you try disabling APIC acceleration (which should be available on that Xeon E5) ? This can be done on the fly using kenv and unloading/reloading vmm.ko (or loader.conf and rebooting) kenv hw.vmm.vmx.use_tpr_shadowing=0 kldunload vmm kldload vmm later, Peter. From owner-freebsd-virtualization@freebsd.org Fri Mar 13 03:51:45 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1BC742731B6 for ; Fri, 13 Mar 2020 03:51:45 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48dsFX6HZyz4JxF; Fri, 13 Mar 2020 03:51:44 +0000 (UTC) (envelope-from wanpengqian@gmail.com) Received: by mail-qt1-x835.google.com with SMTP id n5so6464394qtv.7; Thu, 12 Mar 2020 20:51:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=MqYQEfkEY7sCb9UYsuT4drPXIBgSgnfKLT5H6kv2J3k=; b=HvKjRbSSB2V1VG3Z1N9VRVvm2dsBqNrb/fat0n5lKYhQUbxESXpuM8PCrKXjwg+7Da eCF+pkZk6xi11LMaFPUA5Jp50EeShQ3r1Gzz64X7ICpGq5h7+y9nVEY+r3mEoTMtoijc QIodVsmETRgytsxxlCbjMVaM0rvjjbrYnYdBdgcPQ4Fd23KFGinlbw2c/PZiZMl6yadZ /J9HI3ePYrk2eYQxYs7MQVwqIBvc+MM9utOSUthcyEMkki2vZl4gnSj8VcWa9vxrz3N0 QyOf94WITE+ueRdRidXSQTgyJ2X/xf2HqrygKGFMPkdU2GCokjP1YaPIVQAaj5GWdOul XbtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=MqYQEfkEY7sCb9UYsuT4drPXIBgSgnfKLT5H6kv2J3k=; b=P52DDbYDdyUhiyCAIZ0PpVV73MKEUGRuqnAOc2/WHhQKO0xtedIiJlhYrpD5zmfU8F 7A1bwvuSpcVkCiuxC+jmWNn2EF6mk0lNCKFjlGqYifEpXyDDkyYWhJr+6Zc6RkiXLQkz sf1dIykJrACJeulVkvR1eg8tN7+HjVURm+u/Yd9rtKqXMZCjWPVLCRBb966N5qmUlL74 BaADkiTa9AESj6T4YZmgyPJncMwwfVmrdTi4XYH6O2j7XkCsfA3Xryf5r335zeyjNIOp KIgUNYzZwpRdJPgW2yPxMi3pWrYulruOounLN1jksYC77qDtHCi9lNktKpVT567jLs3R Wo4Q== X-Gm-Message-State: ANhLgQ1lmWM6K3SeWln5EgTTbUS550ntnXSC2NCvU0H9x4Kk6+DsGVXL t0tpJl+nvPrwse3+/qVMIG/zVbJBzSF9/Q0c79oEufex598= X-Google-Smtp-Source: ADFU+vsuulc2TMHk83AZeDwks7tbeUSKKaIeyfCuu+E2EeiJAj47478wK3Y2mifKvSYM1IX23H2VkkR8IRFvj8oDMdo= X-Received: by 2002:ac8:5448:: with SMTP id d8mr10290480qtq.205.1584071502702; Thu, 12 Mar 2020 20:51:42 -0700 (PDT) MIME-Version: 1.0 References: <8357fd6d-d02d-76bb-712a-e5c330ecf434@freebsd.org> In-Reply-To: <8357fd6d-d02d-76bb-712a-e5c330ecf434@freebsd.org> From: Wanpeng Qian Date: Fri, 13 Mar 2020 12:51:21 +0900 Message-ID: Subject: Re: Passthru a VF to Windows guest cause BSOD, chelsio T520BT, SR-IOV To: Peter Grehan Cc: FreeBSD virtualization Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 48dsFX6HZyz4JxF X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Mar 2020 03:51:45 -0000 Hi Peter > Can you try disabling APIC acceleration (which should be available on > that Xeon E5) ? This can be done on the fly using kenv and > unloading/reloading vmm.ko (or loader.conf and rebooting) After disabling tpr_shadowing, the Windows 10 1909 client boots without the IRQL_NOT_LESS_OR_EQUAL BSoD, (Windows 10 is very slow without tpr shadowing) But the driver didn't work. With Windows 10 built in driver, device status is : Windows is still setting up the class configuration for this device. (Code 56) When installing chelsio driver, cause BSoD, which is SYSTEM_THREAD_EXCEPTION_NOT_HANDLE (chnetx64.sys) I also try Windows 2012, same as Windows 10, BSoD while installing driver. And Windows 7 Pro, the chelsio driver didn't have a valid digial signed, not working. Thanks Qian On Fri, Mar 13, 2020 at 11:06 AM Peter Grehan wrote: > > > When I pass the same VF1 to Windows 10 guest, ver 1909, the guest got > > BSOD during installation. I also try to add VF1 post installation. > > also got BSOD. > > the message is IRQL_NOT_LESS_OR_EQUAL. > > I also try Windows server 2012, during installation of Chelsio Driver, BSOD too. > > > > Any suggestions? driver issue or bhyve issue? > > Can you try disabling APIC acceleration (which should be available on > that Xeon E5) ? This can be done on the fly using kenv and > unloading/reloading vmm.ko (or loader.conf and rebooting) > > kenv hw.vmm.vmx.use_tpr_shadowing=0 > kldunload vmm > kldload vmm > > later, > > Peter. > From owner-freebsd-virtualization@freebsd.org Sat Mar 14 04:36:17 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A704E2769C9 for ; Sat, 14 Mar 2020 04:36:17 +0000 (UTC) (envelope-from rebecca@bsdio.com) Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48fVBS17Rkz3LfV for ; Sat, 14 Mar 2020 04:36:16 +0000 (UTC) (envelope-from rebecca@bsdio.com) Received: from in02.mta.xmission.com ([166.70.13.52]) by out03.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jCyWz-00061Y-Ht for freebsd-virtualization@freebsd.org; Fri, 13 Mar 2020 22:36:13 -0600 Received: from mta5.zcs.xmission.com ([166.70.13.69]) by in02.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.87) (envelope-from ) id 1jCyWz-0004Pb-5J for freebsd-virtualization@freebsd.org; Fri, 13 Mar 2020 22:36:13 -0600 Received: from localhost (localhost [127.0.0.1]) by mta5.zcs.xmission.com (Postfix) with ESMTP id F3C1A128099B for ; Fri, 13 Mar 2020 22:36:12 -0600 (MDT) X-Amavis-Modified: Mail body modified (using disclaimer) - mta5.zcs.xmission.com Received: from mta5.zcs.xmission.com ([127.0.0.1]) by localhost (mta5.zcs.xmission.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 5Ac8dzYPpPkT for ; Fri, 13 Mar 2020 22:36:12 -0600 (MDT) Received: from photon.int.bluestop.org (muon.bluestop.org [65.103.231.193]) by mta5.zcs.xmission.com (Postfix) with ESMTPSA id B833D1280875 for ; Fri, 13 Mar 2020 22:36:12 -0600 (MDT) To: freebsd-virtualization@freebsd.org From: Rebecca Cran Message-ID: Date: Fri, 13 Mar 2020 22:36:12 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-XM-SPF: eid=1jCyWz-0004Pb-5J; ; ; mid=; ; ; hst=in02.mta.xmission.com; ; ; ip=166.70.13.69; ; ; frm=rebecca@bsdio.com; ; ; spf=pass X-SA-Exim-Connect-IP: 166.70.13.69 X-SA-Exim-Mail-From: rebecca@bsdio.com X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on sa07.xmission.com X-Spam-Level: X-Spam-Status: No, score=-0.2 required=8.0 tests=ALL_TRUSTED,BAYES_50, DCC_CHECK_NEGATIVE,XM_B_Unicode autolearn=disabled version=3.4.2 X-Spam-Report: * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP * 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60% * [score: 0.5229] * 0.0 XM_B_Unicode BODY: Testing for specific types of unicode * -0.0 DCC_CHECK_NEGATIVE Not listed in DCC * [sa07 1397; IP=ok Body=1 Fuz1=1] [Fuz2=1] X-Spam-DCC: XMission; sa07 1397; IP=ok Body=1 Fuz1=1 Fuz2=1 X-Spam-Combo: ;freebsd-virtualization@freebsd.org X-Spam-Relay-Country: X-Spam-Timing: total 202 ms - load_scoreonly_sql: 0.04 (0.0%), signal_user_changed: 3.3 (1.6%), b_tie_ro: 2.2 (1.1%), parse: 1.24 (0.6%), extract_message_metadata: 4.7 (2.3%), get_uri_detail_list: 0.86 (0.4%), tests_pri_-1000: 2.9 (1.4%), tests_pri_-950: 1.48 (0.7%), tests_pri_-900: 1.19 (0.6%), tests_pri_-90: 17 (8.5%), check_bayes: 16 (7.7%), b_tokenize: 6 (3.0%), b_tok_get_all: 4.4 (2.2%), b_comp_prob: 1.67 (0.8%), b_tok_touch_all: 1.63 (0.8%), b_finish: 0.63 (0.3%), tests_pri_0: 157 (77.8%), check_dkim_signature: 0.52 (0.3%), check_dkim_adsp: 25 (12.4%), poll_dns_idle: 18 (9.1%), tests_pri_10: 3.0 (1.5%), tests_pri_500: 7 (3.6%), rewrite_mail: 0.00 (0.0%) Subject: Bhyve gdb server for system debugging X-Spam-Flag: No X-SA-Exim-Version: 4.2.1 (built Thu, 05 May 2016 13:38:54 -0600) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) X-Rspamd-Queue-Id: 48fVBS17Rkz3LfV X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rebecca@bsdio.com designates 166.70.13.233 as permitted sender) smtp.mailfrom=rebecca@bsdio.com X-Spamd-Result: default: False [2.05 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:166.70.13.0/24]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_LONG(0.92)[0.916,0]; DMARC_NA(0.00)[bsdio.com]; IP_SCORE(0.53)[ip: (-0.48), ipnet: 166.70.0.0/16(1.77), asn: 6315(1.42), country: US(-0.05)]; RCVD_IN_DNSWL_LOW(-0.10)[233.13.70.166.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6315, ipnet:166.70.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 04:36:17 -0000 I know Bhyve supports kgdb debugging of a FreeBSD guest, but I've been looking for a system-level debugging capability to let me debug the system before FreeBSD boots. However, I'm having problems with the -G option to bhyve (e.g. -G 1234): gdb shows "(bad)" in the assembly listing, reports not being able to access memory in some stack frames, and ended up crashing after a while. e.g.: set architecture i386:x86-64:intel target remote localhost:1234 >0xfffcd3e1  (bad)                                                                                                                                            │ │   0xfffcd3e2  (bad)                                                                                                                                            │ │   0xfffcd3e3  (bad)                                                                                                                                            │ │   0xfffcd3e4  (bad)                                                                                                                                            │ │   0xfffcd3e5  (bad)  Should this work? -- Rebecca Cran (null) From owner-freebsd-virtualization@freebsd.org Sat Mar 14 12:50:15 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 115AD258D00 for ; Sat, 14 Mar 2020 12:50:15 +0000 (UTC) (envelope-from erleya@gmail.com) Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48fj8P4bQqz42Ps; Sat, 14 Mar 2020 12:50:13 +0000 (UTC) (envelope-from erleya@gmail.com) Received: by mail-wr1-x432.google.com with SMTP id s14so15523191wrt.8; Sat, 14 Mar 2020 05:50:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=G0hA914vj6W+KotbkUvLEmQ6GOqFx25sKsj7JZQZt6w=; b=uZbcBJVY5CqZHE286XWKH4sudnpdzelgUAyCr32BKcmVXQYgYxBMkwf+R6KxSGk+X+ Ak3kzQZAZj6a3uRxgE5fSthKi4YuJIVEzJJu8H5nNjpIeWiPmHhRga+JhT45Kqi9UzE9 hduN6ytXIzA56g3QPO14fDif0HF3FTPAQNyBvVfikJCFf0Nj5I0ykxYMQm2gnLHfvOPB CdIqcZu0muQf/gmqrVzrEM1Xd8fHxgtv76M8rZH2urnGk1p8O1BZciVq9PPwSuongp8f 3hiitwTQ08th1dM5hY80Rc2bmD5ml+o9qPrH4IvH6O5pYPidFUaxuP9BdC3Q6mW461Y9 KQIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=G0hA914vj6W+KotbkUvLEmQ6GOqFx25sKsj7JZQZt6w=; b=oH5q5FUS02SvBJzwocpnkNjEjMpGN4j2cdDVXDP54mKzV7pJrpxPf1sqBPII7rPuVS AnonxhjQ8zawQLa5+Q9u+h/si3Vv3CaiVlDuTQgeb53vCw4d1yLoCvPvF/WXhR4FqefJ EelCLy9tYsls9X3zTFdiPE4f73RGyjf3O+wbGEfeJEvujOPXsF9MHqXJZa0vvAOC5SF4 R5Q9qXiOXimNuCdWkwYnCu7d7yIB4R5qAWVftPliNvjGl5+JZsGqFShc4mlA1pTaHiSh 5YjkPk5s+7XvsdrEGEqhbd8mC24JQjZggArXbNo2HLV/v4liZHW8aH/zTb8Ap45OPCgS Z+gA== X-Gm-Message-State: ANhLgQ23E+/1l3Jt64eafI4mhM14msMZwaA7DnkO+kf69Z8HtwAnHXHP frGn6TNxtwgq4KAXe6U88jjiHK4J X-Google-Smtp-Source: ADFU+vsKFzbTe0li/pCDtWwKJyi+jbgT08zAJq/KQhpZH8feg/Gd56i/bZY2mwXJ94XHAAqR1DyulQ== X-Received: by 2002:adf:9b19:: with SMTP id b25mr24085356wrc.368.1584190211667; Sat, 14 Mar 2020 05:50:11 -0700 (PDT) Received: from erley.ru (erley.ru. [83.153.157.67]) by smtp.gmail.com with ESMTPSA id f9sm33192780wrc.71.2020.03.14.05.50.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 14 Mar 2020 05:50:11 -0700 (PDT) Received: by erley.ru (OpenSMTPD) with ESMTPSA id f3c18373 (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Sat, 14 Mar 2020 13:50:09 +0100 (CET) Subject: Re: [GPU pass-through] no compatible bridge window for claimed BAR To: Peter Grehan Cc: freebsd-virtualization@freebsd.org References: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> From: Alex Erley Message-ID: Date: Sat, 14 Mar 2020 13:50:39 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 48fj8P4bQqz42Ps X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=uZbcBJVY; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of erleya@gmail.com designates 2a00:1450:4864:20::432 as permitted sender) smtp.mailfrom=erleya@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RECEIVED_SPAMHAUS_PBL(0.00)[67.157.153.83.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (-9.28), ipnet: 2a00:1450::/32(-2.40), asn: 15169(-1.65), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Mar 2020 12:50:15 -0000 Hello, Some new findings to share. 1) Changing PCI_EMUL_MEMBASE64 from 0xD000000000 to any value *below 0x0440000000* makes bhyve fail when starting VM with message: bhyve: failed to initialize BARs for PCI 1/0/0 device emulation initialization error: Cannot allocate memory 2) Having PCI_EMUL_MEMBASE64 set to 0x0440000000 (or above) guest VM can not configure BARs of pass-through device properly. == (a) == On BHyve host ppt device is: > devinfo -rv ... pci0 hostb0 at slot=0 function=0 dbsf=pci0:0:0:0 pcib1 at slot=1 function=0 dbsf=pci0:0:1:0 handle=\_SB_.PCI0.P0P2 I/O ports: 0xe000-0xefff I/O memory addresses: 0x00c0000000-0x00d30fffff <-- covers all child mem windows pci1 ppt0 at slot=0 function=0 dbsf=pci0:1:0:0 pcib1 I/O port window: 0xe000-0xe07f pcib1 memory window: 0x00c0000000-0x00cfffffff <-- 256M 0x00d0000000-0x00d1ffffff <-- 32M 0x00d2000000-0x00d2ffffff <-- 16M ppt1 at slot=0 function=1 dbsf=pci0:1:0:1 pcib1 memory window: 0xd3080000-0xd3083fff <-- 16K ... and there is no other device attached to pci1. == (b) == On guest VM dmesg shows (timestamps are removed): ... BIOS-provided physical RAM map: BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable BIOS-e820: [mem 0x0000000000100000-0x00000000bea95fff] usable BIOS-e820: [mem 0x00000000bea96000-0x00000000bea97fff] reserved BIOS-e820: [mem 0x00000000bea98000-0x00000000bea99fff] ACPI data BIOS-e820: [mem 0x00000000bea9a000-0x00000000beaa8fff] reserved BIOS-e820: [mem 0x00000000beaa9000-0x00000000bfb28fff] usable BIOS-e820: [mem 0x00000000bfb29000-0x00000000bfb58fff] type 20 BIOS-e820: [mem 0x00000000bfb59000-0x00000000bfb7cfff] reserved BIOS-e820: [mem 0x00000000bfb7d000-0x00000000bfb81fff] usable BIOS-e820: [mem 0x00000000bfb82000-0x00000000bfb88fff] ACPI data BIOS-e820: [mem 0x00000000bfb89000-0x00000000bfb8cfff] ACPI NVS BIOS-e820: [mem 0x00000000bfb8d000-0x00000000bffcffff] usable BIOS-e820: [mem 0x00000000bffd0000-0x00000000bffeffff] reserved BIOS-e820: [mem 0x00000000bfff0000-0x00000000bfffffff] usable BIOS-e820: [mem 0x0000000100000000-0x000000043fffffff] usable ^^^-upper limit for adressable memory ... PM: Registered nosave memory: [mem 0xc0000000-0xffffffff] [mem 0xc0000000-0xffffffff] available for PCI devices ... pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] ^-- 128K pci_bus 0000:00: root bus resource [mem 0xc0000000-0xdfffffff window] ^-- 512M pci_bus 0000:00: root bus resource [mem 0xf0000000-0xf07fffff window] ^-- 8M pci_bus 0000:00: root bus resource [bus 00-ff] == (c) == Until now all runs OK. Guest Linux then allocates memory regions for devices. Allocation is done from lower reg (0x10) to higher (0x30) for each device (i.e. from 00.0 to 1f.0) on PCI bus. Here I reordered dmesg output to groups to show continuous RAM regions: (pass-through device is marked with *) pci 0000:00:01.0: reg 0x24: [io 0x2000-0x207f] pci 0000:00:02.0: reg 0x10: [io 0x2080-0x209f] pci 0000:00:03.0: reg 0x10: [io 0x20c0-0x20ff] ... pci 0000:00:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref] pci 0000:00:02.0: reg 0x30: [mem 0x00000000-0x000007ff pref] pci 0000:00:03.0: reg 0x30: [mem 0x00000000-0x000007ff pref] pci 0000:00:1d.0: reg 0x30: [mem 0x00000000-0x000007ff pref] pci 0000:00:1e.0: reg 0x30: [mem 0x00000000-0x000007ff pref] pci 0000:00:1f.0: reg 0x30: [mem 0x00000000-0x000007ff pref] ... pci 0000:00:01.0: reg 0x10:*[mem 0xc0000000-0xc0ffffff] 16M ... 0xc1000000-0xc1ffffff 16M gap pci 0000:00:01.0: reg 0x1c:*[mem 0xc2000000-0xc3ffffff 64bit pref] 32M pci 0000:00:01.1: reg 0x10:*[mem 0xc4000000-0xc4003fff] pci 0000:00:02.0: reg 0x14: [mem 0xc4004000-0xc4005fff] pci 0000:00:03.0: reg 0x14: [mem 0xc4006000-0xc4007fff] pci 0000:00:1d.0: reg 0x10: [mem 0xc4008000-0xc400807f] ... 0xc4008080-0xc4ffffff <16M gap pci 0000:00:1d.0: reg 0x14: [mem 0xc5000000-0xc5ffffff] 16M pci 0000:00:1e.0: reg 0x10: [mem 0xc6000000-0xc6000fff] ... 0xc6001000-0xd2ffffff <208M gap pci 0000:00:01.0: reg 0x30:*[mem 0xd3000000-0xd307ffff pref] 512K 0xd3080000-0xdfffffff <208M gap pci 0000:00:01.0: reg0x14:*[mem 0x440000000-0x44fffffff 64bit pref] 256M ^^^- this value is outside allowed range == (d) == So, there is no window for 256M BAR, although there are 2 big gapes of 208M in 512M space provided for BAR allocation by PCI bus. So, BAR reg 0x14 of size 256M for device 01.0 must be inside provisioned 512M region 0xc0000000-0xdfffffff. But refering to (1) above, setting base address to any value below 0x440000000 breaks bhyve on start. According to (b), this value corresponds to upper addressable memory limit in guest VM. So I'm blocked here at the moment: - Guest VM requires a value which BHyve doesn't like. - Guest VM allocates BARs with huge gapes. I have little knowledge about PCI bus internals, although I already read some articles on internet. Could it be some ACPI trick to do? I'd be happy to hear any ideas... PS I suspect that if I take other OS as a guest VM or other pass-through GPU model, it would probably allocate BARs properly. But this is not what I want for this config. There should be a way to allocate 256M BAR in guest Linux. Have a nice day, Alex