From owner-freebsd-virtualization@freebsd.org Sat Dec 10 20:44:45 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 954B8C7189F for ; Sat, 10 Dec 2016 20:44:45 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 110711F25 for ; Sat, 10 Dec 2016 20:44:44 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id uBAKihWN073495 for ; Sat, 10 Dec 2016 21:44:43 +0100 (CET) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 63B02D50; Sat, 10 Dec 2016 21:44:42 +0100 (CET) Message-ID: <584C6939.9060805@omnilan.de> Date: Sat, 10 Dec 2016 21:44:41 +0100 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: BAR and pci passthrough Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Sat, 10 Dec 2016 21:44:43 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Dec 2016 20:44:45 -0000 In reply to https://lists.freebsd.org/pipermail/freebsd-virtualization/2015-December/004000.html >/>> The problem probably that the length of the bar is smaller and not />/>> aligned with the pagesize. Could the length of the BAR be modified in />/>> order to perform the pci passthrough? />/> />/> Yes, that's correct - the size of the BAR is not a multiple of the />/> page size which leads to the error. If this BAR is mapped into the />/> guest's address space then it will "leak" an additional 3K into the />/> guest (since the minimum nested mapping is 4KB in size). /> The proper fix is to modify the ppt(4) driver so that > it asks the PCI bus to allocate a full page for BARs that are smaller than a > page. Unfortunately the PCI bus driver doesn't currently provide a way to do > that. Even if it did it would not necessarily always work since the device > may be behind a PCI-PCI bridge whose windows can't grow due to existing > allocations of adjacent addresses. Alternatively the hypervisor could trap > all accesses to this page and only permit accesses to the range that contains > the BAR, but that would be quite slow. My stupid idea was to set in device.hints: hint.ppt.0.msize="0x1000" Since John (jhb@) already posted that the PCI bus driver deosn't provide the necessary capability to increase allocation, this clearly doesn't work. But since this hint was one year ago, I wanted to ask if there are news regarding PCI passthru (ppt's) page size handling or other workarrounds. My specific problem reads: sh -c 'while true ; do (bhyve -u -A -H -P -s 0,hostbridge -s 6,virtio-net,vmnet0 -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait -s 31,lpc -s 1,passthru,6/0/0 -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd -S -m 4G -c 4 preed); vmm_exit=$?; if [ ${vmm_exit} -ne 0 ]; then break; fi; done ; /usr/sbin/bhyvectl --destroy --vm=preed' bhyve: passthru device 6/0/0 BAR 5: base 0xc3e10000 or size 0x200 not page aligned Please make sure im on CC, I'm not subscribed. Thanks, -Harry P.S.: Thanks to all byhve hackers, really awsome work! I'm evaluating ESXi reliever... Perhaps this obstacle is interesting for some people here (and one reason I need to passthru SATA controller) https://lists.freebsd.org/pipermail/freebsd-stable/2016-October/086157.html