From owner-freebsd-virtualization@freebsd.org Fri Mar 23 17:25:13 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4277F5C380 for ; Fri, 23 Mar 2018 17:25:12 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 56CD8810C3 for ; Fri, 23 Mar 2018 17:25:12 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (115-166-20-68.dyn.iinet.net.au [115.166.20.68]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w2NHP6WF095954 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 23 Mar 2018 10:25:09 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: bhyve pci passthrough usb controller To: Wouter Snels , freebsd-virtualization@freebsd.org References: <5bd3c8b6-0579-0269-1ada-ab6f0a55ce8f@ofloo.net> From: Julian Elischer Message-ID: <2ca981b4-9651-253a-7549-eaece4a58ff0@freebsd.org> Date: Sat, 24 Mar 2018 01:25:01 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <5bd3c8b6-0579-0269-1ada-ab6f0a55ce8f@ofloo.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Mar 2018 17:25:13 -0000 On 23/3/18 11:40 pm, Wouter Snels wrote: new wrapping is no better It looks like the device you are passing though is not removed from freebsd first. but it's hard to tell. > |Hi, .. I'm trying to pci passhthrough a usb controller to a bhyve > windows 7 virtualization. When I start the virtual machine it freezes > the entire system and this shows up on the console. To be clear the > system runs without any problems without the pci passthrough. | > ||(ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 e0 ff 40 40 29 00 > 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was invalid > (ada0:ahcich0:0:0:0): Error 22, Unretryable error > (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 > 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was invalid > (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ada0 at ahcich0 bus > 0 scbus0 target 0 lun 0 ada0: s/n ZA4292G6 > detached (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 > 00 00 00 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was > invalid (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0: > Poll timeout on slot 28 port 0 ahcich0: is 20000000 cs 10000000 ss > 00000000 rs 10000000 tfd 1d0 serr 00000800 cmd 00005c17 > (aprobe0:ahcich0:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 > 00 00 (aprobe0:ahcich0:0:0:0): CAM status: Command timeout > (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted > (ada0:ahcich0:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 > 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was > invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0: Poll > timeout on slot 30 port 0 ahcich0: is 20000000 cs 40000000 ss 00000000 > rs 40000000 tfd 1d0 serr 00000800 cmd 00005e17 (aprobe0:ahcich0:0:0:0): > SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 > (aprobe0:ahcich0:0:0:0): CAM status: Command timeout > (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted > (ada0:ahcich0:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40 > 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was > invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error > (ada0:ahcich0:0:0:0): Periph destroyed em0: Watchdog timeout Queue[0]-- > resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw > tdh = 428, hw tdt = 471 em0: Tx Queue Status = -2147483648 em0: TX > descriptors avail = 981 em0: Tx Descriptors avail failure = 0 em0: RX > Queue 0 ------ em0: hw rdh = 912, hw rdt = 908 em0: RX discarded packets > = 0 em0: RX Next to Check = 909 em0: RX Next to Refresh = 908 xl0: > watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is > RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 58 > em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 966 em0: > Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0, > hw rdt = 1023 em0: RX discarded packets = 0 em0: RX Next to Check = 0 > em0: RX Next to Refresh = 1023 em0: Watchdog timeout Queue[0]-- > resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw > tdh = 0, hw tdt = 58 em0: Tx Queue Status = -2147483648 em0: TX > descriptors avail = 966 em0: Tx Descriptors avail failure = 0 em0: RX > Queue 0 ------ em0: hw rdh = 0, hw rdt = 1023 em0: RX discarded packets > = 0 em0: RX Next to Check = 0 em0: RX Next to Refresh = 1023 xl0: > watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is > RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 54 > em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 970 em0: > Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0, > hw rdt = 1023 em0: RX discarded packets = 0include GENERIC ident OFL > options IPFIREWALL|||||| > > |Hi, .. I'm trying to pci passhthrough a usb controller to a bhyve > windows 7 virtualization. When I start the virtual machine it freezes > the entire system and this shows up on the console. To be clear the > system runs without any problems without the pci passthrough. | > ||(ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 e0 ff 40 40 29 00 > 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was invalid > (ada0:ahcich0:0:0:0): Error 22, Unretryable error > (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 > 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was invalid > (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ada0 at ahcich0 bus > 0 scbus0 target 0 lun 0 ada0: s/n ZA4292G6 > detached (aprobe0:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 > 00 00 00 00 00 00 (aprobe0:ahcich0:0:0:0): CAM status: CCB request was > invalid (aprobe0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0: > Poll timeout on slot 28 port 0 ahcich0: is 20000000 cs 10000000 ss > 00000000 rs 10000000 tfd 1d0 serr 00000800 cmd 00005c17 > (aprobe0:ahcich0:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 > 00 00 (aprobe0:ahcich0:0:0:0): CAM status: Command timeout > (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted > (ada0:ahcich0:0:0:0): SETFEATURES ENABLE RCACHE. ACB: ef aa 00 00 00 40 > 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was > invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error ahcich0: Poll > timeout on slot 30 port 0 ahcich0: is 20000000 cs 40000000 ss 00000000 > rs 40000000 tfd 1d0 serr 00000800 cmd 00005e17 (aprobe0:ahcich0:0:0:0): > SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00 > (aprobe0:ahcich0:0:0:0): CAM status: Command timeout > (aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted > (ada0:ahcich0:0:0:0): SETFEATURES ENABLE WCACHE. ACB: ef 02 00 00 00 40 > 00 00 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: CCB request was > invalid (ada0:ahcich0:0:0:0): Error 22, Unretryable error > (ada0:ahcich0:0:0:0): Periph destroyed em0: Watchdog timeout Queue[0]-- > resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw > tdh = 428, hw tdt = 471 em0: Tx Queue Status = -2147483648 em0: TX > descriptors avail = 981 em0: Tx Descriptors avail failure = 0 em0: RX > Queue 0 ------ em0: hw rdh = 912, hw rdt = 908 em0: RX discarded packets > = 0 em0: RX Next to Check = 909 em0: RX Next to Refresh = 908 xl0: > watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is > RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 58 > em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 966 em0: > Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0, > hw rdt = 1023 em0: RX discarded packets = 0 em0: RX Next to Check = 0 > em0: RX Next to Refresh = 1023 em0: Watchdog timeout Queue[0]-- > resetting Interface is RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw > tdh = 0, hw tdt = 58 em0: Tx Queue Status = -2147483648 em0: TX > descriptors avail = 966 em0: Tx Descriptors avail failure = 0 em0: RX > Queue 0 ------ em0: hw rdh = 0, hw rdt = 1023 em0: RX discarded packets > = 0 em0: RX Next to Check = 0 em0: RX Next to Refresh = 1023 xl0: > watchdog timeout em0: Watchdog timeout Queue[0]-- resetting Interface is > RUNNING and ACTIVE em0: TX Queue 0 ------ em0: hw tdh = 0, hw tdt = 54 > em0: Tx Queue Status = -2147483648 em0: TX descriptors avail = 970 em0: > Tx Descriptors avail failure = 0 em0: RX Queue 0 ------ em0: hw rdh = 0, > hw rdt = 1023 em0: RX discarded packets = 0include GENERIC ident OFL > options IPFIREWALL options IPFIREWALL_VERBOSE options > IPFIREWALL_DEFAULT_TO_ACCEPT options DUMMYNET options IPDIVERT options > CONSPEED=115200 em0: RX Next to Check = 0 em0: RX Next to Refresh = 1023 |||| > |||I've recompiled the kernel without the xhci driver. I've tested both > usb controllers. Both "|||||Sunrise Point-H USB 3.0 xHCI > Controller|"||||| and "|||uPD720202 USB 3.0 Host Controller|", both > showed the same error on the console screen.| # uname FreeBSD > 11.1-STABLE #1 r331092M # custom kernel include GENERIC ident OFL > options IPFIREWALL options IPFIREWALL_VERBOSE options > IPFIREWALL_DEFAULT_TO_ACCEPT options DUMMYNET options IPDIVERT options > CONSPEED=115200 * note commented "device xhci" in generic # svn info |||Path: . Working Copy Root Path: /usr/src URL: > https://svn0.eu.freebsd.org/base/stable/11 Relative URL: ^/stable/11 > Repository Root: https://svn0.eu.freebsd.org/base Repository UUID: > ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 331092 Node Kind: > directory Schedule: normal Last Changed Author: eadler Last Changed Rev: > 331092 Last Changed Date: 2018-03-17 07:44:39 +0100 (Sat, 17 Mar 2018) > |# vm passthru DEVICE BHYVE ID READY DESCRIPTION hostb0 0/0/0 No Xeon > E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers > pcib1 0/1/0 No Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe > Controller (x16) vgapci0 0/2/0 No HD Graphics 530 |||ppt0| 0/20/0 Yes Sunrise Point-H USB 3.0 xHCI Controller none0 0/20/2 > No Sunrise Point-H Thermal subsystem none1 0/22/0 No Sunrise Point-H > CSME HECI ahci0 0/23/0 No Sunrise Point-H SATA controller [AHCI mode] > pcib2 0/28/0 No Sunrise Point-H PCI Express Root Port pcib4 0/28/6 No > Sunrise Point-H PCI Express Root Port isab0 0/31/0 No Sunrise Point-H > LPC Controller none2 0/31/2 No Sunrise Point-H PMC hdac0 0/31/3 No > Sunrise Point-H HD Audio none3 0/31/4 No Sunrise Point-H SMBus em0 > 0/31/6 No Ethernet Connection (2) I219-V pcib3 2/0/0 No XIO2001 PCI > Express-to-PCI Bridge xl0 3/0/0 No 3c905B 100BaseTX [Cyclone] ppt1 4/0/0 > Yes uPD720202 USB 3.0 Host Controller ||ppt1@pci0:4:0:0: class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 > hdr=0x00 vendor = 'Renesas Technology Corp.' device = 'uPD720202 USB 3.0 > Host Controller' class = serial bus subclass = USB |||ppt|0@pci0:0:20:0: class=0x0c0330 card=0x06ba1028 chip=0xa12f8086 > rev=0x31 hdr=0x00 vendor = 'Intel Corporation' device = 'Sunrise Point-H > USB 3.0 xHCI Controller' class = serial bus subclass = USB # grep -i > pptdevfs loader.conf ||pptdevs="0/20/0 4/0/0" # vm configure ||passthru0="0/20/0"| > > |# vm configure ||passthru0="4/0/0"| > ||Any suggestions on what I'm doing wrong?|| > > Regards, Ofloo > |||||| > > ||options IPFIREWALL_VERBOSE options IPFIREWALL_DEFAULT_TO_ACCEPT options > DUMMYNET options IPDIVERT options CONSPEED=115200 em0: RX Next to Check > = 0 em0: RX Next to Refresh = 1023 |||| > |||I've recompiled the kernel without the xhci driver. I've tested both > usb controllers. Both "|||||Sunrise Point-H USB 3.0 xHCI > Controller|"||||| and "|||uPD720202 USB 3.0 Host Controller|", both > showed the same error on the console screen.| # uname FreeBSD > 11.1-STABLE #1 r331092M # custom kernel include GENERIC ident OFL > options IPFIREWALL options IPFIREWALL_VERBOSE options > IPFIREWALL_DEFAULT_TO_ACCEPT options DUMMYNET options IPDIVERT options > CONSPEED=115200 * note commented "device xhci" in generic # svn info |||Path: . Working Copy Root Path: /usr/src URL: > https://svn0.eu.freebsd.org/base/stable/11 Relative URL: ^/stable/11 > Repository Root: https://svn0.eu.freebsd.org/base Repository UUID: > ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f Revision: 331092 Node Kind: > directory Schedule: normal Last Changed Author: eadler Last Changed Rev: > 331092 Last Changed Date: 2018-03-17 07:44:39 +0100 (Sat, 17 Mar 2018) > |# vm passthru DEVICE BHYVE ID READY DESCRIPTION hostb0 0/0/0 No Xeon > E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers > pcib1 0/1/0 No Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe > Controller (x16) vgapci0 0/2/0 No HD Graphics 530 |||ppt0| 0/20/0 Yes Sunrise Point-H USB 3.0 xHCI Controller none0 0/20/2 > No Sunrise Point-H Thermal subsystem none1 0/22/0 No Sunrise Point-H > CSME HECI ahci0 0/23/0 No Sunrise Point-H SATA controller [AHCI mode] > pcib2 0/28/0 No Sunrise Point-H PCI Express Root Port pcib4 0/28/6 No > Sunrise Point-H PCI Express Root Port isab0 0/31/0 No Sunrise Point-H > LPC Controller none2 0/31/2 No Sunrise Point-H PMC hdac0 0/31/3 No > Sunrise Point-H HD Audio none3 0/31/4 No Sunrise Point-H SMBus em0 > 0/31/6 No Ethernet Connection (2) I219-V pcib3 2/0/0 No XIO2001 PCI > Express-to-PCI Bridge xl0 3/0/0 No 3c905B 100BaseTX [Cyclone] ppt1 4/0/0 > Yes uPD720202 USB 3.0 Host Controller ||ppt1@pci0:4:0:0: class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 > hdr=0x00 vendor = 'Renesas Technology Corp.' device = 'uPD720202 USB 3.0 > Host Controller' class = serial bus subclass = USB |||ppt|0@pci0:0:20:0: class=0x0c0330 card=0x06ba1028 chip=0xa12f8086 > rev=0x31 hdr=0x00 vendor = 'Intel Corporation' device = 'Sunrise Point-H > USB 3.0 xHCI Controller' class = serial bus subclass = USB # grep -i > pptdevfs loader.conf ||pptdevs="0/20/0 4/0/0" # vm configure ||passthru0="0/20/0"| > > |# vm configure ||passthru0="4/0/0"| > ||Any suggestions on what I'm doing wrong?|| > > edit: previous wrapping was all wrong. > > Regards, Ofloo > |||||| > >