From owner-freebsd-virtualization@freebsd.org Tue Feb 23 01:53:05 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFFD9AB08B5; Tue, 23 Feb 2016 01:53:05 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-qg0-f54.google.com (mail-qg0-f54.google.com [209.85.192.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7BC2D1645; Tue, 23 Feb 2016 01:53:05 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mail-qg0-f54.google.com with SMTP id b67so126483572qgb.1; Mon, 22 Feb 2016 17:53:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-type; bh=nyWs5ZwGn8IJSuZsKqUPFPGIPssreYdjm7HGg9g3h5o=; b=WfcQDklg5zH/v8Jz/feJTL7dZOGslTc+tI71jFoxHdWRg5HYAd3RKU61UYSgg/ES53 EnT87M/qHUBjj04lYcu5r2c5Lxq0AKcEPM9p5WvtVJUgwsRPJ2QNbXP1LcAoFUJ00B3s 4Oqq5eGizJeaL/7/B/Ei7VM0gtp2YHOf71wsqPGH1AL7o5ti5h/xL/TlE7OTNaCNeR47 ev0OEVEKLJyDMm0dAgrDfizlD9Rg3Q3TxvI0UL0fi/f1JQ1DyUpfdHzy0mbtklrR2E2K Z8pjUBVPjSN1YiHYbcnVJIJBhRzcRYNWIvUadFRRWWVh9cSIwx9EEJpkwXMMSvH4TY3o euAw== X-Gm-Message-State: AG10YOQlVL/G3wGJgCUc41URO1165tOLdH67SBkfzqs3xKfOXXlX8wX6SvHSvyxzeiRwDg== X-Received: by 10.140.29.202 with SMTP id b68mr37850635qgb.100.1456192383973; Mon, 22 Feb 2016 17:53:03 -0800 (PST) Received: from mail-qg0-f45.google.com (mail-qg0-f45.google.com. [209.85.192.45]) by smtp.gmail.com with ESMTPSA id o60sm7686693qge.34.2016.02.22.17.53.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Feb 2016 17:53:03 -0800 (PST) Received: by mail-qg0-f45.google.com with SMTP id y89so126302946qge.2; Mon, 22 Feb 2016 17:53:03 -0800 (PST) X-Received: by 10.140.102.142 with SMTP id w14mr29926361qge.58.1456192383403; Mon, 22 Feb 2016 17:53:03 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Eric Joyner Date: Tue, 23 Feb 2016 01:52:53 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: sr-iov issues, reset_hw() failed with error -100 To: Ultima Cc: freebsd-current@freebsd.org, freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Feb 2016 01:53:05 -0000 I don't really have any ideas on the error -100. Error -100 means there was a mailbox error, so something failed in the initial communications setup between the PF and VF, but I don't know what exactly went wrong. I'm grasping at straws, but try using a smaller number of VFs initially, like 2? And check to see if VT-d is enabled in your BIOS? (Though I would've expected iovctl to fail). - Eric On Mon, Feb 22, 2016 at 12:01 PM Ultima wrote: > After reboot... > > ifconfig ix1 up > > dhclient ix1 > DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4 > DHCPOFFER from 192.168.1.1 > DHCPREQUEST on ix1 to 255.255.255.255 port 67 > DHCPACK from 192.168.1.1 > bound to 192.168.1.145 -- renewal in 21600 seconds. > > ix0 down > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms > 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms > 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms > > iovctl -Cf /etc/iovctl.conf > > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > ^C > --- 192.168.1.1 ping statistics --- > 29 packets transmitted, 0 packets received, 100.0% packet loss > ifconfig ix1 up > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > ^C > --- 192.168.1.1 ping statistics --- > 12 packets transmitted, 0 packets received, 100.0% packet loss > > ix1 is no longer usable until a restart... > > iovctl -Dd ix1 > ifconfig ix1 up > ping 192.168.1.1 > PING 192.168.1.1 (192.168.1.1): 56 data bytes > ^C > --- 192.168.1.1 ping statistics --- > 9 packets transmitted, 0 packets received, 100.0% packet loss > > > > Is there anything else that maybe useful? > > here is my ifconfig at the end (after ifconfig ix0 up) > > > ix0: flags=8943 metric 0 > mtu 1500 > > options=e400b9 > ether -Hidden- > inet 192.168.1.8 netmask 0xffffff00 broadcast 192.168.1.255 > inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255 > nd6 options=29 > media: Ethernet autoselect (10Gbase-T ) > status: active > ix1: flags=8843 metric 0 mtu 1500 > > options=e407bb > ether -Hidden- > inet 192.168.1.145 netmask 0xffffff00 broadcast 192.168.1.255 > nd6 options=29 > media: Ethernet autoselect (10Gbase-T ) > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > groups: lo > bridge0: flags=8843 metric 0 mtu > 1500 > ether -Hidden- > nd6 options=9 > groups: bridge > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: ix0 flags=143 > ifmaxaddr 0 port 1 priority 128 path cost 2000 > member: epair0a flags=143 > ifmaxaddr 0 port 5 priority 128 path cost 2000 > epair0a: flags=8943 metric > 0 mtu 1500 > options=8 > ether -Hidden- > inet6 fe80::ff:70ff:fe00:50a%epair0a prefixlen 64 scopeid 0x5 > nd6 options=21 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair > > On Mon, Feb 22, 2016 at 1:51 PM, Eric Joyner wrote: > >> Did you do an ifconfig up on ix1 before loading the VF driver? >> >> On Sat, Feb 20, 2016 at 11:57 AM Ultima wrote: >> >>> Decided to do some testing with iovctl to see how sr-iov is coming >>> along. >>> Currently when adding the vf's there are a couple errors, and the network >>> no longer function after iovctl is started. My guess is the reset_hw() >>> call >>> that is failing. Any ideas why this call would fail? I tested this on >>> both >>> ports, ix1 is detached and unused for this test, however inserting a >>> cable >>> results in an unusable port. iovctl -Dd ix1 removes the vf's, however >>> functionality is still not restored without a system restart. >>> >>> FreeBSD S1 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r295736: Wed Feb 17 >>> 21:17:28 EST 2016 root@S1:/usr/obj/usr/src/sys/MYKERNEL amd64 >>> >>> /boot/loader.conf >>> hw.ix.num_queues="4" >>> >>> /etc/iovctl.conf >>> PF { >>> device : ix1; >>> num_vfs : 31; >>> } >>> >>> DEFAULT { >>> passthrough : true; >>> } >>> VF-0 { >>> passthrough : false; >>> } >>> VF-1 { >>> passthrough : false; >>> } >>> >>> # iovctl -C -f /etc/iovctl.conf >>> >>> dmesg >>> ixv0: >> 1.4.6-k> at device 0.129 on pci12 >>> ixv0: Using MSIX interrupts with 2 vectors >>> ixv0: ixgbe_reset_hw() failed with error -100 >>> device_attach: ixv0 attach returned 5 >>> ixv0: >> 1.4.6-k> at device 0.131 on pci12 >>> ixv0: Using MSIX interrupts with 2 vectors >>> ixv0: ixgbe_reset_hw() failed with error -100 >>> device_attach: ixv0 attach returned 5 >>> pci12: at device 0.133 (no driver attached) >>> pci12: at device 0.135 (no driver attached) >>> pci12: at device 0.137 (no driver attached) >>> pci12: at device 0.139 (no driver attached) >>> pci12: at device 0.141 (no driver attached) >>> pci12: at device 0.143 (no driver attached) >>> pci12: at device 0.145 (no driver attached) >>> pci12: at device 0.147 (no driver attached) >>> pci12: at device 0.149 (no driver attached) >>> pci12: at device 0.151 (no driver attached) >>> pci12: at device 0.153 (no driver attached) >>> pci12: at device 0.155 (no driver attached) >>> pci12: at device 0.157 (no driver attached) >>> pci12: at device 0.159 (no driver attached) >>> pci12: at device 0.161 (no driver attached) >>> pci12: at device 0.163 (no driver attached) >>> pci12: at device 0.165 (no driver attached) >>> pci12: at device 0.167 (no driver attached) >>> pci12: at device 0.169 (no driver attached) >>> pci12: at device 0.171 (no driver attached) >>> pci12: at device 0.173 (no driver attached) >>> pci12: at device 0.175 (no driver attached) >>> pci12: at device 0.177 (no driver attached) >>> pci12: at device 0.179 (no driver attached) >>> pci12: at device 0.181 (no driver attached) >>> pci12: at device 0.183 (no driver attached) >>> pci12: at device 0.185 (no driver attached) >>> pci12: at device 0.187 (no driver attached) >>> pci12: at device 0.189 (no driver attached) >>> >>> pciconf -lv >>> ix1@pci0:129:0:1: class=0x020000 card=0x00001458 chip=0x15288086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'Ethernet Controller 10-Gigabit X540-AT2' >>> class = network >>> subclass = ethernet >>> none155@pci0:129:0:129: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> none156@pci0:129:0:131: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt0@pci0:129:0:133: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt1@pci0:129:0:135: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt2@pci0:129:0:137: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt3@pci0:129:0:139: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt4@pci0:129:0:141: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt5@pci0:129:0:143: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt6@pci0:129:0:145: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt7@pci0:129:0:147: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt8@pci0:129:0:149: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt9@pci0:129:0:151: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt10@pci0:129:0:153: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt11@pci0:129:0:155: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt12@pci0:129:0:157: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt13@pci0:129:0:159: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt14@pci0:129:0:161: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt15@pci0:129:0:163: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt16@pci0:129:0:165: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt17@pci0:129:0:167: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt18@pci0:129:0:169: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt19@pci0:129:0:171: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt20@pci0:129:0:173: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt21@pci0:129:0:175: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt22@pci0:129:0:177: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt23@pci0:129:0:179: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt24@pci0:129:0:181: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt25@pci0:129:0:183: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt26@pci0:129:0:185: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt27@pci0:129:0:187: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> ppt28@pci0:129:0:189: class=0x020000 card=0x00001458 chip=0x15158086 >>> rev=0x01 hdr=0x00 >>> vendor = 'Intel Corporation' >>> device = 'X540 Ethernet Controller Virtual Function' >>> class = network >>> subclass = ethernet >>> >>> Ultima >>> _______________________________________________ >>> freebsd-current@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>> To unsubscribe, send any mail to " >>> freebsd-current-unsubscribe@freebsd.org" >>> >> >