From owner-freebsd-current@freebsd.org Mon Feb 22 21:35:57 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 34424AB1F2E for ; Mon, 22 Feb 2016 21:35:57 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA2E11805 for ; Mon, 22 Feb 2016 21:35:56 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22b.google.com with SMTP id a4so182411382wme.1 for ; Mon, 22 Feb 2016 13:35:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=u4XYrwYUosUnCCbx3EWdpBSBdfIIqm1sOPZGohVi3W8=; b=M6gdxOWAmqOTx3khO0dprZhIpeygay3fBQMK7alAertf9OD4d8TvfQUWt8bW4R9eTz o39ycdj3s/xuPX00YwMnQfA6dSX0Ryvk6wBx22RtYKprxXW6nUVru37SjxpFGQMOp5X8 Nxg7DEi2tzsVmWiXnm8mfnQPqvKWjPDqCnVmN/FCTy7RLE89mJsBZd6qOH2IrLqYlTYF W5rbVstraetpfUry4Ll2YSgRoKez/0EY1Pd9g3GhY78ozLymro/CCLoJWTKqJjDlcUee kWI/+Qb1xBMcy709UfvB6xTeKr9neZokLRsJHW5M9eNGX1GboBy3If0cPHk0He0Dlsn9 p89A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=u4XYrwYUosUnCCbx3EWdpBSBdfIIqm1sOPZGohVi3W8=; b=WVoBNe2i4aKZ5HhL8uB9fLO/mCeuf80DdUBpcy6vqgcR3sLIShoCYE/YNbLNdtewMR XSaNqojdeUZXcq0Kw+a/2saT+lUYkqRlYwcKgoheVrlV7C5F0CkHjR3oyI+R1aLLVE2C Lsu/raQGaxWkQHZvv+ERIRx/0D/w4+BN3fbSh82FtSVAif26FLlPOTV5GiLltDO1qNEL DQ30ng95K+t2EVeMQZYGjxZOFAsPQ1Z512AwXx07YRuO1pCatNSDdYga5HuIwxuSed63 6YkLjJD+Kcl4hTanRwwIWTidJEjb+TkxcJp4mHTQnEzRhRiZJycFWR02q/pa8ZgrZQXY wqaA== X-Gm-Message-State: AG10YOSstgRhpv836fcTPL3UxBw9FX8bVesiuiEzA/1pQYA26O8GItq6vd3ugSJHx1eEUb7Q X-Received: by 10.28.49.3 with SMTP id x3mr14295062wmx.4.1456176954973; Mon, 22 Feb 2016 13:35:54 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id 74sm23001347wmn.17.2016.02.22.13.35.53 for (version=TLSv1/SSLv3 cipher=OTHER); Mon, 22 Feb 2016 13:35:53 -0800 (PST) Subject: Re: sr-iov issues, reset_hw() failed with error -100 To: freebsd-current@freebsd.org References: <2A35EA60C3C77D438915767F458D656880942E9A@ORSMSX111.amr.corp.intel.com> From: Steven Hartland Message-ID: <56CB7F3D.2010607@multiplay.co.uk> Date: Mon, 22 Feb 2016 21:35:57 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2016 21:35:57 -0000 isn't that 48 cores (12 real 12 virtual) per CPU? On 22/02/2016 21:26, Ultima wrote: > This system has 24 cores (e5-2670v3)x2 > > Ultima > > On Mon, Feb 22, 2016 at 3:53 PM, Pieper, Jeffrey E < > jeffrey.e.pieper@intel.com> wrote: > >> Just out of curiosity, how many cores does your system have? >> >> Jeff >> >> -----Original Message----- >> From: owner-freebsd-current@freebsd.org [mailto: >> owner-freebsd-current@freebsd.org] On Behalf Of Ultima >> Sent: Monday, February 22, 2016 12:02 PM >> To: Eric Joyner >> Cc: freebsd-current@freebsd.org; freebsd-virtualization@freebsd.org >> Subject: Re: sr-iov issues, reset_hw() failed with error -100 >> >> After reboot... >> >> ifconfig ix1 up >> >> dhclient ix1 >> DHCPDISCOVER on ix1 to 255.255.255.255 port 67 interval 4 >> DHCPOFFER from 192.168.1.1 >> DHCPREQUEST on ix1 to 255.255.255.255 port 67 >> DHCPACK from 192.168.1.1 >> bound to 192.168.1.145 -- renewal in 21600 seconds. >> >> ix0 down >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> 64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.149 ms >> 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.171 ms >> 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.167 ms >> >> iovctl -Cf /etc/iovctl.conf >> >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> ^C >> --- 192.168.1.1 ping statistics --- >> 29 packets transmitted, 0 packets received, 100.0% packet loss >> ifconfig ix1 up >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> ^C >> --- 192.168.1.1 ping statistics --- >> 12 packets transmitted, 0 packets received, 100.0% packet loss >> >> ix1 is no longer usable until a restart... >> >> iovctl -Dd ix1 >> ifconfig ix1 up >> ping 192.168.1.1 >> PING 192.168.1.1 (192.168.1.1): 56 data bytes >> ^C >> --- 192.168.1.1 ping statistics --- >> 9 packets transmitted, 0 packets received, 100.0% packet loss >> >> >> >> Is there anything else that maybe useful? >> >> here is my ifconfig at the end (after ifconfig ix0 up) >> >> >> ix0: flags=8943 metric 0 >> mtu 1500 >> >> options=e400b9 >> ether -Hidden- >> inet 192.168.1.8 netmask 0xffffff00 broadcast 192.168.1.255 >> inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255 >> nd6 options=29 >> media: Ethernet autoselect (10Gbase-T ) >> status: active >> ix1: flags=8843 metric 0 mtu 1500 >> >> options=e407bb >> ether -Hidden- >> inet 192.168.1.145 netmask 0xffffff00 broadcast 192.168.1.255 >> nd6 options=29 >> media: Ethernet autoselect (10Gbase-T ) >> status: active >> lo0: flags=8049 metric 0 mtu 16384 >> options=600003 >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 >> inet 127.0.0.1 netmask 0xff000000 >> nd6 options=21 >> groups: lo >> bridge0: flags=8843 metric 0 mtu >> 1500 >> ether -Hidden- >> nd6 options=9 >> groups: bridge >> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >> member: ix0 flags=143 >> ifmaxaddr 0 port 1 priority 128 path cost 2000 >> member: epair0a flags=143 >> ifmaxaddr 0 port 5 priority 128 path cost 2000 >> epair0a: flags=8943 metric >> 0 mtu 1500 >> options=8 >> ether -Hidden- >> inet6 fe80::ff:70ff:fe00:50a%epair0a prefixlen 64 scopeid 0x5 >> nd6 options=21 >> media: Ethernet 10Gbase-T (10Gbase-T ) >> status: active >> groups: epair >> >> On Mon, Feb 22, 2016 at 1:51 PM, Eric Joyner wrote: >> >>> Did you do an ifconfig up on ix1 before loading the VF driver? >>> >>> On Sat, Feb 20, 2016 at 11:57 AM Ultima wrote: >>> >>>> Decided to do some testing with iovctl to see how sr-iov is coming >> along. >>>> Currently when adding the vf's there are a couple errors, and the >> network >>>> no longer function after iovctl is started. My guess is the reset_hw() >>>> call >>>> that is failing. Any ideas why this call would fail? I tested this on >> both >>>> ports, ix1 is detached and unused for this test, however inserting a >> cable >>>> results in an unusable port. iovctl -Dd ix1 removes the vf's, however >>>> functionality is still not restored without a system restart. >>>> >>>> FreeBSD S1 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r295736: Wed Feb 17 >>>> 21:17:28 EST 2016 root@S1:/usr/obj/usr/src/sys/MYKERNEL amd64 >>>> >>>> /boot/loader.conf >>>> hw.ix.num_queues="4" >>>> >>>> /etc/iovctl.conf >>>> PF { >>>> device : ix1; >>>> num_vfs : 31; >>>> } >>>> >>>> DEFAULT { >>>> passthrough : true; >>>> } >>>> VF-0 { >>>> passthrough : false; >>>> } >>>> VF-1 { >>>> passthrough : false; >>>> } >>>> >>>> # iovctl -C -f /etc/iovctl.conf >>>> >>>> dmesg >>>> ixv0: >>> 1.4.6-k> at device 0.129 on pci12 >>>> ixv0: Using MSIX interrupts with 2 vectors >>>> ixv0: ixgbe_reset_hw() failed with error -100 >>>> device_attach: ixv0 attach returned 5 >>>> ixv0: >>> 1.4.6-k> at device 0.131 on pci12 >>>> ixv0: Using MSIX interrupts with 2 vectors >>>> ixv0: ixgbe_reset_hw() failed with error -100 >>>> device_attach: ixv0 attach returned 5 >>>> pci12: at device 0.133 (no driver attached) >>>> pci12: at device 0.135 (no driver attached) >>>> pci12: at device 0.137 (no driver attached) >>>> pci12: at device 0.139 (no driver attached) >>>> pci12: at device 0.141 (no driver attached) >>>> pci12: at device 0.143 (no driver attached) >>>> pci12: at device 0.145 (no driver attached) >>>> pci12: at device 0.147 (no driver attached) >>>> pci12: at device 0.149 (no driver attached) >>>> pci12: at device 0.151 (no driver attached) >>>> pci12: at device 0.153 (no driver attached) >>>> pci12: at device 0.155 (no driver attached) >>>> pci12: at device 0.157 (no driver attached) >>>> pci12: at device 0.159 (no driver attached) >>>> pci12: at device 0.161 (no driver attached) >>>> pci12: at device 0.163 (no driver attached) >>>> pci12: at device 0.165 (no driver attached) >>>> pci12: at device 0.167 (no driver attached) >>>> pci12: at device 0.169 (no driver attached) >>>> pci12: at device 0.171 (no driver attached) >>>> pci12: at device 0.173 (no driver attached) >>>> pci12: at device 0.175 (no driver attached) >>>> pci12: at device 0.177 (no driver attached) >>>> pci12: at device 0.179 (no driver attached) >>>> pci12: at device 0.181 (no driver attached) >>>> pci12: at device 0.183 (no driver attached) >>>> pci12: at device 0.185 (no driver attached) >>>> pci12: at device 0.187 (no driver attached) >>>> pci12: at device 0.189 (no driver attached) >>>> >>>> pciconf -lv >>>> ix1@pci0:129:0:1: class=0x020000 card=0x00001458 chip=0x15288086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'Ethernet Controller 10-Gigabit X540-AT2' >>>> class = network >>>> subclass = ethernet >>>> none155@pci0:129:0:129: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> none156@pci0:129:0:131: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt0@pci0:129:0:133: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt1@pci0:129:0:135: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt2@pci0:129:0:137: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt3@pci0:129:0:139: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt4@pci0:129:0:141: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt5@pci0:129:0:143: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt6@pci0:129:0:145: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt7@pci0:129:0:147: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt8@pci0:129:0:149: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt9@pci0:129:0:151: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt10@pci0:129:0:153: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt11@pci0:129:0:155: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt12@pci0:129:0:157: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt13@pci0:129:0:159: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt14@pci0:129:0:161: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt15@pci0:129:0:163: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt16@pci0:129:0:165: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt17@pci0:129:0:167: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt18@pci0:129:0:169: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt19@pci0:129:0:171: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt20@pci0:129:0:173: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt21@pci0:129:0:175: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt22@pci0:129:0:177: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt23@pci0:129:0:179: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt24@pci0:129:0:181: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt25@pci0:129:0:183: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt26@pci0:129:0:185: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt27@pci0:129:0:187: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> ppt28@pci0:129:0:189: class=0x020000 card=0x00001458 chip=0x15158086 >>>> rev=0x01 hdr=0x00 >>>> vendor = 'Intel Corporation' >>>> device = 'X540 Ethernet Controller Virtual Function' >>>> class = network >>>> subclass = ethernet >>>> >>>> Ultima >>>> _______________________________________________ >>>> freebsd-current@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>>> To unsubscribe, send any mail to " >> freebsd-current-unsubscribe@freebsd.org >>>> " >>>> >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"