From owner-freebsd-net@freebsd.org Tue Mar 24 14:10:53 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C009B2A6BE0 for ; Tue, 24 Mar 2020 14:10:53 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) by mx1.freebsd.org (Postfix) with ESMTP id 48mtSY08fFz3D8Y for ; Tue, 24 Mar 2020 14:10:36 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: snarspb@) by staff.retn.net (Postfix) with ESMTPSA id F1F82DF8; Tue, 24 Mar 2020 17:08:40 +0300 (MSK) Date: Tue, 24 Mar 2020 17:08:39 +0300 From: Alexandre Snarskii To: freebsd-net@freebsd.org Cc: Alexandre Snarskii Subject: Re: netmap/ixl and crc addition.. Message-ID: <20200324140839.GB26248@staff.retn.net> References: <20200324123721.GA26248@staff.retn.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="J2SCkAp4GZ/dPZZf" Content-Disposition: inline In-Reply-To: <20200324123721.GA26248@staff.retn.net> User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 48mtSY08fFz3D8Y X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=temperror (mx1.freebsd.org: error in processing during lookup of snar@snar.spb.ru: DNS error) smtp.mailfrom=snar@snar.spb.ru X-Spamd-Result: default: False [-1.58 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_FAIL(0.00)[8.0.0.0.0.0.0.0.0.0.0.0.a.2.3.2.f.f.0.2.0.0.0.0.8.d.2.0.2.0.a.2.list.dnswl.org:query timed out]; R_SPF_DNSFAIL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.79)[-0.793,0]; HAS_ATTACHMENT(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; DMARC_NA(0.00)[snar.spb.ru]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.79)[-0.788,0]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:9002, ipnet:2a02:2d8::/32, country:EU]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.00)[country: EU(-0.01)]; ONCE_RECEIVED(0.10)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 14:10:54 -0000 --J2SCkAp4GZ/dPZZf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote: > > Hi! > > Long story short: looks like intel x722 does not by default add CRC to > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is > dropped on the next port.. Fix is simple, attached. ... add missing attach :( > > The same behaviour of 'unconditionally ask card to compute crc' can > be found in both if_ixl: > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=markup#l408 > and in DPDK i40e driver: > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#L1105 > so, I guess, it's safe. > > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while > this upgrade solved the crc problem, this version shows 'stalled tx queue' > problem: after CTXSYNC tail == head == cur, 'no space in ring' condition. > Attempts to ignore this condition led to continuous ring resets in txcsync: > > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t 135 rh 135 rc 135 rt 135 hc 135 ht 135 > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit called for ixl1 TX3 > > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with > two onboard X722 ports (actually, four, but two of them are not > wired). > > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x50654 Family=0x6 Model=0x55 Stepping=4 > > ixl0: mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device 0.0 numa-domain 0 on pci12 > ixl0: using 1024 tx descriptors and 1024 rx descriptors > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0 > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C > ixl0: Using MSIX interrupts with 9 vectors > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active > ixl0: Ethernet address: 3c:2c:30:30:59:85 > ixl0: SR-IOV ready > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024 > > ixl0@pci0:184:0:0: class=0x020000 card=0x00008086 chip=0x37d38086 rev=0x04 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Ethernet Connection X722 for 10GbE SFP+' > class = network > subclass = ethernet > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --J2SCkAp4GZ/dPZZf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="if_ixl_netmap.h.diff" Index: sys/dev/netmap/if_ixl_netmap.h =================================================================== --- sys/dev/netmap/if_ixl_netmap.h (revision 359243) +++ sys/dev/netmap/if_ixl_netmap.h (working copy) @@ -239,7 +239,8 @@ curr->cmd_type_offset_bsz = htole64( ((u64)len << I40E_TXD_QW1_TX_BUF_SZ_SHIFT) | flags | - ((u64)I40E_TX_DESC_CMD_EOP << I40E_TXD_QW1_CMD_SHIFT) + ((u64)I40E_TX_DESC_CMD_EOP << I40E_TXD_QW1_CMD_SHIFT) | + ((u64)I40E_TX_DESC_CMD_ICRC << I40E_TXD_QW1_CMD_SHIFT) ); // XXX more ? /* make sure changes to the buffer are synced */ --J2SCkAp4GZ/dPZZf--