From owner-freebsd-net@freebsd.org Tue Mar 24 12:39:53 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5D5902A4FAE for ; Tue, 24 Mar 2020 12:39:53 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) by mx1.freebsd.org (Postfix) with ESMTP id 48mrRf1NgWz480M for ; Tue, 24 Mar 2020 12:39:40 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: snarspb@) by staff.retn.net (Postfix) with ESMTPSA id 31A4DC87 for ; Tue, 24 Mar 2020 15:37:45 +0300 (MSK) Date: Tue, 24 Mar 2020 15:37:36 +0300 From: Alexandre Snarskii To: freebsd-net@freebsd.org Subject: netmap/ixl and crc addition.. Message-ID: <20200324123721.GA26248@staff.retn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 48mrRf1NgWz480M X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of snar@snar.spb.ru has no SPF policy when checking 2a02:2d8:0:20ff:232a::8) smtp.mailfrom=snar@snar.spb.ru X-Spamd-Result: default: False [2.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.72)[0.725,0]; DMARC_NA(0.00)[snar.spb.ru]; NEURAL_SPAM_LONG(0.90)[0.904,0]; R_SPF_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:9002, ipnet:2a02:2d8::/32, country:EU]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.00)[country: EU(-0.01)]; ONCE_RECEIVED(0.10)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Mar 2020 12:39:53 -0000 Hi! Long story short: looks like intel x722 does not by default add CRC to outbound frames, so with FreeBSD 11-stable netmap-generated traffic is dropped on the next port.. Fix is simple, attached. The same behaviour of 'unconditionally ask card to compute crc' can be found in both if_ixl: https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=markup#l408 and in DPDK i40e driver: https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#L1105 so, I guess, it's safe. PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while this upgrade solved the crc problem, this version shows 'stalled tx queue' problem: after CTXSYNC tail == head == cur, 'no space in ring' condition. Attempts to ignore this condition led to continuous ring resets in txcsync: Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t 135 rh 135 rc 135 rt 135 hc 135 ht 135 Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit called for ixl1 TX3 PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with two onboard X722 ports (actually, four, but two of them are not wired). CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU) Origin="GenuineIntel" Id=0x50654 Family=0x6 Model=0x55 Stepping=4 ixl0: mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device 0.0 numa-domain 0 on pci12 ixl0: using 1024 tx descriptors and 1024 rx descriptors ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0 ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C ixl0: Using MSIX interrupts with 9 vectors ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active ixl0: Ethernet address: 3c:2c:30:30:59:85 ixl0: SR-IOV ready ixl0: netmap queues/slots: TX 8/1024, RX 8/1024 ixl0@pci0:184:0:0: class=0x020000 card=0x00008086 chip=0x37d38086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection X722 for 10GbE SFP+' class = network subclass = ethernet