From owner-freebsd-net@freebsd.org Thu Mar 26 10:39:00 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ADB4527B1D3 for ; Thu, 26 Mar 2020 10:39:00 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) by mx1.freebsd.org (Postfix) with ESMTP id 48p1g95WpVz4TrF; Thu, 26 Mar 2020 10:38:44 +0000 (UTC) (envelope-from snar@snar.spb.ru) Received: from staff.retn.net (staff.retn.net [IPv6:2a02:2d8:0:20ff:232a::8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: snarspb@) by staff.retn.net (Postfix) with ESMTPSA id 56C77CEB; Thu, 26 Mar 2020 13:20:14 +0300 (MSK) Date: Thu, 26 Mar 2020 13:20:08 +0300 From: Alexandre Snarskii To: Vincenzo Maffione Cc: Alexandre Snarskii , "freebsd-net@freebsd.org" Subject: Re: netmap/ixl and crc addition.. Message-ID: <20200326102008.GA54860@staff.retn.net> References: <20200324123721.GA26248@staff.retn.net> <20200324140839.GB26248@staff.retn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 48p1g95WpVz4TrF X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of snar@snar.spb.ru has no SPF policy when checking 2a02:2d8:0:20ff:232a::8) smtp.mailfrom=snar@snar.spb.ru X-Spamd-Result: default: False [0.57 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.12)[-0.117,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[snar.spb.ru]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.31)[-0.312,0]; R_SPF_NA(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:9002, ipnet:2a02:2d8::/32, country:EU]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-0.00)[country: EU(-0.01)]; ONCE_RECEIVED(0.10)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Mar 2020 10:39:00 -0000 On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote: > Hi Alexandre, >   Thanks. Your patch looks good to me. I assume you have tested it? Sure, this patch is already handles production traffic. > I will commit that to stable/11. Thanks. > > The issue you report on stable/12 is more worrisome. The 'no space in TX ring' > condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup (e.g. > TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving forward, > therefore freeing some space to be used in the ring. Unfortunately, further NIOCTXSYNC did not freed queue. Let me explain my observations in details: this issue was reproduceable even under really light load (say, hundreds pps per queue). As the load was ligth, I was able to insert debugging output before and after any calls to ioctl and poll, code is like: while(1) { if (nm_tx_pending(txring)) { printf("%s: pre-txsync cur: %d, head: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); if (ioctl(pa->fd, NIOCTXSYNC) == -1) { log event }; printf("%s: post-txsync cur: %d, haad: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); }; // setup poll descriptors with POLLIN only printf("%s: pre-poll cur: %d, head: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); ret = poll(fds, 2, 1000); printf("%s: post-poll cur: %d, head: %d, tail: %d\n", thread_name, txring->cur, txring->head, txring->tail); Logs in normal situation (only single thread shown): Notice:nz(ixl0:2): pre-txsync cur: 649, head: 649, tail: 647 Notice:nz(ixl0:2): post-txsync cur: 649, head: 649, tail: 648 Notice:nz(ixl0:2): pre-poll cur: 649, head: 649, tail: 648 Notice:nz(ixl0:2): post-poll cur: 649, head: 649, tail: 648 in pre-txsync: cur = head = tail + 2 (sometimes tail + 3), queue is nearly empty. in post-txsync: cur = head = tail + 1, queue is flushed. however, after some time, txcsync call advanced tail not to head - 1, but to head: Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 652 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 and following txcsyncs was not able to free (already empty) queue: Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654 Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654 application entered busy wait.. (in my case most packets just zero-copied from rxring to txring, so if there are no space in txring, packets are not consumed from rx). > However, the ring_reinit means that something is going wrong: either your > application is using the TX ring incorrectly, or there is a bug in the netmap ring_reinit was indeed caused by my incorrect use of txring: I tried to ignore situation head == tail and inject packet anyway.. > iflib code. Since FreeBSD 12, netmap support is provided by iflib, while before > netmap support was provided directly by the ixl driver. > In any case, it would probably help if you could provide some more detailed > info (how to reproduce the problem). > > Cheers, >   Vincenzo > > Il giorno mar 24 mar 2020 alle ore 15:12 Alexandre Snarskii > ha scritto: > > On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote: > > > > Hi! > > > > Long story short: looks like intel x722 does not by default add CRC to > > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is > > dropped on the next port.. Fix is simple, attached. > > ... add missing attach :( > > > > > The same behaviour of 'unconditionally ask card to compute crc' can > > be found in both if_ixl: > > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view= > markup#l408 > > and in DPDK i40e driver: > > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c# > L1105 > > so, I guess, it's safe. > > > > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while > > this upgrade solved the crc problem, this version shows 'stalled tx > queue' > > problem: after CTXSYNC tail == head == cur, 'no space in ring' condition. > > Attempts to ignore this condition led to continuous ring resets in > txcsync: > > > > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue        > ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t > 135 rh 135 rc 135 rt 135 hc 135 ht 135 > > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit        > called for ixl1 TX3 > > > > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with > > two onboard X722 ports (actually, four, but two of them are not > > wired). > > > > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU) > >   Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4 > > > > ixl0: 1.11.9-k> mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device > 0.0 numa-domain 0 on pci12 > > ixl0: using 1024 tx descriptors and 1024 rx descriptors > > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0 > > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C > > ixl0: Using MSIX interrupts with 9 vectors > > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active > > ixl0: Ethernet address: 3c:2c:30:30:59:85 > > ixl0: SR-IOV ready > > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024 > > > > ixl0@pci0:184:0:0:    class=0x020000 card=0x00008086 chip=0x37d38086 rev= > 0x04 hdr=0x00 > >     vendor     = 'Intel Corporation' > >     device     = 'Ethernet Connection X722 for 10GbE SFP+' > >     class      = network > >     subclass   = ethernet > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >