Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 26 Mar 2020 13:20:08 +0300
From:      Alexandre Snarskii <snar@snar.spb.ru>
To:        Vincenzo Maffione <vmaffione@freebsd.org>
Cc:        Alexandre Snarskii <snar@snar.spb.ru>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>
Subject:   Re: netmap/ixl and crc addition..
Message-ID:  <20200326102008.GA54860@staff.retn.net>
In-Reply-To: <CA%2B_eA9jTbqiso_UB0D2jmRVXJXmMMkxCjnH_92FL4GW7SLMOow@mail.gmail.com>
References:  <20200324123721.GA26248@staff.retn.net> <20200324140839.GB26248@staff.retn.net> <CA%2B_eA9jTbqiso_UB0D2jmRVXJXmMMkxCjnH_92FL4GW7SLMOow@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote:
> Hi Alexandre,
>   Thanks. Your patch looks good to me. I assume you have tested it?

Sure, this patch is already handles production traffic.

> I will commit that to stable/11.

Thanks.

> 
> The issue you report on stable/12 is more worrisome. The 'no space in TX ring'
> condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup (e.g.
> TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving forward,
> therefore freeing some space to be used in the ring.

Unfortunately, further NIOCTXSYNC did not freed queue. 

Let me explain my observations in details: this issue was reproduceable 
even under really light load (say, hundreds pps per queue). As the load 
was ligth, I was able to insert debugging output before and after any calls 
to ioctl and poll, code is like:

while(1) {
	if (nm_tx_pending(txring)) {
		printf("%s: pre-txsync cur: %d, head: %d, tail: %d\n",
			thread_name, txring->cur, txring->head, txring->tail);
		if (ioctl(pa->fd, NIOCTXSYNC) == -1) {
			log event
		};
		printf("%s: post-txsync cur: %d, haad: %d, tail: %d\n",
			thread_name, txring->cur, txring->head, txring->tail);
	};
	// setup poll descriptors with POLLIN only
	printf("%s: pre-poll cur: %d, head: %d, tail: %d\n",
		thread_name, txring->cur, txring->head, txring->tail);
    ret = poll(fds, 2, 1000);
	printf("%s: post-poll cur: %d, head: %d, tail: %d\n",
		thread_name, txring->cur, txring->head, txring->tail);
	
Logs in normal situation (only single thread shown):

Notice:nz(ixl0:2): pre-txsync cur: 649, head: 649, tail: 647
Notice:nz(ixl0:2): post-txsync cur: 649, head: 649, tail: 648
Notice:nz(ixl0:2): pre-poll cur: 649, head: 649, tail: 648
Notice:nz(ixl0:2): post-poll cur: 649, head: 649, tail: 648

in pre-txsync: cur = head = tail + 2 (sometimes tail + 3), queue
is nearly empty.
in post-txsync: cur = head = tail + 1, queue is flushed.

however, after some time, txcsync call advanced tail not to head - 1, 
but to head:

Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 652
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654

and following txcsyncs was not able to free (already empty) queue:

Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654

application entered busy wait.. (in my case most packets just zero-copied
from rxring to txring, so if there are no space in txring, packets are not
consumed from rx).

> However, the ring_reinit means that something is going wrong: either your
> application is using the TX ring incorrectly, or there is a bug in the netmap

ring_reinit was indeed caused by my incorrect use of txring: I 
tried to ignore situation head == tail and inject packet anyway.. 

> iflib code. Since FreeBSD 12, netmap support is provided by iflib, while before
> netmap support was provided directly by the ixl driver.
> In any case, it would probably help if you could provide some more detailed
> info (how to reproduce the problem).
> 
> Cheers,
>   Vincenzo
> 
> Il giorno mar 24 mar 2020 alle ore 15:12 Alexandre Snarskii <snar@snar.spb.ru>
> ha scritto:
> 
>     On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote:
>     >
>     > Hi!
>     >
>     > Long story short: looks like intel x722 does not by default add CRC to
>     > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is
>     > dropped on the next port.. Fix is simple, attached.
> 
>     ... add missing attach :(
> 
>     >
>     > The same behaviour of 'unconditionally ask card to compute crc' can
>     > be found in both if_ixl:
>     > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=
>     markup#l408
>     > and in DPDK i40e driver:
>     > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#
>     L1105
>     > so, I guess, it's safe.
>     >
>     > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while
>     > this upgrade solved the crc problem, this version shows 'stalled tx
>     queue'
>     > problem: after CTXSYNC tail == head == cur, 'no space in ring' condition.
>     > Attempts to ignore this condition led to continuous ring resets in
>     txcsync:
>     >
>     > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue       
>     ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t
>     135 rh 135 rc 135 rt 135 hc 135 ht 135
>     > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit       
>     called for ixl1 TX3
>     >
>     > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with
>     > two onboard X722 ports (actually, four, but two of them are not
>     > wired).
>     >
>     > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU)
>     >   Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4
>     >
>     > ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version -
>     1.11.9-k> mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device
>     0.0 numa-domain 0 on pci12
>     > ixl0: using 1024 tx descriptors and 1024 rx descriptors
>     > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0
>     > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C
>     > ixl0: Using MSIX interrupts with 9 vectors
>     > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
>     > ixl0: Ethernet address: 3c:2c:30:30:59:85
>     > ixl0: SR-IOV ready
>     > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
>     >
>     > ixl0@pci0:184:0:0:    class=0x020000 card=0x00008086 chip=0x37d38086 rev=
>     0x04 hdr=0x00
>     >     vendor     = 'Intel Corporation'
>     >     device     = 'Ethernet Connection X722 for 10GbE SFP+'
>     >     class      = network
>     >     subclass   = ethernet
>     >
>     >
>     > _______________________________________________
>     > freebsd-net@freebsd.org mailing list
>     > https://lists.freebsd.org/mailman/listinfo/freebsd-net
>     > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>     _______________________________________________
>     freebsd-net@freebsd.org mailing list
>     https://lists.freebsd.org/mailman/listinfo/freebsd-net
>     To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
> 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200326102008.GA54860>