Date: Thu, 26 Mar 2020 13:20:08 +0300 From: Alexandre Snarskii <snar@snar.spb.ru> To: Vincenzo Maffione <vmaffione@freebsd.org> Cc: Alexandre Snarskii <snar@snar.spb.ru>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: netmap/ixl and crc addition.. Message-ID: <20200326102008.GA54860@staff.retn.net> In-Reply-To: <CA%2B_eA9jTbqiso_UB0D2jmRVXJXmMMkxCjnH_92FL4GW7SLMOow@mail.gmail.com> References: <20200324123721.GA26248@staff.retn.net> <20200324140839.GB26248@staff.retn.net> <CA%2B_eA9jTbqiso_UB0D2jmRVXJXmMMkxCjnH_92FL4GW7SLMOow@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote:
> Hi Alexandre,
> Thanks. Your patch looks good to me. I assume you have tested it?
Sure, this patch is already handles production traffic.
> I will commit that to stable/11.
Thanks.
>
> The issue you report on stable/12 is more worrisome. The 'no space in TX ring'
> condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup (e.g.
> TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving forward,
> therefore freeing some space to be used in the ring.
Unfortunately, further NIOCTXSYNC did not freed queue.
Let me explain my observations in details: this issue was reproduceable
even under really light load (say, hundreds pps per queue). As the load
was ligth, I was able to insert debugging output before and after any calls
to ioctl and poll, code is like:
while(1) {
if (nm_tx_pending(txring)) {
printf("%s: pre-txsync cur: %d, head: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);
if (ioctl(pa->fd, NIOCTXSYNC) == -1) {
log event
};
printf("%s: post-txsync cur: %d, haad: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);
};
// setup poll descriptors with POLLIN only
printf("%s: pre-poll cur: %d, head: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);
ret = poll(fds, 2, 1000);
printf("%s: post-poll cur: %d, head: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);
Logs in normal situation (only single thread shown):
Notice:nz(ixl0:2): pre-txsync cur: 649, head: 649, tail: 647
Notice:nz(ixl0:2): post-txsync cur: 649, head: 649, tail: 648
Notice:nz(ixl0:2): pre-poll cur: 649, head: 649, tail: 648
Notice:nz(ixl0:2): post-poll cur: 649, head: 649, tail: 648
in pre-txsync: cur = head = tail + 2 (sometimes tail + 3), queue
is nearly empty.
in post-txsync: cur = head = tail + 1, queue is flushed.
however, after some time, txcsync call advanced tail not to head - 1,
but to head:
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 652
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
and following txcsyncs was not able to free (already empty) queue:
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
application entered busy wait.. (in my case most packets just zero-copied
from rxring to txring, so if there are no space in txring, packets are not
consumed from rx).
> However, the ring_reinit means that something is going wrong: either your
> application is using the TX ring incorrectly, or there is a bug in the netmap
ring_reinit was indeed caused by my incorrect use of txring: I
tried to ignore situation head == tail and inject packet anyway..
> iflib code. Since FreeBSD 12, netmap support is provided by iflib, while before
> netmap support was provided directly by the ixl driver.
> In any case, it would probably help if you could provide some more detailed
> info (how to reproduce the problem).
>
> Cheers,
> Vincenzo
>
> Il giorno mar 24 mar 2020 alle ore 15:12 Alexandre Snarskii <snar@snar.spb.ru>
> ha scritto:
>
> On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote:
> >
> > Hi!
> >
> > Long story short: looks like intel x722 does not by default add CRC to
> > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is
> > dropped on the next port.. Fix is simple, attached.
>
> ... add missing attach :(
>
> >
> > The same behaviour of 'unconditionally ask card to compute crc' can
> > be found in both if_ixl:
> > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=
> markup#l408
> > and in DPDK i40e driver:
> > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#
> L1105
> > so, I guess, it's safe.
> >
> > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while
> > this upgrade solved the crc problem, this version shows 'stalled tx
> queue'
> > problem: after CTXSYNC tail == head == cur, 'no space in ring' condition.
> > Attempts to ignore this condition led to continuous ring resets in
> txcsync:
> >
> > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue
> ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t
> 135 rh 135 rc 135 rt 135 hc 135 ht 135
> > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit
> called for ixl1 TX3
> >
> > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with
> > two onboard X722 ports (actually, four, but two of them are not
> > wired).
> >
> > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU)
> > Origin="GenuineIntel" Id=0x50654 Family=0x6 Model=0x55 Stepping=4
> >
> > ixl0: <Intel(R) Ethernet Connection 700 Series PF Driver, Version -
> 1.11.9-k> mem 0xfa000000-0xfaffffff,0xfb018000-0xfb01ffff irq 11 at device
> 0.0 numa-domain 0 on pci12
> > ixl0: using 1024 tx descriptors and 1024 rx descriptors
> > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0
> > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C
> > ixl0: Using MSIX interrupts with 9 vectors
> > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
> > ixl0: Ethernet address: 3c:2c:30:30:59:85
> > ixl0: SR-IOV ready
> > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
> >
> > ixl0@pci0:184:0:0: class=0x020000 card=0x00008086 chip=0x37d38086 rev=
> 0x04 hdr=0x00
> > vendor = 'Intel Corporation'
> > device = 'Ethernet Connection X722 for 10GbE SFP+'
> > class = network
> > subclass = ethernet
> >
> >
> > _______________________________________________
> > freebsd-net@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200326102008.GA54860>