Date: Fri, 28 Feb 2020 22:52:18 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Vincenzo Maffione <vmaffione@freebsd.org> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Intel NETMAP performance and packet type Message-ID: <20200228195218.GS8012@zxy.spb.ru> In-Reply-To: <CA%2B_eA9h84Qa8P59_BQ=eQEcdC4D_v6Q-DWW8q=e5m%2BvOUP92zA@mail.gmail.com> References: <20200203204447.GD8028@zxy.spb.ru> <CA%2B_eA9it5iB7ERiDx0b7zN54atPZz64wW75nWeZrqTD58SWyLQ@mail.gmail.com> <20200225150924.GM8012@zxy.spb.ru> <CA%2B_eA9hqsUj7bXnq0%2BprqsxTr7_f9WE-5%2BJwUA==Xy0PvXAG=Q@mail.gmail.com> <20200227201650.GO8012@zxy.spb.ru> <20200228112602.GP8012@zxy.spb.ru> <CA%2B_eA9h84Qa8P59_BQ=eQEcdC4D_v6Q-DWW8q=e5m%2BvOUP92zA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Feb 28, 2020 at 06:37:22PM +0100, Vincenzo Maffione wrote: > Il giorno ven 28 feb 2020 alle ore 12:26 Slawa Olhovchenkov <slw@zxy.spb.ru> > ha scritto: > > > On Thu, Feb 27, 2020 at 11:16:50PM +0300, Slawa Olhovchenkov wrote: > > > > > On Thu, Feb 27, 2020 at 06:51:54PM +0100, Vincenzo Maffione wrote: > > > > > > > Hi, > > > > So, the issue is not the payload. > > > > If you look at the avg_batch statistics reported by pkt-gen, you'll see > > > > that in the ACK-flood experiment you have 4.92, whereas in the > > SYN-flood > > > > case you have 17.5. The batch is the number of packets (well, actually > > > > netmap descriptors, but in this case it's the same) that you receive > > (or > > > > transmit) for each poll() invocation. > > > > So in the first case you end up doing much more poll() calls, hence the > > > > higher per-packet overhead and the lower packet-rate. > > > > > > > > Why is the poll() called more frequently? That depends on packet > > timing and > > > > interrupt rate. There must be something different on your packet > > generator > > > > that produces this effect (e.g. different burstiness, or maybe the > > packet > > > > generator is not able to saturate the 10G link)? > > > > > > No, I am capture netstat output -- raw packet rate is the same. > > > Also, I am change card to chelsio T5 and don't see issuse. > > > > > > This is payload issuse, at driver level. > > > > > > > In any case, I would suggest measuring the RX interrupt rate, and check > > > > that it's higher in the ACK-flood case. Then you can try to lower the > > > > interrupt rate by tuning the interrupt moderation features of the > > Intel NIC > > > > (e,g. limit hw.ix.max_interrupt_rate and disable hw.ix.enable_aim or > > > > similar). > > > > By playing with the interrupt moderation you should be able to > > increase the > > > > avg_batch, and then increase throghput. > > > > > > Already limited. > > > > Also, is this normal (rxd_tail == rxd_head): > > > > dev.ix.0.queue0.rx_discarded: 0 > > dev.ix.0.queue0.rx_copies: 0 > > dev.ix.0.queue0.rx_bytes: 612041623304 > > dev.ix.0.queue0.rx_packets: 9563149414 > > dev.ix.0.queue0.rxd_tail: 1120 > > dev.ix.0.queue0.rxd_head: 1120 > > dev.ix.0.queue0.irqs: 40154885 > > dev.ix.0.queue0.interrupt_rate: 16129 > > dev.ix.0.queue0.tx_packets: 553897984 > > dev.ix.0.queue0.tso_tx: 0 > > dev.ix.0.queue0.txd_tail: 0 > > dev.ix.0.queue0.txd_head: 0 > > > > I am see this RX queue is stoped. > > Ah, may fault. This is different case on same Intel card, this is no ACK-flood. And I am see this on production traffic. > Yes, (rxd_tail == rxd_head) means that the NIC ran out of RX buffers. > rxd_head is the next descriptor that the NIC will use. rxd_tail is the next > descriptor that the driver will replenish. RX buffers are replenished by > the netmap NIOCRXSYNC routine, which is called on poll(). poll() still called frequenced but rxd_head/rxd_tail stalled. > However, rx_discarded is 0, which means that the NIC is not dropping > packets. So the problem should not be that poll() is not called frequently > enough. poll() called for all queue synchronously for multiple queue, stalled only one. > You should check rx_discarded for all the queues. All zero. > Another thing you need to check is how the load is balanced across the > receive queues. How many have you configured? Maybe the two workloads > (SYN-flood and ACK-flood) load different queues in different ways. Sorry, this is different case: after some time (after hour, for example) some queue stalled infinitly. Rest queue handle traffic. This is Intel card, iflib variant.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200228195218.GS8012>