Date: Fri, 28 Feb 2020 18:37:22 +0100 From: Vincenzo Maffione <vmaffione@freebsd.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru>, FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Intel NETMAP performance and packet type Message-ID: <CA%2B_eA9h84Qa8P59_BQ=eQEcdC4D_v6Q-DWW8q=e5m%2BvOUP92zA@mail.gmail.com> In-Reply-To: <20200228112602.GP8012@zxy.spb.ru> References: <20200203204447.GD8028@zxy.spb.ru> <CA%2B_eA9it5iB7ERiDx0b7zN54atPZz64wW75nWeZrqTD58SWyLQ@mail.gmail.com> <20200225150924.GM8012@zxy.spb.ru> <CA%2B_eA9hqsUj7bXnq0%2BprqsxTr7_f9WE-5%2BJwUA==Xy0PvXAG=Q@mail.gmail.com> <20200227201650.GO8012@zxy.spb.ru> <20200228112602.GP8012@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
Il giorno ven 28 feb 2020 alle ore 12:26 Slawa Olhovchenkov <slw@zxy.spb.ru> ha scritto: > On Thu, Feb 27, 2020 at 11:16:50PM +0300, Slawa Olhovchenkov wrote: > > > On Thu, Feb 27, 2020 at 06:51:54PM +0100, Vincenzo Maffione wrote: > > > > > Hi, > > > So, the issue is not the payload. > > > If you look at the avg_batch statistics reported by pkt-gen, you'll see > > > that in the ACK-flood experiment you have 4.92, whereas in the > SYN-flood > > > case you have 17.5. The batch is the number of packets (well, actually > > > netmap descriptors, but in this case it's the same) that you receive > (or > > > transmit) for each poll() invocation. > > > So in the first case you end up doing much more poll() calls, hence the > > > higher per-packet overhead and the lower packet-rate. > > > > > > Why is the poll() called more frequently? That depends on packet > timing and > > > interrupt rate. There must be something different on your packet > generator > > > that produces this effect (e.g. different burstiness, or maybe the > packet > > > generator is not able to saturate the 10G link)? > > > > No, I am capture netstat output -- raw packet rate is the same. > > Also, I am change card to chelsio T5 and don't see issuse. > > > > This is payload issuse, at driver level. > > > > > In any case, I would suggest measuring the RX interrupt rate, and check > > > that it's higher in the ACK-flood case. Then you can try to lower the > > > interrupt rate by tuning the interrupt moderation features of the > Intel NIC > > > (e,g. limit hw.ix.max_interrupt_rate and disable hw.ix.enable_aim or > > > similar). > > > By playing with the interrupt moderation you should be able to > increase the > > > avg_batch, and then increase throghput. > > > > Already limited. > > Also, is this normal (rxd_tail == rxd_head): > > dev.ix.0.queue0.rx_discarded: 0 > dev.ix.0.queue0.rx_copies: 0 > dev.ix.0.queue0.rx_bytes: 612041623304 > dev.ix.0.queue0.rx_packets: 9563149414 > dev.ix.0.queue0.rxd_tail: 1120 > dev.ix.0.queue0.rxd_head: 1120 > dev.ix.0.queue0.irqs: 40154885 > dev.ix.0.queue0.interrupt_rate: 16129 > dev.ix.0.queue0.tx_packets: 553897984 > dev.ix.0.queue0.tso_tx: 0 > dev.ix.0.queue0.txd_tail: 0 > dev.ix.0.queue0.txd_head: 0 > > I am see this RX queue is stoped. > Yes, (rxd_tail == rxd_head) means that the NIC ran out of RX buffers. rxd_head is the next descriptor that the NIC will use. rxd_tail is the next descriptor that the driver will replenish. RX buffers are replenished by the netmap NIOCRXSYNC routine, which is called on poll(). However, rx_discarded is 0, which means that the NIC is not dropping packets. So the problem should not be that poll() is not called frequently enough. You should check rx_discarded for all the queues. Another thing you need to check is how the load is balanced across the receive queues. How many have you configured? Maybe the two workloads (SYN-flood and ACK-flood) load different queues in different ways.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B_eA9h84Qa8P59_BQ=eQEcdC4D_v6Q-DWW8q=e5m%2BvOUP92zA>