From owner-freebsd-net@freebsd.org Fri Feb 28 17:37:35 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2E70F24026C for ; Fri, 28 Feb 2020 17:37:35 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48TcDt4xP3z48rT for ; Fri, 28 Feb 2020 17:37:34 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id 276A83337 for ; Fri, 28 Feb 2020 17:37:34 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-qk1-f175.google.com with SMTP id m2so3722364qka.7 for ; Fri, 28 Feb 2020 09:37:34 -0800 (PST) X-Gm-Message-State: APjAAAXG2IMmzorPhh2uqPlHl49Aci+MDXobg/lQbA4bAzwy3LwqFqM9 wJCx5fsfmtYrGqE6WG+yZNGFGf6slxhL8gWfUqI= X-Google-Smtp-Source: APXvYqwFuchNJkKc+zmev0av1VhvsQ4UyoD0pqODdYIVplrdcbRq/Pqi6JEpYfmAvhfhR62t6aHTu2+avtjvKDYEATA= X-Received: by 2002:ae9:c119:: with SMTP id z25mr4868153qki.407.1582911453736; Fri, 28 Feb 2020 09:37:33 -0800 (PST) MIME-Version: 1.0 References: <20200203204447.GD8028@zxy.spb.ru> <20200225150924.GM8012@zxy.spb.ru> <20200227201650.GO8012@zxy.spb.ru> <20200228112602.GP8012@zxy.spb.ru> In-Reply-To: <20200228112602.GP8012@zxy.spb.ru> From: Vincenzo Maffione Date: Fri, 28 Feb 2020 18:37:22 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Intel NETMAP performance and packet type To: Slawa Olhovchenkov , FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 17:37:35 -0000 Il giorno ven 28 feb 2020 alle ore 12:26 Slawa Olhovchenkov ha scritto: > On Thu, Feb 27, 2020 at 11:16:50PM +0300, Slawa Olhovchenkov wrote: > > > On Thu, Feb 27, 2020 at 06:51:54PM +0100, Vincenzo Maffione wrote: > > > > > Hi, > > > So, the issue is not the payload. > > > If you look at the avg_batch statistics reported by pkt-gen, you'll see > > > that in the ACK-flood experiment you have 4.92, whereas in the > SYN-flood > > > case you have 17.5. The batch is the number of packets (well, actually > > > netmap descriptors, but in this case it's the same) that you receive > (or > > > transmit) for each poll() invocation. > > > So in the first case you end up doing much more poll() calls, hence the > > > higher per-packet overhead and the lower packet-rate. > > > > > > Why is the poll() called more frequently? That depends on packet > timing and > > > interrupt rate. There must be something different on your packet > generator > > > that produces this effect (e.g. different burstiness, or maybe the > packet > > > generator is not able to saturate the 10G link)? > > > > No, I am capture netstat output -- raw packet rate is the same. > > Also, I am change card to chelsio T5 and don't see issuse. > > > > This is payload issuse, at driver level. > > > > > In any case, I would suggest measuring the RX interrupt rate, and check > > > that it's higher in the ACK-flood case. Then you can try to lower the > > > interrupt rate by tuning the interrupt moderation features of the > Intel NIC > > > (e,g. limit hw.ix.max_interrupt_rate and disable hw.ix.enable_aim or > > > similar). > > > By playing with the interrupt moderation you should be able to > increase the > > > avg_batch, and then increase throghput. > > > > Already limited. > > Also, is this normal (rxd_tail == rxd_head): > > dev.ix.0.queue0.rx_discarded: 0 > dev.ix.0.queue0.rx_copies: 0 > dev.ix.0.queue0.rx_bytes: 612041623304 > dev.ix.0.queue0.rx_packets: 9563149414 > dev.ix.0.queue0.rxd_tail: 1120 > dev.ix.0.queue0.rxd_head: 1120 > dev.ix.0.queue0.irqs: 40154885 > dev.ix.0.queue0.interrupt_rate: 16129 > dev.ix.0.queue0.tx_packets: 553897984 > dev.ix.0.queue0.tso_tx: 0 > dev.ix.0.queue0.txd_tail: 0 > dev.ix.0.queue0.txd_head: 0 > > I am see this RX queue is stoped. > Yes, (rxd_tail == rxd_head) means that the NIC ran out of RX buffers. rxd_head is the next descriptor that the NIC will use. rxd_tail is the next descriptor that the driver will replenish. RX buffers are replenished by the netmap NIOCRXSYNC routine, which is called on poll(). However, rx_discarded is 0, which means that the NIC is not dropping packets. So the problem should not be that poll() is not called frequently enough. You should check rx_discarded for all the queues. Another thing you need to check is how the load is balanced across the receive queues. How many have you configured? Maybe the two workloads (SYN-flood and ACK-flood) load different queues in different ways.