From owner-freebsd-net@freebsd.org Fri Feb 28 19:40:05 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B665C243936 for ; Fri, 28 Feb 2020 19:40:05 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48TfyB5DKdz4kJv; Fri, 28 Feb 2020 19:40:02 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1j7lUM-000MgM-4T; Fri, 28 Feb 2020 22:39:58 +0300 Date: Fri, 28 Feb 2020 22:39:58 +0300 From: Slawa Olhovchenkov To: Vincenzo Maffione Cc: FreeBSD Net Subject: Re: Intel NETMAP performance and packet type Message-ID: <20200228193958.GR8012@zxy.spb.ru> References: <20200203204447.GD8028@zxy.spb.ru> <20200225150924.GM8012@zxy.spb.ru> <20200227201650.GO8012@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 48TfyB5DKdz4kJv X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru X-Spamd-Result: default: False [1.51 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.08)[-0.078,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.61)[0.606,0]; IP_SCORE(0.08)[asn: 5495(0.38), country: RU(0.01)]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 19:40:05 -0000 On Fri, Feb 28, 2020 at 06:31:36PM +0100, Vincenzo Maffione wrote: > Il giorno gio 27 feb 2020 alle ore 21:17 Slawa Olhovchenkov > ha scritto: > > > On Thu, Feb 27, 2020 at 06:51:54PM +0100, Vincenzo Maffione wrote: > > > > > Hi, > > > So, the issue is not the payload. > > > If you look at the avg_batch statistics reported by pkt-gen, you'll see > > > that in the ACK-flood experiment you have 4.92, whereas in the SYN-flood > > > case you have 17.5. The batch is the number of packets (well, actually > > > netmap descriptors, but in this case it's the same) that you receive (or > > > transmit) for each poll() invocation. > > > So in the first case you end up doing much more poll() calls, hence the > > > higher per-packet overhead and the lower packet-rate. > > > > > > Why is the poll() called more frequently? That depends on packet timing > > and > > > interrupt rate. There must be something different on your packet > > generator > > > that produces this effect (e.g. different burstiness, or maybe the packet > > > generator is not able to saturate the 10G link)? > > > > No, I am capture netstat output -- raw packet rate is the same. > > Also, I am change card to chelsio T5 and don't see issuse. > > > > This is payload issuse, at driver level. > > > > That's not possible, since netmap does not even look into the payload. Netmap don't look, NIC can look. I mean driver don't coretly/filly switch off all acceleration and ACK flood caused badly handled event in Intel driver. Don't see for chelsio. > Can you please report the per-queue interrupt rate in both cases (ACK-flood > and SYN-flood)? > You can use something like `vmstat -i -w1 | grep ix` to monitor the > interrupt rate. > Or probably you can also use `sysctl -a dev.ix | grep interrupt_rate` > > > > > In any case, I would suggest measuring the RX interrupt rate, and check > > > that it's higher in the ACK-flood case. Then you can try to lower the > > > interrupt rate by tuning the interrupt moderation features of the Intel > > NIC > > > (e,g. limit hw.ix.max_interrupt_rate and disable hw.ix.enable_aim or > > > similar). > > > By playing with the interrupt moderation you should be able to increase > > the > > > avg_batch, and then increase throghput. > > > > Already limited. > > > > Limited to which value? Have you tried to decrease max_interrupt_rate even > more? 16000, don't try different values for this case.