From owner-freebsd-net@freebsd.org Fri Feb 28 17:31:48 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3D526267E20 for ; Fri, 28 Feb 2020 17:31:48 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48Tc6C6ktSz3Pys for ; Fri, 28 Feb 2020 17:31:47 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: vmaffione) by smtp.freebsd.org (Postfix) with ESMTPSA id B22B43334 for ; Fri, 28 Feb 2020 17:31:47 +0000 (UTC) (envelope-from vmaffione@freebsd.org) Received: by mail-qk1-f182.google.com with SMTP id u124so3664635qkh.13 for ; Fri, 28 Feb 2020 09:31:47 -0800 (PST) X-Gm-Message-State: APjAAAW/dZFJI2i6wKYZoM0h6au6NMwdwdikFVUGhyY+hetQZN3tX2By CzAvb1FwObIv9M8z0ZnZFouwoJ2gBh/P+lXSudA= X-Google-Smtp-Source: APXvYqxc3uIz2C7tCvmoQP+E6DFmQlSe4NwwOBGAmMLjFYBZjBByzNnTExAaEudQqJ/kv+JR+nvdA9w3X0kCqb2nfIo= X-Received: by 2002:a37:61c3:: with SMTP id v186mr5420920qkb.96.1582911106943; Fri, 28 Feb 2020 09:31:46 -0800 (PST) MIME-Version: 1.0 References: <20200203204447.GD8028@zxy.spb.ru> <20200225150924.GM8012@zxy.spb.ru> <20200227201650.GO8012@zxy.spb.ru> In-Reply-To: <20200227201650.GO8012@zxy.spb.ru> From: Vincenzo Maffione Date: Fri, 28 Feb 2020 18:31:36 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Intel NETMAP performance and packet type To: Slawa Olhovchenkov , FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 17:31:48 -0000 Il giorno gio 27 feb 2020 alle ore 21:17 Slawa Olhovchenkov ha scritto: > On Thu, Feb 27, 2020 at 06:51:54PM +0100, Vincenzo Maffione wrote: > > > Hi, > > So, the issue is not the payload. > > If you look at the avg_batch statistics reported by pkt-gen, you'll see > > that in the ACK-flood experiment you have 4.92, whereas in the SYN-flood > > case you have 17.5. The batch is the number of packets (well, actually > > netmap descriptors, but in this case it's the same) that you receive (or > > transmit) for each poll() invocation. > > So in the first case you end up doing much more poll() calls, hence the > > higher per-packet overhead and the lower packet-rate. > > > > Why is the poll() called more frequently? That depends on packet timing > and > > interrupt rate. There must be something different on your packet > generator > > that produces this effect (e.g. different burstiness, or maybe the packet > > generator is not able to saturate the 10G link)? > > No, I am capture netstat output -- raw packet rate is the same. > Also, I am change card to chelsio T5 and don't see issuse. > > This is payload issuse, at driver level. > That's not possible, since netmap does not even look into the payload. Can you please report the per-queue interrupt rate in both cases (ACK-flood and SYN-flood)? You can use something like `vmstat -i -w1 | grep ix` to monitor the interrupt rate. Or probably you can also use `sysctl -a dev.ix | grep interrupt_rate` > > In any case, I would suggest measuring the RX interrupt rate, and check > > that it's higher in the ACK-flood case. Then you can try to lower the > > interrupt rate by tuning the interrupt moderation features of the Intel > NIC > > (e,g. limit hw.ix.max_interrupt_rate and disable hw.ix.enable_aim or > > similar). > > By playing with the interrupt moderation you should be able to increase > the > > avg_batch, and then increase throghput. > > Already limited. > Limited to which value? Have you tried to decrease max_interrupt_rate even more? > > > Cheers, > > Vincenzo > > >