From owner-freebsd-drivers@freebsd.org Sun Aug 30 07:20:37 2020 Return-Path: Delivered-To: freebsd-drivers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DF4453CF711 for ; Sun, 30 Aug 2020 07:20:37 +0000 (UTC) (envelope-from support@fleekdemo.me) Received: from host.athar.pk (host.athar.pk [45.32.126.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BfPqy5ZYpz4nx7 for ; Sun, 30 Aug 2020 07:20:30 +0000 (UTC) (envelope-from support@fleekdemo.me) Received: by host.athar.pk (Postfix, from userid 1000) id 5B097D91E; Sun, 30 Aug 2020 07:20:16 +0000 (UTC) To: freebsd-drivers@freebsd.org Subject: =?UTF-8?Q?Wir_bemerkten_eine_ungew=C3=B6hnliche_Aktivit=C3=A4t?= X-PHP-Originating-Script: 1000:art.php Date: Sun, 30 Aug 2020 07:20:16 +0000 From: PayPal Message-ID: MIME-Version: 1.0 X-Rspamd-Queue-Id: 4BfPqy5ZYpz4nx7 X-Spamd-Bar: ++++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of support@fleekdemo.me has no SPF policy when checking 45.32.126.171) smtp.mailfrom=support@fleekdemo.me X-Spamd-Result: default: False [6.23 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; AUTOGEN_PHP_SPAMMY(1.00)[]; HAS_X_POS(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; HAS_PHPMAILER_SIG(0.00)[]; AUTH_NA(1.00)[]; URI_COUNT_ODD(1.00)[1]; RCPT_COUNT_ONE(0.00)[1]; MANY_INVISIBLE_PARTS(0.05)[1]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_SHORT(0.70)[0.698]; NEURAL_SPAM_MEDIUM(0.78)[0.781]; NEURAL_SPAM_LONG(0.97)[0.969]; DMARC_NA(0.00)[fleekdemo.me]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; R_MIXED_CHARSET(0.83)[subject]; ASN(0.00)[asn:20473, ipnet:45.32.96.0/19, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MAILMAN_DEST(0.00)[freebsd-drivers] X-Spam: Yes Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Aug 2020 07:20:38 -0000 PayPal     Wir bemerkten eine ungewöhnliche Aktivität Konto Begrenzung. Wir Benötigen Sie Ihre Hilfe beim Sichern Ihres a-Kontos, um unbefugten Zugriff zu verhindern. Für Ihre Sicherheit kann es einige Einschränkungen für Ihr Konto geben, bis Sie Maßnahmen ergreifen. Schließen Ihr Konto. Wir wird Ihr Konto nach 1 Tag (24 Stunden) schließen und Sie werden gesperrt dauerhaft von unserer Website. Wie um die Schließung Ihres Kontos zu vermeiden. Alles Wir benötigen Ihre Hilfe bei der Sicherung Ihres Kontos, um unbefugten Zugriff zu verhindern. Klicken Sie zu Ihrer Sicherheit auf Secure My αccount, um Ihre Informationen zu bestätigen.     Mein Konto sichern From owner-freebsd-drivers@freebsd.org Wed Sep 2 09:43:52 2020 Return-Path: Delivered-To: freebsd-drivers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4765B3D2810; Wed, 2 Sep 2020 09:43:52 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-oi1-x242.google.com (mail-oi1-x242.google.com [IPv6:2607:f8b0:4864:20::242]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BhJsz2JXQz4WG9; Wed, 2 Sep 2020 09:43:51 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-oi1-x242.google.com with SMTP id y6so3850458oie.5; Wed, 02 Sep 2020 02:43:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CXP6M2e9nMepjz/nQtdC62+uoPYITJvLzcX3ke47eEU=; b=q4bOXUFY5BuHszYdz2+L2d1fRT8eAWqcqfm/PApqQFvmAuVaIlUalVKPfPeLOKSF3X X2zc5Apy/FvaKOmqbzxD+HezruEZAVP3/twNVsZlTuWfDDcC62nURN0SIHlCtn8q6wt2 CF45RbQVkGk6qdJ7r/5oEsRwMlIX8b04tvYlCGqmBcHHUremsJU7UrutbEZiLCzFgWQb Y7PPQIzdN7pTZKQVkLUFTk+fETI3TTujeGFB+p8NvA2iy7zECF81C4Fu1zC9IS/X/+k1 ZhelUDRY34sjUHywhFeti4XQ9FeE6INXwWzjoyQSyb9SyPlg3TwIm/lq+h9vkHyB+nlg yeOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CXP6M2e9nMepjz/nQtdC62+uoPYITJvLzcX3ke47eEU=; b=qgnvtlG94h85qlwME8uBYQngwCn43U/lEcqrgdkQYXUdKNXxj1NQTqBOy0/Vi6hJGN ip2n9so6YdNe1RxvqfSnR7XyqxNptK4oX+6Hj654ThjLc9h1fcf0hpN0Xyyct2G1M4/1 1APqBgYV00WR9OHtw58nkEmdZ1ooyLqHCT+SdjvawtvbPe7LS20Gm85mnZvX/DbsDNjo YshsI8Z2kje9P8NoSHCJlkS+o+hPsgBdB+fqx6Lu+JYdJzhf8RX/bmXA3B72WczGoTv4 41KI+DbkdGd6l/O5XcNKwcuEOjL2846fATYpebn+Y9c5cCtOID8KU6Z5o3H8tviVTnB9 GnLg== X-Gm-Message-State: AOAM532ldixMLh/ylWN0M7/X7G9DaUVA8Wcnzncg4gQv/iYBr5UBX2jf 3cxXP6714j5eOBc5Z3Hbcn5DE/Dx5BE4AEFd5RoZwoij X-Google-Smtp-Source: ABdhPJw3myAWYhREsXYC6rq0gw45yDpAYTa6Z7XrMgjKiWhzrIAJLy4Z7u4LeX4uq7ZK2oA9ZjjnJeT3Wt74FuQcpnM= X-Received: by 2002:aca:49ca:: with SMTP id w193mr1552808oia.46.1599039829785; Wed, 02 Sep 2020 02:43:49 -0700 (PDT) MIME-Version: 1.0 References: <7cfc7c52-b548-19bd-343b-899aca45c654@selasky.org> In-Reply-To: From: Rajesh Kumar Date: Wed, 2 Sep 2020 15:13:39 +0530 Message-ID: Subject: Re: Network throughput not reaching line rate. Need clarification on iflib. To: Daniel Ebdrup Jensen Cc: freebsd-drivers@freebsd.org, FreeBSD Hackers X-Rspamd-Queue-Id: 4BhJsz2JXQz4WG9 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=q4bOXUFY; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2607:f8b0:4864:20::242 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-3.09 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.034]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.04)[-1.035]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::242:from]; NEURAL_HAM_SHORT(-0.02)[-0.018]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-drivers,freebsd-hackers]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Sep 2020 09:43:52 -0000 Hi Guys, Apologies for posting a question on an old thread. I started this thread early this year and it happened to pause working on this topic due to COVID lockdown. I recently resumed working on this and hence posting my questions on the same thread. To restate the issue, I am observing low network throughput numbers in the Receive path (on a 10G link) and trying to improve the performance numbers. As suggested in this thread, I tried using "iperf" instead of "iperf3" and also tried multiple threads. The behavior remains the same between iperf and iperf3. With a single thread I don't see any improvement in the performance numbers. But with multiple threads, I see slight improvement on the numbers, but still not touching the line rate for a 10G link. The base issue here is that the Receive buffers are not getting refilled fast enough for the Received packets. Hence I see a lot of retries (iperf3 output showing retries) and the performance drops. I could see the CPU utilization more than 90% when running iperf/iperf3 traffic. This could be the reason why Receive buffers are not filled faster. Receive path in my driver is pretty straight forward using the iflib framework. "rxd_available" interface returns the number of newly received packets. "rxd_refill" and "rxd_flush" interfaces fills the receive descriptors with buffers from iflib and update the buffer pointer registers. "rxd_pkt_get" interface traverses through all the descriptors for a packet return the received pkt info to iflib. I don't see any unwanted lock, loops and other evident bottlenecks in this path. So, what could be the reason for higher CPU utilization? and How can I debug this? Please let me know if any details are needed. Thanks, Rajesh. On Mon, Mar 2, 2020 at 4:37 AM Rajesh Kumar wrote: > Hi Guys, > > Thanks for your time in responding. > > As you people stated, the problem here doesn't seem to be single threaded > vs multi threaded. I am experience similar behavior with "iperf". It's > mostly to do with iflib and my driver(mostly my driver). It looks like the > receiver is slow in reading the packets than the sender sending the > packets. In this case, iflib drives the packet read through the exposed > interfaces (rxd_available, rxd_pkt_get etc.,). So, how to make the > receiver side read the packet faster with iflib? Is there anything that I > should take care in my driver in this regard? > > Thanks, > Rajesh. > > > > > On Sat, Feb 29, 2020 at 3:47 AM Daniel Ebdrup Jensen > wrote: > >> On Fri, Feb 28, 2020 at 7:39 PM Bruce A. Mah wrote: >> >> > [Resending with a From: address that hopefully works better.] >> > >> > If memory serves me right, Daniel Ebdrup Jensen wrote: >> > > Yes, iperf3 will default to single-threaded packet generation, et al. >> > which >> > > favours fast cores with frequency boosting facilities. >> > > You might want to use iperf2 as that's properly multi-threaded, or you >> > can >> > > use pkt-gen out of src/tools/tools/netmap/ or ports/net/pkt-gen. >> > >> > While it's true that iperf3 is single-threaded, it should be capable of >> > saturating a 10GE link with a single TCP connection, given proper >> > command-line arguments (in particular, specifying a sufficiently large >> > socket-buffer size with the -w option). >> > >> > But based on the symptom of packet loss, I'd say the single-threaded vs. >> > multi-threaded argument might not be relevant to the problem that the OP >> > has. >> > >> > Bruce. >> > >> > > On Fri, Feb 28, 2020 at 10:35 AM Hans Petter Selasky > > >> > > wrote: >> > > >> > >> On 2020-02-28 10:03, Rajesh Kumar wrote: >> > >>> Hi FreeBSD team, >> > >>> >> > >>> I am writing a network driver using iflib framework and using >> "iperf3" >> > >> tool >> > >>> for performance testing. >> > >>> >> > >> >> > >> Is there any difference with "iperf" tool and using multiple >> threads? I >> > >> think iperf3 is single threaded ??? >> > >> >> > >> --HPS >> > >> >> > >> _______________________________________________ >> > >> freebsd-hackers@freebsd.org mailing list >> > >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > >> To unsubscribe, send any mail to " >> > freebsd-hackers-unsubscribe@freebsd.org" >> > >> >> > > _______________________________________________ >> > > freebsd-hackers@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > > To unsubscribe, send any mail to " >> > freebsd-hackers-unsubscribe@freebsd.org" >> > > >> > >> >> Oh, I didn't mean to imply that that wasn't part of the issue - I'm sorry >> if I made it sound like that. >> I was just confirming what Hans was asking, and possibly using the excuse >> to mention some things in base/ports that I think are also pretty neat. :) >> >> Also no longer top-posting, which was rather ghastly of me. I apologise. >> _______________________________________________ >> freebsd-drivers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >> To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org >> " >> >