From owner-freebsd-net@freebsd.org Tue Sep 8 17:40:28 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 701553D0F72 for ; Tue, 8 Sep 2020 17:40:28 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-ot1-x343.google.com (mail-ot1-x343.google.com [IPv6:2607:f8b0:4864:20::343]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmC976x9wz45XL for ; Tue, 8 Sep 2020 17:40:27 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-ot1-x343.google.com with SMTP id o6so4302822ota.2 for ; Tue, 08 Sep 2020 10:40:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=YXSzTz7Ce5diga/y8Ybhi21oIFc42lA4IxXFuoJWNFc=; b=HBZe3WkNeCqG1JLbIubSb4vm5x2oParN47pjPVltplYGZhW6fMacrJ2XQl/3pQJAMZ V6KN6SDfA0lRSvcfijkZiBnPyRPXHXFwS69pzDs9H8Kf9qiLUjSZg5GB36dysTH/tMkm +QypODe72L7W2w63/tudC6zLX8jzHrW2N3Ia5sAcABqWVULyJD9sOyLfCMHgvdiKICFO a0x3KbOHAyOKOjT3sw83Lmv9/phBAhjMa36UoV0LjRrL55PcRRKuUgs++q8AkaLzD7+C PZt/4VyTMN41G0rWTur1gbmMyaZGo5E6Vm+wEYEfd8LBS/n8JIWIpXUA9iCqZsFqCeR6 83tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=YXSzTz7Ce5diga/y8Ybhi21oIFc42lA4IxXFuoJWNFc=; b=b7pCnE+4RrtnryTjceVoqF8rDcq1VKTWtApfBalyD1xd8r5p7g+jRCvu1igepSaScA uzq6vFnLx5yvoFo8yBITsIVakRf7dvpJwbqB2HFJP9tUP0Dwd+9Y5/q4KCBR90CQyCDR QU3YHj39f38YkHjPfaG9Xitruz2wcdx405gWeuLj98KNvxvAcROzRfMjhMtksWfj/pf8 pvZIz71x5expJDEB+odQgwk6rIBQeTs7MpeW38XxlWzthBTx++rZfl1QdC8mb+brzyMx 41Y5LOY9nHkkllASadpV+RWuF4BcFCXHEA1yYbRaurVJ0DXmegc9mvkIltcCg6umJo64 DkUw== X-Gm-Message-State: AOAM532JeU0Sx6S3VaC3/jTR3VtNUURKLWLwBB59MYThMMRgfv+/kKe2 jiKcjdfEXaWgpq0E1ovOAv9dN++LoLokbXDfmLW4LTUX X-Google-Smtp-Source: ABdhPJwNEzd73vSeFZbWwFwcP6Ok9yve8LBaGb+OHJfyQOk+gC5KvqgY3wRYydQltSluDC9GPtgmbAIQkDY7bTo5vYs= X-Received: by 2002:a05:6830:1e0a:: with SMTP id s10mr110628otr.95.1599586826715; Tue, 08 Sep 2020 10:40:26 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Rajesh Kumar Date: Tue, 8 Sep 2020 23:10:14 +0530 Message-ID: Subject: Re: 10G Network interface not reaching wire speed on Receive path with single thread iperf run To: freebsd-net@freebsd.org X-Rspamd-Queue-Id: 4BmC976x9wz45XL X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=HBZe3WkN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2607:f8b0:4864:20::343 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-3.28 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.97)[-0.972]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.99)[-0.993]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::343:from]; NEURAL_HAM_SHORT(-0.32)[-0.315]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-net]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 17:40:28 -0000 Hi Guys, Has anyone got a chance to look into this? Any clues? Thanks, Rajesh. On Mon, Sep 7, 2020 at 11:33 PM Rajesh Kumar wrote: > Hi Michael, > > We use Interrupts for Rx packets. Whereas we poll for the link status. > > Thanks, > Rajesh. > > On Mon, Sep 7, 2020 at 11:17 PM Michael Sierchio > wrote: > >> Are you using device polling or interrupts on the interface? >> >> On Mon, Sep 7, 2020 at 10:40 AM Rajesh Kumar wrote: >> >>> Hi FreeBSD team, >>> >>> I am writing a network driver using the iflib framework and using the >>> "iperf" tool for performance testing. >>> >>> 1) 10G link with 9000 MTU: Seeing line-rate numbers on both Tx and Rx >>> path. >>> 2) 10G link with 1500 MTU. Seeing line-rate numbers on Tx path, but low >>> numbers on Rx path. >>> >>> Base issue seems to be that Rx buffers are not getting re-filled fast >>> enough. With low thread count, I see higher CPU utilization and lower >>> throughput. But with a higher number of threads, I could see better >>> throughput and lesser cpu utilization. >>> >>> iperf with 1 Thread: CPU Utilization is 80% - 95% - Throughput is ~4Gbp= s. >>> iperf with 2 Thread: CPU Utilization is 80% - 90% - Throughput is >>> ~7.5Gbps. >>> iperf with 4 Thread: CPU Utilization is 40% - 75% - Throughput is ~9Gbp= s. >>> iperf with 8 Thread: CPU Utilization is 30% - 70% - Throughput is >>> ~9.2Gbps. >>> >>> So, something is causing CPU utilization higher with lesser thread, whi= ch >>> kind of leads to Rx buffers not getting filled up faster and hence the >>> performance numbers drop. From a code perspective, I couldn't see any >>> evident CPU consuming operations in the Rx path. It's plain hardware >>> descriptors reading/processing. >>> >>> Anyway we can debug to find what is causing more CPU utilization (with >>> lesser threads) in my case? Any clues on debugging this issue? >>> >>> Thanks, >>> Rajesh. >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >> >> >> -- >> >> "Well," Brahm=C4=81 said, "even after ten thousand explanations, a fool = is no >> wiser, but an intelligent person requires only two thousand five hundred= ." >> >> - The Mah=C4=81bh=C4=81rata >> >