From owner-freebsd-drivers@freebsd.org Fri Feb 28 09:03:39 2020 Return-Path: Delivered-To: freebsd-drivers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 63DA025E28A; Fri, 28 Feb 2020 09:03:39 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48TNqt2j9hz4B5q; Fri, 28 Feb 2020 09:03:38 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-wm1-x32b.google.com with SMTP id m10so8858547wmc.0; Fri, 28 Feb 2020 01:03:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=JWM2CyapOK7wXHewObTe2Y3FnRDjW3i4+6ZxSke8TA4=; b=OtwIJ59v10yKJxMgOiKIcsxIfTDU/Rv8ILpnd6WSWCykOMQTuQY4oRTVieRM7EWsbl wyK4Yt6D3VWj7WYMCyDPzZrkMHw+wzZaRJzr8i73YcVmtg4Ga4quz7LK2caUwHdOzK/Y JYgFgf8yPf7C8CSIz4hfUY2dmyiTwD2pYRn8NUNKtEMV4awoYweVvv540tmxv4FJhI1a s1G1I6ThSPZqZJKJ45l7K0qa/iLfYIqwu2WnU5Pbbs1IiC7eVehIoHLFwk5rlNJoRmf0 qCSWu809KjdDADM89n5ZvnX7/8Vn7nizFhzDa4D8Y0+wnuQDY6PF4+W2FBPINXRDgxex H1Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=JWM2CyapOK7wXHewObTe2Y3FnRDjW3i4+6ZxSke8TA4=; b=Dr7nX62V2jNM8VEXHYQUUWgXECeOHZcHFRKaI3oyGulff6t26M2hN0aYgMC1kj6QY6 LMUzDCXz7+rtzQRV9D9zFwYq2fwXOd2Mk8mfGa9A8PkSfh++462FRf0Rzf9KBbgx8XUz lBztzTpkTVm7c6XrSHT13yhNTls4flqcSSAYX9IXX9KAqmiHhf+ZV5YI1+hXO2jJAV+k OqdO/Fy2ybGuNa32bje87tK1usOHMai/ncckT++e4qSDw/7O2xinYqL+p4U8nvqCRVDZ T2njHju4Ynf+yw0qJsu6jfUoimLhShALNGm7JTb6EY2hRmIBlvthvWJIHC0B0Xd6SoHe wbXw== X-Gm-Message-State: APjAAAV52qdw21Jq4wrq7pGUA0LPJ03wHfW/F4rfATUGQSmGTUnYgNfB br+4yR+oweOXJg7B9QOLSiRJFF420BuHmHBynp5biQ== X-Google-Smtp-Source: APXvYqwPHg4hDda0TTO9t1yolVAElyOSffHxMjveVN/JFvEZhQC2MZ3Ko7s7VX4o9+M+4jir4LozMkuhNlXf9bAOHaM= X-Received: by 2002:a1c:6a15:: with SMTP id f21mr3711316wmc.126.1582880616386; Fri, 28 Feb 2020 01:03:36 -0800 (PST) MIME-Version: 1.0 From: Rajesh Kumar Date: Fri, 28 Feb 2020 14:33:25 +0530 Message-ID: Subject: Network throughput not reaching line rate. Need clarification on iflib. To: freebsd-drivers@freebsd.org, FreeBSD Hackers X-Rspamd-Queue-Id: 48TNqt2j9hz4B5q X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=OtwIJ59v; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2a00:1450:4864:20::32b as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; IP_SCORE(0.00)[ip: (-9.15), ipnet: 2a00:1450::/32(-2.41), asn: 15169(-1.67), country: US(-0.05)]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[b.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Feb 2020 09:03:39 -0000 Hi FreeBSD team, I am writing a network driver using iflib framework and using "iperf3" tool for performance testing. 1) 10G link with 9000 MTU. Seeing line-rate numbers on both Tx and Rx path. 2) 10G link with 1500 MTU. Seeing line-rate numbers on Tx path, but low numbers on Rx path. I suspect the driver design here, but need some clarification on the iflib framework. My driver expects two memory buffers per receive descriptor. So I used "2 fls" per rxq (means isc_nrxq = 1, and isc_nfl = 2). For every descriptor, I use the memory buffers at same index from both the free-lists to refill my descriptors. Also, I return the fragments with appropriate index and flid from rxd_pkt_get. This approach works for me. But I am facing performance issue as stated above. So, Is this approach perfect? or any better way of doing? Also, I did a quick analysis of tcpdump on both iperf3 server (my test side) and client side(support side). iperf3 clients pumps traffic to iperf3 server. After a point, server sends an ACK with SACK option (SLE and SRE) to indicate missing frame to the client. And client is re-transmitting that segment as per protocol and the server receives it. But still I see the ACK sent again with SACK option. So client keeps on re-transmitting the packets and hence the performance drops. Any reason why server is not acknowledging the re-transmitted packet? If the driver Rx isn't proper, then I may not see the packet in tcpdump. But I see it. So any idea, what could be wrong here? Please let me know if any more details are needed. Thanks, Rajesh.