From owner-freebsd-net@freebsd.org Tue Feb 4 20:37:12 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8D680233A28 for ; Tue, 4 Feb 2020 20:37:12 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48BxMD08MKz4WqF for ; Tue, 4 Feb 2020 20:37:11 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pf1-x444.google.com with SMTP id i6so10087440pfc.1 for ; Tue, 04 Feb 2020 12:37:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:cc:references:from:autocrypt:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=QhwIqFkHKjVtB+e0cn3u0fIoVNeYaCYJmYc5kGiFph8=; b=eQHFjFzsjTCRWmYitHYmyMZ6asQ6jiocaR5C/uIkLea3Ta6oSFEl73K62/EmPHHbDQ fv2G+aEhibc4WKeAWiVQapXJCmyJdQ3+whFJtOMTpabRwQkHJWS94OawXvBmLiwHVZrV kuBjAaz1aMiZzxCVz+obXG+2/r3s2GWtDN9kZUEm3wLcZptZ92elmyEOlnvpczPbL4VH +gH71Y+zRSXqkpRplNRsVbp25FaodAVamWhQHFnarzUkmS+6VQ5eNmJtS9gDJ+i6xkhe EovNEfD6J+6PsZz4qqIb1y2qMFtQbFyM7GXugToVzv7pIDefy7Qhu4/P8dwSA06h6n80 3w3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:cc:references:from:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=QhwIqFkHKjVtB+e0cn3u0fIoVNeYaCYJmYc5kGiFph8=; b=f0e+ICfuTW0HntNX7tFuTH3k3NQ7xO5Si3v3HFz/9z8livGqF3u0tfnxTvW0PxbR2g rFC9M3jkF3kbF6F1Snw9Gi+dCt09ioyfiHw0kbY2upJ8FkGWD/9MNUIlBHPsrqoR/RHF ps1W5AbiAhTKCrBZHMBzYR8Fpqfp0SfNo0ES6ivEAzeRZXNBOa2xERVzrW0C3bx7MQ/u ZC4y1j+IQPP7feGdOaBwjiXn/xNQorzlk77Oh1MQAm7v7TX4dN08C6dYSEr8U+ktDr/q l3vyiTOAn6deGrO0AhFP+WkLpAAkP9MVk5H+5Objz5JlEcXoFOfzNkwIB6XBy4dz1/Ng NAyw== X-Gm-Message-State: APjAAAV60//FXMiiXplzQd5kU2auUfJVP278GpSzuBbDKley5rru8y6O SOhCpqVgyp26YwjGSBzKNqhAiEOn X-Google-Smtp-Source: APXvYqwnjHxReDxnUbWaxnkUbVE3zUMgfedU9ScPOqXrfJpTC7c8oTZKGGF0Lxygv5CdFYs1T/KTig== X-Received: by 2002:a63:5456:: with SMTP id e22mr32425272pgm.177.1580848630419; Tue, 04 Feb 2020 12:37:10 -0800 (PST) Received: from [10.192.166.0] (stargate.chelsio.com. [12.32.117.8]) by smtp.googlemail.com with ESMTPSA id e17sm26594993pfm.12.2020.02.04.12.37.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 04 Feb 2020 12:37:09 -0800 (PST) Sender: Navdeep Parhar Subject: Re: Chelsio NETMAP performance To: Slawa Olhovchenkov Cc: freebsd-net@freebsd.org References: <20200203201728.GC8028@zxy.spb.ru> <863de9e1-42cc-6f3a-5c1f-1bf737714c9f@FreeBSD.org> <20200203222321.GB8012@zxy.spb.ru> <6868f207-d054-3d45-b60d-eaf7115760c1@FreeBSD.org> <20200204162005.GC8012@zxy.spb.ru> From: Navdeep Parhar Autocrypt: addr=np@FreeBSD.org; prefer-encrypt=mutual; keydata= mQGiBEosaGcRBACOXnXquGEW53BjpMt2jViod/TUf1xgjMekcbDxqOODPeX7eYfrwJ8G6BCN OpGjBmWDu/JcNj4Z+gmTilJ6WLZQ7ecFZfEeO91pt6ys0cyWh0xfO+/mT83D7W81S/kqrJBk QbBIdV6LumevdErHo272r8RcMELC4Ru87eRtX3hmEwCgnnGNJMpQFUfYTt5XE7nY0yQoeV8D /0OcWmJbEZWxX9O7AuliCe3zd2Dw0B4LB9SZ2Dis7+gpVd3xVgYnt5wRE9kM+ThgrMA/wqr8 07qmEG6bcfUsfwwGN9YUtNF3xAN07cXTs026sCIFNZK816PrThBzCgkwR7pDpkMzGWIBr8Wi XXy0eB+JlQ6UV4PEiXuZ5ulzP0b1A/9CZm3wJfrNC0r1gMyrfVedg4zwKU997bmPLGcYs+rW XDTI9CvMseOUYn4CoDZQCp/9zxuHK+VU7Y/w0c/hVE5ERACSn4SjN2unEDstK9njZBMHEPVk Ae/YvSG5cmc97SHlVE+eu/bbLKcvFb6rRLPOaVFQJMJA2VJEGWtYhvP7ZbQfTmF2ZGVlcCBQ YXJoYXIgPG5wQEZyZWVCU0Qub3JnPohgBBMRAgAgBQJKLGhnAhsDBgsJCAcDAgQVAggDBBYC AwECHgECF4AACgkQyrIrk6yriBL0MQCfUJOiS2PbJFDeiav1ylcXXwfpggAAoJRoS7GDENGy M4BzjJ4b0ptZqTLRuQINBEosaGcQCACFCWs47SL4DQA6bNDlVJu4w8wLf8uVOyatuGmdXX8Y /OTVQJgA3vS+ODNVJCxhKVlvhcn7bhBdGdWKS9K+lr8+eEvr4hf2bQpesoHC+uFgKyILkCBN L8raixbhysyq0pfZWWDJMyn+G42BG1yJJi+bykygdpYnbIVA8dYHmBibI8mkPKOHSohjXT1S RfGGn+l1w54OO4NlJhCXMkjTA/Z9Bt4XeaiR85uJi0UUfV8FGZHhgSvT+/P1xIvz+nytuehS P/QLXl13CtAG/nKVkAcZnsT/3NrJ4Z2r45k+c50Wrf210scAaBogrrV5eIHfNGgOANApN8+8 vj+aXO4pXRuXAAMFB/44ea8rd+P5N3OMrfuM8i91Qe1bJ+BIoroKPOr8jvCry0h3QpdfLKUN IgaqbS3JZeBJ8HHnWSGCF+o6H5gzRe1hvylPEclLPDCuPe7T746h9Mzejf2hNDJvOg+BuweD ZW4KhovVbdS+syJEvpGF4bO8qgHT2CKgruXSHbFetdQWbkM0rfMmTuo0GcR2BEVrPb/SPFv6 4ZZyAZzmnGO4vT1bzClnTzJixrDpH74M3vSEYegMB4KdbLYBi8Jx4QUKgVEhJHjJubKWX4et yU/uuehOC3xYrmr1UXvsom3U8r36Dvdo77Yr3dgDVXa7bolNx0TIhdWxZI+R4z9E75QY+/wg iEkEGBECAAkFAkosaGcCGwwACgkQyrIrk6yriBI+JQCfUxgyqGtzZvLh5Al7gsTmRc11PLwA niD3NfWGRcO2+9uxSSQqRH1ywC4n Message-ID: <3a8dfebd-aa26-84ad-a03a-0271b61a89a3@FreeBSD.org> Date: Tue, 4 Feb 2020 12:37:08 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: <20200204162005.GC8012@zxy.spb.ru> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 48BxMD08MKz4WqF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=eQHFjFzs; dmarc=none; spf=pass (mx1.freebsd.org: domain of nparhar@gmail.com designates 2607:f8b0:4864:20::444 as permitted sender) smtp.mailfrom=nparhar@gmail.com X-Spamd-Result: default: False [-3.06 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-0.86)[ip: (-0.52), ipnet: 2607:f8b0::/32(-1.99), asn: 15169(-1.75), country: US(-0.05)]; FORGED_SENDER(0.30)[np@FreeBSD.org,nparhar@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[np@FreeBSD.org,nparhar@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Feb 2020 20:37:12 -0000 On 2/4/20 8:20 AM, Slawa Olhovchenkov wrote: > On Mon, Feb 03, 2020 at 02:39:03PM -0800, Navdeep Parhar wrote: >=20 >> On 2/3/20 2:23 PM, Slawa Olhovchenkov wrote: >>> On Mon, Feb 03, 2020 at 01:39:52PM -0800, Navdeep Parhar wrote: >>> >>>> On 2/3/20 12:17 PM, Slawa Olhovchenkov wrote: >>>>> I am try to use Chelsio T540-CR in netmap mode and see poor (compar= ed >>>>> to Intel 82599ES) performance. >>>> >>>> What approximate FreeBSD version is this? >>> >>> 12.1-STABLE >>> >>>>> >>>>> Same application ac receive only about 8.9Mpss, compared to 12.5Mpp= s >>>>> at Intel. >>>>> >>>>> pmc profile show mostly time spend in: >>>>> >>>>> 49.76% [17802] service_nm_rxq @ /boot/kernel/if_cxgbe.ko >>>>> 100.0% [17802] t4_vi_intr >>>>> 100.0% [17802] ithread_loop @ /boot/kernel/kernel >>>>> 100.0% [17802] fork_exit >>>>> >>>>> >>>>> to be exact at line >>>>> >>>>> while ((d->rsp.u.type_gen & F_RSPD_GEN) =3D=3D nm_rxq->iq_g= en) { >>>>> >>>>> Is this maximum limit for this vendor? >>>> >>>> No, a T540 should be able to sink full 10Gbps (14.88Mpps) on a singl= e rx >>>> queue. Try adding this to your loader.conf: >>>> >>>> hw.cxgbe.toecaps_allowed=3D"0" >>>> >>>> Then try simple netmap "pkt-gen -f rx" instead of any custom app and= see >>>> how many pps it's able to sink. >>> >>> Thanks! `hw.cxgbe.toecaps_allowed=3D"0"` allow recive 14Mpps for may >>> application too! >>> >>> Now I am got only 10% less performance compared to Intel, as I see by= >>> higher Chelsio interrupt cpu time (top show about 30% for every >>> interrupt handler). Is this normal? Is this posible to optimize? >> >> Try changing the interrupt holdoff timer for the netmap rx queues. >> >> This shows the list of timers available (in microseconds): >> # sysctl dev.t5nex.0.holdoff_timers >> >> nm_holdoff_tmr_idx is a 0-based index into the list above. So if the >> tmr idx is 0 you are using the 0th (first) value from the list of >> timers. Try increasing nm_holdoff_tmr_idx and see if that brings down= >> the interrupt rate under control. >> >> # sysctl hw.cxgbe.nm_holdoff_tmr_idx=3D3/4/5 >=20 > OK, interrupt rate go down, but interrupt time about same. > (interrupt rate for intel card about 0, compared to 25% chelsio). I think iflib runs a lot of stuff in taskqueues rather than the driver ithread so the CPU accounting may vary. Use dtrace to see if netmap_rx_irq is being called by an ithread or a taskqueue to figure out what driver does what. Are you also transmitting a lot out of this node or is it mostly Rx? There's no need to worry about Tx updates (and the interrupts they might generate) if this is an Rx-mostly workload. > Most time spent in service_nm_rxq(), in while() check. > Is this posible to do some prefetch? > Trivial `__builtin_prefetch(64+(char*)d);` in body of loop don't > change anything. >=20 > Is this posible to do batch prefetch before cycle? prefetches are not possible here. That while condition is waiting for the ownership bit of the rx descriptor to flip, indicating there is work for the driver to do. Regards, Navdeep