From owner-freebsd-net@freebsd.org Fri Feb 14 06:49:16 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 87EBE2425B0 for ; Fri, 14 Feb 2020 06:49:16 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48JkWJ1VGgz48nS for ; Fri, 14 Feb 2020 06:49:16 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: by mailman.nyi.freebsd.org (Postfix) id 316452425AF; Fri, 14 Feb 2020 06:49:16 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2FE622425AE for ; Fri, 14 Feb 2020 06:49:16 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from sfo.monkeybrains.net (sfo.monkeybrains.net [208.69.40.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.monkeybrains.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48JkWH2Q0Kz48nN for ; Fri, 14 Feb 2020 06:49:14 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from Rudys-iMac.lan (148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged)) (authenticated bits=0) by sfo.monkeybrains.net (8.15.2/8.15.2) with ESMTPSA id 01E6nDn4017740 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 13 Feb 2020 22:49:13 -0800 (PST) (envelope-from crapsh@monkeybrains.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=monkeybrains.net; s=dkim; t=1581662953; bh=t7ZOL0oz4TjMjt7qLQ4pa4Q/7Oq5CZwqNmDcHuRm37A=; h=Subject:From:To:References:Date:In-Reply-To; b=SlZiW6kFlbSZza0Kj0lZ0bU9asyro6C2PWyiu0VPLZdKeIuua3P1nMHCXqj1bOf3Y V9WxGQ4a86eTYgI7EB5no+y83Km5ifD5CdEBF6+fNFVN2sjKuWQ7wC64imNrKbk7h9 yfHcA9P4oATMJQBypCuag0eVUVtiDi3b7J9Iw21s= X-Authentication-Warning: mail.monkeybrains.net: Host 148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged) claimed to be Rudys-iMac.lan Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 From: Rudy To: net@FreeBSD.org References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> Message-ID: Date: Thu, 13 Feb 2020 22:49:13 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: clamav-milter 0.101.4 at mail.monkeybrains.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 48JkWH2Q0Kz48nN X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=monkeybrains.net header.s=dkim header.b=SlZiW6kF; dmarc=pass (policy=none) header.from=monkeybrains.net; spf=pass (mx1.freebsd.org: domain of crapsh@monkeybrains.net designates 208.69.40.9 as permitted sender) smtp.mailfrom=crapsh@monkeybrains.net X-Spamd-Result: default: False [-6.65 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[monkeybrains.net:s=dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[monkeybrains.net.dwl.dnswl.org : 127.0.5.0]; R_SPF_ALLOW(-0.20)[+ptr]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[monkeybrains.net:+]; DMARC_POLICY_ALLOW(-0.50)[monkeybrains.net,none]; RCVD_IN_DNSWL_NONE(0.00)[9.40.69.208.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-3.65)[ip: (-9.48), ipnet: 208.69.40.0/22(-4.81), asn: 32329(-3.91), country: US(-0.05)]; ASN(0.00)[asn:32329, ipnet:208.69.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2020 06:49:16 -0000 On 2/12/20 7:21 PM, Rudy wrote: > > > I'm having issues with a box that is acting as a BGP router for my > network.  3 Chelsio cards, two T5 and one T6.  It was working great > until I turned up our first port on the T6.  It seems like traffic > passing in from a T5 card and out the T6 causes a really high load > (and high interrupts). Update, I have found the chelsio_affinity script and run that, doesn't seem to help.  I did notice the T6 card is on numa-domain 0 and the T5 cards on on numa-domain 1.  Does that matter? Here is the relevant part of dmesg: t6nex0: mem 0xc5300000-0xc537ffff,0xc4000000-0xc4ffffff,0xc5984000-0xc5985fff irq 48 at device 0.4 numa-domain 0 on pci12 cc0: numa-domain 0 on t6nex0 cc1: numa-domain 0 on t6nex0 t6nex0: PCIe gen3 x16, 2 ports, 34 MSI-X interrupts, 66 eq, 33 iq t5nex0: mem 0xed300000-0xed37ffff,0xec000000-0xecffffff,0xed984000-0xed985fff irq 88 at device 0.4 numa-domain 1 on pci18 cxl0: numa-domain 1 on t5nex0 cxl1: numa-domain 1 on t5nex0 cxl2: numa-domain 1 on t5nex0 cxl3: numa-domain 1 on t5nex0 t5nex0: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq t5nex1: mem 0xfb300000-0xfb37ffff,0xfa000000-0xfaffffff,0xfb984000-0xfb985fff irq 98 at device 0.4 numa-domain 1 on pci20 cxl4: numa-domain 1 on t5nex1 cxl5: numa-domain 1 on t5nex1 cxl6: numa-domain 1 on t5nex1 cxl7: numa-domain 1 on t5nex1 t5nex1: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq Box does have 20 cores and hyper-threading is disabled. sysctl hw.ncpu hw.ncpu: 20 I've added updated /boot/loader.conf to the following: hw.cxgbe.nrxq=16 hw.cxgbe.ntxq=16 hw.cxgbe.qsize_rxq=4096 hw.cxgbe.qsize_txq=4096 hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.holdoff_timer_idx=3 machdep.hyperthreading_allowed="0" Box works fine unless I turn on TX on the cc1 device (I am filtering routes learned from that interface via BGP). Rudy