From owner-freebsd-net@freebsd.org Fri Feb 14 17:25:22 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 13FFD23B8FB for ; Fri, 14 Feb 2020 17:25:22 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mail.monkeybrains.net (mail.monkeybrains.net [208.69.40.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.monkeybrains.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48K0dD6nBNz3Q4C for ; Fri, 14 Feb 2020 17:25:20 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from [10.2.86.111] (public.monkeybrains.net [208.69.41.107] (may be forged)) (authenticated bits=0) by mail.monkeybrains.net (8.15.2/8.15.2) with ESMTPSA id 01EHPI2p017079 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 14 Feb 2020 09:25:19 -0800 (PST) (envelope-from crapsh@monkeybrains.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=monkeybrains.net; s=dkim; t=1581701119; bh=Hc0wBl7vWY+WcKtBtZmybHq6/CSEPc+3wpxE3TMsZS8=; h=Subject:To:References:From:Date:In-Reply-To; b=EWEprCVgBtyYqeSlMYz84NvUnQIoSGw9np9g8I3PWUF03B2w85oxD0EhHZpOdX/cT GApNoMcU1PgQFO+T1bRKlh0L84KyThKAZZbXUIyBH92/tLYH6r6ThTDjwvVQs69wmG WS7/AC+e83Yt8yH2KfNOGMfL/QqOzvuzlVwQkKgM= X-Authentication-Warning: mail.monkeybrains.net: Host public.monkeybrains.net [208.69.41.107] (may be forged) claimed to be [10.2.86.111] Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 To: freebsd-net@freebsd.org References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> From: Rudy Message-ID: <428f3cdf-9035-90a7-14f8-f294c2131682@monkeybrains.net> Date: Fri, 14 Feb 2020 09:25:18 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: clamav-milter 0.101.4 at mail.monkeybrains.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 48K0dD6nBNz3Q4C X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=monkeybrains.net header.s=dkim header.b=EWEprCVg; dmarc=pass (policy=none) header.from=monkeybrains.net; spf=pass (mx1.freebsd.org: domain of crapsh@monkeybrains.net designates 208.69.40.19 as permitted sender) smtp.mailfrom=crapsh@monkeybrains.net X-Spamd-Result: default: False [-5.77 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ptr]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[monkeybrains.net:+]; DMARC_POLICY_ALLOW(0.00)[monkeybrains.net,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-3.68)[ip: (-9.66), ipnet: 208.69.40.0/22(-4.79), asn: 32329(-3.90), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_LOW(-0.10)[19.40.69.208.list.dnswl.org : 127.0.5.1]; ASN(0.00)[asn:32329, ipnet:208.69.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[monkeybrains.net.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(0.00)[monkeybrains.net:s=dkim]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; URIBL_PBL(0.01)[calomel.org]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2020 17:25:22 -0000 On 2/12/20 7:21 PM, Rudy wrote: > I'm having issues with a box that is acting as a BGP router for my network.  3 Chelsio cards, two T5 and one T6.  It was working great until I turned up our first port on the T6.  It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Looking better!  I made some changes based on BSDRP which I hadn't known about -- I think ifqmaxlen was the tunable I overlooked. # https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local net.link.ifqmaxlen="16384" Also, I ran chelsio_affinity to bind queues to specific CPU cores.  The script only supports a single t5 card, I am revising and will submit a patch that will do multiple t5 and t6 cards. I made both changes at once, and rebooted, so we'll never know which fixed it.  ;) Right now, I have: #/boot/loader.conf # # https://wiki.freebsd.org/10gFreeBSD/Router hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.holdoff_timer_idx=3 # Before FreeBSD 13, threading bad on router: https://calomel.org/freebsd_network_tuning.html machdep.hyperthreading_allowed="0" hw.cxgbe.nrxq=16 hw.cxgbe.ntxq=16 hw.cxgbe.qsize_rxq=4096 hw.cxgbe.qsize_txq=4096 #hw.cxgbe.pause_settings="0" # https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local net.link.ifqmaxlen="16384" #/etc/sysctl.conf # FRR needs big buffers for OSPF kern.ipc.maxsockbuf=16777216 # Turn FEC off (doesn't work with Cogent) dev.cc.0.fec=0 dev.cc.1.fec=0 # Entropy not from LAN ports... slows them down. kern.random.harvest.mask=65551 net.inet.icmp.icmplim=400 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.blackhole=2          # drop any TCP packets to closed ports net.inet.tcp.msl=7500             # close lost tcp connections in 7.5 seconds (default 30) net.inet.udp.blackhole=1          # drop any UDP packets to closed ports # hw.intr_storm_threshold=9000 net.inet.tcp.tso=0