Date: Fri, 14 Feb 2020 09:25:18 -0800 From: Rudy <crapsh@monkeybrains.net> To: freebsd-net@freebsd.org Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 Message-ID: <428f3cdf-9035-90a7-14f8-f294c2131682@monkeybrains.net> In-Reply-To: <a4c98e33-3aae-f08b-4132-52350a33a56c@monkeybrains.net> References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net> <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net> <a4c98e33-3aae-f08b-4132-52350a33a56c@monkeybrains.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2/12/20 7:21 PM, Rudy wrote: > I'm having issues with a box that is acting as a BGP router for my network. 3 Chelsio cards, two T5 and one T6. It was working great until I turned up our first port on the T6. It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Looking better! I made some changes based on BSDRP which I hadn't known about -- I think ifqmaxlen was the tunable I overlooked. # https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local net.link.ifqmaxlen="16384" Also, I ran chelsio_affinity to bind queues to specific CPU cores. The script only supports a single t5 card, I am revising and will submit a patch that will do multiple t5 and t6 cards. I made both changes at once, and rebooted, so we'll never know which fixed it. ;) Right now, I have: #/boot/loader.conf # # https://wiki.freebsd.org/10gFreeBSD/Router hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.holdoff_timer_idx=3 # Before FreeBSD 13, threading bad on router: https://calomel.org/freebsd_network_tuning.html machdep.hyperthreading_allowed="0" hw.cxgbe.nrxq=16 hw.cxgbe.ntxq=16 hw.cxgbe.qsize_rxq=4096 hw.cxgbe.qsize_txq=4096 #hw.cxgbe.pause_settings="0" # https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local net.link.ifqmaxlen="16384" #/etc/sysctl.conf # FRR needs big buffers for OSPF kern.ipc.maxsockbuf=16777216 # Turn FEC off (doesn't work with Cogent) dev.cc.0.fec=0 dev.cc.1.fec=0 # Entropy not from LAN ports... slows them down. kern.random.harvest.mask=65551 net.inet.icmp.icmplim=400 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports net.inet.tcp.msl=7500 # close lost tcp connections in 7.5 seconds (default 30) net.inet.udp.blackhole=1 # drop any UDP packets to closed ports # hw.intr_storm_threshold=9000 net.inet.tcp.tso=0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?428f3cdf-9035-90a7-14f8-f294c2131682>