Date: Wed, 12 Feb 2020 19:21:31 -0800 From: Rudy <crapsh@monkeybrains.net> To: net@FreeBSD.org Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 Message-ID: <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net> In-Reply-To: <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net> References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I'm having issues with a box that is acting as a BGP router for my network. 3 Chelsio cards, two T5 and one T6. It was working great until I turned up our first port on the T6. It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Attached graphs (will that work on a mailing list?) shows the load going up. The work around at this point is that I block inbound routes (out bound traffic) on cc1. I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't seem to lower Interrupts. # netstat -rn | wc -l 891149 CPU: 0.2% user, 0.0% nice, 0.6% system, 64.4% interrupt, 34.8% idle Traffic (not that much, right?) Dev RX bps TX bps RX PPS TX PPS Error cc0 0 0 0 0 0 cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink, filtering inbound routes to keep TX low) cxl0 287 k 2015 M 353 244 k 0 (our network) cxl1 940 M 3115 M 176 k 360 k 0 (our network) cxl2 634 M 1014 M 103 k 128 k 0 (our network) cxl3 1 k 16 M 1 4 k 0 cxl4 0 0 0 0 0 cxl5 0 0 0 0 0 cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0) cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0) ixl0 913 k 18 M 0 0 0 ixl1 1 M 30 M 0 0 0 lagg0 4019 M 1554 M 491 k 271 k 0 lagg1 1 M 48 M 0 0 0 FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg 1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 # /boot/loader.conf # ZFS! kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" zfs_load="YES" # Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card, 40/100Gbps prots) devices # built into kernel... # if_cxgbe_load="YES" # t5fw_cfg_load="YES" # t6fw_cfg_load="YES" # Before FreeBSD 13, threading bad on router: https://calomel.org/freebsd_network_tuning.html machdep.hyperthreading_allowed="0" # https://wiki.freebsd.org/10gFreeBSD/Router hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" # /etc/sysctl.conf net.inet6.ip6.redirect=0 net.inet.ip.redirect=0 # FRR needs big buffers for OSPF kern.ipc.maxsockbuf=16777216 # Turn FEC off (no link to Cogent otherwise) dev.cc.0.fec=0 dev.cc.1.fec=0 # Entropy not from LAN ports... slows them down. kern.random.harvest.mask=65551 # TODO: try this, https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html # (Default was 2) hw.cxgbe.nm_holdoff_tmr_idx=3 net.inet.icmp.icmplim=400 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports net.inet.tcp.msl=7500 # close lost tcp connections in 7.5 seconds (default 30) net.inet.udp.blackhole=1 # drop any UDP packets to closed ports # vmstat -i interrupt total rate cpu0:timer 54113958 1102 cpu1:timer 55286089 1125 cpu2:timer 53479994 1089 cpu3:timer 55210439 1124 cpu4:timer 25989916 529 cpu5:timer 25908493 527 cpu6:timer 55187512 1123 cpu7:timer 54097494 1101 cpu8:timer 54111676 1101 cpu9:timer 54541125 1110 cpu10:timer 55283621 1125 cpu11:timer 55285466 1125 cpu12:timer 55283322 1125 cpu13:timer 55287200 1125 cpu14:timer 55293560 1126 cpu15:timer 55286002 1125 cpu16:timer 55285700 1125 cpu17:timer 55288330 1125 cpu18:timer 55292101 1126 cpu19:timer 55287826 1125 irq265: xhci0 85784 2 irq266: ahci1 720421 15 irq267: ixl0:aq 1 0 irq268: ixl0:rxq0 8568015 174 irq269: ixl0:rxq1 10726322 218 irq270: ixl0:rxq2 7915180 161 irq271: ixl0:rxq3 8527106 174 irq272: ixl0:rxq4 13080418 266 irq273: ixl0:rxq5 7456434 152 irq274: ixl0:rxq6 8012988 163 irq275: ixl0:rxq7 8807168 179 irq276: ixl0:rxq8 9247761 188 irq277: ixl0:rxq9 8439429 172 irq278: ixl1:aq 1 0 irq279: ixl1:rxq0 7750089 158 irq280: ixl1:rxq1 6604876 134 irq281: ixl1:rxq2 7180431 146 irq282: ixl1:rxq3 5990725 122 irq283: ixl1:rxq4 7385876 150 irq284: ixl1:rxq5 10252020 209 irq285: ixl1:rxq6 9155331 186 irq286: ixl1:rxq7 8384309 171 irq287: ixl1:rxq8 6642749 135 irq288: ixl1:rxq9 10019391 204 irq290: t6nex0:evt 4 0 irq299: t6nex0:1a0 488060047 9935 irq300: t6nex0:1a1 379683827 7729 irq301: t6nex0:1a2 369471291 7521 irq302: t6nex0:1a3 393993838 8020 irq303: t6nex0:1a4 354430416 7215 irq304: t6nex0:1a5 599261919 12199 irq305: t6nex0:1a6 337249778 6865 irq306: t6nex0:1a7 511196035 10406 irq308: t5nex0:evt 8 0 irq309: t5nex0:0a0 3117499 63 irq310: t5nex0:0a1 3123964 64 irq311: t5nex0:0a2 3163023 64 irq312: t5nex0:0a3 3163675 64 irq313: t5nex0:0a4 3036876 62 irq314: t5nex0:0a5 3207249 65 irq315: t5nex0:0a6 2997717 61 irq316: t5nex0:0a7 2861439 58 irq317: t5nex0:1a0 670962265 13658 irq318: t5nex0:1a1 666875099 13575 irq319: t5nex0:1a2 679409070 13830 irq320: t5nex0:1a3 700097218 14251 irq321: t5nex0:1a4 672477227 13689 irq322: t5nex0:1a5 674052457 13721 irq323: t5nex0:1a6 660159886 13438 irq324: t5nex0:1a7 797020409 16224 irq325: t5nex0:2a0 412131923 8389 irq326: t5nex0:2a1 561770787 11435 irq327: t5nex0:2a2 461510991 9394 irq328: t5nex0:2a3 420200582 8554 irq329: t5nex0:2a4 392331601 7986 irq330: t5nex0:2a5 410021075 8346 irq331: t5nex0:2a6 467853129 9524 irq332: t5nex0:2a7 423194899 8615 irq333: t5nex0:3a0 719985 15 irq334: t5nex0:3a1 613535 12 irq335: t5nex0:3a2 635461 13 irq336: t5nex0:3a3 613540 12 irq337: t5nex0:3a4 614360 13 irq338: t5nex0:3a5 613547 12 irq339: t5nex0:3a6 613539 12 irq340: t5nex0:3a7 613556 12 irq342: t5nex1:evt 5 0 irq359: t5nex1:2a0 378696408 7709 irq360: t5nex1:2a1 383630463 7809 irq361: t5nex1:2a2 473624262 9641 irq362: t5nex1:2a3 391170574 7963 irq363: t5nex1:2a4 393111484 8002 irq364: t5nex1:2a5 428581285 8724 irq365: t5nex1:2a6 380367906 7743 irq366: t5nex1:2a7 382910534 7794 irq367: t5nex1:3a0 379035068 7716 irq368: t5nex1:3a1 381198007 7760 irq369: t5nex1:3a2 379586223 7727 irq370: t5nex1:3a3 391500901 7969 irq371: t5nex1:3a4 411892536 8384 irq372: t5nex1:3a5 395213950 8045 irq373: t5nex1:3a6 385156810 7840 irq374: t5nex1:3a7 380410814 7744 Total 20060964625 408359 Thanks for any tips, Rudy
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bb6c3997-c369-28c3-9d85-c9cca526e093>