Date: Wed, 12 Feb 2020 23:05:18 -0800 From: Rudy <crapsh@monkeybrains.net> To: net@FreeBSD.org Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 Message-ID: <d36bcd0c-71c5-e65f-b262-5b0a420a40d9@monkeybrains.net> In-Reply-To: <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net> References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net> <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Load seems to be increasing even though outbound traffic is blocked. I'm seeing 'Discards In' rise up to about 4k/second when doing 4Gbps inbound traffic. I see the tx_pause counter incrementing... dev.cc.1.stats.tx_pause: 2911586 (~ 20/second increase) Any tips would be great, I'm stumped. 20 cores seems like plenty to do much higher PPS based on what I see on the freebsd-net list. Thanks for any pointers, Rudy On 2/12/20 7:21 PM, Rudy wrote: > > > I'm having issues with a box that is acting as a BGP router for my > network. 3 Chelsio cards, two T5 and one T6. It was working great > until I turned up our first port on the T6. It seems like traffic > passing in from a T5 card and out the T6 causes a really high load > (and high interrupts). > > Attached graphs (will that work on a mailing list?) shows the load > going up. > > The work around at this point is that I block inbound routes (out > bound traffic) on cc1. > > I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't > seem to lower Interrupts. > > > # netstat -rn | wc -l > 891149 > > CPU: 0.2% user, 0.0% nice, 0.6% system, 64.4% interrupt, 34.8% idle > > Traffic (not that much, right?) > > Dev RX bps TX bps RX PPS TX PPS Error > cc0 0 0 0 0 0 > cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink, > filtering inbound routes to keep TX low) > cxl0 287 k 2015 M 353 244 k 0 (our network) > cxl1 940 M 3115 M 176 k 360 k 0 (our network) > cxl2 634 M 1014 M 103 k 128 k 0 (our network) > cxl3 1 k 16 M 1 4 k 0 > cxl4 0 0 0 0 0 > cxl5 0 0 0 0 0 > cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0) > cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0) > ixl0 913 k 18 M 0 0 0 > ixl1 1 M 30 M 0 0 0 > lagg0 4019 M 1554 M 491 k 271 k 0 > lagg1 1 M 48 M 0 0 0 > FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg > 1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 > > > # /boot/loader.conf > # ZFS! > kern.geom.label.disk_ident.enable="0" > kern.geom.label.gptid.enable="0" > zfs_load="YES" > > # Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card, > 40/100Gbps prots) devices > # built into kernel... > # if_cxgbe_load="YES" > # t5fw_cfg_load="YES" > # t6fw_cfg_load="YES" > > # Before FreeBSD 13, threading bad on router: > https://calomel.org/freebsd_network_tuning.html > machdep.hyperthreading_allowed="0" > > # https://wiki.freebsd.org/10gFreeBSD/Router > hw.cxgbe.toecaps_allowed="0" > hw.cxgbe.rdmacaps_allowed="0" > hw.cxgbe.iscsicaps_allowed="0" > hw.cxgbe.fcoecaps_allowed="0" > > > > > # /etc/sysctl.conf > > net.inet6.ip6.redirect=0 > net.inet.ip.redirect=0 > > # FRR needs big buffers for OSPF > kern.ipc.maxsockbuf=16777216 > > # Turn FEC off (no link to Cogent otherwise) > dev.cc.0.fec=0 > dev.cc.1.fec=0 > > # Entropy not from LAN ports... slows them down. > kern.random.harvest.mask=65551 > > # TODO: try this, > https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html > # (Default was 2) > hw.cxgbe.nm_holdoff_tmr_idx=3 > > net.inet.icmp.icmplim=400 > net.inet.icmp.maskrepl=0 > net.inet.icmp.log_redirect=0 > net.inet.icmp.drop_redirect=1 > net.inet.tcp.drop_synfin=1 > net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports > net.inet.tcp.msl=7500 # close lost tcp connections in 7.5 > seconds (default 30) > net.inet.udp.blackhole=1 # drop any UDP packets to closed ports > > > > > > # vmstat -i > interrupt total rate > cpu0:timer 54113958 1102 > cpu1:timer 55286089 1125 > cpu2:timer 53479994 1089 > cpu3:timer 55210439 1124 > cpu4:timer 25989916 529 > cpu5:timer 25908493 527 > cpu6:timer 55187512 1123 > cpu7:timer 54097494 1101 > cpu8:timer 54111676 1101 > cpu9:timer 54541125 1110 > cpu10:timer 55283621 1125 > cpu11:timer 55285466 1125 > cpu12:timer 55283322 1125 > cpu13:timer 55287200 1125 > cpu14:timer 55293560 1126 > cpu15:timer 55286002 1125 > cpu16:timer 55285700 1125 > cpu17:timer 55288330 1125 > cpu18:timer 55292101 1126 > cpu19:timer 55287826 1125 > irq265: xhci0 85784 2 > irq266: ahci1 720421 15 > irq267: ixl0:aq 1 0 > irq268: ixl0:rxq0 8568015 174 > irq269: ixl0:rxq1 10726322 218 > irq270: ixl0:rxq2 7915180 161 > irq271: ixl0:rxq3 8527106 174 > irq272: ixl0:rxq4 13080418 266 > irq273: ixl0:rxq5 7456434 152 > irq274: ixl0:rxq6 8012988 163 > irq275: ixl0:rxq7 8807168 179 > irq276: ixl0:rxq8 9247761 188 > irq277: ixl0:rxq9 8439429 172 > irq278: ixl1:aq 1 0 > irq279: ixl1:rxq0 7750089 158 > irq280: ixl1:rxq1 6604876 134 > irq281: ixl1:rxq2 7180431 146 > irq282: ixl1:rxq3 5990725 122 > irq283: ixl1:rxq4 7385876 150 > irq284: ixl1:rxq5 10252020 209 > irq285: ixl1:rxq6 9155331 186 > irq286: ixl1:rxq7 8384309 171 > irq287: ixl1:rxq8 6642749 135 > irq288: ixl1:rxq9 10019391 204 > irq290: t6nex0:evt 4 0 > irq299: t6nex0:1a0 488060047 9935 > irq300: t6nex0:1a1 379683827 7729 > irq301: t6nex0:1a2 369471291 7521 > irq302: t6nex0:1a3 393993838 8020 > irq303: t6nex0:1a4 354430416 7215 > irq304: t6nex0:1a5 599261919 12199 > irq305: t6nex0:1a6 337249778 6865 > irq306: t6nex0:1a7 511196035 10406 > irq308: t5nex0:evt 8 0 > irq309: t5nex0:0a0 3117499 63 > irq310: t5nex0:0a1 3123964 64 > irq311: t5nex0:0a2 3163023 64 > irq312: t5nex0:0a3 3163675 64 > irq313: t5nex0:0a4 3036876 62 > irq314: t5nex0:0a5 3207249 65 > irq315: t5nex0:0a6 2997717 61 > irq316: t5nex0:0a7 2861439 58 > irq317: t5nex0:1a0 670962265 13658 > irq318: t5nex0:1a1 666875099 13575 > irq319: t5nex0:1a2 679409070 13830 > irq320: t5nex0:1a3 700097218 14251 > irq321: t5nex0:1a4 672477227 13689 > irq322: t5nex0:1a5 674052457 13721 > irq323: t5nex0:1a6 660159886 13438 > irq324: t5nex0:1a7 797020409 16224 > irq325: t5nex0:2a0 412131923 8389 > irq326: t5nex0:2a1 561770787 11435 > irq327: t5nex0:2a2 461510991 9394 > irq328: t5nex0:2a3 420200582 8554 > irq329: t5nex0:2a4 392331601 7986 > irq330: t5nex0:2a5 410021075 8346 > irq331: t5nex0:2a6 467853129 9524 > irq332: t5nex0:2a7 423194899 8615 > irq333: t5nex0:3a0 719985 15 > irq334: t5nex0:3a1 613535 12 > irq335: t5nex0:3a2 635461 13 > irq336: t5nex0:3a3 613540 12 > irq337: t5nex0:3a4 614360 13 > irq338: t5nex0:3a5 613547 12 > irq339: t5nex0:3a6 613539 12 > irq340: t5nex0:3a7 613556 12 > irq342: t5nex1:evt 5 0 > irq359: t5nex1:2a0 378696408 7709 > irq360: t5nex1:2a1 383630463 7809 > irq361: t5nex1:2a2 473624262 9641 > irq362: t5nex1:2a3 391170574 7963 > irq363: t5nex1:2a4 393111484 8002 > irq364: t5nex1:2a5 428581285 8724 > irq365: t5nex1:2a6 380367906 7743 > irq366: t5nex1:2a7 382910534 7794 > irq367: t5nex1:3a0 379035068 7716 > irq368: t5nex1:3a1 381198007 7760 > irq369: t5nex1:3a2 379586223 7727 > irq370: t5nex1:3a3 391500901 7969 > irq371: t5nex1:3a4 411892536 8384 > irq372: t5nex1:3a5 395213950 8045 > irq373: t5nex1:3a6 385156810 7840 > irq374: t5nex1:3a7 380410814 7744 > Total 20060964625 408359 > > Thanks for any tips, > Rudy > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d36bcd0c-71c5-e65f-b262-5b0a420a40d9>