Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 12 Feb 2020 19:21:31 -0800
From:      Rudy <crapsh@monkeybrains.net>
To:        net@FreeBSD.org
Subject:   Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
Message-ID:  <bb6c3997-c369-28c3-9d85-c9cca526e093@monkeybrains.net>
In-Reply-To: <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net>
References:  <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> <c921825a-3a9c-cc15-78e6-c7e3776ab12a@monkeybrains.net>

next in thread | previous in thread | raw e-mail | index | archive | help


I'm having issues with a box that is acting as a BGP router for my 
network.  3 Chelsio cards, two T5 and one T6.  It was working great 
until I turned up our first port on the T6.  It seems like traffic 
passing in from a T5 card and out the T6 causes a really high load (and 
high interrupts).

Attached graphs (will that work on a mailing list?) shows the load going 
up.

The work around at this point is that I block inbound routes (out bound 
traffic) on cc1.

I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't 
seem to lower Interrupts.


# netstat -rn | wc -l
  891149

CPU:  0.2% user,  0.0% nice,  0.6% system, 64.4% interrupt, 34.8% idle

Traffic (not that much, right?)

      Dev  RX bps    TX bps    RX PPS    TX PPS Error
      cc0       0         0         0         0         0
      cc1    2212 M       7 M     250 k       6 k 0 (100Gbps uplink, 
filtering inbound routes to keep TX low)
     cxl0     287 k    2015 M     353       244 k 0   (our network)
     cxl1     940 M    3115 M     176 k     360 k 0 (our network)
     cxl2     634 M    1014 M     103 k     128 k 0 (our network)
     cxl3       1 k      16 M       1         4 k       0
     cxl4       0         0         0         0         0
     cxl5       0         0         0         0         0
     cxl6    2343 M     791 M     275 k     137 k 0 (IX , part of lagg0)
     cxl7    1675 M     762 M     215 k     133 k 0 (IX , part of lagg0)
     ixl0     913 k      18 M       0         0 0
     ixl1       1 M      30 M       0         0         0
    lagg0    4019 M    1554 M     491 k     271 k       0
    lagg1       1 M      48 M       0         0         0
FreeBSD 12.1-STABLE orange                 976 Bytes/Packetavg
  1:42PM  up 13:25, 5 users, load averages: 9.38, 10.43, 9.827


# /boot/loader.conf
# ZFS!
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"

# Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card, 
40/100Gbps prots) devices
# built into kernel...
# if_cxgbe_load="YES"
# t5fw_cfg_load="YES"
# t6fw_cfg_load="YES"

# Before FreeBSD 13, threading bad on router: 
https://calomel.org/freebsd_network_tuning.html
machdep.hyperthreading_allowed="0"

# https://wiki.freebsd.org/10gFreeBSD/Router
hw.cxgbe.toecaps_allowed="0"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.fcoecaps_allowed="0"




# /etc/sysctl.conf

net.inet6.ip6.redirect=0
net.inet.ip.redirect=0

# FRR needs big buffers for OSPF
kern.ipc.maxsockbuf=16777216

# Turn FEC off (no link to Cogent otherwise)
dev.cc.0.fec=0
dev.cc.1.fec=0

# Entropy not from LAN ports... slows them down.
kern.random.harvest.mask=65551

# TODO: try this, 
https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html
# (Default was 2)
hw.cxgbe.nm_holdoff_tmr_idx=3

net.inet.icmp.icmplim=400
net.inet.icmp.maskrepl=0
net.inet.icmp.log_redirect=0
net.inet.icmp.drop_redirect=1
net.inet.tcp.drop_synfin=1
net.inet.tcp.blackhole=2          # drop any TCP packets to closed ports
net.inet.tcp.msl=7500             # close lost tcp connections in 7.5 
seconds (default 30)
net.inet.udp.blackhole=1          # drop any UDP packets to closed ports





# vmstat -i
interrupt                          total       rate
cpu0:timer                      54113958       1102
cpu1:timer                      55286089       1125
cpu2:timer                      53479994       1089
cpu3:timer                      55210439       1124
cpu4:timer                      25989916        529
cpu5:timer                      25908493        527
cpu6:timer                      55187512       1123
cpu7:timer                      54097494       1101
cpu8:timer                      54111676       1101
cpu9:timer                      54541125       1110
cpu10:timer                     55283621       1125
cpu11:timer                     55285466       1125
cpu12:timer                     55283322       1125
cpu13:timer                     55287200       1125
cpu14:timer                     55293560       1126
cpu15:timer                     55286002       1125
cpu16:timer                     55285700       1125
cpu17:timer                     55288330       1125
cpu18:timer                     55292101       1126
cpu19:timer                     55287826       1125
irq265: xhci0                      85784          2
irq266: ahci1                     720421         15
irq267: ixl0:aq                        1          0
irq268: ixl0:rxq0                8568015        174
irq269: ixl0:rxq1               10726322        218
irq270: ixl0:rxq2                7915180        161
irq271: ixl0:rxq3                8527106        174
irq272: ixl0:rxq4               13080418        266
irq273: ixl0:rxq5                7456434        152
irq274: ixl0:rxq6                8012988        163
irq275: ixl0:rxq7                8807168        179
irq276: ixl0:rxq8                9247761        188
irq277: ixl0:rxq9                8439429        172
irq278: ixl1:aq                        1          0
irq279: ixl1:rxq0                7750089        158
irq280: ixl1:rxq1                6604876        134
irq281: ixl1:rxq2                7180431        146
irq282: ixl1:rxq3                5990725        122
irq283: ixl1:rxq4                7385876        150
irq284: ixl1:rxq5               10252020        209
irq285: ixl1:rxq6                9155331        186
irq286: ixl1:rxq7                8384309        171
irq287: ixl1:rxq8                6642749        135
irq288: ixl1:rxq9               10019391        204
irq290: t6nex0:evt                     4          0
irq299: t6nex0:1a0             488060047       9935
irq300: t6nex0:1a1             379683827       7729
irq301: t6nex0:1a2             369471291       7521
irq302: t6nex0:1a3             393993838       8020
irq303: t6nex0:1a4             354430416       7215
irq304: t6nex0:1a5             599261919      12199
irq305: t6nex0:1a6             337249778       6865
irq306: t6nex0:1a7             511196035      10406
irq308: t5nex0:evt                     8          0
irq309: t5nex0:0a0               3117499         63
irq310: t5nex0:0a1               3123964         64
irq311: t5nex0:0a2               3163023         64
irq312: t5nex0:0a3               3163675         64
irq313: t5nex0:0a4               3036876         62
irq314: t5nex0:0a5               3207249         65
irq315: t5nex0:0a6               2997717         61
irq316: t5nex0:0a7               2861439         58
irq317: t5nex0:1a0             670962265      13658
irq318: t5nex0:1a1             666875099      13575
irq319: t5nex0:1a2             679409070      13830
irq320: t5nex0:1a3             700097218      14251
irq321: t5nex0:1a4             672477227      13689
irq322: t5nex0:1a5             674052457      13721
irq323: t5nex0:1a6             660159886      13438
irq324: t5nex0:1a7             797020409      16224
irq325: t5nex0:2a0             412131923       8389
irq326: t5nex0:2a1             561770787      11435
irq327: t5nex0:2a2             461510991       9394
irq328: t5nex0:2a3             420200582       8554
irq329: t5nex0:2a4             392331601       7986
irq330: t5nex0:2a5             410021075       8346
irq331: t5nex0:2a6             467853129       9524
irq332: t5nex0:2a7             423194899       8615
irq333: t5nex0:3a0                719985         15
irq334: t5nex0:3a1                613535         12
irq335: t5nex0:3a2                635461         13
irq336: t5nex0:3a3                613540         12
irq337: t5nex0:3a4                614360         13
irq338: t5nex0:3a5                613547         12
irq339: t5nex0:3a6                613539         12
irq340: t5nex0:3a7                613556         12
irq342: t5nex1:evt                     5          0
irq359: t5nex1:2a0             378696408       7709
irq360: t5nex1:2a1             383630463       7809
irq361: t5nex1:2a2             473624262       9641
irq362: t5nex1:2a3             391170574       7963
irq363: t5nex1:2a4             393111484       8002
irq364: t5nex1:2a5             428581285       8724
irq365: t5nex1:2a6             380367906       7743
irq366: t5nex1:2a7             382910534       7794
irq367: t5nex1:3a0             379035068       7716
irq368: t5nex1:3a1             381198007       7760
irq369: t5nex1:3a2             379586223       7727
irq370: t5nex1:3a3             391500901       7969
irq371: t5nex1:3a4             411892536       8384
irq372: t5nex1:3a5             395213950       8045
irq373: t5nex1:3a6             385156810       7840
irq374: t5nex1:3a7             380410814       7744
Total                        20060964625     408359

Thanks for any tips,
Rudy







Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bb6c3997-c369-28c3-9d85-c9cca526e093>