From owner-freebsd-net@freebsd.org Thu Feb 13 03:21:34 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B3C4624BB58 for ; Thu, 13 Feb 2020 03:21:34 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48J1y64PFLz4PSW for ; Thu, 13 Feb 2020 03:21:34 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: by mailman.nyi.freebsd.org (Postfix) id 96ADE24BB57; Thu, 13 Feb 2020 03:21:34 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9642724BB56 for ; Thu, 13 Feb 2020 03:21:34 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from sfo.monkeybrains.net (sfo.monkeybrains.net [208.69.40.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.monkeybrains.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48J1y52995z4PSS for ; Thu, 13 Feb 2020 03:21:32 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from Rudys-iMac.lan (148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged)) (authenticated bits=0) by sfo.monkeybrains.net (8.15.2/8.15.2) with ESMTPSA id 01D3LVDc038035 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 12 Feb 2020 19:21:31 -0800 (PST) (envelope-from crapsh@monkeybrains.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=monkeybrains.net; s=dkim; t=1581564091; bh=6nW/K4gCMp7YToOkrnjwmKqhfckGoZnwJHfZogsq1SU=; h=Subject:From:To:References:Date:In-Reply-To; b=SD+vDJSPiQUZ8uyL3aJtmInwQ0wAnbsDP+PohMZQosPkeK5pXL9W909pECuZoV02Y XKJPoqPJXJi1TK7OwMpuGg+prx0B1JrKCRQ3oj9bst2LIMq1xLbjx8AP7la3t7Ojzb xdDm4BgvmjtQL0ULObT7wykmgIjFy431H1WX8lP8= X-Authentication-Warning: mail.monkeybrains.net: Host 148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged) claimed to be Rudys-iMac.lan Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 From: Rudy To: net@FreeBSD.org References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> Message-ID: Date: Wed, 12 Feb 2020 19:21:31 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Virus-Scanned: clamav-milter 0.101.4 at mail.monkeybrains.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 48J1y52995z4PSS X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=monkeybrains.net header.s=dkim header.b=SD+vDJSP; dmarc=pass (policy=none) header.from=monkeybrains.net; spf=pass (mx1.freebsd.org: domain of crapsh@monkeybrains.net designates 208.69.40.9 as permitted sender) smtp.mailfrom=crapsh@monkeybrains.net X-Spamd-Result: default: False [-2.76 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ptr]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[9]; DKIM_TRACE(0.00)[monkeybrains.net:+]; DMARC_POLICY_ALLOW(0.00)[monkeybrains.net,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; IP_SCORE(-1.77)[ipnet: 208.69.40.0/22(-4.86), asn: 32329(-3.92), country: US(-0.05)]; ASN(0.00)[asn:32329, ipnet:208.69.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[monkeybrains.net.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(0.00)[monkeybrains.net:s=dkim]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain,multipart/related]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; URIBL_PBL(0.01)[calomel.org]; RCVD_IN_DNSWL_NONE(0.00)[9.40.69.208.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2020 03:21:34 -0000 I'm having issues with a box that is acting as a BGP router for my network.  3 Chelsio cards, two T5 and one T6.  It was working great until I turned up our first port on the T6.  It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Attached graphs (will that work on a mailing list?) shows the load going up. The work around at this point is that I block inbound routes (out bound traffic) on cc1. I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't seem to lower Interrupts. # netstat -rn | wc -l  891149 CPU:  0.2% user,  0.0% nice,  0.6% system, 64.4% interrupt, 34.8% idle Traffic (not that much, right?)      Dev  RX bps    TX bps    RX PPS    TX PPS Error      cc0       0         0         0         0         0      cc1    2212 M       7 M     250 k       6 k 0 (100Gbps uplink, filtering inbound routes to keep TX low)     cxl0     287 k    2015 M     353       244 k 0   (our network)     cxl1     940 M    3115 M     176 k     360 k 0 (our network)     cxl2     634 M    1014 M     103 k     128 k 0 (our network)     cxl3       1 k      16 M       1         4 k       0     cxl4       0         0         0         0         0     cxl5       0         0         0         0         0     cxl6    2343 M     791 M     275 k     137 k 0 (IX , part of lagg0)     cxl7    1675 M     762 M     215 k     133 k 0 (IX , part of lagg0)     ixl0     913 k      18 M       0         0 0     ixl1       1 M      30 M       0         0         0    lagg0    4019 M    1554 M     491 k     271 k       0    lagg1       1 M      48 M       0         0         0 FreeBSD 12.1-STABLE orange                 976 Bytes/Packetavg  1:42PM  up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 # /boot/loader.conf # ZFS! kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" zfs_load="YES" # Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card, 40/100Gbps prots) devices # built into kernel... # if_cxgbe_load="YES" # t5fw_cfg_load="YES" # t6fw_cfg_load="YES" # Before FreeBSD 13, threading bad on router: https://calomel.org/freebsd_network_tuning.html machdep.hyperthreading_allowed="0" # https://wiki.freebsd.org/10gFreeBSD/Router hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" # /etc/sysctl.conf net.inet6.ip6.redirect=0 net.inet.ip.redirect=0 # FRR needs big buffers for OSPF kern.ipc.maxsockbuf=16777216 # Turn FEC off (no link to Cogent otherwise) dev.cc.0.fec=0 dev.cc.1.fec=0 # Entropy not from LAN ports... slows them down. kern.random.harvest.mask=65551 # TODO: try this, https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html # (Default was 2) hw.cxgbe.nm_holdoff_tmr_idx=3 net.inet.icmp.icmplim=400 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.blackhole=2          # drop any TCP packets to closed ports net.inet.tcp.msl=7500             # close lost tcp connections in 7.5 seconds (default 30) net.inet.udp.blackhole=1          # drop any UDP packets to closed ports # vmstat -i interrupt                          total       rate cpu0:timer                      54113958       1102 cpu1:timer                      55286089       1125 cpu2:timer                      53479994       1089 cpu3:timer                      55210439       1124 cpu4:timer                      25989916        529 cpu5:timer                      25908493        527 cpu6:timer                      55187512       1123 cpu7:timer                      54097494       1101 cpu8:timer                      54111676       1101 cpu9:timer                      54541125       1110 cpu10:timer                     55283621       1125 cpu11:timer                     55285466       1125 cpu12:timer                     55283322       1125 cpu13:timer                     55287200       1125 cpu14:timer                     55293560       1126 cpu15:timer                     55286002       1125 cpu16:timer                     55285700       1125 cpu17:timer                     55288330       1125 cpu18:timer                     55292101       1126 cpu19:timer                     55287826       1125 irq265: xhci0                      85784          2 irq266: ahci1                     720421         15 irq267: ixl0:aq                        1          0 irq268: ixl0:rxq0                8568015        174 irq269: ixl0:rxq1               10726322        218 irq270: ixl0:rxq2                7915180        161 irq271: ixl0:rxq3                8527106        174 irq272: ixl0:rxq4               13080418        266 irq273: ixl0:rxq5                7456434        152 irq274: ixl0:rxq6                8012988        163 irq275: ixl0:rxq7                8807168        179 irq276: ixl0:rxq8                9247761        188 irq277: ixl0:rxq9                8439429        172 irq278: ixl1:aq                        1          0 irq279: ixl1:rxq0                7750089        158 irq280: ixl1:rxq1                6604876        134 irq281: ixl1:rxq2                7180431        146 irq282: ixl1:rxq3                5990725        122 irq283: ixl1:rxq4                7385876        150 irq284: ixl1:rxq5               10252020        209 irq285: ixl1:rxq6                9155331        186 irq286: ixl1:rxq7                8384309        171 irq287: ixl1:rxq8                6642749        135 irq288: ixl1:rxq9               10019391        204 irq290: t6nex0:evt                     4          0 irq299: t6nex0:1a0             488060047       9935 irq300: t6nex0:1a1             379683827       7729 irq301: t6nex0:1a2             369471291       7521 irq302: t6nex0:1a3             393993838       8020 irq303: t6nex0:1a4             354430416       7215 irq304: t6nex0:1a5             599261919      12199 irq305: t6nex0:1a6             337249778       6865 irq306: t6nex0:1a7             511196035      10406 irq308: t5nex0:evt                     8          0 irq309: t5nex0:0a0               3117499         63 irq310: t5nex0:0a1               3123964         64 irq311: t5nex0:0a2               3163023         64 irq312: t5nex0:0a3               3163675         64 irq313: t5nex0:0a4               3036876         62 irq314: t5nex0:0a5               3207249         65 irq315: t5nex0:0a6               2997717         61 irq316: t5nex0:0a7               2861439         58 irq317: t5nex0:1a0             670962265      13658 irq318: t5nex0:1a1             666875099      13575 irq319: t5nex0:1a2             679409070      13830 irq320: t5nex0:1a3             700097218      14251 irq321: t5nex0:1a4             672477227      13689 irq322: t5nex0:1a5             674052457      13721 irq323: t5nex0:1a6             660159886      13438 irq324: t5nex0:1a7             797020409      16224 irq325: t5nex0:2a0             412131923       8389 irq326: t5nex0:2a1             561770787      11435 irq327: t5nex0:2a2             461510991       9394 irq328: t5nex0:2a3             420200582       8554 irq329: t5nex0:2a4             392331601       7986 irq330: t5nex0:2a5             410021075       8346 irq331: t5nex0:2a6             467853129       9524 irq332: t5nex0:2a7             423194899       8615 irq333: t5nex0:3a0                719985         15 irq334: t5nex0:3a1                613535         12 irq335: t5nex0:3a2                635461         13 irq336: t5nex0:3a3                613540         12 irq337: t5nex0:3a4                614360         13 irq338: t5nex0:3a5                613547         12 irq339: t5nex0:3a6                613539         12 irq340: t5nex0:3a7                613556         12 irq342: t5nex1:evt                     5          0 irq359: t5nex1:2a0             378696408       7709 irq360: t5nex1:2a1             383630463       7809 irq361: t5nex1:2a2             473624262       9641 irq362: t5nex1:2a3             391170574       7963 irq363: t5nex1:2a4             393111484       8002 irq364: t5nex1:2a5             428581285       8724 irq365: t5nex1:2a6             380367906       7743 irq366: t5nex1:2a7             382910534       7794 irq367: t5nex1:3a0             379035068       7716 irq368: t5nex1:3a1             381198007       7760 irq369: t5nex1:3a2             379586223       7727 irq370: t5nex1:3a3             391500901       7969 irq371: t5nex1:3a4             411892536       8384 irq372: t5nex1:3a5             395213950       8045 irq373: t5nex1:3a6             385156810       7840 irq374: t5nex1:3a7             380410814       7744 Total                        20060964625     408359 Thanks for any tips, Rudy