From owner-freebsd-net@freebsd.org Thu Feb 13 07:05:21 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DAD2524F86E for ; Thu, 13 Feb 2020 07:05:21 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48J6wK5YX4z4ZWP for ; Thu, 13 Feb 2020 07:05:21 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: by mailman.nyi.freebsd.org (Postfix) id BCEA824F86D; Thu, 13 Feb 2020 07:05:21 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BB92C24F86C for ; Thu, 13 Feb 2020 07:05:21 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from sfo.monkeybrains.net (sfo.monkeybrains.net [208.69.40.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.monkeybrains.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48J6wJ3FVJz4ZWN for ; Thu, 13 Feb 2020 07:05:20 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from Rudys-iMac.lan (148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged)) (authenticated bits=0) by sfo.monkeybrains.net (8.15.2/8.15.2) with ESMTPSA id 01D75IKA065399 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 12 Feb 2020 23:05:18 -0800 (PST) (envelope-from crapsh@monkeybrains.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=monkeybrains.net; s=dkim; t=1581577518; bh=cACDM0jkD8ousEsXs51NP6YIKwfM6CuM50jfKz5a3xE=; h=Subject:From:To:References:Date:In-Reply-To; b=gEOvs7RU/IBVvcLutoEMZgzvIUGxJR1bhFCq3qXt0xt6BnGj96J09JW4QGT4PYBHj oYQzYu6UfmOgc5v+QcQjd53XhLk39cDIlLzNOoKCDd3TRwJNYQfTUy604vShrStUdT hYk+z1smgv8CE9xr1oAql/ol46qkha9wV8w12STY= X-Authentication-Warning: mail.monkeybrains.net: Host 148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged) claimed to be Rudys-iMac.lan Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 From: Rudy To: net@FreeBSD.org References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> Message-ID: Date: Wed, 12 Feb 2020 23:05:18 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: clamav-milter 0.101.4 at mail.monkeybrains.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 48J6wJ3FVJz4ZWN X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=monkeybrains.net header.s=dkim header.b=gEOvs7RU; dmarc=pass (policy=none) header.from=monkeybrains.net; spf=pass (mx1.freebsd.org: domain of crapsh@monkeybrains.net designates 208.69.40.9 as permitted sender) smtp.mailfrom=crapsh@monkeybrains.net X-Spamd-Result: default: False [-5.66 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ptr]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[monkeybrains.net:+]; DMARC_POLICY_ALLOW(0.00)[monkeybrains.net,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-3.67)[ip: (-9.57), ipnet: 208.69.40.0/22(-4.83), asn: 32329(-3.91), country: US(-0.05)]; ASN(0.00)[asn:32329, ipnet:208.69.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[monkeybrains.net.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(0.00)[monkeybrains.net:s=dkim]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; URIBL_PBL(0.01)[calomel.org]; RCVD_IN_DNSWL_NONE(0.00)[9.40.69.208.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2020 07:05:21 -0000 Load seems to be increasing even though outbound traffic is blocked.  I'm seeing 'Discards In' rise up to about 4k/second when doing 4Gbps inbound traffic. I see the tx_pause counter incrementing...  dev.cc.1.stats.tx_pause: 2911586  (~ 20/second increase) Any tips would be great, I'm stumped. 20 cores seems like plenty to do much higher PPS based on what I see on the freebsd-net list. Thanks for any pointers, Rudy On 2/12/20 7:21 PM, Rudy wrote: > > > I'm having issues with a box that is acting as a BGP router for my > network.  3 Chelsio cards, two T5 and one T6.  It was working great > until I turned up our first port on the T6.  It seems like traffic > passing in from a T5 card and out the T6 causes a really high load > (and high interrupts). > > Attached graphs (will that work on a mailing list?) shows the load > going up. > > The work around at this point is that I block inbound routes (out > bound traffic) on cc1. > > I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't > seem to lower Interrupts. > > > # netstat -rn | wc -l >  891149 > > CPU:  0.2% user,  0.0% nice,  0.6% system, 64.4% interrupt, 34.8% idle > > Traffic (not that much, right?) > >      Dev  RX bps    TX bps    RX PPS    TX PPS Error >      cc0       0         0         0         0         0 >      cc1    2212 M       7 M     250 k       6 k 0 (100Gbps uplink, > filtering inbound routes to keep TX low) >     cxl0     287 k    2015 M     353       244 k 0   (our network) >     cxl1     940 M    3115 M     176 k     360 k 0 (our network) >     cxl2     634 M    1014 M     103 k     128 k 0 (our network) >     cxl3       1 k      16 M       1         4 k       0 >     cxl4       0         0         0         0         0 >     cxl5       0         0         0         0         0 >     cxl6    2343 M     791 M     275 k     137 k 0 (IX , part of lagg0) >     cxl7    1675 M     762 M     215 k     133 k 0 (IX , part of lagg0) >     ixl0     913 k      18 M       0         0 0 >     ixl1       1 M      30 M       0         0         0 >    lagg0    4019 M    1554 M     491 k     271 k       0 >    lagg1       1 M      48 M       0         0         0 > FreeBSD 12.1-STABLE orange                 976 Bytes/Packetavg >  1:42PM  up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 > > > # /boot/loader.conf > # ZFS! > kern.geom.label.disk_ident.enable="0" > kern.geom.label.gptid.enable="0" > zfs_load="YES" > > # Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card, > 40/100Gbps prots) devices > # built into kernel... > # if_cxgbe_load="YES" > # t5fw_cfg_load="YES" > # t6fw_cfg_load="YES" > > # Before FreeBSD 13, threading bad on router: > https://calomel.org/freebsd_network_tuning.html > machdep.hyperthreading_allowed="0" > > # https://wiki.freebsd.org/10gFreeBSD/Router > hw.cxgbe.toecaps_allowed="0" > hw.cxgbe.rdmacaps_allowed="0" > hw.cxgbe.iscsicaps_allowed="0" > hw.cxgbe.fcoecaps_allowed="0" > > > > > # /etc/sysctl.conf > > net.inet6.ip6.redirect=0 > net.inet.ip.redirect=0 > > # FRR needs big buffers for OSPF > kern.ipc.maxsockbuf=16777216 > > # Turn FEC off (no link to Cogent otherwise) > dev.cc.0.fec=0 > dev.cc.1.fec=0 > > # Entropy not from LAN ports... slows them down. > kern.random.harvest.mask=65551 > > # TODO: try this, > https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html > # (Default was 2) > hw.cxgbe.nm_holdoff_tmr_idx=3 > > net.inet.icmp.icmplim=400 > net.inet.icmp.maskrepl=0 > net.inet.icmp.log_redirect=0 > net.inet.icmp.drop_redirect=1 > net.inet.tcp.drop_synfin=1 > net.inet.tcp.blackhole=2          # drop any TCP packets to closed ports > net.inet.tcp.msl=7500             # close lost tcp connections in 7.5 > seconds (default 30) > net.inet.udp.blackhole=1          # drop any UDP packets to closed ports > > > > > > # vmstat -i > interrupt                          total       rate > cpu0:timer                      54113958       1102 > cpu1:timer                      55286089       1125 > cpu2:timer                      53479994       1089 > cpu3:timer                      55210439       1124 > cpu4:timer                      25989916        529 > cpu5:timer                      25908493        527 > cpu6:timer                      55187512       1123 > cpu7:timer                      54097494       1101 > cpu8:timer                      54111676       1101 > cpu9:timer                      54541125       1110 > cpu10:timer                     55283621       1125 > cpu11:timer                     55285466       1125 > cpu12:timer                     55283322       1125 > cpu13:timer                     55287200       1125 > cpu14:timer                     55293560       1126 > cpu15:timer                     55286002       1125 > cpu16:timer                     55285700       1125 > cpu17:timer                     55288330       1125 > cpu18:timer                     55292101       1126 > cpu19:timer                     55287826       1125 > irq265: xhci0                      85784          2 > irq266: ahci1                     720421         15 > irq267: ixl0:aq                        1          0 > irq268: ixl0:rxq0                8568015        174 > irq269: ixl0:rxq1               10726322        218 > irq270: ixl0:rxq2                7915180        161 > irq271: ixl0:rxq3                8527106        174 > irq272: ixl0:rxq4               13080418        266 > irq273: ixl0:rxq5                7456434        152 > irq274: ixl0:rxq6                8012988        163 > irq275: ixl0:rxq7                8807168        179 > irq276: ixl0:rxq8                9247761        188 > irq277: ixl0:rxq9                8439429        172 > irq278: ixl1:aq                        1          0 > irq279: ixl1:rxq0                7750089        158 > irq280: ixl1:rxq1                6604876        134 > irq281: ixl1:rxq2                7180431        146 > irq282: ixl1:rxq3                5990725        122 > irq283: ixl1:rxq4                7385876        150 > irq284: ixl1:rxq5               10252020        209 > irq285: ixl1:rxq6                9155331        186 > irq286: ixl1:rxq7                8384309        171 > irq287: ixl1:rxq8                6642749        135 > irq288: ixl1:rxq9               10019391        204 > irq290: t6nex0:evt                     4          0 > irq299: t6nex0:1a0             488060047       9935 > irq300: t6nex0:1a1             379683827       7729 > irq301: t6nex0:1a2             369471291       7521 > irq302: t6nex0:1a3             393993838       8020 > irq303: t6nex0:1a4             354430416       7215 > irq304: t6nex0:1a5             599261919      12199 > irq305: t6nex0:1a6             337249778       6865 > irq306: t6nex0:1a7             511196035      10406 > irq308: t5nex0:evt                     8          0 > irq309: t5nex0:0a0               3117499         63 > irq310: t5nex0:0a1               3123964         64 > irq311: t5nex0:0a2               3163023         64 > irq312: t5nex0:0a3               3163675         64 > irq313: t5nex0:0a4               3036876         62 > irq314: t5nex0:0a5               3207249         65 > irq315: t5nex0:0a6               2997717         61 > irq316: t5nex0:0a7               2861439         58 > irq317: t5nex0:1a0             670962265      13658 > irq318: t5nex0:1a1             666875099      13575 > irq319: t5nex0:1a2             679409070      13830 > irq320: t5nex0:1a3             700097218      14251 > irq321: t5nex0:1a4             672477227      13689 > irq322: t5nex0:1a5             674052457      13721 > irq323: t5nex0:1a6             660159886      13438 > irq324: t5nex0:1a7             797020409      16224 > irq325: t5nex0:2a0             412131923       8389 > irq326: t5nex0:2a1             561770787      11435 > irq327: t5nex0:2a2             461510991       9394 > irq328: t5nex0:2a3             420200582       8554 > irq329: t5nex0:2a4             392331601       7986 > irq330: t5nex0:2a5             410021075       8346 > irq331: t5nex0:2a6             467853129       9524 > irq332: t5nex0:2a7             423194899       8615 > irq333: t5nex0:3a0                719985         15 > irq334: t5nex0:3a1                613535         12 > irq335: t5nex0:3a2                635461         13 > irq336: t5nex0:3a3                613540         12 > irq337: t5nex0:3a4                614360         13 > irq338: t5nex0:3a5                613547         12 > irq339: t5nex0:3a6                613539         12 > irq340: t5nex0:3a7                613556         12 > irq342: t5nex1:evt                     5          0 > irq359: t5nex1:2a0             378696408       7709 > irq360: t5nex1:2a1             383630463       7809 > irq361: t5nex1:2a2             473624262       9641 > irq362: t5nex1:2a3             391170574       7963 > irq363: t5nex1:2a4             393111484       8002 > irq364: t5nex1:2a5             428581285       8724 > irq365: t5nex1:2a6             380367906       7743 > irq366: t5nex1:2a7             382910534       7794 > irq367: t5nex1:3a0             379035068       7716 > irq368: t5nex1:3a1             381198007       7760 > irq369: t5nex1:3a2             379586223       7727 > irq370: t5nex1:3a3             391500901       7969 > irq371: t5nex1:3a4             411892536       8384 > irq372: t5nex1:3a5             395213950       8045 > irq373: t5nex1:3a6             385156810       7840 > irq374: t5nex1:3a7             380410814       7744 > Total                        20060964625     408359 > > Thanks for any tips, > Rudy > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >