From owner-freebsd-net@freebsd.org Fri Feb 14 22:58:10 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1A9FB245457 for ; Fri, 14 Feb 2020 22:58:10 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48K81F3w7sz4NPr for ; Fri, 14 Feb 2020 22:58:09 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: by mailman.nyi.freebsd.org (Postfix) id 847C1245456; Fri, 14 Feb 2020 22:58:09 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 83004245454 for ; Fri, 14 Feb 2020 22:58:09 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mail.monkeybrains.net (mail.monkeybrains.net [208.69.40.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.monkeybrains.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48K81C5qnHz4NPq for ; Fri, 14 Feb 2020 22:58:07 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from [10.2.86.111] (public.monkeybrains.net [208.69.41.107] (may be forged)) (authenticated bits=0) by mail.monkeybrains.net (8.15.2/8.15.2) with ESMTPSA id 01EMw5OQ095098 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 14 Feb 2020 14:58:06 -0800 (PST) (envelope-from crapsh@monkeybrains.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=monkeybrains.net; s=dkim; t=1581721086; bh=WrApMMRTMixz/I+6NbdsUvPNahJ8vTde8hUJK9dufWM=; h=Subject:To:References:From:Date:In-Reply-To; b=Iujtl0RcecnXbtrkNOrr+zBwGu654hqsfgexBeB5X4kRPC5oobmpXhTzgs0BkSjeo N/hAX2FJ7lLXABeLn8ezUOA7qZ5p0fGAm6lQ9a6LdfVym0cfQW2wntx/jaOU5rnMI2 K2PdLJWB93d8mIoqXwsiz8nJ6+rVqeyAY4q5wTfc= X-Authentication-Warning: mail.monkeybrains.net: Host public.monkeybrains.net [208.69.41.107] (may be forged) claimed to be [10.2.86.111] Subject: Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1 To: "Andrey V. Elsukov" , net@FreeBSD.org References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> From: BulkMailForRudy Message-ID: <317422ba-e052-aecf-aed3-fbb1325220d9@monkeybrains.net> Date: Fri, 14 Feb 2020 14:58:05 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.4.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: clamav-milter 0.101.4 at mail.monkeybrains.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 48K81C5qnHz4NPq X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=monkeybrains.net header.s=dkim header.b=Iujtl0Rc; dmarc=pass (policy=none) header.from=monkeybrains.net; spf=pass (mx1.freebsd.org: domain of crapsh@monkeybrains.net designates 208.69.40.19 as permitted sender) smtp.mailfrom=crapsh@monkeybrains.net X-Spamd-Result: default: False [-6.79 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[monkeybrains.net:s=dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[monkeybrains.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[monkeybrains.net,none]; IP_SCORE(-3.69)[ip: (-9.71), ipnet: 208.69.40.0/22(-4.79), asn: 32329(-3.90), country: US(-0.05)]; FREEMAIL_TO(0.00)[yandex.ru]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[19.40.69.208.list.dnswl.org : 127.0.5.1]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:32329, ipnet:208.69.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[monkeybrains.net.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2020 22:58:10 -0000 On 2/14/20 4:21 AM, Andrey V. Elsukov wrote: > On 13.02.2020 06:21, Rudy wrote: >> >> I'm having issues with a box that is acting as a BGP router for my >> network.  3 Chelsio cards, two T5 and one T6.  It was working great >> until I turned up our first port on the T6.  It seems like traffic >> passing in from a T5 card and out the T6 causes a really high load (and >> high interrupts). >> >> Traffic (not that much, right?) >> >>      Dev  RX bps    TX bps    RX PPS    TX PPS Error >>      cc0       0         0         0         0         0 >>      cc1    2212 M       7 M     250 k       6 k 0 (100Gbps uplink, >> filtering inbound routes to keep TX low) >>     cxl0     287 k    2015 M     353       244 k 0   (our network) >>     cxl1     940 M    3115 M     176 k     360 k 0 (our network) >>     cxl2     634 M    1014 M     103 k     128 k 0 (our network) >>     cxl3       1 k      16 M       1         4 k       0 >>     cxl4       0         0         0         0         0 >>     cxl5       0         0         0         0         0 >>     cxl6    2343 M     791 M     275 k     137 k 0 (IX , part of lagg0) >>     cxl7    1675 M     762 M     215 k     133 k 0 (IX , part of lagg0) >>     ixl0     913 k      18 M       0         0 0 >>     ixl1       1 M      30 M       0         0         0 >>    lagg0    4019 M    1554 M     491 k     271 k       0 >>    lagg1       1 M      48 M       0         0         0 >> FreeBSD 12.1-STABLE orange                 976 Bytes/Packetavg >>  1:42PM  up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 > Hi, > > did you try to use pmcstat to determine what is the heaviest task for > your system? > > # kldload hwpmc > # pmcstat -S inst_retired.any -Tw1 PMC: [inst_retired.any] Samples: 168557 (100.0%) , 2575 unresolved Key: q => exiting... %SAMP IMAGE      FUNCTION             CALLERS  16.6 kernel     sched_idletd         fork_exit  14.7 kernel     cpu_search_highest   cpu_search_highest:12.4 sched_switch:1.4 sched_idletd:0.9  10.5 kernel     cpu_search_lowest    cpu_search_lowest:9.6 sched_pickcpu:0.9   4.2 kernel     eth_tx               drain_ring   3.4 kernel     rn_match             fib4_lookup_nh_basic   2.4 kernel     lock_delay           __mtx_lock_sleep   1.9 kernel     mac_ifnet_check_tran ether_output > > Then capture several first lines from the output and quit using 'q'. > > Do you use some firewall? Also, can you show the snapshot from the `top > -HPSIzts1` output. last pid: 28863;  load averages:  9.30, 10.33, 10.56                                up 0+14:16:08  14:53:23 817 threads:   25 running, 586 sleeping, 206 waiting CPU 0:   0.8% user,  0.0% nice,  6.2% system,  0.0% interrupt, 93.0% idle CPU 1:   2.4% user,  0.0% nice,  0.0% system,  7.9% interrupt, 89.8% idle CPU 2:   0.0% user,  0.0% nice,  0.8% system,  7.1% interrupt, 92.1% idle CPU 3:   1.6% user,  0.0% nice,  0.0% system, 10.2% interrupt, 88.2% idle CPU 4:   0.0% user,  0.0% nice,  0.0% system,  9.4% interrupt, 90.6% idle CPU 5:   0.8% user,  0.0% nice,  0.8% system, 20.5% interrupt, 78.0% idle CPU 6:   1.6% user,  0.0% nice,  0.0% system,  5.5% interrupt, 92.9% idle CPU 7:   0.0% user,  0.0% nice,  0.0% system,  3.1% interrupt, 96.9% idle CPU 8:   0.8% user,  0.0% nice,  0.8% system,  7.1% interrupt, 91.3% idle CPU 9:   0.0% user,  0.0% nice,  0.8% system,  9.4% interrupt, 89.8% idle CPU 10:  0.0% user,  0.0% nice,  0.0% system, 35.4% interrupt, 64.6% idle CPU 11:  0.0% user,  0.0% nice,  0.0% system, 36.2% interrupt, 63.8% idle CPU 12:  0.0% user,  0.0% nice,  0.0% system, 38.6% interrupt, 61.4% idle CPU 13:  0.0% user,  0.0% nice,  0.0% system, 49.6% interrupt, 50.4% idle CPU 14:  0.0% user,  0.0% nice,  0.0% system, 46.5% interrupt, 53.5% idle CPU 15:  0.0% user,  0.0% nice,  0.0% system, 32.3% interrupt, 67.7% idle CPU 16:  0.0% user,  0.0% nice,  0.0% system, 46.5% interrupt, 53.5% idle CPU 17:  0.0% user,  0.0% nice,  0.0% system, 56.7% interrupt, 43.3% idle CPU 18:  0.0% user,  0.0% nice,  0.0% system, 31.5% interrupt, 68.5% idle CPU 19:  0.0% user,  0.0% nice,  0.8% system, 34.6% interrupt, 64.6% idle Mem: 636M Active, 1159M Inact, 5578M Wired, 24G Free ARC: 1430M Total, 327M MFU, 589M MRU, 32K Anon, 13M Header, 502M Other      268M Compressed, 672M Uncompressed, 2.51:1 Ratio Swap: 4096M Total, 4096M Free   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND    12 root        -92    -     0B  3376K WAIT    13  41:13  12.86% intr{irq358: t5nex0:2a1}    12 root        -92    -     0B  3376K WAIT    12  48:08  12.77% intr{irq347: t5nex0:1a6}    12 root        -92    -     0B  3376K CPU13   13  47:40  11.96% intr{irq348: t5nex0:1a7}    12 root        -92    -     0B  3376K WAIT    17  43:46  11.38% intr{irq342: t5nex0:1a1}    12 root        -92    -     0B  3376K WAIT    14  29:17  10.70% intr{irq369: t5nex0:2ac}    12 root        -92    -     0B  3376K WAIT    11  47:55   9.85% intr{irq428: t5nex1:2a5}    12 root        -92    -     0B  3376K WAIT    16  46:11   9.22% intr{irq351: t5nex0:1aa}    12 root        -92    -     0B  3376K WAIT    19  42:28   9.04% intr{irq344: t5nex0:1a3}    12 root        -92    -     0B  3376K WAIT    16  46:45   8.82% intr{irq341: t5nex0:1a0}    12 root        -92    -     0B  3376K RUN     11  48:04   8.33% intr{irq356: t5nex0:1af}    12 root        -92    -     0B  3376K WAIT    10  46:24   8.32% intr{irq355: t5nex0:1ae}    12 root        -92    -     0B  3376K WAIT    10  42:03   8.32% intr{irq345: t5nex0:1a4}    12 root        -92    -     0B  3376K WAIT    14  36:34   8.29% intr{irq441: t5nex1:3a2}    12 root        -92    -     0B  3376K WAIT    19  46:14   8.21% intr{irq354: t5nex0:1ad}    12 root        -92    -     0B  3376K WAIT    14  47:29   8.13% intr{irq349: t5nex0:1a8}    12 root        -92    -     0B  3376K WAIT    11  40:25   7.91% intr{irq346: t5nex0:1a5}    12 root        -92    -     0B  3376K WAIT    15  49:33   7.62% intr{irq350: t5nex0:1a9}    12 root        -92    -     0B  3376K WAIT     5  45:37   7.57% intr{irq322: t6nex0:1af}    12 root        -92    -     0B  3376K WAIT    18  45:41   7.43% intr{irq353: t5nex0:1ac}    12 root        -92    -     0B  3376K WAIT    17  36:43   7.34% intr{irq434: t5nex1:2ab}    12 root        -92    -     0B  3376K WAIT    17  33:30   7.11% intr{irq424: t5nex1:2a1}    12 root        -92    -     0B  3376K WAIT     4  31:43   7.02% intr{irq312: t6nex0:1a5}    12 root        -92    -     0B  3376K WAIT    16  35:01   6.95% intr{irq433: t5nex1:2aa}    12 root        -92    -     0B  3376K WAIT    17  47:03   6.84% intr{irq352: t5nex0:1ab}    12 root        -92    -     0B  3376K WAIT    18  41:33   6.73% intr{irq343: t5nex0:1a2}    12 root        -92    -     0B  3376K WAIT     9  37:02   6.42% intr{irq317: t6nex0:1aa}    12 root        -92    -     0B  3376K WAIT    10  32:22   6.40% intr{irq427: t5nex1:2a4} Thanks.  I did change the chelsio_affinity today to get the cards to bind IRQs to CPU cores in the same numa-domain.  Still, load seems a bit high when using the t6 card compared to just using the T5 cards.