From owner-freebsd-net@freebsd.org Fri Feb 14 05:56:52 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6EB7E241425 for ; Fri, 14 Feb 2020 05:56:52 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48JjLr1m7Rz46Xk for ; Fri, 14 Feb 2020 05:56:52 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: by mailman.nyi.freebsd.org (Postfix) id 3A9C9241424; Fri, 14 Feb 2020 05:56:52 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3A5E5241423 for ; Fri, 14 Feb 2020 05:56:52 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from sfo.monkeybrains.net (sfo.monkeybrains.net [208.69.40.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.monkeybrains.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48JjLq2Nh8z46Xj for ; Fri, 14 Feb 2020 05:56:51 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from Rudys-iMac.lan (148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged)) (authenticated bits=0) by sfo.monkeybrains.net (8.15.2/8.15.2) with ESMTPSA id 01E5uoP1010557 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 13 Feb 2020 21:56:50 -0800 (PST) (envelope-from crapsh@monkeybrains.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=monkeybrains.net; s=dkim; t=1581659810; bh=WQvYu+ipN95Q7dhhN85JyiuwlM8lh209f/BgXVuWAbI=; h=Subject:From:To:References:Date:In-Reply-To; b=BHTYcsrJwCS7udv76yQkpOPi/lFTdzgMWPjsMK0EGg38QJIffrllFFHwVtLGJjdgY QZ0Udlnc1Cez/mNierbSHYp5uDafgSAPzJjv9V2dWZWqy9/6aY1J9Uy2ki4Ed/g3/x 2ITKs0k4dxLzXVRAsbrpH9UytUcCO4vrZIqKRWU8= X-Authentication-Warning: mail.monkeybrains.net: Host 148-64-102-187.PUBLIC.monkeybrains.net [148.64.102.187] (may be forged) claimed to be Rudys-iMac.lan Subject: chelsio_affinity patch to support t6 cards From: Rudy To: net@FreeBSD.org References: <1aa78c6e-e640-623c-73d3-473df132eb72@monkeybrains.net> Message-ID: <05fb814c-b55a-d739-e417-bd1b01d256af@monkeybrains.net> Date: Thu, 13 Feb 2020 21:56:50 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.4.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: clamav-milter 0.101.4 at mail.monkeybrains.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 48JjLq2Nh8z46Xj X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=monkeybrains.net header.s=dkim header.b=BHTYcsrJ; dmarc=pass (policy=none) header.from=monkeybrains.net; spf=pass (mx1.freebsd.org: domain of crapsh@monkeybrains.net designates 208.69.40.9 as permitted sender) smtp.mailfrom=crapsh@monkeybrains.net X-Spamd-Result: default: False [-6.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[monkeybrains.net:s=dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[net@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[monkeybrains.net:+]; DMARC_POLICY_ALLOW(-0.50)[monkeybrains.net,none]; RCVD_IN_DNSWL_NONE(0.00)[9.40.69.208.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-3.63)[ip: (-9.41), ipnet: 208.69.40.0/22(-4.80), asn: 32329(-3.90), country: US(-0.05)]; ASN(0.00)[asn:32329, ipnet:208.69.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[monkeybrains.net.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2020 05:56:52 -0000 Supports t6 as well as t5 cards.  Also, is this desired? Rudy --- chelsio_affinity.orig    2020-02-13 21:04:43.032596000 -0800 +++ chelsio_affinity    2020-02-13 21:50:37.675528000 -0800 @@ -26,8 +26,9 @@  chelsio_affinity_start()  {      # Need a Chelsio NIC -    NQ=$(/sbin/sysctl -ni dev.cxl.0.nrxq) -    if [ -z "${NQ}" ]; then +    NQcxl=$(/sbin/sysctl -ni dev.cxl.0.nrxq) +    NQcc=$(/sbin/sysctl -ni dev.cc.0.nrxq) +    if [ -z "${NQcc}" && -z "${NQcxl}"]; then          echo "No Chelsio NIC port detected"          exit 0;      fi @@ -42,16 +43,30 @@      # Exclude some CPU from binding      NCPU=$((NCPU - ${chelsio_affinity_avoidncpu}))      CPU=${chelsio_affinity_firstcpu} -    cxl=$(sysctl dev.cxl. | grep -c nrxq) -    for i in $(seq 0 $((cxl - 1))); do -        for IRQ in `/usr/bin/vmstat -ai |\ -            /usr/bin/sed -nE "/t5nex0:${i}a/s/irq([[:digit:]]+):.*/\1/p"`; do -            echo "Bind t5nex0:${i}a IRQ ${IRQ} to CPU ${CPU}" -            /usr/bin/cpuset -l ${CPU} -x ${IRQ} -            CPU=$(((CPU + 1) % NCPU)) +    if [ ! -z "${NQcxl}" ]; then +        cxl=$(sysctl dev.cxl. | grep -c nrxq) +        for i in $(seq 0 $((cxl - 1))); do +            for IRQ in `/usr/bin/vmstat -ai |\ +                /usr/bin/sed -nE "/t5nex[01234567]:${i}a/s/irq([[:digit:]]+):.*/\1/p"`; do +                echo "Bind t5nex_:${i}a IRQ ${IRQ} to CPU ${CPU}" +                /usr/bin/cpuset -l ${CPU} -x ${IRQ} +                CPU=$(((CPU + 1) % NCPU)) +            done +            CPU=${chelsio_affinity_firstcpu}          done -        CPU=${chelsio_affinity_firstcpu} -    done +    fi +    if [ ! -z "${NQcc}" ]; then +        cc=$(sysctl dev.cc. | grep -c nrxq) +        for i in $(seq 0 $((cc - 1))); do +            for IRQ in `/usr/bin/vmstat -ai |\ +                /usr/bin/sed -nE "/t6nex[01234567]:${i}a/s/irq([[:digit:]]+):.*/\1/p"`; do +                echo "Bind t6nex_:${i}a IRQ ${IRQ} to CPU ${CPU}" +                /usr/bin/cpuset -l ${CPU} -x ${IRQ} +                CPU=$(((CPU + 1) % NCPU)) +            done +            CPU=${chelsio_affinity_firstcpu} +        done +    fi  }