From owner-freebsd-net@freebsd.org Wed Feb 24 00:22:37 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 05DF355995C for ; Wed, 24 Feb 2021 00:22:37 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "plan-b.pwste.edu.pl", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Dlc7b6rH7z4vB8 for ; Wed, 24 Feb 2021 00:22:35 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from fomalhaut.potoki.eu ([IPv6:2001:470:71:d47:8564:4c1f:4f3a:2bc4]) (authenticated bits=0) by plan-b.pwste.edu.pl (8.16.1/8.16.1) with ESMTPSA id 11O0MPP3057662 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 24 Feb 2021 01:22:25 +0100 (CET) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1614126146; bh=MSRuW9IfvZ/vbFGaOTas5KnPShRwLG2Zagg1WSquKec=; h=From:To:References:Subject:Date:In-Reply-To; b=Spx4xRdAyEP1htVMkcKKx8XM7wQyJ9nB2cNzc4Mw/VcMJuHknO7AvCiHMPIwr0gwE 86ZNv9yJOH4afRu7KyjiVoh7J/x+XgJkoTMVEb2wTN1ijTj19Y3+xg+PIwYd7osBxa qMjxJdMqD9fXvMBeMp7wwclLLL5nfdtHz5Yruhi0OYZ5yAhK6Ujs1MuJ0rRCr0xEj1 NOs7Hern03gPigNIRJdRenquqMviGKrLzrjrlN4Y7Sj+twhKhzyhN1vDijXEKj7yBs +RP7NnjyEvNp2Ehq1cZkJnr/zbsbukFj8knRm0BUbIPIyKx36bw0aUTA+aAwHIzZAT W2LoZNyGk+73A== X-Authentication-Warning: plan-b.pwste.edu.pl: Host [IPv6:2001:470:71:d47:8564:4c1f:4f3a:2bc4] claimed to be fomalhaut.potoki.eu From: Marek Zarychta To: "Alexander V. Chernikov" , freebsd-net@freebsd.org References: <5670cd9a-cd10-2b89-1347-97a6c817c50f@sentex.net> <8696072a-dc25-8eff-04fa-4d1db13bf5cc@plan-b.pwste.edu.pl> <3588111612809020@mail.yandex.ru> <53ef5715-42ce-aad0-9a8b-11d91a9891e3@plan-b.pwste.edu.pl> <301391612827263@mail.yandex.ru> <2642181612914660@mail.yandex.ru> <2951613086198@mail.yandex.ru> <7121614026888@mail.yandex.ru> Subject: Re: option FIB_ALGO and dpdk_lpm4 Message-ID: <7056db64-2e7f-6ad8-54a3-59ff406061cf@plan-b.pwste.edu.pl> Date: Wed, 24 Feb 2021 01:22:24 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <7121614026888@mail.yandex.ru> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Rspamd-Queue-Id: 4Dlc7b6rH7z4vB8 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=plan-b.pwste.edu.pl header.s=plan-b-mailer header.b=Spx4xRdA; dmarc=pass (policy=none) header.from=plan-b.pwste.edu.pl; spf=none (mx1.freebsd.org: domain of zarychtam@plan-b.pwste.edu.pl has no SPF policy when checking 2001:678:618::40) smtp.mailfrom=zarychtam@plan-b.pwste.edu.pl X-Spamd-Result: default: False [-5.55 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[plan-b.pwste.edu.pl:s=plan-b-mailer]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_MED(-2.00)[pwste.edu.pl:dkim]; SPAMHAUS_ZRD(0.00)[2001:678:618::40:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[plan-b.pwste.edu.pl:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[plan-b.pwste.edu.pl,none]; NEURAL_HAM_SHORT(-0.75)[-0.755]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:678:618::40:from]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Feb 2021 00:22:37 -0000 > =C2=A0 > Btw, have you seen any notable dataplane/control plane performance > difference w/ FIB_ALGO ? Dear Alexander, valued subscribers, at last, I have upgraded our BGP router running two instances of net/bird (IP and IPv6) from 11.4-STABLE=C2=A0 to 13.0-STABLE just on time= to catch the "Fix nd6 rib_action() handling". The performance boost is really impressive, but probably not only FIB_ALGO is responsible for it since there were also improvements in PF, lagg(4), aggregated=C2=A0 NICs utilize now iflib and so on. Getting to the point: now when the links are saturated with normal traffic the load on this 8 core ATOM dropped from ~3.5 to ~0.9. The pmc(3) has changed a bit over time and now with PMC: [cpu_clk_unhalted.ref] I get max 1.5% for=C2=A0 rn_match function, b= ut the caller is always pfr_match_addr, so it comes likely from PF. In 11.4=C2=A0 rn_match was peaking up to=C2=A0 9% with links saturated with = normal traffic but there were more callers: fib6_lookup_nh_basic:4.2 pfr_match_addr:1.4 fib4_lookup_nh_basic:1.4. Now only occasionally I observe function=C2=A0 rn_walktree peaking up to ~20% called by rib_walk = what probably comes from net/bird adding/removing routes. At this stage (updating routes) on 11.x-STABLE bird seemed to block/stale the ability to interact with the system for a fraction of second. Now all updates go smooth. So congrats, an excellent job was done, thank you very much for this effort and time spent on making this all happen. Please let me ask only one final question. Does adding: net.route.algo.inet.algo=3Ddpdk_lpm4 net.route.algo.inet6.algo=3Ddpdk_lpm6 to /etc/sysctl.conf make any sense?=C2=A0 I see that right FIB_ALGO is automatically picked up when the module is available. After first reboot into 13-STABLE: Feb 23 22:42:10 rtr kernel: [fib_algo] fib_module_register: attaching radix6_lockless to inet6 Feb 23 22:42:10 rtr kernel: [fib_algo] fib_module_register: attaching radix6 to inet6 Feb 23 22:42:10 rtr kernel: [fib_algo] fib_module_register: attaching bsearch4 to inet Feb 23 22:42:10 rtr kernel: [fib_algo] fib_module_register: attaching radix4_lockless to inet Feb 23 22:42:10 rtr kernel: [fib_algo] fib_module_register: attaching radix4 to inet Feb 23 22:42:10 rtr kernel: [33] [fib_algo] fib_module_register: attaching dpdk_lpm6 to inet6 Feb 23 22:42:10 rtr kernel: [33] [fib_algo] fib_module_register: attaching dpdk_lpm4 to inet Feb 23 22:42:10 rtr kernel: [43] [fib_algo] inet.0 (bsearch4#66) rebuild_fd: switching algo to radix4_lockless Feb 23 22:42:10 rtr kernel: [49] [fib_algo] inet.1 (bsearch4#67) rebuild_fd: switching algo to radix4_lockless Feb 23 22:42:39 rtr kernel: [80] [fib_algo] inet.0 (radix4_lockless#650) rebuild_fd: switching algo to dpdk_lpm4 Feb 23 22:42:42 rtr kernel: [83] [fib_algo] inet6.0 (radix6_lockless#705) rebuild_fd: switching algo to dpdk_lpm6 Though after second reboot with "Fix nd6 rib_action() handling" applied there is no "switching algo to dpdk_lpm6": Feb 23 23:55:24 rtr kernel: [fib_algo] fib_module_register: attaching radix6_lockless to inet6 Feb 23 23:55:24 rtr kernel: [fib_algo] fib_module_register: attaching radix6 to inet6 Feb 23 23:55:24 rtr kernel: [fib_algo] fib_module_register: attaching bsearch4 to inet Feb 23 23:55:24 rtr kernel: [fib_algo] fib_module_register: attaching radix4_lockless to inet Feb 23 23:55:24 rtr kernel: [fib_algo] fib_module_register: attaching radix4 to inet Feb 23 23:55:24 rtr kernel: [10] [fib_algo] fib_module_register: attaching dpdk_lpm6 to inet6 Feb 23 23:55:24 rtr kernel: [10] [fib_algo] fib_module_register: attaching dpdk_lpm4 to inet Feb 23 23:55:24 rtr kernel: [20] [fib_algo] inet.0 (bsearch4#66) rebuild_fd: switching algo to radix4_lockless Feb 23 23:55:24 rtr kernel: [26] [fib_algo] inet.1 (bsearch4#67) rebuild_fd: switching algo to radix4_lockless Feb 23 23:55:57 rtr kernel: [61] [fib_algo] inet.0 (radix4_lockless#571) rebuild_fd: switching algo to dpdk_lpm4 Should I be bothered about it? With kind regards, --=20 Marek Zarychta