From owner-freebsd-net@freebsd.org Sun Mar 28 13:48:36 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6371257A1E1 for ; Sun, 28 Mar 2021 13:48:36 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F7cVq31Fxz4f07; Sun, 28 Mar 2021 13:48:35 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 12SDmU6M063042; Sun, 28 Mar 2021 06:48:30 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 12SDmUcX063041; Sun, 28 Mar 2021 06:48:30 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <202103281348.12SDmUcX063041@gndrsh.dnsmgr.net> Subject: Re: Which cpu/mainboard for fast routing (bgp, full tables) ? In-Reply-To: To: Kurt Jaeger Date: Sun, 28 Mar 2021 06:48:30 -0700 (PDT) CC: Kevin Bowling , freebsd-net@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 4F7cVq31Fxz4f07 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [-1.08 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[69.59.192.140:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[69.59.192.140:from]; NEURAL_HAM_SHORT(-0.98)[-0.984]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Mar 2021 13:48:36 -0000 > Hi! > > > That class of processor has fairly limited memory bandwidth. An E5 v3 or > > greater should get you what you want, although finding a system that makes > > good use of available PCIe lanes with a single socket configuration can > > sometimes be maddening. AMD may have a variety of nice parts for this > > application, although I don?t have any personal experience with routing on > > such hardware. > > Thanks -- I searched for a pair of boxes in my infra with those > specs, found them: > > Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz > 82599ES 10-Gigabit SFI/SFP+ Network Connection > and > Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz > 82599ES 10-Gigabit SFI/SFP+ Network Connection > > and tested. Roughly the same performance, if we use only one > connection. These CPU's have a quad channel memory controller, to achive anything close to the maximal memory bandwidth of 68BG/s you need to have all 4 banks of the memory occupied with identical DIMM's, and preferable DDR4 2133 or DDR4 2400. Also iirc the 82599ES uses the iflib driver code, and I am not sure how performant that is. The Mellanox and Chelsio cards are preferable in places that high speed is needed. > > iperf3 -c > > The boxes were able to reach 10gbit, if we run 3 threads in parallel: > > iperf3 -c -P 3 > > So I have some area where I can investigate further. I am not sure if hyperthreading has any effect on this type setup, I usually disable it on anything I want to be "performant" unless I see some great need for more threads. > pi@FreeBSD.org +49 171 3101372 Now what ? -- Rod Grimes rgrimes@freebsd.org