From owner-freebsd-net@freebsd.org Wed Jun 14 14:48:50 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2AC0AD8FAD1 for ; Wed, 14 Jun 2017 14:48:50 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E45F366B86 for ; Wed, 14 Jun 2017 14:48:49 +0000 (UTC) (envelope-from jjasen@gmail.com) Received: by mail-oi0-x22b.google.com with SMTP id k145so1817429oih.3 for ; Wed, 14 Jun 2017 07:48:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=qcM3K5ELZa0+J6m2bfaO+adwJx6plELG3/lKxxn69rE=; b=UmOzddT6IhRYt5AbyRqSjYANl0p9lH50s/bny0+Z+4v97TnMt0fRXMxW5dPJR7Z8Bt +P6NDwbuSx/EyJR+9SrFnzXlk7R3qe0zWmeyBCZmt/Dpt9CQ/GLINWwIPWU1G2UUX8Mq A+QNkNHpfsPrrd1OcNNCDVWz7rNCreL5HM+8qix3+PCIY2xoRkLsQrs9O9U3NloMQQPi vJrj5bUo0c4KS8k7+Rl1wGffTkwYyIlZlCMCZTh+VEhfdAgoiu74j3oCsRYhxSrjvaQp lmflXaOzNvHBKR2/LSnZD5LWt3I68ZxGSMThArQFNCqU31EjzWLEluhYluBza0+tA0Jk vTxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=qcM3K5ELZa0+J6m2bfaO+adwJx6plELG3/lKxxn69rE=; b=O8zqiA4sg19sBkJlRM7nP1ipHz2l0CGNCjwAOcYUhPoU55uOEqY68AK1F45n3wb/Ru gFcoGMZHUeqW9Mq0YLuM7Hh7UcjO7VFr7BZvu6TKjR66iAyF+A3rgzi8fPa/is1nWdR6 wYEdKDPK1f4Bt6E1+hS8z9hVvIWYzaAg66ulkndhUUZ5Nk0LNwTYYqsFUiOO2pQShu1m nyiukkvH0jhTdZuHgC7jKfOXtKq7X6Ty8X6J4x4WYpMtn7iIoBjojJot+CTgcxVLGY13 sLSdK4TI3HrmkiSVASeLU0jf5TFXH+FNLQkNQU3GjiM/NYw1XhOpFVPdYO04CJf3lHsC 1HxA== X-Gm-Message-State: AKS2vOxjDegUdJvYRWgNi8LN1yB3EPv8AE13YjHDpujzv+9rQjyG7tNN JlHbrqsA84MQ4+pgVXvl0MYLpeZBdkDZ X-Received: by 10.202.74.15 with SMTP id x15mr401111oia.96.1497451728889; Wed, 14 Jun 2017 07:48:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.157.63.100 with HTTP; Wed, 14 Jun 2017 07:48:48 -0700 (PDT) From: John Jasen Date: Wed, 14 Jun 2017 10:48:48 -0400 Message-ID: Subject: state of packet forwarding in FreeBSD? To: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jun 2017 14:48:50 -0000 Our goal was to test whether or not FreeBSD currently is viable, as the operating system platform for high speed routers and firewalls, in the 40 to 100 GbE range. In our investigations, we tested 10.3, 11.0/-STABLE, -CURRENT, and a USB stick from BSDRP using the FreeBSD routing improvements project enhancements (https://wiki.freebsd.org/ProjectsRoutingProposal). We've tried stock and netmap-fwd, have played around a little with netmap itself and dpdk, with the results summarized below. The current testing platform is a Dell PowerEdge R530 with a Chelsio T580-LP-CR dual port 40GbE card. Suggestions, examples for using netmap, etc, all warmly welcomed. Further questions cheerfully answered to the best of our abilities. a) On the positive side, it appears that 11.0 is much faster than 10.0, which we tested several years ago. With appropriate cpuset tuning, 5.5 mpps is achievable using modern hardware. Using slightly older hardware, (such as a Dell R720 with v3 xeons), around 5.2-5.3 mpps can be obtained. b) On the negative side, between the various releases, netmap appeared to be unstable with the Chelsio cards -- sometimes supported, sometimes broken. Also, we're still trying to figure out netmap utilities, such as vale-ctl and bridge, so any advice would be appreciated. b.1) netmap-fwd is admittedly single-threaded and does not support IPv6. These clearly showed in our tests, as we were unable to achieve over 2.5 mpps, saturating a single CPU and letting the others fall asleep. However, bumping a single CPU queue from around 0.6 mpps to 2.5 mpps is nothing to ignore, so it could be useful in some cases. c) The routing improvement project USB stick performed incredibly, achieving 8.5 mpps out of the box. However, it appears (https://wiki.freebsd.org/ProjectsRoutingProposal/ConversionStatus), that many of the changes are still pending review, and that things have not moved much in the last 18 months (https://svnweb.freebsd.org/base/projects/routing/) d) We've not figured out dpdk (dpdk.org) yet. Our first foray into the test examples, and we're stuck trying to get the interfaces online. -- John Jasen (jjasen@gmail.com)