From owner-freebsd-net@freebsd.org Sat Oct 10 14:35:33 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03368A106EB for ; Sat, 10 Oct 2015 14:35:33 +0000 (UTC) (envelope-from jim@netgate.com) Received: from mail-qg0-x22e.google.com (mail-qg0-x22e.google.com [IPv6:2607:f8b0:400d:c04::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C19501C65 for ; Sat, 10 Oct 2015 14:35:32 +0000 (UTC) (envelope-from jim@netgate.com) Received: by qgez77 with SMTP id z77so91490773qge.1 for ; Sat, 10 Oct 2015 07:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netgate.com; s=google; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=QGaUZr47VL0cMSCnkC91W1uhL6SmJkHjBWWNs1w5Iug=; b=XGwsFw1vjN0dqBYOabcj2v55A4vEQgjt5iuUtvhANXpas9HzLTF9osdwqwuPgIDVh7 Ks4WhfzBdpuf9yzVeCoqWKb/Yc5+wTJ0EfpEzYSxXxcSjBPCN+T7ytOeDK9EIi/DeJPZ n1R0ZEcvDeRIgu/0zzSm4g02DCQeUXxvwOaRA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=QGaUZr47VL0cMSCnkC91W1uhL6SmJkHjBWWNs1w5Iug=; b=h4goHTb80JmuNdhYsAwhzc2zjdMCRBzXkeeWSSg1dR6Ip+fkFZoeAHqMcNbUsxREu+ F4ChIkm9xpwcVC2FGtnCKjniGuX3RBlkxLlacJhDTmjYAJHI5JUwErU0FFruAeKuL+z2 c6FsHB21CZEhpOnySDg30Ex0lnGCDOHChiH2RGslHw5dHz6cXKNZhBcWZrkcUVSLC4XG YaormUDoGLo8eKWcE54hIRIp20RiisaPypP8r87OMhdMAaa8C7j89EtJlwckkiqK7eli JWDABz5ebonCJjY0ze4wjPIXKADS0GIArUz39JLSjp4F5XKh/Y620oaBVPvxwtGxV5VC uUfw== X-Gm-Message-State: ALoCoQnS+SbiepNomCD9HQCWy45ZeDY2RxoFlYusy+iAo20xXbuliuq40DikH4lRu06Nhgbh76iv X-Received: by 10.140.25.134 with SMTP id 6mr22329624qgt.96.1444487731794; Sat, 10 Oct 2015 07:35:31 -0700 (PDT) Received: from [10.50.1.76] ([200.19.188.5]) by smtp.gmail.com with ESMTPSA id o65sm2988178qhb.21.2015.10.10.07.35.30 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 10 Oct 2015 07:35:31 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3094\)) Subject: Re: netmap: recommended NIC for 40GbE capture on Linux? From: Jim Thompson In-Reply-To: Date: Sat, 10 Oct 2015 11:35:27 -0300 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Bruce Merry X-Mailer: Apple Mail (2.3094) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2015 14:35:33 -0000 > Before we spend money, I'd love to hear someone report success with = capturing a single flow at >4Mpps, >20Gbps using netmap on Linux and > what NIC they use. You said linux, and this is freebsd-net, but this blog post (from = yesterday) is probably apt. https://blog.cloudflare.com/single-rx-queue-kernel-bypass-with-netmap/ > Ideally this would include transparent mode working so that ARP, ICMP = etc can be handled by the host stack. netmap-fwd (written by loos@ and announced yesterday at BSDcon Brazil) = does this (re-injecting frames designed for =E2=80=99this=E2=80=99 host = on the host ring, though there is a mode to handle ARP and ICMP in the fastpath code). netmap-fwd is an implementation of the ip_fastforward code from FreeBSD = on top of netmap. It currently does around 5Mpps, single-stream, on a = single core using Chelsio or Intel NICs. It=E2=80=99s IPv4-only right now, but once we = add IPv6(*), it will be open sourced. PPS @ 64-byte frames (pkt-gen) through DUT Device under test Network interface = Kernel forwarding Fastforward enabled Tryforward patch netmap-fwd = application C2358 (2 core, 1.7 Ghz) Intel I354 = 123 kbps 217 kbps 213 kbps = 945 kpps C2758 (8 core, 2.4 GHz) Chelsio T520 10G 270 apps = 426 apps 431 apps 1.683 = Mpps Xeon-D 1540 (8 core, 2 GHz) Intel X552 10G 439 kbps = 557 kbps 544 kbps 2.230 = Mpps Xeon E3-1275 (4 core, 3.5 GHz) Intel X520-2 1.058 = Mpps 1.331 Mpps 1.297 Mpps 5.053 = Mpps =E2=80=98Tryforward' above is the code in = https://reviews.freebsd.org/D3737 (We haven=E2=80=99t tried it on linux.) I find it likely that an application that acts as a simple filter like = https://github.com/jibi/nm-single-rx-queue/blob/master/main.c or = https://github.com/erikarn/netmap-tools/tree/master/src/bridge=20 will greatly exceed the 5Mpps figure above. See also: = http://adrianchadd.blogspot.com.br/2015/04/intel-ddio-llc-cache-buffer-ali= gnment.html Jim (*) this work will need to wait until after we can get pfSense 2.3 = released=