From owner-freebsd-net@freebsd.org Thu Mar 9 18:01:56 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7FC4D05686 for ; Thu, 9 Mar 2017 18:01:56 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pg0-x235.google.com (mail-pg0-x235.google.com [IPv6:2607:f8b0:400e:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B758F1A7 for ; Thu, 9 Mar 2017 18:01:56 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: by mail-pg0-x235.google.com with SMTP id b129so29001477pgc.2 for ; Thu, 09 Mar 2017 10:01:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=RvQ0n2o7JIxt94Fthn2SLoP/d3YSHdPFUpgJ85spgeY=; b=FhgRKw6opcWZrT51L+4tFZDaB0U56Jzv67LX6uk+s10OWtu+u14ugMLSzm2TeG7XuO IkxyHYtqnFdEprW+8KCW/2T4EBBMeJWPxzvMC80FpeqMFk64hOW7wMRf3xxi1UP/+NNP DcERM1U2BnRmnoAQTO5D/uBtB4yWduB9cjIgLOvGxdBmYl7HcJineWXrpdO+Bye6GZmM jEaFuNebaKbJUffuk2jyQvqzr8TbYMVnxEeLcV9gT6iJ8MB0E9rnKW/+2/7OcwDTvfKb IJUYjB7912Fbxd5wKKc9pyOoawJY0w/SRYMLeySmLSp6KIEfXNO+7RzOliOdRiUvm3GR jCpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=RvQ0n2o7JIxt94Fthn2SLoP/d3YSHdPFUpgJ85spgeY=; b=DNfVtEWp+7di9+mr5TRB+JN5ZN7JjM2pYE/Xshe2+K6IeANb0c01kvBFswVXGw25F3 GGsEbjY1muGNb1ylZt+u2M84shQ16dONCvHGgfO9YQABFtzhdvOYv1opeObG2gkPl8hh 9prtHCxOakF4mCc38tKfbe0Dx6eYKCYhK9X0k9r8RsRT0VEmhWz9+vGqNvc2t/HRsjXa jRXT5hs9tnzgbhFivBU4dOisXmMvtNesHsMupY0QpoQJ0Sbf8TpR4cCw9z6a/QUSADf+ P6sQgMveP1nmBudRqQkm9U72o/kKiLqxPSzMvy27vQVAkiAsqPBfDD5MFtMmcrSxj/Zg dPdQ== X-Gm-Message-State: AMke39ls5bXGXsFYAZrFZZSZJBkeOKOXFBTMJmFbSKNkLkvqZ56eQbP5wLwXddm9gLCwzQL0bO1xPStLDb0Mdw== X-Received: by 10.99.173.69 with SMTP id y5mr15320072pgo.35.1489082516148; Thu, 09 Mar 2017 10:01:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.100.182.172 with HTTP; Thu, 9 Mar 2017 10:01:55 -0800 (PST) In-Reply-To: References: <93e10699-ecc7-c062-bf85-a26de0de9633@sentex.net> <901224a7-f3ed-84a4-5402-3542cd0e865b@sentex.net> From: Navdeep Parhar Date: Thu, 9 Mar 2017 10:01:55 -0800 Message-ID: Subject: Re: Chelsio netmap support ? (RELENG_11) To: Mike Tancsa Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Mar 2017 18:01:57 -0000 On Wed, Mar 8, 2017 at 6:28 AM, Mike Tancsa wrote: > On 3/7/2017 9:08 PM, Navdeep Parhar wrote: >> On Tue, Mar 7, 2017 at 5:46 PM, Mike Tancsa wrote: >> >>> >>> # dmesg | grep netm >>> netmap: loaded module >>> vcxl0: netmap queues/slots: TX 2/1023, RX 2/1024 >>> vcxl0: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) >>> vcxl1: netmap queues/slots: TX 2/1023, RX 2/1024 >>> vcxl1: 1 txq, 1 rxq (NIC); 1 txq, 1 rxq (TOE); 2 txq, 2 rxq (netmap) >>> igb0: netmap queues/slots: TX 4/1024, RX 4/1024 >>> igb1: netmap queues/slots: TX 4/1024, RX 4/1024 >>> >>> It maxes out at about 800Kpps with and without netmap. Is there a way >> >> Are you actually using a netmap based application that acts as a >> packet router or is this just the vcxl interface running as a normal >> ifnet? > > the later, vcxl running normal ifnet. I thought there would be a benefit > to utilizing netmap ? Sorry, this is not clear to me. The kernel's routing code does not utilize netmap even if it's available. You'll need something like netmap-fwd for netmap based routing. If you're not using netmap there is no need to create the extra vcxl interfaces. > >> >>> to increase the queues for the Chelsio nic, like the onboard igb ? >> >> If you're not running a netmap based router get rid of the num_vis=2 >> and simply try with the cxl0/cxl1 interfaces. They should each have 4 >> rxq/4 txq on your system. In case you want to increase the number of >> queues, use this: > > The tests with the regular cxl also show the box topping out at 0.8Mpps > for forwarding. I would have expected multiple streams to do better. There is a lot of information about forwarding on the bsdrp.net website. Have you tried the tips there? The numbers there are significantly better than what you observe. I suspect your router is CPU-bound. https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_hp_proliant_dl360p_gen8_with_10-gigabit_with_10-gigabit_chelsio_t540-cr https://bsdrp.net/documentation/examples/forwarding_performance_lab_of_a_superserver_5018a-ftn4_with_10-gigabit_chelsio_t540-cr There's a projects/routing branch that does much better than the stock kernel. I'm not sure what work remains to be done before it can be merged into head. https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650-8Cores-Chelsio_T540-CR/forwarding-pf-ipfw/results/fbsd11-routing.r287531/README.md Regards, Navdeep > >> >> The "NIC" queues are the normal tx/rx queues, the "netmap" queues are >> active when the interface is in netmap mode. >> >> Does netsend generate a single flow or multiple flows? If it's a >> single flow it will use a single queue only. > > I think its as a single flow. However, I was using a separate box to > generate a second flow as well. It still topped out at about 800Kpps > before dropping packets. > > ---Mike > > > > -- > ------------------- > Mike Tancsa, tel +1 519 651 3400 > Sentex Communications, mike@sentex.net > Providing Internet services since 1994 www.sentex.net > Cambridge, Ontario Canada http://www.tancsa.com/