From owner-freebsd-net@freebsd.org Sat Jan 23 20:22:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84422A8FB2A for ; Sat, 23 Jan 2016 20:22:14 +0000 (UTC) (envelope-from dudu.meyer@gmail.com) Received: from mail-oi0-x236.google.com (mail-oi0-x236.google.com [IPv6:2607:f8b0:4003:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 42AFE1D04 for ; Sat, 23 Jan 2016 20:22:14 +0000 (UTC) (envelope-from dudu.meyer@gmail.com) Received: by mail-oi0-x236.google.com with SMTP id w75so66883698oie.0 for ; Sat, 23 Jan 2016 12:22:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=gy8MQS87rKNhR/HqSgPWHELU3CuiAtS8iVvouOxUZrs=; b=HgD/JAHNiK5O8yCe+ct54NXjhyqI3CHXBGLfJTwuek6pRA8T2M/th6UQgLFeWhUswv M21k6ZohOM6p59bkbDEk2zXjmLAqNFRFOVgVUHbzPt078ehUNdo7vk0IX9SMv/PJYP8r nRDuVGimL07KOHOo4yCa81kEoodv+e67qpeRVNk4UTuPqmVk37QjIlgEc10i3VoMaojU xrqinl79tm2dDE3oKKL4yRgO8jcnvkw+3BwAp0MJG9AttuGtHeQpQVvIT8TIdwcQCF0W RNVdE61eg5gigb33vEeNI1hyxQgAH1KweFu8nI6rwRxf/nwhzgtCqEwWXygp3oJ9jy6x 3nNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=gy8MQS87rKNhR/HqSgPWHELU3CuiAtS8iVvouOxUZrs=; b=mXYq/9llFBexuy5dae9GR9dt8xuTBx9CNriuFM4dFLoky/u5leBqayz6nPpbwbNGA7 a0QhdswQNUXgSV8OP+iQKICR2drhFVbIege/PeVb7+EBa81N05fmWNfTHRiIP9orv+Ef wyfBxFMcx5hpL9X6XMLGmuVpPnT4YRMqThL7e9hAVM/lQG1n29EPlspRJf73y48lNOKx IanKv+UhgHcmN793ipVuhRENxSc5mV5BAfbPNM7xcBWvz+IPgue4AQ1p+9HCNbIEIAJj dY76r46xKjjB850MRzGZ6pWw1SlH1uioRQk9h5UbpTjLy5zNgPINPUUacklhGTffJUNh S5Zg== X-Gm-Message-State: AG10YOQf6Mek54n0Ui8dr/fQxCQYZuKsRbW+8kWRhKziHKjoFXeHNrsQFvu3da1/YBsh2h/aaoCPP8xocZDddQ== MIME-Version: 1.0 X-Received: by 10.202.73.67 with SMTP id w64mr7147014oia.84.1453580533503; Sat, 23 Jan 2016 12:22:13 -0800 (PST) Received: by 10.182.88.138 with HTTP; Sat, 23 Jan 2016 12:22:13 -0800 (PST) In-Reply-To: References: <20160123184320.CA903A0126@smtp.hushmail.com> Date: Sat, 23 Jan 2016 18:22:13 -0200 Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Eduardo Meyer To: Luigi Rizzo Cc: Marcus Cenzatti , Adrian Chadd , Pavel Odintsov , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 20:22:14 -0000 On Sat, Jan 23, 2016 at 5:49 PM, Luigi Rizzo wrote: > On Sat, Jan 23, 2016 at 10:43 AM, Marcus Cenzatti > wrote: > > > > > > On 1/23/2016 at 1:31 PM, "Adrian Chadd" wrote: > >> > >>For random src/dst ports and IPs and on the chelsio t5 40gig > >>hardware, > >>I was getting what, uhm, 40mil tx pps and around 25ish mil rx pps? > >> > >>The chelsio rx path really wants to be coalescing rx buffers, which > >>the netmap API currently doesn't support. I've no idea if luigi has > >>plans to add that. So, it has the hilarious side effect of "adding > >>more RX queues" translates to "drops in RX performance." :( > >> > >>Thanks, > > > > hello, > > > > I am sorry, are you saying intel and chelsio distribute RX packet load > differently? If I am not mistaken intel will distributed traffic among > queues based on ip addresses flow/hashes/whatever, does chelsio make it per > packet or somethig other? > > > > I think there are several orthogonal issues here: > - traffic distribution has been discussed by Adrian > so please look at the email he just sent; > > - when you use netmap on a single queue ie netmap:ix0-X > the software side is as efficient as it can, as it needs > to check the status of a single queue on poll() or ioctl(..RXSYNC..). > On the contrary, when you access netmap:if0 (i.e. all > queues on a single file descriptor) every system call > has to check all the queues so you are better off with > a smaller number of queues. > > - on the hardware side, distributing traffic to multiple RX queues > has also a cost that increases with the number of queues, as the > NIC needs to update the ring pointers and fetch buffers for > multiple queues, and you can easily run out of PCIe bandwidth for > these transactions. This affects all NICs. > Some (ix ?) have parameters to configure how often to update the rings > and fetch descriptors, mitigating the problem. Some (ixl) don't. > > My opinion is that you should use multiple queues only if you want > to rely on hw-based traffic steering, and/or your workload is > bottlenecked by the CPU rather than bus I/O bandwidth. Even so, > use as few queues as possible. > > Sometimes people use multiple queues to increase the number of > receive buffers and tolerate more latency in the software side, but > this really depends on the traffic distribution, so in the worst case > you are still dealing with a single ring. > > Often you are better off using a single hw queue and have a > process read from it using netmap and demultiplex to different > netmap pipes (zero copy). That reduces bus transactions. > > Another option which I am experimenting these days is forget about > individual packets once you are off the wire, and connect the various > processes in your pipeline with a stream (TCP or similar) where packets > and descriptors are back to back. CPUs and OSes are very efficient in > dealing with streams of data. > > cheers > luigi > > > > Another motivation would be to have more > > Often you are better off d > CPU limited. on multiqueue is that you should use it only if your workload > Thanks for the explanation. What i was trying to achieve is more performance using more than one CPU to actually bridge at line rate, since I have several cores (16 cores) but low CPU clock, and growing up horizontally is easier and cheaper than getting faster CPUs. I though that using 2 queues with two bridge instances or two threads (adrian's bridge) and using that on one of the other 15 idle cores would just allow me to grow from 9Mpps to 14Mpps. It looks like it's not that simple. If I was a developer making a multithreaded netmap application to increase pps rates, is there any other/better strategy than using multiple queues? Should I distribute the load along the application reading and writing to one single Q or something better? I mean, a multithreaded bridge to check how many packets we have in the queue and distributing a constant number of packets to each thread, is it possible/efficient? If I don't have the constant number of packets I should be able to see TAIL via netmap, right? Thank you all for all the details on how things actually work. -- =========== Eduardo Meyer pessoal: dudu.meyer@gmail.com profissional: ddm.farmaciap@saude.gov.br