From owner-freebsd-net@freebsd.org Wed Jan 20 18:46:43 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EAB74A8A28B for ; Wed, 20 Jan 2016 18:46:42 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4C8F19FA for ; Wed, 20 Jan 2016 18:46:42 +0000 (UTC) (envelope-from pavel.odintsov@gmail.com) Received: by mail-io0-x236.google.com with SMTP id g73so30056256ioe.3 for ; Wed, 20 Jan 2016 10:46:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vUx5L7ZidlXG7AAgjvdfrRMvmi+7Y+dWjBQDN6ESZOU=; b=kHIKpUNr9qV+AZeiX7BUxu2pZLwODBciTm2izX95/DkKWSleJXKRHCKvI53C7s5Pao z1I5zUo0OFAPJRmYaK9h5jFFUZjvY/AoEy9PC/ojVo/3Qc6tsaSdzbGHN85c6N+yCnGG Pnlx0abme2s9Tgo4bSXYTxzChi68Vu7rQxQ8WGye2TmKpUKguW8wwnt7gcC4BtdlhhwX ZtndTrr7UxNqFKE4CVq3TMFZOAGetdn9vw5DcpSbpt7NHOFApN0JjSvqx9/wQS2Rlflk 7h5qx3/Ck6mXk5ON6HH9pwbB7BLXAztbFkkOSC82pUPQVJ90s5onRlOYzqjaiYEBfE/O 8a4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=vUx5L7ZidlXG7AAgjvdfrRMvmi+7Y+dWjBQDN6ESZOU=; b=Z1bh7yva7sBik2iBc+HLcYa6H/EuLOxTTGE2qlLU6FbL40opRMq8C+uOd4MM7hrYCN QTtqKFFvquCxAML/QeRwXp2KrgS0XuN3n/qjgqJRsN6cUxx2wxxwWUXMX+moB7pfgzrB pttxTFMMPAUb53I+h+YSIsZU5fCdU5rsIh6smaFD5dIkAHelMr2wdOvnY6wU1daewjyw GyscJU+FQsA93hMZ5gXhgNoyrCCGLSkassO4/XM00MQrn2osCAnPx4fsgsxBGBwFAKIr An0fxiVRGvVvDDuc5RC9VgpAM3Ib+yYkiSKXiPArhomct1S3Nw/4shSGwsYrLcitNuy4 WvtA== X-Gm-Message-State: ALoCoQkH8prU0DJjl6eqlv/EYn9NvDkCd9YnQvnubKvHwk+b0nnd0h/30B5yhfa3BCSomuCLQfZ21wNlm2+je0dt6BGPq9vHyQ== MIME-Version: 1.0 X-Received: by 10.107.16.95 with SMTP id y92mr37995071ioi.194.1453315602114; Wed, 20 Jan 2016 10:46:42 -0800 (PST) Received: by 10.79.36.65 with HTTP; Wed, 20 Jan 2016 10:46:42 -0800 (PST) In-Reply-To: References: Date: Wed, 20 Jan 2016 21:46:42 +0300 Message-ID: Subject: Re: netmap design question - accessing netmap:X-n individual queues on FreeBSD From: Pavel Odintsov To: Eduardo Meyer Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jan 2016 18:46:43 -0000 Hi Yes, this approach working really well on Linux. But I have never tried to do same on FreeBSD. I'm using similar approach in dastnetmon abd read data from the network card in X threads where each thread assigned to physical queue. So for Linux you should use my custom (based on Intel's drivers from Sourceforge with netmap patches) vbecause vanilla drivers haven't support for multi queue mode. On Wednesday, 20 January 2016, Eduardo Meyer wrote: > Hello all, > > I have some doubts regarding netmap design direct queue usage. > > If open netmap:ix0 I am opening all 0-7 queues. Are those queues FIFO among > themselves? I mean first packeds will be available on netmap:ix0-0 and if > this queue fills up the next packets will be on netmap:ix0-1, and via > netmap:ix0 I have all queues from 0 to 7, is this understanding correct? > > Or something else happens, like, only 1 queue is used if I open netmap:ix0? > > A later question, if I open netmap:ix0-0 and nothing else is it supposed to > work? > > On my tests I can see that "pkt-gen -f tx -i ix0-0" will work, but "pkt-gen > -f rx -i ix0-0" will not. I can transmit but cant receive on a given queue. > Why is that? And how could I make something like this, work (code change > required?): > > bridge -i netmap:ix0-0 -i netmap:ix1-0 > > Or should it already work? > > Mr Pavel Odiltsov, the author from fastnetmon mentioned he can run on Linux > and it works: > > "kipfw netmap:eth0-n netmap:eth1-n" > > But this or the above bridge example won't work on FreeBSD. Is that any > different? (I did not try on Linux). > > I could also notice performance differences I would like to understand, if > I run: > > pkt-gen -i ix0 -f tx -s 192.168.0.2 -d 192.168.0.1 > > I have 14.8Mpps (like rate). > > If I run: > > pkt-gen -i ix0-1 -f tx -s 192.168.0.2 -d 192.168.0.1 > > I can have only 11Mpps. In fact I have 11Mpps if I run on ix0-2, ix0-3, ... > ix0-7. > > I can understand if I run all queues I can have better pps rates than only > one single queue, sure, however if I run: > > pkt-gen -i ix0-0 -f tx -s 192.168.0.2 -d 192.168.0.1 > > I also have 14.8Mpps. So yeah, ix0 or ix0-0 both give me 14.8Mpps while any > other queue give me 11Mpps. How should I understand this? > > I am asking this because I want to hack (for learning) into the bridge code > to make it multithreaded, and I want to have a thread opening each one of > the 8 available queues allocated on each one of my 8 CPUs. > > However both opening ix0-1 and ix1-1 on source code or as a parameter to > the bridge application, I can't have it working. > > I also looked on "pkt-gen -p 8 -c 8" and although debug shows I have 8 > threads on 8 CPU with 8 queues I still only see 1 thread working, all other > threads go IDLE. I expected to see at least 2 threads working, say, for > ix0-0 and ix0-1 to fill line rate. > > Thank you in advance. > > -- > =========== > Eduardo Meyer > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org > " > -- Sincerely yours, Pavel Odintsov