From owner-freebsd-net@freebsd.org  Fri Jan  8 11:02:33 2016
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3388EA686A5
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Fri,  8 Jan 2016 11:02:33 +0000 (UTC)
 (envelope-from brde@optusnet.com.au)
Received: from mail107.syd.optusnet.com.au (mail107.syd.optusnet.com.au
 [211.29.132.53]) by mx1.freebsd.org (Postfix) with ESMTP id B96A5162E
 for <freebsd-net@freebsd.org>; Fri,  8 Jan 2016 11:02:32 +0000 (UTC)
 (envelope-from brde@optusnet.com.au)
Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au
 (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197])
 by mail107.syd.optusnet.com.au (Postfix) with ESMTPS id 26BF4D48AB6;
 Fri,  8 Jan 2016 22:02:27 +1100 (AEDT)
Date: Fri, 8 Jan 2016 22:02:24 +1100 (EST)
From: Bruce Evans <brde@optusnet.com.au>
X-X-Sender: bde@besplex.bde.org
To: Adrian Chadd <adrian.chadd@gmail.com>
cc: Mark Delany <c2h@romeo.emu.st>, FreeBSD Net <freebsd-net@freebsd.org>
Subject: Re: Does FreeBSD have sendmmsg or recvmmsg system calls?
In-Reply-To: <CAJ-VmonYPhcN-gikuYQU_k5GaTAqTijoxR_0ORV4BZqsHMRJSg@mail.gmail.com>
Message-ID: <20160108204606.G2420@besplex.bde.org>
References: <20160104101747.58347.qmail@f5-external.bushwire.net>
 <20160104194044.GD3625@kib.kiev.ua>
 <20160104210741.32812.qmail@f5-external.bushwire.net>
 <CAP=KkTwfpjec2Tgnm4PRR3u8t4GEqN9Febm5HRcqapifBG-B6g@mail.gmail.com>
 <CA+hQ2+h4NNz9tgSpjJdv7fXteq5tAR7o3LvjV=u08NHjRLPwmA@mail.gmail.com>
 <CAP=KkTzFUDsZwDDLD3n97xJW0qLVZMPduZGSX+eXC3UuLpVjMg@mail.gmail.com>
 <20160107161213.GZ3625@kib.kiev.ua>
 <CA+hQ2+g6OB3MmZrW5hzNSnkcqKaKf1XGDraHfWXtSrowxKuL5g@mail.gmail.com>
 <20160107192840.GF3625@kib.kiev.ua> <20160108172323.W1815@besplex.bde.org>
 <20160108075815.3243.qmail@f5-external.bushwire.net>
 <CAJ-VmonYPhcN-gikuYQU_k5GaTAqTijoxR_0ORV4BZqsHMRJSg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
X-Optus-CM-Score: 0
X-Optus-CM-Analysis: v=2.1 cv=PfoC/XVd c=1 sm=1 tr=0
 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=L9H7d07YOLsA:10
 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10
 a=Cyi0gIR--wBvI8ew5kYA:9 a=CjuIK1q_8ugA:10
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Jan 2016 11:02:33 -0000

On Fri, 8 Jan 2016, Adrian Chadd wrote:

> On 7 January 2016 at 23:58, Mark Delany <c2h@romeo.emu.st> wrote:
>> On 08Jan16, Bruce Evans allegedly wrote:
>>> If the NIC can't reach line rate
>>
>>> Network stack overheads are also enormous.
>>
>> Bruce makes some excellent points.
>>
>> I challenge anyone to get line rate UDP out of FBSD (or Linux) for a
>> 1G NIC yet alone a 10G NIC listening to a single port. It was exactly
>> my frustration with UDP performance that led me down the path of
>> *mmsg() and netmap.
>>
>> Frankly this is an opportunity for FBSD as UDP performance appears to
>> be a neglected area.
>
> I'm there, on 16 threads.
>
> I'd rather we do it on two or three, as a lot of time is wasted in
> producer/consumer locking. but yeah, 500k tx/rx should be doable per
> CPU with only locking changes.

Line rate for 1 Gbps is about 1500 kpps (small packets).

With I218V2 (em), I see enormous lock contention above 3 or 4 (user)
threads, and 8 are slightly slower than 1.  1 doesn't saturate the NIC,
and 2 is optimal.

Bruce