From owner-freebsd-net@freebsd.org  Mon Aug  8 12:19:18 2016
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECF97BA9265
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Mon,  8 Aug 2016 12:19:18 +0000 (UTC)
 (envelope-from eugen@grosbein.net)
Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified))
 by mx1.freebsd.org (Postfix) with ESMTPS id 800441CB3
 for <freebsd-net@freebsd.org>; Mon,  8 Aug 2016 12:19:17 +0000 (UTC)
 (envelope-from eugen@grosbein.net)
Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221])
 by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u78CJ6Wl054514
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT);
 Mon, 8 Aug 2016 14:19:07 +0200 (CEST)
 (envelope-from eugen@grosbein.net)
X-Envelope-From: eugen@grosbein.net
X-Envelope-To: sepherosa@gmail.com
Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1])
 by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id u78CIuMs028253;
 Mon, 8 Aug 2016 19:18:56 +0700 (KRAT)
 (envelope-from eugen@grosbein.net)
Subject: Re: 40Gbps http client benchmark
To: Sepherosa Ziehau <sepherosa@gmail.com>
References: <57A62668.7020309@grosbein.net>
 <CAMOc5cyHsLZ93R9jQncBgReii0nCTnW0ZoBFbA4RkHpTcwHcnw@mail.gmail.com>
Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>
From: Eugene Grosbein <eugen@grosbein.net>
Message-ID: <57A878B0.9040004@grosbein.net>
Date: Mon, 8 Aug 2016 19:18:56 +0700
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101
 Thunderbird/38.4.0
MIME-Version: 1.0
In-Reply-To: <CAMOc5cyHsLZ93R9jQncBgReii0nCTnW0ZoBFbA4RkHpTcwHcnw@mail.gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Spam-Status: No, score=3.3 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_96_Q, 
 LOCAL_FROM autolearn=no version=3.3.2
X-Spam-Report: * 3.0 DATE_IN_FUTURE_96_Q Date: is 4 days to 4 months after
 Received: date
 * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1%
 *      [score: 0.0000] *  2.6 LOCAL_FROM From my domains
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net
X-Spam-Level: ***
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 08 Aug 2016 12:19:19 -0000

On 08.08.2016 08:57, Sepherosa Ziehau wrote:
> We have an optimized wrk here:
> https://github.com/sepherosa/wrk

Thank you, I'll try it.

> It greatly reduces the # of kqueue syscalls and avoids unnecessary
> setsockopt etc.  BTW, how many concurrent connections and threads are
> you testing w/?  Did you reduce the MSL on your client machines?
> Default local port range probably should be ok, but it still worth
> checking.

I use default port ranges and get best results with command like this:

cpuset -l 0-7 wrk -t8 -c60 -d60s http://x.x.x.x/index

This way, I get 4.38GB out of 4x10G lagg (lagghash l4), that is about 36Gbps.

It seems, uneven lagg load balancing prevents it from reaching higher numbers.
I tried to increase connection number from -c60 to large values but then
unpatched wrk overhelms CPU with syscalls.