From owner-freebsd-net@freebsd.org Sat Aug 6 18:03:38 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26A36BAFADC for ; Sat, 6 Aug 2016 18:03:38 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A27A1D61 for ; Sat, 6 Aug 2016 18:03:37 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u76I3WJs044829 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sat, 6 Aug 2016 20:03:33 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u76I3OeQ012899 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Sun, 7 Aug 2016 01:03:24 +0700 (KRAT) (envelope-from eugen@grosbein.net) To: "freebsd-net@freebsd.org" From: Eugene Grosbein Subject: 40Gbps http client benchmark Message-ID: <57A62668.7020309@grosbein.net> Date: Sun, 7 Aug 2016 01:03:20 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Aug 2016 18:03:38 -0000 Hi! Is there any high performance benchmark acting as http client for outer http server capable to receive 40Gbps without overwhelming CPU with insane number of syscalls? I've tried benchmarks/wrk version 4.0.2 and it works just fine upto 20Gbps for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection combined to single lagg interface (lagghash l4). But each worker pthread of wrk generates too many kqueue() system calls polling for incoming data and eats 100% of its CPU core and cannot receive more. Or, it may be some kqueue() kernel level lock contention, I do not know. More worker threads, more overloaded CPU cores, no increase of transfer over about 20Gbps. I transfer 1MByte-sized files for the benchmark (tried 8MB-sized too). I set sysctl net.inet.tcp.recvspace=4194304 (4MB). I've even patched wrk to to use read buffer sized 4MB instead of its compiled-in default 8k, as it does not change default socket options other than TCP_NODELAY: --- src/wrk.h.orig 2016-08-06 23:20:16.205906000 +0700 +++ src/wrk.h 2016-08-06 23:20:20.460579000 +0700 @@ -18,7 +18,7 @@ #include "ae.h" #include "http_parser.h" -#define RECVBUF 8192 +#define RECVBUF 4194304 #define MAX_THREAD_RATE_S 10000000 #define SOCKET_TIMEOUT_MS 2000 Nothing helps to decrease consumed system time and ktrace(1) assures that's because of kevent() calls.