From owner-freebsd-net@freebsd.org Sat Aug 6 19:05:22 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D651BBB0947 for ; Sat, 6 Aug 2016 19:05:22 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 623581782 for ; Sat, 6 Aug 2016 19:05:21 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u76J5HTf044988 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sat, 6 Aug 2016 21:05:17 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u76J5CnX013454 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Sun, 7 Aug 2016 02:05:12 +0700 (KRAT) (envelope-from eugen@grosbein.net) Subject: Re: 40Gbps http client benchmark To: "freebsd-net@freebsd.org" References: <57A62668.7020309@grosbein.net> From: Eugene Grosbein Message-ID: <57A634E5.9060702@grosbein.net> Date: Sun, 7 Aug 2016 02:05:09 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <57A62668.7020309@grosbein.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Aug 2016 19:05:22 -0000 07.08.2016 1:03, Eugene Grosbein пишет: > Hi! > > Is there any high performance benchmark acting as http client for outer http server > capable to receive 40Gbps without overwhelming CPU with insane number of syscalls? > > I've tried benchmarks/wrk version 4.0.2 and it works just fine upto 20Gbps > for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz > with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection > combined to single lagg interface (lagghash l4). > > But each worker pthread of wrk generates too many kqueue() system calls > polling for incoming data and eats 100% of its CPU core and cannot receive more. > Or, it may be some kqueue() kernel level lock contention, I do not know. > More worker threads, more overloaded CPU cores, no increase of transfer over about 20Gbps. Hmm, it seems that's not number of system calls that hurts me but some kernel-level problem. This is NUMA system running under 10.3-STABLE r303291. It has two NUMA domains with first physical CPU (cores 0-7) and first dual-port ix adapter belonging to one domain and second CPU (cores 8-11) and second dual-port adatper belonging to another domain: http://www.grosbein.net/img/r4.svg Just prepend of "cpuset -l 0-7" to wrk invocation boots transfer upto nearly 30Gbps without any other changes. That's strange.