From owner-freebsd-net@freebsd.org Mon Aug 8 01:57:51 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CB8DBA7AFA for ; Mon, 8 Aug 2016 01:57:51 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from mail-ua0-x234.google.com (mail-ua0-x234.google.com [IPv6:2607:f8b0:400c:c08::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF1051EB8 for ; Mon, 8 Aug 2016 01:57:50 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by mail-ua0-x234.google.com with SMTP id 35so236453130uap.1 for ; Sun, 07 Aug 2016 18:57:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=HGvkg2zBBAlaraGBrYp4CbnMXFVdgl30YzMfWx6xyHQ=; b=NFktKtI/XL3Z9+/4r5BMNooZzykmGedpFAuLSr2m0KwahBn/VTkLAOY8tKP4MqxFvX zt+yeXxwnsd+gaMotWpwOvwmVYP7s4til7uBOjd/66pbBpBYodSp2Ll11U1WMgZ36BN+ ZQq97q3cXnANpiTSUkDHsce/x8zWGN0lcUeet4QddL9CoQyI5lkSUVfdI/K5E0MvuK3F 2vAueEDa6J1GeayMQvytsE6Ahi4d4HA6jLQ2bsdaXhFLESMPG3hqPNx5qojJ959JF2q2 vSfmUi6piBxEDS/HBbNBJ36msFs0pAd1M8GGZlzsT5FTgigC1TKRn6vhlefX5OLTjop/ Kyrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=HGvkg2zBBAlaraGBrYp4CbnMXFVdgl30YzMfWx6xyHQ=; b=AdD5jvUOPg0T8ANw2JKsCs+Y2am0n5OuljzmBZWBVgGt/GzNFY/ScUo166iRiPI0mh no60NHT/bFv968SGujBlcmIEEtxm1RFKF33+RhCNbkpSE61VmBYgKwd9tJOadf5U5XtT lSnc5Or6012B7v8pdM8oojA36Igq6Q3UUsf9WBy6E4uNJ69tDgNSu/MKi73KZNXZnYBj pbwaonLEl7imqWwYpnAuucaDyBE66qMEfHtMWRo49aoxfv5MXoexf6ppTUgP8oWu2b7Y boLgLNr4BwzgogCGbVhk/wQ1GKO7Ad062YQarVVQAGxjRUJYbCuGAExwuxcYeBnftMYc dDfw== X-Gm-Message-State: AEkoouupCzcC77cwfZ6U43FiE3DgywrG5/r/jKzZ29mD1ptFegAz/rgftt7/Fsv4UAsWFmpaMMWSCKFSlMkh7w== X-Received: by 10.159.38.47 with SMTP id 44mr1111540uag.25.1470621469782; Sun, 07 Aug 2016 18:57:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.1.54 with HTTP; Sun, 7 Aug 2016 18:57:49 -0700 (PDT) In-Reply-To: <57A62668.7020309@grosbein.net> References: <57A62668.7020309@grosbein.net> From: Sepherosa Ziehau Date: Mon, 8 Aug 2016 09:57:49 +0800 Message-ID: Subject: Re: 40Gbps http client benchmark To: Eugene Grosbein Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Aug 2016 01:57:51 -0000 We have an optimized wrk here: https://github.com/sepherosa/wrk It greatly reduces the # of kqueue syscalls and avoids unnecessary setsockopt etc. BTW, how many concurrent connections and threads are you testing w/? Did you reduce the MSL on your client machines? Default local port range probably should be ok, but it still worth checking. Thanks, sephe On Sun, Aug 7, 2016 at 2:03 AM, Eugene Grosbein wrote: > Hi! > > Is there any high performance benchmark acting as http client for outer http > server > capable to receive 40Gbps without overwhelming CPU with insane number of > syscalls? > > I've tried benchmarks/wrk version 4.0.2 and it works just fine upto 20Gbps > for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 v3 @ > 2.40GHz > with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection > combined to single lagg interface (lagghash l4). > > But each worker pthread of wrk generates too many kqueue() system calls > polling for incoming data and eats 100% of its CPU core and cannot receive > more. > Or, it may be some kqueue() kernel level lock contention, I do not know. > More worker threads, more overloaded CPU cores, no increase of transfer over > about 20Gbps. > > I transfer 1MByte-sized files for the benchmark (tried 8MB-sized too). > I set sysctl net.inet.tcp.recvspace=4194304 (4MB). > I've even patched wrk to to use read buffer sized 4MB instead of its > compiled-in > default 8k, as it does not change default socket options other than > TCP_NODELAY: > > --- src/wrk.h.orig 2016-08-06 23:20:16.205906000 +0700 > +++ src/wrk.h 2016-08-06 23:20:20.460579000 +0700 > @@ -18,7 +18,7 @@ > #include "ae.h" > #include "http_parser.h" > > -#define RECVBUF 8192 > +#define RECVBUF 4194304 > > #define MAX_THREAD_RATE_S 10000000 > #define SOCKET_TIMEOUT_MS 2000 > > Nothing helps to decrease consumed system time and ktrace(1) assures > that's because of kevent() calls. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Tomorrow Will Never Die