From owner-freebsd-net@freebsd.org Wed Mar 8 12:25:59 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 137EED022F1 for ; Wed, 8 Mar 2017 12:25:59 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-oi0-x22c.google.com (mail-oi0-x22c.google.com [IPv6:2607:f8b0:4003:c06::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D21261177 for ; Wed, 8 Mar 2017 12:25:58 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-oi0-x22c.google.com with SMTP id m124so17729528oig.1 for ; Wed, 08 Mar 2017 04:25:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=MPaeUAslqWkN3sAyZQBjftr9toXHigH7tAXnkcJ9dLo=; b=MFW4UxZnPz/OlD+mSd8biY/oEfeyCT3FhSp+Le0t12G/0onC9OBrSv+AEZHWZf0ILv F/rI1Pa0GYMbSMbT/azD9jJznURJ6Cc6GCEegcUtsDQOYMoaG4AXVNWESarTP1plxbOw aS8XzVGQH9b7vx8PH4djvbvAUv4QqbXDPf/nU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=MPaeUAslqWkN3sAyZQBjftr9toXHigH7tAXnkcJ9dLo=; b=UCVnY5C0iM3rdyexg2+BJSPJiCCSeyMYs9/fX04Q2KloLIKPm+/HC/pLjjnzWcv14B JqlrKjl08TCSMUzWU618a3PcTMpzcpCr6O+Cev7Wd88WzcpAjCx2GFpyBwai7xaBAEb4 WlMcpa77G1eqT09qWDk4qk9emEEMNuiqEx1c0zJc4sGNnc1Hq5KkF4xMgVCxpJAtmMEp pSWPD2JjatgMm3CJ5Xz3wLOJb3YmHHs7oNWFi6GdgXHQXYEKTx3l59oZR5xzQRk8mxNj Wbr8HbvKV2qzIZtdJwSxo8t4o3TdIctCE7zmcTBLGKxyJH2HeBFO9UqD0HwpcH3kOlQC NbXw== X-Gm-Message-State: AMke39mLTUiUoA6aFY7wbq3rRmJyQ2vIaGVds1liH0SUoydH5ShUrpRFsjYIy/ISHIRDWbn53djAu8HbuDAOHw== X-Received: by 10.202.105.195 with SMTP id e186mr3392878oic.160.1488975957926; Wed, 08 Mar 2017 04:25:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.23.208 with HTTP; Wed, 8 Mar 2017 04:25:57 -0800 (PST) In-Reply-To: References: From: Kevin Bowling Date: Wed, 8 Mar 2017 05:25:57 -0700 Message-ID: Subject: Re: about that DFBSD performance test To: "Eugene M. Zheganin" Cc: freebsd-net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Mar 2017 12:25:59 -0000 Right off the bat, FreeBSD doesn't really understand NUMA in any sufficient capacity. Unfortunately at companies like the one I work at, we take that to mean "OK buy a high bin CPU and only populate one socket" which serves us well and may ultimately be the best value but does nothing to address the reality that multi-socket NUMA systems are common place and FreeBSD should run in the same league on them as other operating systems to be taken seriously. I'd be interested in seeing how the contenders look by removing one of the CPUs as it might at least put a spotlight on that issue. With respect to SO_REUSEPORT, we are investigating implementing this round robin behavior right now. We think librss is a better way to go for content serving, but software will get written with Linux APIs in mind so I think it'll be a good idea to have a distributing SO_REUSEPORT anyway for software that does not understand librss natively. I have no commercial need for kernel IP forwarding but Matt Macy is doing a lot of driver work for me and we are at least trying to keep it in line with the legacy drivers. Since sephe's test was ixgbe, it'd be interesting to test https://github.com/mattmacy/networking/commits/HEAD_MERGE/iflib-ixgbe-current which makes the FreeBSD ixgbe driver much more like cxgbe (same ring code, improved coalescing, various prefetching insights for 10g+). In talking with Matt there's probably some low hanging fruit at the driver layer for this workload around batching and distribution. I do think the use of polling(4) in the benchmark is a bit hacky, it should be possible to get similar performance to Linux by doing batching and distribution as we are fine tuning in the iflib project. I'd be looking at netmap before polling(4) in any new product development that does heavy forwarding. But there's probably more to do up stack in routing and I don't know of anyone working on it with significant commercial backing. Ultimately I have to say congrats to DFBSD. They pulled that performance off with a small team and very little to no commercial backing. FreeBSD has a lot of full time commercial contributors, and while this is generally a good thing for longevity, I think workplace culture (even/especially at fabled places, that just means you're super busy) defeats some of the lucid insight and tenacity that ardent personal contributors are able to pull off of their own accord. Regards, On Tue, Mar 7, 2017 at 9:00 PM, Eugene M. Zheganin wrote: > Hi. > > Some have probably seen this already - http://lists.dragonflybsd.org/ > pipermail/users/2017-March/313254.html > > So, could anyone explain why FreeBSD was owned that much. Test is split > into two parts, one is nginx part, and the other is the IPv4 forwarding > part. I understand that nginx ownage was due to SO_REUSEPORT feature, which > we do formally have, but in DFBSD and Linux it does provide a kernel socket > multiplexor, which eliminates locking, and ours does not. I have only found > traces of discussion that DFBSD implementation is too hackish. Well, > hackish or not, but it's 4 times faster, as it turns out. The IPv4 > forwarding loss is pure defeat though. > > Please not that although they use HEAD it these tests, they also mention > that this is the GENERIC-NODEBUG kernel which means this isn't related to > the WITNESS stuff. > > Please also don't consider this trolling, I'm a big FreeBSD fan through > the years, so I'm asking because I'm kind of concerned. > > Eugene. > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >