From owner-freebsd-net@FreeBSD.ORG Mon Jul 28 08:08:34 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:1900:2254:206a::19:2]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 21989C7B; Mon, 28 Jul 2014 08:08:34 +0000 (UTC) Received: from butcher-nb.yandex.net (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) by mx2.freebsd.org (Postfix) with ESMTP id 5500828BA; Mon, 28 Jul 2014 08:08:31 +0000 (UTC) Message-ID: <53D604EA.2000203@FreeBSD.org> Date: Mon, 28 Jul 2014 12:08:10 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: John Jasen , FreeBSD Net , "Alexander V. Chernikov" Subject: Re: fastforward/routing: a 3 million packet-per-second system? References: <53CE80DD.9090109@gmail.com> <53D0D0B2.6080600@yandex.ru> <53D10055.1050304@gmail.com> In-Reply-To: <53D10055.1050304@gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 08:08:34 -0000 On 24.07.2014 16:47, John Jasen wrote: > On 07/24/2014 05:24 AM, Andrey V. Elsukov wrote: >> On 22.07.2014 19:18, John Jasen wrote: >>> Feedback and/or tips and tricks more than welcome. >>> >>> Outstanding questions: >>> >>> Would increasing the number of processor cores help? >> AFAIR, increasing the number of cores will lead to worse results. >> With patched and tuned FreeBSD we able to route (with fastforwarding) >> about 7 Mpps IPv4 and 2.5Mpps IPv6. But the stock system is far from >> even half of this results. >> > > Increasing the physical CPU count can (and probably will) result in > performance degradation. However, from what I've seen, balancing IRQs > across the cores on a single physical CPU seems to help. > > I am curious as well, as to how you achieved 7 Mpps. Can you share the > system specs, the patches and tuning? Usually the systems have 2x8 cores (e.g. Intel Xeon E5-2660), 16-32G RAM, 2 ixgbe(4) network cards (82599EB). We slowly trying merge our patches to FreeBSD, but most of them are hackish and needs rethinking. We need reduce lock contention in routing code and optimize structures to fit cache lines. Some of patches were published here by melifaro@. Some are already committed. Also v6-related patches included in my svn branch user/ae/inet6. Tuning depends from your hardware. It is good to pin network adapter's taskq threads and interrupts. andre@ committed this functional recently in head/. -- WBR, Andrey V. Elsukov