From owner-freebsd-net@freebsd.org Mon Jan 23 15:47:38 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4A10CBEFB9 for ; Mon, 23 Jan 2017 15:47:38 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from out.smtpout.orange.fr (out03.smtpout.orange.fr [193.252.22.212]) by mx1.freebsd.org (Postfix) with ESMTP id 4037D3A0 for ; Mon, 23 Jan 2017 15:47:37 +0000 (UTC) (envelope-from olivier@freebsd.org) Received: from mail-yw0-f178.google.com ([209.85.161.178]) by mwinf5d52 with ME id brfx1u00x3rEd8Z03rfy6i; Mon, 23 Jan 2017 16:39:59 +0100 X-ME-Helo: mail-yw0-f178.google.com X-ME-Auth: Y29jaGFyZC1sYWJiZS5vbGl2aWVyQG9yYW5nZS5mcg== X-ME-Date: Mon, 23 Jan 2017 16:39:59 +0100 X-ME-IP: 209.85.161.178 Received: by mail-yw0-f178.google.com with SMTP id w75so138937118ywg.1; Mon, 23 Jan 2017 07:39:58 -0800 (PST) X-Gm-Message-State: AIkVDXJDNMef4P9MiFXpXnLBLQ6aVAMGk2ySpbYdo4aiyc3NJlz4zCnu0trJ3r4zr3UWzIr5mX4lRig99kNTtA== X-Received: by 10.55.151.199 with SMTP id z190mr26400127qkd.166.1485185997280; Mon, 23 Jan 2017 07:39:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.200.49.99 with HTTP; Mon, 23 Jan 2017 07:39:36 -0800 (PST) In-Reply-To: <159902b73ed.10775291e21533.7488368455500235608@nextbsd.org> References: <30f21c75-d3a2-edcd-1999-d5ed9f970c06@freebsd.org> <1598d97bf2a.c6bcb76838987.6501340920645175463@nextbsd.org> <574a7ac7-4842-9518-8286-a4d89a9f7a27@freebsd.org> <6c6cb534-73c7-464b-8af1-7445a9c0188c@freebsd.org> <1598f29d379.ea6360351471.8752933472741761813@nextbsd.org> <1598f3f8588.d20017893749.339651164872952258@nextbsd.org> <1598f42ad77.eeec05be4113.9201780237587761460@nextbsd.org> <159902b73ed.10775291e21533.7488368455500235608@nextbsd.org> From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Date: Mon, 23 Jan 2017 16:39:36 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending To: Matthew Macy Cc: Sean Bruno , "freebsd-net@freebsd.org" , "freebsd-current@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jan 2017 15:47:39 -0000 On Thu, Jan 12, 2017 at 1:54 AM, Matthew Macy wrote: > > A flame graph for the core cycle count and a flame graph with cache > miss stats from pmc would be a great start. > > > > > > =E2=80=8BI didn't know the exact event name to use for cache miss stat= s, but > here are the flame graphs for CPU_CLK_UNHALTED_CORE: > > http://dev.bsdrp.net/netgate.r311848.CPU_CLK_UNHALTED_CORE.svg > > http://dev.bsdrp.net/netgate.r311849.CPU_CLK_UNHALTED_CORE.svg > > Thanks. Having twice as many txqs would definitely help. It's also clear > that there may be some sort of peformance issue in iflib_txq_drain. > Although it could just be non-stop cache misses on the packet headers. > > > =E2=80=8BAny news about the performance issue in iflib_txq_drain ? On a different hardware (PC Engine APU2), I've got -20% performance drop: x head r311848: packets per second + head r311849: packets per second +--------------------------------------------------------------------------= + | ++ x= | |+++ x xx x= | | |_A_|= | ||A| = | +--------------------------------------------------------------------------= + N Min Max Median Avg Stddev x 5 580021 588650 585676 585406.1 3550.8673 + 5 463865 467599 465428 465638.6 1437.9347 Difference at 95.0% confidence -119768 +/- 3950.78 -20.4589% +/- 0.558328% (Student's t, pooled s =3D 2708.9) =E2=80=8B =E2=80=8BBecause it's an AMD processor I didn't found the pmc equivalent of CPU_CLK_UNHALTED_CORE, then I've used BU_CPU_CLK_UNHALTED but I've no idea if it's the good one. http://dev.bsdrp.net/apu2.r311848.BU_CPU_CLK_UNHALTED.svg http://dev.bsdrp.net/apu2.r311849.BU_CPU_CLK_UNHALTED.svg =E2=80=8B =E2=80=8BThanks=E2=80=8B