From owner-svn-src-head@freebsd.org Mon Aug 3 13:06:22 2015 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 620B79B25BB; Mon, 3 Aug 2015 13:06:22 +0000 (UTC) (envelope-from cochard@gmail.com) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC2021F2D; Mon, 3 Aug 2015 13:06:21 +0000 (UTC) (envelope-from cochard@gmail.com) Received: by labow3 with SMTP id ow3so16164314lab.1; Mon, 03 Aug 2015 06:06:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=iOMyHenElxoCyae+96euAY/lKwgXg9l3lfGqpT08OSw=; b=nRBAGnX+ZP+SV0hdLLbaH8bA9ebURKaJfnP7LL0hOK0DtfYFqGQYscre2fCJRZ5ZVZ ZyRwi2Ji6PLslASUHi6mpAuPc6qAfcsSG3k1v+k6mpegJoQPSIr3RQtspjyl1k62fuDz /lIk9p2rwWYcTin1sEfAlTb8fa6Un5QOLg2uRtRbnUAVF57p7R9NrVcqjBMW9fheSnjd LldjcZ0vI2jg48ttLx1Bn7Hyio+qHjZtKFnok1CEnVPxL5Wn27OcVlHukiFfqBh463y3 3QBdAroPKPhxtqufpkkyYkO/4y23qzBrXPH5hHksHegzcsEV9r8P+8Nrmg4NRE/hmjzb LSyQ== X-Received: by 10.112.144.69 with SMTP id sk5mr16793734lbb.6.1438607179582; Mon, 03 Aug 2015 06:06:19 -0700 (PDT) MIME-Version: 1.0 Sender: cochard@gmail.com Received: by 10.152.115.140 with HTTP; Mon, 3 Aug 2015 06:06:00 -0700 (PDT) In-Reply-To: <20150728124220.GW72729@FreeBSD.org> References: <201507021810.t62IAgCc003272@repo.freebsd.org> <20150728124220.GW72729@FreeBSD.org> From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Date: Mon, 3 Aug 2015 15:06:00 +0200 X-Google-Sender-Auth: -7vzaOBPz8xukJkEsV3qIFMZqvc Message-ID: Subject: Re: svn commit: r285051 - head/sys/netinet To: Gleb Smirnoff Cc: =?UTF-8?Q?Ermal_Lu=C3=A7i?= , svn-src-head , svn-src-all , src-committers Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 13:06:22 -0000 On Tue, Jul 28, 2015 at 2:42 PM, Gleb Smirnoff wrote: > > Here is suggested patch. Ermal and Oliver, can you please test/benchmark > it? > =E2=80=8BHi, this patch reduce performanece :-( Here are the results regarding forwarding: x r285046.pps.forwarding (IPSEC compiled but not used) + r285051.pps.forwarding (IPSEC compiled but not used) * r285051-glebius-patched.pps.forwarding (IPSEC compiled but not used) +--------------------------------------------------------------------------= ------+ |xxx xx **** * + +++| ||_M_A___| |_A_| |AM|| +--------------------------------------------------------------------------= ------+ N Min Max Median Avg Stddev x 5 397733 406951 399300 401613.8 4324.9755 + 5 478095 482079 480869 480543.6 1666.0282 Difference at 95.0% confidence 78929.8 +/- 4779.72 19.6532% +/- 1.19013% (Student's t, pooled s =3D 3277.27) * 5 424720 430745 427014 427378.4 2351.7439 Difference at 95.0% confidence 25764.6 +/- 5076.98 6.41527% +/- 1.26415% (Student's t, pooled s =3D 3481.1) =E2=80=8B PMC stats during forwarding bench: [root@netgate]/data# pmcannotate pmc.forwarding.out /data/debug/boot/kernel/kernel.symbols CONVERSION STATISTICS: #samples/total 33880 Profile trace for function: __rw_rlock() [6.29%] Profile trace for function: ip_forward() [4.68%] Profile trace for function: ip_output() [4.64%] Profile trace for function: binuptime() [4.05%] Profile trace for function: igb_mq_start_locked() [3.79%] Profile trace for function: igb_rxeof() [3.46%] Profile trace for function: tsc_get_timecount_low_lfence() [3.25%] Profile trace for function: ether_output() [3.03%] Profile trace for function: rtalloc1_fib() [2.77%] Profile trace for function: random_ivy_read() [2.64%] Profile trace for function: _rw_runlock_cookie() [2.64%] Profile trace for function: ether_nh_input() [2.63%] Profile trace for function: ip_input() [2.55%] Profile trace for function: key_allocsp_default() [2.39%] Profile trace for function: igb_mq_start() [2.39%] Profile trace for function: bzero() [2.08%] Profile trace for function: uma_zalloc_arg() [1.95%] Profile trace for function: memcpy() [1.84%] Profile trace for function: _mtx_lock_spin_cookie() [1.83%] Profile trace for function: bcopy() [1.76%] Profile trace for function: random_harvest_queue() [1.63%] Profile trace for function: __mtx_lock_sleep() [1.56%] Profile trace for function: uma_zfree_arg() [1.47%] Profile trace for function: arpresolve() [1.39%] Profile trace for function: in_cksumdata() [1.25%] Profile trace for function: bounce_bus_dmamap_load_buffer() [1.22%] Profile trace for function: bcmp() [1.13%] Profile trace for function: rtalloc_ign_fib() [1.11%] Profile trace for function: rn_match() [1.03%] Profile trace for function: netisr_dispatch_src() [1.03%] Profile trace for function: critical_exit() [1.02%] Profile trace for function: bus_dmamap_load_mbuf_sg() [0.87%] Profile trace for function: spinlock_exit() [0.79%] Profile trace for function: in_cksum_skip() [0.75%] Profile trace for function: ip_ipsec_output() [0.75%] Profile trace for function: acpi_cpu_c1() [0.74%] Profile trace for function: in_broadcast() [0.74%] Profile trace for function: spinlock_enter() [0.74%] Profile trace for function: igb_refresh_mbufs() [0.71%] Profile trace for function: in_lltable_lookup() [0.71%] Profile trace for function: ip_fastforward() [0.68%] Profile trace for function: m_adj() [0.65%] Profile trace for function: ether_demux() [0.65%] Profile trace for function: _key_freesp() [0.61%] Profile trace for function: lockstat_nsecs() [0.60%] Profile trace for function: m_freem() [0.58%] Profile trace for function: critical_enter() [0.56%] Profile trace for function: m_copydata() [0.55%] Profile trace for function: mb_free_ext() [0.54%] Profile trace for function: pmap_kextract() [0.50%] =E2=80=8B =E2=80=8BAnd about fastforwarding: =E2=80=8B x 285046.pps.fastforwarding (IPSEC compiled but not used) + 285051.pps.fastforwarding (IPSEC compiled but not used) * r285051-glebius-patched.pps.fastforwarding (IPSEC compiled but not used) +--------------------------------------------------------------------------= ------+ |* + | |* * * * ++ +x x+ xx x| ||____AM__| |_MA__|__AM_| | +--------------------------------------------------------------------------= ------+ N Min Max Median Avg Stddev x 5 742683 754709 750940 749015.6 4944.9059 + 5 736459 747593 738197 740112.4 4511.1611 Difference at 95.0% confidence -8903.2 +/- 6902.82 -1.18865% +/- 0.921585% (Student's t, pooled s =3D 4733) * 5 638356 652855 646668 645263 6599.8604 Difference at 95.0% confidence -103753 +/- 8504.75 -13.8519% +/- 1.13546% (Student's t, pooled s =3D 5831.39) And pmcstat during fastforwarding: [root@netgate]/data# pmcannotate pmc.fastforwading.out /data/debug/boot/kernel/kernel.symbols CONVERSION STATISTICS: #samples/total 37656 Profile trace for function: __rw_rlock() [9.12%] Profile trace for function: ip_fastforward() [5.80%] Profile trace for function: binuptime() [5.73%] Profile trace for function: rtalloc1_fib() [4.92%] Profile trace for function: tsc_get_timecount_low_lfence() [4.71%] Profile trace for function: igb_rxeof() [4.51%] Profile trace for function: _rw_runlock_cookie() [4.14%] Profile trace for function: igb_mq_start_locked() [4.12%] Profile trace for function: ether_output() [3.36%] Profile trace for function: ether_nh_input() [3.11%] Profile trace for function: igb_mq_start() [3.02%] Profile trace for function: random_ivy_read() [2.88%] Profile trace for function: bzero() [2.70%] Profile trace for function: memcpy() [2.54%] Profile trace for function: random_harvest_queue() [2.32%] Profile trace for function: __mtx_lock_sleep() [1.91%] Profile trace for function: bcmp() [1.78%] Profile trace for function: _mtx_lock_spin_cookie() [1.73%] Profile trace for function: igb_txeof() [1.70%] Profile trace for function: rn_match() [1.69%] Profile trace for function: bounce_bus_dmamap_load_buffer() [1.52%] Profile trace for function: rtalloc_ign_fib() [1.30%] Profile trace for function: mb_free_ext() [1.29%] Profile trace for function: arpresolve() [1.28%] Profile trace for function: bcopy() [1.22%] Profile trace for function: spinlock_exit() [1.06%] Profile trace for function: mb_ctor_pack() [1.04%] Profile trace for function: uma_zalloc_arg() [1.02%] Profile trace for function: spinlock_enter() [1.00%] Profile trace for function: bus_dmamap_load_mbuf_sg() [0.93%] Profile trace for function: critical_exit() [0.90%] Profile trace for function: igb_refresh_mbufs() [0.89%] Profile trace for function: acpi_cpu_c1() [0.89%] Profile trace for function: netisr_dispatch_src() [0.81%] Profile trace for function: uma_zfree_arg() [0.80%] Profile trace for function: lockstat_nsecs() [0.79%] Profile trace for function: ether_demux() [0.76%] Profile trace for function: in_lltable_lookup() [0.75%] Profile trace for function: ip_findroute() [0.66%] Profile trace for function: in_localip() [0.65%] Profile trace for function: _mtx_trylock_flags_() [0.57%] Profile trace for function: _bus_dmamap_load_buffer() [0.54%] =E2=80=8BRegards, Olivier=E2=80=8B