From owner-svn-src-all@freebsd.org Mon Aug 3 15:05:43 2015 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 762E99AE2F1; Mon, 3 Aug 2015 15:05:43 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: from mail-yk0-x229.google.com (mail-yk0-x229.google.com [IPv6:2607:f8b0:4002:c07::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32AFDD0D; Mon, 3 Aug 2015 15:05:43 +0000 (UTC) (envelope-from ermal.luci@gmail.com) Received: by ykoo205 with SMTP id o205so22511977yko.0; Mon, 03 Aug 2015 08:05:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=51Na3GClefhjU3ulb9xD1yLJf/Hw/rG/S7TKQrm/kZE=; b=WRzPmwH5AztzD7VwCr1sEHvbLGG/0AovRvPehAol1XShl6Kxcw6+SSXWnkGm70cpKM 1DThlQTmCXpe3BWyKUm5Pb4/QKdC4gXkyww+eQfPnYWcHKuvnl2E1R4cXtG8wOw7i7w5 x2eladXyeGKC0lXiDVBahi19O3UrXxNW/bRMwoDmWXwNe25MPMvpIEGCWl3wkg6SCwJI 9My2h7WLfBhDbJ2EPlrvTeEVJbi3anPBJS1xK62TLhDTaKpWeEfR2uxK9xaQSrJJc/6m 3cd1HyW5s4lkGBeM4oGwaCnSpLt2hsLrGrYg7UEbRjr3dkj3lnuBkJYouVFPw06p/Jd8 Gysw== MIME-Version: 1.0 X-Received: by 10.170.198.144 with SMTP id p138mr20800610yke.70.1438614342295; Mon, 03 Aug 2015 08:05:42 -0700 (PDT) Sender: ermal.luci@gmail.com Received: by 10.129.83.139 with HTTP; Mon, 3 Aug 2015 08:05:42 -0700 (PDT) In-Reply-To: References: <201507021810.t62IAgCc003272@repo.freebsd.org> <20150728124220.GW72729@FreeBSD.org> Date: Mon, 3 Aug 2015 17:05:42 +0200 X-Google-Sender-Auth: 9XThOUgrTxJV_yDg27MNOba6GRw Message-ID: Subject: Re: svn commit: r285051 - head/sys/netinet From: =?UTF-8?Q?Ermal_Lu=C3=A7i?= To: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= Cc: Gleb Smirnoff , svn-src-head , svn-src-all , src-committers Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Aug 2015 15:05:43 -0000 Hello Olivier, its strange seeing so much contention on the arp tables on your PMC stats. Do you run ping(to prepopulate arp) or static arp to remove the noise from that interaction? Also do you run with flowtable active? On Mon, Aug 3, 2015 at 3:06 PM, Olivier Cochard-Labb=C3=A9 wrote: > On Tue, Jul 28, 2015 at 2:42 PM, Gleb Smirnoff > wrote: > >> >> Here is suggested patch. Ermal and Oliver, can you please test/benchmark >> it? >> > > =E2=80=8BHi, > > this patch reduce performanece :-( > > Here are the results regarding forwarding: > > x r285046.pps.forwarding (IPSEC compiled but not used) > + r285051.pps.forwarding (IPSEC compiled but not used) > * r285051-glebius-patched.pps.forwarding (IPSEC compiled but not used) > > +------------------------------------------------------------------------= --------+ > |xxx xx **** > * + +++| > ||_M_A___| > |_A_| |AM|| > > +------------------------------------------------------------------------= --------+ > N Min Max Median Avg Stdd= ev > x 5 397733 406951 399300 401613.8 4324.97= 55 > + 5 478095 482079 480869 480543.6 1666.02= 82 > Difference at 95.0% confidence > 78929.8 +/- 4779.72 > 19.6532% +/- 1.19013% > (Student's t, pooled s =3D 3277.27) > * 5 424720 430745 427014 427378.4 2351.74= 39 > Difference at 95.0% confidence > 25764.6 +/- 5076.98 > 6.41527% +/- 1.26415% > (Student's t, pooled s =3D 3481.1) > =E2=80=8B > > > PMC stats during forwarding bench: > [root@netgate]/data# pmcannotate pmc.forwarding.out > /data/debug/boot/kernel/kernel.symbols > CONVERSION STATISTICS: > #samples/total 33880 > Profile trace for function: __rw_rlock() [6.29%] > Profile trace for function: ip_forward() [4.68%] > Profile trace for function: ip_output() [4.64%] > Profile trace for function: binuptime() [4.05%] > Profile trace for function: igb_mq_start_locked() [3.79%] > Profile trace for function: igb_rxeof() [3.46%] > Profile trace for function: tsc_get_timecount_low_lfence() [3.25%] > Profile trace for function: ether_output() [3.03%] > Profile trace for function: rtalloc1_fib() [2.77%] > Profile trace for function: random_ivy_read() [2.64%] > Profile trace for function: _rw_runlock_cookie() [2.64%] > Profile trace for function: ether_nh_input() [2.63%] > Profile trace for function: ip_input() [2.55%] > Profile trace for function: key_allocsp_default() [2.39%] > Profile trace for function: igb_mq_start() [2.39%] > Profile trace for function: bzero() [2.08%] > Profile trace for function: uma_zalloc_arg() [1.95%] > Profile trace for function: memcpy() [1.84%] > Profile trace for function: _mtx_lock_spin_cookie() [1.83%] > Profile trace for function: bcopy() [1.76%] > Profile trace for function: random_harvest_queue() [1.63%] > Profile trace for function: __mtx_lock_sleep() [1.56%] > Profile trace for function: uma_zfree_arg() [1.47%] > Profile trace for function: arpresolve() [1.39%] > Profile trace for function: in_cksumdata() [1.25%] > Profile trace for function: bounce_bus_dmamap_load_buffer() [1.22%] > Profile trace for function: bcmp() [1.13%] > Profile trace for function: rtalloc_ign_fib() [1.11%] > Profile trace for function: rn_match() [1.03%] > Profile trace for function: netisr_dispatch_src() [1.03%] > Profile trace for function: critical_exit() [1.02%] > Profile trace for function: bus_dmamap_load_mbuf_sg() [0.87%] > Profile trace for function: spinlock_exit() [0.79%] > Profile trace for function: in_cksum_skip() [0.75%] > Profile trace for function: ip_ipsec_output() [0.75%] > Profile trace for function: acpi_cpu_c1() [0.74%] > Profile trace for function: in_broadcast() [0.74%] > Profile trace for function: spinlock_enter() [0.74%] > Profile trace for function: igb_refresh_mbufs() [0.71%] > Profile trace for function: in_lltable_lookup() [0.71%] > Profile trace for function: ip_fastforward() [0.68%] > Profile trace for function: m_adj() [0.65%] > Profile trace for function: ether_demux() [0.65%] > Profile trace for function: _key_freesp() [0.61%] > Profile trace for function: lockstat_nsecs() [0.60%] > Profile trace for function: m_freem() [0.58%] > Profile trace for function: critical_enter() [0.56%] > Profile trace for function: m_copydata() [0.55%] > Profile trace for function: mb_free_ext() [0.54%] > Profile trace for function: pmap_kextract() [0.50%] > > =E2=80=8B > > =E2=80=8BAnd about fastforwarding: > =E2=80=8B > x 285046.pps.fastforwarding (IPSEC compiled but not used) > + 285051.pps.fastforwarding (IPSEC compiled but not used) > * r285051-glebius-patched.pps.fastforwarding (IPSEC compiled but not used= ) > > +------------------------------------------------------------------------= --------+ > |* > + | > |* * * * ++ +x > x+ xx x| > ||____AM__| > |_MA__|__AM_| | > > +------------------------------------------------------------------------= --------+ > N Min Max Median Avg Stdd= ev > x 5 742683 754709 750940 749015.6 4944.90= 59 > + 5 736459 747593 738197 740112.4 4511.16= 11 > Difference at 95.0% confidence > -8903.2 +/- 6902.82 > -1.18865% +/- 0.921585% > (Student's t, pooled s =3D 4733) > * 5 638356 652855 646668 645263 6599.86= 04 > Difference at 95.0% confidence > -103753 +/- 8504.75 > -13.8519% +/- 1.13546% > (Student's t, pooled s =3D 5831.39) > > And pmcstat during fastforwarding: > > [root@netgate]/data# pmcannotate pmc.fastforwading.out > /data/debug/boot/kernel/kernel.symbols > CONVERSION STATISTICS: > #samples/total 37656 > Profile trace for function: __rw_rlock() [9.12%] > Profile trace for function: ip_fastforward() [5.80%] > Profile trace for function: binuptime() [5.73%] > Profile trace for function: rtalloc1_fib() [4.92%] > Profile trace for function: tsc_get_timecount_low_lfence() [4.71%] > Profile trace for function: igb_rxeof() [4.51%] > Profile trace for function: _rw_runlock_cookie() [4.14%] > Profile trace for function: igb_mq_start_locked() [4.12%] > Profile trace for function: ether_output() [3.36%] > Profile trace for function: ether_nh_input() [3.11%] > Profile trace for function: igb_mq_start() [3.02%] > Profile trace for function: random_ivy_read() [2.88%] > Profile trace for function: bzero() [2.70%] > Profile trace for function: memcpy() [2.54%] > Profile trace for function: random_harvest_queue() [2.32%] > Profile trace for function: __mtx_lock_sleep() [1.91%] > Profile trace for function: bcmp() [1.78%] > Profile trace for function: _mtx_lock_spin_cookie() [1.73%] > Profile trace for function: igb_txeof() [1.70%] > Profile trace for function: rn_match() [1.69%] > Profile trace for function: bounce_bus_dmamap_load_buffer() [1.52%] > Profile trace for function: rtalloc_ign_fib() [1.30%] > Profile trace for function: mb_free_ext() [1.29%] > Profile trace for function: arpresolve() [1.28%] > Profile trace for function: bcopy() [1.22%] > Profile trace for function: spinlock_exit() [1.06%] > Profile trace for function: mb_ctor_pack() [1.04%] > Profile trace for function: uma_zalloc_arg() [1.02%] > Profile trace for function: spinlock_enter() [1.00%] > Profile trace for function: bus_dmamap_load_mbuf_sg() [0.93%] > Profile trace for function: critical_exit() [0.90%] > Profile trace for function: igb_refresh_mbufs() [0.89%] > Profile trace for function: acpi_cpu_c1() [0.89%] > Profile trace for function: netisr_dispatch_src() [0.81%] > Profile trace for function: uma_zfree_arg() [0.80%] > Profile trace for function: lockstat_nsecs() [0.79%] > Profile trace for function: ether_demux() [0.76%] > Profile trace for function: in_lltable_lookup() [0.75%] > Profile trace for function: ip_findroute() [0.66%] > Profile trace for function: in_localip() [0.65%] > Profile trace for function: _mtx_trylock_flags_() [0.57%] > Profile trace for function: _bus_dmamap_load_buffer() [0.54%] > > =E2=80=8BRegards, > > Olivier=E2=80=8B > --=20 Ermal