From owner-freebsd-stable@FreeBSD.ORG Wed Mar 18 21:14:58 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ECD7C8AB for ; Wed, 18 Mar 2015 21:14:58 +0000 (UTC) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A6428C76 for ; Wed, 18 Mar 2015 21:14:58 +0000 (UTC) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t2ILEv7B053833 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 18 Mar 2015 14:14:57 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t2ILEv5L053832; Wed, 18 Mar 2015 14:14:57 -0700 (PDT) (envelope-from jmg) Date: Wed, 18 Mar 2015 14:14:57 -0700 From: John-Mark Gurney To: Mike Tancsa Subject: Re: 35-40% performance drop releng9 vs releng10 openvpn Message-ID: <20150318211457.GL51048@funkthat.com> References: <5506250A.2000506@sentex.net> <20150316132055.GQ32288@funkthat.com> <5509D6C6.4050204@sentex.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5509D6C6.4050204@sentex.net> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Wed, 18 Mar 2015 14:14:57 -0700 (PDT) Cc: FreeBSD-STABLE Mailing List X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Mar 2015 21:14:59 -0000 Mike Tancsa wrote this message on Wed, Mar 18, 2015 at 15:49 -0400: > On 3/16/2015 9:20 AM, John-Mark Gurney wrote: > > > > Since you have at test framework ready, you could generate some flame > > graphs[1] using dtrace to help see where things might be having an > > impact... > > > > These are very easy to generate, and posting them would be useful... > > > > [1] http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html > > > Hi, > I went through the steps to generate one. What args should I use for > dtrace to generate the information that is helpful / useful ? For my > setup, I have > > server1---------apu-------------server2 > > server1 has an openvpn tunnel to the apu > I route server2's IP address across the VPN tunnel, so if I ping from > server1 to server2's IP, it goes via the tunnel > > on the > > dtrace -x ustackframes=100 -n 'profile-99 /execname == "openvpn" && > arg1/ { @[ustack()] = count(); } tick-30s { exit(0); }' -o 10.stacks > > which generated > http://tancsa.com/10.svg So, I would first identify the machine w/ the cpu limited load.. I assume that is apu... Then I would look at where most of the cpu time is being spent, be it openvpn itself, or in the kernel... Most likely it is the kernel, so getting stacks from the kernel would be more useful than the one you generated... Use the command: # dtrace -x stackframes=100 -n 'profile-997 /arg0/ { @[stack()] = count(); } tick-60s { exit(0); }' -o out.kern_stacks Also, another thing you can do is to compare the two using differential flame graphs: http://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html Which will highlight where the performances differ... As I've never used OpenVPN before and their docs don't go into saying what it's using.. Is OpenVPN a kernel or userland VPN? Do they use IPSec in the kernel? or are they just using UDP or TCP for their connections? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."