From owner-freebsd-stable@FreeBSD.ORG Fri Mar 20 23:53:43 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7263B1BF; Fri, 20 Mar 2015 23:53:43 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [IPv6:2001:470:1:117::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 536D0F34; Fri, 20 Mar 2015 23:53:43 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [12.229.62.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 6D978EAB1; Fri, 20 Mar 2015 16:53:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1426895622; x=1426910022; bh=vINHaNFrwDsaH2rl0ITQNEygkJ2+UgliKVH/7pTfhTw=; h=Date:From:Reply-To:To:CC:Subject:References:In-Reply-To; b=ehE7Ff2+PSHRXnDBjrhd6NrR3NdXthKqvwf1Kl07zWXhd6wy7t4hvDrfDZF3XdYzd nZPrXmoza4ifhg2Qc49+nCLgWsPG2Ae2QSNbYOfL2UZmG/3seFyxufCa44fxJbFE+h deoiHJdaspR4zKHZh3zfXDZ7LY8e4tdxh4YPgTes= Message-ID: <550CB306.7030405@delphij.net> Date: Fri, 20 Mar 2015 16:53:42 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: Mike Tancsa , John-Mark Gurney Subject: Re: RELENG_10 performance regression (was Re: 35-40% performance drop releng9 vs releng10 openvpn References: <5506250A.2000506@sentex.net> <20150316132055.GQ32288@funkthat.com> <5509D6C6.4050204@sentex.net> <20150318211457.GL51048@funkthat.com> <550B6950.8060806@sentex.net> <550C5AAF.9060502@sentex.net> <550C8AEE.4090408@sentex.net> In-Reply-To: <550C8AEE.4090408@sentex.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Cc: FreeBSD-STABLE Mailing List , John Baldwin X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Mar 2015 23:53:43 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/20/15 14:02, Mike Tancsa wrote: > OK, I think I found where the RELENG_10 performance loss happened. > It seems > https://lists.freebsd.org/pipermail/svn-src-stable-10/2015-March/004778.html > > is the issue. > > Testing with a kernel from r279796 I get 76-77Mb of throughput. > With r279848 it drops to about 60Mb Hrm, looking at 'diffgraph.svg', it suggest that writing through devfs have incurred a tax on writev(2), which is basically this codepath, if my reading is correct, in sys/fs/devfs/devfs.c: 1676 static int 1677 devfs_write_f(struct file *fp, struct uio *uio, struct ucred *cred, 1678 int flags, struct thread *td) 1679 { ... 1703 if (uio->uio_resid != resid || (error == 0 && resid != 0)) { 1704 vfs_timestamp(&dev->si_ctime); 1705 dev->si_mtime = dev->si_ctime; 1706 } Further looking at the code, in devfs vfs_timestamp() is also called when reading from a device node. When the setting was 0, the code would return time_second directly instead of attempting to read the timestamp. For the purpose of devfs, does it make sense to bump timestamps like normal filesystems for each read/write operation? Looks like Mac OS X will bump timestamps for each operation but Debian don't. Cheers, > ---Mike > > > > On 3/20/2015 1:36 PM, Mike Tancsa wrote: >> OK, just to refocus, >> >> I had been tracking down what I thought was a regression between >> RELENG9 and RELENG10, but looks more like an issue that cropped >> up somewhere between the beginning of March and now. For >> RELENG9, I was actually using a kernel from sources back on Jan >> 29th by accident. If I bring RELENG9 upto today, I get a similar >> performance loss. >> >> Again, I am testing a simple VPN router setup >> >> server1 --- apu --- server2 >> >> where server1 connections to the apu via an OpenVPN tunnel and >> server1 sends packets via netblast across the tunnel to server2. >> >> I get the following throughput using netblast through the tunnel >> on 10 >> >> Using # netblast 1.1.2.2 500 1200 15 (server1 to server2) on >> 10.x >> >> Kernel Mb/s rev r277684 76.7563 r279978 59.3233 >> >> All good at r278533, r278534, r279467 >> >> But at r279978 its quite a bit slower. So somewhere between >> r279467 and r279978. I will keep trying to narrow it down... >> >> >> ---Mike >> >> >> >> >> >> On 3/19/2015 8:26 PM, Mike Tancsa wrote: >>> On 3/18/2015 5:14 PM, John-Mark Gurney wrote: >>>> # dtrace -x stackframes=100 -n 'profile-997 /arg0/ { >>>> @[stack()] = count(); } tick-60s { exit(0); }' -o >>>> out.kern_stacks >>>> >>>> Also, another thing you can do is to compare the two using >>>> differential flame graphs: >>>> http://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html >>>> >>>> >>>> >>>> >>>> >>>> Which will highlight where the performances differ... >>> >>> OK, some more data points. It seems a performance regression >>> happened in RELENG_10 somewhere between r277684 (late January >>> 2015) and now. Using r277684 on RELENG_10, I can get about >>> 75Mb/s of throughput on OpenVPN. Still not as good as the >>> 83-85Mb on RELENG_9, but much better than the 61Mb using >>> RELENG_10 from the start of this week, >>> >>> For the differential graph, see >>> >>> http://tancsa.com/diffgraph.svg >>> >>> and >>> >>> http://tancsa.com/10-r277684.svg >>> http://tancsa.com/10-r277684-kern.svg >>> >>> ---Mike >>> >>> >>> >>> >> >> > > - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.1.2 (FreeBSD) iQIcBAEBCgAGBQJVDLMDAAoJEJW2GBstM+nsmjgQAKGNdw70A2lfrVn8ngLympDM HeJKVCRtjaZselJpVnkyy8igBU6TX0Ae66R4IPNBpi9aLcGb/xscDgYKNh2dZc4H LjfyiXW62JwggXt+PCUjrLF3sbrlCI8ySTNLnILTs/CRFwdJXALFHWXDEtuMg+AT Mto3HxKlRZlEbWHaKbWmR4uysks6KNYJXSCnw/VtSFvlVUSwpyNz+soygrajZPKC PPOpjDqx2rvVxQwFaHF+ysTzKhvQSuJg+C5M9iKu4wgHd5QNuJ8IliAocVvPig9u gushcc2MxGPyzG24Telc3Vc5T+pbCexiP3hRtFY0/okADG4O47tuoJOY09gp28Tq 5X3eXCXkBGfCtriv6aPf//jAQ6duDrGTDmYcI+FGIDp3/FG71adNTqZ6fcqFAL/+ JHO39qpytOruVVjFyjEvQBslYEKWUZJBGZ82q6kuE+811TZxdMGVLh3PICxiJw0l SJcTR52UEYNp1kX+KhFXpz6cvV0Za5bJh/qHnC/1uzHvLYE7MDTinwDPb7LyK01G fg+Kls6YCAv6rh4mDzWJCg8aTaWmbtfn2+ETwx74if6Z8LjTB1VdmZ4y336YRY2f mFhV6eYAvvziIIMUvTQjAwyIcCtc65dvseOpgDdnWPFq/N+K4Jfg9Ast6mSHBYF2 FjvUzNg8OYvyfq5/TdPB =djPp -----END PGP SIGNATURE-----