From owner-freebsd-hackers@FreeBSD.ORG Thu Nov 13 19:17:24 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 97C6A511 for ; Thu, 13 Nov 2014 19:17:24 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 68A68909 for ; Thu, 13 Nov 2014 19:17:24 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id sADJHMRE046623 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 13 Nov 2014 11:17:22 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id sADJHL1J046622; Thu, 13 Nov 2014 11:17:21 -0800 (PST) (envelope-from jmg) Date: Thu, 13 Nov 2014 11:17:21 -0800 From: John-Mark Gurney To: Mehmet Erol Sanliturk Subject: Re: Slow nfsd write performance, tweaks needed Message-ID: <20141113191721.GJ24601@funkthat.com> Mail-Followup-To: Mehmet Erol Sanliturk , Rick Macklem , FreeBSD Hackers , Mark Schouten References: <2826701214-10966@kerio.tuxis.nl> <1841043385.12560911.1415883615679.JavaMail.root@uoguelph.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 13 Nov 2014 11:17:23 -0800 (PST) Cc: FreeBSD Hackers , Rick Macklem , Mark Schouten X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 19:17:24 -0000 Mehmet Erol Sanliturk wrote this message on Thu, Nov 13, 2014 at 05:14 -0800: > On Thu, Nov 13, 2014 at 5:00 AM, Rick Macklem wrote: > > > Mark Schouten wrote: > > > Hi, > > > > > > > > > I am in the process of switching from a ZFS On Linux-based NFS-server > > > to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is > > > way superiour over ZoL, and the box serves as storage for a > > > virtualizationplatform, so stability is welcome. :) > > > > > > > > > The box is stable, but performs terribly. Surely, I'm doing something > > > wrong, but I would like some tips and tricks to speed things up. > > > > > > > > > > > > > > > Here's my setup: > > > CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is > > > enabled) > > > RAM: 64GB > > > NIC: 2x igb in lagg0 (loadbalancing) > > Oops, I didn't see this before my last post. igb had problems with > > the 64K TSO issue and I'd try to get rid of lagg as well. > > (You might be much better off just using a single net interface > > without lagg.) > > > > Again, good luck with it, rick > > > > > Disks: > > > > > > export1 1.81T 914G 942G 49% 1.00x ONLINE - > > > mirror 928G 457G 471G - > > > da0 - - - - > > > da1 - - - - > > > mirror 928G 457G 471G - > > > da2 - - - - > > > da3 - - - - > > > mirror 9.94G 173M 9.77G - > > > da4p1 - - - - > > > da5p1 - - - - > > > cache - - - - - - > > > da4p2 223G 223G 8M - > > > da5p2 223G 223G 8M - > > > > > > > > > da0-3 are 1TB WDs > > > da4-5 are 240GB Samsung SSD 840s > > > > > > > > > Here's (related) info from rc.conf. > > > > > > nfs_server_enable="YES" > > > nfs_server_flags="-u -t -n 128" > > > rpcbind_enable="YES" > > > mountd_enable="YES" > > > rpc_lockd_enable="YES" > > > rpc_statd_enable="YES" > > > > > > > > > > > > > > > I have compression enabled on all the ZFS-filesystems, and > > > jumboframes are enabled on the nics. > > > > > > > > > As soon as one of the (Linux) clients start to do some IO, NFS > > > responsetimes go up bigtime (yesterday up to 13 seconds), while the > > > hardware is pretty much idle, I must be doing something very wrong. > > > I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is > > > appreciated. > > > > > > > > > Regards, > > > > > > > > > -- > > > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ > > > Mark Schouten | Tuxis Internet Engineering > > > KvK: 61527076 | http://www.tuxis.nl/ > > > T: 0318 200208 | info@tuxis.nl > > > > > > A terrible write performance may come from Linux mount options in fstab : > > > There is sync / async for the NFS connection : > > I was set sync : A file requiring 30 ( thirty ) SECONDs to write become 30 > MINUTEs to write . > After working to solve this problems by trying diffrerent parameter setting > over many days , there only remained sync / async . > When I selected async , writes turned to NORMAL . > > I do not know effect of compression ( my opinion is that it will not be > "terrible" ) , but my suggestion is to check as a possible trouble point : > > sync / async in Linux computer ( in fstab or mount statement ) This sounds like you might want a ZIL added to your server... Make sure you use an SSD.. Since you're using an NFS server, it cannot reply success to an operation till it is committed to stable storage which it sounds like is slow... With a ZIL on an SSD, it can more quickly commit the NFS request to stable storage and return success much quicker... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."