From owner-freebsd-hackers@FreeBSD.ORG Thu Nov 13 13:14:46 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 66B78592 for ; Thu, 13 Nov 2014 13:14:46 +0000 (UTC) Received: from mail-yk0-x22b.google.com (mail-yk0-x22b.google.com [IPv6:2607:f8b0:4002:c07::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 21B44945 for ; Thu, 13 Nov 2014 13:14:46 +0000 (UTC) Received: by mail-yk0-f171.google.com with SMTP id 142so2039592ykq.2 for ; Thu, 13 Nov 2014 05:14:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=A/1P3+OZHFW7PdILPPUFq5Lj8mYquGgnUNkVCJnrB2I=; b=i6uQngxEb0Q8guTRIv88dmfbN1zOeCefROvCEo2N506LfWoEqloK4wjZeYd0czY2Kf Vd3LMH+ibACFFVOjAMAilzB3dOI+fOf4lNdmQq9knsec8HYf5+AKmnIHnF58nw5HeZHb xedL+HbSUVi4+aCy44QyVLgeStJc3BtAW061W1BykBXjX7JTyuzL57xWtkC63ZBONJ1v SZpht1i4WiuLKgzIu3h1TGltaxVTHHIiKWuFbMO3frw3Vedf7jQiISvbNYNKsRqzZ0mk zDS/nSMjNtla0gan7Q2KohxWFXPHudmkBpMX6HhBfQmKYI+FGUPyrLXWFn4Tpkmw2N8r LOOA== MIME-Version: 1.0 X-Received: by 10.170.196.72 with SMTP id n69mr2747882yke.41.1415884485344; Thu, 13 Nov 2014 05:14:45 -0800 (PST) Received: by 10.170.84.133 with HTTP; Thu, 13 Nov 2014 05:14:45 -0800 (PST) In-Reply-To: <1841043385.12560911.1415883615679.JavaMail.root@uoguelph.ca> References: <2826701214-10966@kerio.tuxis.nl> <1841043385.12560911.1415883615679.JavaMail.root@uoguelph.ca> Date: Thu, 13 Nov 2014 05:14:45 -0800 Message-ID: Subject: Re: Slow nfsd write performance, tweaks needed From: Mehmet Erol Sanliturk To: Rick Macklem Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD Hackers , Mark Schouten X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 13:14:46 -0000 On Thu, Nov 13, 2014 at 5:00 AM, Rick Macklem wrote: > Mark Schouten wrote: > > Hi, > > > > > > I am in the process of switching from a ZFS On Linux-based NFS-server > > to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is > > way superiour over ZoL, and the box serves as storage for a > > virtualizationplatform, so stability is welcome. :) > > > > > > The box is stable, but performs terribly. Surely, I'm doing something > > wrong, but I would like some tips and tricks to speed things up. > > > > > > > > > > Here's my setup: > > CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is > > enabled) > > RAM: 64GB > > NIC: 2x igb in lagg0 (loadbalancing) > Oops, I didn't see this before my last post. igb had problems with > the 64K TSO issue and I'd try to get rid of lagg as well. > (You might be much better off just using a single net interface > without lagg.) > > Again, good luck with it, rick > > > Disks: > > > > export1 1.81T 914G 942G 49% 1.00x ONLINE - > > mirror 928G 457G 471G - > > da0 - - - - > > da1 - - - - > > mirror 928G 457G 471G - > > da2 - - - - > > da3 - - - - > > mirror 9.94G 173M 9.77G - > > da4p1 - - - - > > da5p1 - - - - > > cache - - - - - - > > da4p2 223G 223G 8M - > > da5p2 223G 223G 8M - > > > > > > da0-3 are 1TB WDs > > da4-5 are 240GB Samsung SSD 840s > > > > > > Here's (related) info from rc.conf. > > > > nfs_server_enable="YES" > > nfs_server_flags="-u -t -n 128" > > rpcbind_enable="YES" > > mountd_enable="YES" > > rpc_lockd_enable="YES" > > rpc_statd_enable="YES" > > > > > > > > > > I have compression enabled on all the ZFS-filesystems, and > > jumboframes are enabled on the nics. > > > > > > As soon as one of the (Linux) clients start to do some IO, NFS > > responsetimes go up bigtime (yesterday up to 13 seconds), while the > > hardware is pretty much idle, I must be doing something very wrong. > > I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is > > appreciated. > > > > > > Regards, > > > > > > -- > > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ > > Mark Schouten | Tuxis Internet Engineering > > KvK: 61527076 | http://www.tuxis.nl/ > > T: 0318 200208 | info@tuxis.nl > A terrible write performance may come from Linux mount options in fstab : There is sync / async for the NFS connection : I was set sync : A file requiring 30 ( thirty ) SECONDs to write become 30 MINUTEs to write . After working to solve this problems by trying diffrerent parameter setting over many days , there only remained sync / async . When I selected async , writes turned to NORMAL . I do not know effect of compression ( my opinion is that it will not be "terrible" ) , but my suggestion is to check as a possible trouble point : sync / async in Linux computer ( in fstab or mount statement ) Thank you very much . Mehmet Erol Sanliturk