From owner-freebsd-fs@FreeBSD.ORG Mon Dec 8 08:36:33 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7EEF5C99 for ; Mon, 8 Dec 2014 08:36:33 +0000 (UTC) Received: from smtp.unix-experience.fr (195-154-176-227.rev.poneytelecom.eu [195.154.176.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3B3CDD7B for ; Mon, 8 Dec 2014 08:36:32 +0000 (UTC) Received: from smtp.unix-experience.fr (unknown [192.168.200.21]) by smtp.unix-experience.fr (Postfix) with ESMTP id 16B45220B4; Mon, 8 Dec 2014 08:36:24 +0000 (UTC) X-Virus-Scanned: scanned by unix-experience.fr Received: from smtp.unix-experience.fr ([192.168.200.21]) by smtp.unix-experience.fr (smtp.unix-experience.fr [192.168.200.21]) (amavisd-new, port 10024) with ESMTP id ZB2wtjMPoJw0; Mon, 8 Dec 2014 08:36:22 +0000 (UTC) Received: from mail.unix-experience.fr (repo.unix-experience.fr [192.168.200.30]) by smtp.unix-experience.fr (Postfix) with ESMTPSA id D8119220A6; Mon, 8 Dec 2014 08:36:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=unix-experience.fr; s=uxselect; t=1418027782; bh=0CeMJ0G7XaB0j+b9/ZZGanFCD5urDyZlD6qvnMCJCjE=; h=Date:From:Subject:To:Cc:In-Reply-To:References; b=X+tg4GpFvWY4K54JFzCX5pfdWMAGEzrVTIKEhRS2gAeuAS/n8iTXVIGExmSgzvJfj ARhPjilf7unWee5LCSgt89lSMMLHhDBhQwSfr6NQUvpCPgEkbQ11Vehu4L8dDedcSD iXgPYhRa7QqO/rDejXy+N9IqIu+lRZ3Nu0rGVTVE= Mime-Version: 1.0 Date: Mon, 08 Dec 2014 08:36:21 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-ID: X-Mailer: RainLoop/1.6.10.182 From: "=?utf-8?B?TG/Dr2MgQmxvdA==?=" Subject: Re: High Kernel Load with nfsv4 To: "Rick Macklem" In-Reply-To: <581583623.5730217.1417788866930.JavaMail.root@uoguelph.ca> References: <581583623.5730217.1417788866930.JavaMail.root@uoguelph.ca> Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2014 08:36:33 -0000 Hi Rick,=0AI stopped the jails this week-end and started it this morning,= i'll give you some stats this week.=0A=0AHere is my nfsstat -m output (w= ith your rsize/wsize tweaks)=0A=0Anfsv4,tcp,resvport,hard,cto,sec=3Dsys,a= cdirmin=3D3,acdirmax=3D60,acregmin=3D5,acregmax=3D60,nametimeo=3D60,negna= metimeo=3D60,rsize=3D32768,wsize=3D32768,readdirsize=3D32768,readahead=3D= 1,wcommitsize=3D773136,timeout=3D120,retrans=3D2147483647=0A=0AOn server = side my disks are on a raid controller which show a 512b volume and write= performances are very honest (dd if=3D/dev/zero of=3D/jails/test.dd bs= =3D4096 count=3D100000000 =3D> 450MBps)=0A=0ARegards,=0A=0ALo=C3=AFc Blot= ,=0AUNIX Systems, Network and Security Engineer=0Ahttp://www.unix-experie= nce.fr=0A=0A5 d=C3=A9cembre 2014 15:14 "Rick Macklem" a =C3=A9crit: =0A> Loic Blot wrote:=0A> =0A>> Hi,=0A>> i'm trying to = create a virtualisation environment based on jails.=0A>> Those jails are = stored under a big ZFS pool on a FreeBSD 9.3 which=0A>> export a NFSv4 vo= lume. This NFSv4 volume was mounted on a big=0A>> hypervisor (2 Xeon E5v3= + 128GB memory and 8 ports (but only 1 was=0A>> used at this time).=0A>>= =0A>> The problem is simple, my hypervisors runs 6 jails (used 1% cpu an= d=0A>> 10GB RAM approximatively and less than 1MB bandwidth) and works=0A= >> fine at start but the system slows down and after 2-3 days become=0A>>= unusable. When i look at top command i see 80-100% on system and=0A>> co= mmands are very very slow. Many process are tagged with nfs_cl*.=0A> =0A>= To be honest, I would expect the slowness to be because of slow response= =0A> from the NFSv4 server, but if you do:=0A> # ps axHl=0A> on a client = when it is slow and post that, it would give us some more=0A> information= on where the client side processes are sitting.=0A> If you also do somet= hing like:=0A> # nfsstat -c -w 1=0A> and let it run for a while, that sho= uld show you how many RPCs are=0A> being done and which ones.=0A> =0A> # = nfsstat -m=0A> will show you what your mount is actually using.=0A> The o= nly mount option I can suggest trying is "rsize=3D32768,wsize=3D32768",= =0A> since some network environments have difficulties with 64K.=0A> =0A>= There are a few things you can try on the NFSv4 server side, if it appea= rs=0A> that the clients are generating a large RPC load.=0A> - disabling = the DRC cache for TCP by setting vfs.nfsd.cachetcp=3D0=0A> - If the serve= r is seeing a large write RPC load, then "sync=3Ddisabled"=0A> might help= , although it does run a risk of data loss when the server=0A> crashes.= =0A> Then there are a couple of other ZFS related things (I'm not a ZFS g= uy,=0A> but these have shown up on the mailing lists).=0A> - make sure yo= ur volumes are 4K aligned and ashift=3D12 (in case a drive=0A> that uses = 4K sectors is pretending to be 512byte sectored)=0A> - never run over 70-= 80% full if write performance is an issue=0A> - use a zil on an SSD with = good write performance=0A> =0A> The only NFSv4 thing I can tell you is th= at it is known that ZFS's=0A> algorithm for determining sequential vs ran= dom I/O fails for NFSv4=0A> during writing and this can be a performance = hit. The only workaround=0A> is to use NFSv3 mounts, since file handle af= finity apparently fixes=0A> the problem and this is only done for NFSv3.= =0A> =0A> rick=0A> =0A>> I saw that there are TSO issues with igb then i'= m trying to disable=0A>> it with sysctl but the situation wasn't solved.= =0A>> =0A>> Someone has got ideas ? I can give you more informations if y= ou=0A>> need.=0A>> =0A>> Thanks in advance.=0A>> Regards,=0A>> =0A>> Lo= =C3=AFc Blot,=0A>> UNIX Systems, Network and Security Engineer=0A>> http:= //www.unix-experience.fr=0A>> ___________________________________________= ____=0A>> freebsd-fs@freebsd.org mailing list=0A>> http://lists.freebsd.o= rg/mailman/listinfo/freebsd-fs=0A>> To unsubscribe, send any mail to "fre= ebsd-fs-unsubscribe@freebsd.org"