From owner-freebsd-fs@FreeBSD.ORG Fri Dec 5 14:15:36 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C85ED996 for ; Fri, 5 Dec 2014 14:15:36 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 21711138 for ; Fri, 5 Dec 2014 14:15:35 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgwFAIm8gVSDaFve/2dsb2JhbABZg1hYBIMBw0AKhURPAoEzAQEBAQF9hAIBAQEDAQEBASArIAsFFhgCAg0ZAikBCSYGCAcEARwEiBEJDb9xlm8BAQEBAQUBAQEBAQEBARqBKI5WAQEbATMHgjE+EYE2BYowiRSDUINtNI4Hg2KEDSEwB4EFOX4BAQE X-IronPort-AV: E=Sophos;i="5.07,522,1413259200"; d="scan'208";a="175711504" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 05 Dec 2014 09:14:26 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id E6C27B4060; Fri, 5 Dec 2014 09:14:26 -0500 (EST) Date: Fri, 5 Dec 2014 09:14:26 -0500 (EST) From: Rick Macklem To: =?utf-8?B?TG/Dr2M=?= Blot Message-ID: <581583623.5730217.1417788866930.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: High Kernel Load with nfsv4 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Dec 2014 14:15:37 -0000 Loic Blot wrote: > Hi, > i'm trying to create a virtualisation environment based on jails. > Those jails are stored under a big ZFS pool on a FreeBSD 9.3 which > export a NFSv4 volume. This NFSv4 volume was mounted on a big > hypervisor (2 Xeon E5v3 + 128GB memory and 8 ports (but only 1 was > used at this time). >=20 > The problem is simple, my hypervisors runs 6 jails (used 1% cpu and > 10GB RAM approximatively and less than 1MB bandwidth) and works > fine at start but the system slows down and after 2-3 days become > unusable. When i look at top command i see 80-100% on system and > commands are very very slow. Many process are tagged with nfs_cl*. >=20 To be honest, I would expect the slowness to be because of slow response from the NFSv4 server, but if you do: # ps axHl on a client when it is slow and post that, it would give us some more information on where the client side processes are sitting. If you also do something like: # nfsstat -c -w 1 and let it run for a while, that should show you how many RPCs are being done and which ones. # nfsstat -m will show you what your mount is actually using. The only mount option I can suggest trying is "rsize=3D32768,wsize=3D32768"= , since some network environments have difficulties with 64K. There are a few things you can try on the NFSv4 server side, if it appears that the clients are generating a large RPC load. - disabling the DRC cache for TCP by setting vfs.nfsd.cachetcp=3D0 - If the server is seeing a large write RPC load, then "sync=3Ddisabled" might help, although it does run a risk of data loss when the server crashes. Then there are a couple of other ZFS related things (I'm not a ZFS guy, but these have shown up on the mailing lists). - make sure your volumes are 4K aligned and ashift=3D12 (in case a drive that uses 4K sectors is pretending to be 512byte sectored) - never run over 70-80% full if write performance is an issue - use a zil on an SSD with good write performance The only NFSv4 thing I can tell you is that it is known that ZFS's algorithm for determining sequential vs random I/O fails for NFSv4 during writing and this can be a performance hit. The only workaround is to use NFSv3 mounts, since file handle affinity apparently fixes the problem and this is only done for NFSv3. rick > I saw that there are TSO issues with igb then i'm trying to disable > it with sysctl but the situation wasn't solved. >=20 > Someone has got ideas ? I can give you more informations if you > need. >=20 > Thanks in advance. > Regards, >=20 > Lo=C3=AFc Blot, > UNIX Systems, Network and Security Engineer > http://www.unix-experience.fr > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"