From owner-freebsd-stable@FreeBSD.ORG Wed Sep 11 12:51:19 2013 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 551C2D1A for ; Wed, 11 Sep 2013 12:51:19 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-ie0-f171.google.com (mail-ie0-f171.google.com [209.85.223.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2509C2A07 for ; Wed, 11 Sep 2013 12:51:18 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id 16so7742678iea.2 for ; Wed, 11 Sep 2013 05:51:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=Al5IQR7Ca/qVBB+TTmCRLU2f5QuiKjqe7zJOda+0Xfo=; b=PLVc/sc5R+4Gn1Holak6yA44PPEkb2JHVc6l69Yad2dpXFmel8KNrNn3touHiJN/9+ k8Mtg5zjaRb5rNLZhkhtG9DzVh+ijDnZ/9dO93yvHXCKnhiVUxG0BBIgNVg8B56gwmIL CDObTsW/EmzLISPETLeaWTwXlwuctjyF4QOkdBw/00fV6aN9fMO8GnMBFDkw0gr5Notf GV6U16sGPLaom5tDhyx25p2O3E3GBRM7gDOWyxVWN5+sazeSv1CV21ZoGB+ykWeuwvEF LBFr4ihoK7K75xxh3HnZCP4XIF6iX+WD0qo0BlMlqvc4QfmDkk8LHHu2UoLBAIvNyPi1 Hjhw== X-Gm-Message-State: ALoCoQns1gNx2NmGzKk01R/tx4DaU8zMiEG88ATm3gw4jS/z4wGLH4PCYTRLo8p8ppHIB1ncrQ+d X-Received: by 10.50.43.170 with SMTP id x10mr313162igl.45.1378903878038; Wed, 11 Sep 2013 05:51:18 -0700 (PDT) Received: from [97.61.211.27] (27.sub-97-61-211.myvzw.com. [97.61.211.27]) by mx.google.com with ESMTPSA id w4sm2955033igb.5.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 11 Sep 2013 05:51:16 -0700 (PDT) References: <995078453.21651811.1378900450650.JavaMail.root@uoguelph.ca> Mime-Version: 1.0 (1.0) In-Reply-To: <995078453.21651811.1378900450650.JavaMail.root@uoguelph.ca> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: X-Mailer: iPhone Mail (10B329) From: Mark Saad Subject: Re: nfsd CPU usage? Date: Wed, 11 Sep 2013 08:51:05 -0400 To: Rick Macklem Cc: "stable@freebsd.org" , Lars Eggert X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Sep 2013 12:51:19 -0000 Rick Would this affect 9.2-RCn ?=20 --- Mark saad | mark.saad@longcount.org On Sep 11, 2013, at 7:54 AM, Rick Macklem wrote: > Lars Eggert wrote: >> Hi, >>=20 >> I'm seeing extremely high CPU usage withssh-st the new nfsd: >>=20 >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 2280 root 102 0 9932K 1376K *nfs_c 0 320:11 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K CPU7 7 319:47 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K CPU5 5 318:25 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K CPU6 6 318:20 100.00% >> nfsd{nfsd: service} >> 2280 root 52 0 9932K 1376K CPU0 0 317:32 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K *nfs_c 1 315:41 99.17% >> nfsd{nfsd: service} >> 2280 root 52 0 9932K 1376K *nfs_c 4 320:22 98.78% >> nfsd{nfsd: master} >> 2280 root 102 0 9932K 1376K *nfs_c 1 317:10 98.10% >> nfsd{nfsd: service} >>=20 >> And this is at a few hundred KB/s with only a few clients: >>=20 >> ifstat -i igb1 10 >> igb1 >> KB/s in KB/s out >> 796.56 208.66 >> 431.19 232.36 >> 316.11 280.31 >> 1005.96 523.42 >> 1077.74 342.25 >> 340.63 217.73 >> 1067.96 330.56 >> 487.91 235.61 >>=20 >> Any ideas? >>=20 >> FreeBSD stanley.muccbc.hq.netapp.com 9.2-PRERELEASE FreeBSD >> 9.2-PRERELEASE #7: Wed Sep 4 11:06:31 CEST 2013 >> root@stanley.muccbc.hq.netapp.com:/usr/obj/usr/src/sys/STANLEY >> amd64 >>=20 >> Thanks, >> Lars > There is a patch in head (r254337) that I believe handles this. > It will be MFC'd to stable/9 in about a week, unless someone finds > problems with it before then. > If you want a semantically equivalent (but uglier code) patch, > you can find it here: > http://people.freebsd.org/~rmacklem/drc4-stable9.patch > After applying the patch, you need to set sysctl variable(s), > to avoid the aggressive trimming of stale DRC entries. Garrett > Wollman suggests the following for a large server: > vfs.nfsd.tcphighwater=3D100000 > vfs.nfsd.tcpcachetimeout=3D300 (5 minutes instead of default of several hr= s) >=20 > You can also use the sysctl > vfs.nfsd.cachetcp=3D0 > to disable use of the DRC for TCP. >=20 > The old nfs server did not use the DRC for TCP. The assumption being that > TCP layer retransmits are good enough to maintain reliable RPC transport. > Unfortuantely, you can get file corruption when the server reboots or > there is a network partitioning, if the client chooses to redo the RPC > over TCP (clients always do this after having to create a new TCP connecti= on). > In other words, vfs.nfsd.cachetcp=3D0 is roughly what the old nfsd did. >=20 > If you don't want to patch the 9.2 code, you can edit the sources > (sys/fs/nfsserver/nfs_nfsdcache.c) and change the line: > static int nfsrc_tcpnonidempotent =3D 1; > to > static int nfsrc_tcpnonidempotent =3D 0; > to do the same thing as vfs.nfsd.cachetcp=3D0 >=20 > rick > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"