From owner-freebsd-fs@FreeBSD.ORG Fri Oct 8 17:06:40 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C8F1106564A for ; Fri, 8 Oct 2010 17:06:40 +0000 (UTC) (envelope-from gvidals@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id CA0958FC13 for ; Fri, 8 Oct 2010 17:06:39 +0000 (UTC) Received: by fxm4 with SMTP id 4so535571fxm.13 for ; Fri, 08 Oct 2010 10:06:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:reply-to:date :message-id:subject:from:to:content-type; bh=OH2ZTs60mzopinUmffKQG6IXrIFMPUH+56VaCsrPXgY=; b=ismkF9jXsyXB8t5JLF8Uk42h0UGHYXZaODvIw9QQ2Nfi16IhifXaYK0sgkSW/mtMQw dY25N095ub/SxtlnG1nfwwfPpRwFEH5JxPwztCV5V/ocybt//W7g5j1CCi/4owSAXw0Q +FvffiXX8s32vbFrccmvztl0hhkckBq6OBuDE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; b=YTF5/8aZZ7+S4SF7bq4Pm9Sb/J0bE25QopdDAG6P1jFkf5vyvmi2DIhYJZgJKrukal COb4wUhxymakccXVo2wkezqej4qfJAvTE2BJLzi/tC+sBy/+2GqtsMRmULkySTkajz2+ bBkD1hKSTYDVfMncp5yaBR4ItGGbq1o63AdTU= MIME-Version: 1.0 Received: by 10.239.193.68 with SMTP id h4mr149503hbi.144.1286557598634; Fri, 08 Oct 2010 10:06:38 -0700 (PDT) Received: by 10.239.153.75 with HTTP; Fri, 8 Oct 2010 10:06:38 -0700 (PDT) Date: Fri, 8 Oct 2010 10:06:38 -0700 Message-ID: From: Gil Vidals To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: rpcsvc wcpu 500% X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gil@vidals.net List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Oct 2010 17:06:40 -0000 My FreeBSD 8.1 running 64-bit / 8 GB of RAM from two mirrored HP USB Sticks. The FreeBSD 8.1 server is a new NAS for my ESX Hosting environment. The file system I am using is ZFS and the ESX hosts connect as clients to the NAS. I am using link aggregation on two NICs. The FreeBSD NAS hung when WCPU went to 500 percent. It happened after I started a second mount point from my ESX Host to the NFS store. Since I read a thread from another soul who experience the similar situation when launching VMs on a SECOND mount point (NFS), the assumption was that FreeBSD 8.1 doesn't run more than one NFS Server and that is what causes the crash, but I've been corrected by others in this forum. FreeBSD 8.1 runs one NFS Master and Server, but many threads. I do now see that with the ps wauxH | grep nfs command. In any case, I'm back to square one. How should I go about investigating what caused the RPCSVC spike to 500% causing my VMs to hang and I had to hard reboot the server? (i tried shutting down nfsd, but that just hung). last pid: 3899; load averages: 0.00, 0.00, 0.00 up 0+15:58:33 09:21:15 62 processes: 1 running, 61 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 18M Active, 11M Inact, 2163M Wired, 4K Cache, 557M Buf, 5719M Free Swap: 3942M Total, 3942M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1214 root 24 44 0 5804K 1588K rpcsvc 2 8:38 0.00% nfsd <======= WCPU had spiked to 500% zambia# ps wauxH | grep nfs root 1213 0.0 0.0 5804 1508 ?? Is 5:23PM 0:00.03 nfsd: master (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.79 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:22.25 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.02 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.17 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.77 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.31 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:20.15 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.99 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.86 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:23.37 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:22.36 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:22.57 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:20.13 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:22.62 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.30 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:22.47 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:20.99 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:22.24 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.26 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:22.09 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.72 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:20.60 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.07 nfsd: server (nfsd) root 1214 0.0 0.0 5804 1588 ?? S 5:23PM 0:21.03 nfsd: server (nfsd) Thanks, Gil Vidals