From owner-freebsd-stable@FreeBSD.ORG Tue Sep 16 06:02:15 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E5C3106566C for ; Tue, 16 Sep 2008 06:02:15 +0000 (UTC) (envelope-from gphoto6@gmail.com) Received: from mail-gx0-f17.google.com (mail-gx0-f17.google.com [209.85.217.17]) by mx1.freebsd.org (Postfix) with ESMTP id 41CC58FC1D for ; Tue, 16 Sep 2008 06:02:14 +0000 (UTC) (envelope-from gphoto6@gmail.com) Received: by gxk10 with SMTP id 10so24825266gxk.19 for ; Mon, 15 Sep 2008 23:02:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=f/5WkKTBUY0WlWw8dAxvnByMlc0lwsFaPCi744nX0j0=; b=Jp4kmCGpvK5NrYfcnKGSnxcZEEugC0HX2nfjY6gST8NhsZ7HwncLA8siXdBcFSLHG+ 2pkL8kgn0PO0y/bv1FVfp5lGlBA8GUiwsHzobS5UTI3p+8pIZPfCWY9sS3YCubaIC/0q yg0XpdBlRUUscA5i3XwKMz9AOPl4RZT6fW84k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=DLPmE84U+8M2z4X2aMOxK/7gWPNZ7DSHKP++DHgg5oefo4A6kPc/KRuVBspj6g+RjH dCLahj4n568eIPplg6CW8JZpUCaaYqhBaB00N7xmKtvYNcLPSNlmGS5ImanJvCbYaoxm SHEUN0I8FoeZBuzYRTO6H1gyXVEeDi4g2bn28= Received: by 10.151.141.16 with SMTP id t16mr897915ybn.4.1221544934262; Mon, 15 Sep 2008 23:02:14 -0700 (PDT) Received: by 10.151.41.19 with HTTP; Mon, 15 Sep 2008 23:02:14 -0700 (PDT) Message-ID: <1f51039c0809152302s2e6c1471n89588b058069f73d@mail.gmail.com> Date: Tue, 16 Sep 2008 14:02:14 +0800 From: "Tim Chen" To: freebsd-stable@freebsd.org In-Reply-To: <200809151606.23933.jhb@freebsd.org> MIME-Version: 1.0 References: <1f51039c0809150857l50b6be8eu848e21189a4175d6@mail.gmail.com> <200809151606.23933.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Suddenly frozen fcntl/stat call on NFS over TCP with MTU 9000 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Sep 2008 06:02:15 -0000 On Tue, Sep 16, 2008 at 4:06 AM, John Baldwin wrote: > On Monday 15 September 2008 11:57:02 am Tim Chen wrote: > > Currently I was running a mail server using a netapp filer as backend > > storage. > > >From time to time, the whole system get stuck and lasted for 3-5 > minutes. > > But > > after that, everything recovers normally. During the "stuck" moment, > using > > ps > > auxw shows 200-300 of mail delivery agent(MDA) processes staying in "D" > > status. > > The command df certainly does not reponse either. > > Can you use 'ps axl' to determine the wait mesg ("wchan") of the stuck > threads > when they hang? If it is "lockf", then make sure you have an up-to-date > RELENG_6 kernel as there was a recent fix for a "lockf" hang. > Thanks for your suggestion. After trying to 'ps axl', it seems all the "D status" process were in nfs,nfsreq,nfsreq. Can you give some hint how to keep delving the problem? My system is RELENG_7 within one week, I always make world to keep my system up to date. > > Alternatively, if things are stuck in "nfsreq", it may be useful to use > tcpdump to look at the NFS requests your client is making. nfsstat can > also > be useful as you can see which counters are increasing during a hang. > > When system was stuck, counters of nfsstat grows slowly. It seems only read, write, create, remove in RPC counts were increased. As to tcpdump, since I am not familiar with that, I will try to read some doc and make some tests. Thanks very much for your kindly help. Hope the problem can be solved soon. Sincerely, Tim Chen