From owner-freebsd-stable@FreeBSD.ORG Thu Feb 14 16:42:54 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EF1E148D; Thu, 14 Feb 2013 16:42:54 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 930E1E89; Thu, 14 Feb 2013 16:42:54 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EACMTHVGDaFvO/2dsb2JhbABEhkm6NnOCHwEBBSMEUhsOCgICDRkCWQYTiBKrBZJFgSOPToETA4hmjT6QU4Mlggk X-IronPort-AV: E=Sophos;i="4.84,665,1355115600"; d="scan'208";a="14077934" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu.net.uoguelph.ca with ESMTP; 14 Feb 2013 11:41:45 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id CA028B4080; Thu, 14 Feb 2013 11:41:45 -0500 (EST) Date: Thu, 14 Feb 2013 11:41:45 -0500 (EST) From: Rick Macklem To: Marc Fournier Message-ID: <1422247357.3019523.1360860105806.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <9A149E78-BB4F-414D-AAE5-331C5934FF82@hub.org> Subject: Re: 9-STABLE -> NFS -> NetAPP: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: Konstantin Belousov , freebsd-stable@freebsd.org, Kostik Belousov , John Baldwin X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Feb 2013 16:42:55 -0000 Marc Fournier wrote: > On 2013-02-13, at 3:54 PM, Rick Macklem wrote: >=20 > >> > > The pid that is in "T" state for the "ps auxlH". >=20 > Different server, last kernel update on Jan 22nd, https process this > time instead of du last time. >=20 > I've attached: >=20 > ps auxlH > ps auxlH of just the processes that are in TJ state (6 httpd servers) > procstat output for each of the 6 process >=20 >=20 >=20 >=20 > They are included as attachments =E2=80=A6 if these don't make it through= , let > me know, just figured I'd try and keep it compact ... Ok, I took a look and the interesting process seems to be 16693. It is stopped ("T" state) and several of its threads (22, but not all) have a procstat like this: 16693 104135 httpd - mi_switch+0x186 thread_suspe= nd_check+0x19f sleepq_catch_signals+0x1c5 sleepq_timedwait_sig+0x19 _sleep+0x2ca clnt_vc_call+0x763 clnt_reconnect= _call+0xfb newnfs_request+0xadb nfscl_request+0x72 nfsrpc_accessrpc+0x1df nfs34_acc= ess_otw+0x56 nfs_access+0x306 vn_open_cred+0x5a8 kern_openat+0x20a amd64_syscall+0x540 Xfast_syscall+0= xf7=20 The sleep in clnt_vc_call is waiting for an RPC reply (while a vnode lock is held) with PCATCH | PBDRY flags, since it interruptible. I can see that the thread_suspend_check() has a 1 argument (return_instead = =3D=3D 1), since there is only one call to thread_suspend_check() in sleepq_catch_sign= als(). When looking at thread_suspend_check(), I basically got lost, although it seems that it can only "return_instead" if there is a single thread and not multiple threads doing this. If these threads are stuck here and won't return from msleep(), that would explain the hang. If they would wakeup and return from the msleep() when a wakeup occurs, it would suggest that there is a lost reply or similar, so the wakeup isn't occurring. I also don't know if a timeout of the msleep() will still occur and make the msleep() return? Although it wasn't done to fix this, it looks like jhb@'s recent patch to head (r246417) might fix this, since it reworks how STOP signals are handle= d for interruptible mounts. Hopefully kib or jhb can provide more insight. Btw Marc, if you just want this problem to go away, I suspect getting rid of the "intr" mount option would do that. rick