From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 25 03:36:02 2004 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 21E0116A4CF for ; Fri, 25 Jun 2004 03:36:02 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8977B43D31 for ; Fri, 25 Jun 2004 03:36:01 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5P3ZfDk027260; Thu, 24 Jun 2004 23:35:41 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5P3ZemC027257; Thu, 24 Jun 2004 23:35:40 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 24 Jun 2004 23:35:40 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Danny Braniss In-Reply-To: <20040623163251.71F4A43D2F@mx1.FreeBSD.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Matt Freitag cc: freebsd-hackers@freebsd.org Subject: Re: waiting on sbwait X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jun 2004 03:36:02 -0000 On Wed, 23 Jun 2004, Danny Braniss wrote: > sometimes we get > load: 0.04 cmd: dmesg 13453 [nfsrcvlk] 0.00u 0.00s 0% 148k > > and looking through the code, there might be some connection between > sbwait and nfsrcvlk, but i doubt that it's sockets that im running out > off, neither mbufs, since: > > foundation> netstat -m > 326/1216/26624 mbufs in use (current/peak/max): > 326 mbufs allocated to data > 321/428/6656 mbuf clusters in use (current/peak/max) > 1160 Kbytes allocated to network (5% of mb_map in use) > 0 requests for memory denied > 0 requests for memory delayed > 0 calls to protocol drain routines > > also, the process enters sbwait either in sosend or soreceive, make me > believe that it's some resource, rather than data, that is missing. > > the fact that this 'unresponsivness' happens sometimes is making this > rather challenging, but try to tell this to the users :-) sbwait() occurs when a thread is blocked in a socket waiting for space in the socket to send, or for data in the socket on a receive. This can happen either because a process is directly performing socket I/O -- for example, sending or receiving on a TCP or UDP socket -- or, it can happen when a process is using a facility that performs socket I/O in its kernel thread. For example, the NFS client. So the sbwait state could be a result of filled buffers of NFS. If I had to guess, it might well be NFS. However, there are actually ways to tell :-). The easiest is to compile your kernel with DDB, and when a process hangs with those symptoms, break into the debugger and do a trace on its pid. You'll get back a stack trace. If it's using a send/recv system call that terminates in the socket code without hitting VFS/NFS, it's blocked on network I/O, perhaps because it's sending or receiving a lot of data and hasn't finished. If you see it pass through NFS-related functions, then it's waiting for NFS network I/O, which could reflect a busy NFS server, network segment, packet loss, etc. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research