From owner-freebsd-stable@FreeBSD.ORG Tue Mar 18 16:10:48 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6C9E106564A for ; Tue, 18 Mar 2008 16:10:48 +0000 (UTC) (envelope-from alex-goncharov@comcast.net) Received: from QMTA03.westchester.pa.mail.comcast.net (qmta03.westchester.pa.mail.comcast.net [76.96.62.32]) by mx1.freebsd.org (Postfix) with ESMTP id 7E86D8FC15 for ; Tue, 18 Mar 2008 16:10:48 +0000 (UTC) (envelope-from alex-goncharov@comcast.net) Received: from OMTA08.westchester.pa.mail.comcast.net ([76.96.62.12]) by QMTA03.westchester.pa.mail.comcast.net with comcast id 2fUU1Z0030Fqzac5305700; Tue, 18 Mar 2008 15:59:46 +0000 Received: from daland.home ([24.61.21.4]) by OMTA08.westchester.pa.mail.comcast.net with comcast id 2g0m1Z00C05H7zL3U00000; Tue, 18 Mar 2008 16:00:47 +0000 X-Authority-Analysis: v=1.0 c=1 a=mDUhab2H8L8A:10 a=rITDv7nW5hcA:10 a=BUmY6nS4aFZ3ObDiYnwA:9 a=K9iLOqqnb7zMXshCEqwA:7 a=xUAexLCCzV1GIvSgfvjGkpAgtYcA:4 a=si9q_4b84H0A:10 a=XF7b4UCPwd8A:10 Received: from algo by daland.home with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JbeF4-0002Fb-LB for freebsd-stable@freebsd.org; Tue, 18 Mar 2008 12:00:46 -0400 From: Alex Goncharov To: freebsd-stable@freebsd.org Message-Id: Sender: Alex Goncharov Date: Tue, 18 Mar 2008 12:00:46 -0400 Subject: Plentiful NFS debug messages in /var/log/messages X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Alex Goncharov List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Mar 2008 16:10:48 -0000 The system is built from source so that: ======================================== # uname -sr FreeBSD 7.0-RELEASE # cat KERNEL-CONFIG | FILTER-DEBUG options KDB options KDB_TRACE options DDB options WITNESS options WITNESS_SKIPSPIN ======================================== An NFS client (an application using an NFS file system) runs unbelievably slow, compared with when a local file system is used. The NFS server shows this: ---------------------------------------- # vmstat 4 200 procs memory page disk faults cpu r b w avm fre flt re pi po fr sr ad0 in sy cs us sy id 3 0 0 142408 317124 10 0 0 0 10 0 0 69 80 409 0 1 99 2 0 0 142408 316756 0 0 0 0 0 0 1 406 3288 1336 1 99 0 2 0 0 142408 316276 0 0 0 0 4 0 5 409 4837 1350 1 98 1 2 0 0 142408 315812 168 0 0 0 143 0 6 401 5251 1341 1 98 1 0 0 0 142408 315396 0 0 0 0 44 0 25 365 4396 1252 1 80 20 0 0 0 142408 314892 0 0 0 0 27 0 34 287 2449 1101 1 37 62 0 0 0 142408 314244 0 0 0 0 16 0 18 272 1215 1051 0 20 80 0 0 0 142408 313828 0 0 0 0 6 0 14 197 545 862 0 9 91 0 0 0 142408 313236 0 0 0 0 6 0 10 232 536 933 0 9 91 0 0 0 142408 312680 0 0 0 0 10 0 14 229 977 944 1 14 85 0 0 0 142408 312208 0 0 0 0 12 0 16 216 1053 895 0 16 84 0 0 0 142408 311936 0 0 0 0 4 0 7 133 410 668 0 6 93 ---------------------------------------- These messages are meanwhile streaming into `/var/log/messages': -------------------- Mar 18 11:47:42 fasolt kernel: --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c4a2b, esp = 0xbfbfeb2c, ebp = 0xbfb feb48 --- Mar 18 11:47:42 fasolt kernel: uma_zalloc_arg: zone "mbuf" with the following non-sleepable locks held: Mar 18 11:47:42 fasolt kernel: exclusive sleep mutex nfsd_mtx r = 0 (0xc09456a0) locked @ /mnt/wdx/freebsd/7.0/usr/src/s ys/nfsserver/nfs_srvsock.c:654 KDB: stack backtrace: db_trace_self_wrapper(c08485d5,d5f1faf8,c0615eed,c084895a,d5f1fb0c,...) at db_trace_self_ kdb_backtrace(c084895a,d5f1fb0c,4,1,0,...) at kdb_backtrace+0x29 witness_warn(5,0,c085e59c,c084c9c8,d5f1fb1c,...) at witness_warn+0x1cd uma_zalloc_arg(c1067d20,d5f1fb70,2,8,c2bf94a4,...) at uma_zalloc_arg+0x34 nfs_realign(c09456a0,0,c08595b8,28e,0,...) at nfs_realign+0x6f nfsrv_rcv(c2daf948,c2bf9480,2,160,0,...) at nfsrv_rcv+0x42a nfssvc(c2c0caa0,d5f1fcfc,8,c2c0caa0,c089eac8,...) at nfssvc+0x664 syscall(d5f1fd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 -------------------- I could, of course, rebuild the kernel without the debug options but perhaps somebody sees a problem with this behavior and can advise me on a better course of actions. Thanks, -- Alex -- alex-goncharov@comcast.net --