From owner-freebsd-bugs@FreeBSD.ORG Thu Jan 16 19:10:00 2014 Return-Path: Delivered-To: freebsd-bugs@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB46193C for ; Thu, 16 Jan 2014 19:10:00 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 956651E71 for ; Thu, 16 Jan 2014 19:10:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s0GJA0EP000121 for ; Thu, 16 Jan 2014 19:10:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id s0GJA0Oj000120; Thu, 16 Jan 2014 19:10:00 GMT (envelope-from gnats) Resent-Date: Thu, 16 Jan 2014 19:10:00 GMT Resent-Message-Id: <201401161910.s0GJA0Oj000120@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, John Hickey Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 18B86931 for ; Thu, 16 Jan 2014 19:09:48 +0000 (UTC) Received: from oldred.freebsd.org (oldred.freebsd.org [IPv6:2001:1900:2254:206a::50:4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DDE701E67 for ; Thu, 16 Jan 2014 19:09:47 +0000 (UTC) Received: from oldred.freebsd.org ([127.0.1.6]) by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id s0GJ9lon081570 for ; Thu, 16 Jan 2014 19:09:47 GMT (envelope-from nobody@oldred.freebsd.org) Received: (from nobody@localhost) by oldred.freebsd.org (8.14.5/8.14.5/Submit) id s0GJ9loJ081566; Thu, 16 Jan 2014 19:09:47 GMT (envelope-from nobody) Message-Id: <201401161909.s0GJ9loJ081566@oldred.freebsd.org> Date: Thu, 16 Jan 2014 19:09:47 GMT From: John Hickey To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: kern/185827: Kernel Panic after upgrading NFS server from FreeBSD 9.1 to 9.2 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jan 2014 19:10:00 -0000 >Number: 185827 >Category: kern >Synopsis: Kernel Panic after upgrading NFS server from FreeBSD 9.1 to 9.2 >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Jan 16 19:10:00 UTC 2014 >Closed-Date: >Last-Modified: >Originator: John Hickey >Release: FreeBSD 9.2-RELEASE-p3 #0 r260700: Wed Jan 15 22:17:33 PST 2014 root@:/usr/obj/usr/src/sys/USERS92 i386 >Organization: USC - ISI / DETER Project >Environment: FreeBSD 9.2-RELEASE-p3 #0 r260700: Wed Jan 15 22:17:33 PST 2014 root@:/usr/obj/usr/src/sys/USERS92 i386 >Description: After upgrading a heavily used NFS server from FreeBSD 9.1 i386 to 9.2 i386, we get a repeatable kernel panic around the time the NFS server starts during boot. This happened at every boot with the same trace until I went back to FreeBSD 9.1: panic: stack overflow detected; backtrace may be corrupted cpuid = 5 KDB: stack backtrace: #0 0xc0b180ef at kdb_backtrace+0x4f #1 0xc0adf36f at panic+0x16f #2 0xc0b0a482 at __stack_chk_fail+0x12 #3 0xc0cc9e83 at fha_assign+0x433 #4 0xc0a09ee0 at fhanew_assign+0x20 #5 0xc0ce2237 at svc_run_internal+0x767 #6 0xc0ce25f0 at svc_thread_start+0x10 #7 0xc0aaad9f at fork_exit+0xcf #8 0xc0f36664 at fork_trampoline+0x8 Uptime: 7m22s Physical memory: 3558 MB Dumping 265 MB: 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10 #0 doadump (textdump=1) at pcpu.h:249 249 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=1) at pcpu.h:249 #1 0xc0adf0b5 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:449 #2 0xc0adf3b2 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:637 #3 0xc0b0a482 in __stack_chk_fail () at /usr/src/sys/kern/stack_protector.c:17 #4 0xc0cc9e83 in fha_assign (this_thread=0xcf274500, req=0xcf328000, softc=0xc12cd860) at /usr/src/sys/nfs/nfs_fha.c:463 #5 0xc0a09ee0 in fhanew_assign (this_thread=0xcf274500, req=0xcf328000) at /usr/src/sys/fs/nfsserver/nfs_fha_new.c:271 #6 0xc0ce2237 in svc_run_internal (pool=0xcf272980, ismaster=0) at /usr/src/sys/rpc/svc.c:1109 #7 0xc0ce25f0 in svc_thread_start (arg=0xcf272980) at /usr/src/sys/rpc/svc.c:1200 #8 0xc0aaad9f in fork_exit (callout=0xc0ce25e0 , arg=0xcf272980, frame=0xe876dd08) at /usr/src/sys/kern/kern_fork.c:992 #9 0xc0f36664 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:279 (kgdb) >How-To-Repeat: I am not sure how to repeat this outside of our environment. We have a large number to NFS clients served by this machine. It could have to do with a race condition exposed by such a large number of connections happening at boot time. >Fix: >Release-Note: >Audit-Trail: >Unformatted: