From owner-freebsd-questions@FreeBSD.ORG Sun Dec 9 18:40:53 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DD25ADAC for ; Sun, 9 Dec 2012 18:40:53 +0000 (UTC) (envelope-from matsuoka@nj-k.org) Received: from smtp.nj-k.org (nj-k.org [219.117.240.4]) by mx1.freebsd.org (Postfix) with ESMTP id 7E15F8FC12 for ; Sun, 9 Dec 2012 18:40:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.nj-k.org (Postfix) with ESMTP id C37E9417FC for ; Mon, 10 Dec 2012 03:40:51 +0900 (JST) X-Virus-Scanned: Debian amavisd-new at smtp.nj-k.org Received: from smtp.nj-k.org ([127.0.0.1]) by localhost (debian.nj-k.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id utGAqIE7NhyX for ; Mon, 10 Dec 2012 03:40:50 +0900 (JST) Received: from www.nj-k.org (freebsd.nj-k.org [192.168.3.35]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.nj-k.org (Postfix) with ESMTPSA id 05B974070C for ; Mon, 10 Dec 2012 03:40:49 +0900 (JST) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 10 Dec 2012 03:40:49 +0900 From: Takahide Matsuoka To: Subject: Kernel panic after excessive access to NFSv4 as client (FreeBSD-9.1) Message-ID: <8f2896c16bfd5ca064437a528e216b3e@nj-k.org> X-Sender: matsuoka@nj-k.org User-Agent: Roundcube Webmail/RCMAIL_VERSION X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Dec 2012 18:40:54 -0000 Hi, My system has been encountering kernel panic every time it accesses large number of files over NFSv4. The backtrace left after reboot (I haven't saved the core dump): Dec 4 17:35:56 freebsd syslogd: kernel boot file is /boot/kernel/kernel Dec 4 17:35:56 freebsd kernel: panic: stack overflow detected; backtrace may be corrupted Dec 4 17:35:56 freebsd kernel: cpuid = 1 Dec 4 17:35:56 freebsd kernel: KDB: stack backtrace: Dec 4 17:35:56 freebsd kernel: #0 0xffffffff809274a6 at kdb_backtrace+0x66 Dec 4 17:35:56 freebsd kernel: #1 0xffffffff808f13fe at panic+0x1ce Dec 4 17:35:56 freebsd kernel: #2 0xffffffff8091a452 at __stack_chk_fail+0x12 Dec 4 17:35:56 freebsd kernel: #3 0xffffffff81613ee7 at krb5_unwrap_old+0x407 Dec 4 17:35:56 freebsd kernel: #4 0xffffffff8161413c at krb5_unwrap+0x12c Dec 4 17:35:56 freebsd kernel: #5 0xffffffff816446e4 at xdr_rpc_gss_unwrap_data+0x164 Dec 4 17:35:56 freebsd kernel: #6 0xffffffff81642a9d at rpc_gss_validate+0x1bd Dec 4 17:35:56 freebsd kernel: #7 0xffffffff80ad09be at clnt_vc_call+0x8de Dec 4 17:35:56 freebsd kernel: #8 0xffffffff80acec0b at clnt_reconnect_call+0xfb Dec 4 17:35:56 freebsd kernel: #9 0xffffffff807f07e5 at newnfs_request+0x595 Dec 4 17:35:56 freebsd kernel: #10 0xffffffff80826c42 at nfscl_request+0x72 Dec 4 17:35:56 freebsd kernel: #11 0xffffffff808117dd at nfsrpc_lookup+0x1bd Dec 4 17:35:56 freebsd kernel: #12 0xffffffff80821982 at nfs_lookup+0x292 Dec 4 17:35:56 freebsd kernel: #13 0xffffffff80c6ee60 at VOP_LOOKUP_APV+0x40 Dec 4 17:35:56 freebsd kernel: #14 0xffffffff8097abaf at lookup+0x41f Dec 4 17:35:56 freebsd kernel: #15 0xffffffff8097bcab at namei+0x4db Dec 4 17:35:56 freebsd kernel: #16 0xffffffff8098d203 at kern_statat_vnhook+0xb3 Dec 4 17:35:56 freebsd kernel: #17 0xffffffff8098d3c5 at kern_statat+0x15 Dec 4 17:35:56 freebsd kernel: Uptime: 21h22m34s Dec 4 17:35:56 freebsd kernel: Automatic reboot in 15 seconds - press a key on the console to abort Dec 4 17:35:56 freebsd kernel: Rebooting... Here's another backtrace: Dec 2 00:15:33 freebsd syslogd: kernel boot file is /boot/kernel/kernel Dec 2 00:15:33 freebsd kernel: panic: stack overflow detected; backtrace may be corrupted Dec 2 00:15:33 freebsd kernel: cpuid = 3 Dec 2 00:15:33 freebsd kernel: KDB: stack backtrace: Dec 2 00:15:33 freebsd kernel: #0 0xffffffff809274a6 at kdb_backtrace+0x66 Dec 2 00:15:33 freebsd kernel: #1 0xffffffff808f13fe at panic+0x1ce Dec 2 00:15:33 freebsd kernel: #2 0xffffffff8091a452 at __stack_chk_fail+0x12 Dec 2 00:15:33 freebsd kernel: #3 0xffffffff81613ee7 at krb5_unwrap_old+0x407 Dec 2 00:15:33 freebsd kernel: #4 0xffffffff8161413c at krb5_unwrap+0x12c Dec 2 00:15:33 freebsd kernel: #5 0xffffffff816446e4 at xdr_rpc_gss_unwrap_data+0x164 Dec 2 00:15:33 freebsd kernel: #6 0xffffffff81642a9d at rpc_gss_validate+0x1bd Dec 2 00:15:33 freebsd kernel: #7 0xffffffff80ad09be at clnt_vc_call+0x8de Dec 2 00:15:33 freebsd kernel: #8 0xffffffff80acec0b at clnt_reconnect_call+0xfb Dec 2 00:15:33 freebsd kernel: #9 0xffffffff807f07e5 at newnfs_request+0x595 Dec 2 00:15:33 freebsd kernel: #10 0xffffffff80826c42 at nfscl_request+0x72 Dec 2 00:15:33 freebsd kernel: #11 0xffffffff8080fcd4 at nfsrpc_write+0x4e4 Dec 2 00:15:33 freebsd kernel: #12 0xffffffff8081ea72 at ncl_writerpc+0x62 Dec 2 00:15:33 freebsd kernel: #13 0xffffffff80829f26 at ncl_doio+0x196 Dec 2 00:15:33 freebsd kernel: #14 0xffffffff8082e002 at nfssvc_iod+0xc2 Dec 2 00:15:33 freebsd kernel: #15 0xffffffff808c1f0f at fork_exit+0x11f Dec 2 00:15:33 freebsd kernel: #16 0xffffffff80bc8c3e at fork_trampoline+0xe Dec 2 00:15:33 freebsd kernel: Uptime: 2h6m33s Dec 2 00:15:33 freebsd kernel: Automatic reboot in 15 seconds - press a key on the console to abort Dec 2 00:15:33 freebsd kernel: Rebooting... # uname -a FreeBSD freebsd.nj-k.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #5 r243328: Sat Nov 24 03:01:20 JST 2012 root@:/usr/obj/usr/src/sys/GENERIC amd64 Disabling rpcbind stopped the kernel panic for a while. # cat /etc/rc.conf (truncated lines for apache and etc.) hostname="freebsd.nj-k.org" keymap="us.iso.kbd" ifconfig_re0="DHCP" ntpd_enable="YES" ntpd_sync_on_start="YES" nscd_enable="YES" ##rpcbind_enable="YES" gssd_enable="YES" nfsuserd_enable="YES" nfsuserd_flags="-domain nj-k.org" nfscbd_enable="YES" nfs_client_enable="YES" sendmail_enable="YES" # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable dumpdev="NO" # cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/vtbd0p2 / ufs rw 1 1 /dev/vtbd0p3 none swap sw 0 0 nfs.nj-k.org:/ /mnt/nfs nfs rw,nfsv4,sec=krb5p,rsize=32768,wsize=32768,intr 0 0 # klist Credentials cache: FILE:/tmp/krb5cc_0 Principal: host/freebsd.nj-k.org@NJ-K.ORG Issued Expires Principal Dec 10 00:45:00 Dec 10 10:45:00 krbtgt/NJ-K.ORG@NJ-K.ORG Dec 10 00:46:00 Dec 10 10:45:00 ldap/debian.nj-k.org@NJ-K.ORG I previously had the same problem with 9.0-RELEASE, and thought installing 9-STABLE might solve the problem. After deinstalling security/krb5, I have rebuilt world, kernel and ports. I have following lines in /etc/krb5.conf under [libdefaults]: forwardable = true dns_lookup_kdc = true dns_lookup_realm = true allow_weak_crypto = true default_etypes = des-cbc-crc default_etypes_des = des-cbc-crc fcc-mit-ticketflags = true Lines after allow_weak_crypto were added because I thought the problem was related to the encryption type. The user of the service which accesses the nfs is www which is in /etc/passwd and ldap # id www (truncated some groups) uid=80(www) gid=80(www) groups=80(www),110(ssl-cert) # su www -c 'KRB5CCNAME=FILE:/tmp/krb5cc_80 klist -v' Credentials cache: FILE:/tmp/krb5cc_80 Principal: www@NJ-K.ORG Cache version: 4 Server: krbtgt/NJ-K.ORG@NJ-K.ORG Client: www@NJ-K.ORG Ticket etype: des-cbc-crc, kvno 1 Ticket length: 204 Auth time: Dec 10 00:55:00 2012 End time: Dec 10 10:55:00 2012 Ticket flags: forwardable, initial, pre-authenticated Addresses: addressless I am new to FreeBSD (I installed my first FreeBSD system 6 months ago), but willing to get adapted to it. This system is on kvm so I can freely create another instance of the system for test. Any help or advice is appreciated!!! Please tell me if this mail was not a well-formed question (this is my first mail to any mailing lists). Best regards, Tak