Date: Mon, 10 Dec 2012 03:40:49 +0900 From: Takahide Matsuoka <matsuoka@nj-k.org> To: <freebsd-questions@freebsd.org> Subject: Kernel panic after excessive access to NFSv4 as client (FreeBSD-9.1) Message-ID: <8f2896c16bfd5ca064437a528e216b3e@nj-k.org>
next in thread | raw e-mail | index | archive | help
Hi,
My system has been encountering kernel panic every time it accesses
large number of files over NFSv4.
The backtrace left after reboot (I haven't saved the core dump):
Dec 4 17:35:56 freebsd syslogd: kernel boot file is
/boot/kernel/kernel
Dec 4 17:35:56 freebsd kernel: panic: stack overflow detected;
backtrace may be corrupted
Dec 4 17:35:56 freebsd kernel: cpuid = 1
Dec 4 17:35:56 freebsd kernel: KDB: stack backtrace:
Dec 4 17:35:56 freebsd kernel: #0 0xffffffff809274a6 at
kdb_backtrace+0x66
Dec 4 17:35:56 freebsd kernel: #1 0xffffffff808f13fe at panic+0x1ce
Dec 4 17:35:56 freebsd kernel: #2 0xffffffff8091a452 at
__stack_chk_fail+0x12
Dec 4 17:35:56 freebsd kernel: #3 0xffffffff81613ee7 at
krb5_unwrap_old+0x407
Dec 4 17:35:56 freebsd kernel: #4 0xffffffff8161413c at
krb5_unwrap+0x12c
Dec 4 17:35:56 freebsd kernel: #5 0xffffffff816446e4 at
xdr_rpc_gss_unwrap_data+0x164
Dec 4 17:35:56 freebsd kernel: #6 0xffffffff81642a9d at
rpc_gss_validate+0x1bd
Dec 4 17:35:56 freebsd kernel: #7 0xffffffff80ad09be at
clnt_vc_call+0x8de
Dec 4 17:35:56 freebsd kernel: #8 0xffffffff80acec0b at
clnt_reconnect_call+0xfb
Dec 4 17:35:56 freebsd kernel: #9 0xffffffff807f07e5 at
newnfs_request+0x595
Dec 4 17:35:56 freebsd kernel: #10 0xffffffff80826c42 at
nfscl_request+0x72
Dec 4 17:35:56 freebsd kernel: #11 0xffffffff808117dd at
nfsrpc_lookup+0x1bd
Dec 4 17:35:56 freebsd kernel: #12 0xffffffff80821982 at
nfs_lookup+0x292
Dec 4 17:35:56 freebsd kernel: #13 0xffffffff80c6ee60 at
VOP_LOOKUP_APV+0x40
Dec 4 17:35:56 freebsd kernel: #14 0xffffffff8097abaf at lookup+0x41f
Dec 4 17:35:56 freebsd kernel: #15 0xffffffff8097bcab at namei+0x4db
Dec 4 17:35:56 freebsd kernel: #16 0xffffffff8098d203 at
kern_statat_vnhook+0xb3
Dec 4 17:35:56 freebsd kernel: #17 0xffffffff8098d3c5 at
kern_statat+0x15
Dec 4 17:35:56 freebsd kernel: Uptime: 21h22m34s
Dec 4 17:35:56 freebsd kernel: Automatic reboot in 15 seconds - press
a key on the console to abort
Dec 4 17:35:56 freebsd kernel: Rebooting...
Here's another backtrace:
Dec 2 00:15:33 freebsd syslogd: kernel boot file is
/boot/kernel/kernel
Dec 2 00:15:33 freebsd kernel: panic: stack overflow detected;
backtrace may be corrupted
Dec 2 00:15:33 freebsd kernel: cpuid = 3
Dec 2 00:15:33 freebsd kernel: KDB: stack backtrace:
Dec 2 00:15:33 freebsd kernel: #0 0xffffffff809274a6 at
kdb_backtrace+0x66
Dec 2 00:15:33 freebsd kernel: #1 0xffffffff808f13fe at panic+0x1ce
Dec 2 00:15:33 freebsd kernel: #2 0xffffffff8091a452 at
__stack_chk_fail+0x12
Dec 2 00:15:33 freebsd kernel: #3 0xffffffff81613ee7 at
krb5_unwrap_old+0x407
Dec 2 00:15:33 freebsd kernel: #4 0xffffffff8161413c at
krb5_unwrap+0x12c
Dec 2 00:15:33 freebsd kernel: #5 0xffffffff816446e4 at
xdr_rpc_gss_unwrap_data+0x164
Dec 2 00:15:33 freebsd kernel: #6 0xffffffff81642a9d at
rpc_gss_validate+0x1bd
Dec 2 00:15:33 freebsd kernel: #7 0xffffffff80ad09be at
clnt_vc_call+0x8de
Dec 2 00:15:33 freebsd kernel: #8 0xffffffff80acec0b at
clnt_reconnect_call+0xfb
Dec 2 00:15:33 freebsd kernel: #9 0xffffffff807f07e5 at
newnfs_request+0x595
Dec 2 00:15:33 freebsd kernel: #10 0xffffffff80826c42 at
nfscl_request+0x72
Dec 2 00:15:33 freebsd kernel: #11 0xffffffff8080fcd4 at
nfsrpc_write+0x4e4
Dec 2 00:15:33 freebsd kernel: #12 0xffffffff8081ea72 at
ncl_writerpc+0x62
Dec 2 00:15:33 freebsd kernel: #13 0xffffffff80829f26 at
ncl_doio+0x196
Dec 2 00:15:33 freebsd kernel: #14 0xffffffff8082e002 at
nfssvc_iod+0xc2
Dec 2 00:15:33 freebsd kernel: #15 0xffffffff808c1f0f at
fork_exit+0x11f
Dec 2 00:15:33 freebsd kernel: #16 0xffffffff80bc8c3e at
fork_trampoline+0xe
Dec 2 00:15:33 freebsd kernel: Uptime: 2h6m33s
Dec 2 00:15:33 freebsd kernel: Automatic reboot in 15 seconds - press
a key on the console to abort
Dec 2 00:15:33 freebsd kernel: Rebooting...
# uname -a
FreeBSD freebsd.nj-k.org 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #5
r243328: Sat Nov 24 03:01:20 JST 2012
root@:/usr/obj/usr/src/sys/GENERIC amd64
Disabling rpcbind stopped the kernel panic for a while.
# cat /etc/rc.conf (truncated lines for apache and etc.)
hostname="freebsd.nj-k.org"
keymap="us.iso.kbd"
ifconfig_re0="DHCP"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
nscd_enable="YES"
##rpcbind_enable="YES"
gssd_enable="YES"
nfsuserd_enable="YES"
nfsuserd_flags="-domain nj-k.org"
nfscbd_enable="YES"
nfs_client_enable="YES"
sendmail_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
# cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/vtbd0p2 / ufs rw 1 1
/dev/vtbd0p3 none swap sw 0 0
nfs.nj-k.org:/ /mnt/nfs nfs rw,nfsv4,sec=krb5p,rsize=32768,wsize=32768,intr 0 0
# klist
Credentials cache: FILE:/tmp/krb5cc_0
Principal: host/freebsd.nj-k.org@NJ-K.ORG
Issued Expires Principal
Dec 10 00:45:00 Dec 10 10:45:00 krbtgt/NJ-K.ORG@NJ-K.ORG
Dec 10 00:46:00 Dec 10 10:45:00 ldap/debian.nj-k.org@NJ-K.ORG
I previously had the same problem with 9.0-RELEASE, and thought
installing 9-STABLE might solve the problem.
After deinstalling security/krb5, I have rebuilt world, kernel and
ports.
I have following lines in /etc/krb5.conf under [libdefaults]:
forwardable = true
dns_lookup_kdc = true
dns_lookup_realm = true
allow_weak_crypto = true
default_etypes = des-cbc-crc
default_etypes_des = des-cbc-crc
fcc-mit-ticketflags = true
Lines after allow_weak_crypto were added because I thought the problem
was related to the encryption type.
The user of the service which accesses the nfs is www which is in
/etc/passwd and ldap
# id www (truncated some groups)
uid=80(www) gid=80(www) groups=80(www),110(ssl-cert)
# su www -c 'KRB5CCNAME=FILE:/tmp/krb5cc_80 klist -v'
Credentials cache: FILE:/tmp/krb5cc_80
Principal: www@NJ-K.ORG
Cache version: 4
Server: krbtgt/NJ-K.ORG@NJ-K.ORG
Client: www@NJ-K.ORG
Ticket etype: des-cbc-crc, kvno 1
Ticket length: 204
Auth time: Dec 10 00:55:00 2012
End time: Dec 10 10:55:00 2012
Ticket flags: forwardable, initial, pre-authenticated
Addresses: addressless
I am new to FreeBSD (I installed my first FreeBSD system 6 months ago),
but willing to get adapted to it.
This system is on kvm so I can freely create another instance of the
system for test.
Any help or advice is appreciated!!!
Please tell me if this mail was not a well-formed question (this is my
first mail to any mailing lists).
Best regards,
Tak
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8f2896c16bfd5ca064437a528e216b3e>