Date: Fri, 23 Sep 2011 21:27:01 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Clinton Adams <clinton.adams@gmail.com> Cc: freebsd-fs@freebsd.org Subject: Re: kernel panics with RPCSEC_GSS Message-ID: <1461855405.1940757.1316827621857.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <CAEuopLYJ6fvZ_7hCHQdjKgdNC39kzK1DwyGib7XvvrqE5Pqozw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_1940756_505940954.1316827621854 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Clinton Adams wrote: > Hi, > > On 8.2-RELEASE-p2, kernel occassionaly panics during relatively high > nfs usage (usually morning logins). Frequency of crashes have > decreased as we have reduced the number of clients, about twice a week > with 10 clients versus daily with 15. > > Server is running nfsv4 with mit kerberos, clients are linux (ubuntu > 10.04). > > Backtraces from last 2 cores - > > #1 0xffffffff805cbb5e in boot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:419 > #2 0xffffffff805cbf91 in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:592 > #3 0xffffffff808d25c0 in trap_fatal (frame=0xc, eva=Variable "eva" is > not available. > ) > at /usr/src/sys/amd64/amd64/trap.c:783 > #4 0xffffffff808d299f in trap_pfault (frame=0xffffff8096bb7790, > usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:699 > #5 0xffffffff808d2e7f in trap (frame=0xffffff8096bb7790) > at /usr/src/sys/amd64/amd64/trap.c:449 > #6 0xffffffff808baf74 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:224 > #7 0xffffffff807db8d8 in svc_rpc_gss_forget_client (client=0x0) > at /usr/src/sys/rpc/rpcsec_gss/svc_rpcsec_gss.c:616 Oops, I realized that if multiple threads did the call at line#642 concurrently, it could try to remove it from the tailq twice. Please try this attached patch instead of the one I posted a few minutes ago (I think it avoids this race). Thanks for reporting this and please let us know if this patch helps, rick > #8 0xffffffff807dc1c3 in svc_rpc_gss (rqst=0xffffff005708c000, > msg=0xffffff8096bb7b20) at > /usr/src/sys/rpc/rpcsec_gss/svc_rpcsec_gss.c:642 > #9 0xffffffff807d49d3 in svc_run_internal (pool=0xffffff003d03d600, > ismaster=0) at /usr/src/sys/rpc/svc.c:837 > #10 0xffffffff807d518b in svc_thread_start (arg=Variable "arg" is not > available. > ) > at /usr/src/sys/rpc/svc.c:1200 > #11 0xffffffff805a2798 in fork_exit ( > callout=0xffffffff807d5180 <svc_thread_start>, arg=0xffffff003d03d600, > frame=0xffffff8096bb7c40) at /usr/src/sys/kern/kern_fork.c:845 > #12 0xffffffff808bb43e in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:565 > > > #1 0xffffffff805cbabe in boot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:419 > #2 0xffffffff805cbed3 in panic (fmt=0x0) > at /usr/src/sys/kern/kern_shutdown.c:592 > #3 0xffffffff808d239d in trap_fatal (frame=0xffffff0004c89460, > eva=Variable "eva" is not available. > ) > at /usr/src/sys/amd64/amd64/trap.c:783 > #4 0xffffffff808d275f in trap_pfault (frame=0xffffff8096c0d790, > usermode=0) > at /usr/src/sys/amd64/amd64/trap.c:699 > #5 0xffffffff808d2b5f in trap (frame=0xffffff8096c0d790) > at /usr/src/sys/amd64/amd64/trap.c:449 > #6 0xffffffff808bada4 in calltrap () > at /usr/src/sys/amd64/amd64/exception.S:224 > #7 0xffffffff807db856 in svc_rpc_gss_forget_client > (client=0xffffff001c015200) > at atomic.h:158 > #8 0xffffffff807dc0e3 in svc_rpc_gss (rqst=0xffffff0004a24000, > msg=0xffffff8096c0db20) at > /usr/src/sys/rpc/rpcsec_gss/svc_rpcsec_gss.c:642 > #9 0xffffffff807d48f3 in svc_run_internal (pool=0xffffff0004ca6200, > ismaster=0) at /usr/src/sys/rpc/svc.c:837 > #10 0xffffffff807d50ab in svc_thread_start (arg=Variable "arg" is not > available. > ) > at /usr/src/sys/rpc/svc.c:1200 > #11 0xffffffff805a26f8 in fork_exit ( > callout=0xffffffff807d50a0 <svc_thread_start>, arg=0xffffff0004ca6200, > frame=0xffffff8096c0dc40) at /usr/src/sys/kern/kern_fork.c:845 > #12 0xffffffff808bb26e in fork_trampoline () > at /usr/src/sys/amd64/amd64/exception.S:565 > > Kernel is generic except for > device crypto > options KGSSAPI. > > Ash /etc/make.conf > WITHOUT_X11=yes > KRB5_HOME=/usr/local > KRB5_IMPL=mit > # added by use.perl 2011-09-02 11:38:57 > PERL_VERSION=5.10.1 > > I'm happy to provide any additional info. > > Thanks for any help, > Clinton > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" ------=_Part_1940756_505940954.1316827621854 Content-Type: text/x-patch; name=svcrpcsec.patch Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=svcrpcsec.patch LS0tIHJwYy9ycGNzZWNfZ3NzL3N2Y19ycGNzZWNfZ3NzLmMuc2F2CTIwMTEtMDktMjMgMjA6MTQ6 MDcuMDAwMDAwMDAwIC0wNDAwCisrKyBycGMvcnBjc2VjX2dzcy9zdmNfcnBjc2VjX2dzcy5jCTIw MTEtMDktMjMgMjE6MTY6MzQuMDAwMDAwMDAwIC0wNDAwCkBAIC02MjUsNiArNjI1LDIzIEBAIHN2 Y19ycGNfZ3NzX2ZvcmdldF9jbGllbnQoc3RydWN0IHN2Y19ycGMKIAlzdmNfcnBjX2dzc19yZWxl YXNlX2NsaWVudChjbGllbnQpOwogfQogCisvKgorICogU2FtZSBhcyBhYm92ZSwgZXhjZXB0IHRo YXQgdGhpcyBvbmUgZXhwZWN0cyBzdmNfcnBjX2dzc19sb2NrIHRvCisgKiBiZSBoZWxkIHdoZW4g aXQgaXMgY2FsbGVkLiBJdCByZWxlYXNlcyB0aGlzIGxvY2suCisgKi8KK3N0YXRpYyB2b2lkCitz dmNfcnBjX2dzc19mb3JnZXRfY2xpZW50X2xvY2tlZChzdHJ1Y3Qgc3ZjX3JwY19nc3NfY2xpZW50 ICpjbGllbnQpCit7CisJc3RydWN0IHN2Y19ycGNfZ3NzX2NsaWVudF9saXN0ICpsaXN0OworCisJ bGlzdCA9ICZzdmNfcnBjX2dzc19jbGllbnRfaGFzaFtjbGllbnQtPmNsX2lkLmNpX2lkICUgQ0xJ RU5UX0hBU0hfU0laRV07CisJVEFJTFFfUkVNT1ZFKGxpc3QsIGNsaWVudCwgY2xfbGluayk7CisJ VEFJTFFfUkVNT1ZFKCZzdmNfcnBjX2dzc19jbGllbnRzLCBjbGllbnQsIGNsX2FsbGxpbmspOwor CXN2Y19ycGNfZ3NzX2NsaWVudF9jb3VudC0tOworCXN4X3h1bmxvY2soJnN2Y19ycGNfZ3NzX2xv Y2spOworCXN2Y19ycGNfZ3NzX3JlbGVhc2VfY2xpZW50KGNsaWVudCk7Cit9CisKIHN0YXRpYyB2 b2lkCiBzdmNfcnBjX2dzc190aW1lb3V0X2NsaWVudHModm9pZCkKIHsKQEAgLTYzOCwxNiArNjU1 LDIzIEBAIHN2Y19ycGNfZ3NzX3RpbWVvdXRfY2xpZW50cyh2b2lkKQogCSAqIEZpcnN0IGVuZm9y Y2UgdGhlIG1heCBjbGllbnQgbGltaXQuIFdlIGtlZXAKIAkgKiBzdmNfcnBjX2dzc19jbGllbnRz IGluIExSVSBvcmRlci4KIAkgKi8KLQl3aGlsZSAoc3ZjX3JwY19nc3NfY2xpZW50X2NvdW50ID4g Q0xJRU5UX01BWCkKLQkJc3ZjX3JwY19nc3NfZm9yZ2V0X2NsaWVudChUQUlMUV9MQVNUKCZzdmNf cnBjX2dzc19jbGllbnRzLAotCQkJICAgIHN2Y19ycGNfZ3NzX2NsaWVudF9saXN0KSk7CisJc3hf eGxvY2soJnN2Y19ycGNfZ3NzX2xvY2spOworCWNsaWVudCA9IFRBSUxRX0xBU1QoJnN2Y19ycGNf Z3NzX2NsaWVudHMsIHN2Y19ycGNfZ3NzX2NsaWVudF9saXN0KTsKKwl3aGlsZSAoc3ZjX3JwY19n c3NfY2xpZW50X2NvdW50ID4gQ0xJRU5UX01BWCAmJiBjbGllbnQgIT0gTlVMTCkgeworCQlzdmNf cnBjX2dzc19mb3JnZXRfY2xpZW50X2xvY2tlZChjbGllbnQpOyAvKiByZWxlYXNlcyBsb2NrICov CisJCXN4X3hsb2NrKCZzdmNfcnBjX2dzc19sb2NrKTsKKwkJY2xpZW50ID0gVEFJTFFfTEFTVCgm c3ZjX3JwY19nc3NfY2xpZW50cywKKwkJICAgIHN2Y19ycGNfZ3NzX2NsaWVudF9saXN0KTsKKwl9 CiAJVEFJTFFfRk9SRUFDSF9TQUZFKGNsaWVudCwgJnN2Y19ycGNfZ3NzX2NsaWVudHMsIGNsX2Fs bGxpbmssIG5jbGllbnQpIHsKIAkJaWYgKGNsaWVudC0+Y2xfc3RhdGUgPT0gQ0xJRU5UX1NUQUxF CiAJCSAgICB8fCBub3cgPiBjbGllbnQtPmNsX2V4cGlyYXRpb24pIHsKIAkJCXJwY19nc3NfbG9n X2RlYnVnKCJleHBpcmluZyBjbGllbnQgJXAiLCBjbGllbnQpOwotCQkJc3ZjX3JwY19nc3NfZm9y Z2V0X2NsaWVudChjbGllbnQpOworCQkJc3ZjX3JwY19nc3NfZm9yZ2V0X2NsaWVudF9sb2NrZWQo Y2xpZW50KTsKKwkJCXN4X3hsb2NrKCZzdmNfcnBjX2dzc19sb2NrKTsKIAkJfQogCX0KKwlzeF94 dW5sb2NrKCZzdmNfcnBjX2dzc19sb2NrKTsKIH0KIAogI2lmZGVmIERFQlVHCg== ------=_Part_1940756_505940954.1316827621854--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1461855405.1940757.1316827621857.JavaMail.root>