From owner-freebsd-fs@freebsd.org Sun Jun 4 23:19:52 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F10FABF2481 for ; Sun, 4 Jun 2017 23:19:52 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660064.outbound.protection.outlook.com [40.107.66.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 82B1664F96; Sun, 4 Jun 2017 23:19:51 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0190.CANPRD01.PROD.OUTLOOK.COM (10.165.218.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1143.10; Sun, 4 Jun 2017 23:19:49 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.1143.018; Sun, 4 Jun 2017 23:19:49 +0000 From: Rick Macklem To: Bryan Drewery CC: FreeBSD FS Subject: Re: NFS panic: newnfs_copycred: negative nfsc_ngroups Thread-Topic: NFS panic: newnfs_copycred: negative nfsc_ngroups Thread-Index: AQHS3LrT/PEtl29yjUaGma/5tOYV1aIT2FCAgAF/Tys= Date: Sun, 4 Jun 2017 23:19:49 +0000 Message-ID: References: <2b7a77df-8291-d399-6d1f-c454fbb2a5d9@FreeBSD.org>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: FreeBSD.org; dkim=none (message not signed) header.d=none;FreeBSD.org; dmarc=none action=none header.from=uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTXPR01MB0190; 7:rnb5ltKkfnktPa60dLongLYyMTW8zOiSc+xDRdrw78+SvWvwTQXtgZjScbTY7PSA321OZfy9kOnXYP8JRnDne57paxi4rsVPJRfVqwIctZ3D8KDp3BRy576FarJHcgzh9IZbTMgo9AoQPa+wFPvyc4EvFypheu5e0OmmTmdUA0YBXvkjlk922/SCxx0q4IxC/HrkQTaVADruUoNyrF9AE+lcgRhc3uKed0I5K2A4voST/+Vp+bkCO0qc+z0v6t0uTCd1ZL8TUWdH2Q/uYZ0dgX2KOWnJsDsxpwF3bjOFdnC985rNfey/ZsGF3V0CXSRUmx9CdUbAoID0xZCX3oUXKA== x-ms-traffictypediagnostic: YTXPR01MB0190: x-ms-office365-filtering-correlation-id: 75c6338d-b286-49f0-7db7-08d4aba03288 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254075)(201703131423075)(201703031133081)(201702281549075); SRVR:YTXPR01MB0190; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(102415395)(6040450)(2401047)(8121501046)(5005006)(10201501046)(93006095)(93001095)(3002001)(100000703101)(100105400095)(6041248)(20161123555025)(20161123560025)(20161123558100)(20161123564025)(201703131423075)(201702281529075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:YTXPR01MB0190; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:YTXPR01MB0190; x-forefront-prvs: 03283976A6 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(39410400002)(39400400002)(39450400003)(377454003)(24454002)(81166006)(102836003)(74482002)(6506006)(2900100001)(122556002)(6436002)(74316002)(38730400002)(110136004)(6246003)(305945005)(55016002)(53936002)(189998001)(6306002)(5890100001)(2906002)(5660300001)(77096006)(9686003)(966005)(3280700002)(3660700001)(33656002)(86362001)(14454004)(8936002)(478600001)(8676002)(99936001)(2950100002)(53546009)(25786009)(50986999)(450100002)(4326008)(54356999)(76176999)(6916009); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0190; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/mixed; boundary="_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_" MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Jun 2017 23:19:49.6067 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0190 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jun 2017 23:19:53 -0000 --_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable You could try the attached little patch (untested) and see if the panics go away. It is weird that no one else seems to see this, but I can see that it might be possible for the code to create an open structure and not initialize the nfso_cred structure in it. This patch makes sure it is set t= o the credentials at the time the open is created, which I think is harmless. If this stops the crashes, I can easily come up with a better patch for thi= s and commit it to head. rick ________________________________________ From: Bryan Drewery Sent: Saturday, June 3, 2017 8:24:11 PM To: Rick Macklem Cc: FreeBSD FS Subject: Re: NFS panic: newnfs_copycred: negative nfsc_ngroups On 6/3/2017 3:43 PM, Bryan Drewery wrote: > Last reported here but I forgot to follow-up > https://lists.freebsd.org/pipermail/freebsd-current/2013-July/042996.html > > I still get this quite often. > > Server is: 10.2-RELEASE-p2 > > Client is: 12.0-CURRENT #5 r318116M > > mount is (no soft or intr since 2013): > >> tank:/tank/distfiles/freebsd /mnt/distfiles = nfs rw,bg,noatime,rsize=3D65536,wsize=3D65536,readahead=3D8,nfs= v4,rdirplus 0 0 > > I have a core for debugging... >> (kgdb) bt >> #0 __curthread () at ./machine/pcpu.h:232 >> #1 doadump (textdump=3D1) at /usr/src/sys/kern/kern_shutdown.c:318 >> #2 0xffffffff803abf3c in db_fncall_generic (addr=3D, rv= =3D, nargs=3D, args=3D) at /us= r/src/sys/ddb/db_command.c:581 >> #3 db_fncall (dummy1=3D, dummy2=3D, dummy= 3=3D, dummy4=3D) at /usr/src/sys/ddb/db_comma= nd.c:629 >> #4 0xffffffff803abaaf in db_command (last_cmdp=3D, cmd_t= able=3D, dopager=3D) at /usr/src/sys/ddb/db_c= ommand.c:453 >> #5 0xffffffff803ab7e4 in db_command_loop () at /usr/src/sys/ddb/db_comm= and.c:506 >> #6 0xffffffff803ae89f in db_trap (type=3D, code=3D) at /usr/src/sys/ddb/db_main.c:248 >> #7 0xffffffff80a9fda3 in kdb_trap (type=3D3, code=3D-61456, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 >> #8 0xffffffff80ee9286 in trap (frame=3D0xfffffe355f840540) at /usr/src/= sys/amd64/amd64/trap.c:537 >> #9 >> #10 kdb_enter (why=3D0xffffffff81455661 "panic", msg=3D) = at /usr/src/sys/kern/subr_kdb.c:444 >> #11 0xffffffff80a5d759 in vpanic (fmt=3D, ap=3D0xfffffe35= 5f8406d0) at /usr/src/sys/kern/kern_shutdown.c:772 >> #12 0xffffffff80a5d59f in _kassert_panic (fatal=3D1, fmt=3D0xffffffff814= 34d8b "newnfs_copycred: negative nfsc_ngroups") at /usr/src/sys/kern/kern_s= hutdown.c:669 >> #13 0xffffffff80946ec2 in newnfs_copycred (nfscr=3D0xfffff8047b3eb530, c= r=3D0xfffff80122cfa500) at /usr/src/sys/fs/nfs/nfs_commonport.c:244 >> #14 0xffffffff8094bddc in nfscl_getstateid (vp=3D, nfhp= =3D0xfffff80501233902 "\233\262\tM\336\006\236\313\n", fhlen=3D28, mode=3D1= , fords=3D, cred=3D, p=3D, sta= teidp=3D, lckpp=3D) at /usr/src/sys/fs/nfscli= ent/nfs_clstate.c:630 >> #15 0xffffffff8095ca88 in nfsrpc_read (vp=3D0xfffff8030bc209c0, uiop=3D0= xfffffe355f840af8, cred=3D0xfffff80122cfa500, p=3D0x0, nap=3D0xfffffe355f84= 09d0, attrflagp=3D0xfffffe355f840aa4, stuff=3D) at /usr/src/= sys/fs/nfsclient/nfs_clrpcops.c:1396 >> #16 0xffffffff8096b90a in ncl_readrpc (vp=3D0xfffff8030bc209c0, uiop=3D0= xfffffe355f840af8, cred=3D0xfffff801b4913300) at /usr/src/sys/fs/nfsclient/= nfs_clvnops.c:1375 >> #17 0xffffffff80976656 in ncl_doio (vp=3D0xfffff8030bc209c0, bp=3D0xffff= fe349a268750, cr=3D, td=3D0x0, called_from_strategy=3D) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:1643 >> #18 0xffffffff80978694 in nfssvc_iod (instance=3D) at /us= r/src/sys/fs/nfsclient/nfs_clnfsiod.c:302 >> #19 0xffffffff80a1e394 in fork_exit (callout=3D0xffffffff80978420 , arg=3D0xffffffff81c7de64 , frame=3D0xfffffe355f8= 40c00) at /usr/src/sys/kern/kern_fork.c:1038 >> #20 >> (kgdb) p *nfscr >> $3 =3D {nfsc_uid =3D 3735929054, nfsc_groups =3D {3735929054 }, nfsc_ngroups =3D -559038242} >> (kgdb) frame 17 >> #17 0xffffffff80976656 in ncl_doio (vp=3D0xfffff8030bc209c0, bp=3D0xffff= fe349a268750, cr=3D, td=3D0x0, called_from_strategy=3D) at /usr/src/sys/fs/nfsclient/nfs_clbio.c:1643 >> (kgdb) p vp->v_mount->mnt_stat.f_mntonname >> $8 =3D "/mnt/distfiles", '\000' > > I had some bogus -domain in my nfsuserd options on the client that I > removed after the recent panic. Not sure if it is relevant. > No that had no impact, I've hit it 3 times since sending the last email. -- Regards, Bryan Drewery --_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_ Content-Type: application/octet-stream; name="opencredpanic.patch" Content-Description: opencredpanic.patch Content-Disposition: attachment; filename="opencredpanic.patch"; size=746; creation-date="Sun, 04 Jun 2017 23:19:37 GMT"; modification-date="Sun, 04 Jun 2017 23:19:37 GMT" Content-Transfer-Encoding: base64 LS0tIGZzL25mc2NsaWVudC9uZnNfY2xzdGF0ZS5jLmNyZWRwYW5pYwkyMDE3LTA2LTA0IDEwOjQ2 OjA5LjI3MzYxMzAwMCAtMDQwMAorKysgZnMvbmZzY2xpZW50L25mc19jbHN0YXRlLmMJMjAxNy0w Ni0wNCAxMTowMToyMS41MzYwOTMwMDAgLTA0MDAKQEAgLTI5MCw2ICsyOTAsMTkgQEAgbmZzY2xf b3Blbih2bm9kZV90IHZwLCB1X2ludDhfdCAqbmZocCwgaQogCSAgICBuZXdvbmVwKTsKIAogCS8q CisJICogSWYgbmZocCAhPSBOVUxMICYmIG5vcCA9PSBOVUxMLCBhIG5ldyBPcGVuIHN0cnVjdHVy ZSB3YXMgYWxsb2NhdGVkCisJICogdXNpbmcgKm5vcC4gIEZvciB0aGlzIGNhc2UsIHNldCB0aGUg Y3JlZGVudGlhbHMgaW4gdGhlIE9wZW4sIHNvCisJICogdGhhdCB0aGV5IGFyZSBuZXZlciB1bmlu aXRpYWxpemVkLgorCSAqLworCWlmIChuZmhwICE9IE5VTEwgJiYgbm9wID09IE5VTEwpIHsKKwkJ S0FTU0VSVCgqbmV3b25lcCAhPSAwLCAoIiVzOiBuZXcgb3BlbiB3YXMgYWxsb2NhdGVkXG4iLAor CQkgICAgX19mdW5jX18pKTsKKwkJS0FTU0VSVChvcCAhPSBOVUxMLCAoIiVzOiBOZXcgb3BlbiBt dXN0IGJlIHJldHVybmVkXG4iLAorCQkgICAgX19mdW5jX18pKTsKKwkJbmV3bmZzX2NvcHlpbmNy ZWQoY3JlZCwgJm9wLT5uZnNvX2NyZWQpOworCX0KKworCS8qCiAJICogTm93LCBjaGVjayB0aGUg bW9kZSBvbiB0aGUgb3BlbiBhbmQgcmV0dXJuIHRoZSBhcHByb3ByaWF0ZQogCSAqIHZhbHVlLgog CSAqLwo= --_002_YTXPR01MB0189942CAB532478E7002855DDF50YTXPR01MB0189CANP_--