From owner-freebsd-net@freebsd.org Tue Jan 12 23:29:32 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 580D1A808CE for ; Tue, 12 Jan 2016 23:29:32 +0000 (UTC) (envelope-from flewis@panasas.com) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0066.outbound.protection.outlook.com [207.46.100.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 227211174 for ; Tue, 12 Jan 2016 23:29:31 +0000 (UTC) (envelope-from flewis@panasas.com) Received: from DM2PR08MB446.namprd08.prod.outlook.com (10.141.86.18) by DM2PR08MB413.namprd08.prod.outlook.com (10.141.85.142) with Microsoft SMTP Server (TLS) id 15.1.361.13; Tue, 12 Jan 2016 23:29:24 +0000 Received: from DM2PR08MB447.namprd08.prod.outlook.com (10.141.86.19) by DM2PR08MB446.namprd08.prod.outlook.com (10.141.86.18) with Microsoft SMTP Server (TLS) id 15.1.361.13; Tue, 12 Jan 2016 23:29:22 +0000 Received: from DM2PR08MB447.namprd08.prod.outlook.com ([10.141.86.19]) by DM2PR08MB447.namprd08.prod.outlook.com ([10.141.86.19]) with mapi id 15.01.0361.006; Tue, 12 Jan 2016 23:29:22 +0000 From: "Lewis, Fred" To: "freebsd-net@freebsd.org" CC: "Pokala, Ravi" , "Lewis, Fred" Subject: Kernel panic from lagg_ioctl and lagg_port_ioctl Thread-Topic: Kernel panic from lagg_ioctl and lagg_port_ioctl Thread-Index: AQHRTZERJxBzi0RqBU+/N+qQfMI0yA== Date: Tue, 12 Jan 2016 23:29:22 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=flewis@panasas.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [66.31.107.140] x-microsoft-exchange-diagnostics: 1; DM2PR08MB446; 5:VjPXI9Oegh3eCtCiytXZjOIscTtkAbGRF3B/fbFyDkpx3B6UU4AKa6wX4QhigVWWIAKQZdb0P/7awd1wSxveiLjNCceHf3+zGXsnUVrXkNjozo/yJ5gSFsTBHkpPMqr8CdyxK8G7mYgvhYw4N4nldg==; 24:m/mPX6lab5fNm7Rirn80p6u0K7GDA1dPgDHFwIc46uxQRSZhumzJMslYn/nyH5I1lkfWWpRudp7x8J2HoSoIFSi/NPbJloU6joRZpvPbsfU= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:DM2PR08MB446; x-ms-office365-filtering-correlation-id: 7b879c44-642f-4196-8457-08d31ba833e1 x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(8121501046)(520078)(3002001)(10201501046); SRVR:DM2PR08MB446; BCL:0; PCL:0; RULEID:; SRVR:DM2PR08MB446; x-forefront-prvs: 081904387B x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(189002)(164054003)(199003)(110136002)(87936001)(5004730100002)(81156007)(97736004)(2351001)(54356999)(2900100001)(105586002)(5002640100001)(1096002)(4001430100002)(4326007)(1220700001)(6116002)(92566002)(107886002)(11100500001)(102836003)(99286002)(3846002)(50986999)(106116001)(122556002)(86362001)(66066001)(5008740100001)(40100003)(229853001)(10400500002)(5001960100002)(189998001)(2906002)(36756003)(106356001)(450100001)(586003)(101416001)(2501003)(77096005); DIR:OUT; SFP:1101; SCL:1; SRVR:DM2PR08MB446; H:DM2PR08MB447.namprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: panasas.com does not designate permitted sender hosts) Content-Type: text/plain; charset="us-ascii" Content-ID: <2EF450DB0D2172439EB44A03CB1753BF@namprd08.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Jan 2016 23:29:22.4338 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR08MB446 X-Microsoft-Exchange-Diagnostics: 1; DM2PR08MB413; 2:XJ+VCsP+foLhJA9mJoiWEO22L866KOwi9N5s910fOIAFBtJjqNsj08cf02DsN4yuFt0cfbgi5Y1poYeT8Tp8JvELVxlgzHBb0Tt2YTIa+5p8GETdVa/CRGVnAXHx1hfsSv2zahi8gbeacmDXfwPaRg==; 23:2gADQ+3tNjs87aEJNJAiRFEQUyWoKvSpP+/tzQnJyMC/kK64JUmLdIMtWDVoBTO2cseavN5Etf0fU1KWCoiFHP5tZ2nty0uzhin300jnxict4Jc/o+9qMax0Iid+fmdPMV78z1MroypY/qWsBno8vCPcrQNUF/BZAFvnM0lg95InhM3Mmi1I6vEMi4n4z7fI X-OriginatorOrg: panasas.com X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2016 23:29:32 -0000 Hi FreeBSD Networking folks. We are seeing a kernel panics on stable/10 that are being caused by lagg_ioctl() and lagg_port_ioctl(). The panic occurs when moving from an lacp configuration to, say, a failover configuration. Please double-check me, but what appears to be happening is that the software context is not getting cleaned up properly on a mode change and lacp_portreq() is getting called when the lagg is set to failover mode. In particular, sc->sc_portreq is left pointing to lacp_portreq when the mode is no longer lacp. In earlier versions of lagg_ioctl() (e.g. stable/10/r171247) all of the callout vectors are set to NULL which I think will prevent the problem. Similar NULLing code is also in stable/7. I didn't check other releases. case SIOCSLAGG: if (sc->sc_proto !=3D LAGG_PROTO_NONE) { LAGG_WLOCK(sc); error =3D sc->sc_detach(sc); /* Reset protocol and pointers */ sc->sc_proto =3D LAGG_PROTO_NONE; sc->sc_detach =3D NULL; sc->sc_start =3D NULL; sc->sc_input =3D NULL; sc->sc_port_create =3D NULL; sc->sc_port_destroy =3D NULL; sc->sc_linkstate =3D NULL; sc->sc_init =3D NULL; sc->sc_stop =3D NULL; sc->sc_lladdr =3D NULL; sc->sc_req =3D NULL; sc->sc_portreq =3D NULL; } Looks like the above code was taken out via r287723. Evidently this has been made moot in HOL via r272170 and r272178 (maybe others). Here is one of the backtrace snippets: panic() at panic+0x155/frame 0xfffffe201e3df2e0 trap_fatal() at trap_fatal+0x38f/frame 0xfffffe201e3df340 trap_pfault() at trap_pfault+0x308/frame 0xfffffe201e3df3e0 trap() at trap+0x47a/frame 0xfffffe201e3df5f0 calltrap() at calltrap+0x8/frame 0xfffffe201e3df5f0 --- trap 0xc, rip =3D 0xffffffff804b9811, rsp =3D 0xfffffe201e3df6b0, rbp = =3D 0xfffffe201e3df730 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x1a1/frame 0xfffffe201e3df730 __mtx_lock_flags() at __mtx_lock_flags+0x5a/frame 0xfffffe201e3df750 lacp_portreq() at lacp_portreq+0x2f/frame 0xfffffe201e3df780 lagg_port2req() at lagg_port2req+0x62/frame 0xfffffe201e3df7b0 lagg_port_ioctl() at lagg_port_ioctl+0x14b/frame 0xfffffe201e3df820 ifioctl() at ifioctl+0x162b/frame 0xfffffe201e3df8e0 kern_ioctl() at kern_ioctl+0x255/frame 0xfffffe201e3df950 sys_ioctl() at sys_ioctl+0x13c/frame 0xfffffe201e3df9a0 Is there any chance of getting this fixed in stable/10 before code freeze? We have tested a set of diffs that fix the issue and will submit them for review shortly. Thanks, -Fred