From owner-freebsd-current@freebsd.org Mon Sep 5 23:51:06 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB655B96671 for ; Mon, 5 Sep 2016 23:51:06 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qk0-x22a.google.com (mail-qk0-x22a.google.com [IPv6:2607:f8b0:400d:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7FDF6A3D for ; Mon, 5 Sep 2016 23:51:06 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qk0-x22a.google.com with SMTP id v123so202331594qkh.2 for ; Mon, 05 Sep 2016 16:51:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Tv5PdM9TcmPY96mrSXGiHoAeH4YDEaFcez4d42pZwxM=; b=lB8p4Uf98BwBfCJm2mxD5BhfrsEGrvrXpaIZsOEBottTcCxb0nw71g+7VTjcg8GTa7 oviwG5hwhYAbk/eHE1CWk9O3j/uQiR8Hln3G3zgNPJoAgbV/tG3Sj4aBsnmCSjl/BnIj Smz4mwSA28tmP1Ojh+JJli73rivNNR/ZRrLmGtjXusE1URqNO8t40xWEeUlkcevefk8w /Rpm2OG/rk8lAjQDhrQY+1l5gG/qwAAt5zQZz7oXvsU31+mVeDDdnuvsvjGgXT2yXerL mIXaC3siNv0Qvs7Ppx1htIMxsmMBy3QDhXnxA+WfSEy6Z74RWT2aagBSjnymSNdKzARk b1Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Tv5PdM9TcmPY96mrSXGiHoAeH4YDEaFcez4d42pZwxM=; b=EHbzQHdFfhyG6NAQYKdmJ3+lIvUBG0mxnp1iWAarS2pAOX4Io9ReCFW5e2pxECHLb6 Gjy0O4dQL2oI+WM98RI2vIeJEu+Sa6xV+vg/9Lq4B5mbiRaDedLIs4bt5rpkSyBNq4s1 bMeMU078gjaLCj4ZhTLfS1xp7nHa1fOergtzyhRWc++7DjwVMoFvWJcvML4fdMoBH8Hl 9UMt/3HWeiB+VxJxKzw0PbXpwxWKvhTLTy88VGQY2FJz+SJ6mw7+G9bPkxazGkJhHDkC pT6/WtE1c8F5gawzhXOGE341ZQ8C0PnvnTCzigs14xgkvD6c6MK4+qbOdnI+mDTR968a 9oxQ== X-Gm-Message-State: AE9vXwOFYnYK/NGfe7oZQbFs444iKDaQM4rpClJ4jm4sabsEpLHH40qnoa8apyUrn1tv7nV1 X-Received: by 10.55.44.134 with SMTP id s128mr39884121qkh.177.1473119465544; Mon, 05 Sep 2016 16:51:05 -0700 (PDT) Received: from mutt-hardenedbsd (pool-100-16-219-226.bltmmd.fios.verizon.net. [100.16.219.226]) by smtp.gmail.com with ESMTPSA id z52sm5513851qta.40.2016.09.05.16.51.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 05 Sep 2016 16:51:04 -0700 (PDT) Date: Mon, 5 Sep 2016 19:51:02 -0400 From: Shawn Webb To: Mark Johnston Cc: freebsd-current@freebsd.org, mmacy@nextbsd.org Subject: Re: taskqgroup_adjust kernel panic Message-ID: <20160905235102.GA42492@mutt-hardenedbsd> References: <20160905175538.GA81799@mutt-hardenedbsd> <20160905215454.GE70066@wkstn-mjohnston.west.isilon.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="4bRzO86E/ozDv8r1" Content-Disposition: inline In-Reply-To: <20160905215454.GE70066@wkstn-mjohnston.west.isilon.com> X-Operating-System: FreeBSD mutt-hardenedbsd 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Sep 2016 23:51:06 -0000 --4bRzO86E/ozDv8r1 Content-Type: multipart/mixed; boundary="C7zPtVaVf+AK4Oqc" Content-Disposition: inline --C7zPtVaVf+AK4Oqc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 05, 2016 at 02:54:54PM -0700, Mark Johnston wrote: > On Mon, Sep 05, 2016 at 01:55:38PM -0400, Shawn Webb wrote: > > Hey all, > >=20 > > I'm at revision 3872750 of the hardened/current/drm-next-4.7 branch in > > the HardenedBSD/hardenedBSD-playground repo. I've gotten this kernel > > panic a couple times when booting. I'm using full-disk encryption with > > ZFS and encrypted swap. The hardware is a Purism 15 2K laptop. > >=20 > > The panic doesn't happen often nor is there a way I can reproduce it > > 100%. > >=20 > > Here's my `uname -a` output: > >=20 > > FreeBSD hbsd-dev-laptop 12.0-CURRENT-HBSD FreeBSD 12.0-CURRENT-HBSD #0 = 3872750(hardened/current/drm-next-4.7): Tue Aug 30 17:41:53 EDT 2016 sh= awn@hbsd-dev-laptop:/usr/obj/usr/src/sys/LATT-SEC amd64 > >=20 > > Here's a couple pictures of the panic I took: > >=20 > > https://goo.gl/photos/P5kiwabPYjwQX7Kr8 > > https://goo.gl/photos/BWtvBnq7QLnwgRP28 >=20 > Based on the faulting instruction, the panic probably happened because > qid is uninitialized in the loop that starts with >=20 > while ((gtask =3D LIST_FIRST(>ask_head))) { >=20 > I don't know this code very well, so I'm not sure how that can happen. I > suspect iflib_irq_alloc_generic() is buggy: it calls >=20 > taskqgroup_attach_cpu(... CPU_FFS(&cpus) ...); >=20 > and CPU_FFS returns 1-indexed IDs, but taskqgroup_attach_cpu() pretty > clearly expects 0-indexed CPU IDs. There's a similar bug in find_nth() > in iflib.c. I think you hit the nail right on the head. Attached is a patch that doesn't fix the underlying issue, but at least detects improperly setting qid. It'll throw a KASSERT if qid isn't set properly. I'll study this code a bit more within the next couple days and I hope to have a full patch to address the underlying issue. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --C7zPtVaVf+AK4Oqc Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="2016-09-05_subr_gtaskqueue.c-rev-01.patch" Content-Transfer-Encoding: quoted-printable diff --git a/sys/kern/subr_gtaskqueue.c b/sys/kern/subr_gtaskqueue.c index 9423dd8..1a086c9 100644 --- a/sys/kern/subr_gtaskqueue.c +++ b/sys/kern/subr_gtaskqueue.c @@ -790,6 +790,7 @@ _taskqgroup_adjust(struct taskqgroup *qgroup, int cnt, = int stride) =20 while ((gtask =3D LIST_FIRST(>ask_head))) { LIST_REMOVE(gtask, gt_list); + qid=3D-1; if (gtask->gt_cpu =3D=3D -1) qid =3D taskqgroup_find(qgroup, gtask->gt_uniq); else { @@ -799,6 +800,7 @@ _taskqgroup_adjust(struct taskqgroup *qgroup, int cnt, = int stride) break; } } + KASSERT(qid !=3D -1, ("_taskgroup_adjust: qid cannot be -1")); qgroup->tqg_queue[qid].tgc_cnt++; LIST_INSERT_HEAD(&qgroup->tqg_queue[qid].tgc_tasks, gtask, gt_list); --C7zPtVaVf+AK4Oqc-- --4bRzO86E/ozDv8r1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXzgTjAAoJEGqEZY9SRW7uQzsQAIZiFyBaxOlWQwMas6f3s81F InMi/HP5keo7sToBxky5lBIDC9tQ3N/YVhDyt6IGDHuuNequ7H2KAl3g6IF+/Nj2 rL06tbVwvKfrMNhO0ADMXQ7Y6xj/KlZOEYS0UbDlA7WOP7gTkgygvKQElb7YKUbC TTiZo9me9jNeYewY66dDACasSVSPvoCdQFDyubp1C+8FNSxpSfiuAV5QeLRD/UAO iu7svhL0O8luMuk9IhymKSHVSp6NUDBcfeo4OTa/FknEz1e5d+I27PvP15+1OlZb NsSEA+vsNe0Go9M2hZ32ZV+HK1tE2BkYXozINc1H7oiwGtaLDoekB03iZIwbO+lz pHu7UbgZukjQpygMncHlaC+C4VtaU0xaF6ZVVF4wjPywrqEzk5lpK7eBcsLgcBGa 3Y0cuMADKkfxC27/UErsXWdiWS6l9dBkXFVxFmw/erbLKA7b1PGFLmYE1R3NRLbm 7LulOvC1dEhodxPZ/8Gl1EEnOm3GvbiCdaoapYsPCJQmL1ZWmY2/SxeD5mB92FOx PLejNJxRwj7mpibDJCGFgLb0bBvO0b8U+weQrSlync6zEMOcSy+FC1Hs0Eg+YkVi rzZgEAfPfcBmzIJKTJexVe/8N9GW+0VAIZE8zB806wFiq2upZB76pog7DygYWdgv G6EpvIU0dffbM+drnlRd =S8W4 -----END PGP SIGNATURE----- --4bRzO86E/ozDv8r1--