From owner-freebsd-stable@FreeBSD.ORG Wed Jun 3 03:06:57 2015 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CA52CB87 for ; Wed, 3 Jun 2015 03:06:57 +0000 (UTC) (envelope-from smkelly@smkelly.org) Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com [66.111.4.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9544D132C for ; Wed, 3 Jun 2015 03:06:57 +0000 (UTC) (envelope-from smkelly@smkelly.org) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 31B9F20B16 for ; Tue, 2 Jun 2015 23:06:50 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute4.internal (MEProxy); Tue, 02 Jun 2015 23:06:50 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=smkelly.org; h=cc :content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=DNi5d sRU2LV2j3lJRZDR9VMXENA=; b=rjldF7JnYpuXFvnMQd6GzqAiBw8vToCSqWPHV /HOEj/mqS4QluvaTzhZCdwwLrviy1OqJ4pPpLVXZjHK8zpd50JSGh0/pv7TI/dx1 88EDFEi5F5POy8nQ/RBNs/sMsD586nDN/sKAZRWhg1ZDTTPcUQlqGDgd+drmrxJJ zGR9aw= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=DNi5dsRU2LV2j3lJRZDR9VMXENA=; b=CB/tZ oK5ViECUKERLAkZZmVbhhy6M0Lrcn8ULodX6ldPSobK3VoCP6Z3/KD9A8ZvLU+xO oKA0LrgruU3ofTsny9dVf1Av9HFE12vTrvjBB15NCyCyPbME7jqvRCtoeNcVIi7A G+GpRjPjM1ce5y8VwkzGsMkgMLl1PMroRA2jOc= X-Sasl-enc: /viiZh7tbFMu4qkd9rUaZS3lSq7jxMnrb3gucSlo5+X6 1433300809 Received: from vpn-3.gwp.corp.flightaware.com (unknown [38.100.147.146]) by mail.messagingengine.com (Postfix) with ESMTPA id A64C5680141; Tue, 2 Jun 2015 23:06:49 -0400 (EDT) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2100\)) Subject: Re: 10.1 NVMe kernel panic From: Sean Kelly In-Reply-To: Date: Tue, 2 Jun 2015 22:06:48 -0500 Cc: FreeBSD-STABLE Mailing List Message-Id: References: <90B2D392-01FD-415A-B3D9-3CEDFC8373C4@smkelly.org> To: Jim Harris X-Mailer: Apple Mail (2.2100) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 03:06:57 -0000 Jim, Thanks for the reply. I set hw.nvme.force_intx=3D1 and get a new form of = kernel panic: http://smkelly.org/stuff/nvme_crash_force_intx.txt = It looks like the NVMes are just failing to initialize at all now. As = long as that tunable is in the kenv, I get this behavior. If I kldload = them after boot, the init fails as well. But if I kldunload, kenv -u, = kldload, it then works again. The only difference is kldload doesn=E2=80=99= t result in a panic, just timeouts initializing them all. I also compiled and tried stable/10 and it crashed in a similar way, but = i=E2=80=99ve not captured the panic yet. It crashes even without the = tunable in place. I=E2=80=99ll see if I can capture it. --=20 Sean Kelly smkelly@smkelly.org http://smkelly.org > On Jun 2, 2015, at 6:10 PM, Jim Harris wrote: >=20 >=20 >=20 > On Thu, May 21, 2015 at 8:33 AM, Sean Kelly > wrote: > Greetings. >=20 > I have a Dell R630 server with four of Dell=E2=80=99s 800GB NVMe SSDs = running FreeBSD 10.1-p10. According to the PCI vendor, they are some = sort of rebranded Samsung drive. If I boot the system and then load = nvme.ko and nvd.ko from a command line, the drives show up okay. If I = put > nvme_load=3D=E2=80=9CYES=E2=80=9D > nvd_load=3D=E2=80=9CYES=E2=80=9D > in /boot/loader.conf, the box panics on boot: > panic: nexus_setup_intr: NULL irq resource! >=20 > If I boot the system with =E2=80=9CSafe Mode: ON=E2=80=9D from the = loader menu, it also boots successfully and the drives show up. >=20 > You can see a full =E2=80=98boot -v=E2=80=99 here: > http://smkelly.org/stuff/nvme-panic.txt = = > >=20 > Anyone have any insight into what the issue may be here? Ideally I = need to get this working in the next few days or return this thing to = Dell. >=20 > Hi Sean, >=20 > Can you try adding hw.nvme.force_intx=3D1 to /boot/loader.conf? >=20 > I suspect you are able to load the drivers successfully after boot = because interrupt assignments are not restricted to CPU0 at that point - = see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199321 = for a = related issue. Your logs clearly show that vectors were allocated for = the first 2 NVMe SSDs, but the third could not get its full allocation. = There is a bug in the INTx fallback code that needs to be fixed - you do = not hit this bug when loading after boot because bug #199321 only = affects interrupt allocation during boot. >=20 > If the force_intx test works, would you able to upgrade your nvme = drivers to the latest on stable/10? There are several patches (one = related to interrupt vector allocation) that have been pushed to = stable/10 since 10.1 was released, and I will be pushing another patch = for the issue you have reported shortly. >=20 > Thanks, >=20 > -Jim >=20 >=20 > =20 >=20 > Thanks! >=20 > -- > Sean Kelly > smkelly@smkelly.org > http://smkelly.org >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable = > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org = "