From owner-freebsd-stable@FreeBSD.ORG Fri Oct 17 02:19:59 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EA5BFB3 for ; Fri, 17 Oct 2014 02:19:59 +0000 (UTC) Received: from mx12-out4.antispamcloud.com (mx12-out4.antispamcloud.com [46.165.232.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7B79CF for ; Fri, 17 Oct 2014 02:19:58 +0000 (UTC) Received: from net06.redwood.com ([91.233.6.246] helo=mail.redwood.com) by mx12.antispamcloud.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.84) (envelope-from ) id 1XertA-0005F5-0P; Thu, 16 Oct 2014 22:43:15 +0200 Received: from exceu2.rwd.lan ([fe80::e42a:a959:5640:381a]) by exceu2.rwd.lan ([fe80::e42a:a959:5640:381a%10]) with mapi id 14.01.0438.000; Thu, 16 Oct 2014 22:43:11 +0200 From: Maikel Verheijen To: Jim Harris Subject: Re: nvd disk on nvme controller not detected at boot-time Thread-Topic: nvd disk on nvme controller not detected at boot-time Thread-Index: AQHP54ttNqAodkLuEEmDqzIhnS+RzpwzAKSAgAAz24A= Date: Thu, 16 Oct 2014 20:43:10 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US, nl-NL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.5.141003 x-originating-ip: [10.31.1.9] MIME-Version: 1.0 X-Filter-ID: s0sct1PQhAABKnZB5plbIV6jvtGXzsRLyDxHGPnOSosTzNYXe+HZr4WowsEZlxIc1tZWiYINMnD4 agKXMg0NtBlCFCOA7kScelqCJFYKursKpTLhc3scs8lIU/v8u+rLonq0jcl6y1B4xrk1sOdNSWS2 BpcZp5MieEYXpn6TUALZI/Mr0BskU9hAYZ3Ymm+Y77VjOXfkZEVwPyGgPVOYnvLF3n+yP+L+jcBn IV6ubcZIn5ObN5P05LGWFQlP+VmPBIIZGAiIku3Cx523pDpt3x48vBcmKs9GYcu0FLW7EWPlgTl6 fJxyntEfhZCKje4Zx0bFDp3mGK8Y7ug+kgE2tol2UKe9D6SGnFl2eHdGD56roVjH0lzFpZqUycuK 16uwBmJ8Q0iagechgWqTL3vIA1Jh9nHh5xBVP9pvcu1pgYqSoC5VPlOdrUr8E6UB0EPg3VyTyNUA us5wK5dhtldfyDQttY3c9bWbvNE9580khkH/asDO0V6SUvEY03codo+QZd7VZNM714SJ3DaW+K4f VsdxRa5Rvm6JFujnJaOZwdtixEWhmbBXdtEvx2xJnvEp4oIKD0HBt4sWrEIlE8VJalYFvf25LVON YbYifH5OzZANAwxW2AvNktcRxKgEznEpH50YYHE3/09I4g/iRQecfi+orTeREDUbREcPuO+7veLj r9jD+xEtCAUWMNpEVddcVmUctY5jL1s3weViixsNRA== X-Report-Abuse-To: spam@mx99.antispamcloud.com X-Filter-Fingerprint: IFrWXGses7OKB5S5G8/dJbmLbQULIXlEhuVRudChxyrJUWjZ8+qhjyB23tbDuyLOYL8Ff78gYsez 4Rl08xudmXi4esCQ0R1MchVjt7wblGlvhFgW0MjUMRkF5sMCDfftTXNFDzN17hnrWeZYOJvLq0Ic WjZ+XcEjj/7Pkld0zkmvziDInX9WdMov2kn2yXjdwv61T+KDYyYtREgszdyFwv8IxCB3p/oCKvxr eyISh3JGb7OS5oVgiO+kDxZrVPLz3MmEGC2PrUKqLq5WmHK+Nw== X-Originating-IP: 91.233.6.246 X-Spampanel-Domain: redwood.com X-Spampanel-Username: sysadmin Authentication-Results: antispamcloud.com; auth=pass (login) smtp.auth=sysadmin X-Spampanel-Outgoing-Class: ham X-Spampanel-Outgoing-Evidence: Combined (0.07) X-Recommended-Action: accept Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Oct 2014 02:19:59 -0000 Hi Jim, On Tue, Oct 14, 2014 at 1:47 AM, Maikel Verheijen > wrote: Hi list! We recently purchased a HP DL380e G8 server to serve as our backup server w= ith an Intel P3600 that uses the nvme interface. We added load_nvme=3D"YES"= and load_nvd=3D"YES" to our loader.conf, and they both get loaded, however= the ssd disk device is not detected at boot. When we unload and reload the= nvd module the disk does get detected. Is there a way to see if the load-o= rder is correct? We added verbose_loading=3D"YES" to the loader.conf, but d= mesg doesn't show me the actual loading. One thing I did see related to the nvme in the dmesg (full dmesg attached) = output is this: Hi Maikel, Which version of FreeBSD are you using? I believe both of these issues hav= e been fixed on HEAD and merged to stable/9 and stable/10. 9.3 and 10.1-RC= would have these fixes. 9.2 and 10.0 would not. I'm currently running FreeBSD 10.0 with updates from freebsd-update. Would = FreeBSD 10.1-RC be safe for a production environment or would it be better = to wait until the real release? The machine is not used for production yet,= but we do want to go live with it in the near future. I currently use a = =93workaround=94, I=92m not loading ZFS at boot time but use a custom RC sc= ript to load the nvd module, test if the nvd0 disk is there, then load the = ZFS module. That seems to work for now, but having it work =93out of the bo= x=94 would be better, and a lot less prone to manual mistakes made by me :) nvme0: mem 0xfbdf0000-0xfbdf3fff irq 16 at device 0.0= on pica =85 nvme0: SET FEATURES (09) sqid:0 cid:9 nsid:0 cdw10:00000080 cdw11:00000000 nvme0: INTERNAL DEVICE ERROR (00/06) sqid:0 cid:9 cdw0: This was fixed in r263277. Incidentally, the error message here is harmles= s and not associated with the primary issue you have described above. Thanks, this was the only thing I thought that might be relevant to the iss= ue, glad to hear it=92s harmless. After unloading and reloading the nvd module the disk does get detected: nvd0: NVMe namespace nvd0: 381554MB (781422768 512 byte sectors) The load on boot issue was fixed in r263310. That is good news, that means we can probably drop my rc script in the near= future and use the proper loading procedures. Best regards, -Jim Thanks again and warm regards, Maikel Verheijen