From owner-freebsd-stable@freebsd.org Tue Sep 27 15:51:20 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 908C5BEC31A for ; Tue, 27 Sep 2016 15:51:20 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id 5FB95A4B for ; Tue, 27 Sep 2016 15:51:20 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from sweettea.beer.town (unknown [76.164.8.130]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 530D556488; Tue, 27 Sep 2016 10:51:19 -0500 (CDT) Subject: Re: Status of PCIe Hotplug? To: Borja Marcos , Jan Henrik Sylvester References: <791ee95b-ab4e-f07e-6b8d-0e0b6c49ceb0@janh.de> Cc: FreeBSD-STABLE Mailing List From: Eric van Gyzen Message-ID: <58f84457-d775-3023-5727-a326bc8df722@FreeBSD.org> Date: Tue, 27 Sep 2016 10:51:15 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Sep 2016 15:51:20 -0000 On 09/27/2016 08:57, Borja Marcos wrote: > >> On 27 Sep 2016, at 15:48, Jan Henrik Sylvester wrote: >> >> On 09/27/2016 12:16, Borja Marcos wrote: >>> I have noticed that the GENERIC kernel in 11-STABLE includes the >>> PCI_HP option, and the hotplug bits seem to be present in the >>> kernel, but I don’t see any userland support for it. >>> >>> Is it somewhat complete and in that case am I missing something? >> >> I do not know kind of userland support you mean. I just tried: >> >> Plugging in my USB 3.0 ExpressCard while 11.0 is running, the >> controller was detected and I was able to use USB devices with it. >> Great. > > Thanks :) > > I was hoping (and I assume it’s the ultimate goal of the project) to > be able to hot plug PCIe devices such as NVMe drives. > > On Solaris you can replace them provided you power them off > previously (there’s a command for that, “hotplug”). > > On FreeBSD I’ve tried using devctl but powering off, disabling a > device and enabling it again has led to a panic. > > Interestingly, I disabled nvme0 using devctl and "nvmecontrol > devlist" didn’t find any nvme controllers despite having 10 > controllers and 10 drives. However, the ZFS pool of 10 NVMe drives > was working happily. Degraded of course, with one NVMe missing. To my knowledge, all the necessary PCIe-layer code is present. However, that's just one layer: Many drivers will likely need changes in order to cope with surprise removal of their devices. For that reason, HotPlug needs a lot of testing on a variety of platforms. The FreeBSD developer base is much smaller than its user base, of course, so the variety of our testing is rather limited. You can help immensely by giving us detailed bug reports, either on a mailing list or in Bugzilla. For a panic, the panic messages and stack trace of the current thread will be very helpful. Complete crashinfo(8) output would be great. The most relevant userland tool is devctl, followed closely by devinfo and pciconf. In the case of Jan's USB 3.0 ExpressCard, it's possible that one or all of the USB controller drivers (xhci, ehci, uhci) didn't cope with the surprise removal of the controller. Before removing the card, try detaching the driver(s) with "devctl detach xhciN". There might be more than one device. Use "pciconf -lc" to find the HotPlug-capable pcib devices (bridges). Use devinfo to find which one is your ExpressCard slot and find all the devices attached to it. Then use devctl to detach the devices. There could be a tree of devices; in that case, you can usually start at the level immediately under pcibN; you don't need to detach every device from the bottom up. Once all the devices are detached, you should be able to remove the card safely. Eric