From owner-freebsd-current@freebsd.org Mon Oct 15 14:37:13 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA5B210DD0B6 for ; Mon, 15 Oct 2018 14:37:12 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A5DF76A55 for ; Mon, 15 Oct 2018 14:37:12 +0000 (UTC) (envelope-from yuripv@yuripv.net) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id B563520FAD; Mon, 15 Oct 2018 10:37:11 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 15 Oct 2018 10:37:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.net; h= subject:to:references:from:message-id:date:mime-version :in-reply-to:content-type; s=fm3; bh=gXWMu1lRmNMiYy3dYr5YFE8rBMp VZwHI/8hvK8WuvEc=; b=CuhbEfueH+Fh03LuKHrKrXn9xYsLSqd7yu9GWGh+XRs xDT4PxNHV1TJBgIZLvPpuBXavTWh6XhHBpty7LdeNC3EYSBai85a87NnIcdBb//r GdFIILPKPxkCj2hVJd6Q3hFR56dl5joO8symhiMsr4BdBKUAZBHoQcuUjhZar++A ibSfR+4HV7x4aRsF6QjzYihFOy240by6dq5kqTpHT188qchHZEnJqXJYtoWOcRll dI4qt8Fk5DLr+TgBTdoBWmQt/Coi7XhIOlA/faIC/EKpXWdCA8XhPRlW4gVEpxCk z4Oy3HtddpiPW9sSXkZXZSwQ08KSK+iWuH0E/rjNSeA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=gXWMu1 lRmNMiYy3dYr5YFE8rBMpVZwHI/8hvK8WuvEc=; b=R1y71FblpZTf44wyhhrfbT zis7qH7rfT0C/OmzGK1KaKv7kCvg21GM/2D8ZZLgAHfQyeLCdm6h8mdzubHQZPhV +h8mN3UiLSrJr9P6kAJ7VpezxPq/ofztZbLUNhTEPCYF8D7ZYDhNvxOuF/q61NCJ sJjiFEJcvlX8CsIPtfJefxxtehUZQ5DfJq/I3KlsKyaOs1O5ZEIhfs3KjtIq47/X PdAuhamJqYyCDdTnbatM3738ak+IXnYGBMCmSQyn+B4KADLTwGZU0JcvF0NSVbQw 4N7U3Jkt4uFvvtB2er7bQ9gY3kRbrFE6YQC/FcYs8/N11C+Oj7xrq69Xirk3lYFQ == X-ME-Sender: X-ME-Proxy: Received: from [192.168.1.2] (unknown [94.233.224.82]) by mail.messagingengine.com (Postfix) with ESMTPA id DDF98E40E6; Mon, 15 Oct 2018 10:37:09 -0400 (EDT) Subject: Re: vm_fault on boot with NVMe/nda To: Daniel Nebdal , Current References: From: Yuri Pankov Openpgp: preference=signencrypt Autocrypt: addr=yuripv@yuripv.net; keydata= xsBNBFu8u6IBCADB11gP0QwnorrHjqAtKLHKHNHskhy0s7jqJKfx0YqXgVBKGLJ9/mjLAz0F CBNvemHSDDTs0mEZ9cBKKi6cmsav6+UQgr//yai6hvXLBJqKchSFO4MhmdvBtsGFq1yKz5Zi uhjmimKyIpgBgvMdbgGbGq6cnSB2uEPmZuJr419SVRODOkXukU+F5WHgaHzDdHAIu1asCt2B +6msxqIqlFWcXyZyTGicTGGvC/PFIsVRUtD1dIJANTC876g7DTb7LZXWiWwJpSJ4GKMXMHVX Ct9BoQ4i3nhKbOxb6Io1wsy+NFyWsTJ9KYrxKKPJP3oG8BWb/cqlFqnE4eNSsiq2q7krABEB AAHNH1l1cmkgUGFua292IDx5dXJpcHZAeXVyaXB2Lm5ldD7CwJcEEwEIAEECGwMFCQWjmoAF CwkIBwMFFQoJCAsFFgMCAQACHgECF4AWIQT4arc+w94tPi0v/3CTi+B/sSrhbAUCW708wAIZ AQAKCRCTi+B/sSrhbPxBB/961alcU091O+yKT5/oReHVc/PX0Tz4sW3V44AcgLfYlrZavCro EFz90qmCrl0xqEwuAKcC4bjmL8SjPWAhSN6IH9nxdw+HeZnAPiHm/q679Bu47+nHBl3qD/9p +t1PkKeKZfaWToFMt1nq06ytSu6VLMCwLdlDNe6DReX0ex/afEqKsuaIZSKL4UYjRwklp8PU Uf98QkrfapyHB67hQMzfI4tPeJaYyv0cTgfq3kUWJx1V6Xi0b6Zxj4ZrB2TXvaMO5g7yhU9E E3WWAvoe4FgB3a7dHe8atnHhq5+Cuvm6+LD4Jh7jvMAE5UMN+xxQpnGpNghHjaCy4vXrLRBZ nhRYzsBNBFu8u6IBCADKih3Q933rDNj4ZA8FhBQ2RlmBgvwOLcDPIL3h0V7h38y3+HisgFSc XACDsdrTlYZ1bRXkD9FHENynBcv0l/3uGJDk8jaGIDE0TP8OQBRp+IaU9/BHnAqrKxTJGIol Dahy2m+yx2yhdc6B4ujWMDqCF1rWOD+ymOWw+VLllOkrHcZa5PJtX9UOGbApZl8ZTM8El4CA NN8F1bg9MWzUi+8LYoGWGc+BwsFS1OUB1c4SPgMu5fD4Wfsr9yRl06fdpEA2YT7B/j5/5RSC 0sE2Zs/tmJ/JRflHJ12ycj59ma2xQMfEJF40hZDpMFQmZvbVqgEg3ocQcltjbxlIKZ/mjC4z ABEBAAHCwHwEGAEKACYWIQT4arc+w94tPi0v/3CTi+B/sSrhbAUCW7y7ogIbDAUJBaOagAAK CRCTi+B/sSrhbIDcCACqAZMcoxUBLZa40a5b24j5i1jplvCYYb3h+Q5lt5+BFJ87kCb4dJuU D3kh2i29BrxWQWa9WNue9ozxeYkbkfXubQYXexVolRsnh64OdGsE8KvorBFBB3zdK/GRt2Jy +jsnTfUWuQllbzMP0MfhCDMk1Mo8WvDH2/cOEP/yLKf20a+cd6nLs7bidjmGXo9pyuBKAtV6 Kv+VRu54AL+A/UBYu/eB3Dtvzcnut+1Zq6KaP++kUwPwINLIk04OBDwN0zRNTiqMAFYYyz2v ZHBB6E1th/l//ZC5b9Dk0ZpFI1bYdL9ymnrZe1MqbGPnDCToQxu00T/pZCm6Z92YrZQYuNwl Message-ID: <1bb0a463-7630-e182-edb6-d02a868704d8@yuripv.net> Date: Mon, 15 Oct 2018 17:37:02 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="KQWMvaLxIP300y8z7Touyh582pK6OUwtv" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2018 14:37:13 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --KQWMvaLxIP300y8z7Touyh582pK6OUwtv Content-Type: multipart/mixed; boundary="qaqygAKhLHaZsDXqdSu1R7YaOl75EeLA9"; protected-headers="v1" From: Yuri Pankov To: Daniel Nebdal , Current Message-ID: <1bb0a463-7630-e182-edb6-d02a868704d8@yuripv.net> Subject: Re: vm_fault on boot with NVMe/nda References: In-Reply-To: --qaqygAKhLHaZsDXqdSu1R7YaOl75EeLA9 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Daniel Nebdal wrote: > Hi. I have a 12-ALPHA9 / r339331 amd64 system (a HPE ProLiant ML30 G9),= > with a Kingston NVMe SSD ("KINGSTON SKC1000480G") on a PCIe card. >=20 > By default, it shows up as /dev/nvd0, and this is how I installed the > system. It has a single large UFS2 (with SJ and TRIM support) partition= > mounted as /. (There's also a few other partitions on it that should be= > irrelevant for this.) This works, but it does sometimes slow down for > minutes at the time with disturbing queue lengths in gstat; on the orde= r of > tens of thousands. As I understand it, this is due to how TRIM operatio= ns > take precedence over everything else when using nvd ? >=20 > Looking around, I noticed the nda driver for NVMe-through-CAM. To test = it, > I added hw.nvme.use_nvd=3D0 to loader.conf. On one level, this works: T= he > drive shows up as /dev/nda0 . On the other hand, trying to mount nda0p2= as > / floods the console with "vm_fault: pager read error, pid 1 (init)", a= nd > never finishes booting. >=20 > What is more interesting is that if I boot from the drive, but mount an= > alpha9 usb stick as /, I can then mount the nda device just fine, and t= he > very minimal testing I did (using bin/cat and COPYRIGHT on the NVMe dri= ve) > seems to work. >=20 > So - is nda meant to be bootable, or am I a bit over-eager in trying to= do > so? > If not, is there anything smart I can do to get better performance out = of > nvd? > (Or have I just overlooked something obvious?) >=20 > Dmesg from a normal nvd boot here: > https://openbenchmarking.org/system/1810159-RA-SSD30089593/SSD/dmesg FWIW, I set hw.nvme.use_nvd=3D0 in the installer, got 12-ALPHA8 installed= on nda0, and it's happily booting from it (using ZFS, though), so it's certainly meant to be bootable. --qaqygAKhLHaZsDXqdSu1R7YaOl75EeLA9-- --KQWMvaLxIP300y8z7Touyh582pK6OUwtv Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEE+Gq3PsPeLT4tL/9wk4vgf7Eq4WwFAlvEphUACgkQk4vgf7Eq 4Wyobwf9Ez7D5OftUenrLU+dqGQISRylanCtI4z6M/iCMUwDNHZ/rghWPfPNs5Nd g8G3w+BE/Nq+H8HvzRJ+F4Grc6WnoGJzv67OYW5CLh9o30cakuBUGuysfaS7k0gm zORXdij5RDVom2zLG6jFXmq0rKCTdrn/mkx+XcyP+IiNT8k7wPmJtwEIhQ2xkjag PzUQ8H7TYZp0sgrQaJHTML9ULBcRd77trZYjsFrw2dPWiuYz+vgwmfRgXxWsxuxu 6yvSEzwjf0yAJtVMgWc6vkngR3t889b1e1OhbdWgu9yPJoNbFoiSZqeqirHtifYl qvmTiNlTy5l80IiDgZ+Fs9jS+Y9MvA== =RtxT -----END PGP SIGNATURE----- --KQWMvaLxIP300y8z7Touyh582pK6OUwtv--