From owner-freebsd-current@freebsd.org Mon Oct 15 14:32:17 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C636310DCEFF for ; Mon, 15 Oct 2018 14:32:17 +0000 (UTC) (envelope-from dnebdal@gmail.com) Received: from mail-yw1-xc30.google.com (mail-yw1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5FA6B767EB for ; Mon, 15 Oct 2018 14:32:17 +0000 (UTC) (envelope-from dnebdal@gmail.com) Received: by mail-yw1-xc30.google.com with SMTP id v198-v6so7575519ywg.12 for ; Mon, 15 Oct 2018 07:32:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=FmRusg7PFN2Uj0YHAdU5IrenC6hfl6LNl+13kAdkq4A=; b=B/vzuisAULeE0Eb2FFf0nbJvWAGAqZndQp7Ml47mkLfyB3n/TRuuAJw+4vZKIizOme reH1m/jzt0san0D442FeGcY/t78XJQKDIxAHXvofygVvb/SSbwHPee1qsq+0wg77JDIe Lueb1BWaB7/OEdO9QW1gVvEn5q6WYyp5O2l2WlaPmCqSfDyv3PIvDyEv1mdw9IrSG713 BDdpger40mLbwbWSEr/zlZ1EQgb6go5sQY+XQDm2+Ry1oHl4Bi1y73LyUM+zJyulTGO8 9oSJFxR8TAaj4KcINdeiLMn7D8ni/AcFY5GInpXRPpQJaIBp8jAsPj4ZMAYS5hq6BY5E vPjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=FmRusg7PFN2Uj0YHAdU5IrenC6hfl6LNl+13kAdkq4A=; b=BBuM9/m+pwcVcA+bu4QHigcyppv+2+b8bvgEckm26F7Wrk2TKaD9BZaScZjrPAscM8 cQ+Y3n5Gydke77if1b0VHAKv1hkRGz7KHUbvewCQB+3jT2rSSCnv9htI6wF3+7MpC/Mz u54ArXbIBH+W8bBhm8x8GH0jUAS/gDouW+IMsEa6IH91bUXvx3Z5qNgD+4xAVjpkwyHn avErWkknYjXQechaU1Zn/H2hQD1rbHvWesvEKp74o+mvoHC4R5/nwEcCskGD7lqEgGeM BWJTW35P1vAbGfL0IAJqWTkEI8fcDebFPe6wtyLJf2aPUWtN+lyNjTYWZQumJk1Jr6lL a0hQ== X-Gm-Message-State: ABuFfoh+BTz70uhVQixTgRV+n6YfqsjHAzDzvRCFkjLU524qcWhnfoPK prgcnQhHf78KFAHm2a//5HV8cwOQ09qejSRK34fpUEh8 X-Google-Smtp-Source: ACcGV63VLfb9qTy3sE5nGAn87kKxfips64GbmJ0ZY+bWi8KGOxwpZ4uTvLIn/Vel8AuWs/hT0LzkOdmtoEwVLSxFEfE= X-Received: by 2002:a0d:e706:: with SMTP id q6-v6mr8826959ywe.436.1539613936111; Mon, 15 Oct 2018 07:32:16 -0700 (PDT) MIME-Version: 1.0 From: Daniel Nebdal Date: Mon, 15 Oct 2018 16:31:38 +0200 Message-ID: Subject: vm_fault on boot with NVMe/nda To: Current Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2018 14:32:18 -0000 Hi. I have a 12-ALPHA9 / r339331 amd64 system (a HPE ProLiant ML30 G9), with a Kingston NVMe SSD ("KINGSTON SKC1000480G") on a PCIe card. By default, it shows up as /dev/nvd0, and this is how I installed the system. It has a single large UFS2 (with SJ and TRIM support) partition mounted as /. (There's also a few other partitions on it that should be irrelevant for this.) This works, but it does sometimes slow down for minutes at the time with disturbing queue lengths in gstat; on the order of tens of thousands. As I understand it, this is due to how TRIM operations take precedence over everything else when using nvd ? Looking around, I noticed the nda driver for NVMe-through-CAM. To test it, I added hw.nvme.use_nvd=0 to loader.conf. On one level, this works: The drive shows up as /dev/nda0 . On the other hand, trying to mount nda0p2 as / floods the console with "vm_fault: pager read error, pid 1 (init)", and never finishes booting. What is more interesting is that if I boot from the drive, but mount an alpha9 usb stick as /, I can then mount the nda device just fine, and the very minimal testing I did (using bin/cat and COPYRIGHT on the NVMe drive) seems to work. So - is nda meant to be bootable, or am I a bit over-eager in trying to do so? If not, is there anything smart I can do to get better performance out of nvd? (Or have I just overlooked something obvious?) Dmesg from a normal nvd boot here: https://openbenchmarking.org/system/1810159-RA-SSD30089593/SSD/dmesg -- Daniel Nebdal