From owner-freebsd-current@freebsd.org Mon Apr 9 23:17:48 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C114BF86EFB for ; Mon, 9 Apr 2018 23:17:48 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B18B829FB for ; Mon, 9 Apr 2018 23:17:48 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.200.3] (c-73-216-227-39.hsd1.va.comcast.net [73.216.227.39]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id 4C17027000AB for ; Mon, 9 Apr 2018 19:11:51 -0400 (EDT) DMARC-Filter: OpenDMARC Filter v1.3.1 duke.cs.duke.edu 4C17027000AB Authentication-Results: duke.cs.duke.edu; dmarc=none header.from=cs.duke.edu DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail0816; t=1523315511; bh=rrrRRfF5tx9GyCfn1G2OjiZ1ko0Dv4JH8hMvJwaC7Z8=; h=To:From:Subject:Date:From; b=JqrVWiutfmC8cF3bXyDyWac0q9UtpfLEXrcv4PNHLlyRp3jRlWFSYgsAhEtyZThBh +qj+709hftSuojavmDm0WBY3J1lb8FIGrPRhsMB4NSP88BOiXEi0ae4QNe1zkAM/1P nuKQbqJ/hw7WuHL1xcWZW6Zkmr4k55ge5F4DvvpfPYT403nCLCYLAzy5f/FM/GdGbV OUzazD+8sFcpkKN6nwAzsCtTyaspOZNWc5kgxAyADyBp4HG7touWtD9kWmAl91yXn8 LUxFumegZpUTpLCYFLO2LTtwtE7I+vXUM+vagpTMnH7hVMIimFE8xFoivYyH80YeYK SgqthrYr1P/ww== To: freebsd-current@freebsd.org From: Andrew Gallatin Subject: Odd ZFS boot module issue on r332158 Message-ID: Date: Mon, 9 Apr 2018 19:11:50 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Apr 2018 23:17:49 -0000 I updated my main amd64 workstation to r332158 from something much earlier (mid Jan). Upon reboot, all seemed well. However, I later realized that the vmm.ko module was not loaded at boot, because bhyve PCI passthru did not work. My loader.conf looks like (I'm passing a USB interface through): ####### vmm_load="YES" opensolaris_load="YES" zfs_load="YES" nvidia_load="YES" nvidia-modeset_load="YES" # Tune ZFS Arc Size - Change to adjust memory used for disk cache vfs.zfs.arc_max="4096M" hint.xhci.2.disabled="1" pptdevs="8/0/0" hw.dmar.enable="0" cuse_load="YES" ####### The problem seems "random". I rebooted into single-user to see if somehow, vmm.ko was loaded at boot and something was unloading vmm.ko. However, on this boot it was loaded. I then ^D'ed and continued to multi-user, where X failed to start because this time, the nvidia modules were not loaded. (but nvidia had been loaded on the 1st boot). So it *seems* like different modules are randomly not loaded by the loader, at boot. The ZFS config is: config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p2 ONLINE 0 0 0 da3p2 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ada1p2 ONLINE 0 0 0 da0p2 ONLINE 0 0 0 cache da2s1d ONLINE 0 0 0 The data drives in the pool are all exactly like this: => 34 9767541101 ada0 GPT (4.5T) 34 6 - free - (3.0K) 40 204800 1 efi (100M) 204840 9763209216 2 freebsd-zfs (4.5T) 9763414056 4096000 3 freebsd-swap (2.0G) 9767510056 31079 - free - (15M) There is about 1.44T used in the pool. I have no idea how ZFS mirrors work, but I'm wondering if somehow this is a 2T problem, and there are issues with blocks on difference sides of the mirror being across the 2T boundary. Sorry to be so vague.. but this is the one machine I *don't* have a serial console on, so I don't have good logs. Drew