From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 21:40:54 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB392BB3; Tue, 2 Jun 2015 21:40:54 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 572BF16B7; Tue, 2 Jun 2015 21:40:54 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wgme6 with SMTP id e6so150890321wgm.2; Tue, 02 Jun 2015 14:40:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AqszvN3bekAfkFZa2VCt1U89CykEgQ3bB5eBFadirAY=; b=uHptRQnwildZm0PTN7cK9gJNZjOlI/RIXLTjJtUeVsm1EVQFIz6nVv1/c3CVTYieJr YOlyloVqAun+erylm2ZpOLoY5Q1bhM5MQrMqayF/72vPGoowfRxpLysiZYLCWZXMYejk pzO9s2qZQPlP0VW6RZzopguyQi0z33h9GVvSdOJCf7aPhgBCS7XcxnM8uuZ4AkPgMYo4 bAgQqFdG/ogrgs2QZ0Okt/ft0JBUNStHcBkjGwTx2xVtxTO+R5J0I3PynEs4ugcUmOgR MTIKiAwVScPl+NleEYdlVnF2HYRtlSimzux8XCTTsllFfW6a1ewzrLV26wdMY9dGX1w5 qExw== MIME-Version: 1.0 X-Received: by 10.194.62.201 with SMTP id a9mr53841138wjs.63.1433281251768; Tue, 02 Jun 2015 14:40:51 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Tue, 2 Jun 2015 14:40:51 -0700 (PDT) In-Reply-To: <556DDDA9.6090005@FreeBSD.org> References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> Date: Tue, 2 Jun 2015 14:40:51 -0700 Message-ID: Subject: Re: bhyve: corrupting zfs pools? From: Neel Natu To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 21:40:54 -0000 Hi Andriy, On Tue, Jun 2, 2015 at 9:45 AM, Andriy Gapon wrote: > On 02/06/2015 14:14, Andriy Gapon wrote: >> >> I am doing a simple experiment. >> >> I get FreeBSD image from here: >> ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz >> >> Then I run in bhyve with two additional "disks" created with truncate -s 4g: >> $ bhyveload -m 1G -d >> ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test >> $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s >> 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img >> -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s >> 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l >> com2,/dev/nmdm0A -c 2 -m 1g test >> >> Note sectorsize=512/4096 options. Not sure if it's them that cause the trouble. >> >> Then, in the VM: >> $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 >> $ zfs create -p l2arc-test/ROOT/initial >> $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - >> >> Afterwards, zpool status -v reports no problem. >> But then I run zpool scrub and get the following in the end: >> $ zpool status -v >> pool: l2arc-test >> state: ONLINE >> status: One or more devices has experienced an error resulting in data >> corruption. Applications may be affected. >> action: Restore the file in question if possible. Otherwise restore the >> entire pool from backup. >> see: http://illumos.org/msg/ZFS-8000-8A >> scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 >> config: >> >> NAME STATE READ WRITE CKSUM >> l2arc-test ONLINE 0 0 9 >> mirror-0 ONLINE 0 0 18 >> vtbd1 ONLINE 0 0 25 >> vtbd2 ONLINE 0 0 23 >> >> errors: Permanent errors have been detected in the following files: >> >> /l2arc-test/ROOT/initial/usr/bin/svnlitesync >> /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz >> /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz >> >> /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a >> >> >> The same issue is reproducible with ahci-hd. >> >> My host system is a recent amd64 CURRENT as well. The hardware platform is AMD. >> > > I used the following monstrous command line to reproduce the test in qemu: > $ qemu-system-x86_64 -smp 2 -m 1024 -drive > file=/tmp/livecd2/R2.img,format=raw,if=none,id=bootd -device > virtio-blk-pci,drive=bootd -drive > file=/tmp/l2arc-test/hdd1,if=none,id=hdd1,format=raw -device > virtio-blk-pci,drive=hdd1,logical_block_size=4096 -drive > file=/tmp/l2arc-test/hdd2,id=hdd2,if=none,format=raw -device > virtio-blk-pci,drive=hdd2,logical_block_size=4096 -drive > file=/tmp/l2arc-test/ssd,id=ssd,if=none,format=raw -device > virtio-blk-pci,drive=ssd,logical_block_size=4096 ... > > And several other variations of logical_block_size and physical_block_size. > The tests a re very slow, but there are no checksum errors. > > So, I suspect guest memory corruption caused by bhyve. Perhaps the problem is > indeed specific to AMD-V. > Perhaps, but I wasn't able to repro this. I tried your recipe to repro on two systems running -current at r283917. - Intel Sandybridge server: Xeon E52650 with 8 cores/16 threads and 64GB memory - hdd1/hdd2 on UFS - hdd1/hdd2 on ZFS - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory - hdd1/hdd2 on UFS Can you provide some more details about your setup? I can then try to repro on a system similar to your setup. - uname -a - sysctl hw.model - sysctl hw.ncpu - sysctl hw.physmem - sysctl hw.vmm - host filesystem underlying hdd1 and hdd2 best Neel > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org"