From owner-freebsd-xen@freebsd.org Mon Feb 11 09:30:23 2019 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5757B14CFF99 for ; Mon, 11 Feb 2019 09:30:23 +0000 (UTC) (envelope-from prvs=938cbf5a0=roger.pau@citrix.com) Received: from SMTP.EU.CITRIX.COM (smtp.eu.citrix.com [185.25.65.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.citrix.com", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E45228BDC0 for ; Mon, 11 Feb 2019 09:30:19 +0000 (UTC) (envelope-from prvs=938cbf5a0=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.58,358,1544486400"; d="scan'208";a="85758748" Date: Mon, 11 Feb 2019 10:30:09 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Eric Bautsch CC: Subject: Re: Issues with XEN and ZFS Message-ID: <20190211093009.lnzak3l4ub65b67n@mac> References: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL02.citrite.net (10.69.22.126) X-Rspamd-Queue-Id: E45228BDC0 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of prvs=938cbf5a0=roger.pau@citrix.com designates 185.25.65.24 as permitted sender) smtp.mailfrom=prvs=938cbf5a0=roger.pau@citrix.com X-Spamd-Result: default: False [-2.35 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.86)[-0.860,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; ENVFROM_PRVS(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[citrix.com]; FORGED_SENDER_VERP_SRS(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[24.65.25.185.list.dnswl.org : 127.0.9.2]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[smtp.citrix.com.au,smtp02.citrix.com,smtp.eu.citrix.com,smtp03.citrix.com,smtp.citrix.com]; NEURAL_HAM_SHORT(-0.46)[-0.462,0]; IP_SCORE(-0.02)[country: GB(-0.09)]; FORGED_SENDER(0.00)[roger.pau@citrix.com,prvs=938cbf5a0=roger.pau@citrix.com]; RCVD_COUNT_ZERO(0.00)[0]; R_DKIM_NA(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:60825, ipnet:185.25.64.0/23, country:GB]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,prvs=938cbf5a0=roger.pau@citrix.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Feb 2019 09:30:23 -0000 Thanks for the testing! On Fri, Feb 08, 2019 at 07:35:04PM +0000, Eric Bautsch wrote: > Hi. > > > Brief abstract: I'm having ZFS/Xen interaction issues with the disks being > declared unusable by the dom0. > > > The longer bit: > > I'm new to FreeBSD, so my apologies for all the stupid questions. I'm trying > to migrate from Linux as my virtual platform host (very bad experiences with > stability, let's leave it at that). I'm hosting mostly Solaris VMs (that > being my choice of OS, but again, Betamax/VHS, need I say more), as well as > a Windows VM (because I have to) and a Linux VM (as a future desktop via > thin clients as and when I have to retire my SunRay solution which also runs > on a VM for lack of functionality). > > So, I got xen working on FreeBSD now after my newbie mistake was pointed out to me. > > However, I seem to be stuck again: > > I have, in this initial test server, only two disks. They are SATA hanging > off the on-board SATA controller. The system is one of those Shuttle XPC > cubes, an older one I had hanging around with 16GB memory and I think 4 > cores. > > I've given the dom0 2GB of memory and 2 core to start with. 2GB might be too low when using ZFS, I would suggest 4G as a minimum when using ZFS for reasonable performance, even 8G. ZFS is quite memory hungry. > The root filesystem is zfs with a mirror between the two disks. > > The entire thing is dead easy to blow away and re-install as I was very > impressed how easy the FreeBSD automatic installer was to understand and > pick up, so I have it all scripted. If I need to blow stuff away to test, no > problem and I can always get back to a known configuration. > > > As I only have two disks, I have created a zfs volume for the Xen domU thus: > > zfs create -V40G -o volmode=dev zroot/nereid0 > > > The domU nereid is defined thus: > > cat - << EOI > /export/vm/nereid.cfg > builder = "hvm" > name = "nereid" > memory = 2048 > vcpus = 1 > vif = [ 'mac=00:16:3E:11:11:51,bridge=bridge0', > 'mac=00:16:3E:11:11:52,bridge=bridge1', > 'mac=00:16:3E:11:11:53,bridge=bridge2' ] > disk = [ '/dev/zvol/zroot/nereid0,raw,hda,rw' ] > vnc = 1 > vnclisten = "0.0.0.0" > serial = "pty" > EOI > > nereid itself also auto-installs, it's a Solaris 11.3 instance. > > > As it tries to install, I get this in the dom0: > > Feb 8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): > WRITE_FPDMA_QUEUED. ACB: 61 18 a0 ef 88 40 46 00 00 00 00 00 > Feb 8 18:57:16 bianca.swangage.co.uk last message repeated 4 times > Feb 8 18:57:16 bianca.swangage.co.uk kernel: (ada1:ahcich1:0:0:0): CAM > status: CCB request was invalid That's weird, and I would say it's not related to ZFS, the same could likely happen with UFS since this is an error message from the disk controller hardware. Can you test whether the same happens _without_ Xen running? Ie: booting FreeBSD without Xen and then doing some kind of disk stress test, like fio [0]. Thanks, Roger. [0] https://svnweb.freebsd.org/ports/head/benchmarks/fio/