From owner-freebsd-isp@FreeBSD.ORG Mon Jul 29 18:51:57 2013 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 27018299 for ; Mon, 29 Jul 2013 18:51:57 +0000 (UTC) (envelope-from florian.heigl@gmail.com) Received: from mail-bk0-x22e.google.com (mail-bk0-x22e.google.com [IPv6:2a00:1450:4008:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AB2232883 for ; Mon, 29 Jul 2013 18:51:56 +0000 (UTC) Received: by mail-bk0-f46.google.com with SMTP id 6so422322bkj.5 for ; Mon, 29 Jul 2013 11:51:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=OKddDSqWUzkmsZsELpQpfg937IZyZKH6dQ4Grm4OVzA=; b=CB5OVyxbrXZBI6Z3K/fzpxDib+lk3l2pi5QJkOOgwoGNAAGuXXTpt84fC+YegnaDRk JHiIjehALgyZcNbyjDXYyPIDNliDoXYPh7Vxs0cSp0q7dBUjaO94iylIy5pRQRM/LmHu JsP4LojXKmlId9QlNVW5ru/BNOZJZIcWwlJ7TtQvkkeYOCfFfIDFtjz4AW4X5kC3/s4M gAV8EovqD5AU6CtwEaPiDHfImp8k2w5PTGsy0dj6YoVJlP8OIIJPmYMgnwajkDhDpBKi hdWkKZpXcReROm82ALak+pPVn0hoYGsNHY7i24ErE7AkAsXnz6qcouf7adzzQ4qnO80M qgcg== X-Received: by 10.204.70.1 with SMTP id b1mr8910300bkj.3.1375123913932; Mon, 29 Jul 2013 11:51:53 -0700 (PDT) Received: from ?IPv6:2001:a60:113f:b801:a934:fd8b:334c:3078? ([2001:a60:113f:b801:a934:fd8b:334c:3078]) by mx.google.com with ESMTPSA id da7sm8441509bkb.1.2013.07.29.11.51.50 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 29 Jul 2013 11:51:51 -0700 (PDT) Subject: Re: Virtualizing FreeBSD... Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Florian Heigl In-Reply-To: Date: Mon, 29 Jul 2013 20:51:49 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <91400827-DCA1-4B2F-AC6F-3287E0C85600@gmail.com> References: To: Karl Pielorz X-Mailer: Apple Mail (2.1283) Cc: freebsd-isp@freebsd.org X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Jul 2013 18:51:57 -0000 Hi, On 29.07.2013, at 16:23, Karl Pielorz wrote: > Anyone got any recommended / "works for them" advice on what's the = best virtualization platform to run FreeBSD under? (apart from FreeBSD = itself) - we're looking for something commercial / with management etc. = (e.g. HyperV/ESXi etc.) >=20 > We've got some experience running it under VMware - but are looking = for one that offers good driver support (i.e. not 'emulated hardware' = for NIC / disks)? >=20 > The experience we've had so far hasn't been brilliant from an I/O = point of view (hence the push to find out who/what supports FreeBSD = better for I/O drivers) - there's only so far an emulated E1000 can go = :) I've been running FreeBSD on two platforms so far: - Xen (non-support "real" Xen) in paravirtualized and HVM modes; for my = own use I've sticked with that since the days of Xen2. The performance and low of PV mode was what kept me trying it, but most = of the times there have been intolerable bugs, regressions etc. that = were unfixed for years. The worst thing is that bugs generally tend to not get fixed for years = until the next FreeBSD or Xen version somehow makes them go away. This may sound bitter, but I've been working with FreeBSD on Xen for 8 = years now and I'm just saying I've seen this happen. PV Benefits that I also cared about were i.e. online RAM = increase/decrease, that at least worked some time during those years; = with HVM it's not possible. PV is still mostly dead / bugged so it's out of the question. (FYI = XenServer can run both modes too) So, if I want a FreeBSD VM for compiling, testing etc. I'll fire up a = Xen HVM domU; this is exactly what you'd have in XenServer, too. Running XENHVM is something that has worked nicely for me ever since = FreeBSD 8 with performance being very good for networking and "ok" for = disk IO. A really stupid fact is that those two tiny net/disk drivers that you = need for Xen HVM didn't make it into GENERIC in more than two years. You still need to run GENERIC off horribly slow emulated disk devices so = you can finally build your real kernel. Really, there's no single non-stupid reason for that. Xen(Server) has a nice advantage over VMWare in not having per-CPU = memory overhead. I'll also add that I've supported a XenServer farm from 3.x to 6.x and = the time we spent in supporting / fixing it was significantly more than = expected.=20 - VMWare ESXi This is what I have also tried. I'm quite sure I have had the = e1000/vmxnet autoswitching working fine and the overall experience was = just fine. No bugs, no troubles. The platform is perfectly robust and plays nice with FreeBSD (what I'm = trying to say: You can completely skip worrying about platform issues) I've now also inherited management of a medium sized platform running = FreeBSD on ESXi, some 45 VMs I guess. What I notice there is disk throughput being not great and disk IO = latency feels abysmal, especially to someone used PV Xen VMs. My tinderboxes run around 2 days for under 1500 packages. I suspect that my own servers are a bit more powerful than what's under = those VMs. Still, often it's wiser to have a physical, super-responsive FreeBSD box = with jails than many tiny but not snappy FreeBSD VMs on VMWare. Pro sides I didn't mention, for VMWare: - Page sharing will be easier to use than in Xen (and HyperV just went = to a corner, crying) - Snapshots, stuff like VEEAM, VAAI, or better Filer-side snapshots on = NFS and freezing the VM is aeons ahead of XenServer (imho) - HA really exists, if you afford it. (XenServer HA exists but the scope = differs) That brings me to the last points, lessons learned: - Alignment:=20 For all I can tell stock 9.x FreeBSD hasn't gotten the news about disk = alignment, meaning you need to invest some time to fix that, or you = would waste a lot of your performance. - Avoid optimized CFlags if you go with XenServer, or you might run into = bad surprises i.e. if you change your hardware platform. I ended up needing to recompile for AMD on the old server before being = able to move to new hosts. VMWare would have just run GENERIC and never have caused me that trouble = in the first place. Greetings, Florian=