From nobody Fri Feb 16 17:42:35 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TbzmL5JXGz53tbk for ; Fri, 16 Feb 2024 17:42:58 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from fhigh5-smtp.messagingengine.com (fhigh5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TbzmL0ClYz57wC for ; Fri, 16 Feb 2024 17:42:58 +0000 (UTC) (envelope-from chuck@tuffli.net) Authentication-Results: mx1.freebsd.org; none Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id E362411400B4; Fri, 16 Feb 2024 12:42:56 -0500 (EST) Received: from imap51 ([10.202.2.101]) by compute7.internal (MEProxy); Fri, 16 Feb 2024 12:42:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli.net; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1708105376; x=1708191776; bh=ncRK1GaDUL RsBm3ERSjSEvhW98FhfG9w2XNIHJD44Ps=; b=drLypW1B0Av4C8jFnG6reuAuau DVUyofctx7n2rU8l0sm+jsTyxO80UGoasC4imIv7k8EqVX3wauVuQ5+XtemIMzRD C/+G3TxJmSR0/Wgtvge7KubMYFhgn80v3ci4dfkX7k0XIxfyZiJrrfRt3JAIDQPR Wqg2yC5kOhfxnXUyawhFwawRsI5x7O4U3vYx2ij3Snr1pYBGUZm9YTqbeqy55Krt 4JnGivPDVV+Q2Q53O6w1Wv3q9xkJjxPJbm76IDkqFEw2vm/bwL6sMnaWzFgtrNAy pCzEIKhyH7JNo8QvaFBB94XeSwm8H7iixo83CgSMHUpztUpNdWidwqjcscZg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1708105376; x=1708191776; bh=ncRK1GaDULRsBm3ERSjSEvhW98Fh fG9w2XNIHJD44Ps=; b=MulIyMR2CFO17I7jngZfQfDTdpU1FjB7/eG+nhMOo/wy +49kHGsf+dJ350yweIWoW1Ww2vzxq/QfrGC/ITlwrj75aLT/e0IRz/1t+Zlc7g8m pb34+hzXbHLGe1iTDKPA+CVPdp4XziUZOKxmkqyBj04+V8OdO4GvbnicsUfu+yQz hJn72/+wCLHM9f7rw5fhkFFUMbM9XK7h8EuWhsKjJjtUW4SkRNC1Cl6zpnHa0pxI GdCNxwsJrQc4lkl6644xJtKzrR7dI5q+NuKr6l+UnInx3rZsQcfBxNwHyz0HqS/n qdJFW/k2o8QHF7nJctI6S4TN0ZRRf5WqCWrhhlSgQA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddvgddutdefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreerjeenucfhrhhomhepfdevhhhu tghkucfvuhhffhhlihdfuceotghhuhgtkhesthhufhhflhhirdhnvghtqeenucggtffrrg htthgvrhhnpeevjeevtddtteffkeeigefhveekjefhtedutddtiefgjefgffejheevtdet hfeftdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gthhhutghksehtuhhffhhlihdrnhgvth X-ME-Proxy: Feedback-ID: ib6f94606:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 393EEB60092; Fri, 16 Feb 2024 12:42:56 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Message-Id: <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> In-Reply-To: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> Date: Fri, 16 Feb 2024 09:42:35 -0800 From: "Chuck Tuffli" To: "Matthew Grooms" , "FreeBSD virtualization" Subject: Re: bhyve disk performance issue Content-Type: multipart/alternative; boundary=4a4f83f7edf3401cad2a2c472cf96382 X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4TbzmL0ClYz57wC X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:209242, ipnet:103.168.172.0/24, country:US] --4a4f83f7edf3401cad2a2c472cf96382 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: > Hi All, >=20 >=20 >=20 > I'm in the middle of a project that involves building out a handful of= servers to host virtual Linux instances. Part of that includes testing = bhyve to see how it performs. The intent is to compare host storage opti= ons such as raw vs zvol block devices and ufs vs zfs disk images using h= ardware raid vs zfs managed disks. It would also involve >=20 >=20 =E2=80=A6 > Here is a list of a few other things I'd like to try: >=20 >=20 > 1) Wiring guest memory ( unlikely as it's 32G of 256G ) > 2) Downgrading the host to 13.2-RELEASE FWIW we recently did a similar exercise and saw significant performance = differences on ZFS backed disk images when comparing 14.0 and 13.2. We d= idn=E2=80=99t have time to root cause the difference, so it could simply= be some tuning difference needed for 14.=20 =E2=80=94chuck --4a4f83f7edf3401cad2a2c472cf96382 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Fri, Feb 16,= 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the mi= ddle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


=E2=80= =A6

Here is a list of a few other things I'd l= ike to try:


1) Wiring guest memory (= unlikely as it's 32G of 256G )
2) Downgrading the host t= o 13.2-RELEASE

FWIW we recentl= y did a similar exercise and saw significant performance differences on = ZFS backed disk images when comparing 14.0 and 13.2. We didn=E2=80=99t h= ave time to root cause the difference, so it could simply be some tuning= difference needed for 14. 

=E2=80=94c= huck
--4a4f83f7edf3401cad2a2c472cf96382--