From nobody Fri Feb 16 17:45:47 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tbzqk2gjjz53twY for ; Fri, 16 Feb 2024 17:45:54 +0000 (UTC) (envelope-from pprocacci@gmail.com) Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tbzqk0tfTz592y for ; Fri, 16 Feb 2024 17:45:54 +0000 (UTC) (envelope-from pprocacci@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x635.google.com with SMTP id a640c23a62f3a-a3ddc13bbb3so165523566b.0 for ; Fri, 16 Feb 2024 09:45:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708105552; x=1708710352; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=N7mnvjMwUoxZoKGfYbsM0FM+oh1q+JWqQJ0z+LLz2wc=; b=AeyPeuYtw6iuZZz4U2sizeM4YPfzos4B96KencUwtil20RmNR39Ix/Fg1kdxVVleLn Q59IA/JhA/KbjcVlS6s9615o9MGaQhJ0KaQI/kNKk/C3AUp7EhGpaaMh1FmpqBmAVlFw 5GwzR1zosMEq02DWp0YLz0J4S2Dc2VMqEXmG4YlpR94N+gNx0KUJ5hP5BYfIaSixjxXf ixx8z6HYm0Qpr7gvL23kod7ci0kXTAtiLL7xEfmbZRzzBVULHP0CI5X9WCJ675fKaQiA M11ez+SZ4z9okLuyP+F4/BUKCcfFeIQym0TiBTHVbn7NfFxaWjcnVyT98umfdUoNXWaB Sfpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708105552; x=1708710352; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=N7mnvjMwUoxZoKGfYbsM0FM+oh1q+JWqQJ0z+LLz2wc=; b=RTKKnFz9d7Iwkdu8RPQbCfj58Iu08Ma4GlvLI0rpcezJ1DVNR0aE8a74BlQZzHLAY3 OoyA9vPrUWjsFjDD+EC1vk3r83MjNFePtLmBdEGA/usPbI4+h9LdfFoJEqPTKXHAGG5t mCggJ32v1UgrNgsae63beq1yckawxEi1WK0jvaXNi1c4q9QsyM91LWEV6a9vdprpjEl2 Z2F/lj1YRR0ibob676BH5DRgp69H28spqskXuBaCHytz1lvP/UU01DelGLSpYsDoBatf lnTR57sDj9RyZ3slh89N3v1/FJ5yL8FBhDXsDVe6FQP3KuQmala+lh1j5/GAECiUI58t Ojlw== X-Forwarded-Encrypted: i=1; AJvYcCXjQKqOz+JSuzDaJiRc3bqCJT/IOPih6cUruhBh9opZQ1pJtB2OTrVIR3+Um7bmYzUIoyyi0aYJcccTq62+avK5x8sgaxr75AN3PqKGUqHyfvuQ X-Gm-Message-State: AOJu0Yxfm2wzx9MhQ4VWOJazpY3fdqu26wyQp/We4PVU0yNev6ThhK8X xozFYxGRBhb9BoIiF9DkdLFSLb9ylok476UL/405Uup82Z5eBfL2qrFEXV0LD4Ue0Sh12zyTuG3 IADO2XQ4i62yyZfy1WaXTlp1DO+WPJTDjXg== X-Google-Smtp-Source: AGHT+IGlFOx4f6b5Qn6EkhCuk4JQN2E+zuCd+wMYvq9TV5bKsvw+mh4wT3l1mW7RZopYjF9rkH3ZCKUuMJbD+13ebTA= X-Received: by 2002:a17:906:5f84:b0:a3c:1e4d:725a with SMTP id a4-20020a1709065f8400b00a3c1e4d725amr6725980eju.37.1708105552217; Fri, 16 Feb 2024 09:45:52 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> In-Reply-To: <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> From: Paul Procacci Date: Fri, 16 Feb 2024 12:45:47 -0500 Message-ID: Subject: Re: bhyve disk performance issue To: Chuck Tuffli Cc: Matthew Grooms , FreeBSD virtualization Content-Type: multipart/alternative; boundary="000000000000adcbaa06118350cb" X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Tbzqk0tfTz592y X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] --000000000000adcbaa06118350cb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Feb 16, 2024 at 12:43=E2=80=AFPM Chuck Tuffli wr= ote: > On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: > > Hi All, > > > I'm in the middle of a project that involves building out a handful of > servers to host virtual Linux instances. Part of that includes testing > bhyve to see how it performs. The intent is to compare host storage optio= ns > such as raw vs zvol block devices and ufs vs zfs disk images using hardwa= re > raid vs zfs managed disks. It would also involve > > > =E2=80=A6 > > Here is a list of a few other things I'd like to try: > > 1) Wiring guest memory ( unlikely as it's 32G of 256G ) > 2) Downgrading the host to 13.2-RELEASE > > > FWIW we recently did a similar exercise and saw significant performance > differences on ZFS backed disk images when comparing 14.0 and 13.2. We > didn=E2=80=99t have time to root cause the difference, so it could simply= be some > tuning difference needed for 14. > > =E2=80=94chuck > I myself am actually doing something very very similar. I was seeing atrocious disk performance until I set the disk type to nvme. Now it's screaming fast. disk0_type=3D"nvme" Not sure what yours is set at, but it might be worth looking into. ~Paul --=20 __________________ :(){ :|:& };: --000000000000adcbaa06118350cb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Feb 16, 2024 at 12:43=E2= =80=AFPM Chuck Tuffli <chuck@tuffli.= net> wrote:
On Fri, Feb 16, = 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm i= n the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


=E2=80=A6=

Here is a list of a few other things I'= d like to try:


1) Wiring guest memory (= unlikely as it's 32G of 256G )
2) Downgrading the host = to 13.2-RELEASE

FWIW we recently = did a similar exercise and saw significant performance differences on ZFS b= acked disk images when comparing 14.0 and 13.2. We didn=E2=80=99t have time= to root cause the difference, so it could simply be some tuning difference= needed for 14.=C2=A0

=E2=80=94chuck

I myself am actuall= y doing something very very similar.
I was seeing atrocious disk = performance until I set the disk type to nvme.
Now it's s= creaming fast.

disk0_type=3D"nvme"

Not s= ure what yours is set at, but it might be worth looking into.
~Paul
-- <= /span>
__________________
=
:(){ :|:& };:
--000000000000adcbaa06118350cb--