From nobody Tue Feb 27 10:21:43 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TkYSc45fZz5CtZD for ; Tue, 27 Feb 2024 10:22:08 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TkYSb6GPhz4qxv for ; Tue, 27 Feb 2024 10:22:07 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=icCT19zs; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of gusev.vitaliy@gmail.com designates 2a00:1450:4864:20::132 as permitted sender) smtp.mailfrom=gusev.vitaliy@gmail.com Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-513143d3c40so94989e87.3 for ; Tue, 27 Feb 2024 02:22:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709029325; x=1709634125; darn=freebsd.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=u35HOfZx09c0Jkx1DaHouJbTlLDHGu3cYqXhRUVbGIw=; b=icCT19zsC+ZKBGyrBNJToUHkzFnWbFi2LIPyPluBVqwBNmTVbY6VLsUAMciheRVrXO t03biV4rKNMX1mSee/qTDPVv5QInssA0cUMWslxCwXu2hT731gHTpdXRo307Do+3PhlX ZbuV/TYgoZZ7vlFWEMuDtQ6yKRn0g48P4JkxV4ClvyTTSy8Iyvu9Y+0uJ10vtB8Qp1T5 fRE0f0XcAU3CHvtFnj+B42M7E2FolJplDUAb2inMxDDlze6N6FA6aZ+vAA69ulJ3o0C8 5uO1XfCQ5x3crADpjuW7Devjaeae6nzoe6Z6M6J4FK0MlcyLCAchHvc3CTW5L40eZzTJ xH1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709029325; x=1709634125; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u35HOfZx09c0Jkx1DaHouJbTlLDHGu3cYqXhRUVbGIw=; b=cTJRQ7ZVW+ldIplltKQ+egmrs+6VDnEMmrAWA9xDSdy8FRZ3YL5ujPds5BDS8RLQjJ XnAgXsqJ6CwL8liMZMIL4TrWjUGC23A1jXOGf7gyH/f8uxwLBg8mSrGKB5r/+zBw59uB C9pFIfiGf3nXve5JtuFwsMdOIOI0oPiLB5NCAcOgYyG1cbUJPITyhsgXeU1i7OgKkSuD 8c9d7IfPJhElAle3u2JNxH34wFSqwwztQcUK23JboOPbm/UU2P5F4gY/h/N3DJTZkPMT iszS/z85Lh/OLm5zOSM4lJiRoTHfhBGFDgjfTupPyM+3jDmW+Hdg6/rgS3/JqBZsw47v l1lA== X-Gm-Message-State: AOJu0Yx13yreQMz16SJNTaTzf0qeZm46jrr4efqX6LlZapWpV0FMcyWS cFDDEz9e8dDC6pZlUH/EhQBI+TZtxUBamOU1vYp6QgHMTTWeJZvQiBlJFmr6oh2jmw== X-Google-Smtp-Source: AGHT+IHttSmbTXvL1KiiSWxtGaUrv1BZ0niufj5YCqPffhupgstsD3uRJB/xpDzcOIUnhSFiNgFuaw== X-Received: by 2002:a05:6512:234d:b0:512:d89e:946 with SMTP id p13-20020a056512234d00b00512d89e0946mr7427163lfu.44.1709029324554; Tue, 27 Feb 2024 02:22:04 -0800 (PST) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id u7-20020a056512040700b00513131a218fsm70990lfk.97.2024.02.27.02.22.04 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 27 Feb 2024 02:22:04 -0800 (PST) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\)) Subject: Re: bhyve disk performance issue Date: Tue, 27 Feb 2024 13:21:43 +0300 In-Reply-To: Cc: virtualization@freebsd.org To: Matthew Grooms References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net> X-Mailer: Apple Mail (2.3774.400.31) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.48)[-0.483]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; APPLE_MAILER_COMMON(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::132:from] X-Rspamd-Queue-Id: 4TkYSb6GPhz4qxv --Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, > On 23 Feb 2024, at 18:37, Matthew Grooms wrote: >=20 >> ... > The problem occurs when an image file is used on either ZFS or UFS. = The problem also occurs when the virtual disk is backed by a raw disk = partition or a ZVOL. This issue isn't related to a specific underlying = filesystem. >=20 Do I understand right, you ran testing inside VM inside guest VM on = ext4 filesystem? If so you should be aware about additional overhead in = comparison when you were running tests on the hosts. I would suggest to run fio (or even dd) on raw disk device inside VM, = i.e. without filesystem at all. Just do not forget do =E2=80=9Cecho 3 > = /proc/sys/vm/drop_caches=E2=80=9D in Linux Guest VM before you run = tests.=20 Could you also give more information about: 1. What results did you get (decode bonnie++ output)? 2. What results expecting? 3. VM configuration, virtio-blk disk size, etc. 4. Full command for tests (including size of test-set), bhyve, etc. 5. Did you pass virtio-blk as 512 or 4K ? If 512, probably you should = try 4K. 6. Linux has several read-ahead options for IO schedule, and it could = be related too. Additionally could also you play with =E2=80=9Csync=3Ddisabled=E2=80=9D = volume/zvol option? Of course it is only for write testing. =E2=80=94=E2=80=94 Vitaliy --Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi,


On 23 Feb 2024, at 18:37, Matthew Grooms = <mgrooms@shrew.net> wrote:

...
The problem occurs when an image file is = used on either ZFS or UFS. The problem also occurs when the virtual disk = is backed by a raw disk partition or a ZVOL. This issue isn't related to = a specific underlying = filesystem.


Do I = understand right, you ran testing inside VM inside guest VM  on = ext4 filesystem? If so you should be aware about additional overhead in = comparison when you were running tests on the = hosts.

I would suggest to run fio (or even dd) = on raw disk device inside VM, i.e. without filesystem at all.  Just = do not forget do =E2=80=9Cecho 3 > = /proc/sys/vm/drop_caches=E2=80=9D in Linux Guest VM before you = run tests. 

Could you also give more = information about:

 1. What results did = you get (decode bonnie++ output)?
 2. What results = expecting?
 3. VM configuration, virtio-blk disk size, = etc.
 4. Full command for tests (including size of = test-set), bhyve, etc.
 5. Did you pass virtio-blk as 512 = or 4K ? If 512, probably you should try 4K.
 6. Linux has = several read-ahead options for IO schedule, and it could be related = too.

Additionally could also you play with = =E2=80=9Csync=3Ddisabled=E2=80=9D volume/zvol option? Of course it is = only for write = testing.

=E2=80=94=E2=80=94
Vitaliy

= --Apple-Mail=_2DCCC81D-8089-4C12-92E2-6A195E662DF6--