From owner-freebsd-virtualization@freebsd.org Mon May 3 11:14:57 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E009C62F345 for ; Mon, 3 May 2021 11:14:57 +0000 (UTC) (envelope-from xi@borderworlds.dk) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FYgNw65vXz4jTZ for ; Mon, 3 May 2021 11:14:56 +0000 (UTC) (envelope-from xi@borderworlds.dk) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id E1F0A5C0085 for ; Mon, 3 May 2021 07:14:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 03 May 2021 07:14:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=borderworlds.dk; h=subject:to:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm2; bh=u c1rAcnIH+rwjiHGGaghdlUXVWc9hCqox2duLTH7odg=; b=OtlicXL+KBiGHZR73 qLuze4IecGE2rBA2emOwCSiWgPNwZvXplknvcmUrfNnD61oMj7EH68qx6qi/1bBO 3bgR7/GBUaKoeuAnRNZ3dN5JqXPl8jnsCdevi1WnQxCF6yKlQSvD5rOzq5j3WQIa eGS/bCvk1Ik0uOTfEsZ/8fcg2IjDh3EsdSEsbJ6ENw3LCKiGlwmzdEk/Rx5M3gOd wq0HGgyRr8Z87I1qy6oZDNcfld3Zv7/x9qjXnjlhkh1nR/gqZyi+eyZV4tZ8L/U6 6ckDNJkqOk65xJTA7ZYcYPrGT5afzPL6j1VjpRdohr8hA7y+q9HbSKmsf+JPiSX6 RvnLg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=uc1rAcnIH+rwjiHGGaghdlUXVWc9hCqox2duLTH7o dg=; b=N3PibY+npE/myBeu856wOenNY130myVdnion+kOXnkfqj4DRqvHhvPnT/ yyz7p/UsoYwyhoaqpx6tUF5Bb+StejvSmJKfGjtGTDFJpnUyEJivJCNfIBNNpzXU U+/PfYsm111kZRTCm3yepZGYKqpNyYemUtWNCVtCohPw/jcmtzuNlmNMfgO+i103 AvqSLUPFf2bjfkGdW7QUD6mdzXy3zrKlulw5sDMv6uMYhX8bh3lD3gOnzbjodZWm KEzs8SBS0uUgUhdriqCdBfencH8EgrozgRTdYIpgJucsClfEteYr8AInlykpzld4 ZpP4K9iP3Hd+HtjAUhV1rOomIm0yw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduledrvdefgedgfeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefuvfhfhfhokffffgggjggtgfesth ekredttdefjeenucfhrhhomhepvehhrhhishhtihgrnhgpufhtoohrkhcuoeigihessgho rhguvghrfihorhhlughsrdgukheqnecuggftrfgrthhtvghrnhephffffeeluedtfefhje ekleevtddtvdehgfehfefhudetkefggfduueeuffevvdegnecuffhomhgrihhnpegsohhr uggvrhifohhrlhgushdrughknecukfhppeeivddruddtjedrjedrudeivdenucevlhhush htvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeigihessghorhguvghr fihorhhlughsrdgukh X-ME-Proxy: Received: from baku.borderworlds.dk (3e6b07a2.rev.stofanet.dk [62.107.7.162]) by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 3 May 2021 07:14:55 -0400 (EDT) Subject: Re: FreeBSD 13 on bhyve freezes under load To: freebsd-virtualization@freebsd.org References: <587b3653-0433-ef46-a53d-d6ab0841f993@borderworlds.dk> <20210503123144.134c209c@bsd64.grem.de> From: =?UTF-8?Q?Christian_St=c3=a6rk?= Organization: The Border Worlds Message-ID: <9ddd42be-e316-c813-8132-b7b90b694f34@borderworlds.dk> Date: Mon, 3 May 2021 13:14:53 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <20210503123144.134c209c@bsd64.grem.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 4FYgNw65vXz4jTZ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=borderworlds.dk header.s=fm2 header.b=OtlicXL+; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=N3PibY+n; dmarc=none; spf=pass (mx1.freebsd.org: domain of xi@borderworlds.dk designates 66.111.4.25 as permitted sender) smtp.mailfrom=xi@borderworlds.dk X-Spamd-Result: default: False [-1.60 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[66.111.4.25:from]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.25:c]; TO_DN_NONE(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[borderworlds.dk:+,messagingengine.com:+]; RECEIVED_SPAMHAUS_PBL(0.00)[62.107.7.162:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[66.111.4.25:from]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[borderworlds.dk:s=fm2,messagingengine.com:s=fm2]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.25:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[borderworlds.dk]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[66.111.4.25:from:127.0.2.255]; NEURAL_SPAM_SHORT(1.00)[1.000]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 May 2021 11:14:57 -0000 Hi Michael, I am not using ZFS and have plenty of free diskspace: Filesystem     Size    Used   Avail Capacity Mounted on /dev/ada0p2    190G     92G     83G    52%    / top looks like this on a not-hung system: last pid: 38292;  load averages:  0.91, 0.87,  0.88    up 0+01:13:16  13:12:12 32 processes:  2 running, 29 sleeping, 1 zombie CPU:  5.5% user,  0.0% nice,  7.8% system,  7.0% interrupt, 79.7% idle Mem: 59M Active, 3549M Inact, 1090M Wired, 610M Buf, 2250M Free Swap: 24G Total, 24G Free   PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND 38079 root          1  22    0    99M    43M RUN      0:03  14.02% pkg-static 38285 xi            1  20    0    14M  3612K RUN      0:00   0.10% top 37889 root          1  22    0    15M  3680K nanslp   0:02   0.05% sh   759 root          1  20    0    17M  6200K select   0:04   0.01% tmux 37881 xi            1  20    0    21M  9252K select   0:00   0.00% sshd   600 ntpd          1  20    0    17M  4980K select   0:00   0.00% ntpd   749 xi            1  20    0    21M  8804K select   0:03   0.00% sshd 37882 xi            1  20    0    15M  5584K pause    0:00   0.00% zsh 37603 root          1  20    0    15M  3900K select   0:02   0.00% sh 78529 root          1  26    0    15M  3892K piperd   0:00   0.00% sh 36203 root          1  47    0    23M    10M wait     0:00   0.00% perl   750 xi            1  25    0    15M  5296K pause    0:00   0.00% zsh   542 root          1  20    0    13M  2632K select   0:00   0.00% syslogd   341 root          1  20    0    11M  1424K select   0:00   0.00% devd   753 root          1  20    0    16M  4428K pause    0:00   0.00% csh 37826 root          1  20    0    21M  9228K select   0:00   0.00% sshd 37790 root          1  52    0    15M  3896K wait     0:00   0.00% sh It occurred to me that I could try using ahci-hd instead of virtio-blk, so I am trying that now to see if it makes a difference. Best regards Christian On 5/3/21 12:31 PM, Michael Gmelin wrote: > > > On Mon, 3 May 2021 11:14:45 +0200 > Christian Stærk wrote: > >> Hello, >> >> I recently upgraded one of my bhyve guests to FreeBSD 13.0-RELEASE. >> The host is running 12.2-RELEASE-p4. >> >> The guest in question is used for building packages with poudriere >> and is always the first one to get upgraded to new releases. >> >> The guest freezes consistently every time poudriere tries to build >> the thunderbird package. Although it is not a complete freeze. It >> responds to ping and programs like top and systat continue updating. >> However, all I/O operations seem to hang. >> >> systat looks like this when it happens: >> https://borderworlds.dk/~xi/FreeBSD%2013%20frozen.png >> >> >> What is the best way to debug this? > Sharing top output might help (memory/swap and ZFS arc usage). Also, > make sure you're not low on diskspace (these are the two most common > issues I encountered with hanging bhyve VMs - out of non-virtual memory > and out of diskspace of the host machine filesystem). > > Some basic info on the host and your VM configuration might also help > others to reproduce the problem. > > -m >