From nobody Wed Mar 2 20:07:34 2022 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B938B19F5426 for ; Wed, 2 Mar 2022 20:07:38 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (ns.NFBCAL.ORG [157.22.230.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "nfbcal.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K84sj3lVpz4q3Z for ; Wed, 2 Mar 2022 20:07:37 +0000 (UTC) (envelope-from buhrow@nfbcal.org) Received: from nfbcal.org (localhost [127.0.0.1]) by nfbcal.org (8.15.2/8.14.1-NFBNETBSD) with ESMTPS id 222K7YTq006622 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 2 Mar 2022 12:07:34 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.2 at lothlorien.nfbcal.org Received: (from buhrow@localhost) by nfbcal.org (8.15.2/8.12.11) id 222K7Y7I013387; Wed, 2 Mar 2022 12:07:34 -0800 (PST) Message-Id: <202203022007.222K7Y7I013387@nfbcal.org> From: Brian Buhrow Date: Wed, 2 Mar 2022 12:07:34 -0800 In-Reply-To: X-Mailer: Mail User's Shell (7.2.6 beta(4.pl1)+dynamic 20000103) To: Ze Dupsys Subject: Re: ZFS + FreeBSD XEN dom0 panic Cc: freebsd-xen@freebsd.org, buhrow@nfbcal.org X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (nfbcal.org [127.0.0.1]); Wed, 02 Mar 2022 12:07:35 -0800 (PST) X-Rspamd-Queue-Id: 4K84sj3lVpz4q3Z X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of buhrow@nfbcal.org designates 157.22.230.125 as permitted sender) smtp.mailfrom=buhrow@nfbcal.org X-Spamd-Result: default: False [-3.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:ns.nfbcal.org:c]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nfbcal.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-xen]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7091, ipnet:157.22.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org Hello. Given Roger's message and the presentation of the errors you're seeing, I think we're both in agreement that you're running out of memory. Below I've listed some suggestions on how you might go about making your system more stable, and, possibly figuring out where the trouble is, exactly. When I was building my infrastructure, I ran into a bunch of edge conditions and weird bugs as well, so some ideas that may not seem obvious, may, in fact, work, simply because they cause you to skirt some latent bug in the system. Here's what my setup looks like, in case it helps. 1. Use FreeBSD-12.1 or 12.2, and xen-4.13 or xen-4.14 (Using FreeBSD as dom0.) 2. Use a minimum of 8G of RAM for the dom0. 3. Put the root filesystem on a mirrored set of partitions or drives; I do not use ZFS as a root filesystem, but put root and swap on a mirrored media using the gmirror(8) utility and mirror driver. In this way, if the system needs to use swap, it can do so without touching ZFS and, hopefully, without requiring additional memory to get to the swap. 4. Create a network bridge using the bridge(4) driver for attaching the domu's to the network. You can either attach this bridge to one of your physical network interfaces, or you can attach it to a vlan(4) interface, thus allowing separation between the network the dom0 lives on and the network(s) the domu's live on. I do not do any firewalling or natting on the dom0, it's just a packet forwarder. 5. Create a zpool for your domu disks. It can be comprised of partitions on the same disks you use for booting, or it can live on separate disks. 6. Create zvols for the disks you want to attach to your domu's. I typically attach 1 or two disks to each domu. I've not experimented with attaching large numbers of disks to a single domu, instead partitioning the virtual disks from inside the domu itself if I want separation. I use the xbd(4) block driver for attaching the disks as raw block devices to the domu's. I've not played with the other virtual drivers. ( I have played with HVM hosts using the qemu drivers and found things sort of work, but are not stable; though the dom0 is fine.) While I've not tried oversubscribing the RAM reserved for domu use on the systems I run, if the domu's you're running support the balloon driver, I think you will have success in doing that. I disagree with your comment that adding memory just puts off the inevitable problem. Once you figure out how much memory your dom0 needs to do its work, then it's just a matter of deciding how much memory you want for your domu's. Having said that, however, it's important to figure out what the minimum amount of memory you need for your dom0 is; going below that threshold will result in instability. As I think abut your issue, I'm guessing you'll find things are much more stable and you'll be happy with 8G for the dom0 if you put the root and swap on a mirror and get it out of ZFS. I think you're runing into a situation where your system is short of memory, it tries to swap, the swap is on ZFS, which needs more memory to fulfill the paging request and, well, you get the idea. The system I'm sending this from, for example, has 8GB devoted to the dom0 and it's currently using 36Mb of swap. It's currently been up for 69 days, which is the time since the last power failure. I have another system, the one running pfsense as one of its domu's, which has 8GB of memory, no swap configured, and it's been up for 740 days. Hope these notes are helpful. -Brian