From nobody Mon Jun 17 08:21:15 2024 X-Original-To: freebsd-xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4W2jXH3WKDz5Ndmw for ; Mon, 17 Jun 2024 08:21:35 +0000 (UTC) (envelope-from roger.pau@cloud.com) Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4W2jXG3bPCz4K9W for ; Mon, 17 Jun 2024 08:21:34 +0000 (UTC) (envelope-from roger.pau@cloud.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=citrix.com header.s=google header.b=meLXV7Uq; dmarc=pass (policy=reject) header.from=citrix.com; spf=pass (mx1.freebsd.org: domain of roger.pau@cloud.com designates 2607:f8b0:4864:20::735 as permitted sender) smtp.mailfrom=roger.pau@cloud.com Received: by mail-qk1-x735.google.com with SMTP id af79cd13be357-7979198531fso303146885a.0 for ; Mon, 17 Jun 2024 01:21:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1718612493; x=1719217293; darn=freebsd.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ozVvVH2rVQ0iNNvlFeXELhTlIxBkL/Cz7ft6ZryBX6s=; b=meLXV7UqNRshhY08CEz6h+/yolIAicepcGsvxFQ4hYfUj5CT7V36nzXzJu7SV3Q1Jv k8k1uXrxXcFwRK5c9siciSXcpGTC7y1s8mQvqm1p89PyZdSw08v1ZCS7qUUGnxo2L3yr +QA3ELUX6iLG5m/sm14IWP97pR3DVmIvRzi6o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718612493; x=1719217293; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ozVvVH2rVQ0iNNvlFeXELhTlIxBkL/Cz7ft6ZryBX6s=; b=HBK4i4xpOnbPYn3RjWjTt9d377LztSeW/IHcmXhuJGB1BO3n1Ob1xw+7VZuKLjOrNq z1lbo3UDx8rjDwc0P/J2n7o5mNpGLB/Z4yp3Rmh+cS3OTtNj6vTcg603uoVzn0dO4lUK 9hdpQ2ahbhe2gzMpze9+vwzX5/p0RblDWjRpKfbHOny/v+zeBwccxzwHifxTr8iFxM9s 2FUvaleQtXGeBqesGcMrGW1e3f2zjg2y0Cwltvpe8VgR2VlKye8zLUOsGGA1bhpZd9pX Ham/saLg/fyOWTXRP5qmCO7fRp6VRxO/w+m3pQyhl001WBIUVpXbluFYhF1WE2BkO9nf FFnw== X-Gm-Message-State: AOJu0Yzn0jux0gMAif8UjNJ7yjQbvblX6P1WspaCB8AFApwQ6j/R1+G8 82OUBgZTC3drd0ae4IHRg925T9b8LxI+AG2+opysVGTvuPSDBkhUPuXqZpE4qCYsaOJl1yPY392 7 X-Google-Smtp-Source: AGHT+IEbs7cOHNWHy20caarAhMMdxVVZsuqgy1yFTAk8e2U9vfsjm/Jz7P5QyWfZHDUaVmfL5xynMA== X-Received: by 2002:ad4:4381:0:b0:6ae:e4f7:6c5f with SMTP id 6a1803df08f44-6b2afd6156cmr87749206d6.52.1718612493298; Mon, 17 Jun 2024 01:21:33 -0700 (PDT) Received: from localhost ([213.195.124.163]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b2a5bf2832sm53029486d6.24.2024.06.17.01.21.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jun 2024 01:21:32 -0700 (PDT) Date: Mon, 17 Jun 2024 10:21:15 +0200 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Marcin Cieslak Cc: freebsd-xen@freebsd.org Subject: Re: 13.2-RELEASE-p11 dom0 fails to boot unlike -p10 Message-ID: References: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-xen@freebsd.org Sender: owner-freebsd-xen@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.995]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[citrix.com,reject]; FORGED_SENDER(0.30)[roger.pau@citrix.com,roger.pau@cloud.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[citrix.com:s=google]; MIME_GOOD(-0.10)[text/plain]; MISSING_XM_UA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[roger.pau@citrix.com,roger.pau@cloud.com]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-xen@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::735:from]; RCPT_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[citrix.com:+] X-Rspamd-Queue-Id: 4W2jXG3bPCz4K9W On Sat, Jun 15, 2024 at 04:58:16PM +0000, Marcin Cieslak wrote: > (resend, wrong address) > > Hello, > > After upgrading 13.2-RELEASE-p10 amd64 machine to 13.2-RELEASE-p11, > the dom0 stops booting, last lines being: > > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > > After the pci0 the following would normally follow: > > pcib1: irq 16 at device 1.0 on pci0 > pci1: on pcib1 > vgapci0: port 0xf000-0xf03f mem > 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 > vgapci0: Boot video device > > > GENERIC kernel. > Xen version used: > > # pkg -c /mnt info -x xen > edk2-xen-x64-g202308_4 > xen-kernel-4.18.2.20240411 > xen-tools-4.18.2.20240411_1 > > xen 4.18.0 also didn't boot. > > I realized that reverting 8f1f4e60ceb9b8e5eddd54cf1fde62944f56eaa4 > fixes the issue. That's weird. 8f1f4e60ceb9b8e shouldn't have anything to do with Xen. I suspect if anything this is more likely a code generation issue with clang/llvm, rather than source code issue. Tested right now and the current HEAD: ba95b4aea789 cxgbe(4): New knob to limit driver to the specified types of doorbells. Seems to work fine for me. A couple of things to test: Can you check whether just re-building f5ac4e174fdd3497749e351c27aafb34171c5730 without any reverts also leads to a non-working kernel? Can you boot with the following in loader.conf: boot_verbose="YES" xen_kernel="/boot/xen-debug" And see if it makes any difference regarding the printed messages? Does the box have a serial console so you can get Xen messages? Regards, Roger. From nobody Mon Jun 17 15:17:52 2024 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4W2tmg0gSFz5P3rL for ; Mon, 17 Jun 2024 15:17:55 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4W2tmf5phwz52LB; Mon, 17 Jun 2024 15:17:54 +0000 (UTC) (envelope-from kevans@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1718637474; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UMtZC0CjiTK7DC9XTezfnQg6Cmvs/2WbjLUrhetfgqw=; b=uUvwSosX4Afn1HzmKu6Pf7HqYSJWcUdkcCG3ti/NMu3goRyN6ed4RfcwHAzMxnfVZxTZdZ GChcdSHdAuyG7ziOJbpstj8+IRLY89IfdYm5ax9v1O//iKufIG4kk4PdOQ0RPNdE4ahP+p VP0sYyTbzobT1BNZAKKon2Z1tuj2ATBLxwgUyDTo+KK+rhg93ddz9brfspGR/7Ao3cwHzv A1Kb+2i115y+wvob7IK7ZnvQY1Y1Y+bTHrj1eS5xJgx2LkhPV+pvZ4xVfbbZ4MgyMhnFGB G1lA/U4ZkaxmRWFDNjPGK05wRpwcUEJaEdjHNkmKa5IviTQeL6Mn6Lk4SCFGeQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1718637474; a=rsa-sha256; cv=none; b=eAxDOPSauMqpbUptTmGCG4Wp+VRqJVLYM7osBG5Ql/L3FQNyBzbGb5bxeYNbDH70459uu0 fJEbYZA8LnY1PhJ7jhhUyzeNfYAoLGuOqh9V0RQa702pQmAQSW85K2y39gKCKRBCoyWuEw lS2JC4Dd/suQ8TbzU0mqP2OFb72bioFvIAFH9mrhQ7cpfbP5EduY+6MrMkOVtJbbyTMrJV ef2qn+EpDTMIAoJoYSzjnMYc7nrQsvvAUm2UO3oVWDLi5CCaF6Mt1iShE32+gUDu+N9OR7 j1snRK5Q0Vc3oKdr+WfT24ckLZT30Q5reWDtB+oVmE4seQdkrY9fV/HCDvl31w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1718637474; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UMtZC0CjiTK7DC9XTezfnQg6Cmvs/2WbjLUrhetfgqw=; b=CXhxYuaCerOiSkJAcRKx0nXO37okB+jhoRok72/aKO6tytOBhjZFCy5WKfPr0NAvcHtDLR 83ILgyqboWRC5tf9Gd24Qu7FKP4biFE5d71w4iPvJ2B25zJ+S7Q/NCbqkC1e3hUbRdX4vY abrllEo3LyZ0cLkWW7BdVcZNZB4AIj1dkTojv7ccW5NsHOVJqVNh2WBb50oVY6hO5Mtf4j baWfarW1YVXcd0WJZx4pHRFNmaiS6E0Y36rGweT6E1DflJAdQUKZ2W3HfKB9Cs0EhIX85I NOQwa+lHmQpjw3lZamfyebGVsGi/shygF2PVxaKUnmP4pxdlsaZMhWzKS3c0rw== Received: from [10.9.4.95] (unknown [209.182.120.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: kevans/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4W2tmf3DBSzRQL; Mon, 17 Jun 2024 15:17:54 +0000 (UTC) (envelope-from kevans@FreeBSD.org) Message-ID: <102314ab-334c-4652-aa9b-d155c49fd186@FreeBSD.org> Date: Mon, 17 Jun 2024 10:17:52 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-xen@freebsd.org Sender: owner-freebsd-xen@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: 13.2-RELEASE-p11 dom0 fails to boot unlike -p10 To: Marcin Cieslak , xen@freebsd.org References: Content-Language: en-US Cc: cperciva@FreeBSD.org From: Kyle Evans In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 6/15/24 11:24, Marcin Cieslak wrote: > Hello, > > After upgrading 13.2-RELEASE-p10 amd64  machine to 13.2-RELEASE-p11, > the dom0 stops booting, last lines being: > > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > > (see screenshot) > > After the pci0 the following would normally follow: > > pcib1: irq 16 at device 1.0 on pci0 > pci1: on pcib1 > vgapci0: port 0xf000-0xf03f mem > 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 > vgapci0: Boot video device > > > GENERIC kernel. > Xen version used: > > # pkg -c /mnt info -x xen > edk2-xen-x64-g202308_4 > xen-kernel-4.18.2.20240411 > xen-tools-4.18.2.20240411_1 > > xen 4.18.0 also didn't boot. > > I realized that reverting 8f1f4e60ceb9b8e5eddd54cf1fde62944f56eaa4 > fixes the issue. > > I am a bit puzzled, since I don't even load if_wg, but > checking out f5ac4e174fdd3497749e351c27aafb34171c5730 > and reverting 8f1f4e60 helped. > There's 0% chance of that change directly causing this, especially if you don't even load the module or build it into your kernel. Even if you did, this is too early for tunnels to be constructed and pass traffic which would certainly be a prerequisite. > Reverting f3195cc08ccc99365ec00900a3c1abc59ceefc9c alone didn't fix > it (I tried this first). > > What's going on..? > I would rather suspect that for some reason or another your kernel is linking in a different order between the two builds and you're getting really unlucky (e.g., maybe there's a pair of sysinits that, when flipped, are causing this). CC'ing cperciva@ because he's done more recent work with sysinit reordering (though I don't think anything that landed in 13.x) in case this particular case rings a bell. > Marcin Thanks, Kyle Evans