From nobody Fri Aug 18 00:06:48 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RRhxj1wVDz4qM6L; Fri, 18 Aug 2023 00:06:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RRhxh5Lz3z3DcM; Fri, 18 Aug 2023 00:06:48 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692317208; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=K8dWgF3evVKxHqyNvc8pNNNtuHpZz7ICMjsci5ZrTdA=; b=kSAy0JcaHyk9pBHGnZ+zidGYUy+A45fGMiU2PFrV9eEbxeywgKxKMCi6WGpNhc8XZuSWGi T3hyXe21b4tjpgZTWg6E3K60ApybTJgLJgHb2CNGtv8GNmNy6eaJBOQPj8C50MA9UR6Xnm rWuIBr0Fo/Z/tsZeMXR2rFcDkN1/vZfQv06gkp82SqpM58q4MICFPoK4YlSbQsQsurKW7b r4C2ztmEHvQZOkEo7ma8efrS0lzKM6fa3a9MnGfWXlv81vXG8hOeXg1a1B9UL6CdAPtcCo xzN9sPs4FjJ7PA17sJm6CIeSBvSg/+45IylpRkX15dz3dNr+Y6q/vLI9sLBzug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1692317208; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=K8dWgF3evVKxHqyNvc8pNNNtuHpZz7ICMjsci5ZrTdA=; b=XgOhIS0Of4ILOo5X4OUH6LhWteuOK//JoCCO/GYjN9axK025rNRkaALfEEudBTVIXpO7kT RWjB1O/G18ihKh1r1OH4db3aNl4axGzAk9h0MbGozp7cX/MYm6YVmEUsxT/Op1TJ4tuQ2P SzJZJ7HQrzLRErfiAUsrPboDmOJI7oV5h/xLkBWUvnEuEyWp7PwqV89huR8/gROLZFenzC EXmeaTz4XbVSkZ/cvWPAWWakiO4kuAAFN7RRmhpzkaj9p4LcMHNJ2p7j9cuiSbreMwF+EG Nega8TVjtI3H5qXaFl0pQqu6DZ6cz6mW3bq6X7EzKbIid6FRlRcbZDn8k/tX8A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1692317208; a=rsa-sha256; cv=none; b=KHLsprMiGfBGHZiVb7gRIMvvZdA9PjHkq4zEjgswAKz5eWbLtdenikT+hXWUkdKU5hNoRk AzMX6dPgiS6ZNPjpvEs+ZHpYCqQVTVhFf/YM6Vgtvyuz1lKdwoGXmR3Tx/idPmcAmRCn6/ IZqSBPTTY7mA/5grOumZyLGgROmXv30zjbIYd0KlOI9uad33/JZkN3N7UyqHqxdsC27EpL WdJaHzRLqoWJh4gGcyseTGBD9aRO6Qv/cidm7ILyMoqA+X5SpEDlZAYU7iVaXFyHSODJs0 j1hRNZvCpsRKoOiCJLUhFqqs0jBnffOnofXnFaK2v2VnTBiQcfNp+iiyWf5FMQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RRhxh4S5Gzv0j; Fri, 18 Aug 2023 00:06:48 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 37I06mJT096791; Fri, 18 Aug 2023 00:06:48 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 37I06mOf096788; Fri, 18 Aug 2023 00:06:48 GMT (envelope-from git) Date: Fri, 18 Aug 2023 00:06:48 GMT Message-Id: <202308180006.37I06mOf096788@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Ed Maste Subject: git: fa5f94140a83 - main - msi: handle error from BUS_REMAP_INTR in msi_assign_cpu List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: emaste X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: fa5f94140a83b4704c654ababd67cd9addb7cd29 Auto-Submitted: auto-generated The branch main has been updated by emaste: URL: https://cgit.FreeBSD.org/src/commit/?id=fa5f94140a83b4704c654ababd67cd9addb7cd29 commit fa5f94140a83b4704c654ababd67cd9addb7cd29 Author: Ed Maste AuthorDate: 2023-08-14 16:56:12 +0000 Commit: Ed Maste CommitDate: 2023-08-18 00:03:48 +0000 msi: handle error from BUS_REMAP_INTR in msi_assign_cpu Previously errors from BUS_REMAP_INTR were silently ignored, and we ended up with non-functional interrupts. Now we allocate and enable new vectors, but postpone assignment of new APIC IDs and vectors where we can, until after BUS_REMAP_INTR is successful. We then disable and free the old vectors. If BUS_REMAP_INTR fails we restore the old configuration, and disable and free the new, unused vectors. Thanks to AMD for providing hardware (with APIC IDs above 255) for testing. Reviewed by: jhb MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D41455 --- sys/x86/x86/msi.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/sys/x86/x86/msi.c b/sys/x86/x86/msi.c index 8751c621a5e1..7f4d87c09453 100644 --- a/sys/x86/x86/msi.c +++ b/sys/x86/x86/msi.c @@ -244,7 +244,7 @@ msi_assign_cpu(struct intsrc *isrc, u_int apic_id) struct msi_intsrc *sib, *msi = (struct msi_intsrc *)isrc; int old_vector; u_int old_id; - int i, vector; + int error, i, vector; /* * Only allow CPUs to be assigned to the first message for an @@ -274,31 +274,48 @@ msi_assign_cpu(struct intsrc *isrc, u_int apic_id) if (vector == 0) return (ENOSPC); + /* Must be set before BUS_REMAP_INTR as it may call back into MSI. */ msi->msi_cpu = apic_id; msi->msi_vector = vector; if (msi->msi_intsrc.is_handlers > 0) apic_enable_vector(msi->msi_cpu, msi->msi_vector); - if (bootverbose) - printf("msi: Assigning %s IRQ %d to local APIC %u vector %u\n", - msi->msi_msix ? "MSI-X" : "MSI", msi->msi_irq, - msi->msi_cpu, msi->msi_vector); for (i = 1; i < msi->msi_count; i++) { sib = (struct msi_intsrc *)intr_lookup_source(msi->msi_irqs[i]); - sib->msi_cpu = apic_id; - sib->msi_vector = vector + i; if (sib->msi_intsrc.is_handlers > 0) - apic_enable_vector(sib->msi_cpu, sib->msi_vector); - if (bootverbose) - printf( - "msi: Assigning MSI IRQ %d to local APIC %u vector %u\n", - sib->msi_irq, sib->msi_cpu, sib->msi_vector); + apic_enable_vector(apic_id, vector + i); } - BUS_REMAP_INTR(device_get_parent(msi->msi_dev), msi->msi_dev, + error = BUS_REMAP_INTR(device_get_parent(msi->msi_dev), msi->msi_dev, msi->msi_irq); + if (error == 0) { + if (bootverbose) { + printf("msi: Assigning %s IRQ %d to local APIC %u vector %u\n", + msi->msi_msix ? "MSI-X" : "MSI", msi->msi_irq, + msi->msi_cpu, msi->msi_vector); + } + for (i = 1; i < msi->msi_count; i++) { + sib = (struct msi_intsrc *)intr_lookup_source( + msi->msi_irqs[i]); + sib->msi_cpu = apic_id; + sib->msi_vector = vector + i; + if (bootverbose) + printf("msi: Assigning MSI IRQ %d to local APIC %u vector %u\n", + sib->msi_irq, sib->msi_cpu, + sib->msi_vector); + } + } else { + device_printf(msi->msi_dev, + "remap irq %u to APIC ID %u failed (error %d)\n", + msi->msi_irq, apic_id, error); + msi->msi_cpu = old_id; + msi->msi_vector = old_vector; + old_id = apic_id; + old_vector = vector; + } /* * Free the old vector after the new one is established. This is done - * to prevent races where we could miss an interrupt. + * to prevent races where we could miss an interrupt. If BUS_REMAP_INTR + * failed then we disable and free the new, unused vector(s). */ if (msi->msi_intsrc.is_handlers > 0) apic_disable_vector(old_id, old_vector); @@ -309,7 +326,7 @@ msi_assign_cpu(struct intsrc *isrc, u_int apic_id) apic_disable_vector(old_id, old_vector + i); apic_free_vector(old_id, old_vector + i, msi->msi_irqs[i]); } - return (0); + return (error); } void