From owner-freebsd-bugs@FreeBSD.ORG Thu Apr 9 16:59:08 2015 Return-Path: Delivered-To: freebsd-bugs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF08A87A for ; Thu, 9 Apr 2015 16:59:08 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C6AE779 for ; Thu, 9 Apr 2015 16:59:08 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t39Gx8kl081746 for ; Thu, 9 Apr 2015 16:59:08 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 199321] Total MSI-X vector allocation limited to 191 vectors Date: Thu, 09 Apr 2015 16:59:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jimharris@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Apr 2015 16:59:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321 Bug ID: 199321 Summary: Total MSI-X vector allocation limited to 191 vectors Product: Base System Version: 11.0-CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: jimharris@FreeBSD.org System Configuration: 2x 18C E5-v3 CPU (SMT enabled) 2x X520 Intel Ethernet 10Gb Adapter (4 ports total) 11.0-CURRENT (r281283) ixgbe driver tries to allocate full range of 64 MSI-X vectors per port - first two ports succeed, but third and fourth ports fail and fall back to a single MSI vector. msix_alloc() tries to use intr_next_cpu() to round-robin MSI-X vectors across all cores. But driver initialization happens before APs are started, so intr_next_cpu() just returns the BSP's apic ID 0. Each local APIC is limited to APIC_NUM_IOINTS==191 vectors, so a single local APIC cannot handle the 64 vectors for each of the 4 ports. intr_shuffle_irqs runs in SI_SUB_SMP phase and is intended to shuffle iras off of core 0 to APs, but at this point it is too late - pci_alloc_msix() already reported to ixgbe(4) that it could not allocate all 64 vectors. x86 defines NUM_MSI_INTS as 512, but this bug really restricts number of MSI-X vectors to APIC_NUM_IOINTS (191). Impact is that some drivers/devices which can take advantage of multiple queues and one interrupt per queue must fall back to a single interrupt. There may also be inconsistency across platforms based on device enumeration - whichever devices enumerate first will get its allotment of MSI-X vectors. This problem will become exacerbated as NVMe SSDs become more prevalent, which also try to allocate 32-64 MSIx vectors each. There are systems available already with 8 NVMe SSDs in a single system. -- You are receiving this mail because: You are the assignee for the bug.