From nobody Thu Feb 8 00:25:22 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVd753vQrz58PQD for ; Thu, 8 Feb 2024 00:25:37 +0000 (UTC) (envelope-from loren.lang@intel.com) Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mgamail.intel.com", Issuer "Sectigo RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVd7354tQz4WJd for ; Thu, 8 Feb 2024 00:25:35 +0000 (UTC) (envelope-from loren.lang@intel.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=intel.com header.s=Intel header.b=VZSh88h8; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:reject}"); dmarc=pass (policy=none) header.from=intel.com; spf=pass (mx1.freebsd.org: domain of loren.lang@intel.com designates 192.198.163.13 as permitted sender) smtp.mailfrom=loren.lang@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707351935; x=1738887935; h=date:from:to:subject:message-id: content-transfer-encoding:in-reply-to:mime-version; bh=6vZY9QxBG2cHVJrdILo3F9VDknOidKiJ0vPQQHwZSsk=; b=VZSh88h8CJZsuwjbaHt9yW5MknbHDobMAWxPSS8LeL3rX2b48YUlIJcU w3U5VlTDfUmvuxE7G7tn0Wq2alHWD5iqzKZVIU+eICD3WUEyBG9R8G2NY vnf100X3/5t+eqUbVDhDMNpYyC0jHHuifaNKe8jau4O20NqPf4HOve235 Ekd3+prlCwKc+HDy75TQ+x80QHJ1nGXOrVHaHYbXN5h0iY/STPzsUjZWU RPUTog4yYYrac2bcddDXe/36hqB5mtVF6on9mCC8hBqnTPukz5xjwB4qq oOgN9AD3Xb1hsbxxME8dNUAaTXiX5hHpXefFVwO4vJJsJtRMaYmXhn6ZJ Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10977"; a="4098908" X-IronPort-AV: E=Sophos;i="6.05,252,1701158400"; d="scan'208";a="4098908" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Feb 2024 16:25:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,252,1701158400"; d="scan'208";a="1814080" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by orviesa006.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 07 Feb 2024 16:25:28 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Wed, 7 Feb 2024 16:25:27 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Wed, 7 Feb 2024 16:25:27 -0800 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (104.47.73.168) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 7 Feb 2024 16:25:26 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BXeVCPzodv2yVDsdTGh5qPaT7GZiMVeO6HsJOfqGiL9xuVn9ZS9kLi91loI2UxFTB31CFwejG0BjzfPAnhhIlwq/wwfFoVKlZB+jsExerezkL1XTdBK/80zv12GnLby/S3nkwr0D4En3K9FVYIZNIQNJrKRAEEM0GiqexbGvbjcECwJDKJ5DwrIL27rH5nltQrwIXQxd6gS3S3qgCxuez3SZ8ZphvJvrYpHvbSxAmzF5wqi7tm9zActxdCw3Qi1xvX5rALOGWdG6kI28+reuD97MEbA6n6o7Lx3zJ7zXpDTrOCsZw9QV9oeIFQ/bFxlwuaukfaPMVHVBbyKxTNhtjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jPuCU2LoAcRHUVlYJeO7HVKqxdoZz1DhEvBD5maw6Z8=; b=AgzpLHrz3yOtUqCjYiPYGMLjkxC/U2JvvPcR4BhLaM0ZvjzYyxZNCmjN0JKjFqkFlRjlv8SVvzoZgfxTJ0xYrqBzDqMG+BXeQphJhTB4g4c4+2I4dJP/H5NHuiXBBYpi0wrhZ9LeFUHwazFJiKHnIlSTa3q3opjc/CeJSXfep6dNbJ0MTnKYj8/6DJRopszxHUrK0EQpw9OI7DE4CeJZlvNlX2oVlYxU5rmbjPhPaqoG5bYAuKfpM9kt1WwdIg4dOKwcTxYCV1rx/ffw1eOOS4AtgK/3qDEOuc70Oc/4Ne6xU1QGRXLBSvJy3Ysi2jdcp1c7xHotrLLiIrshOw8Ktw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from LV2PR11MB6048.namprd11.prod.outlook.com (2603:10b6:408:178::19) by CYYPR11MB8330.namprd11.prod.outlook.com (2603:10b6:930:b8::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7249.38; Thu, 8 Feb 2024 00:25:25 +0000 Received: from LV2PR11MB6048.namprd11.prod.outlook.com ([fe80::16db:9885:b358:ad64]) by LV2PR11MB6048.namprd11.prod.outlook.com ([fe80::16db:9885:b358:ad64%6]) with mapi id 15.20.7249.035; Thu, 8 Feb 2024 00:25:25 +0000 Date: Wed, 7 Feb 2024 16:25:22 -0800 From: "Loren M. Lang" To: "freebsd-virtualization@freebsd.org" Subject: Re: if_ixl: Admin Queue memory allocation issue hangs the server Message-ID: Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <7F49AE24-D3BC-4103-A522-75CB7BE477D0@freebsd.am> X-ClientProxiedBy: MW4PR02CA0008.namprd02.prod.outlook.com (2603:10b6:303:16d::27) To LV2PR11MB6048.namprd11.prod.outlook.com (2603:10b6:408:178::19) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR11MB6048:EE_|CYYPR11MB8330:EE_ X-MS-Office365-Filtering-Correlation-Id: cee96db2-2552-4e39-4ce5-08dc283c71df X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bPYHNVAEcjn3EdGJp2A3F0IlaRJeSxbQxXQQ3pFb6y0gAbNahAA9J6Ot5R395Ijn+kNpEiOI5EGEuNznr6adFIuHHOquCVGrg8VYNoDBJ0yGlJaql37vLoFiZeeWjNKrnC2j61NBbx9QRkW67yT2UpAbzPbCxz4h+71t9t6H0m5OrG8UEtnwS/C27xdgZyB4+WMB+bQGNcSBHugCBUCW4g513H4gUovbcv64deHud86t0lkMbc0LdLwVuOw3+GWufLI7TAYH8pE36+8bKdNi5+08PSmJ9CmybHTxyTXvR+Cmeazdco4PIAonne11hkEj3QLgSWt8n3Ga5KhcT6kaz8vKjhu6e+Vzlp3vlDVEaerW90SrjuW3C0v9+B43ldZZ5h0htWVa5URMkC6EQxUJpCwMI9ttXh9iiwqv4e1RoqNjfTzY6FAScbVhAoEKWc8qDAagJmenbx0fnNiDZTI2FXDK2yNfuJcyztJ/DZMSsdtOBfwPSdTJWbKTIYkbCRgwNtJ+Pq2W8Jyd8tDUx1aIxMss75izlduswWZgSmuP4kA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR11MB6048.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(7916004)(376002)(396003)(39860400002)(346002)(366004)(136003)(230922051799003)(186009)(64100799003)(1800799012)(451199024)(2906002)(5660300002)(41300700001)(82960400001)(33716001)(83380400001)(26005)(38100700002)(86362001)(6506007)(9686003)(6512007)(66946007)(6916009)(66556008)(66476007)(6666004)(478600001)(6486002)(966005)(8676002)(8936002)(316002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UGxwbXJlaXVYVHRwcnZ3dnQ4RWlLREFVRjA1SEdXSWNhQ0hjaTVYbmZJZzdK?= =?utf-8?B?Q1NYeURUZTlMaEF0Zis0bjE0NkcyYVpCcWVhalM0R2VGOXdJRUR1dnM2dWo2?= =?utf-8?B?T2JZb3VWQUN6dExkOXZra1BhT0pEUWVkQXFKY2FEM3ZjMFhlVlhKVlZkU285?= =?utf-8?B?TktqTC83T0tNb1pScEg1UUlXM3BVamJDeXRSTkVNUEJKMjF1dWtYemhWeEh4?= =?utf-8?B?SWJMMkJEUndBbmpIZjVObVVUL3pnTlhLUzNESGRpc0FkeEQyU0xacWVraUhX?= =?utf-8?B?UWgrY1ovdXhJeDFTd3hlejJGcmVkR3hLelJsL1lSUk84c1hMU2d3TWdZbUNG?= =?utf-8?B?a2xmZkY1SVVOSG9mZkdVNS9wb0NJSlJFRXJ2d282OFRGS2l3Q3p3dUlCRUtu?= =?utf-8?B?NEFmVENObEFrdktNWHQ2Vmh0VmV4VmxiTTJNMHJKelg4dEI0YlExZEg3cGdZ?= =?utf-8?B?ekJNU3E4ZWtBNko1RTFZcG1PYTMvNDlMVEE3M3BUcmRDOGJIKzF3L0xxVUtr?= =?utf-8?B?M1Yxb1RhcCtCamw3VXI0L2JaMUNPL2pNUE5OWHRoMDh6eWUwUW5MWGUrYXlL?= =?utf-8?B?SWJBbXhSbUdLa2ZLL0pkS1pITVJIckNnMlg1aEZqYTd0ZnRZTnZQMHZ1ajZp?= =?utf-8?B?OWRMSWxMd29iT3lvWS81b3N6WTdhQU5OSHc1T2hyTjBMVXFJRHE1SEdIYlpR?= =?utf-8?B?SW1qWFNCTmd1OEdCL0tGaDJZZE1YRkh0V2FNWUN5NVFRNVZ3TE9xT2RYZ0R0?= =?utf-8?B?UnJ4MWpsWUM4VE1Lb3ROQTRRejJaK081ZFJvanh0WENMZFlFa3lJN2V4N0dh?= =?utf-8?B?Z25qbHk1bkNjVjVtdWtyRDE2SG90RDF3a21DNmV4ZlNjWWRIYm45aXFkWG45?= =?utf-8?B?WlRTWkRmMVRxRlEwM1A2VVE0SlU5OThyZWl4QW1MSlpFc0NpSUxXSEoyV2Fx?= =?utf-8?B?UXhqSUZDOVBtb0prYTlZRm5UNTZGL01pbFZQbjVMK2Fram5rdHJJb2sxK254?= =?utf-8?B?citjdms0NmFtVEd5d0pEb2h4NFZXdVlOTjRnbWNOUVNTdElFNFJhLyszaTBx?= =?utf-8?B?dlNyTjlMa2V6TzhqMXlzaEd6N0V3K2EvMGgvM1JJQmdHYVJaOTlTSi91cVRK?= =?utf-8?B?QTg3M1dDck1PM1ZEL08relNiNnd4WEwvTFZWWHpsWlRaWGE0VEcyeEZMQ3kx?= =?utf-8?B?UitSVHZITEJENVc1Y0lid2JJSnFhSVlGdjFqbjBvRmc0ekxlM2c4cGtseEMx?= =?utf-8?B?a3d1am5kZS9WT1RoaFJudXN2ZTI5YmtPQnJQVTlnQWdtcVd2WUhhOU92YSty?= =?utf-8?B?OFQxamFFL2kxVklFZ2czRndXM3ZLNUF6K0lrRU9HSjdVOXU1dExDL1I3cFhx?= =?utf-8?B?U2w5S3BtcEZTQVlJaExaMEtZR2pPTlRYMCswR1hBaDdYTXBPRTNXaDZRT0Z6?= =?utf-8?B?TVVnYkFzVThhbHpKcElRL25HQUdJakxyQVlTanJRQW5aVkJTeGx3SVM0RGVC?= =?utf-8?B?K1plM1dhNmZOR1Bod3hpZWlUS0N0Q3N0bThaTjh2R1VVODBWaGNOWDhHWE5H?= =?utf-8?B?MG9JTUhEVGhZT0phc1JpWUYyVm05K2hERGt6WjlzclBLRG53MzlHbG1lSGlj?= =?utf-8?B?ZThZWkl5UjJGWUZ5Qmt2T21ROTBZTzRmQzFZY29Db3JBTWt6TFB1cG5JQXU1?= =?utf-8?B?MUozWFdzYmErS1JUbWtTQ2l2LzZyVmZmdFhXdFZzdk1Ha3FORGVHSVZ2UUQ1?= =?utf-8?B?Ym5BcDhkNUdBRkszMFhJUlNteGpnR3kwY3FRSFUyd2hYZnV1U0VnM2pjeDJ0?= =?utf-8?B?Mk52bEVHRkU3K2xYV29mdk02L3pTQmUwMnQ3SlloQXpjNnZiRllGRHhLajUy?= =?utf-8?B?bEloOXljZnNnRmRYMDRIc3VtLzYxOWtYdEl1SFA3dDJSRVVPRFBsSEhaeGMz?= =?utf-8?B?aTJhUTI0aGgzd1BqeFZDTEtZT1B4eGhPVU81cklHNjk4bko4MUFxOVN6VnRn?= =?utf-8?B?TTU1c1YyRGh6aTRDdWg1dG1vYktSc0phUms5c1J1VXh6a3JUbG01Q25hVnc2?= =?utf-8?B?aWVwSldlcUd2dE5QN0pZVEZJckNLZ202TjlraitseFd5dnc2NklLeHZzOTVv?= =?utf-8?Q?xQsogeHDz0UK7wv42J4+EQlf+?= X-MS-Exchange-CrossTenant-Network-Message-Id: cee96db2-2552-4e39-4ce5-08dc283c71df X-MS-Exchange-CrossTenant-AuthSource: LV2PR11MB6048.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2024 00:25:25.5495 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ojpu81PT0tCRO3gGt9oEUYzYz+U1TY4kSRKBysl9wwsp9df0NFM+jhdJ5RIzvg0405F9SmiifUpseIvwwS7lug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR11MB8330 X-OriginatorOrg: intel.com X-Spamd-Bar: ------- X-Spamd-Result: default: False [-7.50 / 15.00]; WHITELIST_SPF_DKIM(-3.00)[intel.com:d:+,intel.com:s:+]; DWL_DNSWL_MED(-2.00)[intel.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; ARC_REJECT(1.00)[signature check failed: fail, {[1] = sig:microsoft.com:reject}]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[intel.com,none]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[intel.com:s=Intel]; R_SPF_ALLOW(-0.20)[+ip4:192.198.163.0/26]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:4983, ipnet:192.198.162.0/23, country:US]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_SEVEN(0.00)[7]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[104.47.73.168:received]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[intel.com:+] X-Rspamd-Queue-Id: 4TVd7354tQz4WJd On Thu, 01 Feb 2024 16:07:11 UTC Antranig Vartanian wrote: >Greetings y’all, > >We have a massive server with two IXL NICs (Ethernet Controller X710 >for 10GBASE-T). One of them is on the host and the other is passed to a >guest running in a bhyve VM. > >Every once in a while, the system hangs. We cannot SSH, we cannot ping >nor we can login using console (IPMI remote screen). > >However, on the console, I see the following messages: > >"ixl0: ixl_process_adminq: Unable to allocate memory for Admin Queue >event!” > >And it keeps repeating. I would need to look into seeing if we've had a similar issue posted previously. Can you post a detailed report of the issue with version, log output on the forum here: https://community.intel.com/t5/Ethernet-Products/bd-p/ethernet-products > >The only way to get access to the host is to… reboot it. > >We don’t use SR-IOV (although when we did, the same thing happened), we >don’t use large MTUs (1500) nor we have a memory issue (the system has >2TB of RAM, and right before hanging around 250G is available). > >Any tip would be appreciated. > >P.S. we also tried the kmod from ports, still seeing the same issue. > >Kind regards, Thanks, -- Loren M. Lang Intel