From nobody Tue May 13 20:20:56 2025 X-Original-To: freebsd-cloud@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZxntS0CD7z5vvcX for ; Tue, 13 May 2025 20:21:20 +0000 (UTC) (envelope-from dan@langille.org) Received: from fout-b4-smtp.messagingengine.com (fout-b4-smtp.messagingengine.com [202.12.124.147]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZxntR2Klmz3jpl for ; Tue, 13 May 2025 20:21:19 +0000 (UTC) (envelope-from dan@langille.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=langille.org header.s=fm3 header.b=V3vQPxqd; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=Y42op1bB; spf=pass (mx1.freebsd.org: domain of dan@langille.org designates 202.12.124.147 as permitted sender) smtp.mailfrom=dan@langille.org; dmarc=pass (policy=none) header.from=langille.org Received: from phl-compute-01.internal (phl-compute-01.phl.internal [10.202.2.41]) by mailfout.stl.internal (Postfix) with ESMTP id D8BC411400B0 for ; Tue, 13 May 2025 16:21:17 -0400 (EDT) Received: from phl-imap-18 ([10.202.2.89]) by phl-compute-01.internal (MEProxy); Tue, 13 May 2025 16:21:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=langille.org; h= cc:content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1747167677; x=1747254077; bh=tKpyAfezFBaNoB7gf0+YYGrK2ZTQVZsWqD0aboU+78c=; b= V3vQPxqduYzgdzks0ZNHCjvAbbKss2H/einb+XdaFRfE4gkDWD2cHEGB4njJrc3V 91KiKp9HR/fC+V2bF8OWtcqkGaQddgS513+jcRi/Ek9vUyqk+UcXxJYATHzqjmGC ro+hvlQ9hexytfsJ8gc4hVQXaRz4tZ6MEjnd0017ekRuI6fwZGPsq74692wmT3s+ +krSU07J6XaUE+RYiBmp+KObaQ2zlpl6R9WsySppeFO/QSNgh5UYTB3D7FLzc1GW NJi4MjsItkMHyoAy65A2xGIKEujnXkeSty29a4MOwMwqU66gDFZmJ5AB6iRQeD1a d3osEVTrAbh1Ny+9a47iVA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; t=1747167677; x=1747254077; bh=t KpyAfezFBaNoB7gf0+YYGrK2ZTQVZsWqD0aboU+78c=; b=Y42op1bBsuKjph4Ja S1ZKpGMJSiGL/Gd5VFTx8rcrwxuMoRnf1rvz3e53IYYAaLPde2nZiUpJpX0y91TT Rj8yGlvKDeMLHXLU+0AJgeS7ALfoiSCiIu3FUU8ZBG3IEUwEAYSWA6d6jVH62q5m GXonMSSUlWGC/xtlE3Nu9XxatmSPQLA3PLC/Y6fFNL0hKtuh7KzM2e+8EJVvU2hH EMRy9w3q5Ox00lGa/fewDMYUmzxrfUf3Aw132NMwqgdlqjQyPQ9O/nGzsjwhKRtL C8qbPfsqNppbYLTT5u/ajN59fHJt1yDgcZxcwxetR/V2Nqnb6oVbtKccoa8ptQDj /5smQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgdeftdehtdehucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggv pdfurfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpih gvnhhtshculddquddttddmnecujfgurhepofggfffhvffkjghfufgtgfesthejredtredt tdenucfhrhhomhepfdffrghnucfnrghnghhilhhlvgdfuceouggrnheslhgrnhhgihhllh gvrdhorhhgqeenucggtffrrghtthgvrhhnpedtveegjeeihfffteduteekvdfgveehffdt gefgvddtgeehfeejheejtdfgueeugeenucevlhhushhtvghrufhiiigvpedtnecurfgrrh grmhepmhgrihhlfhhrohhmpegurghnsehlrghnghhilhhlvgdrohhrghdpnhgspghrtghp thhtohepuddpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepfhhrvggvsghsugdqtg hlohhuugesfhhrvggvsghsugdrohhrgh X-ME-Proxy: Feedback-ID: ifbf9424e:Fastmail Received: by mailuser.phl.internal (Postfix, from userid 501) id 7074A15C0069; Tue, 13 May 2025 16:21:17 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface List-Id: FreeBSD on cloud platforms (EC2, GCE, Azure, etc.) List-Archive: https://lists.freebsd.org/archives/freebsd-cloud List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-cloud@FreeBSD.org MIME-Version: 1.0 X-ThreadId: Tf2ec06965819b1c3 Date: Tue, 13 May 2025 16:20:56 -0400 From: "Dan Langille" To: "Application Certification Support via freebsd-cloud" Message-Id: <707702db-2eb9-475b-9170-bed740efd0c2@app.fastmail.com> In-Reply-To: <2fe4e22b-acde-4a43-9359-bd6a4e028a37@nomadlogic.org> References: <2fe4e22b-acde-4a43-9359-bd6a4e028a37@nomadlogic.org> Subject: Re: ena(4) tx timeout messages in dmesg Content-Type: text/plain Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4ZxntR2Klmz3jpl X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.36 / 15.00]; NEURAL_HAM_SHORT(-0.81)[-0.811]; NEURAL_HAM_MEDIUM(-0.78)[-0.778]; NEURAL_HAM_LONG(-0.69)[-0.685]; DMARC_POLICY_ALLOW(-0.50)[langille.org,none]; R_DKIM_ALLOW(-0.20)[langille.org:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:202.12.124.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[202.12.124.147:from]; XM_UA_NO_VERSION(0.01)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[dan]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-cloud@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; PREVIOUSLY_DELIVERED(0.00)[freebsd-cloud@freebsd.org]; DKIM_TRACE(0.00)[langille.org:+,messagingengine.com:+] On Tue, May 13, 2025, at 1:22 PM, Pete Wright wrote: > On 5/12/25 11:04, Pete Wright wrote: >> hey there - i have an ec2 instance that i'm using as a nfs server and >> have noticed the following messages in my dmesg buffer: >> >> ena0: Found a Tx that wasn't completed on time, qid 2, index 593. 10 >> msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. >> ena0: Found a Tx that wasn't completed on time, qid 2, index 220. 1 >> msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. >> ena0: Found a Tx that wasn't completed on time, qid 3, index 240. 1 >> msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. >> ena0: Found a Tx that wasn't completed on time, qid 3, index 974. 1 >> msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. >> ena0: Found a Tx that wasn't completed on time, qid 2, index 730. 1 >> msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. >> ena0: Found a Tx that wasn't completed on time, qid 2, index 864. 10 >> msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. >> ena0: Found a Tx that wasn't completed on time, qid 3, index 998. 1 >> msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. >> >> > > > So I've found an interesting pattern, the above messages get printed to > /var/log/messages and the dmesg buffer when i "su" to root apparently: > > May 9 19:19:23 airflow-nfs su[66523]: ec2-user to root on /dev/pts/3 > May 9 19:19:23 airflow-nfs kernel: Found a Tx that wasn't completed on > time, qid 2, index 593. 10 msecs have passed since last cleanup. Missing > Tx timeout value 5000 msecs. > May 9 19:19:23 airflow-nfs kernel: Found a Tx that wasn't completed on > time, qid 2, index 220. 1 msecs have passed since last cleanup. Missing > Tx timeout value 5000 msecs. > M > > > May 12 17:55:25 airflow-nfs su[29272]: ec2-user to root on /dev/pts/0 > May 12 17:55:25 airflow-nfs kernel: Found a Tx that wasn't completed on > time, qid 3, index 998. 1 msecs have passed since last cleanup. Missing > Tx timeout value 5000 msecs. > May 12 17:55:25 airflow-nfs kernel: Found a Tx that wasn't completed on > time, qid 1, index 975. 1 msecs have passed since last cleanup. Missing > Tx timeout value 5000 msecs. > May 12 17:55:25 airflow-nfs kernel: Found a Tx that wasn't completed on > time, qid 1, index 428. 1 msecs have passed since last cleanup. Missing > Tx timeout value 5000 msecs. > > > May 13 17:17:14 airflow-nfs su[16099]: ec2-user to root on /dev/pts/0 > May 13 17:17:14 airflow-nfs kernel: Found a Tx that wasn't completed on > time, qid 1, index 289. 1 msecs have passed since last cleanup. Missing > Tx timeout value 5000 msecs. > May 13 17:17:14 airflow-nfs kernel: Found a Tx that wasn't completed on > time, qid 1, index 159. 1 msecs have passed since last cleanup. Missing > Tx timeout value 5000 msecs. > > > I have no idea what that means, but certainly feels like an interesting > data-point. i'm ssh'ing as the ec2-user, then "su -" to become root and > as you can see from the timestamps something triggers those log events. > i'm not seeing any other occurances of these log messages outside of > su'ing too. this is a very vanilla system, not krb auth or other > network interactions should happen when i become root. I get them too: May 9 21:20:50 aws-1 kernel: ena0: Found a Tx that wasn't completed on time, qid 0, index 105. 1 msecs have passed since last cleanup. Missing Tx timeout value 5000 msecs. -- Dan Langille dan@langille.org