From owner-svn-src-stable@freebsd.org Fri Nov 13 16:38:31 2020 Return-Path: Delivered-To: svn-src-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6B6EC2E7297; Fri, 13 Nov 2020 16:38:31 +0000 (UTC) (envelope-from martin@vx.sk) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:191:9029::4]) by mx1.freebsd.org (Postfix) with ESMTP id 4CXkg94CjXz4bCT; Fri, 13 Nov 2020 16:38:29 +0000 (UTC) (envelope-from martin@vx.sk) Received: from mail.vx.sk (localhost [127.0.0.1]) by mail.vx.sk (Postfix) with ESMTP id EE7621D9BDF; Fri, 13 Nov 2020 17:38:21 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=vx.sk; h= content-transfer-encoding:mime-version:content-type:content-type :user-agent:in-reply-to:subject:subject:from:from:message-id :date:date:received:received:received; s=1; t=1605285501; x= 1607099902; bh=Du8iAPMSEvWijtnChlun4boFtUp6gASBvA0or8WvVJM=; b=l jfEHBpSzI8ZxztM7kogZMb4bLcoFIpWPBtofp8erj5Aroox92OAx4WIUt/ueEii9 +NGRjRb06f9PSr45tX23HPfFPgk6LOTqGo68ebuDJvgGWiDA0EZtUeCNBoDXIG2l ufAqNuiCYljgWT7QanInatbvxSZKzfmrZtO1TZb/mWM/Ik8qoSffmtPBq4pLBp0B 1f3A78JVTJ1ypMm0XfoBa6MrWkD6z0XhLYwqAY81kcPQFSgWVwbrsMwJHkPSs/6N R5LoV8yrbm8Pt1POhgvHquRki0scz6qPaUo9KVogeKD1l6Z7ymxWkydf3JFQJ9qE lRHnWmWrLRYjjT4mcwbbA== X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by mail.vx.sk (amavisd-new, unix socket) with LMTP id jR8_fe4ZpbcX; Fri, 13 Nov 2020 17:38:21 +0100 (CET) Received: from mail.vx.sk (localhost [IPv6:::1]) by mail.vx.sk (Postfix) with ESMTPSA id A01CD1D9C49; Fri, 13 Nov 2020 17:38:21 +0100 (CET) Received: from 188-167-136-198.dynamic.chello.sk (188-167-136-198.dynamic.chello.sk [188.167.136.198]) by mail.vx.sk (Horde Framework) with HTTPS; Fri, 13 Nov 2020 17:38:21 +0100 Date: Fri, 13 Nov 2020 17:38:21 +0100 Message-ID: <20201113173821.Horde.jq1A9wp72tHUvM10uhYxhOq@mail.vx.sk> From: martin@vx.sk To: Alexander Motin Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: Re: svn commit: r367359 - stable/12/sys/dev/pci In-Reply-To: <202011050257.0A52veoR086637@repo.freebsd.org> User-Agent: Horde Application Framework 5 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4CXkg94CjXz4bCT X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=vx.sk header.s=1 header.b=l jfEHBp; dmarc=none; spf=pass (mx1.freebsd.org: domain of martin@vx.sk designates 2a01:4f8:191:9029::4 as permitted sender) smtp.mailfrom=martin@vx.sk X-Spamd-Result: default: False [-3.40 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; FREEFALL_USER(0.00)[martin]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[vx.sk]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_FIVE(0.00)[5]; SPAMHAUS_ZRD(0.00)[2a01:4f8:191:9029::4:from:127.0.2.255]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[vx.sk:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_NO_DN(0.00)[]; R_DKIM_ALLOW(-0.20)[vx.sk:s=1]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:4f8:191:9029::4:from]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MAILMAN_DEST(0.00)[svn-src-all,svn-src-stable-12,svn-src-stable]; RECEIVED_SPAMHAUS_PBL(0.00)[188.167.136.198:received] Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Content-Description: =?utf-8?b?U3Byw6F2YQ==?= s =?utf-8?b?xI1pc3TDvW0=?= textom X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Nov 2020 16:38:31 -0000 I am encountering a problem with hw.pci.enable_aspm: System: ASUS RS500A-E10-RS12U CPU: AMD EPYC 7502 32-Core. NVMe drive model: Micron 9300 PRO Number of NVMe drives: 10 Using X2APIC When hw.pci.enable_aspm is enabled, PCIe hot plug of Micron NVMe =20 drives does not work (including detection on boot) and the system =20 reports the following errors for the vast majority of the drives: Nov 13 15:11:30 xxx kernel: pcib9: Timed out waiting for Data Link =20 Layer Active Nov 13 15:12:17 xxx kernel: pcib30: Timed out waiting for Data Link =20 Layer Active Nov 13 15:12:44 xxx kernel: pcib29: Timed out waiting for Data Link =20 Layer Active Nov 13 15:13:04 xxx kernel: pcib11: Timed out waiting for Data Link =20 Layer Active ... Flipping the tunable back to 0 workarounds the problem. Reproduced both on latest head and stable/12. Best regards, mm Quoting Alexander Motin : > Author: mav > Date: Thu Nov=C2=A0 5 02:57:40 2020 > New Revision: 367359 > URL: https://svnweb.freebsd.org/changeset/base/367359 > > Log: > MFC r364038: Enable hw.pci.enable_aspm tunable by default. > > While effects on power saving is only a guess, effects on hot-plug are > clearly visible.=C2=A0 Lets try to enable it and see what happen. > > Modified: > stable/12/sys/dev/pci/pci.c > Directory Properties: > stable/12/=C2=A0 =C2=A0(props changed) > > Modified: stable/12/sys/dev/pci/pci.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > --- stable/12/sys/dev/pci/pci.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 Thu Nov=C2=A0= 5 02:12:33 2020=C2=A0 =C2=A0 =C2=A0 =20 > =C2=A0 (r367358) > +++ stable/12/sys/dev/pci/pci.c=C2=A0 =C2=A0 =C2=A0 =C2=A0 Thu Nov=C2=A0= 5 02:57:40 2020=C2=A0 =C2=A0 =C2=A0 =20 > =C2=A0 (r367359) > @@ -411,7 +411,7 @@ static int pci_enable_ari =3D 1; > SYSCTL_INT(_hw_pci, OID_AUTO, enable_ari, CTLFLAG_RDTUN, &pci_enable_ar= i, > =C2=A0 =C2=A0 0, "Enable support for PCIe Alternative RID Interpretatio= n"); > > -int pci_enable_aspm; > +int pci_enable_aspm =3D 1; > SYSCTL_INT(_hw_pci, OID_AUTO, enable_aspm, CTLFLAG_RDTUN, =20 > &pci_enable_aspm,=C2=A0 =C2=A0 0, "Enable support for PCIe Active State= Power =20 > Management");