From nobody Mon Dec 4 08:16:28 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SkGj91Ytmz535DC for ; Mon, 4 Dec 2023 08:16:45 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-23.consmr.mail.gq1.yahoo.com (sonic304-23.consmr.mail.gq1.yahoo.com [98.137.68.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SkGj83MTJz3X9p for ; Mon, 4 Dec 2023 08:16:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=RdwiSaOq; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1701677802; bh=VPwSwAiAA3ddbwVobDneCnfd2F3xy0gOURgurqkmYpQ=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=RdwiSaOqyk07AKlUiNvt5uWdgPvxml3hV5T5EWYcAAQbeWP48K2umADk9S0NDrtMZyOYPbfmiNIPDLpa/Q5fIPjwEsOAJtx9hCJ2a1/Aupb6dVbCUgMKh4hZOxcRR3WqPqi3g//6GmHD7W706KZPX1g3RSuY4yyBZ2yLlJIpy/2zuznWzW+OSs6Tfeai45bs47AHeEahIBZIEOXKxJE1+qO4rRGOkwuUFYK1/W5BSWGAF9qb2ha0XPu8ExrWo/qkHAGoh8l4gwdh6Y5wFsJJKT9etV9WGjnws6tVzu2WOG3W2WrU2T188b1hBLz0lj9okxWvRnFkQ+DOwlhkUtZlxw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1701677802; bh=1Hkborj7jS5xo0TBzLphfCxIrXHyv+oigwJEFL3aUIP=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=oNdAFgYuXRLoTQ/T2GgHj4wHEIzUcfVzTsaK+2Qga5pkPl7apRKPr94VHkGW6yUEDb7TT4CZHDx4bRqf/4RKIoicjvknhqFz7hzo73lTvd3eRsdIks6IOWKmjWHeZSH2iht6jxGMQcjVX0qwQnBPkKZQEBGyuOAwzQCNeN3UPPIhX2WgiYG5tC6pVxSGE80fBp9xS+WRjsiJhpwFQ8y9ftiEpQrO1f5K+x+IpbboC6NAqJZ/nF286LpltkT1KQM2WgtoTp1Mge1jOHtmb9an/VlDoukd8YEpojflLVB98ysikef/e6WKwQDOCiWUO+mhX59WBQeBOMqPYzcp9MQsyw== X-YMail-OSG: i.9nB7cVM1lNb6zBxZK8QvH5d5iCPHUr4TouN.YqexVf0D6cY9ExTMBKjrYj0y. dZ0O.QPF20iX1PPNA.OHcUzQ8XekD_wFqvGVf1VJCHlLYEXYbw9gau.OiBtKaRNUQZUnqUGLwZe. ccdfkIfbtvnETALm2IQthuz1eNZMQfuSNMICRdKsmPy8w6EDfLRaxV7KcSIt9XAVIOa3jX8qnjxL xzOOPdCFvaIf3MLQ8jrXkJI.Srh42OJngaYW.hhyrV_ueYwvg.wQctxYjsoDxFwAE1BtGIZ2n480 3DqeZ0zeShmgTwO.zUktbzwxBkxHjOj6RyguHAqmvqSNRyyqxa65.xv751aqTq20u1lUJ2hm6r7o tJUw4sygHYDrRI3HzhMl9e8GXmuu1zdp4vLRnNggU9s.1U6CzVfqxKW10_1UM_73C6ePjpoqSO1Y 18OQ.QpKXsRIhwAHYApXmk4PMdFI7HhRHBYLDdY0Wli3L58wBGWjnRSauFCTXeNIQDOfROfjFRph qhQmFNsOJfdIPNZmohXxyJU4ACZ9pyGH0ZumDBWxmPBWI3he0V71NT.0ivD9Tnm_FkeLgjCrOd6v N8UskbITP9HlG78WEKtw46qjGmciqTcJNf9SdtphG9ws79ZFV4HAwXjaVsH8yMC_lQdrNFILj7yD pQ.jUHIwRo5ZByygNlI1XC1ceHkkSoq4tAnuVW_rbbVMLMex2lASGXI5neuLurhawcUXndLtdoUS i_D53c3UYPpBXJAnt_3rAzE5M1mfowcyWYzHM.SlD2VaFIbWe3FOQV5.vjaGD.q_GT0n0ELdxPNe FchIoKwXttIA2oItY9fpXpe4eJJsCnZrIe6OD9JMvy_3clAgUgbTS4uzj8ivHbJoer9RIGcuXKAl CCg5rTCPY30GNnYaRWgXvkGKTss96PD8r6jlR5AXCgkAXFqV5A077It.8DJnjvcimfKZEOyXe504 PnC6TwIkMZn2fN_woduEeVVo09PljQpV_nNRk8uVvIeJl7Ons2P6u7pQ1xC21fmPEw6_BebX964O Sv220RTDDZ.wmq78Q5OoJIsQ4QyfpMFZqAxPZtBGxRJA.SC3Roc_ia.rEst4rOWZKsLOA2qnczul Nil3wsKGi.5vRMZrtHzn_g9l_HGZrOkRBtrhSQ.rbYfbzJ55N7_46gXZ.EFz_C_uZqsBzWpfi5oy MM8rMkZ7YGCcSXFWB.BHMOyuq3Rj7jDLtiAcl_TckRQh51OuVoFVA6ykx1KG43CPR.aRO2EOhgKw tuilaaxrAFPwaEXPr6IO.QbfLDsIh9WtYNXiLji_9UgMZ112und3Bj.stH6jZvPLdJcIcmdB0jrN 9T79b1f7p34iL6g3OBmOHKWOaSzNV9bvRHqUZ._7pWRV8nJbZG3AN3Qmee6J_rgRBe_iMtOJwu1V ayf4h0jN9T1yWkV__WnheW243udPqRgD3TzT3n5.LbjwFKg3r6c9DYOlwZNMEKB8Lm5U3yJZDCy2 L2yctT38vED9MDjX_itYi9O0zyPNUAloa31mwKKTBQhMOhbjA0idLi.1iw_IRdv3i5vnVaylaY_P sQRqIltqUnU6_xtZeAL.MJ8j_4jbDyna_KPAngrVJZUFY2ZsJ4SOH8HrbWqX0VFdAwlp8C4pHW69 A_kapHHpO2PlnDsAE2EN.wM0XfqKcTr3uM38Rcr27oGWkcfFLgak45ZJ.70t9c_TVaAkodo8GX3r MwJcowCNDRkVlcBQevGmASeegSIVoYT7wcU6vURvgSP76P.dSOg4R4pBfEMW97fdr4nVWppk09Yf jDEvOaPmGhJPbOSmCRDGDm3.EncFtm0njWQ73fCZxO0o_qtYwm4T4z4NTtNdgcZ50p9..I.24WYc khq6VZIQoD5_m94MiM3TFEhYV_giLl2G5iIDKg4d4FnqelRE__smaznK2Lck5l8.vNgvV3j4wimH Lt8BYsKt1EtB6jrHCKQrGIfBR0Ihk__FvLo5yxXyDTisDCwAtB2hswAuxFBNojEFSKjGZCp2ofRN ukCY61bYAagZO_J52mQUTB2O5wzOMlJ0HgfCcNCBXXGighzXVsbem4XVYFusUCOAct7HAFe68EBN GdAQdKz.bb4uqQ94zyQh1Z75p0uVNRdY.C1DAbEE7s9BTMkj4r_.hG_Xuz5xfv0ayNe347jzuY0V 3e0t3Dj2feBJbnggyoavApej7P9uaqz8PqXQwnMbkFPcL1px7wS8Rc7QaMAep5FVTTjH0eF7B.X9 UF4oHfPhuh2YuepPROlaXsBa1mkOd3f57zveHsVPftyDX5qNEhuavxJOGcans X-Sonic-MF: X-Sonic-ID: a60b613e-b067-4f66-9df3-115555c04292 Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Mon, 4 Dec 2023 08:16:42 +0000 Received: by hermes--production-gq1-5cf8f76c44-2v26z (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID bce5fb8ec71e91fa7fce47887703683a; Mon, 04 Dec 2023 08:16:39 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: main [so: 15] context, 7950X3D and RTL8251/8153 based Ethernet dongle: loss of state, example log information Message-Id: <41913B2D-381A-4EEC-9B37-531445645F71@yahoo.com> Date: Mon, 4 Dec 2023 00:16:28 -0800 To: Current FreeBSD X-Mailer: Apple Mail (2.3774.200.91.1.1) References: <41913B2D-381A-4EEC-9B37-531445645F71.ref@yahoo.com> X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.80)[-0.796]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.204:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.204:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SkGj83MTJz3X9p X-Spamd-Bar: --- The following sort of thing is happening a lot: Ryzen 9 7950X3D using a USB Ethernet dongle that I've historically used on occasion, sometimes for long periods . . . Example contexts for the issue have been: FreeBSD 15.0-CURRENT #126 main-n266130-d521abdff236-dirty: Tue Oct 24 = 18:17:40 PDT 2023 and: FreeBSD 15.0-CURRENT #131 main-n266749-ed31b3f4a146-dirty: Wed Nov 29 = 16:53:33 PST 2023 Both UFS and ZFS boot media, here part of a ed31b3f4a146 UFS example is = shown Nov 29 18:26:27 7950X3D-UFS kernel: miibus0: on ure0 Nov 29 18:26:27 7950X3D-UFS kernel: rgephy0: PHY 0 on miibus0 Nov 29 18:26:27 7950X3D-UFS kernel: rgephy0: none, 10baseT, = 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, = 1000baseT-FDX-master, auto Nov 29 18:26:27 7950X3D-UFS kernel: ue0: on ure0 Nov 29 18:26:27 7950X3D-UFS kernel: ue0: Ethernet address: REDACTED Nov 29 18:26:27 7950X3D-UFS kernel: ue0: link state changed to DOWN . . . Nov 29 18:26:27 7950X3D-UFS kernel: ue0: link state changed to UP . . . (no ue0: messages, then) . . . Nov 30 03:23:18 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:18 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:18 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:18 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:18 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:18 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:19 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:20 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:20 7950X3D-UFS kernel: ue0: 2 link states coalesced Nov 30 03:23:20 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:20 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:20 7950X3D-UFS dhclient[53725]: New IP Address (ue0): = 192.168.1.157 Nov 30 03:23:20 7950X3D-UFS dhclient[53730]: New Subnet Mask (ue0): = 255.255.255.0 Nov 30 03:23:20 7950X3D-UFS dhclient[53734]: New Broadcast Address = (ue0): 192.168.1.255 Nov 30 03:23:20 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:20 7950X3D-UFS kernel: ue0: 3 link states coalesced Nov 30 03:23:20 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:20 7950X3D-UFS dhclient[53771]: New Routers (ue0): = 192.168.1.1 Nov 30 03:23:21 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:21 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:21 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:21 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:21 7950X3D-UFS kernel: ue0: link state changed to DOWN Nov 30 03:23:21 7950X3D-UFS kernel: ue0: 3 link states coalesced Nov 30 03:23:21 7950X3D-UFS kernel: ue0: link state changed to UP Nov 30 03:23:21 7950X3D-UFS kernel: ue0: 2 link states coalesced Nov 30 03:23:21 7950X3D-UFS kernel: ue0: link state changed to UP . . . (lots more) . . . Other FreeBSD system on the Ethernet are not getting such but none of them are currently using such a dongle. Note: The ASUS X670-P WiFi board's built in Ethernet is not supported. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Tue Dec 5 08:10:01 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SktWG2Yxgz53RQ6 for ; Tue, 5 Dec 2023 08:10:18 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-oi1-x22e.google.com (mail-oi1-x22e.google.com [IPv6:2607:f8b0:4864:20::22e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SktWF02Gqz4FHN for ; Tue, 5 Dec 2023 08:10:16 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=iitbombay-org.20230601.gappssmtp.com header.s=20230601 header.b="Z/Galm35"; spf=pass (mx1.freebsd.org: domain of bakul@iitbombay.org designates 2607:f8b0:4864:20::22e as permitted sender) smtp.mailfrom=bakul@iitbombay.org; dmarc=none Received: by mail-oi1-x22e.google.com with SMTP id 5614622812f47-3b9b8522d91so990353b6e.0 for ; Tue, 05 Dec 2023 00:10:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20230601.gappssmtp.com; s=20230601; t=1701763813; x=1702368613; darn=freebsd.org; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=5f1zWxhS5wkxpRoY/EX13ltJCFpg4iUUy6eFY8VgXAs=; b=Z/Galm35UgayUlRVeNkLKEgSLAo2DCHftbbXUDgPF1nBtRtFR14Da1JIrDH5bPQiQ7 0X21AjMsLtEbuk3XCumxZglDW2/S93YmaE+gYp1zVy9WvDxTB7ftyFF580+zWwmvodzr lOqkuSmvSOn6cw9I6v2oTRlSxRk2x0yZs1JaB7Fu5to8WL8IShknwV/Eeu0WqLJ7+sWO sRrGupKmlw+jt6QjjVVKwfH7hhAlo/OTy3ET7hM39ssRPD1c0+CQxcGCkzspxxbrn1Hi zq5i7gS/OusrA3KxzXvZz/YF6ErpHEL7pzJFIjn1hEQJHoW8FdcnfK9YCLQX17vGyQFH YlcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701763813; x=1702368613; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5f1zWxhS5wkxpRoY/EX13ltJCFpg4iUUy6eFY8VgXAs=; b=cuhSidEJr0rnTKPZ4iSn42WpBSCR6iFTzSvZGJ3ZwrbUkS371qmzmp5hHLvO48AVce CMI7ziPcLM5zK3fXbHys+D9Ylx/EBcs0qjuc0z/jbHlOvAxUG+j5bFc2i122qxJM9TOz aia5O2sMlR72rQydKByAYh+2hd4skaks3LcKfzzQv8ZWXbjNay4kOfnMjWIy0wQKZ/Mf LrL2D/tYbWtBpJDx+mV7DDWXwhD9v2Cqtxr8cWb5AOpQVKUjwKUhrxJSm83lxDWOp+8z bA5l6bCCOpz3/PlHaz8OBebMK2OSKM3AIoNg39yJj5qYluanLY2u8YEWd32TCT0EHOgA HvmA== X-Gm-Message-State: AOJu0YybuoQuPV1O6EtkVUyD8sQbLUdR66IIz7MEHJ9jKMKnlyCJCpxc +QcRvnUkB5tGkoiQCrmDe8LZRjTT6bgivlnEvjI= X-Google-Smtp-Source: AGHT+IHj4sy4RQy3CLSZnK+kIbYXTcTKwM1x0jxDrgLAgGhIleByNvYi5tO3lAnMA8/RbkEDKoVNZA== X-Received: by 2002:a05:6808:180b:b0:3a7:d566:8b5e with SMTP id bh11-20020a056808180b00b003a7d5668b5emr6683940oib.44.1701763813601; Tue, 05 Dec 2023 00:10:13 -0800 (PST) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id u30-20020a056a00099e00b006cba5c61a01sm8834529pfg.159.2023.12.05.00.10.12 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Dec 2023 00:10:13 -0800 (PST) From: Bakul Shah Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: installkernel with PORTS_MODULES+=graphics/drm-515-kmod stopped working Message-Id: <8138561E-B09D-4F5F-BC10-2B04E245DA63@iitbombay.org> Date: Tue, 5 Dec 2023 00:10:01 -0800 To: FreeBSD CURRENT X-Mailer: Apple Mail (2.3774.200.91.1.1) X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[iitbombay-org.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::22e:from]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[iitbombay-org.20230601.gappssmtp.com:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[bakul]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DMARC_NA(0.00)[iitbombay.org]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4SktWF02Gqz4FHN X-Spamd-Bar: -- I build{world,kernel} on one machine but install everything from the target machine, with /usr/obj and /usr/src from the build machine nfs mounted. /etc/src.conf (on target and build machines) has PORTS_MODULES+=3Dgraphics/drm-515-kmod to build this port at the same time. This used to work at least until Nov 1, 2023 but now stopped working. What I don't understand is why is install trying to install to /usr/obj/..., which is really on the build tree. It this known? Is this style of building .ko ports still supported? The make infrastructure has gotten far too complicated and baroque over the years so before diving into it I am looking any shortcuts! Thanks! [note: /usr/ports is mounted from the build machine /usr/src is symlinked to /home/FreeBSD/stable-14 /etc/src.conf is the same on both machines. =46rom target (as root) I can create root owned files under /usr/obj on the target] =3D=3D=3D> drm-515-kmod not installed, skipping =3D=3D=3D> Staging for drm-515-kmod-5.15.118_2 No name and/or group mapping for uid,gid:(0,0) No name and/or group mapping for uid,gid:(0,0) ... No name and/or group mapping for uid,gid:(0,0) =3D=3D=3D> Generating temporary packing list =3D=3D=3D> dmabuf (install) install -T release -o root -g wheel -m 555 dmabuf.ko = /usr/obj/home/FreeBSD/stable-14/amd64.amd64/sys/GENERIC/usr/ports/graphics= /drm-515-kmod/work/stage/boot/modules/ =3D=3D=3D> ttm (install) install -T release -o root -g wheel -m 555 ttm.ko = /usr/obj/home/FreeBSD/stable-14/amd64.amd64/sys/GENERIC/usr/ports/graphics= /drm-515-kmod/work/stage/boot/modules/ =3D=3D=3D> drm (install) install -T release -o root -g wheel -m 555 drm.ko = /usr/obj/home/FreeBSD/stable-14/amd64.amd64/sys/GENERIC/usr/ports/graphics= /drm-515-kmod/work/stage/boot/modules/ No name and/or group mapping for uid,gid:(0,0) install: = /usr/obj/home/FreeBSD/stable-14/amd64.amd64/sys/GENERIC/usr/ports/graphics= /drm-515-kmod/work/stage/boot/modules/drm.ko: chown/chgrp: Operation not = permitted *** Error code 71 Stop. make[6]: stopped in = /usr/obj/home/FreeBSD/stable-14/amd64.amd64/sys/GENERIC/usr/ports/graphics= /drm-515-kmod/work/drm-kmod-drm_v5.15.118_3/drm *** Error code 1 Stop. make[5]: stopped in = /usr/obj/home/FreeBSD/stable-14/amd64.amd64/sys/GENERIC/usr/ports/graphics= /drm-515-kmod/work/drm-kmod-drm_v5.15.118_3 *** Error code 1 From nobody Thu Dec 7 04:31:13 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sm1Yv23b4z53YJb for ; Thu, 7 Dec 2023 04:31:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-25.consmr.mail.gq1.yahoo.com (sonic311-25.consmr.mail.gq1.yahoo.com [98.137.65.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sm1Ys5r7Nz3MD2 for ; Thu, 7 Dec 2023 04:31:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=sbJtE6BE; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1701923487; bh=bqy9alElPtOLehJaAGfTGM393FoswTPIBgNyMJDVDqU=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=sbJtE6BE+Cs33M3VN5HUCL+sa+yyCn9Cw8UVrl68BY1bPbL3Zi7ekoNWCQirwCYhoPgPT+HEGMmPi9+TBdPinBgmdyy5HTgt+sVqNuD38MSO0/9FoiLOSYMtmn20dSzb06O2C0RHsd/2V4A0qjhWiuvMK9LTWr+C9zOJyX4mp2LXJV1IkEICq7EZc/BNpBQ9fTVoR8r9QGYm9T/nYosejN1+VwM721YfM27/tlIJY+Yr3uYYgy4t8G7/P0jUoyc6CJWLW3sDJbK/tj4sRvZvEMsm1sN0HFTyaVnOyMUayXE/ZinJJ/AtHZ4HIUJHHizmOOwwLxPJo6lQpLHbBD032Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1701923487; bh=h2v33qlk/8vHaRLno3Y1U5xVKpqDkIYdzLdglMaf6H1=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=NTOOfrgbMHUb2djcP1e5rLucOEoiZ7ik6YkvI/Zx8R3cMhRvr7j+dzg2XdJL51YnU7fJ697UJtJ8VcEkW6FOC565uTU6QaLrxmg/2rHxlFHUDXXm6AqjLoc9PhXvL+3HFh6EcscPmKQ68GYvGUW6urnwncyD7U9RgYqtia3G1EpzhMUDTT74Xc71xnAmCafs1xCwoT+PbWEZ8Vt4jWWr8UWEAB1UW3KYTzrK/VUmsEK6Uwgs4CwGtLXvfMFsEe7/c4rMHoJg7Kh45JApDcnDVUv/4hIoBEDs9uzFkNhmIe0bwoBJuwTqCtUifPiCEeVbhZoEDzxA2QIOIDt7TydwaQ== X-YMail-OSG: Ty1C_xkVM1lOs0q6hy12BGcXju6FVFfwyYTzi2aVquxddWUJ2YrUXRWPxyKT1P8 ZctAE_71P_Y4MxmrZUYRMmd65X73iWYRuCL.xjnVdjKJ9l4mmB9uThmIBdLO5ZJXxlMsyLAEO7bY _r6JbAwj9qgqk1wYBp_SMMWTwO5GhFWab7HnGThrRx5qKA3ucR04jy5wx7XbbbgEWlQvBXqIYaeR 2JfjM7GaWxUujgDu0sm47_DQRwud.jf5UBo7vBH1B.sC_wX7zW8lplykGrGjryTrOn23M5m90nCj d.dxpGESngB3fEqOqyGasFUxxZSctuAKZFzf.RzGlE.OYM8YILoJrZ.QtHdcGG8z6hgrAhvXeUBl unjcwfZvS7ArqmsXbRTRwW3ZKCQMzVzfoyuA980eyh5kgbO0tI4jMyluhYCwgtS21Loeiw0N.42l CRNQb4Ej8kMAg7BDppelZwFhyWmY8Lrf8OQ5iopxRvQyOsRy9RStVZoIBCp5kojJ5WmBVNeIICa3 YDysEgWOeH2fmK7VOOJQ3Da77h2BQZNJ2Upe5MiyQ6P1goyZt1GRqvcOSAvZ0A5Uv3U2zP6iwZvX nxkacOh5uUQ.cw.xrX_IPf4.2UUDDkNLuPHDif5s67kdPXX6VJmuxdu.Ag.JNU0mibLewXzZBZ5E YM.taF4xwTL2424SKKFwe63Vjq.58W0RzLX0VUYsFoVJqjxpXSSERuKYYuIEbZbmiOmLz4PC93Bf Kv5oWsnvz5XX6.TYU_QJn4tizRuTM3UtLO0ilw7tTKyH143LcvJ96GyP2mfW5G0TfA0QtV8KFQVk _M0heEV5gyjxKXL_NLbMoP2fDYZjbNGCUMiOqqJ5NND8pc_EinmlEPISx1obzKCQlLbwR2O2Xu9a KLbtxFubwsd0n_KWL7Ndj1mQzL7lGSsYbpPVWxLhKL5MyWMiSuI7cFhsICA3YS2Gk6iQt9k3vU2l .Ek8_EtIFSiB.KvcEIDYtlmdVhloio68WWQfjdU_7kMzxqtq7MQmURcCy4GKoMXDofEZj_NcbpHM vZStVlbmNGWI1a7UPBeo6aR_6poPoRrap6dN9pRruHHFuhNG.27Fnd19_ldbu7Im.xOdGFYeZSyD 0QrkJ42hxfdODBlfUiCEnXy5zgvP0WsySg.9coidnot_N5ZJVKK4CQRT57sQdl1cE2Fye8_5TrSI eeZABRU25uupGKUATu63YQoLJGIGYpwkqDKxtvV2d3EOR7VdMTS9TBtV7b_biCR0gAI16jk9BV5B WmIv_zdB42aY1HafJB66i9pjZIkwgEc34Fo1GyuUSqP9BsFo8TuLZU4x0JxXlMNAXzuYRwpczkI3 sdWb4PliDd5huYB_GIZr8xK3Zii4jKAVX48e1OuEQlJnX7bsFD7ebne7RtaZBza22BHq7XRsWBi5 LpWtXczQfMNVXf1joA4eJfu2hawLqWKLDj7nKeJCFGTffu.QlS0irQYtHDhABC4G_6BfBnP8awe9 2N9H0PoMnT0Ga4kBLcL9f7xVkVAFmQBBAvqR6OAUneH3BgXejSE_vO3Zvy80OrnGW1hzXYG.VD9S qHkLP9WCi7J_gHpTBox2b._u6y0hLGHXPLXp1PtCXADm2DPIw86nAqf2XMPw3ziFIW397cqFnJlB RAn_wCACN5B3rIkYd0PXkNyHLhLXOgrwj8hC1M2g1lg_B.T2iMsNngS5Miu8W8eBXXOzOXTDRxDb JmAQ6TP1rZhPyXAGd2N5lRldtHZK6kr.z.BrDqk_AKPk4SWgmu.nNJK.MLm2UqPt2qsdOFX2.I3r TkMGvWg3ZBVXDSCtzZzoKJQPXUnNeqSG4QsmF2Pg5sJ78xtCD5q6rIrxrptMiJvAYnIiRkANeuYm ti4z54g2gDzqShPo7R4u1XKuSXJzqs8rCC5b8AyF8fvSpIG4NrLM3_C7gaX4pR.r24fyr4AeSNau e5eUrAdJtAD9TqdMe2mtKPf3HO3wg2WwUFr9g7_aNVNqNs.Pbh0v4BgEvVKIsGU0TgbLDBE5kHWr ex2reeNRXarLamqlo5dJ1ij2nPiguWb3NhV2R5mxUzSFQcmMVXWxqQWkYdPghjtxPO8zcTYVN9j1 mIo7MeOSRBqqJa87528l9nuUXUDpwP._GvqoAuwXizCPGA3mGLfR9BCbm2nUyz6uPuGhkiRnL3nq _tewRBnZeEa1vZ8LPxN2CI.hPAeVh60wFCHTKFBT0wQ8Rv8BndIM5.md7NXIAntQibwygYpjKKd9 r9jDbM.ug7sKzvaBINlXjUK1ApAzOBxeLGs_9coiIN_RXLPWU9foBo_tmCFGZajCZf4NicaWB7T0 iCyqLfA-- X-Sonic-MF: X-Sonic-ID: c1a297b4-c203-4fa9-aa27-5107d2b96bff Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Thu, 7 Dec 2023 04:31:27 +0000 Received: by hermes--production-gq1-64499dfdcc-m6m9b (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1f8c2da4d6900a2db2a7a3cf6c36dfba; Thu, 07 Dec 2023 04:31:23 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports Message-Id: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D@yahoo.com> Date: Wed, 6 Dec 2023 20:31:13 -0800 Cc: dan@langille.org To: Warner Losh , Current FreeBSD , FreeBSD-STABLE Mailing List X-Mailer: Apple Mail (2.3774.200.91.1.1) References: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.206:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.206:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Sm1Ys5r7Nz3MD2 X-Spamd-Bar: --- man arch reports: QUOTE Some machines support more than one FreeBSD ABI. Typically these = are 64-bit machines, where the =E2=80=9Cnative=E2=80=9D LP64 execution = environment is accompanied by the =E2=80=9Clegacy=E2=80=9D ILP32 environment, = which was the historical 32-bit predecessor for 64-bit evolution. Examples are: LP64 ILP32 counterpart amd64 i386 powerpc64 powerpc aarch64 armv6/armv7 aarch64 will support execution of armv6 or armv7 binaries if the = CPU implements AArch32 execution state, however older armv4 and armv5 binaries aren't supported. END QUOTE (I take "armv6 or armv7 binaries" as what was built targeting a FreeBSD architecture triple for one of those. FreeBSD keeps them distinct.) However, the armv6 part of that is wrong: The infrastructure supports only one 32-bit alternative for a given kernel, not a family of them at once . . . sys/kern/kern_mib.c : static const char * proc_machine_arch(struct proc *p) { if (p->p_sysent->sv_machine_arch !=3D NULL) return (p->p_sysent->sv_machine_arch(p)); #ifdef COMPAT_FREEBSD32 if (SV_PROC_FLAG(p, SV_ILP32)) return (MACHINE_ARCH32); #endif return (MACHINE_ARCH); } . . . static int sysctl_kern_supported_archs(SYSCTL_HANDLER_ARGS) { const char *supported_archs; supported_archs =3D #ifdef COMPAT_FREEBSD32 compat_freebsd_32bit ? MACHINE_ARCH " " MACHINE_ARCH32 : #endif MACHINE_ARCH; return (SYSCTL_OUT(req, supported_archs, strlen(supported_archs) = + 1)); } sys/arm64/include/param.h : #define MACHINE_ARCHES MACHINE_ARCH " " MACHINE_ARCH32 . . . #define MACHINE_ARCH32 "armv7" (There is no "armv6" alternative present.) But with something like: #define MACHINE_ARCH32 "armv7 armv6" MACHINE_ARCH32 is not interpreted as a list of alternatives, each supported. There is code that would have to be reworked to allow a list of alternatives to work. One can build a custom kernel with: #define MACHINE_ARCH32 "armv6" and then, having booted that kernel, then run armv6 on aarch64 --but, then, not armv7. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256132 is about this and has my messy notes as I explored and discovered that multiple 32-bit alternatives did not work. I see that I forgot various quote (") symbols. This note was prompted by: = https://lists.freebsd.org/archives/freebsd-hackers/2023-December/002728.ht= ml that mentions "the list of valid MACHINE_ARCH" that reminded me of this old issue. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Dec 7 09:19:26 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sm7yS4lvdz53tdh; Thu, 7 Dec 2023 09:19:44 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sm7yS4G3Wz4S8X; Thu, 7 Dec 2023 09:19:44 +0000 (UTC) (envelope-from dim@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701940784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+7yN9OEkgGgFe0ohUOL4ljYavKJXghk+9c0DO9vPNUs=; b=dhj1pSd/w5FvDiI6/WyZtTUuMq9cTzwh0iz0cqgeWwYTemuFpiCP5uj+DWCZWZtZQQW+7r XGZv8H5QzDUejYYPpckjGW9ufsCp071Wo6w0NFccd6SfzR/6J6UuVob+ZTaYYp7lA2eFN2 Z+ZmmEOfPetI/GRjSeS+l6AqY0s9izZwYrZ/N3v80ImcSXak55gWR+A8Ueb/hEsOM+DYmm Ln+g6uiM+hD+Z+Q566Sl5K2VNK6rilnbhZh1UaoQ/0PRV3+ce+ptxCqQm7gBjoP11YFDXG hGmJ7N8r+wzd96X+aPcVnEfU4zZDFDxDsvq8WonbdKbEpbMoVBp57F2iKxJcxQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1701940784; a=rsa-sha256; cv=none; b=e9aQIl7Ocxdz3g2y5grA/sHH4E0QuG/QiFa4085Jas+RXW5erPYbHtMo+azWcw+NWTf0Cz mdmG82AdaXTkx3sVYucO5IO2TZBlzu5hN8T272BEHI4fRr/KNZpBDqdmQ/8TkGuIvUaFAg gdexDkNjC4pVVSXNyi8oEkVE8+5p6I3xq6xm1WwS9hC1oRZnTfFFuiRelmnD0cI4iOx2KU qMXSD5ZV50+4mUz1guX0ecr17bJUHy8bMlQ43jj+lk703SPeJwYpgCx02ba4qc9gfttIvV OSJDYw9EPq6Wcy8WCbl8uIcVwUHukKaKEbad5/2tazkt+tjZiGON3FcvrVmg7A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701940784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+7yN9OEkgGgFe0ohUOL4ljYavKJXghk+9c0DO9vPNUs=; b=Xyt0LHNsyGGATdEDmLgZbxZE0DweD3iervXL8R/f/qwJV+IfOeY+vSXnxzkptxsF2TbU42 0uBavJEQgsYXq7GZHzVOo5xnRMKDeftYIo6Ars4HyLVJ3qmsdHoMUckKqJlH7jOOheLfoL /I5yOEY94tOuhfSQkMnXVVTQk3UqX5Ev/arzO1tXtkJb1l1Y3w/cbuwQP/MzLtL2Kq0rxX qSZ9CIFH4mPLB5GtbeZxfeC3cCDgX8JjJR75f4SoF+0gg7JRuP2WCBmDiarFLvNZqxbqB7 PbiF4U9pAmxDvLb1wzven/71kqzw0nzAFNxhBf8fUXA+o1b61SqvV1ljEL2TuQ== Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "tensor.andric.com", Issuer "R3" (verified OK)) (Authenticated sender: dim) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Sm7yS2g78z3Nl; Thu, 7 Dec 2023 09:19:44 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from smtpclient.apple (longrow.home.andric.com [192.168.0.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 4AAE1284BF; Thu, 7 Dec 2023 10:19:42 +0100 (CET) Content-Type: multipart/signed; boundary="Apple-Mail=_F15A5000-3925-42D9-80D8-80DB8FFAB5C2"; protocol="application/pgp-signature"; micalg=pgp-sha1 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports From: Dimitry Andric In-Reply-To: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D@yahoo.com> Date: Thu, 7 Dec 2023 10:19:26 +0100 Cc: Warner Losh , Current FreeBSD , FreeBSD-STABLE Mailing List , dan@langille.org Message-Id: <2F5E2EA1-246A-4098-A964-3C8DEA2EF297@FreeBSD.org> References: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D.ref@yahoo.com> <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D@yahoo.com> To: Mark Millard X-Mailer: Apple Mail (2.3731.700.6) --Apple-Mail=_F15A5000-3925-42D9-80D8-80DB8FFAB5C2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On 7 Dec 2023, at 05:31, Mark Millard wrote: >=20 > man arch reports: >=20 > QUOTE > Some machines support more than one FreeBSD ABI. Typically these = are > 64-bit machines, where the =E2=80=9Cnative=E2=80=9D LP64 execution = environment is > accompanied by the =E2=80=9Clegacy=E2=80=9D ILP32 environment, = which was the historical > 32-bit predecessor for 64-bit evolution. Examples are: >=20 > LP64 ILP32 counterpart > amd64 i386 > powerpc64 powerpc > aarch64 armv6/armv7 So, this might be replaced with "armv6^armv7" or "armv6 xor armv7", = then? -Dimitry --Apple-Mail=_F15A5000-3925-42D9-80D8-80DB8FFAB5C2 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCZXGOHgAKCRCwXqMKLiCW o1ZvAJ9KH0xs8FDDxFxyfKCQBq8Uamj6sQCg/Zp6P2J3uIM8nNcRxurUuVQj7+8= =Sx35 -----END PGP SIGNATURE----- --Apple-Mail=_F15A5000-3925-42D9-80D8-80DB8FFAB5C2-- From nobody Thu Dec 7 10:33:08 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sm9bN03P0z540q9; Thu, 7 Dec 2023 10:33:20 +0000 (UTC) (envelope-from fuz@fuz.su) Received: from fuz.su (fuz.su [IPv6:2001:41d0:8:e508::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "fuz.su", Issuer "fuz.su" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sm9bM2FCzz4b4f; Thu, 7 Dec 2023 10:33:19 +0000 (UTC) (envelope-from fuz@fuz.su) Authentication-Results: mx1.freebsd.org; none Received: from fuz.su (localhost [127.0.0.1]) by fuz.su (8.17.1/8.17.1) with ESMTPS id 3B7AX9qh087594 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 7 Dec 2023 11:33:09 +0100 (CET) (envelope-from fuz@fuz.su) Received: (from fuz@localhost) by fuz.su (8.17.1/8.17.1/Submit) id 3B7AX8OI087593; Thu, 7 Dec 2023 11:33:08 +0100 (CET) (envelope-from fuz) Date: Thu, 7 Dec 2023 11:33:08 +0100 From: Robert Clausecker To: Mark Millard Cc: Warner Losh , Current FreeBSD , FreeBSD-STABLE Mailing List , dan@langille.org Subject: Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports Message-ID: References: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D.ref@yahoo.com> <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D@yahoo.com> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D@yahoo.com> X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:16276, ipnet:2001:41d0::/32, country:FR] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Sm9bM2FCzz4b4f Hi; Am Wed, Dec 06, 2023 at 08:31:13PM -0800 schrieb Mark Millard: > man arch reports: > > QUOTE > Some machines support more than one FreeBSD ABI. Typically these are > 64-bit machines, where the “native” LP64 execution environment is > accompanied by the “legacy” ILP32 environment, which was the historical > 32-bit predecessor for 64-bit evolution. Examples are: > > LP64 ILP32 counterpart > amd64 i386 > powerpc64 powerpc > aarch64 armv6/armv7 > > aarch64 will support execution of armv6 or armv7 binaries if the CPU > implements AArch32 execution state, however older armv4 and armv5 > binaries aren't supported. > END QUOTE > > (I take "armv6 or armv7 binaries" as what was built targeting a FreeBSD > architecture triple for one of those. FreeBSD keeps them distinct.) > > However, the armv6 part of that is wrong: The infrastructure supports > only one 32-bit alternative for a given kernel, not a family of them at > once . . . See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256132 Yours, Robert Clausecker -- () ascii ribbon campaign - for an encoding-agnostic world /\ - against html email - against proprietary attachments From nobody Thu Dec 7 10:36:05 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sm9fr1s24z540y9 for ; Thu, 7 Dec 2023 10:36:20 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sm9fq4YWNz4ddM for ; Thu, 7 Dec 2023 10:36:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62c.google.com with SMTP id a640c23a62f3a-a1c7b20f895so83604166b.2 for ; Thu, 07 Dec 2023 02:36:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701945378; x=1702550178; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YRyu65Zu4HWwiLpjg4YtKBvk8ET3nb7Q0hBODLOoF5Y=; b=scOdVfKCdQJPYNILuLXq6WFNVxH7sW/FiLpaViukHu+ySzjwlizEiKT5kkZioJ7iOd Pmb1aD/SCEv/J3h3dsDW7pEaxIK702y5JYXkxYuuu1r0eCIsYpqPe/+0k5rqFvPETKyE CFmPH0ZKkaW7ude2V2IrKhwq4WK7pdvino+80U8OZHQTi+1korN4N8c5CdBQJpwl1aea 2VU5ANA6i7KfJX7p+o8aLOmdcsnwir5+xeTv5+sHaod54CqoixvnIZKcpF66FJysh3qz mdVSEAXienvOjwYMDVRbj3gaR5XT/HEsHlJpWAo+pePygphe3MlprXHMTO0/k7MAbnqy Jakg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701945378; x=1702550178; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YRyu65Zu4HWwiLpjg4YtKBvk8ET3nb7Q0hBODLOoF5Y=; b=V2tNxT7TDdxjMKLAdzp6umS1eClgFaSl4jQmfCFCXFjYHWp0WwCEqmYVG6GgfJTQ2v sYkQ0etJ75Qi6af3AMWO4/8c6duvk5tdsxy2WNHv+DVnz616FZT7ojF52AVq6wxFgtET JN4yfTY2f2TPZ95jGxMaT1BMv3zUEaJbyPVEtvuEx1rt2ARcOTITvYo36fBLO1daen6r /vR7NIHE4w0lqEqwBeeJflzboK4R5rUkdJaCmhxIebqmAb+nRD+5VQ4uYFOKmFG41D5w eSrgv2Sel2DfvxZ3tW5OhTcPUra9XbH2GIRw8rhJhdY/1jyh7HIbhbBink6Phba0ZHkv 2F1Q== X-Gm-Message-State: AOJu0Yzc7SOqv6lXfLR5XOwQ9N+2kLDG9M1e2Jfn76mbQOOsuiCmt1sx k1dbglzufxX/xfceLFzPnLIRnv3vXVohlWx0hy62zMB6B3tUOd9z X-Google-Smtp-Source: AGHT+IEahUuSnWviLUc7qIX+/hjxxo05qcFDJgGd4RVR0bMyL4PGtXWpGy8oJ7yPtwApoEkyfgJT+Uw+DG2DQuQmx8E= X-Received: by 2002:a17:906:5352:b0:a1d:1fb9:5e39 with SMTP id j18-20020a170906535200b00a1d1fb95e39mr1566562ejo.15.1701945377899; Thu, 07 Dec 2023 02:36:17 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D.ref@yahoo.com> <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D@yahoo.com> <2F5E2EA1-246A-4098-A964-3C8DEA2EF297@FreeBSD.org> In-Reply-To: <2F5E2EA1-246A-4098-A964-3C8DEA2EF297@FreeBSD.org> From: Warner Losh Date: Thu, 7 Dec 2023 03:36:05 -0700 Message-ID: Subject: Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports To: Dimitry Andric Cc: Mark Millard , Current FreeBSD , FreeBSD-STABLE Mailing List , Dan Langille Content-Type: multipart/alternative; boundary="000000000000ad589a060be909eb" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Sm9fq4YWNz4ddM --000000000000ad589a060be909eb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Dec 7, 2023, 2:19 AM Dimitry Andric wrote: > On 7 Dec 2023, at 05:31, Mark Millard wrote: > > > > man arch reports: > > > > QUOTE > > Some machines support more than one FreeBSD ABI. Typically these a= re > > 64-bit machines, where the =E2=80=9Cnative=E2=80=9D LP64 execution = environment is > > accompanied by the =E2=80=9Clegacy=E2=80=9D ILP32 environment, whic= h was the > historical > > 32-bit predecessor for 64-bit evolution. Examples are: > > > > LP64 ILP32 counterpart > > amd64 i386 > > powerpc64 powerpc > > aarch64 armv6/armv7 > > So, this might be replaced with "armv6^armv7" or "armv6 xor armv7", then? > The binaries are basically the same. But you need one set of libraries or the other since the calling conversations differ. I think you'll need to enhance the current sysctl to be per jail. Or you could migrate away from armv6. It's days are numbered in main. Warner > -Dimitry > > --000000000000ad589a060be909eb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Dec 7, 2023, 2:19 AM Dimitry Andric <dim@freebsd.org> wrote:
On 7 Dec 2023, at 05:31, Mark Millard <marklmi@= yahoo.com> wrote:
>
> man arch reports:
>
> QUOTE
>=C2=A0 =C2=A0 =C2=A0Some machines support more than one FreeBSD ABI.=C2= =A0 Typically these are
>=C2=A0 =C2=A0 =C2=A064-bit machines, where the =E2=80=9Cnative=E2=80=9D= LP64 execution environment is
>=C2=A0 =C2=A0 =C2=A0accompanied by the =E2=80=9Clegacy=E2=80=9D ILP32 e= nvironment, which was the historical
>=C2=A0 =C2=A0 =C2=A032-bit predecessor for 64-bit evolution.=C2=A0 Exam= ples are:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0LP64=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0ILP32 counterpart
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0amd64=C2=A0 =C2=A0 =C2=A0 =C2= =A0 i386
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0powerpc64=C2=A0 =C2=A0 powerpc=
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0aarch64=C2=A0 =C2=A0 =C2=A0 ar= mv6/armv7

So, this might be replaced with "armv6^armv7" or "armv6 xor = armv7", then?

=
The binaries are basically the same. But you need one set= of libraries or the other since the calling conversations differ. I think = you'll need to enhance the current sysctl to be per jail.

Or you could migrate away from armv6.= It's days are numbered in main.

Warner


-Dimitry

--000000000000ad589a060be909eb-- From nobody Thu Dec 7 10:46:35 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Sm9v01L0Lz541VJ for ; Thu, 7 Dec 2023 10:46:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-54.consmr.mail.gq1.yahoo.com (sonic308-54.consmr.mail.gq1.yahoo.com [98.137.68.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Sm9tz60BPz3DGn for ; Thu, 7 Dec 2023 10:46:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1701946008; bh=tJoS8yldyOObP/NxvaFESMrct9BZAwJTv6JjpYp7lgw=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=B9LNuZUSi3boMkrZ4Mun0FZPVi8lcXrL2ozYygfDnpMc15WZuOrtaEmNJsTiGfrx+HV/ebQUuXvG+O0M1NiJeLfw4OJrdOoxYoJBgy52va81L/dfC46o8ozMRcO+X/dUPW5r5DsPI45K/zq6YHOdkwgrnNtcktCawE2HgrAxRIyHFnksUL8p5UOxF2UFUGBRv8banhFHMjUSSm8cvzDXtEnnyCz1Y4ZB13N5lyvqcemhjq2j3rEn2C51PmL1+d3FGqi2b+D0huDfRBU7YGh2QWm4aS2aRmWtFncS+REKTjZJw590uUZRqZDwwp1ygAUSsAs5VSR21lO3TO/2+gd0JQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1701946008; bh=wvVV+doOqyBwbxUwgLY5YJ8kfwYYksR8XPpSDQk2Nt9=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=e9H6+5Bf46QcNmsdLGkUjHmVnIEfpOf40s8fmhS0d0ULZcfH5Tlei/+87AakG0a1DpdSzh5cFhiCLgjO8WIfpj1aO/TbNiBdHPd/A3If0DHqSYoHrzcekUbmtO7mVIoEs/dECEE7cGUcYkWitt19fB5DTHF8bajQjOFHMstgyrNi+2MoRhHtIZPO9m+tCAJN677nn9TS0gjWDYfMSa67LgoF9ItaxgZV+RThdHKU6pdt1Cj+L7QoAwg0mR/FQu9uPMHI3VfRkSkwZDB1cyHb2jSGdRYGpJYCKrJINE4xUKk/M/WPrEdw6pw8Ei7n4Tzt0Hp6CQB+3XfOmWTpxE5QGg== X-YMail-OSG: X3GGa3cVM1kt_SblMETFHPprIBazfx1NsQXAiqKcr3vgPwDcSrkeCXkOjoE4m5u oidfHMzZgZk4Xx.cNOv9zYfKvcXubVe56Nr2j197.62yLld8wHQWYeNelrFkR9ZEjOTgsSRz84Hv a_yNxMAhShaeOHv_Vu8YnPeg1Cmcv0xywKWFfiGZZFuS4ssdH6nxNi2qXsM0s70bgnM5oWiasW_w WoRGE1CT3iHJ3li58k6oFWtAbxe1hKrdHtNj9WmDva0DY64aYDbd7RdaKPQ5S0WDLzGTr9UQ4Lwc EvMyhLdZLYWdjlyfUhs4hXvxF50XV6_SOeIl3T8JkQyp3Dl.Ra7y1chipU4G2DLVYIqOjqQwftZf 2T7EqW2Ncljv806wopyHxtpezRL_nr8Wz0a34qpplyIkX3It.Bz0zrcHLkj3ve7dGbbZfesw6K2j U0RpbxImWjqgoB9dDkXVl7BiSQsXId4bA1NLA62O2pRXw49TJOaD6Z7Fk76uE52WMZYeYdc5.E0d q5K2fZ97hIk4hLGuA2Ehy.k4nD9yZ4GaM7UkUVYxiSQx01D_7XOliejXRjRfRpvmLmwtw9Sgqbqm mDxYUBuOeVXjjHLW8erlCPULKdfXMSoqaZtC2Zq3FQMhp.lud0MZaV0JATL66Phfpz2APzLaRZn_ fxLopGXrJVvL7IP772Ax5eC1cka4Pugki3306vsS4pto3xQZdkxAnrAmmFJ1XI_M9ioiEU9mJI9z jy5Ik96_UHlRoOVMWgyeAauHrtW7gIF_lc5RbqVI3T.raXqUZeav1Orjm0psuda4kfNxvlvBdM.d HpuUiC9cX_rZA2E1muUsRnmp9X3wgtsR5JA13aWrnYc0xuFw_hbpW2hza.JMa.nOuEcznEfqmYe2 ar9Z1JaJL8f3CBELLjdNsOamPJq_0kkTycEVF0BjmrLancflusYTXEIf_z3k8DU8B8.STpqGNJiL HLEdmCIJuBVLzsP554PtCgBge22_cdsypTkp9QkORxoIRYHRpQnuqh9FLVjy1Az7z645vKW07zJP S86NWDc8G1xr9Y5KG0KG_YODXwWb85A9FuGETSRMXzTo_MTUrqnIQtQpIVae7OzJx1d6BJWwPLZq jbtvOU9gWRsMK4eFslh8oPh201D6pnEKZKtmcoOAx2OLBZZwXBTbeBJDh_DLC8e7OFpoSpPTmbBP xaOmBgJtqgx_JY69Gw3HtcaLxs.KHc89dalcEz25jkesGwedV6IG5D5.rBOj5nqRlAldYZn8Ay0v dSKCxqKR7lU.xTlnX8phR7ueQ6KygOfiJ8j48jrSEpAH_ARYvbE.ipsRBVUtJ2ML7.s04ilxKegb yO01SBfy2Pygp9P116d95DjCy3eLjKCu4CifnFZ6GMcRGXVuLG3aabBcSHsMGnCZQSBFyHZPfmck H0sc5GA.3glU6zulers8PbqXF4.qAUx.zozsCOok4GbMJTsJ0Wf2FlRA7KdwJDLOmUHKHWEhot.F JyBu2BeuoiVXHtcs95Xux2u8CebRMw_K2adlZkji6bQX33TIcvNn_guaPSUpU6efGDEDCSF0DfkZ 8paHV4Et1QsfORLR.jAqEzFGdhajrfTjB6GUG7dpU_iD._XzBcCH0XaXoMNkTvmDsUdUWMJOiaXJ m_2nmUI13V2vJAdKG5RL0izYjLZ2qHoFiFvOyIeSGR6pmcQBPK6LtrGbdcWpMgD.kmscn0YBM92o SWVxjn1JgAmhd9j927aNJvZYthb0yUiFCXe.oDyEAy8NCF2VJxilCnXnY.ageJ_kfxy1dYr7c4LE hBLjb2GH2Y6MdwE_Y38H1OkeOR6Mbezhhx8ZhR6nI3hu.Q2eQhMvw_dfyicNKY1xyNsCpob0QJwt 1ya0Wn5tfmgUteNk2TvF.UZyMA3IhJElrl.V74TtYUndTMbQhLuExVwBm48wEknOJxvuXnGoR23O G8P4TVGHqkSGc1J8rugH.eCQHZvNDrp9cZMNqalj1FUO0VktUPi1c118ol4mwOP4Dmzs15KccGHe 1383ORs5sMoaejh9YSSJSJC_JM5lMFwVmbd4NdKTIbk5FXVJCK2gb4RnhiOm32CKlAY0yzb9JGkL DbAzY.ZFL1smHvcEA9wMdHKS8XleOyunbDDg.uXJbqY36NClKNNxaRGcQD9rJskVxKxc80WmQZdy qwsomMPpqimeUX9rdJDvwXfKdzcspgioZqJxcGGzwd0y3rEg1GkF9B8HPTXwK54CmBq.R1stj4d7 eNKGH7j_hIZ7VGJnEd9OW_gNSK9nCLaolB.uyTepzuaYb0SWa2IzHcgUPM.mHIHW_Pryp5G.yLrJ hog-- X-Sonic-MF: X-Sonic-ID: c2b0902b-c7db-45eb-b2ee-c9a549c60f96 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Thu, 7 Dec 2023 10:46:48 +0000 Received: by hermes--production-gq1-64499dfdcc-pxg8l (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 21ebad9e45431473a536abcc75eb83f5; Thu, 07 Dec 2023 10:46:46 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: Re: aarch64 and armv6 vs. armv7 support: armv6 is not supported, despite what "man arch" reports From: Mark Millard In-Reply-To: <2F5E2EA1-246A-4098-A964-3C8DEA2EF297@FreeBSD.org> Date: Thu, 7 Dec 2023 02:46:35 -0800 Cc: Warner Losh , Current FreeBSD , FreeBSD-STABLE Mailing List , dan@langille.org, Mike Karels Content-Transfer-Encoding: quoted-printable Message-Id: <1B672854-4011-4F4D-BFBC-4C3ABF6FA4FB@yahoo.com> References: <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D.ref@yahoo.com> <3C33BBA5-A84D-4F46-B38E-B67F9FE0196D@yahoo.com> <2F5E2EA1-246A-4098-A964-3C8DEA2EF297@FreeBSD.org> To: Dimitry Andric X-Mailer: Apple Mail (2.3774.200.91.1.1) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Sm9tz60BPz3DGn On Dec 7, 2023, at 01:19, Dimitry Andric wrote: > On 7 Dec 2023, at 05:31, Mark Millard wrote: >>=20 >> man arch reports: >>=20 >> QUOTE >> Some machines support more than one FreeBSD ABI. Typically these = are >> 64-bit machines, where the =E2=80=9Cnative=E2=80=9D LP64 execution = environment is >> accompanied by the =E2=80=9Clegacy=E2=80=9D ILP32 environment, = which was the historical >> 32-bit predecessor for 64-bit evolution. Examples are: >>=20 >> LP64 ILP32 counterpart >> amd64 i386 >> powerpc64 powerpc >> aarch64 armv6/armv7 >=20 > So, this might be replaced with "armv6^armv7" or "armv6 xor armv7", = then? Only for folks that build from source. For those folks, a footnote about updating MACHINE_ARCH32 in sys/arm64/include/param.h would be appropriate. It is not exactly obvious or commonly known. Hmm, thinking more about the old bugzilla information . . . I'll also note that my information predated lib32 on aarch64: just chroot/jail sorts of use back then, and I just tested chroot back then. I've never tested a lib32 context for armv6 on aarch64 for an adjusted MACHINE_ARCH32. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Dec 7 22:38:37 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmThS4RPzz53kN3 for ; Thu, 7 Dec 2023 22:38:48 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmThR0ZY3z3Nm2 for ; Thu, 7 Dec 2023 22:38:47 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=nomadlogic.org header.s=04242021 header.b=dipBMxDg; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 66.165.241.226 as permitted sender) smtp.mailfrom=pete@nomadlogic.org; dmarc=pass (policy=quarantine) header.from=nomadlogic.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1701988718; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QTANYxtBfVGjHsqhqZcODUXkF5P9yPX0UOBF2xVn3DU=; b=dipBMxDgH1mVmKERKvKPbJKNjML1zpDTP7vWAuVI9YMFp3mgLHqTNez35/Eu9aj/B9QNPQ RINYd0OZJ5fSIC1l8/ksaZ0XlhJLpkktTgwj+DbF6DPgZQ/9aH0RqWRPgUawyxliPIWd2e Qf6uRMyZtaZheH0TEPaE/kZNcs68Lu8= Received: from [192.168.1.240] (024-024-168-214.inf.spectrum.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id fc295d87 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 7 Dec 2023 22:38:38 +0000 (UTC) Message-ID: Date: Thu, 7 Dec 2023 14:38:37 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: nvme timeout issues with hardware and bhyve vm's Content-Language: en-US To: Warner Losh Cc: FreeBSD Current References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> From: Pete Wright In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.95 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.960]; DMARC_POLICY_ALLOW(-0.50)[nomadlogic.org,quarantine]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[nomadlogic.org:s=04242021]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[nomadlogic.org:+]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; RCVD_TLS_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4SmThR0ZY3z3Nm2 X-Spamd-Bar: --- On 10/13/23 7:34 PM, Warner Losh wrote: > > > the messages i posted in the start of the thread are from the VM itself > (13.2-RELEASE).  The zpool on the hypervisor (13.2-RELEASE) showed no > such issues. > > Based on your comment about the improvements in 14 I'll focus my > efforts > on my workstation, it seemed to happen regularly so hopefully i can > find > a repo case. > > > Let me now if you see similar messages in stable/14. I think I've fixed > all the > issues with timeouts, though you shouldn't ever seem them in a vm setup > unless something else weird is going on. > Hi Warner, just resurfacing this thread because I've had a few lockups on my workstation running 14.0-STABLE. I was able to capture a photo of the hang and this seems to be the most important line: nvme0: Resetting controller due to a timeout and possible hot unplug. When I scan the device after reboot I don't see any errors, but if there is a particular thing I should check via nvmecontrol please let me know. Also, since it mentions possible hot unplug I wonder if this is hardware/firmware related to my system? Anyway, haven't found a repro case yet but it has locked up a few times the past two weeks. -pete -- Pete Wright pete@nomadlogic.org From nobody Thu Dec 7 22:49:03 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmTwX5bywz53l1M for ; Thu, 7 Dec 2023 22:49:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmTwX3TxTz3R3G for ; Thu, 7 Dec 2023 22:49:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-54dcfca54e0so1608177a12.1 for ; Thu, 07 Dec 2023 14:49:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701989355; x=1702594155; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=W6wsczIHjzUX84vUc+gXeX5UQmTe3nQkJFjqVWTTeNg=; b=nOsW+O8XIedJn08nLbJ1/flkKcH7f3FP/smjNd71gX2d2tHagSPG6ehTUzZSfoz3dn sA9bPAzWbmWzBhg1eLW/t1x59uS/GSNW4ob4d4wUUaIht+yKInsTXOjMF9twwmUeU5IY XjGbXI+iCd9wpr+3oYHtNV/iKKSn9qUkAQ+Y+GIy+5nzb86MVtv70aUylx0UQ7GSRAFL Rps9jRQ/5Nvkrxrqkn1tIK+UhZrBc9ry0x4ZR5HTg3l0snfxnBYm+olReoi4Q96c29ri PM9AKbXRwsV1lijT7mDBv/+9+7PuARiAnmyNn+uwvE7BWHled/w8eU4JtOyDJfEFhCLG zV5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701989355; x=1702594155; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=W6wsczIHjzUX84vUc+gXeX5UQmTe3nQkJFjqVWTTeNg=; b=gGgAKsSgHP1bgNqw6sTugeOlwnt0rkAORcDUJNMriy+hRlw50XNZgsBQhz6tBFAhWL 5iPtaS7Io+1KQg+QxDnQa1VuUO5tV0PzSLpnLEXu5ZT1QdVGNKl62U1mWsX6X8J148vC cs35WUYTIxJ/4SrSo9/Xjehe53gQgaCZCclq+HAvzg2RfX9nf0Bix2zEU26tzWF0Ge2H PA9pCYzKLDX0RCsUJkOa66uFpdM/pc1l0gPQbLJ3+m5sTB/ZdJpt+4K8hX6jGV/0dmIW KODu6kMfwYMhGqfOWso9peH2yugTj9Lggl4m4yNruzgK1sq4Wu9azmjStU2iqMN1hnw6 Q/Kg== X-Gm-Message-State: AOJu0Yz4oU5ND88xJVijf8xyEcMOIl3q/i80Uzc38XrOgCFxRlG0nblP pWi0SX7KBCgr88xPYHgvHujFtvWqt5Q8NnwUUy66in0LZk/We2vH X-Google-Smtp-Source: AGHT+IH+OSl/uZVqHcFJksYc+fPhEH0kKJcI9lkPdeQmUYSKXftmLmRuiQSsfyui8Lrtv5aj8oN3kOQtAye11K4OKxU= X-Received: by 2002:a05:6402:1619:b0:54d:ae:1759 with SMTP id f25-20020a056402161900b0054d00ae1759mr1863876edv.24.1701989354909; Thu, 07 Dec 2023 14:49:14 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> In-Reply-To: From: Warner Losh Date: Thu, 7 Dec 2023 15:49:03 -0700 Message-ID: Subject: Re: nvme timeout issues with hardware and bhyve vm's To: Pete Wright Cc: FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000e942a4060bf34636" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmTwX3TxTz3R3G --000000000000e942a4060bf34636 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Dec 7, 2023 at 3:38=E2=80=AFPM Pete Wright wr= ote: > > > On 10/13/23 7:34 PM, Warner Losh wrote: > > > > > > > the messages i posted in the start of the thread are from the VM > itself > > (13.2-RELEASE). The zpool on the hypervisor (13.2-RELEASE) showed = no > > such issues. > > > > Based on your comment about the improvements in 14 I'll focus my > > efforts > > on my workstation, it seemed to happen regularly so hopefully i can > > find > > a repo case. > > > > > > Let me now if you see similar messages in stable/14. I think I've fixed > > all the > > issues with timeouts, though you shouldn't ever seem them in a vm setup > > unless something else weird is going on. > > > > > Hi Warner, just resurfacing this thread because I've had a few lockups > on my workstation running 14.0-STABLE. I was able to capture a photo of > the hang and this seems to be the most important line: > > nvme0: Resetting controller due to a timeout and possible hot unplug. > > When I scan the device after reboot I don't see any errors, but if there > is a particular thing I should check via nvmecontrol please let me know. > Also, since it mentions possible hot unplug I wonder if this is > hardware/firmware related to my system? > > Anyway, haven't found a repro case yet but it has locked up a few times > the past two weeks. > What the message means is that (a) we stopped getting interrupts from the device and (b) when we went to check on the status of the device it read back like missing hardware. So is this from inside the VM running under bhyve, or in the host that's hosting the VM? We have different next steps depending on where it is. Warner --000000000000e942a4060bf34636 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Dec 7, 2023 at 3:38=E2=80=AFP= M Pete Wright <pete@nomadlogic.or= g> wrote:


On 10/13/23 7:34 PM, Warner Losh wrote:
>

>
>=C2=A0 =C2=A0 =C2=A0the messages i posted in the start of the thread ar= e from the VM itself
>=C2=A0 =C2=A0 =C2=A0(13.2-RELEASE).=C2=A0 The zpool on the hypervisor (= 13.2-RELEASE) showed no
>=C2=A0 =C2=A0 =C2=A0such issues.
>
>=C2=A0 =C2=A0 =C2=A0Based on your comment about the improvements in 14 = I'll focus my
>=C2=A0 =C2=A0 =C2=A0efforts
>=C2=A0 =C2=A0 =C2=A0on my workstation, it seemed to happen regularly so= hopefully i can
>=C2=A0 =C2=A0 =C2=A0find
>=C2=A0 =C2=A0 =C2=A0a repo case.
>
>
> Let me now if you see similar messages in stable/14. I think I've = fixed
> all the
> issues with timeouts, though you shouldn't ever seem them in a vm = setup
> unless something else weird is going on.
>


Hi Warner, just resurfacing this thread because I've had a few lockups =
on my workstation running 14.0-STABLE.=C2=A0 I was able to capture a photo = of
the hang and this seems to be the most important line:

nvme0: Resetting controller due to a timeout and possible hot unplug.

When I scan the device after reboot I don't see any errors, but if ther= e
is a particular thing I should check via nvmecontrol please let me know. =C2=A0 Also, since it mentions possible hot unplug I wonder if this is
hardware/firmware related to my system?

Anyway, haven't found a repro case yet but it has locked up a few times=
the past two weeks.

What the message me= ans is that (a) we stopped getting interrupts from the device and (b) when = we went to check on the status of the device it read back like missing hard= ware.

So is this from inside the VM running under = bhyve, or in the host that's hosting the VM? We have different next ste= ps depending on where it is.

Warner=C2=A0
--000000000000e942a4060bf34636-- From nobody Thu Dec 7 23:09:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmVN65t0kz53mMS for ; Thu, 7 Dec 2023 23:09:42 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmVN50DMcz3T8p for ; Thu, 7 Dec 2023 23:09:40 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of junchoon@dec.sakura.ne.jp has no SPF policy when checking 153.125.133.21) smtp.mailfrom=junchoon@dec.sakura.ne.jp; dmarc=none Received: from kalamity.joker.local (123-1-22-158.area1b.commufa.jp [123.1.22.158]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 3B7N9Tfs098580 for ; Fri, 8 Dec 2023 08:09:29 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Fri, 8 Dec 2023 08:09:29 +0900 From: Tomoaki AOKI To: freebsd-current@freebsd.org Subject: Re: nvme timeout issues with hardware and bhyve vm's Message-Id: <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> In-Reply-To: References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-1.47 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.974]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; R_DKIM_NA(0.00)[]; HAS_ORG_HEADER(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[sakura.ne.jp]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4SmVN50DMcz3T8p X-Spamd-Bar: - On Thu, 7 Dec 2023 14:38:37 -0800 Pete Wright wrote: > > > On 10/13/23 7:34 PM, Warner Losh wrote: > > > > > > > the messages i posted in the start of the thread are from the VM itself > > (13.2-RELEASE).  The zpool on the hypervisor (13.2-RELEASE) showed no > > such issues. > > > > Based on your comment about the improvements in 14 I'll focus my > > efforts > > on my workstation, it seemed to happen regularly so hopefully i can > > find > > a repo case. > > > > > > Let me now if you see similar messages in stable/14. I think I've fixed > > all the > > issues with timeouts, though you shouldn't ever seem them in a vm setup > > unless something else weird is going on. > > > > > Hi Warner, just resurfacing this thread because I've had a few lockups > on my workstation running 14.0-STABLE. I was able to capture a photo of > the hang and this seems to be the most important line: > > nvme0: Resetting controller due to a timeout and possible hot unplug. > > When I scan the device after reboot I don't see any errors, but if there > is a particular thing I should check via nvmecontrol please let me know. > Also, since it mentions possible hot unplug I wonder if this is > hardware/firmware related to my system? > > Anyway, haven't found a repro case yet but it has locked up a few times > the past two weeks. > > -pete > > > -- > Pete Wright > pete@nomadlogic.org If I myself encounter this kind of problem ON BARE METAL HARDWARE, I would usually suspect *Overheating caused hang of NVMe controller or PCI bridge on SSD, or *Unstable physical connection (bad contact) first. -- Tomoaki AOKI From nobody Thu Dec 7 23:16:51 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmVXT1QNGz53mk5 for ; Thu, 7 Dec 2023 23:16:57 +0000 (UTC) (envelope-from leres@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmVXT0wr2z3VpX; Thu, 7 Dec 2023 23:16:57 +0000 (UTC) (envelope-from leres@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701991017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yc+B62csqYEeO6mzC+/T87tiAtO9eFnMTxuQBN3wBZM=; b=DMlaVe+r5iodxOIKv9qHs8UBZ+DN6X+4Wmq7QNEiAsqV/fA+nug1W0h++I0aTV0GDhXkMM uLKfhz7TE1CUHvNzkDiXex0bByR7D4B6g9Bp9VMx2LghVTGnJfzC6a298pX+V8uAeTh+Ai T/1AWqslXfkSBTv+F4Jv66fgCUK9y7ckLAwbtXNdwBog5Za7/1vXzE0QvReJH88R2+GlqB TgeZHu04CBjdx8mFDAA31omkKBP5+321FWlgSGhLHDaQtDcoS00lmmfhOCcXBqMyIs6ISf ixsgZrSTTOWvcUjTuB/7zlNRt36gdSv1rlU7xbkmQirM4mijUmzc7fkCiuV5cA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1701991017; a=rsa-sha256; cv=none; b=VRCNPHI2McOO7lgOzuWqOuz8Zg5uFwe1j5+y0Y8wR9S6LqZkwtqldEyQhpr5Uhrl5p0O08 9I4v9+Soy+vok2O71oVazLICidYQ4XwP4lTyG0ugfYlhQGYtyq1MuwNln7Oo0kQbTX08nm +XmhJMnCoNVDFOjT8EDdKWQTIRwj1LBniegHsYA5+VFhyxd6p3+exSPauT48rjYPEesSFk KEzbq/0MzmlIJ72rFs2sc2P1vVkMEIsVn8/pSgR1sdWytR813MWqhZD5hnSm+RfPc19icl DbdOTQiZBJ3Rj1tsJohIi8hL7F64OshCY2GKKP4Ygs31WjXw6r5+pho6FmUnbw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1701991017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yc+B62csqYEeO6mzC+/T87tiAtO9eFnMTxuQBN3wBZM=; b=JKjqhR0DoqiGYfO3Ny0gZmyl6iEz/LROIS6JFfdOLYzELwrPkITpGzsAFJ5nNZtOuzYiPO Hq7XzbzdnSbOVvBn6cFkouvVfc8hMDdmQDC3DTqkpd1NrE+tO8MTMQTl/Lzr0+A7Hx0Gcg YM5/YRGTcmFAjl809v59QrrsAcsRCscKO+3Ohh5Te8rAPONOMZ7cmvZkSW8bve/LY4CP99 Lxf91N8T+NH2JnNspNNr8m8B0XFelx00LyzGFJ0t/3ALwXC53gGi1Kly0whuEB8bXL/LFq hAVBOtzcokQf0x1PbayLJa2eiOEI+RcFkI41Lf6hy9ZJwQ3pvQRUbzsy979pgw== Received: from [IPV6:fd:1965::2] (unknown [IPv6:2600:1700:ab1b:6800:2e0:edff:fece:8f27]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: leres) by smtp.freebsd.org (Postfix) with ESMTPSA id 4SmVXS4lDTzqBW; Thu, 7 Dec 2023 23:16:56 +0000 (UTC) (envelope-from leres@freebsd.org) Message-ID: <60642a1e-1603-495f-a2d3-5f3d3c82e474@freebsd.org> Date: Thu, 7 Dec 2023 15:16:51 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: nvme timeout issues with hardware and bhyve vm's Content-Language: en-US To: Tomoaki AOKI , freebsd-current@freebsd.org References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> From: Craig Leres In-Reply-To: <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 12/7/23 15:09, Tomoaki AOKI wrote: > If I myself encounter this kind of problem ON BARE METAL HARDWARE, > I would usually suspect > > *Overheating caused hang of NVMe controller or PCI bridge on SSD, or This would also be my first guess. Five years ago I had an nmve in an intel nuc that would sometimes "go to sleep", here's the thread https://lists.freebsd.org/pipermail/freebsd-hackers/2018-May/052783.html @imp helpfully suggested running "nvmecontrol logpage -p 2 nvme0" which showed mine was hot (60° C/140° F)! I adjusted the fan settings in the bios and have never had an issue since. Craig From nobody Thu Dec 7 23:44:07 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmW7t1W40z53pnM for ; Thu, 7 Dec 2023 23:44:10 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmW7s5fwvz3bZy for ; Thu, 7 Dec 2023 23:44:09 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1701992647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VSnQTWGs1LNvpfDMCLB26FWZSIKDxeESs4Y74OZ4EbA=; b=BffRX15zEXeWKH4LvHKJLWVLUTpY9nu9vszA5XjRXgwgtwWljRbDqQsuc0ChHEJNy1x1kt EkWDNLOQE/CBYNAV4qy89q6X4TRkRyTSnWISR0xOC5bwgDrmnuplacJbxj+ffWlDPWQq4m frj7Nk4UwLZYEHrtugyBH6uDAgmwQFU= Received: from [192.168.1.240] (024-024-168-214.inf.spectrum.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id b9910b89 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 7 Dec 2023 23:44:07 +0000 (UTC) Message-ID: Date: Thu, 7 Dec 2023 15:44:07 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: nvme timeout issues with hardware and bhyve vm's Content-Language: en-US To: Warner Losh Cc: FreeBSD Current References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> From: Pete Wright In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmW7s5fwvz3bZy On 12/7/23 2:49 PM, Warner Losh wrote: > > > On Thu, Dec 7, 2023 at 3:38 PM Pete Wright > wrote: > > > > On 10/13/23 7:34 PM, Warner Losh wrote: > > > > > > >     the messages i posted in the start of the thread are from the > VM itself > >     (13.2-RELEASE).  The zpool on the hypervisor (13.2-RELEASE) > showed no > >     such issues. > > > >     Based on your comment about the improvements in 14 I'll focus my > >     efforts > >     on my workstation, it seemed to happen regularly so hopefully > i can > >     find > >     a repo case. > > > > > > Let me now if you see similar messages in stable/14. I think I've > fixed > > all the > > issues with timeouts, though you shouldn't ever seem them in a vm > setup > > unless something else weird is going on. > > > > > Hi Warner, just resurfacing this thread because I've had a few lockups > on my workstation running 14.0-STABLE.  I was able to capture a > photo of > the hang and this seems to be the most important line: > > nvme0: Resetting controller due to a timeout and possible hot unplug. > > When I scan the device after reboot I don't see any errors, but if > there > is a particular thing I should check via nvmecontrol please let me > know. >   Also, since it mentions possible hot unplug I wonder if this is > hardware/firmware related to my system? > > Anyway, haven't found a repro case yet but it has locked up a few times > the past two weeks. > > > What the message means is that (a) we stopped getting interrupts from > the device and (b) when we went to check on the status of the device it > read back like missing hardware. > > So is this from inside the VM running under bhyve, or in the host that's > hosting the VM? We have different next steps depending on where it is. > OK awesome thanks for that context, so this is on a bare metal workstation. -pete -- Pete Wright pete@nomadlogic.org From nobody Thu Dec 7 23:49:29 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmWG41RMdz53ptr for ; Thu, 7 Dec 2023 23:49:32 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmWG36Wwsz3cSK; Thu, 7 Dec 2023 23:49:31 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1701992970; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZogneQpV8kLBCw8K9/WmfTZ4gSVCIkvr8uKsaWeRUKo=; b=CedN+6UdedYYjpGe4VbpeqpHXP5R554LPJIOFqpdwolwb8HLLM/yt/gcaMsUGKFVxIRm2t 8usSXF3wdpXXfmpn/mKUL20CF8FY9E0krLtBQOilbyjKph9VLtkqKc4eJhT0qUPa30QNph Ma+SftE7MkebAMvExGOqtkbAPW9BhUM= Received: from [192.168.1.240] (024-024-168-214.inf.spectrum.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id fd23d911 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Thu, 7 Dec 2023 23:49:30 +0000 (UTC) Message-ID: <4384561a-2600-43fc-bba8-e7f5e2c676a3@nomadlogic.org> Date: Thu, 7 Dec 2023 15:49:29 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: nvme timeout issues with hardware and bhyve vm's Content-Language: en-US To: Craig Leres , Tomoaki AOKI , freebsd-current@freebsd.org References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> <60642a1e-1603-495f-a2d3-5f3d3c82e474@freebsd.org> From: Pete Wright In-Reply-To: <60642a1e-1603-495f-a2d3-5f3d3c82e474@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmWG36Wwsz3cSK On 12/7/23 3:16 PM, Craig Leres wrote: > On 12/7/23 15:09, Tomoaki AOKI wrote: >> If I myself encounter this kind of problem ON BARE METAL HARDWARE, >> I would usually suspect >> >>   *Overheating caused hang of NVMe controller or PCI bridge on SSD, or > > This would also be my first guess. > > Five years ago I had an nmve in an intel nuc that would sometimes "go to > sleep", here's the thread > > > https://lists.freebsd.org/pipermail/freebsd-hackers/2018-May/052783.html > > @imp helpfully suggested running "nvmecontrol logpage -p 2 nvme0" which > showed mine was hot (60° C/140° F)! I adjusted the fan settings in the > bios and have never had an issue since. > oh interesting, i'll run that next time it locks up. the box is well ventilated, but that's not to say its not overheating. right now its at: Temperature: 314 K, 40.85 C, 105.53 F nvemecontrol doesn't list any errors or warnings though: Media errors: 0 No. error info log entries: 0 Warning Temp Composite Time: 0 Error Temp Composite Time: 0 thanks for the tip! -pete -- Pete Wright pete@nomadlogic.org From nobody Thu Dec 7 23:59:08 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmWTN60vDz53qTy for ; Thu, 7 Dec 2023 23:59:20 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmWTN45nCz3fMj for ; Thu, 7 Dec 2023 23:59:20 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52b.google.com with SMTP id 4fb4d7f45d1cf-54f4b31494fso375211a12.1 for ; Thu, 07 Dec 2023 15:59:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1701993559; x=1702598359; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0+/GpNECtoZAKMpsNfdLxZeZQs2P98BU6TkVJDPFo4k=; b=mQ293sw+Sd9sIcnQnDhE/C/OyBt/1ROb4pQe5rOLNmOyIp4dZagwe6rvPlp4TlbFCk xYJUzmpPwtVU9rSEfd1+JpMuGnmghgeP/yXZJ3uvsqYnUlf5L3+uxAnGKQD6p4ChqPJE fcmiC6qsz4LjGjTCaMaBHq94fC1r+uE68l/n9OxkOfEFy6Pq+nkYxE5bZLmkj6fGd3W9 R6j1kXkaxmmmEt8FELXtWqJQKAKNdu2uYx0tlyM29bgRJ1bAEaIv10FLPCPYAEWR1IJp A5zD/0muhQdeaR4R37wxlXcve3gtuACY0PYMJclLNZeHQAOwbLBn/vcJd1KvObitO1zR ZKtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701993559; x=1702598359; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0+/GpNECtoZAKMpsNfdLxZeZQs2P98BU6TkVJDPFo4k=; b=Wh84uaWKQqOlZQeXxaosocKDD+n2ruLj3hTlgQ/XwAzeaZEgpxM3xpDqy4Y2NWOE6R 4o1xfHDykfL+lkOLNFQfmHSXeivfjyuJz+sU/B0sXH3LcQR08COQsrk2wWT11CbdOgFU t+RUm8yqUYcXKgyLzVN3JPv6bu8ygQcG8RlWvxV3Z+biT0fOi2MqeOezj055GMKj4uGc Gs8Ryc+8hKDNxbhbaf4cl8PRpqmn4RbUGPU9I4B0gO15iNweSmY22tnn07aq1srAPtb4 TcJ3DyqJ9uQX0CZCmeyWTnnZYCjJ4boLd6fvMLoDyLQU+0vS9X7mai2Hzk9tl9B6jDAj w3Uw== X-Gm-Message-State: AOJu0YxgY76G6QBUPUiS9kV821bR5lVE2kcF0OVLp4FRbTq93cLDNCIw FM5/pHBfTVnG9i2kwZ4Ki8BK+KVUkNb8du4flhTWxqLeboFCM6vg X-Google-Smtp-Source: AGHT+IFCX+vDQ/YbtVGqndvi6pHEz9tPTwaOXJJtEFw/X1qF3eOwl8ztU6BkzZ+KIsYWBYreo3Rs5KY4R7eJHjeY+RE= X-Received: by 2002:a50:d59c:0:b0:54c:7235:92a0 with SMTP id v28-20020a50d59c000000b0054c723592a0mr18266edi.43.1701993559209; Thu, 07 Dec 2023 15:59:19 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> In-Reply-To: <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> From: Warner Losh Date: Thu, 7 Dec 2023 16:59:08 -0700 Message-ID: Subject: Re: nvme timeout issues with hardware and bhyve vm's To: Tomoaki AOKI Cc: freebsd-current@freebsd.org Content-Type: multipart/alternative; boundary="00000000000081ccc6060bf44134" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmWTN45nCz3fMj --00000000000081ccc6060bf44134 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Dec 7, 2023 at 4:09=E2=80=AFPM Tomoaki AOKI wrote: > On Thu, 7 Dec 2023 14:38:37 -0800 > Pete Wright wrote: > > > > > > > On 10/13/23 7:34 PM, Warner Losh wrote: > > > > > > > > > > > the messages i posted in the start of the thread are from the VM > itself > > > (13.2-RELEASE). The zpool on the hypervisor (13.2-RELEASE) showe= d > no > > > such issues. > > > > > > Based on your comment about the improvements in 14 I'll focus my > > > efforts > > > on my workstation, it seemed to happen regularly so hopefully i c= an > > > find > > > a repo case. > > > > > > > > > Let me now if you see similar messages in stable/14. I think I've > fixed > > > all the > > > issues with timeouts, though you shouldn't ever seem them in a vm set= up > > > unless something else weird is going on. > > > > > > > > > Hi Warner, just resurfacing this thread because I've had a few lockups > > on my workstation running 14.0-STABLE. I was able to capture a photo o= f > > the hang and this seems to be the most important line: > > > > nvme0: Resetting controller due to a timeout and possible hot unplug. > > > > When I scan the device after reboot I don't see any errors, but if ther= e > > is a particular thing I should check via nvmecontrol please let me know= . > > Also, since it mentions possible hot unplug I wonder if this is > > hardware/firmware related to my system? > > > > Anyway, haven't found a repro case yet but it has locked up a few times > > the past two weeks. > > > > -pete > > > > > > -- > > Pete Wright > > pete@nomadlogic.org > > If I myself encounter this kind of problem ON BARE METAL HARDWARE, > I would usually suspect > > *Overheating caused hang of NVMe controller or PCI bridge on SSD, or > Yes. Most drive's firmware when it overheats resets. There might be something that the pci code can do when this happens to retrain the link, reprogram the config registers, etc. > *Unstable physical connection (bad contact) > Yea, hot plug controller is required for this, but this will be bouncing. Warner --00000000000081ccc6060bf44134 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Dec 7, 2023 at 4:09=E2=80=AFP= M Tomoaki AOKI <junchoon@de= c.sakura.ne.jp> wrote:
On Thu, 7 Dec 2023 14:38:37 -0800
Pete Wright <pe= te@nomadlogic.org> wrote:

>
>
> On 10/13/23 7:34 PM, Warner Losh wrote:
> >
>
> >
> >=C2=A0 =C2=A0 =C2=A0the messages i posted in the start of the thre= ad are from the VM itself
> >=C2=A0 =C2=A0 =C2=A0(13.2-RELEASE).=C2=A0 The zpool on the hypervi= sor (13.2-RELEASE) showed no
> >=C2=A0 =C2=A0 =C2=A0such issues.
> >
> >=C2=A0 =C2=A0 =C2=A0Based on your comment about the improvements i= n 14 I'll focus my
> >=C2=A0 =C2=A0 =C2=A0efforts
> >=C2=A0 =C2=A0 =C2=A0on my workstation, it seemed to happen regular= ly so hopefully i can
> >=C2=A0 =C2=A0 =C2=A0find
> >=C2=A0 =C2=A0 =C2=A0a repo case.
> >
> >
> > Let me now if you see similar messages in stable/14. I think I= 9;ve fixed
> > all the
> > issues with timeouts, though you shouldn't ever seem them in = a vm setup
> > unless something else weird is going on.
> >
>
>
> Hi Warner, just resurfacing this thread because I've had a few loc= kups
> on my workstation running 14.0-STABLE.=C2=A0 I was able to capture a p= hoto of
> the hang and this seems to be the most important line:
>
> nvme0: Resetting controller due to a timeout and possible hot unplug.<= br> >
> When I scan the device after reboot I don't see any errors, but if= there
> is a particular thing I should check via nvmecontrol please let me kno= w.
>=C2=A0 =C2=A0Also, since it mentions possible hot unplug I wonder if th= is is
> hardware/firmware related to my system?
>
> Anyway, haven't found a repro case yet but it has locked up a few = times
> the past two weeks.
>
> -pete
>
>
> --
> Pete Wright
> pete@nomadlog= ic.org

If I myself encounter this kind of problem ON BARE METAL HARDWARE,
I would usually suspect

=C2=A0*Overheating caused hang of NVMe controller or PCI bridge on SSD, or<= br>

Yes. Most drive's firmware when it = overheats resets. There might be something
that the pci code can = do when this happens to retrain the link, reprogram the
config re= gisters, etc.
=C2=A0
=C2=A0*Unstable physical connection (bad contact)

=
Yea, hot plug controller is required for this, but this will be = bouncing.

Warner
--00000000000081ccc6060bf44134-- From nobody Fri Dec 8 00:07:39 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmWgF423Zz53rFM for ; Fri, 8 Dec 2023 00:07:53 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmWgD4fPTz4BCD for ; Fri, 8 Dec 2023 00:07:52 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1d04c097e34so12588495ad.0 for ; Thu, 07 Dec 2023 16:07:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20230601.gappssmtp.com; s=20230601; t=1701994071; x=1702598871; darn=freebsd.org; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=m930/4sQ4Ccoomm8xO5S6vefcZGatith/TVLHR1dHgQ=; b=crHXtvwG4Qxargs8PxEpH8ikwTYotEo8cjVgd9qXV2XogJZXm9OyK+s3/UWTjfNgU9 WFym+1zwbt5d0T0auHne7rpwgNFaVAYSmUqT5gNkZcJI7mkwNgIHUeTfbPXh6KvUjRTU PZWazKzqVtgarsNiYh8SONcblsXi0g2NLXfSRnzb6t6TeR9hGnyxRUmAZLDpIaPvTe3K MGOEx9mY4BlJAX93RjS4Nuui8MRwSSj2L8W6otVIFQ1bUYa04qPT+oQSQa0uUQjvV7Py TnPZzTOZHXBbJvZ9gMtMgVhYSBXB26kjvg+Y0n9mkcjg5E2hJVb0R7WQRML20uijxjd4 /90Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701994071; x=1702598871; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m930/4sQ4Ccoomm8xO5S6vefcZGatith/TVLHR1dHgQ=; b=Jv6f1bNowRmklOrwFWz4rZFKTMaxcCWjdS0QDoAnbjGXtWGe1JlKCJ0xwvKmfnCivv JoIbRWAx+pUNEfczhCf5YDDrMUJLyb6uCGdoohg0506sC+wvweku9nKi4bsE5Vbcnp1r FzTpmzE93rZzJSzWhRhPoyX1IPiNfFmX1KQfQcbT1kng/tBVCXPrVa5ipK99cv/hiUVb KZf/RmqPho14EtMTJlB0BMA8Fr8p/3mWk03AJcORCGKmCpSpzqN/p1xPTwV+6H+nGkH1 wYiMlmQKHfIIOUNL9WBuhUhTnY6rZLfXUCRlgICtSHe7lmH8kCrCmOxsZNSlpGJnaEVU VZvA== X-Gm-Message-State: AOJu0Yy7G/C5pv+laZxK7F0FVvQGotFAaLxyfZIvRAaSJ5IZNWnppNt3 ygtO7+6uYSfOlM5mniEcQO4AxQ== X-Google-Smtp-Source: AGHT+IET6aOwL/y7EuYhun6rSdBzQksBIohAlV86KmK9/P96T0vp+SXz6ibs3ZK/zttkvtS4ujR5CQ== X-Received: by 2002:a17:902:d342:b0:1cf:fc91:e4d0 with SMTP id l2-20020a170902d34200b001cffc91e4d0mr68769plk.16.1701994071005; Thu, 07 Dec 2023 16:07:51 -0800 (PST) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id q19-20020a170902bd9300b001cc2ebd2c2csm384426pls.256.2023.12.07.16.07.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Dec 2023 16:07:50 -0800 (PST) Content-Type: text/plain; charset=utf-8 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: Re: nvme timeout issues with hardware and bhyve vm's From: Bakul Shah In-Reply-To: Date: Thu, 7 Dec 2023 16:07:39 -0800 Cc: Tomoaki AOKI , freebsd-current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <10FD2FC6-1F39-4F7D-8BA8-976ADC0AE37A@iitbombay.org> References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> To: Warner Losh X-Mailer: Apple Mail (2.3774.200.91.1.1) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmWgD4fPTz4BCD On Dec 7, 2023, at 3:59=E2=80=AFPM, Warner Losh wrote: >=20 >=20 > *Overheating caused hang of NVMe controller or PCI bridge on SSD, or >=20 > Yes. Most drive's firmware when it overheats resets. There might be = something > that the pci code can do when this happens to retrain the link, = reprogram the > config registers, etc. How quickly can the device heat up? Can it be queried frequently enough act before it overheats by throttling io? From nobody Fri Dec 8 00:19:12 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmWwc29xFz52sfK for ; Fri, 8 Dec 2023 00:19:28 +0000 (UTC) (envelope-from ctuffli@gmail.com) Received: from mail-vs1-f54.google.com (mail-vs1-f54.google.com [209.85.217.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmWwb41Szz4FLj for ; Fri, 8 Dec 2023 00:19:27 +0000 (UTC) (envelope-from ctuffli@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vs1-f54.google.com with SMTP id ada2fe7eead31-46495e57cdeso345830137.3 for ; Thu, 07 Dec 2023 16:19:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701994764; x=1702599564; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HAVrHdtV1jEq+MCAk+4FhZjIcnz9rKMTam6ZnqeuaJY=; b=tcdpEAHmkNM10HJMaFRZfv3jL2MkgFf/x+4OgUSvVx9IU0U3IT9BmFO/FL0kv+BfGj Q3k4O1hqwxVK6iTYg0F+w3wEJr1O+jAyDyGAe429CoVdA4j9sXhArlufRiY6vHrLv4BH 2mBV95pexAUZpZxHDqnEQjblUzKaBxNquLTJldqTP+/MW5g7e7m2IgGyq7usiFCxnu86 dh6TBeFHhmSKQne/RutvI520d+Bppb5FajbZWv6aSYMAN5yCFfKM46pwJFQ+/xsULQ61 mquIKpWJ6KZZJ5Q96tbFKSFFKcz9Z2oTfwELD5t7nO132tX6FSOLSaepABp9hlgj9QHS 9lyg== X-Gm-Message-State: AOJu0Yx1NRIaoYTf1Ey51WGtxYC8+VCBsFdn6ZkDUGSrJrsiHc9QiTwo rMz8brbyJzg67IagoGcVF2+6WYTtYTo= X-Google-Smtp-Source: AGHT+IFrv8OOwBRqBpyqw61GCd2HfCWaNGLLcvwmJ9HrDSPwIxwlvULf8Uun15XXAs83LymKr4bszA== X-Received: by 2002:a1f:cc42:0:b0:495:bd61:a184 with SMTP id c63-20020a1fcc42000000b00495bd61a184mr1989096vkg.2.1701994764262; Thu, 07 Dec 2023 16:19:24 -0800 (PST) Received: from mail-vs1-f53.google.com (mail-vs1-f53.google.com. [209.85.217.53]) by smtp.gmail.com with ESMTPSA id o189-20020a1f73c6000000b004b2bf0a3e95sm84190vkc.31.2023.12.07.16.19.23 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 07 Dec 2023 16:19:24 -0800 (PST) Received: by mail-vs1-f53.google.com with SMTP id ada2fe7eead31-46495e57cdeso345823137.3 for ; Thu, 07 Dec 2023 16:19:23 -0800 (PST) X-Received: by 2002:a1f:6ecb:0:b0:4b3:da:e36f with SMTP id j194-20020a1f6ecb000000b004b300dae36fmr2227160vkc.9.1701994763577; Thu, 07 Dec 2023 16:19:23 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> In-Reply-To: From: Chuck Tuffli Date: Thu, 7 Dec 2023 16:19:12 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: nvme timeout issues with hardware and bhyve vm's To: Pete Wright Cc: Warner Losh , FreeBSD Current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmWwb41Szz4FLj On Thu, Dec 7, 2023 at 2:39=E2=80=AFPM Pete Wright wr= ote: ... > Hi Warner, just resurfacing this thread because I've had a few lockups > on my workstation running 14.0-STABLE. I was able to capture a photo of > the hang and this seems to be the most important line: > > nvme0: Resetting controller due to a timeout and possible hot unplug. > > When I scan the device after reboot I don't see any errors, but if there > is a particular thing I should check via nvmecontrol please let me know. > Also, since it mentions possible hot unplug I wonder if this is > hardware/firmware related to my system? Does the device support Persistent Log pages (LID=3D0x0d)? If so, it might be interesting to dump those. --chuck From nobody Fri Dec 8 01:02:57 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmXv336Mhz52x26 for ; Fri, 8 Dec 2023 01:03:11 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmXv31QQGz4JyC for ; Fri, 8 Dec 2023 01:03:11 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-f53.google.com with SMTP id 2adb3069b0e04-50bee606265so1514268e87.2 for ; Thu, 07 Dec 2023 17:03:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701997389; x=1702602189; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=6nyC/tcUF4wO4trQqSXTC1MoLSqXCK8svkcBURynzjo=; b=EvlDKtvT+oa8sdx+X8aAu5VbUW6hGLN7+wC+T2aQJ7FXxgyrJtPaB9LJTiWIMiIC1/ 7GsoS/8MxeVJmF4mM+akhtmWipTZk1Z3JQM3Y3zgJFeNMr/Ap9rIyk5A/7HK9NYMGWGO SPA7+5ZAx1rE94yr2MmHYH0OeWTFuVtnpbS9++ghBG7o5Vp8LmNnSwQlO7Kwjf/S5oia +r9815+B9MY0ZV3PzQz6p5z2x1r3uCYTXi31xQ8ibMXRbv+XyRgKY6fmkIlwjBYW3RP4 rDU6p7Gs3iBG3ZuuAMXWLS+pnkFFB+VqabOWuMLDS7+8VPwZIolDAYeVg7MK+4v/El7r Q+YA== X-Gm-Message-State: AOJu0YzOg2KlP9BlpJUl4ok+8aLoaaUweSk6gioDS4zFYjuHvwHrXWDU xFXg7PoRE/PaPw5BeyKmVL87dDk1HZdfibUdroVXGtVbNHfb55h3eSU= X-Google-Smtp-Source: AGHT+IGmNSwdA0Bl4msa9a34LkYCGgTPEIrwqkrBjNOgivdhR0OsTm/lGYD8HhnBPB9pWk5NISuIssYVIHpJIYx1yd0= X-Received: by 2002:ac2:46ed:0:b0:50c:e34:aefb with SMTP id q13-20020ac246ed000000b0050c0e34aefbmr1916974lfo.28.1701997388887; Thu, 07 Dec 2023 17:03:08 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> <10FD2FC6-1F39-4F7D-8BA8-976ADC0AE37A@iitbombay.org> In-Reply-To: <10FD2FC6-1F39-4F7D-8BA8-976ADC0AE37A@iitbombay.org> From: Maxim Sobolev Date: Thu, 7 Dec 2023 17:02:57 -0800 Message-ID: Subject: Re: nvme timeout issues with hardware and bhyve vm's To: Bakul Shah Cc: Warner Losh , Tomoaki AOKI , FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000c61723060bf525a2" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmXv31QQGz4JyC --000000000000c61723060bf525a2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable How quickly it heats up depends on lots of factors. Usually those devices burn some 3-7 watts per stick at 100% load, so maybe this would give you some idea. At least some of them support several toggleable performance modes, which use throttling internally to limit power consumption to a certain level (man nvmecontril). It helped me recently to make a system stable, which otherwise would hang with timeout after reaching 70-75C until I got the chance to take it apart and attach a heatsinks to the nvmes. Once the temperature dropped to <=3D 50C the drives become 100% stable. -Max On Thu, Dec 7, 2023, 4:07=E2=80=AFPM Bakul Shah wrote= : > On Dec 7, 2023, at 3:59=E2=80=AFPM, Warner Losh wrote: > > > > > > *Overheating caused hang of NVMe controller or PCI bridge on SSD, or > > > > Yes. Most drive's firmware when it overheats resets. There might be > something > > that the pci code can do when this happens to retrain the link, > reprogram the > > config registers, etc. > > How quickly can the device heat up? Can it be queried frequently > enough act before it overheats by throttling io? > > > > > --000000000000c61723060bf525a2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
How quickly it heats up depends on lots of factors. Usual= ly those devices burn some 3-7 watts per stick at 100% load, so maybe this = would give you some idea. At least some of them support several toggleable = performance modes, which use throttling internally to limit power consumpti= on to a certain level (man nvmecontril). It helped me recently to make a sy= stem stable, which otherwise would hang with timeout after reaching 70-75C = until I got the chance to take it apart and attach a heatsinks to the nvmes= . Once the temperature dropped to <=3D 50C the drives become 100% stable= .

-Max

On Thu, Dec 7, 202= 3, 4:07=E2=80=AFPM Bakul Shah <ba= kul@iitbombay.org> wrote:
On= Dec 7, 2023, at 3:59=E2=80=AFPM, Warner Losh <imp@bsdimp.com> wrote:=
>
>
>=C2=A0 *Overheating caused hang of NVMe controller or PCI bridge on SSD= , or
>
> Yes. Most drive's firmware when it overheats resets. There might b= e something
> that the pci code can do when this happens to retrain the link, reprog= ram the
> config registers, etc.

How quickly can the device heat up? Can it be queried frequently
enough act before it overheats by throttling io?




--000000000000c61723060bf525a2-- From nobody Fri Dec 8 01:33:02 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SmYYn0tZvz530gh for ; Fri, 8 Dec 2023 01:33:17 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SmYYm6MQLz4Mpj for ; Fri, 8 Dec 2023 01:33:16 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-5c690c3d113so1318522a12.1 for ; Thu, 07 Dec 2023 17:33:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20230601.gappssmtp.com; s=20230601; t=1701999195; x=1702603995; darn=freebsd.org; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=BNZcCFcPFI4jh30wbIZm24FjVocuSoXlnqXsnlmde4w=; b=SFtwObuR3VJ7J6VFjD/xDGMriIg3y2San+F9H4SD+rQha6EZjQTzuiZjJiqKtNf232 3QIMXIRTxwV8xVH4V9m6QgSCuQpyzAsyf7wOW6STolnXuFPbMBziGLjDSLcvhLA1yCog p+XJEsv4tJPwuDcEaUdUkig9NCPCW7FkU38Q5vGDKL+l4P7IV5Pbk8EbMk04BXBsWgn9 S8v1FmxB8VKJcW9i4YA/OTfHupyce/jn2eucsHbzCQsSEuF0DMqy2mPo9bEPW05ooISO 6yOxeKfWq6HflwweEJwOp81SR90JPuAeOXFWHtPilWaZkQp/PqaFbXy6Svo90dtxlEVa 4uIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701999195; x=1702603995; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BNZcCFcPFI4jh30wbIZm24FjVocuSoXlnqXsnlmde4w=; b=IOU5eiT5bV5Lnf1QYfuQQiU+zls18uVo9DhgARnw1Rp4s9hL/aol73NvR4uFBp9VRM BwEirgyeNUtvxpFTbmCQQlXmpC50TkmtLV/QaYRbWWi/h0KbX3dMvDLlKX2Cj2aHvJkt vf66WJVx+ysuREG/pj5nP39Mopcgabx2oWHg6CqsCGkbqxjEuHzVABN9/wBCbK82d2LR dz1sEAbjrwnoe9zuAF0vsTIoYhwDn+dZa1Cv+RArFfvrN8MfZhTe3O0lDsIpH+y+vHHc tExHRqnsOt0luDDoJ/XTE0njtQPkqBIbltYRybIGd7+qKfxE5lcFhazYeORV1oQiL6FN QZnw== X-Gm-Message-State: AOJu0YwC1lawG8k0C4ONcJsrhvX4V1ByOlyLINqvqeEqB/bB7hEXgYpI UwiPeeK4INkVc7Kagb/I51aCEw== X-Google-Smtp-Source: AGHT+IFT6P08OXkrmk0rByXDa+0q6FOKCn+1gjF1v/BLUsMkxtzlIY8C6cNSOKen5sxhMFnPCSRAGw== X-Received: by 2002:a17:90a:df82:b0:286:6cc0:cace with SMTP id p2-20020a17090adf8200b002866cc0cacemr3650921pjv.69.1701999194941; Thu, 07 Dec 2023 17:33:14 -0800 (PST) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id d5-20020a17090ac24500b0028559a67729sm2072203pjx.42.2023.12.07.17.33.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Dec 2023 17:33:14 -0800 (PST) From: Bakul Shah Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_15189078-E915-4290-888A-F62380A8027C" List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\)) Subject: Re: nvme timeout issues with hardware and bhyve vm's Date: Thu, 7 Dec 2023 17:33:02 -0800 In-Reply-To: Cc: Warner Losh , Tomoaki AOKI , FreeBSD Current To: Maxim Sobolev References: <90d3e532-8ea7-4eea-8e31-8c363285a156@nomadlogic.org> <0ad493d5-1c1e-4370-977a-118f46ebd677@nomadlogic.org> <0c4f8149-89dd-4635-a5ed-4766fffd2553@nomadlogic.org> <20231208080929.cfd9fca421fea81d89d2380b@dec.sakura.ne.jp> <10FD2FC6-1F39-4F7D-8BA8-976ADC0AE37A@iitbombay.org> X-Mailer: Apple Mail (2.3774.200.91.1.1) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SmYYm6MQLz4Mpj --Apple-Mail=_15189078-E915-4290-888A-F62380A8027C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Thanks. It may be worth checking the temp periodically and warning the user in = case it is too high (70=C2=BAC+ or something). Even for devices that = allow internal throttling, a user might wish to know whether the device = neads a (better) heatsink. > On Dec 7, 2023, at 5:02=E2=80=AFPM, Maxim Sobolev = wrote: >=20 > How quickly it heats up depends on lots of factors. Usually those = devices burn some 3-7 watts per stick at 100% load, so maybe this would = give you some idea. At least some of them support several toggleable = performance modes, which use throttling internally to limit power = consumption to a certain level (man nvmecontril). It helped me recently = to make a system stable, which otherwise would hang with timeout after = reaching 70-75C until I got the chance to take it apart and attach a = heatsinks to the nvmes. Once the temperature dropped to <=3D 50C the = drives become 100% stable. >=20 > -Max >=20 > On Thu, Dec 7, 2023, 4:07=E2=80=AFPM Bakul Shah > wrote: >> On Dec 7, 2023, at 3:59=E2=80=AFPM, Warner Losh > wrote: >> >=20 >> >=20 >> > *Overheating caused hang of NVMe controller or PCI bridge on SSD, = or >> >=20 >> > Yes. Most drive's firmware when it overheats resets. There might be = something >> > that the pci code can do when this happens to retrain the link, = reprogram the >> > config registers, etc. >>=20 >> How quickly can the device heat up? Can it be queried frequently >> enough act before it overheats by throttling io? >>=20 >>=20 >>=20 >>=20 --Apple-Mail=_15189078-E915-4290-888A-F62380A8027C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Thanks.

It may be worth checking = the temp periodically and warning the user in case it is too high = (70=C2=BAC+ or something). Even for devices that allow internal = throttling, a user might wish to know whether the device neads a = (better) heatsink.

On = Dec 7, 2023, at 5:02=E2=80=AFPM, Maxim Sobolev = <sobomax@freebsd.org> wrote:

How quickly = it heats up depends on lots of factors. Usually those devices burn some = 3-7 watts per stick at 100% load, so maybe this would give you some = idea. At least some of them support several toggleable performance = modes, which use throttling internally to limit power consumption to a = certain level (man nvmecontril). It helped me recently to make a system = stable, which otherwise would hang with timeout after reaching 70-75C = until I got the chance to take it apart and attach a heatsinks to the = nvmes. Once the temperature dropped to <=3D 50C the drives become = 100% stable.

-Max

On Thu, Dec 7, 2023, 4:07=E2=80=AFPM = Bakul Shah <bakul@iitbombay.org> = wrote:
On Dec 7, 2023, at = 3:59=E2=80=AFPM, Warner Losh <imp@bsdimp.com> wrote:
>
>
>  *Overheating caused hang of NVMe controller or PCI bridge on = SSD, or
>
> Yes. Most drive's firmware when it overheats resets. There might be = something
> that the pci code can do when this happens to retrain the link, = reprogram the
> config registers, etc.

How quickly can the device heat up? Can it be queried frequently
enough act before it overheats by throttling io?





= --Apple-Mail=_15189078-E915-4290-888A-F62380A8027C--