From owner-freebsd-stable@freebsd.org Mon Feb 22 20:07:26 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9492F54CDDC for ; Mon, 22 Feb 2021 20:07:26 +0000 (UTC) (envelope-from flo@smeets.xyz) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DktWf1KVsz4g5r for ; Mon, 22 Feb 2021 20:07:26 +0000 (UTC) (envelope-from flo@smeets.xyz) Received: by mailman.nyi.freebsd.org (Postfix) id 2B5C354CAFB; Mon, 22 Feb 2021 20:07:26 +0000 (UTC) Delivered-To: stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2B17554CF08 for ; Mon, 22 Feb 2021 20:07:26 +0000 (UTC) (envelope-from flo@smeets.xyz) Received: from mail-out.smeets.xyz (mail-out.smeets.xyz [IPv6:2a01:4f8:10a:3543:ffff:0:25:11]) by mx1.freebsd.org (Postfix) with ESMTP id 4DktWd0C8Qz4gY2 for ; Mon, 22 Feb 2021 20:07:24 +0000 (UTC) (envelope-from flo@smeets.xyz) Received: from mail.smeets.xyz (mail.smeets.xyz [IPv6:2a01:4f8:10a:3543::25:3]) by mail-out.smeets.xyz (Postfix) with ESMTP id 89DCF1C65E for ; Mon, 22 Feb 2021 21:07:16 +0100 (CET) Received: from amavis.smeets.xyz (amavis.smeets.xyz [IPv6:2a01:4f8:10a:3543::aa:4]) by mail.smeets.xyz (Postfix) with ESMTP id 82E9AB0823 for ; Mon, 22 Feb 2021 21:07:16 +0100 (CET) X-Virus-Scanned: amavisd-new at smeets.xyz Received: from mail.smeets.xyz ([IPv6:2a01:4f8:10a:3543::25:3]) by amavis.smeets.xyz (amavis.smeets.xyz [IPv6:2a01:4f8:10a:3543::aa:4]) (amavisd-new, port 10025) with ESMTP id iDdZlnHruqnu for ; Mon, 22 Feb 2021 21:07:16 +0100 (CET) Received: from nibbler.home.lan (p20030006312ead437137e8738847c905.dip0.t-ipconnect.de [IPv6:2003:6:312e:ad43:7137:e873:8847:c905]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by mail.smeets.xyz (Postfix) with ESMTPSA id 3A242B0661 for ; Mon, 22 Feb 2021 21:07:14 +0100 (CET) To: stable@freebsd.org From: Florian Smeets Subject: 13.0-BETA3 frequent file system (UFS) hangs Message-ID: <01190f48-1071-c2ee-0012-9a9ea00530dd@smeets.xyz> Date: Mon, 22 Feb 2021 21:07:12 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="HvcoC49cvNSMQOeD7lsABYlXQjG0kSlsd" X-Rspamd-Queue-Id: 4DktWd0C8Qz4gY2 X-Spamd-Bar: ----- X-Spamd-Result: default: False [-6.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[smeets.xyz:+]; DMARC_POLICY_ALLOW(-0.50)[smeets.xyz,reject]; NEURAL_HAM_SHORT(-1.00)[-0.999]; SIGNED_PGP(-2.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:+,4:+,5:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:4f8:10a:3543:ffff:0:25:11:from]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_COUNT_FIVE(0.00)[5]; FREEFALL_USER(0.00)[flo]; FROM_HAS_DN(0.00)[]; R_DKIM_ALLOW(-0.20)[smeets.xyz:s=dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain,application/pgp-keys]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a01:4f8:10a:3543:ffff:0:25:11:from:127.0.2.255]; MAILMAN_DEST(0.00)[stable] X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Feb 2021 20:07:26 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --HvcoC49cvNSMQOeD7lsABYlXQjG0kSlsd Content-Type: multipart/mixed; boundary="L3qNHQtesxqEj4kt3nMpHWxM9LIO3vJnp"; protected-headers="v1" From: Florian Smeets To: stable@freebsd.org Message-ID: <01190f48-1071-c2ee-0012-9a9ea00530dd@smeets.xyz> Subject: 13.0-BETA3 frequent file system (UFS) hangs --L3qNHQtesxqEj4kt3nMpHWxM9LIO3vJnp Content-Type: multipart/mixed; boundary="------------4768F994E39AE9E194749B4A" Content-Language: en-US This is a multi-part message in MIME format. --------------4768F994E39AE9E194749B4A Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi, on Sunday I upgraded a lightly loaded web server (Nginx+PHP-FPM+MariaDB) = from 12.2-RELEASE-p3 to 13.0-BETA3. The server has been hanging every=20 few hours (between 2-12). Until now the only remedy I found was a reboot.= When the issue appears I can still read from the file system, I was able = to touch a (new) file, but not delete it. load: 0.49 cmd: rm 78200 [biowr] 7.31r 0.00u 0.28s 1% 2192k mi_switch+0xc1 _sleep+0x1cb bwait+0x6e bufwrite+0x206 ffs_update+0x2d0=20 ffs_syncvnode+0x552 softdep_prelink+0x14b ufs_remove+0x85=20 VOP_REMOVE_APV+0x27 kern_funlinkat+0x2d5 sys_unlink+0x28=20 amd64_syscall+0x10c fast_syscall_common+0xf8 The system is UFS only. /dev/da0s1a on / (ufs, local, noatime, journaled soft-updates, writes:=20 sync 19483 async 3361, reads: sync 24501 async 4191, fsid 374666591e63cde= d) For some of the reboots I see the following in the log before rebooting: Feb 21 15:21:57 web01 kernel: Waiting (max 60 seconds) for system=20 process `vnlru' to stop... done Feb 21 15:21:57 web01 kernel: Waiting (max 60 seconds) for system=20 process `syncer' to stop... Feb 21 15:21:57 web01 kernel: Syncing disks, vnodes remaining... 20=20 fsync: giving up on dirty (error =3D 35) 0xfffff80003815b70: type VCHR Feb 21 15:21:57 web01 kernel: usecount 1, writecount 0, refcount=20 1074 seqc users 0 rdev 0xfffff80004c3f000 Feb 21 15:21:57 web01 kernel: hold count flags () Feb 21 15:21:57 web01 kernel: flags () Feb 21 15:21:57 web01 kernel: v_object 0xfffff800030d0c60 ref 0=20 pages 45500 cleanbuf 1071 dirtybuf 1 Feb 21 15:21:57 web01 kernel: lock type mntfs: EXCL by thread=20 0xfffffe00c33fc000 (pid 24, syncer, tid 100097) Feb 21 15:21:57 web01 kernel: 9 5 0 0 done Is this related to the thread "FreeBSD 13.0-BETA2 and slow IO" ? I have a stable/13 kernel on there now, but that also hung once already. Do I need to disable SU/SUJ or put a main kernel on there to get it stabl= e? Thanks, Florian --------------4768F994E39AE9E194749B4A-- --L3qNHQtesxqEj4kt3nMpHWxM9LIO3vJnp-- --HvcoC49cvNSMQOeD7lsABYlXQjG0kSlsd Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEE7LNouHkIv7aRTXJp71uk3NWp88AFAmA0DvAFAwAAAAAACgkQ71uk3NWp88Bi YRAAjp4SSO9veOrtkYhm5SxPVHNO7iyfn5I6+f5eAehyfa5mh3eMngp9McPGUgXFnuREG3ESZWf4 SqUMpGHwA5+m0NG5qcPLIP2pjYYseujMDqMieD8OmWsnUcy5+Tg6j4D84Ry/VmQZCMDHKzTQsF12 9O581F+JTYG34MKiFCNWGpT785XtI51ldFCdAjuEM2PO3zckemMvGnGoNVlaMQDLa5a9COe/z6x+ 6Ccvuz0rfLR5ps1TeKjfWE0I1zVx285FFWbc/2ENbJu0Zq1JR7l8qb360g0J0Rf8bMN5FNbdVtFS gvq3LaGU2eJrq1O9EGoCaLyZ7YTIWzB7Pj7D/LKVJdAUqd7BTIMbSt8XqKWERL1/kwkY/jrqKYB+ mq5MZEPaT5113V+NNGlEPzGLrwero61ufg9y0wAjSxnVJN3MVieCvxlYJ878f2mqiwlOspj0P1JH hMdRDcHBn7oNGMOPMWVjd0/CiGHgZ6/FcT3xbs6/6yVpn0hjBDKx4KXkF5RTP6DjvNLOIKp+c2It eQorfBwCN+rgCY8umE2BJMGJYCPBTTKsQO3V9KWnPhcRvWU5edSZMJ4KmpPqcq91A9J3rkeYmQ6R ocHwOB5a5IuBvRHIVSGBUOJSfK0BibCGjyp4gSk5X47/3u2qelSS5AwB4MC2Zc7HJnhR8bj9OteD oGQ= =g5Vd -----END PGP SIGNATURE----- --HvcoC49cvNSMQOeD7lsABYlXQjG0kSlsd--