From nobody Wed Sep 1 13:10:53 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1F1BF179E28B for ; Wed, 1 Sep 2021 13:11:02 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4H04F16FQ4z3Ldr for ; Wed, 1 Sep 2021 13:11:01 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [129.217.43.37] (kalamos.cs.uni-dortmund.de [129.217.43.37]) (authenticated bits=0) by unimail.uni-dortmund.de (8.17.1/8.17.1) with ESMTPSA id 181DArre022506 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Wed, 1 Sep 2021 15:10:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1630501853; bh=Kl2Ngfb49otDAjAOnin+0dmimRh4DtF4XS18j9mVw7g=; h=To:Cc:References:From:Subject:Date:In-Reply-To; b=dto8G686ZN1M4H1OV8P83OZ27/YhebN2cdIcCtwsnwiPJd7f4JQK2rlTPoEp9XRhq YEH4NUxMBMO2h6Y/6uUULDaEHgmlW2xqg2kXb/bvMStDhmYxVGr+DaDiqZPJaL0rap AcZ7iKzJiFc9RNLZSzNXTA9p+L6PeJFXUQ626sdQ= To: Konstantin Belousov Cc: freebsd-fs , Horst Schirmeier References: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> <380bdcc8-bede-2a64-8e5e-031552231d82@tu-dortmund.de> <46649402-d28a-6f81-f0a8-39180b681f4c@tu-dortmund.de> From: Alexander Lochmann Subject: Re: Various unprotected accesses to buf and vnode Message-ID: Date: Wed, 1 Sep 2021 15:10:53 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="04TxWz7gO8SglOxkkiEWFPFYUunGc16dc" X-Rspamd-Queue-Id: 4H04F16FQ4z3Ldr X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --04TxWz7gO8SglOxkkiEWFPFYUunGc16dc Content-Type: multipart/mixed; boundary="WfgMYjAh9e85p3j6EnDfzFPMssO4F8VSz"; protected-headers="v1" From: Alexander Lochmann To: Konstantin Belousov Cc: freebsd-fs , Horst Schirmeier Message-ID: Subject: Re: Various unprotected accesses to buf and vnode References: <55f3661e-2173-793e-4834-bbcd79d3d99e@tu-dortmund.de> <380bdcc8-bede-2a64-8e5e-031552231d82@tu-dortmund.de> <46649402-d28a-6f81-f0a8-39180b681f4c@tu-dortmund.de> In-Reply-To: --WfgMYjAh9e85p3j6EnDfzFPMssO4F8VSz Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 31.08.21 16:44, Konstantin Belousov wrote: >> So in all of those call sequences the buffer lock is not acquired. >> However, I'd not rule out that our tooling could be broken as well. > Buffer is locked inside UFS_BALLOC(), which returns it to the ffs_write= () > use. I took a deep dive into our data, and had a closer look at two samples. In both cases the b_lock is not acquired. Since the debug information seems to be damaged, I had to use 'objdump -S' to translate the pc of the unguarded memory access to a source code position. It seems to be vp->v_lasta =3D bp->b_blkno; in https://thasos.cs.tu-dortmund.de/freebsd-lockdoc/lockdoc-v13.0-0.6/source= /sys/kern/vfs_cluster.c#L802. It was release in binsfree() and bq_insert(): https://thasos.cs.tu-dortmund.de/freebsd-lockdoc/latest/source/sys/kern/v= fs_bio.c#L1537 https://thasos.cs.tu-dortmund.de/freebsd-lockdoc/latest/source/sys/kern/v= fs_bio.c#L1977 Right before the entry that records the unlock, there was a memory access recorded including the stracketrace. I assume that memory access belongs to the unlock operation, and translated the stacktrace. For binsfree(): /opt/kernel/freebsd/src/sys/sys/lockdoc.h:104 /opt/kernel/freebsd/src/sys/kern/kern_lock.c:247 (inlined by) /opt/kernel/freebsd/src/sys/kern/kern_lock.c:1408 /opt/kernel/freebsd/src/sys/sys/lockmgr.h:107 (inlined by) /opt/kernel/freebsd/src/sys/kern/vfs_bio.c:1537 /opt/kernel/freebsd/src/sys/kern/vfs_bio.c:2437 /opt/kernel/freebsd/src/sys/kern/vfs_cluster.c:775 /opt/kernel/freebsd/src/sys/ufs/ffs/ffs_vnops.c:894 /opt/kernel/freebsd/obj/lockdoc-kernproc/vnode_if.c:1108 /opt/kernel/freebsd/obj/lockdoc-kernproc/./vnode_if.h:569 (inlined by) /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1093 /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1158 /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1276 /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1398 /opt/kernel/freebsd/src/sys/sys/file.h:327 (inlined by) /opt/kernel/freebsd/src/sys/kern/sys_generic.c:564 /opt/kernel/freebsd/src/sys/kern/sys_generic.c:491 /opt/kernel/freebsd/src/sys/i386/i386/../../kern/subr_syscall.c:189 For bq_insert(): /opt/kernel/freebsd/src/sys/sys/lockdoc.h:104 /opt/kernel/freebsd/src/sys/kern/kern_lock.c:247 (inlined by) /opt/kernel/freebsd/src/sys/kern/kern_lock.c:1408 /opt/kernel/freebsd/src/sys/sys/lockmgr.h:107 (inlined by) /opt/kernel/freebsd/src/sys/kern/vfs_bio.c:1977 /opt/kernel/freebsd/src/sys/kern/vfs_bio.c:1552 /opt/kernel/freebsd/src/sys/kern/vfs_bio.c:2437 /opt/kernel/freebsd/src/sys/kern/vfs_cluster.c:775 /opt/kernel/freebsd/src/sys/ufs/ffs/ffs_vnops.c:894 /opt/kernel/freebsd/obj/lockdoc-kernproc/vnode_if.c:1108 /opt/kernel/freebsd/obj/lockdoc-kernproc/./vnode_if.h:569 (inlined by) /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1093 /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1158 /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1276 /opt/kernel/freebsd/src/sys/kern/vfs_vnops.c:1398 /opt/kernel/freebsd/src/sys/sys/file.h:327 (inlined by) /opt/kernel/freebsd/src/sys/kern/sys_generic.c:564 /opt/kernel/freebsd/src/sys/kern/sys_generic.c:491 /opt/kernel/freebsd/src/sys/i386/i386/../../kern/subr_syscall.c:189 > Read e.g. sys/ufs/ufs/inode.h gerald comment above struct inode definit= ion. > It provides more detailed exposure. Aaah. Thx. This is about the struct inode. So I assume it also applies for a vnode belonging to an inode. Am I right?> Vnode lock is a lock obtained with vn_lock(). It is up to filesystem > to implement VOP_LOCK() which locks the vnode. >=20 > Default VOP_LOCK() locks vp->v_vnlock, which again by default points > to &vp->v_lock. >=20 > There are several special cases. For instance, for FFS and snapshot vn= odes, > v_vnlock points to the snapdata->sn_lock (snapdata is unique per FFS mo= unt). > For nullfs non-reclaimed vnodes, v_vnlock points to the lower vnode loc= k. >=20 Thx! Is this written down somewhere? --=20 Technische Universit=C3=A4t Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al --WfgMYjAh9e85p3j6EnDfzFPMssO4F8VSz-- --04TxWz7gO8SglOxkkiEWFPFYUunGc16dc Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEElhZsUHzVP0dbkjCRWT7tBbw+9v0FAmEve90FAwAAAAAACgkQWT7tBbw+9v0+ Kw//eUQMK0XnS1gE8LdLDDImOqyN+/Avb+UnWNtia+npz8t3+kVNex25pTb6/yFJyHCbZTLelZYs NAA+uGnuYyAK1D67v5R8KSrKjof39DRbXErD1OfkTnyTm9lrJyqx7mCult+9XfDivAPBEOfN+qBT gACsguJ5jDmaqS4Wb6W8xk2ga/1LbrfNdYmavWHK7Pc2oQdaUapN927t5gREqlasfHN5Cj7Rp2i9 pmHgJmrHjf+X14uyiUGYsoMBBHeRgVf4Ow523PTHv3hO0ZFxbuam9AZxoM+48dowWRoVvAsbIGX0 ebKdAMZ84aPATOmdypK5D7GJajuMwM718Bg4oBYjuH/jrriBS4fRIn6hq9pDFsr4J5dQaErAi1CH YTxQ+PlRFok6NUMOQs0bXFNK2cTD5UR2u/Nrt/DVmZn2rWiFviT+BD8WV6FV7eYvzqii5ReKiT+7 0Wa4y1TWfYEM1qDWiA7B5NOlNNp/Bxn8NpTL9r2KrQrR/uoXADSN8lTLP0JcSGsN0GjRgVJ0AN6P D3MNsgqyQ3MNyb3QCbE6npZjR0SquGoqvITZvSewpzpycb7Z/0U3UBTNxiOFSgdyVmK0zEMN6CAx ZVqqx4mZ3wA5lw84AkN2TeQyFoFvzWCGSxI2nC9Ld594EU3O5hJxtvOIaa2NYOY1KJ1wKxrYT7zm b9Y= =z0tu -----END PGP SIGNATURE----- --04TxWz7gO8SglOxkkiEWFPFYUunGc16dc--