From owner-freebsd-fs@freebsd.org Mon Apr 19 20:06:42 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9EF8A5FDDEA for ; Mon, 19 Apr 2021 20:06:42 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FPHry0YZZz4p0S for ; Mon, 19 Apr 2021 20:06:41 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [192.168.111.113] (p4fd97aed.dip0.t-ipconnect.de [79.217.122.237]) (authenticated bits=0) by unimail.uni-dortmund.de (8.16.1/8.16.1) with ESMTPSA id 13JK6e2V023897 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Mon, 19 Apr 2021 22:06:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1618862800; bh=EiOUGVOSRtoeWmnoR/ZuKrIdW/FlQiTT1khU4bpa9xs=; h=To:Cc:References:From:Subject:Date:In-Reply-To; b=P0DuIC4xu69KgY+rsIhLdSLoKvlhL9WGgNRQNFmhNwDSDodBR33xVAEj++qKrDVj9 dzujcrJBLp+Iz9uzt9GZjVBpeM5ONwdJNzGeMGe72Cxl7cM61ZcefKX1+NN3NXude0 yMnZh/eTSQJ291q8jbsowhes0pNzQIcKBPjk5SBA= To: Konstantin Belousov Cc: freebsd-fs@freebsd.org References: <792c8a3d-8ea6-073f-3fda-b3eb793ef2b9@tu-dortmund.de> <4ade0f5d-d4f4-616a-b198-fc58f947070d@tu-dortmund.de> From: Alexander Lochmann Subject: Re: [struct buf] Unlocked access to b_vflags? Message-ID: <6f370c67-f3ba-00ce-5da7-7ee400a4180b@tu-dortmund.de> Date: Mon, 19 Apr 2021 22:06:40 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="HFWgVQWvOw8vw2RbNT5bdXY3emhVZWJKN" X-Rspamd-Queue-Id: 4FPHry0YZZz4p0S X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tu-dortmund.de header.s=unimail header.b=P0DuIC4x; dmarc=none; spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de designates 129.217.128.51 as permitted sender) smtp.mailfrom=alexander.lochmann@tu-dortmund.de X-Spamd-Result: default: False [-6.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24:c]; HAS_ATTACHMENT(0.00)[]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RWL_MAILSPIKE_GOOD(0.00)[129.217.128.51:from]; DKIM_TRACE(0.00)[tu-dortmund.de:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[79.217.122.237:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[tu-dortmund.de:s=unimail]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; DMARC_NA(0.00)[tu-dortmund.de]; DWL_DNSWL_LOW(-1.00)[tu-dortmund.de:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Apr 2021 20:06:42 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --HFWgVQWvOw8vw2RbNT5bdXY3emhVZWJKN Content-Type: multipart/mixed; boundary="50Rj90fW8Fc0EkjBE07aP6mNyo9ZDIM1h"; protected-headers="v1" From: Alexander Lochmann To: Konstantin Belousov Cc: freebsd-fs@freebsd.org Message-ID: <6f370c67-f3ba-00ce-5da7-7ee400a4180b@tu-dortmund.de> Subject: Re: [struct buf] Unlocked access to b_vflags? References: <792c8a3d-8ea6-073f-3fda-b3eb793ef2b9@tu-dortmund.de> <4ade0f5d-d4f4-616a-b198-fc58f947070d@tu-dortmund.de> In-Reply-To: --50Rj90fW8Fc0EkjBE07aP6mNyo9ZDIM1h Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: de-DE Content-Transfer-Encoding: quoted-printable On 19.04.21 22:00, Konstantin Belousov wrote: > On Mon, Apr 19, 2021 at 09:43:47PM +0200, Alexander Lochmann wrote: >> Out of curiosity: What's the path from function entry to the access in=20 line >> 759? >> The lock is acquired upon function entry in line 7197, and never relea= sed >> afterwards (except for the lines 7212 and 7217). > The bufobj mutex is interlocked with the buffer locks. Basically, each= > time LK_INTERLOCK is used, the bo_lock is dropped. Is this done by BUF_LOCK() in line 7233? Is the interlocked behavior done automatically, or just because both=20 LK_INTERLOCK and BO_LOCKPTR(bo) are specified? > > Or do you ask about something else? No. I think that's it. - Alex >=20 >> >> - Alex >> >> On 13.04.21 12:25, Konstantin Belousov wrote: >>> On Mon, Apr 12, 2021 at 11:19:05PM +0200, Alexander Lochmann wrote: >>>> Hi folks, >>>> >>>> I'm was digging through our data set when I encountered a strange si= tuation: >>>> According to the code in trunc_dependencies() in sys/ufs/ffs/ffs_sof= tdep.c, >>>> the bo_lock should be held. At least that's how I read the code. >>>> However, we see several thousands of accesses to b_vflags without th= e >>>> bo_lock held. >>>> At least the own b_lock is acquired. >>>> The access happens in line 7549: bp->b_vflags |=3D BV_SCANNED; [1] >>>> Can you please shed some light on this situation? >>>> Is the b_lock sufficeint, and somehow overrules the bo_lock? >>>> Am I missing something? >>> I think you found a valid race. There is one more place where BV_SCA= NNED >>> was manipulated without owning bufobj lock. Patch below should fix b= oth. >>> >>> commit a678470b1307542c5a46b930c119b2358863e0d2 >>> Author: Konstantin Belousov >>> Date: Tue Apr 13 13:22:56 2021 +0300 >>> >>> b_vflags update requries bufobj lock >>> Reported by: Alexander Lochmann (trunc_dependencies()) >>> >>> diff --git a/sys/ufs/ffs/ffs_softdep.c b/sys/ufs/ffs/ffs_softdep.c >>> index 0091b5dcd3b8..23c0cf6e128b 100644 >>> --- a/sys/ufs/ffs/ffs_softdep.c >>> +++ b/sys/ufs/ffs/ffs_softdep.c >>> @@ -7546,7 +7546,9 @@ trunc_dependencies(ip, freeblks, lastlbn, lasto= ff, flags) >>> BO_LOCK(bo); >>> goto cleanrestart; >>> } >>> + BO_LOCK(bo); >>> bp->b_vflags |=3D BV_SCANNED; >>> + BO_UNLOCK(bo); >>> bremfree(bp); >>> if (blkoff !=3D 0) { >>> allocbuf(bp, blkoff); >>> diff --git a/sys/ufs/ffs/ffs_vnops.c b/sys/ufs/ffs/ffs_vnops.c >>> index dc638595eb7b..05eb19c0ee13 100644 >>> --- a/sys/ufs/ffs/ffs_vnops.c >>> +++ b/sys/ufs/ffs/ffs_vnops.c >>> @@ -321,8 +321,9 @@ ffs_syncvnode(struct vnode *vp, int waitfor, int = flags) >>> if (BUF_LOCK(bp, >>> LK_EXCLUSIVE | LK_SLEEPFAIL | LK_INTERLOCK, >>> BO_LOCKPTR(bo)) !=3D 0) { >>> + BO_LOCK(bo); >>> bp->b_vflags &=3D ~BV_SCANNED; >>> - goto next; >>> + goto next_locked; >>> } >>> } else >>> continue; >>> @@ -385,6 +386,7 @@ ffs_syncvnode(struct vnode *vp, int waitfor, int = flags) >>> * to start from a known point. >>> */ >>> BO_LOCK(bo); >>> +next_locked: >>> nbp =3D TAILQ_FIRST(&bo->bo_dirty.bv_hd); >>> } >>> if (waitfor !=3D MNT_WAIT) { >>> >> >> --=20 >> Technische Universit=C3=A4t Dortmund >> Alexander Lochmann PGP key: 0xBC3EF6FD >> Otto-Hahn-Str. 16 phone: +49.231.7556141 >> D-44227 Dortmund fax: +49.231.7556116 >> http://ess.cs.tu-dortmund.de/Staff/al >> >=20 >=20 >=20 --=20 Technische Universit=C3=A4t Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al --50Rj90fW8Fc0EkjBE07aP6mNyo9ZDIM1h-- --HFWgVQWvOw8vw2RbNT5bdXY3emhVZWJKN Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEElhZsUHzVP0dbkjCRWT7tBbw+9v0FAmB94tAFAwAAAAAACgkQWT7tBbw+9v0N JQ//aeEs/I94nNXCWGFUfEMGlF18dTGS2wAhRozhAs3SargGtMkjPWOD25NwCgZdhwGksnGdcmHY nCTTN3NVlHp801+puXx7Q4UEfWANSn1L1LlEy/XJl+5K811xxS8+qfQoBz+coMdmO1OwbRmQUppB 5hIJjRdHrG+9SIfYlXC31xkVDOMPUpXJpCpC4Nm2mdc7oW0AGDFhiUvZwAE1ZnvdGxYIopIdiTHa kY9lmSIqukw4KNkoF2TmTuBuJNCrIM6i/aC3nJliRapwUPAvwUm54Su8pOkQoOPLBeQXgsvv/xx6 XQw1KXh7d9K1J7/BQ2+u3pTXEd9tr07HvGkR7LG+EYLZ1K9E027wBf7jJHQmgEXzaNMZNII1Bvq2 pFAgyHBQin837jAlFbSv3DLnpQKtLw7Ylg07UjQXZiKgBRFXqgiCpcMG37rFNjqzPVAYDrtLM51k bXw9Fgf3Md76qpg0tr14f2O0ICN5yQ0uphPktuQAF4ZXZsdDSH3+Au8fR9eD9sIkhGAf+Ay0TeY7 fYB0Ossd4cDBWQX3wWReSdkGNeJxLETNnxN7TBLLpcpi8IYVgXHIbyC41Lb5wFXXgaUcu5sWxCOJ NSLqud5WsKHoO9z26wTqhXCGnaq5CJydSgjXfA1QDGh8E/M1li7CYdsmLrKeh8RkTleFrBuK4wec Qs0= =CfoM -----END PGP SIGNATURE----- --HFWgVQWvOw8vw2RbNT5bdXY3emhVZWJKN--