From owner-freebsd-fs@freebsd.org Sat Mar 13 12:59:26 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4AFA5576BEB for ; Sat, 13 Mar 2021 12:59:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DyN705MCjz4r8w for ; Sat, 13 Mar 2021 12:59:24 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 12DCxG3f041777 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 13 Mar 2021 14:59:19 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 12DCxG3f041777 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 12DCxGfR041776; Sat, 13 Mar 2021 14:59:16 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 13 Mar 2021 14:59:16 +0200 From: Konstantin Belousov To: Alexander Lochmann Cc: freebsd-fs@freebsd.org Subject: Re: [RFC] Understanding the locking of struct buf Message-ID: References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DyN705MCjz4r8w X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.98 / 15.00]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-0.98)[-0.979]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_SOFTFAIL(0.00)[~all:c]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.999]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2021 12:59:26 -0000 On Sat, Mar 13, 2021 at 01:03:59PM +0100, Alexander Lochmann wrote: > > > On 13.03.21 04:30, Konstantin Belousov wrote: > >> E.g. any read is permitted without a lock being held. > >> Can b_bcount, for example, be read without a lock? > > Sure you can read it without lock. Question is, what do you intent to do > > with this information. > We're performing lock analysis on the FreeBSD kernel, and I want > understand what kind of general assumptions are made. > In the Linux kernel, for example, every word-sized value is considered > to be read without a lock if consistency doesn't matter. What consistency? If you are talking about multithreading memory model as expected by the FreeBSD kernel, look at atomic(9). It has assumptions like atomicity of naturally-aligned word-sized integer accesses written out explicitly. > In FreeBSD, do I have to use lock X in any case except Y and Z? > Or is it the other way round: Do I need no lock at all except for case X > and Y? I do not understand this question(?). > > > Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr > > (kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In > > particular, it does not report unlock of such lock to witness. > First of all, I want to understand how things in FreeBSD work. > >From what I understand now: When setting up an asynchronous IO request, > buf.b_lock is taken by thread X. Later on LK_KERNPROC is used to hand > over the ownership to the kernel. The lock itself is still acquired. The lock is acquired in the sense that nobody else can take the buffer' lock until the call to lockmgr(LK_RELEASE). But from this point, there is no thread owning the lock. Consider that the lock was converted to the 1-counting semaphore. > The completion of the IO job is performed in the kernel's context, which > releases buf.b_lock in brelse(). The completion for async IO is performed in the context of some other thread, typically either geom io up thread, or direct completion thread of some disk driver. This is why this somewhat strange LK_KERNPROC business is needed at all. For sync IO, that thread only signals original thread that io completed. In this case, no LK_KERNPROC trick is performed. > So there is no explicit lock call in the latter context, is it? No lock call, but there is an unlock. > > However, I think there is indeed a call to WITNESS_UNLOCK() in > __lockmgr_disown(): > https://github.com/freebsd/freebsd-src/blob/main/sys/kern/kern_lock.c#L1641 > > For the following stack trace, we observe several writes to buf.b_bcount > without any lock held. > xpt_done_td > xpt_done_process > adadone > g_disk_done > g_io_devliver > g_std_done > g_io_deliver > g_std_done > g_io_deliver > g_vfs_done > bufdone > ffs_backgroundwritedone > bufdine > brelse Assuming you wrote the stack bottom-up, this is exactly what I wrote above: xpt_done_td is CAM IO completion thread, and it performs actions after hw informed that io request (bio) was completed. > allocbuf When brelse() notes that buffer was marked as 'no cache', it demolishes the buffer right after async io finishes. Perhaps this is the case that you observed. > > There are several reasons for those observations: > a) Due to the call to WITNESS_UNLOCK() in __lockmgr_disown(), which we > instrumented, our approach assumes no lock is held. > b) Our instrumentation missed a lock call. > c) Down the path describes above, no locks are held at all. > d) Something else.... > Can you pls shed some light on our observation? > > - Alex > > -- > Technische Universität Dortmund > Alexander Lochmann PGP key: 0xBC3EF6FD > Otto-Hahn-Str. 16 phone: +49.231.7556141 > D-44227 Dortmund fax: +49.231.7556116 > http://ess.cs.tu-dortmund.de/Staff/al