From owner-freebsd-fs@freebsd.org Sun Mar 14 08:35:40 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3BEC5577AAA for ; Sun, 14 Mar 2021 08:35:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DytDC0Hctz3Cwl for ; Sun, 14 Mar 2021 08:35:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 12E8ZPZU019088 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 14 Mar 2021 10:35:28 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 12E8ZPZU019088 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 12E8ZOt8019080; Sun, 14 Mar 2021 10:35:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 14 Mar 2021 10:35:24 +0200 From: Konstantin Belousov To: Alexander Lochmann Cc: freebsd-fs@freebsd.org Subject: Re: [RFC] Understanding the locking of struct buf Message-ID: References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de> <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <48913698-d2e8-9721-ee1a-4828a9265e55@tu-dortmund.de> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4DytDC0Hctz3Cwl X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; R_SPF_SOFTFAIL(0.00)[~all]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-fs]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Mar 2021 08:35:40 -0000 On Sat, Mar 13, 2021 at 05:29:39PM +0100, Alexander Lochmann wrote: > > > On 13.03.21 13:59, Konstantin Belousov wrote: > > What consistency? If you are talking about multithreading memory model > > as expected by the FreeBSD kernel, look at atomic(9). It has assumptions > > like atomicity of naturally-aligned word-sized integer accesses written > > out explicitly. > > > Yeah, I think that's what I meant. Thx! > > In sys/sys/buf.h, it says "All fields are protected by the buffer lock > except those marked [...]". How does this fit together with atomic(9)? > If I had to implement some extension to vfs_bio.c, for example, I would > follow those locking rules without knowing that I may access some fields > without any lock. You cannot write any working code that uses buffer cache, without studying both buffer cache code itself, and examples using it from filesystems. Short (even the whole-screen short) herald comment cannot explain all the nuances that grown in the 40+ years codebase. > Shouldn't that particular piece of documentation be updated? > For example: b_bcount /* w: b_lock, r: no lock needed */ Do you understand the purpose of the locking? What do you intent to do with the read value of b_bcount, if you do not own the buffer? What is you next action with that value? > >> In FreeBSD, do I have to use lock X in any case except Y and Z? > >> Or is it the other way round: Do I need no lock at all except for case X > >> and Y? > > I do not understand this question(?). > > > I'm sry. Maybe my question is just nuts.>> > >>> Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr > >>> (kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In > >>> particular, it does not report unlock of such lock to witness. > >> First of all, I want to understand how things in FreeBSD work. > >> >From what I understand now: When setting up an asynchronous IO request, > >> buf.b_lock is taken by thread X. Later on LK_KERNPROC is used to hand > >> over the ownership to the kernel. The lock itself is still acquired. > > The lock is acquired in the sense that nobody else can take the buffer' > > lock until the call to lockmgr(LK_RELEASE). > So can we safely assume, for our analysis, that the b_lock is still > acquired during bufdone and the other related fns? LK_KERNPROC buffer is logically owned by the io completion thread(s). > But from this point, there > > is no thread owning the lock. Consider that the lock was converted to > > the 1-counting semaphore.I'm not sure. What do you mean bei 1-counting semaphore? > > > >> The completion of the IO job is performed in the kernel's context, which > >> releases buf.b_lock in brelse(). > > The completion for async IO is performed in the context of some other > > thread, typically either geom io up thread, or direct completion thread > > of some disk driver. This is why this somewhat strange LK_KERNPROC > > business is needed at all. > > > > For sync IO, that thread only signals original thread that io completed. > > In this case, no LK_KERNPROC trick is performed. > > > That is clear to me. > >> So there is no explicit lock call in the latter context, is it? > > No lock call, but there is an unlock. > Are you talking about BUF_UNLOCK()? > > > Assuming you wrote the stack bottom-up, this is exactly what I wrote above: > > xpt_done_td is CAM IO completion thread, and it performs actions after hw > > informed that io request (bio) was completed. > > > Thx. > >> allocbuf > > When brelse() notes that buffer was marked as 'no cache', it demolishes > > the buffer right after async io finishes. Perhaps this is the case that > > you observed. > > Yeah, but maybe our approach is just inaccurate. Due to the > WITNESS_UNLOCK() call in __lockmgr_disown(), we assume the b_lock has > already been released. > > - Alex > > -- > Technische Universität Dortmund > Alexander Lochmann PGP key: 0xBC3EF6FD > Otto-Hahn-Str. 16 phone: +49.231.7556141 > D-44227 Dortmund fax: +49.231.7556116 > http://ess.cs.tu-dortmund.de/Staff/al