From owner-freebsd-fs@freebsd.org Sat Mar 13 12:04:03 2021 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 510A8575B17 for ; Sat, 13 Mar 2021 12:04:03 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "unimail.tu-dortmund.de", Issuer "DFN-Verein Global Issuing CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DyLv64HQ7z4nlP for ; Sat, 13 Mar 2021 12:04:02 +0000 (UTC) (envelope-from alexander.lochmann@tu-dortmund.de) Received: from [192.168.111.103] (p2e513b89.dip0.t-ipconnect.de [46.81.59.137]) (authenticated bits=0) by unimail.uni-dortmund.de (8.16.1/8.16.1) with ESMTPSA id 12DC3xe3027099 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT); Sat, 13 Mar 2021 13:03:59 +0100 (CET) To: Konstantin Belousov Cc: freebsd-fs@freebsd.org References: <49130618-349a-bfc7-6d26-0c3763904dc5@tu-dortmund.de> From: Alexander Lochmann Subject: Re: [RFC] Understanding the locking of struct buf Message-ID: Date: Sat, 13 Mar 2021 13:03:59 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4DyLv64HQ7z4nlP X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of alexander.lochmann@tu-dortmund.de designates 129.217.128.51 as permitted sender) smtp.mailfrom=alexander.lochmann@tu-dortmund.de X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:129.217.128.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tu-dortmund.de]; ARC_NA(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[129.217.128.51:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[129.217.128.51:from]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-0.997]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:129.217.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-fs]; RECEIVED_SPAMHAUS_PBL(0.00)[46.81.59.137:received] X-Mailman-Approved-At: Sat, 13 Mar 2021 18:22:34 +0000 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2021 12:04:03 -0000 On 13.03.21 04:30, Konstantin Belousov wrote: >> E.g. any read is permitted without a lock being held. >> Can b_bcount, for example, be read without a lock? > Sure you can read it without lock. Question is, what do you intent to do > with this information. We're performing lock analysis on the FreeBSD kernel, and I want understand what kind of general assumptions are made. In the Linux kernel, for example, every word-sized value is considered to be read without a lock if consistency doesn't matter. In FreeBSD, do I have to use lock X in any case except Y and Z? Or is it the other way round: Do I need no lock at all except for case X and Y? > Are you reporting a bug or just asking about LK_KERNPROC. Lockmgr > (kern_lock.c)is careful enough to handle LK_KERNPROC owner specially. In > particular, it does not report unlock of such lock to witness. First of all, I want to understand how things in FreeBSD work. >From what I understand now: When setting up an asynchronous IO request, buf.b_lock is taken by thread X. Later on LK_KERNPROC is used to hand over the ownership to the kernel. The lock itself is still acquired. The completion of the IO job is performed in the kernel's context, which releases buf.b_lock in brelse(). So there is no explicit lock call in the latter context, is it? However, I think there is indeed a call to WITNESS_UNLOCK() in __lockmgr_disown(): https://github.com/freebsd/freebsd-src/blob/main/sys/kern/kern_lock.c#L1641 For the following stack trace, we observe several writes to buf.b_bcount without any lock held. xpt_done_td xpt_done_process adadone g_disk_done g_io_devliver g_std_done g_io_deliver g_std_done g_io_deliver g_vfs_done bufdone ffs_backgroundwritedone bufdine brelse allocbuf There are several reasons for those observations: a) Due to the call to WITNESS_UNLOCK() in __lockmgr_disown(), which we instrumented, our approach assumes no lock is held. b) Our instrumentation missed a lock call. c) Down the path describes above, no locks are held at all. d) Something else.... Can you pls shed some light on our observation? - Alex -- Technische Universität Dortmund Alexander Lochmann PGP key: 0xBC3EF6FD Otto-Hahn-Str. 16 phone: +49.231.7556141 D-44227 Dortmund fax: +49.231.7556116 http://ess.cs.tu-dortmund.de/Staff/al