From owner-freebsd-hackers@freebsd.org Mon Nov 21 16:41:17 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57774C4D894 for ; Mon, 21 Nov 2016 16:41:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F3453165D for ; Mon, 21 Nov 2016 16:41:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uALGf9GT005122 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 21 Nov 2016 18:41:09 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uALGf9GT005122 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uALGf9XM005121; Mon, 21 Nov 2016 18:41:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 21 Nov 2016 18:41:09 +0200 From: Konstantin Belousov To: Volker Lendecke Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121164109.GC54029@kib.kiev.ua> References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> <20161121151040.GA54029@kib.kiev.ua> <20161121152542.GA31733@sernet.de> <20161121155823.GB54029@kib.kiev.ua> <20161121161454.GA32128@sernet.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121161454.GA32128@sernet.de> User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 16:41:17 -0000 On Mon, Nov 21, 2016 at 05:14:54PM +0100, Volker Lendecke wrote: > On Mon, Nov 21, 2016 at 05:58:23PM +0200, Konstantin Belousov wrote: > > On Mon, Nov 21, 2016 at 04:25:42PM +0100, Volker Lendecke wrote: > > > On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote: > > > > Please see the libthr(3) man page, in particular, read the RUN-TIME > > > > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent > > > > sysctl. > > > > > > > > Does setting the sysctl to 1 allow your program to run ? > > > > > > Yes, that does make it work. The description says that the umtx vnode > > > is dropped on the last munmap. If I #if 0 the middle mmap, it works, > > > although there is no mmap around anymore. So the description is not > > > 100% accurate I'd say. > > No, not umtx vnode is dropped, but the shared umutexes attached to the > > file' page. > > > > What you observed is the consequence of an implementation detail. It is > > impossible to execute umtx cleanup while dereferencing vm object, due to > > locking issues. An asynchronous task is scheduled to perform the cleanup. > > But when the task is run, it is quite possible that your process is already > > executed second mmap() and pthread_mutex_lock(), creating another reference > > on the umtx data. > > Hmm. If I do a poll(NULL, 0, 60000) between the munmap and mmap > without the intervening mmap, it still works. It's really the > mmap(NULL,0xb0...) that kills it. Yes, because at that time, the cleanup task already completed. So the mutex is auto-reinited with default attributes, but as shared. When the cleanup is pending, the mutex off-page data is marked for pending removal, and lookup of that data in pthread_mutex_lock() returns an error. This is unfortunate consequence of the initially limiting ABI which we are trying to preserve still. > > > Look at the Single Unix Specification, particularly to the following > > paragraph in the mmap() description: > > > > The state of synchronization objects such as mutexes, semaphores, > > barriers, and conditional variables placed in shared memory mapped > > with MAP_SHARED becomes undefined when the last region in any process > > containing the synchronization object is unmapped. > > Thanks! Hidden deep in mmap(2)... No hint in any of the pthread calls. > > So -- all of the above discussion becomes irrelevant if I change tdb > such that it keeps the mutex area mmappe'd at least once? Then no GC > will kick in regardless of the sysctl? This would be possible, because > we use mutexes on so-called CLEAR_IF_FIRST databases only. When the last > process closes the db, it will be wiped on the next open. If the file is mmaped, then yes, the mutex must be not destroyed. If it is, then there is a bug in the current implementation.