From owner-freebsd-threads@freebsd.org Wed Dec 23 17:25:41 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F9BFA4FDCC for ; Wed, 23 Dec 2015 17:25:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8C79A1EE4 for ; Wed, 23 Dec 2015 17:25:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 88FFCA4FDCB; Wed, 23 Dec 2015 17:25:41 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E897A4FDC9; Wed, 23 Dec 2015 17:25:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 18A221EE2; Wed, 23 Dec 2015 17:25:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBNHPS51064753 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 23 Dec 2015 19:25:28 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBNHPS51064753 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBNHPSaZ064752; Wed, 23 Dec 2015 19:25:28 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 23 Dec 2015 19:25:28 +0200 From: Konstantin Belousov To: threads@freebsd.org, arch@freebsd.org Subject: libthr shared locks Message-ID: <20151223172528.GT3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 17:25:41 -0000 A well-known limitation of the FreeBSD libthr implementation of the pthread locking objects is the missed support for the process-shared locks. The hardest part of implementing the support is the neccessity of providing ABI compatibility for the current implementation. Right now, the ABI-visible handle to the locks is a single pointer word. As an example which allows to make the description less vague, let consider pthread_mutex_t. It is defined in sys/sys/_pthreadtypes.h as typedef struct pthread_mutex *pthread_mutex_t; The pointer points to the following structure, after the pthread_mutex_init(3) is called struct pthread_mutex { struct umutex m_lock; int m_flags; struct pthread *m_owner; int m_count; int m_spinloops; int m_yieldloops; TAILQ_ENTRY(pthread_mutex) m_qe; }; struct umutex { volatile __lwpid_t m_owner; /* Owner of the mutex */ __uint32_t m_flags; /* Flags of the mutex */ __uint32_t m_ceilings[2]; /* Priority protect ceiling */ __uint32_t m_spare[4]; }; Would the ABI modified to make the pthread_mutex_t large enough to hold struct pthread_mutex, the rest of the implementation of the shared mutex is relatively trivial, if not already done. Changing this ABI is very hard. libthr provides the symbol versioning, which allows to provide compatible shims for the previous ABI variant. But since userspace tends to use the pthread objects in the layouts of the library objects, this causes serious ABI issues when mixing libraries built against different default versions of libthr. My idea to provide the shared locks, while not changing the ABI for libthr, is to use marker pointers to indicate the shared objects. The real struct pthread_mutex, which carries the locking information, is allocated by at the off-page from some anonymous posix shared memory object. The marker is defined as #define THR_PSHARED_PTR \ ((void *)(uintptr_t)((1ULL << (NBBY * sizeof(long) - 1)) | 1)) The bit-pattern is 1000....00001. There are two tricks used: 1. All correctly allocated objects in all supported ABIs are at least word-aligned, so the least-significant bit cannot be set. This should made the THR_PSHARED_PTR pattern unique against non-shared allocations. 2. The high bit is set, which makes the address non-canonical on amd64, causing attempts to dereference the pointer guaranteed to segfault, instead of relying of not having the corresponding page not mapped on the weakly-aligned arches. The majority of the libthr modifications follow the easy pattern where the library must store the THR_PSHARED_PTR upon the initialization of the shared objects, allocate the off-page and initialize the lock there. If a call assumes that the object is already initialized, then the we must not instantiate the off-page. To speed-up the lookup, a cache is kept at the userspace which translates address of locks to the off-page. Note that we can safely ignore possible unmapping of the locks, since correct pthread_* API use assumes the call to pthread_*_destroy() on the end of the object lifecycle. If the lock is remapped in the usermode, then userspace off-page translation cache fails, but kernel returns the same shm for lookup, and we end with two off-page mappings, which is acceptable. Kernel holds a lookup table which translates the (vm_object, offset) pair, obtained by the dereference of the user-space address, into the posix shared memory object. The lifecycle of the shm objects is bound to the existence of the corresponding vm object. Note that lifecycle of the kernel objects does not correspond well to the lifecycle of the vnode vm object. Closed vnode could be recycled by VFS for whatever reasons, and then we would loose the entry in the registry. I am not sure if this is very serious issue, since I suppose that typical use case assumes the anonymous shared memory backing. Right now kernel drops the off-page shm object on the last vnode unmap. Due to backing by the kernel objects, the implementation imposes per-uid limits on the amount of the shared objects created. An issue is that there are no such limits in other implementations. Overhead of the implementation, comparing with the non-process shared locks, is due to the mandatory off-page lookup, which is mostly ammortized by the (read-locked) userspace cache. Also, for each shared lock we get an additional page of memory, which works fine assuming the applications use limited amount of the shared locks. Cost for the non-shared locks is a single memory load for each pthread_* call. Below are the timing results of my implementation on the 4-core sandy against the Fedora 22 glibc, done with the same program on the same hardware (https://www.kib.kiev.ua/kib/pshared/pthread_shared_mtx1.c). [FreeBSD] # time /root/pthread_shared_mtx1-64 iter1 10000000 aiter1 10000000 iter2 10000000 aiter2 10000000 ./pthread_shared_mtx1-64 2.47s user 3.27s system 166% cpu 3.443 total [Fedora] [kostik@sandy tests]$ /usr/bin/time ./pthread_shared_mtx1-linux64 iter1 10000000 aiter1 10000000 iter2 10000000 aiter2 10000000 1.38user 2.46system 0:01.95elapsed 196%CPU (0avgtext+0avgdata 1576maxresident)k 0inputs+0outputs (0major+142minor)pagefaults 0swaps The implementation in the patch https://www.kib.kiev.ua/kib/pshared/pshared.1.patch gives shared mutexes, condvars, rwlocks and barriers. I did some smoke-testing, only on amd64. Not implementated are the robust mutexes. I want to finalize this part of work before implementing robustness, but some restructuring in the patch, which seems to be arbitrary, like the normal/pp queues rework to live in arrays, is a preparation to the robustness feature. The work was sponsored by The FreeBSD Foundation, previous and current versions of idea and previous patch were discussed with John Baldwin and Ed Maste. From owner-freebsd-threads@freebsd.org Wed Dec 23 18:08:32 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D064FA50C79 for ; Wed, 23 Dec 2015 18:08:32 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B625C189A for ; Wed, 23 Dec 2015 18:08:32 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B26DEA50C77; Wed, 23 Dec 2015 18:08:32 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97F87A50C76; Wed, 23 Dec 2015 18:08:32 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5728A1899; Wed, 23 Dec 2015 18:08:32 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBNI3x8L002901; Wed, 23 Dec 2015 13:03:59 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Wed, 23 Dec 2015 13:03:59 -0500 (EST) Date: Wed, 23 Dec 2015 13:03:59 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151223172528.GT3625@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 18:08:33 -0000 On Wed, 23 Dec 2015, Konstantin Belousov wrote: > A well-known limitation of the FreeBSD libthr implementation of the > pthread locking objects is the missed support for the process-shared > locks. The hardest part of implementing the support is the neccessity > of providing ABI compatibility for the current implementation. > > Right now, the ABI-visible handle to the locks is a single pointer > word. As an example which allows to make the description less vague, > let consider pthread_mutex_t. It is defined in > sys/sys/_pthreadtypes.h as > typedef struct pthread_mutex *pthread_mutex_t; > The pointer points to the following structure, after the > pthread_mutex_init(3) is called > struct pthread_mutex { > struct umutex m_lock; > int m_flags; > struct pthread *m_owner; > int m_count; > int m_spinloops; > int m_yieldloops; > TAILQ_ENTRY(pthread_mutex) m_qe; > }; > struct umutex { > volatile __lwpid_t m_owner; /* Owner of the mutex */ > __uint32_t m_flags; /* Flags of the mutex */ > __uint32_t m_ceilings[2]; /* Priority protect ceiling */ > __uint32_t m_spare[4]; > }; > > Would the ABI modified to make the pthread_mutex_t large enough to > hold struct pthread_mutex, the rest of the implementation of the > shared mutex is relatively trivial, if not already done. > > Changing this ABI is very hard. libthr provides the symbol > versioning, which allows to provide compatible shims for the previous > ABI variant. But since userspace tends to use the pthread objects in > the layouts of the library objects, this causes serious ABI issues > when mixing libraries built against different default versions of > libthr. > > My idea to provide the shared locks, while not changing the ABI for > libthr, is to use marker pointers to indicate the shared objects. The > real struct pthread_mutex, which carries the locking information, is > allocated by at the off-page from some anonymous posix shared memory > object. I'd rather just change the pthread_foo lock types to include the space. I really don't like the way libc, rtld, etc have to jump through hoops to initialize the locks. If the lock types included the storage, then they wouldn't have to. My idea was to keep the first storage unit of the lock as a pointer/self-reference, so that it is easy to keep the ABI. You can set the 0 bit in the pointer to indicate whether the lock was the old ABI, then just clear it before using it. And we could hide old ABI implementation compilation behind WITH_FOO to avoid overhead of checking (ptr & 0x1) != 0. I think this is similar to what you've done below. > > The marker is defined as > #define THR_PSHARED_PTR \ > ((void *)(uintptr_t)((1ULL << (NBBY * sizeof(long) - 1)) | 1)) > The bit-pattern is 1000....00001. There are two tricks used: [ ... ] I also don't like to get the kernel involved if it isn't necessary. -- DE From owner-freebsd-threads@freebsd.org Wed Dec 23 18:27:37 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 354A6A4F298 for ; Wed, 23 Dec 2015 18:27:37 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2493A11BD for ; Wed, 23 Dec 2015 18:27:37 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1F704A4F295; Wed, 23 Dec 2015 18:27:37 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1E948A4F293; Wed, 23 Dec 2015 18:27:37 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E0ABF11BB; Wed, 23 Dec 2015 18:27:36 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBNIRZOS022271; Wed, 23 Dec 2015 13:27:35 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Wed, 23 Dec 2015 13:27:35 -0500 (EST) Date: Wed, 23 Dec 2015 13:27:35 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151223172528.GT3625@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 18:27:37 -0000 On Wed, 23 Dec 2015, Konstantin Belousov wrote: [ ... ] > Would the ABI modified to make the pthread_mutex_t large enough to > hold struct pthread_mutex, the rest of the implementation of the > shared mutex is relatively trivial, if not already done. > > Changing this ABI is very hard. libthr provides the symbol > versioning, which allows to provide compatible shims for the previous > ABI variant. But since userspace tends to use the pthread objects in > the layouts of the library objects, this causes serious ABI issues > when mixing libraries built against different default versions of > libthr. I think this is only if the libraries (or apps) pass pthread lock types between them, that one has initialized with one ABI but the other library uses another ABI. We should really exclude this as part of our ABI compatibility. Mixing libraries built with different pthread ABIs should not be a problem as long as they don't directly operate on the other's pthread lock types. There is also our ability to do a library version bump as a last resort. -- DE From owner-freebsd-threads@freebsd.org Wed Dec 23 19:05:27 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7ED16A4FD84 for ; Wed, 23 Dec 2015 19:05:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 60E631395 for ; Wed, 23 Dec 2015 19:05:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5BF8DA4FD82; Wed, 23 Dec 2015 19:05:27 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41499A4FD80; Wed, 23 Dec 2015 19:05:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DFA551392; Wed, 23 Dec 2015 19:05:26 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBNJ5K4l088363 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 23 Dec 2015 21:05:20 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBNJ5K4l088363 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBNJ5JCE088358; Wed, 23 Dec 2015 21:05:19 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 23 Dec 2015 21:05:19 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151223190519.GU3625@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 19:05:27 -0000 On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote: > On Wed, 23 Dec 2015, Konstantin Belousov wrote: > > [ ... ] > > Would the ABI modified to make the pthread_mutex_t large enough to > > hold struct pthread_mutex, the rest of the implementation of the > > shared mutex is relatively trivial, if not already done. > > > > Changing this ABI is very hard. libthr provides the symbol > > versioning, which allows to provide compatible shims for the previous > > ABI variant. But since userspace tends to use the pthread objects in > > the layouts of the library objects, this causes serious ABI issues > > when mixing libraries built against different default versions of > > libthr. > > I think this is only if the libraries (or apps) pass pthread > lock types between them, that one has initialized with one ABI > but the other library uses another ABI. We should really > exclude this as part of our ABI compatibility. Applications commonly embed pthread locks into their objects, including the exposed ABI in the third-party libraries. struct my_object { pthread_mutex_t obj_lock; ... }; Changing the size of the pthread locks causes uncontrolled breakage of the ABI for significant number of third-party code. The issue is similar to the effect of the attempted ino_t expansion to the 64 bit, which changes the struct stat layout and then triggers cascade of the ABI issues. With struct stat, most of it is localized in the src/, which makes handling easier because all of the changed staff is under our control. For libpthread, the approach would cause *big* blow up for ports and for software compiled locally, and we cannot do anything. It would become the flag day, with subtle bugs all over there after the transition on a machine where ABIs are mixed. This is not an issue for e.g. appliance vendors, but IMO is unnaceptable for general-purpose OS. And, why pay the cost of the flag day for the feature that was not present up to today ? > > Mixing libraries built with different pthread ABIs should not > be a problem as long as they don't directly operate on the > other's pthread lock types. > > There is also our ability to do a library version bump as a last > resort. Bumping libthr version would be not enough. It is very much possible to get both libthr.so.3 and libthr.so.4 into the same process. Versions for the symbols must be adjusted to properly bind new and old ABI' consumers. I did evaluated this route, and I am between being very skeptical that it is workable and completely denying the viability. As I pointed out earlier, this causes enormous amount of bugs due to the ABI change in third-party, you cannot mix. Clean recompilation for the new ABI would solve it, but it is not acceptable for FreeBSD pretending to support upgrades. From owner-freebsd-threads@freebsd.org Wed Dec 23 19:48:59 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EADCA50CEB for ; Wed, 23 Dec 2015 19:48:59 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3CBD61D68 for ; Wed, 23 Dec 2015 19:48:59 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 36F88A50CE8; Wed, 23 Dec 2015 19:48:59 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3676BA50CE7; Wed, 23 Dec 2015 19:48:59 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 036211D66; Wed, 23 Dec 2015 19:48:58 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBNJmu0b014361; Wed, 23 Dec 2015 14:48:56 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Wed, 23 Dec 2015 14:48:56 -0500 (EST) Date: Wed, 23 Dec 2015 14:48:56 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151223190519.GU3625@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <20151223190519.GU3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 19:48:59 -0000 On Wed, 23 Dec 2015, Konstantin Belousov wrote: > On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote: >> On Wed, 23 Dec 2015, Konstantin Belousov wrote: >> >> [ ... ] >>> Would the ABI modified to make the pthread_mutex_t large enough to >>> hold struct pthread_mutex, the rest of the implementation of the >>> shared mutex is relatively trivial, if not already done. >>> >>> Changing this ABI is very hard. libthr provides the symbol >>> versioning, which allows to provide compatible shims for the previous >>> ABI variant. But since userspace tends to use the pthread objects in >>> the layouts of the library objects, this causes serious ABI issues >>> when mixing libraries built against different default versions of >>> libthr. >> >> I think this is only if the libraries (or apps) pass pthread >> lock types between them, that one has initialized with one ABI >> but the other library uses another ABI. We should really >> exclude this as part of our ABI compatibility. > Applications commonly embed pthread locks into their objects, including > the exposed ABI in the third-party libraries. > > struct my_object { > pthread_mutex_t obj_lock; > ... > }; > > Changing the size of the pthread locks causes uncontrolled breakage of > the ABI for significant number of third-party code. If the application creates the object itself or allocates storage for it, basically, if it isn't opaque, yes. But we can bump port revisions for affected libraries (probably just searching /usr/local/include/... for pthread_mutex, pthread_cond, etc types to see possible problems. I did a search for the installed ports on my system and found a few that might cause problems. I think we're just putting off the inevitable. Do we not want to change our pthread sync types anyway, to get rid of an extra dereference per lock? To get rid of the hacks in libc, rtld, etc? If the answer is no, we never want to do that, then ok. -- DE From owner-freebsd-threads@freebsd.org Wed Dec 23 20:18:44 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B6FFA4F5BD for ; Wed, 23 Dec 2015 20:18:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 80743199E for ; Wed, 23 Dec 2015 20:18:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 7D4DDA4F5BC; Wed, 23 Dec 2015 20:18:44 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64789A4F5BA; Wed, 23 Dec 2015 20:18:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0E561199D; Wed, 23 Dec 2015 20:18:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBNKIcen006067 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 23 Dec 2015 22:18:38 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBNKIcen006067 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBNKIbPs006066; Wed, 23 Dec 2015 22:18:37 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 23 Dec 2015 22:18:37 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151223201837.GW3625@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <20151223190519.GU3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 20:18:44 -0000 On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote: > On Wed, 23 Dec 2015, Konstantin Belousov wrote: > > > On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote: > >> On Wed, 23 Dec 2015, Konstantin Belousov wrote: > >> > >> [ ... ] > >>> Would the ABI modified to make the pthread_mutex_t large enough to > >>> hold struct pthread_mutex, the rest of the implementation of the > >>> shared mutex is relatively trivial, if not already done. > >>> > >>> Changing this ABI is very hard. libthr provides the symbol > >>> versioning, which allows to provide compatible shims for the previous > >>> ABI variant. But since userspace tends to use the pthread objects in > >>> the layouts of the library objects, this causes serious ABI issues > >>> when mixing libraries built against different default versions of > >>> libthr. > >> > >> I think this is only if the libraries (or apps) pass pthread > >> lock types between them, that one has initialized with one ABI > >> but the other library uses another ABI. We should really > >> exclude this as part of our ABI compatibility. > > Applications commonly embed pthread locks into their objects, including > > the exposed ABI in the third-party libraries. > > > > struct my_object { > > pthread_mutex_t obj_lock; > > ... > > }; > > > > Changing the size of the pthread locks causes uncontrolled breakage of > > the ABI for significant number of third-party code. > > If the application creates the object itself or allocates storage > for it, basically, if it isn't opaque, yes. But we can bump port > revisions for affected libraries (probably just searching > /usr/local/include/... for pthread_mutex, pthread_cond, etc > types to see possible problems. I did a search for the installed > ports on my system and found a few that might cause problems. This relegates the issue to an attempt to do the full rebuild. But I do not see how the port bump would fix it, assume that you are updating from the 10.x to 11.x and have the mix of the libraries, some of which were built during the 10.x lifetime but with the bumped ports version. It is not feasible to do a reliable audit of the 24+ Kports. > > I think we're just putting off the inevitable. Do we not want > to change our pthread sync types anyway, to get rid of an extra > dereference per lock? To get rid of the hacks in libc, rtld, > etc? > > If the answer is no, we never want to do that, then ok. An answer to this question requires a) consensus b) a workforce that would do the decided transition. I evaluated my opinion, potential consequences and efforts required, and ended up with the posted patch. From owner-freebsd-threads@freebsd.org Wed Dec 23 21:31:00 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5B47A50CCB for ; Wed, 23 Dec 2015 21:31:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B25591FA5 for ; Wed, 23 Dec 2015 21:31:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id AF86EA50CC7; Wed, 23 Dec 2015 21:31:00 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AEB55A50CC5; Wed, 23 Dec 2015 21:31:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7A7A31FA2; Wed, 23 Dec 2015 21:31:00 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBNLUwLg005586; Wed, 23 Dec 2015 16:30:58 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Wed, 23 Dec 2015 16:30:58 -0500 (EST) Date: Wed, 23 Dec 2015 16:30:58 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151223201837.GW3625@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <20151223190519.GU3625@kib.kiev.ua> <20151223201837.GW3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 21:31:00 -0000 On Wed, 23 Dec 2015, Konstantin Belousov wrote: > On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote: >> >> If the application creates the object itself or allocates storage >> for it, basically, if it isn't opaque, yes. But we can bump port >> revisions for affected libraries (probably just searching >> /usr/local/include/... for pthread_mutex, pthread_cond, etc >> types to see possible problems. I did a search for the installed >> ports on my system and found a few that might cause problems. > > This relegates the issue to an attempt to do the full rebuild. But I > do not see how the port bump would fix it, assume that you are updating > from the 10.x to 11.x and have the mix of the libraries, some of which > were built during the 10.x lifetime but with the bumped ports version. > > It is not feasible to do a reliable audit of the 24+ Kports. Is it really that hard to do a port run and insert a grep for pthread_{mutex,cond,rwlock}_t in a ports installed header files? Then just blindly bumping portrevisions for those ports and those depending on it? Other than errors caused by storage layouts, libthr can easily be instrumented to emit a warning when a sync type is used with the wrong versioned function. >> I think we're just putting off the inevitable. Do we not want >> to change our pthread sync types anyway, to get rid of an extra >> dereference per lock? To get rid of the hacks in libc, rtld, >> etc? >> >> If the answer is no, we never want to do that, then ok. > > An answer to this question requires a) consensus b) a workforce that > would do the decided transition. I evaluated my opinion, potential > consequences and efforts required, and ended up with the posted patch. There's an old patch here: http://people.freebsd.org/~davidxu/pshared/patch6.diff -- DE From owner-freebsd-threads@freebsd.org Thu Dec 24 06:00:59 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9DD59A4F4A2 for ; Thu, 24 Dec 2015 06:00:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 7F05D1D41 for ; Thu, 24 Dec 2015 06:00:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6ED93A4F49E; Thu, 24 Dec 2015 06:00:59 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54067A4F49C; Thu, 24 Dec 2015 06:00:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 33B2D1D3D; Thu, 24 Dec 2015 06:00:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4D005B980; Thu, 24 Dec 2015 01:00:58 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Cc: Konstantin Belousov , Daniel Eischen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Date: Wed, 23 Dec 2015 13:22:16 -0800 Message-ID: <4199356.DlQeWDh27F@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <20151223201837.GW3625@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <20151223201837.GW3625@kib.kiev.ua> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 24 Dec 2015 01:00:58 -0500 (EST) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 06:00:59 -0000 On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote: > On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote: > > On Wed, 23 Dec 2015, Konstantin Belousov wrote: > > > > > On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote: > > >> On Wed, 23 Dec 2015, Konstantin Belousov wrote: > > >> > > >> [ ... ] > > >>> Would the ABI modified to make the pthread_mutex_t large enough to > > >>> hold struct pthread_mutex, the rest of the implementation of the > > >>> shared mutex is relatively trivial, if not already done. > > >>> > > >>> Changing this ABI is very hard. libthr provides the symbol > > >>> versioning, which allows to provide compatible shims for the previous > > >>> ABI variant. But since userspace tends to use the pthread objects in > > >>> the layouts of the library objects, this causes serious ABI issues > > >>> when mixing libraries built against different default versions of > > >>> libthr. > > >> > > >> I think this is only if the libraries (or apps) pass pthread > > >> lock types between them, that one has initialized with one ABI > > >> but the other library uses another ABI. We should really > > >> exclude this as part of our ABI compatibility. > > > Applications commonly embed pthread locks into their objects, including > > > the exposed ABI in the third-party libraries. > > > > > > struct my_object { > > > pthread_mutex_t obj_lock; > > > ... > > > }; > > > > > > Changing the size of the pthread locks causes uncontrolled breakage of > > > the ABI for significant number of third-party code. > > > > If the application creates the object itself or allocates storage > > for it, basically, if it isn't opaque, yes. But we can bump port > > revisions for affected libraries (probably just searching > > /usr/local/include/... for pthread_mutex, pthread_cond, etc > > types to see possible problems. I did a search for the installed > > ports on my system and found a few that might cause problems. > This relegates the issue to an attempt to do the full rebuild. But I > do not see how the port bump would fix it, assume that you are updating > from the 10.x to 11.x and have the mix of the libraries, some of which > were built during the 10.x lifetime but with the bumped ports version. > > It is not feasible to do a reliable audit of the 24+ Kports. As a bit of a devil's advocate, I think the 64-bit ino_t change will in fact require this for 11. I suspect 3rd pary apps embed struct stat in various structures as well and that that ABI change will require not mixing old and new libraries. One other point in favor of Konstantin's approach (IMO) is that keeping the structures private prevents having to maintain the ABI of those structures in the future. I'm already keenly aware of how painful a problem that can be with our non-opaque FILE (and which we cannot now make opaque even though the standard APIs would work fine with an opaque object). -- John Baldwin From owner-freebsd-threads@freebsd.org Thu Dec 24 10:46:04 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 384B4A50D96 for ; Thu, 24 Dec 2015 10:46:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2252D12BE for ; Thu, 24 Dec 2015 10:46:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1E581A50D92; Thu, 24 Dec 2015 10:46:04 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1BDFBA50D90; Thu, 24 Dec 2015 10:46:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B3A712BA; Thu, 24 Dec 2015 10:46:03 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBOAjuRt090339 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 24 Dec 2015 12:45:57 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBOAjuRt090339 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBOAjuZE090325; Thu, 24 Dec 2015 12:45:56 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 24 Dec 2015 12:45:56 +0200 From: Konstantin Belousov To: John Baldwin Cc: freebsd-arch@freebsd.org, Daniel Eischen , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151224104556.GX3625@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <20151223201837.GW3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4199356.DlQeWDh27F@ralph.baldwin.cx> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 10:46:04 -0000 On Wed, Dec 23, 2015 at 01:22:16PM -0800, John Baldwin wrote: > On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote: > > On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote: > > > On Wed, 23 Dec 2015, Konstantin Belousov wrote: > > > > > > > On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote: > > > >> On Wed, 23 Dec 2015, Konstantin Belousov wrote: > > > >> > > > >> [ ... ] > > > >>> Would the ABI modified to make the pthread_mutex_t large enough to > > > >>> hold struct pthread_mutex, the rest of the implementation of the > > > >>> shared mutex is relatively trivial, if not already done. > > > >>> > > > >>> Changing this ABI is very hard. libthr provides the symbol > > > >>> versioning, which allows to provide compatible shims for the previous > > > >>> ABI variant. But since userspace tends to use the pthread objects in > > > >>> the layouts of the library objects, this causes serious ABI issues > > > >>> when mixing libraries built against different default versions of > > > >>> libthr. > > > >> > > > >> I think this is only if the libraries (or apps) pass pthread > > > >> lock types between them, that one has initialized with one ABI > > > >> but the other library uses another ABI. We should really > > > >> exclude this as part of our ABI compatibility. > > > > Applications commonly embed pthread locks into their objects, including > > > > the exposed ABI in the third-party libraries. > > > > > > > > struct my_object { > > > > pthread_mutex_t obj_lock; > > > > ... > > > > }; > > > > > > > > Changing the size of the pthread locks causes uncontrolled breakage of > > > > the ABI for significant number of third-party code. > > > > > > If the application creates the object itself or allocates storage > > > for it, basically, if it isn't opaque, yes. But we can bump port > > > revisions for affected libraries (probably just searching > > > /usr/local/include/... for pthread_mutex, pthread_cond, etc > > > types to see possible problems. I did a search for the installed > > > ports on my system and found a few that might cause problems. > > This relegates the issue to an attempt to do the full rebuild. But I > > do not see how the port bump would fix it, assume that you are updating > > from the 10.x to 11.x and have the mix of the libraries, some of which > > were built during the 10.x lifetime but with the bumped ports version. > > > > It is not feasible to do a reliable audit of the 24+ Kports. > > As a bit of a devil's advocate, I think the 64-bit ino_t change will in > fact require this for 11. I suspect 3rd pary apps embed struct stat in > various structures as well and that that ABI change will require not > mixing old and new libraries. The stat change is different in this. I do think (and quick overview of includes seems to support the opinion) that the struct stat is rarely embedded into the user objects. I already wrote about this, when I said that 'struct stat' changes are mostly confined to the src/ tree, in one of the previous messages. When I thought about the ino_t 64bit change, the only example of affected app that I could remember, without looking at the code, was rogue/nethack. > > One other point in favor of Konstantin's approach (IMO) is that keeping > the structures private prevents having to maintain the ABI of those > structures in the future. I'm already keenly aware of how painful a > problem that can be with our non-opaque FILE (and which we cannot now > make opaque even though the standard APIs would work fine with an opaque > object). My patch does not prevent future change to use in-place pthread_mutex_t, since the patch does not change current ABI. If somebody is willing to go that route, well, fine. Nobody productized David' work for ten (?) years. And the reason is, I believe, that the ABI change required to the fundamental facility (threads did become ubiquios) at that late point in the OS lifecycle is destructive. The issue we have after the embedded structures layout modification is exactly the issue which cannot be handled by symbol versioning. From owner-freebsd-threads@freebsd.org Thu Dec 24 13:45:41 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18EB7A50E30 for ; Thu, 24 Dec 2015 13:45:41 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id F29C91DC1 for ; Thu, 24 Dec 2015 13:45:40 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id EFDA7A50E2A; Thu, 24 Dec 2015 13:45:40 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D42CFA50E26; Thu, 24 Dec 2015 13:45:40 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 920171DBE; Thu, 24 Dec 2015 13:45:40 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBODjcld041978; Thu, 24 Dec 2015 08:45:38 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Thu, 24 Dec 2015 08:45:38 -0500 (EST) Date: Thu, 24 Dec 2015 08:45:38 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: John Baldwin cc: freebsd-arch@freebsd.org, Konstantin Belousov , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <4199356.DlQeWDh27F@ralph.baldwin.cx> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <20151223201837.GW3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 13:45:41 -0000 On Wed, 23 Dec 2015, John Baldwin wrote: > On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote: >> On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote: >>> On Wed, 23 Dec 2015, Konstantin Belousov wrote: >>> >>>> On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote: >>>>> On Wed, 23 Dec 2015, Konstantin Belousov wrote: >>>>> >>>>> [ ... ] >>>>>> Would the ABI modified to make the pthread_mutex_t large enough to >>>>>> hold struct pthread_mutex, the rest of the implementation of the >>>>>> shared mutex is relatively trivial, if not already done. >>>>>> >>>>>> Changing this ABI is very hard. libthr provides the symbol >>>>>> versioning, which allows to provide compatible shims for the previous >>>>>> ABI variant. But since userspace tends to use the pthread objects in >>>>>> the layouts of the library objects, this causes serious ABI issues >>>>>> when mixing libraries built against different default versions of >>>>>> libthr. >>>>> >>>>> I think this is only if the libraries (or apps) pass pthread >>>>> lock types between them, that one has initialized with one ABI >>>>> but the other library uses another ABI. We should really >>>>> exclude this as part of our ABI compatibility. >>>> Applications commonly embed pthread locks into their objects, including >>>> the exposed ABI in the third-party libraries. >>>> >>>> struct my_object { >>>> pthread_mutex_t obj_lock; >>>> ... >>>> }; >>>> >>>> Changing the size of the pthread locks causes uncontrolled breakage of >>>> the ABI for significant number of third-party code. >>> >>> If the application creates the object itself or allocates storage >>> for it, basically, if it isn't opaque, yes. But we can bump port >>> revisions for affected libraries (probably just searching >>> /usr/local/include/... for pthread_mutex, pthread_cond, etc >>> types to see possible problems. I did a search for the installed >>> ports on my system and found a few that might cause problems. >> This relegates the issue to an attempt to do the full rebuild. But I >> do not see how the port bump would fix it, assume that you are updating >> from the 10.x to 11.x and have the mix of the libraries, some of which >> were built during the 10.x lifetime but with the bumped ports version. >> >> It is not feasible to do a reliable audit of the 24+ Kports. > > As a bit of a devil's advocate, I think the 64-bit ino_t change will in > fact require this for 11. I suspect 3rd pary apps embed struct stat in > various structures as well and that that ABI change will require not > mixing old and new libraries. > > One other point in favor of Konstantin's approach (IMO) is that keeping > the structures private prevents having to maintain the ABI of those > structures in the future. I'm already keenly aware of how painful a > problem that can be with our non-opaque FILE (and which we cannot now > make opaque even though the standard APIs would work fine with an opaque > object). We would include extra/spare words in the struct from day 1. The public struct should just consist of an array of storage units or similar, so that nothing can be gleaned from the elements of the struct. This is what Solaris does (or at least used to). Going forward, I think the sync structures just need to be able to be properly initialized with PTHREAD_FOO_INITIALIZER. If 10 years out the spare words are not enough, we could still allocate remaining storage on first use like we do now. -- DE From owner-freebsd-threads@freebsd.org Thu Dec 24 14:29:51 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79E04A51ADC for ; Thu, 24 Dec 2015 14:29:51 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 669CD1948 for ; Thu, 24 Dec 2015 14:29:51 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6591AA51ADB; Thu, 24 Dec 2015 14:29:51 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64F97A51AD9; Thu, 24 Dec 2015 14:29:51 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 304F91947; Thu, 24 Dec 2015 14:29:50 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBOETneO001930; Thu, 24 Dec 2015 09:29:49 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Thu, 24 Dec 2015 09:29:49 -0500 (EST) Date: Thu, 24 Dec 2015 09:29:49 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: John Baldwin cc: freebsd-arch@freebsd.org, Konstantin Belousov , threads@freebsd.org, arch@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <4199356.DlQeWDh27F@ralph.baldwin.cx> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <20151223201837.GW3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 14:29:51 -0000 On Wed, 23 Dec 2015, John Baldwin wrote: > On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote: [ much snipped for brevity ] >> >> It is not feasible to do a reliable audit of the 24+ Kports. > > As a bit of a devil's advocate, I think the 64-bit ino_t change will in > fact require this for 11. I suspect 3rd pary apps embed struct stat in > various structures as well and that that ABI change will require not > mixing old and new libraries. > > One other point in favor of Konstantin's approach (IMO) is that keeping > the structures private prevents having to maintain the ABI of those > structures in the future. I'm already keenly aware of how painful a > problem that can be with our non-opaque FILE (and which we cannot now > make opaque even though the standard APIs would work fine with an opaque > object). This seems to be David's latest patch: http://people.freebsd.org/~davidxu/patch/pshared0607.diff It is only 3 years old (2012). I have email from David that says he got 8-10% speedup in mysql OLTP from making the synch types structures. The patch also implements robust and priority inheritence mutexes bumps shared library versions. -- DE From owner-freebsd-threads@freebsd.org Thu Dec 24 17:25:43 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7EEBA51198; Thu, 24 Dec 2015 17:25:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C24C41324; Thu, 24 Dec 2015 17:25:42 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4D855B95B; Thu, 24 Dec 2015 12:25:41 -0500 (EST) From: John Baldwin To: Daniel Eischen Cc: freebsd-arch@freebsd.org, Konstantin Belousov , freebsd-threads@freebsd.org Subject: Re: libthr shared locks Date: Thu, 24 Dec 2015 09:25:33 -0800 Message-ID: <5496837.TbTQtANDNj@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 24 Dec 2015 12:25:41 -0500 (EST) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 17:25:43 -0000 On Thursday, December 24, 2015 08:45:38 AM Daniel Eischen wrote: > On Wed, 23 Dec 2015, John Baldwin wrote: > > > On Wednesday, December 23, 2015 10:18:37 PM Konstantin Belousov wrote: > >> On Wed, Dec 23, 2015 at 02:48:56PM -0500, Daniel Eischen wrote: > >>> On Wed, 23 Dec 2015, Konstantin Belousov wrote: > >>> > >>>> On Wed, Dec 23, 2015 at 01:27:35PM -0500, Daniel Eischen wrote: > >>>>> On Wed, 23 Dec 2015, Konstantin Belousov wrote: > >>>>> > >>>>> [ ... ] > >>>>>> Would the ABI modified to make the pthread_mutex_t large enough to > >>>>>> hold struct pthread_mutex, the rest of the implementation of the > >>>>>> shared mutex is relatively trivial, if not already done. > >>>>>> > >>>>>> Changing this ABI is very hard. libthr provides the symbol > >>>>>> versioning, which allows to provide compatible shims for the previous > >>>>>> ABI variant. But since userspace tends to use the pthread objects in > >>>>>> the layouts of the library objects, this causes serious ABI issues > >>>>>> when mixing libraries built against different default versions of > >>>>>> libthr. > >>>>> > >>>>> I think this is only if the libraries (or apps) pass pthread > >>>>> lock types between them, that one has initialized with one ABI > >>>>> but the other library uses another ABI. We should really > >>>>> exclude this as part of our ABI compatibility. > >>>> Applications commonly embed pthread locks into their objects, including > >>>> the exposed ABI in the third-party libraries. > >>>> > >>>> struct my_object { > >>>> pthread_mutex_t obj_lock; > >>>> ... > >>>> }; > >>>> > >>>> Changing the size of the pthread locks causes uncontrolled breakage of > >>>> the ABI for significant number of third-party code. > >>> > >>> If the application creates the object itself or allocates storage > >>> for it, basically, if it isn't opaque, yes. But we can bump port > >>> revisions for affected libraries (probably just searching > >>> /usr/local/include/... for pthread_mutex, pthread_cond, etc > >>> types to see possible problems. I did a search for the installed > >>> ports on my system and found a few that might cause problems. > >> This relegates the issue to an attempt to do the full rebuild. But I > >> do not see how the port bump would fix it, assume that you are updating > >> from the 10.x to 11.x and have the mix of the libraries, some of which > >> were built during the 10.x lifetime but with the bumped ports version. > >> > >> It is not feasible to do a reliable audit of the 24+ Kports. > > > > As a bit of a devil's advocate, I think the 64-bit ino_t change will in > > fact require this for 11. I suspect 3rd pary apps embed struct stat in > > various structures as well and that that ABI change will require not > > mixing old and new libraries. > > > > One other point in favor of Konstantin's approach (IMO) is that keeping > > the structures private prevents having to maintain the ABI of those > > structures in the future. I'm already keenly aware of how painful a > > problem that can be with our non-opaque FILE (and which we cannot now > > make opaque even though the standard APIs would work fine with an opaque > > object). > > We would include extra/spare words in the struct from day 1. > The public struct should just consist of an array of storage > units or similar, so that nothing can be gleaned from the > elements of the struct. This is what Solaris does (or at least > used to). > > Going forward, I think the sync structures just need to be able > to be properly initialized with PTHREAD_FOO_INITIALIZER. If > 10 years out the spare words are not enough, we could still > allocate remaining storage on first use like we do now. You can't allocate extra storage for the PSHARED case. Any changes to PSHARED primitives that require altering the layout are full-blown ABI breakages the same as the one being contemplated here. -- John Baldwin From owner-freebsd-threads@freebsd.org Thu Dec 24 18:46:32 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6FB74A50FAD; Thu, 24 Dec 2015 18:46:32 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A6CB19FA; Thu, 24 Dec 2015 18:46:32 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBOIkPHN001165; Thu, 24 Dec 2015 13:46:25 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Thu, 24 Dec 2015 13:46:25 -0500 (EST) Date: Thu, 24 Dec 2015 13:46:25 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: John Baldwin cc: freebsd-arch@freebsd.org, Konstantin Belousov , freebsd-threads@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <5496837.TbTQtANDNj@ralph.baldwin.cx> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 18:46:32 -0000 On Thu, 24 Dec 2015, John Baldwin wrote: > On Thursday, December 24, 2015 08:45:38 AM Daniel Eischen wrote: >> We would include extra/spare words in the struct from day 1. >> The public struct should just consist of an array of storage >> units or similar, so that nothing can be gleaned from the >> elements of the struct. This is what Solaris does (or at least >> used to). >> >> Going forward, I think the sync structures just need to be able >> to be properly initialized with PTHREAD_FOO_INITIALIZER. If >> 10 years out the spare words are not enough, we could still >> allocate remaining storage on first use like we do now. > > You can't allocate extra storage for the PSHARED case. Any changes > to PSHARED primitives that require altering the layout are full-blown > ABI breakages the same as the one being contemplated here. Yes, I know. With spare slots and being able to move anything required for shared to near the beginning of the struct, I think we'd be good for quite a while anyway. Can we still implement things like robust and priority mutexes in a pshared mutex with Konstantin's implementation? The rest of the world (Linux, Solaris anyway) use structs and get by just fine ABI-wise. -- DE From owner-freebsd-threads@freebsd.org Thu Dec 24 19:14:15 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67D55A517E2; Thu, 24 Dec 2015 19:14:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D05331608; Thu, 24 Dec 2015 19:14:14 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBOJE89k074931 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 24 Dec 2015 21:14:09 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBOJE89k074931 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBOJE80T074930; Thu, 24 Dec 2015 21:14:08 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 24 Dec 2015 21:14:08 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151224191408.GA3625@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 19:14:15 -0000 On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote: > Can we still implement things like robust and priority mutexes in a > pshared mutex with Konstantin's implementation? The rest of the world > (Linux, Solaris anyway) use structs and get by just fine ABI-wise. Yes, we do. I intend to do this as the next stage, planned the implementation and did some preparations for it in the current patch. This raises the question, why do you suppose that my approach with the off-page would not allow it ? I am asking not to argument, but to understand a possible shortcoming in the approach, which I missed. >From the implementation PoV, off-page is completely equivalent to the in-line structs approach, and the difference is only in the initial translation of say pthread_mutex_t to the structure (off-page lookup vs. identity). The issues with maintaining the lists of the owned robust of pi mutexes are same, and in fact you could see some non-trivial traces of this in the _mutex_fork() reimplementation in my patch. Linux and Solaris get (large enough) structs early enough to avoid the ABI issues. I remember Linux changing FILE layout in early days of glibc 2.0 and resulting pain, while they already had the GNU symbol versioning working and used. From owner-freebsd-threads@freebsd.org Fri Dec 25 18:18:16 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E084AA51B1F; Fri, 25 Dec 2015 18:18:16 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AA862197E; Fri, 25 Dec 2015 18:18:16 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBPIIEvS003623; Fri, 25 Dec 2015 13:18:14 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Fri, 25 Dec 2015 13:18:14 -0500 (EST) Date: Fri, 25 Dec 2015 13:18:14 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151224191408.GA3625@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Dec 2015 18:18:17 -0000 On Thu, 24 Dec 2015, Konstantin Belousov wrote: > On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote: >> Can we still implement things like robust and priority mutexes in a >> pshared mutex with Konstantin's implementation? The rest of the world >> (Linux, Solaris anyway) use structs and get by just fine ABI-wise. > > Yes, we do. I intend to do this as the next stage, planned the > implementation and did some preparations for it in the current patch. > This raises the question, why do you suppose that my approach with > the off-page would not allow it ? I am asking not to argument, but to > understand a possible shortcoming in the approach, which I missed. No, I was merely inquiring, if this is on your agenda, that's good. > From the implementation PoV, off-page is completely equivalent to the > in-line structs approach, and the difference is only in the initial > translation of say pthread_mutex_t to the structure (off-page lookup vs. > identity). The issues with maintaining the lists of the owned robust of > pi mutexes are same, and in fact you could see some non-trivial traces > of this in the _mutex_fork() reimplementation in my patch. > > Linux and Solaris get (large enough) structs early enough to avoid the ABI > issues. I remember Linux changing FILE layout in early days of glibc 2.0 > and resulting pain, while they already had the GNU symbol versioning > working and used. I guess the issue now is the apparent speedups without having the extra de-reference. 8-10% for things like mysql might be worth considering the userland structs. -- DE From owner-freebsd-threads@freebsd.org Sat Dec 26 02:46:57 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9088A51654 for ; Sat, 26 Dec 2015 02:46:57 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 97E741DD1 for ; Sat, 26 Dec 2015 02:46:57 +0000 (UTC) (envelope-from julian@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 96C09A51653; Sat, 26 Dec 2015 02:46:57 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 96337A51651; Sat, 26 Dec 2015 02:46:57 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 740041DD0; Sat, 26 Dec 2015 02:46:57 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-234-233.lns20.per1.internode.on.net [121.45.234.233]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id tBQ2kpSX003335 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Fri, 25 Dec 2015 18:46:54 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: libthr shared locks To: Konstantin Belousov , threads@freebsd.org, arch@freebsd.org References: <20151223172528.GT3625@kib.kiev.ua> From: Julian Elischer Message-ID: <567DFF95.70904@freebsd.org> Date: Sat, 26 Dec 2015 10:46:45 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <20151223172528.GT3625@kib.kiev.ua> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Dec 2015 02:46:57 -0000 On 24/12/2015 1:25 AM, Konstantin Belousov wrote: > A well-known limitation of the FreeBSD libthr implementation of the > pthread locking objects is the missed support for the process-shared > locks. The hardest part of implementing the support is the neccessity > of providing ABI compatibility for the current implementation. > [...] > > Changing this ABI is very hard. libthr provides the symbol > versioning, which allows to provide compatible shims for the previous > ABI variant. But since userspace tends to use the pthread objects in > the layouts of the library objects, this causes serious ABI issues > when mixing libraries built against different default versions of > libthr. which is one reason I think symbol versioning is over-rated.. just make a new library version. > From owner-freebsd-threads@freebsd.org Sat Dec 26 10:54:19 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC0D2A52959; Sat, 26 Dec 2015 10:54:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3CFEB17CF; Sat, 26 Dec 2015 10:54:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBQAs9dI042457 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sat, 26 Dec 2015 12:54:09 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBQAs9dI042457 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBQAs93R042456; Sat, 26 Dec 2015 12:54:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 26 Dec 2015 12:54:09 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151226105409.GH3625@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Dec 2015 10:54:19 -0000 On Fri, Dec 25, 2015 at 01:18:14PM -0500, Daniel Eischen wrote: > On Thu, 24 Dec 2015, Konstantin Belousov wrote: > > > On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote: > >> Can we still implement things like robust and priority mutexes in a > >> pshared mutex with Konstantin's implementation? The rest of the world > >> (Linux, Solaris anyway) use structs and get by just fine ABI-wise. > > > > Yes, we do. I intend to do this as the next stage, planned the > > implementation and did some preparations for it in the current patch. > > This raises the question, why do you suppose that my approach with > > the off-page would not allow it ? I am asking not to argument, but to > > understand a possible shortcoming in the approach, which I missed. > > No, I was merely inquiring, if this is on your agenda, that's > good. > > > From the implementation PoV, off-page is completely equivalent to the > > in-line structs approach, and the difference is only in the initial > > translation of say pthread_mutex_t to the structure (off-page lookup vs. > > identity). The issues with maintaining the lists of the owned robust of > > pi mutexes are same, and in fact you could see some non-trivial traces > > of this in the _mutex_fork() reimplementation in my patch. > > > > Linux and Solaris get (large enough) structs early enough to avoid the ABI > > issues. I remember Linux changing FILE layout in early days of glibc 2.0 > > and resulting pain, while they already had the GNU symbol versioning > > working and used. > > I guess the issue now is the apparent speedups without having the > extra de-reference. 8-10% for things like mysql might be worth > considering the userland structs. I am sorry for the delay in answering, I read the David' patch you pointed out, and it took time to get through 9KLOCs patch which is not applicable to the current sources. I should note that the patch is not immediately useable, not due to the source code drift, but because the patch combines many unrelated things, and I do not agree with everything in it. E.g., I found quite interesting the notion that our libthr fork() does not work when called from the signal handler. I do not quite understand how the detection of the situation is done in the patch, and I do not agree with the cleanup of the 'in sighandler' state on longjmp. I think that this part of the patch is obsoleted by libthr intercepting signals and postponing signal delivery until a critical section is exited. Patch adds yet another private malloc(), now to libthr. I think that cooperating with rtld and share the same allocator would be more reasonable. Notes above are to explain why I think that productizing the David' patch is huge work, IMO significantly more than just 'test and commit'. Returning to the point of potential gain in performance due to the ABI change. As I already stated in my previous replies, I am quite worried about the impact of the ABI change, and I think that potential destabilizations, which would manifest itself in the subtle and hard to diagnose ways (i.e. it is not a sigsegv outright on the start) is huge setback for the change. OTOH, we should be able to plan such change, which requires much more drastic measures to be implementable. I started thinking about it, and I noted that what is needed is solved by renaming libthr to something else, e.g. libthr2. One of the issue which is not solved by the dso version bump (not to even mention versioned symbols bump) is the static linker bringing both libthr.so.3 and libthr.so.4 into the same process, by the mere use of -lpthread by different objects at different times. Then, libthr and hypothetical libthr2 should check and prevent activation of image which loads both. But these are longtime to evaluate and implement. The feature at hand does not require ABI change, as my patch demostrated. Yes, shared mutexes would be more naturally implemented with the inline locks, and avoidance of an indirection in the libthr is also good, but lets not mix significant ABI change and some less ambitious feature. My approach does not lock out future strategies. From owner-freebsd-threads@freebsd.org Sat Dec 26 17:15:46 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32D96A52A03; Sat, 26 Dec 2015 17:15:46 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL SHA256 CA - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A0D31B21; Sat, 26 Dec 2015 17:15:45 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.15.1/8.15.1/NETPLEX) with ESMTP id tBQHFhXY016956; Sat, 26 Dec 2015 12:15:44 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Sat, 26 Dec 2015 12:15:44 -0500 (EST) Date: Sat, 26 Dec 2015 12:15:43 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Konstantin Belousov cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks In-Reply-To: <20151226105409.GH3625@kib.kiev.ua> Message-ID: References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Dec 2015 17:15:46 -0000 On Sat, 26 Dec 2015, Konstantin Belousov wrote: > On Fri, Dec 25, 2015 at 01:18:14PM -0500, Daniel Eischen wrote: >> On Thu, 24 Dec 2015, Konstantin Belousov wrote: >> >>> On Thu, Dec 24, 2015 at 01:46:25PM -0500, Daniel Eischen wrote: >>>> Can we still implement things like robust and priority mutexes in a >>>> pshared mutex with Konstantin's implementation? The rest of the world >>>> (Linux, Solaris anyway) use structs and get by just fine ABI-wise. >>> >>> Yes, we do. I intend to do this as the next stage, planned the >>> implementation and did some preparations for it in the current patch. >>> This raises the question, why do you suppose that my approach with >>> the off-page would not allow it ? I am asking not to argument, but to >>> understand a possible shortcoming in the approach, which I missed. >> >> No, I was merely inquiring, if this is on your agenda, that's >> good. >> >>> From the implementation PoV, off-page is completely equivalent to the >>> in-line structs approach, and the difference is only in the initial >>> translation of say pthread_mutex_t to the structure (off-page lookup vs. >>> identity). The issues with maintaining the lists of the owned robust of >>> pi mutexes are same, and in fact you could see some non-trivial traces >>> of this in the _mutex_fork() reimplementation in my patch. >>> >>> Linux and Solaris get (large enough) structs early enough to avoid the ABI >>> issues. I remember Linux changing FILE layout in early days of glibc 2.0 >>> and resulting pain, while they already had the GNU symbol versioning >>> working and used. >> >> I guess the issue now is the apparent speedups without having the >> extra de-reference. 8-10% for things like mysql might be worth >> considering the userland structs. > > I am sorry for the delay in answering, I read the David' patch you pointed > out, and it took time to get through 9KLOCs patch which is not applicable > to the current sources. It is the holidays, I'm sure everyone is busy with other things. I am in the same camp :-) > I should note that the patch is not immediately useable, not due to > the source code drift, but because the patch combines many unrelated > things, and I do not agree with everything in it. Yeah, I would advocate only committing part of David's patch, in stages. Just the part that implements the struct change and shared synch primitives. Other changes may no longer apply or can be done later. > E.g., I found quite interesting the notion that our libthr fork() does > not work when called from the signal handler. I do not quite understand > how the detection of the situation is done in the patch, and I do not > agree with the cleanup of the 'in sighandler' state on longjmp. I > think that this part of the patch is obsoleted by libthr intercepting > signals and postponing signal delivery until a critical section is > exited. I think libthr has always done that (postpoining until critical section is exited). But there are libc (and perhaps other) locks that may be left locked on a fork(). Really (according to POSIX), a fork() from a threaded program can only be used to exec() something. > Patch adds yet another private malloc(), now to libthr. I think that > cooperating with rtld and share the same allocator would be more > reasonable. I tend to agree. > Notes above are to explain why I think that productizing the David' patch > is huge work, IMO significantly more than just 'test and commit'. > > Returning to the point of potential gain in performance due to the > ABI change. As I already stated in my previous replies, I am quite > worried about the impact of the ABI change, and I think that potential > destabilizations, which would manifest itself in the subtle and hard to > diagnose ways (i.e. it is not a sigsegv outright on the start) is huge > setback for the change. I guess I am just not that worried about the ABI change when combined with a library version bump. We use to do this for every new major release, it isn't anything that hasn't been done before. We never supported 2 thread libraries in the same executable, and it is obvious when that happens (I think we even instrumented libc_r, or was it rtld?, to emit a warning when that happened. My memory is fuzzy). With poudriere, our new package system, I would think we would be better off now than 10 years ago? I do note that I was initially wrong in thinking that port revision bumps could help alleviate any pain. A port could be built in -stable with the revision bump just as well as in -current (or 11). What would be needed is for pkg to understand that any port built for 10 and previous could not run on 11+ (just to be on the safe side). > OTOH, we should be able to plan such change, which requires much more > drastic measures to be implementable. I started thinking about it, and > I noted that what is needed is solved by renaming libthr to something > else, e.g. libthr2. One of the issue which is not solved by the dso > version bump (not to even mention versioned symbols bump) is the > static linker bringing both libthr.so.3 and libthr.so.4 into the same > process, by the mere use of -lpthread by different objects at different > times. Then, libthr and hypothetical libthr2 should check and prevent > activation of image which loads both. > > But these are longtime to evaluate and implement. The feature at hand > does not require ABI change, as my patch demostrated. Yes, shared > mutexes would be more naturally implemented with the inline locks, and > avoidance of an indirection in the libthr is also good, but lets not mix > significant ABI change and some less ambitious feature. My approach does > not lock out future strategies. I agree, but the work that you are doing now would be basically thrown out later on. I will not stand in your way and appreciate any work you do. I would just rather that the struct change be made now for 11, even without any pshared or other changes. For once the struct change is made, pshared or other additions can be made afterward, even in the 11 branch because they would not break the ABI. -- DE From owner-freebsd-threads@freebsd.org Sat Dec 26 20:04:23 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E6AEA536DD for ; Sat, 26 Dec 2015 20:04:23 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2D49B1F79 for ; Sat, 26 Dec 2015 20:04:23 +0000 (UTC) (envelope-from sjg@juniper.net) Received: by mailman.ysv.freebsd.org (Postfix) id 2A9EFA536DB; Sat, 26 Dec 2015 20:04:23 +0000 (UTC) Delivered-To: threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A124A536D9; Sat, 26 Dec 2015 20:04:23 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from na01-by2-obe.outbound.protection.outlook.com (mail-by2on0121.outbound.protection.outlook.com [207.46.100.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7AD0D1F77; Sat, 26 Dec 2015 20:04:21 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from BLUPR05CA0052.namprd05.prod.outlook.com (10.141.20.22) by BN3PR0501MB1379.namprd05.prod.outlook.com (10.160.117.13) with Microsoft SMTP Server (TLS) id 15.1.361.13; Sat, 26 Dec 2015 20:04:14 +0000 Received: from BN1BFFO11FD029.protection.gbl (2a01:111:f400:7c10::1:166) by BLUPR05CA0052.outlook.office365.com (2a01:111:e400:855::22) with Microsoft SMTP Server (TLS) id 15.1.361.13 via Frontend Transport; Sat, 26 Dec 2015 20:04:13 +0000 Authentication-Results: spf=softfail (sender IP is 66.129.239.19) smtp.mailfrom=juniper.net; freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=juniper.net; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.19 as permitted sender) Received: from p-emfe01b-sac.jnpr.net (66.129.239.19) by BN1BFFO11FD029.mail.protection.outlook.com (10.58.144.92) with Microsoft SMTP Server (TLS) id 15.1.355.15 via Frontend Transport; Sat, 26 Dec 2015 20:04:13 +0000 Received: from magenta.juniper.net (172.17.27.123) by p-emfe01b-sac.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.123.3; Sat, 26 Dec 2015 12:04:12 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.21.16.28]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id tBQK4BD61357; Sat, 26 Dec 2015 12:04:11 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos (localhost [IPv6:::1]) by chaos.jnpr.net (Postfix) with ESMTP id 009FE580A9; Sat, 26 Dec 2015 12:04:10 -0800 (PST) To: Julian Elischer CC: Konstantin Belousov , , , Subject: Re: libthr shared locks In-Reply-To: <567DFF95.70904@freebsd.org> References: <20151223172528.GT3625@kib.kiev.ua> <567DFF95.70904@freebsd.org> Comments: In-reply-to: Julian Elischer message dated "Sat, 26 Dec 2015 10:46:45 +0800." From: "Simon J. Gerraty" X-Mailer: MH-E 8.6; nmh 1.6; GNU Emacs 24.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <10836.1451160250.1@chaos> Date: Sat, 26 Dec 2015 12:04:10 -0800 Message-ID: <6581.1451160250@chaos> X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; BN1BFFO11FD029; 1:9UrTFZCds+bXbeQEuMFaYQ+JL77mxw309RZby9iR9XNjpzw6Fmr/Md2CMsBMi6vcMbrQWebMvlrFJiStE774IlyyG+l6N1tCixtfVlL1EisM1dOoAlsozwY4CEabqA+gVXRK/GkXnFqpHqQqn2Xkoyl0AeiwpnNVSluvuJ+UypLeU6ABi7bZWdIkAdJnS7IgG4mHGp88PSx1eVc+E1QZRvMwke797a+9Ok2nVF0aQ7kcZN8ThZ6h6Vv03aHsD5NfcHcUHuIW5UBn30sXfaliLnmm8W25HvW4U6waP6E/u1FrbX/Co51w/XpWbDA0uVn2b/ej+99xeWBvzasl9lBbcw== X-Forefront-Antispam-Report: CIP:66.129.239.19; CTRY:US; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10019020)(6009001)(2980300002)(24454002)(189002)(199003)(5001960100002)(105596002)(47776003)(76506005)(46406003)(57986006)(33716001)(19580395003)(69596002)(19580405001)(117636001)(11100500001)(1220700001)(23726003)(1096002)(586003)(5008740100001)(106466001)(76176999)(4001430100002)(81156007)(50986999)(50466002)(6806005)(97736004)(2950100001)(50226001)(97756001)(87936001)(77096005)(86362001)(110136002)(92566002)(189998001)(107886002)(62816006)(42262002); DIR:OUT; SFP:1102; SCL:1; SRVR:BN3PR0501MB1379; H:p-emfe01b-sac.jnpr.net; FPR:; SPF:SoftFail; PTR:InfoDomainNonexistent; MX:1; A:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0501MB1379; 2:mC+M5or8ANqfMQElrVxTbO+sbj8oE/Sy/C4mY3b6rIsrbF+tmEhzeSnPoHxi4k8ZmzR4+xMhLbnksbGgzJ9B7ixrQLwO//36k72VLohTuEP5L3qdmqYTmRd9yE1p5FKHdFzEcx0KzS3hf1GB+jlXTQ==; 3:14xUo1NpOEepLLDrnfl/0PA+WBLRliIF3bEipC7MWcWj7YgBLYLTbCsNx8bjIJ3ZYGyfm2W22AIfgNm41MUU6F5mVwyqzc2pZLRrA6zyn7gfvHK9IB+DdqB0ItOF2uIUSRLZh3sQP+tqS2lbUUWO4ZM9nkf1JQUTJHH/W9tTGQSbBoCj8QPEDYWEHVsinGseDE3ARyb8H1cms5RK5A5vE/EMD+MVj9xGbUySxoAqQeE=; 25:N7+Me8sfLC4MhDdMS7CCol+fBQVtu2wmZhGK5SLN/T5Jni+BSKtK64YPfRpsMw0gIKDBZD3ARHtEc5wv+p6uKIhFphGquzDfuKpeDDZH2eqnG14fAwFU2+KKf0xx6o42dwIVRMRZGKss7PlAne7dIsodwJENzm5VBhVHn3FlT/i73k7/AR3QTBJVzPwT5qnotFiIR9A0IT/gvd3sMmfdT+R62tc44ZLiQy/DmH5yYyQo/EtycC7KV46jj6DDf9mOLhXANXuY9dD5VvYmvYIAvg== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN3PR0501MB1379; X-Microsoft-Exchange-Diagnostics: 1; BN3PR0501MB1379; 20:wTc813n82oOEVBJ0P0LUCRzE2aed5s1c1UpjsSOx+GGYagzOiYU3gRWF4VG4sqcINVRqAn4Lgu5rMQCjHRlpBMkD3mzsh0dmnM4FOp3BhDdWLVYbrgY1kVoZNsNA8pHxB4IFdzyp/ccSWgMdY95msY2oXCiMdwsXhnZoMCtDXIfhfAb6v/vkOw6o9DhG07jUAkc9Zi/pOGs/ORW8nCvVFREZ5oMwfTvJ5018Avg0hGo88YfyPFkmIox9DzwC+zCzgihA1E3fBqHj+lfrvO8XsUBp0iUF0prdC4I+Kz66CTLJHoUNqrtZ/ZjIjKRaWs7nWjXKegWg+LWO4YiSIXSp0mXsIVw0hjEdudXSPEGPN4mPC2gyFbg7IE1ZX8ygBAIIcJivFPn6N55OsaNU/+Vf4JFMAz7jQcR97PnQv3TL3ZAS2pk6CIL/ZvmcawprSZRpGJjWCK+F7Gp2bdqy2tDOZJ1KbRdbjFfpinpDrLQLYJEZhr5wTFsWvgRe0fpRr4vx; 4:C5vTOUK4t7NMyID1sfwtdL/F6k9Ag68ddFUePDB/0H/RfublqduIMRdrkBQ42SD9YdZ2RS/TnKWARHp/3BGF2/vCyNu+kkH3Wv3c+04DtUGwicCee6zV5F0+eGeOqjhC030+njMpj/VNKMTX/wyVh7FElg406KhmFqPidLYtMZKnN3RiLEBJc1btoirrNDbm+ubHoG9wQJsclmcyCanxlvuS4cAjXwKE7ql/ymVBzNky5sJKSoKNc6RWzIZss18ee5UjR8sHarrxR4LBgDTd9gCTlhdwmHVadxIEDobuZTEXKKVKz6wPuSx/ehmApnhvP6i6d40g2VFzwiMPEkD1CXyjGEtqV8m60DlxaAHldhHQpSKDLOtH6HdMEgSKS1M2 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(10201501046)(3002001); SRVR:BN3PR0501MB1379; BCL:0; PCL:0; RULEID:; SRVR:BN3PR0501MB1379; X-Forefront-PRVS: 0802ADD973 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BN3PR0501MB1379; 23:NCEPYg4AYp3ShCIDSki84UCtjX9+5tbe4xWIGZ+?= =?us-ascii?Q?mPVHwK8QkGH1Wx0s4T10nuYNqvkpoKAVidVs8Aa4itcW8hPdnqd5/pLe+WWa?= =?us-ascii?Q?K8Aw/2QppOTx0kVMshThKuoSfun/BdmVdkD5jTzX5kPwX7ijCBPjvpI2CK1j?= =?us-ascii?Q?o9jDocVXN09z6H5CuxFlHVIOZIS2vxKmnss2UPPRMy0Iyz95G4zuacv2wk7l?= =?us-ascii?Q?x5oP3E2K/JUvdL+54md49rgCz4tfh99Ov8beLY2i1s3rE+8DKDkQt7+SWIY7?= =?us-ascii?Q?JnCrUTB3IkbVZkmAZ7YWQXxqhiglFxHHvhKcsFaB8ooqNfDcw8Y5eUcicJ92?= =?us-ascii?Q?M7+jHA2NtKcNGWzv80HFEIC6F/5o1GT8Bw01XRc38YsJ3hIXNaAF7aJdcnaU?= =?us-ascii?Q?IYJHb9QdYzBNaif0iYd2JKvjLRWUZSdjcr7ncEpVz+kV6d/SU00kt/FmDlyK?= =?us-ascii?Q?gvDOhc/F1ULY/QqyOKuuUqFWdLQoZoBPNQjOS1UTZI1ZRzMLkfqhBvD+DtHl?= =?us-ascii?Q?9IQoE8fEpBwrBLNWWHF1yWT0ruj7cKcA+8ux3S+8UFOUfwba8/dmgZxvPwxp?= =?us-ascii?Q?pzd6gNHGrv8paC5je1ifkoKl0YwvtK6LGLE/UYjJ67h0iIpnQc73jQE8JF6t?= =?us-ascii?Q?Q2PiLWMiWadIXuvsxcjtzzFiJJWOiqyMQU9CNsoQ6Xk1EAlDmUvvbvQmCM6x?= =?us-ascii?Q?d6s0aK8XyMcYxLz+XqPadBwV55t67PUXrGBH2FlsC8I79en2uXNY5LO3gwdE?= =?us-ascii?Q?DKRwU8I3zR8mjazqpDsBhUprALTq1MA6CcIngqPkf+XYgN4v7mT4vaeWPV88?= =?us-ascii?Q?ZRK9CDrn0CbxtjPK/n9dD2o6HnpptblHKn/8A8KnQOWFZjWZztrkrzWpQ1zJ?= =?us-ascii?Q?w8oC3JOv6Tsm4oMgF6viGKAl0+MccF0WrpCQQXO4x2YspYr9UrTm6iYc2S31?= =?us-ascii?Q?bI9+rgQ9hLxdmPfJgR8cu4YDQdT6nlCcWfS8m3YPxg18L6jY/4ozNNGtwbhT?= =?us-ascii?Q?qJqtA8/7nE/UqqtWM3AUMZPcwgtiQM8nSwea2QfgEXK4i5dXNNs12/eHBauO?= =?us-ascii?Q?OJGLUPtulfMNaRVFplPQS1Uaywb3LMu6RnN43G6XCVnJ0Ew+cmJ9miEBBA3t?= =?us-ascii?Q?3DAfvbEWIY4U=3D?= X-Microsoft-Exchange-Diagnostics: 1; BN3PR0501MB1379; 5:35AvF3rUBxFs2YpWGccg6CQpkT/ByyEAdeqXgh2X+PS2R8BnbjJY5WJXYsLk8GzTweeQ3f0xTfnpjXDmhP7sgzbVEZCNwp5bCm21X4cI0KTkwxcw5Wj6bSI+Ej6ZNWRxTzOK6bplzpqP0RaeFkZzDA==; 24:s+7jXNhjsFFcYIWC/zKu/HQ3tMR9hWn+FgMyB3ezctMQkxOJY+vfZHeokV1Q7lK95yjCblLoI129+8IIE8qRn7NrARhNTdfQUJ5aq6PztzA= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Dec 2015 20:04:13.3208 (UTC) X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.19]; Helo=[p-emfe01b-sac.jnpr.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR0501MB1379 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Dec 2015 20:04:23 -0000 Julian Elischer wrote: > > ABI variant. But since userspace tends to use the pthread objects in > > the layouts of the library objects, this causes serious ABI issues > > when mixing libraries built against different default versions of > > libthr. > > which is one reason I think symbol versioning is over-rated.. symbol versioning works fine - provided you have opaque structs. > just make a new library version. as previously noted, this rarely solves anything since to avoid problems you need to bump the major version of all shared libs, and you cannot do that for 3rd party code you don't even know exists. From owner-freebsd-threads@freebsd.org Sat Dec 26 23:44:32 2015 Return-Path: Delivered-To: freebsd-threads@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAA5AA52CA2; Sat, 26 Dec 2015 23:44:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 631A01735; Sat, 26 Dec 2015 23:44:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tBQNiQN5096105 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 27 Dec 2015 01:44:26 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tBQNiQN5096105 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tBQNiOA0096104; Sun, 27 Dec 2015 01:44:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 27 Dec 2015 01:44:24 +0200 From: Konstantin Belousov To: Daniel Eischen Cc: John Baldwin , freebsd-arch@freebsd.org, freebsd-threads@freebsd.org Subject: Re: libthr shared locks Message-ID: <20151226234424.GJ3625@kib.kiev.ua> References: <20151223172528.GT3625@kib.kiev.ua> <4199356.DlQeWDh27F@ralph.baldwin.cx> <5496837.TbTQtANDNj@ralph.baldwin.cx> <20151224191408.GA3625@kib.kiev.ua> <20151226105409.GH3625@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Dec 2015 23:44:32 -0000 On Sat, Dec 26, 2015 at 12:15:43PM -0500, Daniel Eischen wrote: > I think libthr has always done that (postpoining until critical > section is exited). But there are libc (and perhaps other) > locks that may be left locked on a fork(). Really (according > to POSIX), a fork() from a threaded program can only be used > to exec() something. libthr only started to do the interception in r212076, which was committed by David on Sep 1, 2010. Posix left the behaviour undefined after the fork, but real world rejects implementations which make C runtime broken after fork in mt process. Rtld must work in the child anyway (and in sighandler as well). We were forced to ensure also that malloc(3) and pthread_create(3) work in the child after the fork, and any transient regressions in this area where immediately reported. One of the biggest abusers of that are Qt/KDE. > I guess I am just not that worried about the ABI change when combined > with a library version bump. We use to do this for every new major > release, it isn't anything that hasn't been done before. We never > supported 2 thread libraries in the same executable, and it is > obvious when that happens (I think we even instrumented libc_r, > or was it rtld?, to emit a warning when that happened. My memory > is fuzzy). With poudriere, our new package system, I would think > we would be better off now than 10 years ago? We did significant work to avoid requiring complete recompilation of all user binaries on the major version updates. Part of it is the symbol versioning, another part is the stronger discipline and future planning when doing ABI-sensitive changes. At least starting with stable/8, when you binary depends only on the basic C runtime (i.e. rtld + libc + libm + libthr), you are no longer required to recompile it, and you do not need to install any compat libraries. Things just work. We are not there yet, since some libraries are not managed that good, e.g. libutil is not symver-ed but at least usually bumped correctly. Some libraries are handled much worse, e.g. libcam _interface_ depends on volatile kernel parts and libcam version is not always bumped when needed. Unfortunately, libcam is the dependency of high-profile user applications, from Gnome and KDE. This is what prevents us from stopping declaring 'recompile ports on major upgrade', but it is a goal. Still, basic C runtime is in much better shape WRT ABI stability than it was, say, in the 6.x-7.x time. This is why ino_t 64bit is moved with so much caution, and should also explain why I am so nervous to lock inlining. Doing libthr bump would break things. > > I do note that I was initially wrong in thinking that port revision > bumps could help alleviate any pain. A port could be built in > -stable with the revision bump just as well as in -current (or 11). > What would be needed is for pkg to understand that any port built > for 10 and previous could not run on 11+ (just to be on the safe > side). BTW, I tried to explain exactly this scenario in one of the previous replies. > > > OTOH, we should be able to plan such change, which requires much more > > drastic measures to be implementable. I started thinking about it, and > > I noted that what is needed is solved by renaming libthr to something > > else, e.g. libthr2. One of the issue which is not solved by the dso > > version bump (not to even mention versioned symbols bump) is the > > static linker bringing both libthr.so.3 and libthr.so.4 into the same > > process, by the mere use of -lpthread by different objects at different > > times. Then, libthr and hypothetical libthr2 should check and prevent > > activation of image which loads both. > > > > But these are longtime to evaluate and implement. The feature at hand > > does not require ABI change, as my patch demostrated. Yes, shared > > mutexes would be more naturally implemented with the inline locks, and > > avoidance of an indirection in the libthr is also good, but lets not mix > > significant ABI change and some less ambitious feature. My approach does > > not lock out future strategies. > > I agree, but the work that you are doing now would be basically > thrown out later on. I will not stand in your way and appreciate > any work you do. I would just rather that the struct change be > made now for 11, even without any pshared or other changes. For > once the struct change is made, pshared or other additions can > be made afterward, even in the 11 branch because they would not > break the ABI. Lock inlining was not done for ten years, now cost of doing it is extremely high, as discussed above. Who would drive the change, and with what time frame ? If me, I seriosly consider renaming libthr to libthr2, but I had no time to think much about it. Right now, I think that I want to commit my current patch and implement robust mutexes as the next step, without ABI breakage. At least, this seems to have fixed time-frame and can be made ready for 11.x. Lock inlining might be not. Are there serious objections against the plan, except that (lock inlining + pshared) is ideal situation, while the plan is not (but more practical) ?