From owner-freebsd-doc@FreeBSD.ORG Sat Sep 20 17:07:06 2014 Return-Path: Delivered-To: doc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 51998908; Sat, 20 Sep 2014 17:07:06 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CBC8DDE; Sat, 20 Sep 2014 17:07:05 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8KH6xBo096804 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 20 Sep 2014 20:06:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8KH6xBo096804 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8KH6wFY096803; Sat, 20 Sep 2014 20:06:58 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 20 Sep 2014 20:06:58 +0300 From: Konstantin Belousov To: John Baldwin Subject: Re: libthr and main thread stack size Message-ID: <20140920170658.GE2210@kib.kiev.ua> References: <53E36E84.4060806@ivan-labs.com> <20140916081324.GQ2737@kib.kiev.ua> <5242716.s4iaScq0Bu@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="pAwQNkOnpTn9IO2O" Content-Disposition: inline In-Reply-To: <5242716.s4iaScq0Bu@ralph.baldwin.cx> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: "Justin T. Gibbs" , "Ivan A. Kosarev" , freebsd-current@freebsd.org, doc@freebsd.org X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Sep 2014 17:07:06 -0000 --pAwQNkOnpTn9IO2O Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 19, 2014 at 03:27:25PM -0400, John Baldwin wrote: > I suspect it was done out of reasons of being overly conservative in > interpreting RLIMIT_STACK. I think it is quite surprising behavior > though and would rather we make your option the default and implement > what the Open Group says above. Ok, below is the patch. I felt bad about adding yet another magic and undocumented tunable to our libthr. Since there seems to be no alternative than a tunable to enforce old behaviour, I documented the quirks I am aware of. Doc people, please review the man page in the patch. diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c index 9bf0e29..72a067a 100644 --- a/lib/libthr/thread/thr_init.c +++ b/lib/libthr/thread/thr_init.c @@ -445,7 +445,7 @@ init_private(void) struct rlimit rlim; size_t len; int mib[2]; - char *env; + char *env, *env_bigstack, *env_splitstack; =20 _thr_umutex_init(&_mutex_static_lock); _thr_umutex_init(&_cond_static_lock); @@ -473,8 +473,9 @@ init_private(void) len =3D sizeof (_usrstack); if (sysctl(mib, 2, &_usrstack, &len, NULL, 0) =3D=3D -1) PANIC("Cannot get kern.usrstack from sysctl"); - env =3D getenv("LIBPTHREAD_BIGSTACK_MAIN"); - if (env !=3D NULL) { + env_bigstack =3D getenv("LIBPTHREAD_BIGSTACK_MAIN"); + env_splitstack =3D getenv("LIBPTHREAD_SPLITSTACK_MAIN"); + if (bigstack !=3D NULL || env_splitstack =3D=3D NULL) { if (getrlimit(RLIMIT_STACK, &rlim) =3D=3D -1) PANIC("Cannot get stack rlimit"); _thr_stack_initial =3D rlim.rlim_cur; diff --git a/share/man/man7/libthr.7 b/share/man/man7/libthr.7 new file mode 100644 index 0000000..16d916f --- /dev/null +++ b/share/man/man7/libthr.7 @@ -0,0 +1,254 @@ +.\" Copyright (c) 2014 The FreeBSD Foundation, Inc. +.\" All rights reserved. +.\" +.\" This documentation was written by Konstantin Belousov +.\" under sponsorship from the FreeBSD Foundation. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PUR= POSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIAB= LE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUEN= TIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, ST= RICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY = WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd September 20, 2014 +.Dt libthr +.Os +.Sh NAME +.Nm libthr +.Nd FreeBSD implementation of the Posix threading library +.Sh LIBRARY +.Lb libpthread +.Sh DESCRIPTION +The man page documents the quirks and tunables of the +.Fx +implementation for the +.Lb libpthread . +When linking with the +.Li -lpthread , +the run-time dependency +.Dv libthr.so.3 +library is recorded in the produced object. +.Pp +The library is tigthly integrated with the Run-time Link-editor +.Xr ld-elf.so.1 1 +and +.Lb libc , +all three components must be built from the same source tree. +Mixing +.Dv libc.so +and +.Nm +libraries from different versions of +.Fx +is not supported. +The run-time linker +.Li ld-elf.so.1 +has some code to ensure backward-compatibility with older +.Nm . +.Sh MUTEX ACQUISITION +The locked mutex (see +.Xr pthread_mutex_lock 3 ) +is represented by a volatile variable of type +.Dv lwpid_t , +which records the global system identifier of the thread +owning the lock. +The +.Nm +performs a congested mutex acquisition in three stages, each of which +is more resource-consuming than the previous. +.Pp +First, the +.Li spin loop +is performed, where the library attempts to acquire the lock by +.Xr atomic 9 +operations. +The loop count is controlled by the +.Ev LIBPTHREAD_SPINLOOPS +environment variable. +.Pp +If the +.Li spin loop +was unable to acquire the mutex, the +.Li yield loop +is executed, performing the same +.Xr atomic 9 +acquisition attempts as +.Li spin loop , +but each attempt is followed by yield of the CPU time of the thread by +.Xr sched_yield 2 +syscall. +By default, the +.Li yield loop +is not executed. +This is controlled by +.Ev LIBPTHREAD_YIELDLOOPS +environment variable. +.Pp +If both +.Li spin +and +.Li yield loops +failed to acquire the lock, the thread is taken off the CPU and +put to sleep in kernel with the +.Xr umtx 2 +syscall. +Kernel wakes up a thread and hands the ownership of the lock to +the woken thread. +.Sh THREADS STACKS +Each thread is provided with the private stack area used by C runtime. +The size of the main (initial) thread stack is set by kernel, and is +controlled by the +.Dv RLIMIT_STACK +process resource limit (see +.Xr getrlimit 2 ) . +.Pp +By default, the main thread size is equal to the value of resource +.Dv RLIMIT_STACK +for the process. +If the +.Dv LIBPTHREAD_SPLITSTACK_MAIN +environment variable is present (its value does not matter), +the main thread size if chomped to 4MB on 64bit architectures, and to +2MB on 32bit architectures, on the threading library initialization. +The rest of the address space area reserved by the kernel for initial +process stack, is used for non-initial threads stack in this case. +The presence of the +.Dv LIBPTHREAD_BIGSTACK_MAIN +environment variable overrides the +.Dv LIBPTHREAD_SPLITSTACK_MAIN , +it is kept for backward-compatibility. +.Pp +The size of the stacks for threads created by the process at run-time +with the +.Xr pthread_create 3 +call, is controlled by thread attributes, see +.Xr pthread_attr 3 , +in particular, the +.Xr pthread_attr_setstacksize 3 , +.Xr pthread_attr_setguardsize 3 +and +.Xr pthread_attr_setstackaddr 3 . +If no attributes for the thread stack size are specified, the default +non-initial thread stack size is 2MB for 64bit architectures, and 1MB +for 32bit architectures. +.Sh RUN-TIME SETTINGS +The following environment variables are recognized by +.Dv libthr +and adjust the operation of the library at run-time: +.Bl -tag -width LIBPTHREAD_SPLITSTACK_MAIN +.It Ev LIBPTHREAD_BIGSTACK_MAIN +Disables the chomp of the initial thread stack, enabled by +.Ev LIBPTHREAD_SPLITSTACK_MAIN . +.It Ev LIBPTHREAD_SPLITSTACK_MAIN +Causes the chomp of the initial thread stack, as described in the +section +.Li THREAD_STACKS . +This was the default behaviour of the +.Nm +before +.Fx 11.0 . +.It Ev LIBPTHREAD_SPINLOOPS +The integer value of the variable overrides the default count of +iterations in the +.Li spin loop +of the mutex acquisition. +The default count is 2000, set by the +.Dv MUTEX_ADAPTIVE_SPINS +define in the +.Nm +sources. +.It Ev LIBPTHREAD_YIELDLOOPS +The non-zero integer value of the variable allows the +.Li yield loop +in the process of the mutex acquisition. +The value is the counter of loop operations. +.It Ev LIBPTHREAD_QUEUE_FIFO +The integer value of the variable specifies how often the blocked +threads are put into the head of the sleep queue, instead of it tail. +The bigger value reduces the frequency of the FIFO discipline. +The value must be between 0 and 255. +.El +.Sh INTERACTION WITH RUN-TIME LINKER. +The +.Nm +library must appear before +.Dv libc +in the global order of depended objects. +.Pp +.Pp +Loading the +.Nm +library with the +.Xr dlopen 3 +call in the process after the program binary is activated, +is not supported, and causes miscellaneous and hard to diagnose misbehavio= ur. +This is due to +.Nm +interposing several important +.Dv libc +symbols to provide thread-safe services. +In particular, +.Dv errno +and locking stubs from +.Dv libc +are affected. +This requirement is currently not enforced. +.Pp +If the program loads the modules at run-time, and modules may require +the threading services, the main program binary must be linked with +.Dv libpthread , +even if it does not require any service from the library. +.Pp +The library cannot be unloaded, the +.Xr dlclose 3 +function does not perform any action when called with a handle for +.Dv libthr . +One of the reason is that interposing of +.Dv libc +functions cannot be undone. +.Sh SIGNALS +The implementation also interposes the user-installed +.Xr signal 3 +handlers. +The interposing is done to postpone signal delivery to threads which +entered the (libthr-internal) critical sections, where the calling +of the signal handler is unsafe. +Example of such situation is owning the internal library lock. +When the signal is delivered while signal handler cannot be safely +called, call is postponed and performed after the critical section +is left. +This should be taken into account when interpreting the +.Xr ktrace 1 +logs. +.Sh SEE ALSO +.Xr atomic 9 , +.Xr dlclose 3 , +.Xr dlopen 3 , +.Xr errno 3 , +.Xr getenv 3 , +.Xr getrlimit 2 , +.Xr ktrace 1 , +.Xr ld-elf.so.1 1 , +.Xr libc 3 , +.Xr pthread_attr 3 , +.Xr pthread_attr_setstacksize 3 , +.Xr pthread_create 3 , +.Xr signal 3 , +.Xr umtx 2 . --pAwQNkOnpTn9IO2O Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUHbQxAAoJEJDCuSvBvK1BJowQAIkGc0gokVCcaGUUXT+WPkuE 2u49WokeTewPizBT0040afss1FKeyeDKI5Gn+mogNl0IIh7XCOLNpL0B3qFtWf44 Ie3hkuuSN+p47e1auaN7KBuRAyWVHCVdJsXsEEO/MiJT8tqVt+fZiLfvSmNAWrse 9hgYupxmQRsNRRTz6m22zdRxAWZVhvM56pt/oFaHbM5DGXw2C9nNqg3EKZjnpqtc A7uXJelz95k5T/mXK0yxr/aoG/rHv4ON57slvp1cRD+6PiSlsOsR3TWYRPmUgFZO KEjBknEhSsuaXlh02fjMbY/Ht0r9VqTdijmYffnwxmF3cn4flCewn86zO+6mGo9B 162AS5XpxNMJsudCK/sP9+4VyxICB5A/V7LXESBNYVt67Et6jJBNBZXu50xbMTU+ l5wrx2mMc4XlDQZZM+KRnynzxF3B8FoAUBOvQV2Rw58hLqooJ1HR5rf1HJIdZ2yw nJr7GH6hIsLYOTDGLHiZpMAYJMwgsAlm++ZPqQAf6a8P3Fqs9A0X40BC07JHS+G5 dCSJgtFd9nzuo75V7qyFpCZVPM4+q9jayWILihcrdjFi6ZuDDpMLjuqaxRHVVP/G dw9FiV/W8u+Gx2cFdwUPcYQTbxgjgb5DFuTnmln9pRqiWTndwqk7hM6foMEG8qT9 f3QBuQLeUJrKqWpNAzfB =z6GS -----END PGP SIGNATURE----- --pAwQNkOnpTn9IO2O--