From owner-svn-src-all@freebsd.org Wed Dec 20 00:34:30 2017 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 034EEE862BD; Wed, 20 Dec 2017 00:34:30 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from c.mail.sonic.net (c.mail.sonic.net [64.142.111.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DFDAF735B3; Wed, 20 Dec 2017 00:34:29 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from helicon.physics.ucla.edu (helicon.physics.ucla.edu [169.232.156.253]) (authenticated bits=0) by c.mail.sonic.net (8.15.1/8.15.1) with ESMTPSA id vBK0YPw4009889 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 19 Dec 2017 16:34:26 -0800 Subject: Re: svn commit: r326758 - in head/sys/i386: conf include To: John Baldwin , Eugene Grosbein , Konstantin Belousov , Conrad Meyer Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <201712110432.vBB4WbnE021090@repo.freebsd.org> <20171211091943.GF2272@kib.kiev.ua> <5A2E5D44.9030904@grosbein.net> <4a9c76c9-8063-9420-b198-14487b089840@FreeBSD.org> From: Nathan Whitehorn Message-ID: <9b4b021b-dbf5-b788-2e23-6a49efa95da8@freebsd.org> Date: Tue, 19 Dec 2017 16:34:25 -0800 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <4a9c76c9-8063-9420-b198-14487b089840@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Sonic-CAuth: UmFuZG9tSVZBX1ky9ANZODUU+Fr5DEkYXnT+PRhdax6eloy8PokTIYoPjwrcFwUVRux8U4fe0cr6qJVqXCN66KkdjeR7G6xkGNP2Da4Xr08= X-Sonic-ID: C;2tMyiB3l5xGiZusnWtmBlw== M;apCViB3l5xGiZusnWtmBlw== X-Spam-Flag: No X-Sonic-Spam-Details: 0.0/5.0 by cerberusd X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Dec 2017 00:34:30 -0000 On 12/12/17 11:32, John Baldwin wrote: > On 12/11/17 5:26 AM, Eugene Grosbein wrote: >> On 11.12.2017 16:19, Konstantin Belousov wrote: >>> On Mon, Dec 11, 2017 at 04:32:37AM +0000, Conrad Meyer wrote: >>>> Author: cem >>>> Date: Mon Dec 11 04:32:37 2017 >>>> New Revision: 326758 >>>> URL: https://svnweb.freebsd.org/changeset/base/326758 >>>> >>>> Log: >>>> i386: Bump KSTACK_PAGES default to match amd64 >>> i386 is not amd64, the change is wrong. >>> >>> i386 has the word size two times smaller than amd64, which makes typical >>> frame smaller by 30-40% over same code on amd64. Also i386 has much >>> smaller available KVA size (tens of MB) and KVA fragmentation is both >>> more severe and more fatal due to this. I expect that your change will >>> make any non-trivial load which creates enough threads to either fail >>> randomly or deadlock. >>> >>> If somebody tries to fit large load onto i386 machine, he must know what to >>> do and how to configure the kernel to adapt to the load (which does not >>> require the recompilation). >> Its very easy to get kernel stack overflow with 11-STABLE/i386 >> without any significant load due to abuse of kernel stack in many kernel subsystems >> as shown in the https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476 >> >> Contrary, "enough threads" seems to be very non-trivial number of threads >> and pretty unusual load pattern for i386 as I run several such systems >> with kern.kstack_pages=4 for quite long time and have no problems. >> No random fails, no deadlocks. And with 2 pages only 11-STABLE/i386 is just unusable >> if one utilizes SCTP, IPv6, some WiFi connectivity, IPSEC or even very small ZFS pool. >> >> I still wonder if there is really such load pattern that creates "enough threads" >> for i386 to make 4-pages stack troublesome. > I suspect two things are at play in running out of stack in 10.x and later. > 1) Usage of XSAVE for AVX, etc. state uses more of the kstack (as kib@ alludes > to), and 2) clang. Certainly for MIPS I have found that compiling with clang > instead of gcc for mips64 gives a kernel that panics for stack overflow for any > use of NFS. It might be that this is due to something MIPS-specific, but it > might be worthwhile retesting with kstack_pages=2 and building the kernel > with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package. For what it's worth, I see the same NFS-related stack overflows on mips64 with in-tree GCC 4.2.1. -Nathan