From owner-svn-src-head@freebsd.org Mon Dec 11 19:33:41 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 037C1E9BE69 for ; Mon, 11 Dec 2017 19:33:41 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A32DB7521E; Mon, 11 Dec 2017 19:33:40 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBBJXVp2008451 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 11 Dec 2017 20:33:32 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: rgrimes@freebsd.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBBJXRTT081930 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 12 Dec 2017 02:33:27 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: svn commit: r326758 - in head/sys/i386: conf include To: rgrimes@freebsd.org References: <201712111912.vBBJCaFO082664@pdx.rh.CN85.dnsmgr.net> Cc: Michael Tuexen , Konstantin Belousov , Conrad Meyer , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org From: Eugene Grosbein Message-ID: <5A2EDD82.1040404@grosbein.net> Date: Tue, 12 Dec 2017 02:33:22 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <201712111912.vBBJCaFO082664@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Dec 2017 19:33:41 -0000 12.12.2017 2:12, Rodney W. Grimes wrote: >>> Not sure about IPSEC and ZFS. But I was NOT able to reproduce the issue by just using >>> an SCTP association on a 32-bit VM using FreeBSD 11.1. So right now I don't know what >>> is wrong and therefore what could be fixed... >> >> You just got lucky. Kernel stack abuse by multiple parts of FreeBSD kernel it matter of fact. >> But it is not like you always get a panic because sometimes network traffic >> is passed through NETISR queues to reduce stack usage and sometimes it is not, >> depending on several run-time factors. You may find it stable with one NIC driver >> and crashing with another even just because of different stack usage. >> > > I agree with that, there is and has been a kernel stack space issue on > i386 for some time. But lets all agree just cranking the size of the > kernel stack up is a poor long term solution as the larger you crank > this the less number of processes you can have due to limited KVA. OTOH, it seems to be totally unrealistic to hope the problem can be solved in feasible future as such crashes were not fixed for several *years* (at least since 10.x era). > So, first off, ZFS, well that is fairly well documented to not be > a good idea on i386, and well documented to be a bad idea with only > 4G of memory. That's just incorrect. With 4 kstack pages, you could utilize ZFS for compressed src/obj/ports with virtualized FreeBSD 9 and 1GB memory using following loader.conf settings: zfs_load="YES" vfs.zfs.arc_max="40M" vfs.zfs.vdev.cache.size="8M" vfs.zfs.prefetch_disable="1" And that was pretty stable as ARC was really limited. This was broken in 10.x and later so that ARC grows order of magnitude (x10) larger, though. > So make sure that this stays documented, along with > what one should tweak to make it work, most of which is already in > https://wiki.freebsd.org/ZFSTuningGuide#i386 > someone please verify that you really need to increase KSTACK, cause > last time I played with ZFS on i386 I just used that tuning guide and > all was happy. If it is found necessary to add KSTACK pages, document > that above and include in that document the side effects of doing so. > > As far as IPSEC and SCTP, maybe someone can take a once over through > the code looking for stack abuse with local variables that > really should be malloced, passing large stuff in function call > arguments rather than a pointer. > > Possibly even doing some dtrace to find out how close we get to the limits. That's not just ZFS, IPSEC or SCTP. It spreads all over kernel parts: wifi subsystem and drivers, libkern/zlib, crypto, scsi/cam/da/ada drivers, vm subsystem itself, NFS etc. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476