From owner-freebsd-current@FreeBSD.ORG Wed Jan 23 10:52:44 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 42C3B16A417 for ; Wed, 23 Jan 2008 10:52:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from relay02.kiev.sovam.com (relay02.kiev.sovam.com [62.64.120.197]) by mx1.freebsd.org (Postfix) with ESMTP id CA94E13C468 for ; Wed, 23 Jan 2008 10:52:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from [212.82.216.226] (helo=skuns.kiev.zoral.com.ua) by relay02.kiev.sovam.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1JHdDl-000LYb-Ju for freebsd-current@freebsd.org; Wed, 23 Jan 2008 12:52:42 +0200 Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by skuns.kiev.zoral.com.ua (8.14.1/8.14.1) with ESMTP id m0NAq8BG054728 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 23 Jan 2008 12:52:08 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2) with ESMTP id m0NAqPEU069628; Wed, 23 Jan 2008 12:52:25 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.2/8.14.2/Submit) id m0NAqOf7069627; Wed, 23 Jan 2008 12:52:24 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 23 Jan 2008 12:52:24 +0200 From: Kostik Belousov To: d@delphij.net Message-ID: <20080123105224.GY57756@deviant.kiev.zoral.com.ua> References: <4796801C.7010601@delphij.net> <20080123051215.GU57756@deviant.kiev.zoral.com.ua> <4796D7C5.3020307@delphij.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="QD2trd1dTxsoIJGg" Content-Disposition: inline In-Reply-To: <4796D7C5.3020307@delphij.net> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: ClamAV version 0.91.2, clamav-milter version 0.91.2 on skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Scanner-Signature: eacc24c597fa83a26e5f02f00d1ff66c X-DrWeb-checked: yes X-SpamTest-Envelope-From: kostikbel@gmail.com X-SpamTest-Group-ID: 00000000 X-SpamTest-Info: Profiles 2033 [Jan 23 2008] X-SpamTest-Info: helo_type=3 X-SpamTest-Info: {received from trusted relay: not dialup} X-SpamTest-Method: none X-SpamTest-Method: Local Lists X-SpamTest-Rate: 0 X-SpamTest-Status: Not detected X-SpamTest-Status-Extended: not_detected X-SpamTest-Version: SMTP-Filter Version 3.0.0 [0255], KAS30/Release Cc: FreeBSD Current Subject: Re: RELENG_7_0: vm_thread_new: kstack allocation failed X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jan 2008 10:52:44 -0000 --QD2trd1dTxsoIJGg Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jan 22, 2008 at 09:59:33PM -0800, Xin LI wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 >=20 > Kostik Belousov wrote: > > On Tue, Jan 22, 2008 at 03:45:32PM -0800, Xin LI wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> Hi, > >> > >> I have got a lot of this in dmesg output for RELENG_7_0 as of today: > >> > >> vm_thread_new: kstack allocation failed > >> vm_thread_new: kstack allocation failed > >> vm_thread_new: kstack allocation failed > >> vm_thread_new: kstack allocation failed > >> vm_thread_new: kstack allocation failed > >> vm_thread_new: kstack allocation failed > >> > >> Any idea? > >=20 > > Does it cause any problems aside from printing these messages ? >=20 > It causes some fork() to fail. >=20 > > What workload do you put on the machine ? >=20 > It was an rsync from NFS to ZFS with ~15M of files, and rsync will > consume basically all physical memory. I end up with some 2GB active, > 4GB wired thing. (The system has 8GB of RAM), and I added a "make -j9 > buildworld" into the chaos to see if things get worse, and it did :-) >=20 > > The messages came from the failure of the kernel to allocate address > > space for the kernel stack for a thread being created. Previously, the > > system would panic encountering this situation. >=20 > Yes, I knew, previously it just panic and hangs there, and thanks a lot > for fixing it =3D-) >=20 > > This may happen due to kernel_map address space depletion, for instance, > > by having a lot (on i386 machines with > 1Gb memory, ~40000) threads. >=20 > It seems that I have hit some sort of "leak" or some exhaustion issue. > Say, when the workload is gone, the system did not recover from the > situation, and reboot worked fine. >=20 > The system is sort of in production and it is about 20 miles away from > my office. Do you want me to do some experiments for this? Yes, I want to know what exactly leaked. Ideally, I would like to see the series of the output of the vmstat -z and vmstat -m for some time before the system is bogged down. But, even the one snapshot of the vmstat -z/-m output immediately before things stop working would be good to look at. Output of the ps auxwwH is helpful too. --QD2trd1dTxsoIJGg Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkeXHGgACgkQC3+MBN1Mb4iQbwCg79R8T9RFPR5tXS4ZjoaVVKUz 4PUAn2YsAkbFnh1WnFwv7+zo7SdfeJhs =lY8d -----END PGP SIGNATURE----- --QD2trd1dTxsoIJGg--