From owner-freebsd-stable@freebsd.org Tue Feb 2 20:07:55 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2A2EA99D09 for ; Tue, 2 Feb 2016 20:07:55 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8904D10E7 for ; Tue, 2 Feb 2016 20:07:55 +0000 (UTC) (envelope-from peter@rulingia.com) Received: by mailman.ysv.freebsd.org (Postfix) id 86AC0A99D05; Tue, 2 Feb 2016 20:07:55 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C8CAA99D04 for ; Tue, 2 Feb 2016 20:07:55 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (unknown [IPv6:2001:388:f000::349d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.rulingia.com", Issuer "Let's Encrypt Authority X1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B73010E5; Tue, 2 Feb 2016 20:07:54 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (c122-106-195-17.belrs5.nsw.optusnet.com.au [122.106.195.17]) by vps.rulingia.com (8.15.2/8.15.2) with ESMTPS id u12K7iZO041829 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 3 Feb 2016 07:07:51 +1100 (AEDT) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id u12K7dGW062876 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 3 Feb 2016 07:07:39 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id u12K7cNn062875; Wed, 3 Feb 2016 07:07:38 +1100 (AEDT) (envelope-from peter) Date: Wed, 3 Feb 2016 07:07:38 +1100 From: Peter Jeremy To: Hajimu UMEMOTO Cc: stable@FreeBSD.org, mckusick@FreeBSD.org Subject: Re: 10-STABLE hangups frequently Message-ID: <20160202200738.GA78969@server.rulingia.com> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="3MwIy2ne0vdjdPXF" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.24 (2015-08-30) X-Greylist: Sender succeeded STARTTLS authentication, not delayed by milter-greylist-4.4.3 (vps.rulingia.com [103.243.244.15]); Wed, 03 Feb 2016 07:07:51 +1100 (AEDT) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Feb 2016 20:07:55 -0000 --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2016-Feb-02 16:55:46 +0900, Hajimu UMEMOTO wrote: >I'm disturbed by a frequent hangup of my 10-STABLE boxes since this >year. It seems occur during running the periodic daily scripts. >I've narrowed which commit causes this problem. It seems r292895 >causes it. I see many `Resource temporarily unavailable' message just >before hangup occurs. >Any idea? As others have said, you need to provide lots more detail on your configuration. That said, I'm seeing something potentially similar on a Google Compute Engine f1-micro instance (1 vCPU, 0.6GB RAM) that is running FreeBSD 10-stable/amd64 with ZFS but basically idle. (Yes, I realize that's very little RAM for ZFS but I previously had no problems with things like buildworld). There were no problems at r290231 but after I upgraded to r295005, I started seeing "out of swap" errors and hangs during the periodic daily runs. I'm not seeing this on 1GB instances - though they are all running UFS. Some experimentation suggested that just "find /" was enough to wedge my system. I did some experimenting and found that the following loader config was enough to prevent it hanging: vfs.zfs.arc_max=3D"128M" vfs.zfs.arc_meta_limit=3D"50M" vfs.zfs.arc_min=3D"25M" (previously, I had no ZFS tuning at all). One odditity was that I would semi-regularly see: kernel: pid 67431 (ntpd), uid 0, was killed: out of swap space I haven't worked out why the OOM killer preferred ntpd to anything else - it didn't seem to be bigger. And I didn't see any signs that swap space was being consumed (though I haven't done a scientific examination). (Note that swap is on a raw partition). The behaviour is definitely a regression and my initial suspicion is ZFS, though I haven't identified any smoking gun. Unfortunately, GCE only offers read access to the console, so I can't use DDB to poke around after it wedges. --=20 Peter Jeremy --3MwIy2ne0vdjdPXF Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJWsQyKXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs0b80QAJJuXHGlnlpnAmKoh9X3Tejt 0jZuhQ9zHwQJJAJ1c8eZsROZXsrJSMyAaLoUXsp+t0vFT/3VHZ9+vBC0XyaO3ScW wcFbZvCCjoPg0EdqgDJ0oibscJBYMxJUtK5tsoH9pDL0rOsi9/vjnCU1jH60mubA O+Knrt/fTdbrn5B+gbxAz4Nlsl3j3u5FuHJWX0u45PpEOHi6yKkCBhd56QqhtyuC itZ289sC7c3ddZKGejMf8o+Yt0yYMljXY14Eb5N7bAzSEdvLGySX8Nn40bN/UBce cv1QPOuq0y8UKGdofxzhgpmFzKi/wGKTkY/MJfDW027M3gLP2pYFGuAoUCP+cviX +7b5C3LgQxMNBNkat9L4vapkDE23iWwIwukqh2r9Pdi4h3UQfEuRbVgDcoZQatg0 slundqkP4qk/XBKCirfK8ij2Yj1QylC/rdpggoECJM+2q1nkuG8gR50KMRwTj32u zpPHcRN+iTWRfcFqvFelxv3qYJ+4tVTZRjI+TxlKZLoLzoutq56NznzGfqzp+Kqm SB7ScvCHwIzBsmzKWzVQ2E2IGkxkotXAD6+WFcIzQdQpzpJEN05qZzfBmyMEKnw8 +j994kx6iC0GoIAxVte5kmEHfPTBtNR5IIx5oCUlWepRIz69dWH7jWvwqKDRnpH9 /EmuA3vhk4x8E+CfJWMC =M0kL -----END PGP SIGNATURE----- --3MwIy2ne0vdjdPXF--