From owner-freebsd-stable@freebsd.org  Mon Nov 21 18:30:00 2016
Return-Path: <owner-freebsd-stable@freebsd.org>
Delivered-To: freebsd-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FFBCC4D1E1
 for <freebsd-stable@mailman.ysv.freebsd.org>;
 Mon, 21 Nov 2016 18:30:00 +0000 (UTC)
 (envelope-from gpalmer@freebsd.org)
Received: from mail.in-addr.com (mail.in-addr.com
 [IPv6:2a01:4f8:191:61e8::2525:2525])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 1DF6BB24
 for <freebsd-stable@freebsd.org>; Mon, 21 Nov 2016 18:30:00 +0000 (UTC)
 (envelope-from gpalmer@freebsd.org)
Received: from gjp by mail.in-addr.com with local (Exim 4.87 (FreeBSD))
 (envelope-from <gpalmer@freebsd.org>)
 id 1c8tLq-000Kr6-Br; Mon, 21 Nov 2016 18:29:58 +0000
Date: Mon, 21 Nov 2016 18:29:58 +0000
From: Gary Palmer <gpalmer@freebsd.org>
To: Pete French <petefrench@ingresso.co.uk>
Cc: freebsd-stable@freebsd.org
Subject: Re: Help! two machines ran out of swap and corrupted their zpools!
Message-ID: <20161121182958.GC99344@in-addr.com>
References: <E1c8sgj-0003Q2-It@dilbert.ingresso.co.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <E1c8sgj-0003Q2-It@dilbert.ingresso.co.uk>
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: gpalmer@freebsd.org
X-SA-Exim-Scanned: No (on mail.in-addr.com); SAEximRunCond expanded to false
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, 
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 21 Nov 2016 18:30:00 -0000

On Mon, Nov 21, 2016 at 05:47:29PM +0000, Pete French wrote:
> So, I am off sick and my colleagues decided to load test our set of five
> servers excesively. All ran out of swap. So far so irritating, but whats has
> happened is that twoof them now will not boot, as it appears the ZFS pool
> they are booting from has become corrupted.
> 
> One starts to boot, then crases importing the root pool. The other doenst
> even get that far with gptzfsboot saying it can't find the pool to boot from!
> 
> Now I can recover these, but I am a bit worried, that it got like this at
> all, as I havent ever seen ZFS corrupt a pool like this. Anyone got any insights,
> or suggstions as to how to stop it happening again ?
> 
> We are swapping to a separate partition, not to the pool by theway.

Silly question - have you checked that the swap partition does not
overlap your boot pool partition?  It could well be that the end of
the swap partition intrudes into the affected ZFS pool

Gary