From owner-freebsd-stable@freebsd.org Mon Nov 21 17:59:00 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FE9BC4C3DA for ; Mon, 21 Nov 2016 17:59:00 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-yw0-x232.google.com (mail-yw0-x232.google.com [IPv6:2607:f8b0:4002:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E2789DF9 for ; Mon, 21 Nov 2016 17:58:59 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: by mail-yw0-x232.google.com with SMTP id r204so218696553ywb.0 for ; Mon, 21 Nov 2016 09:58:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=oEpBhcj3BeNmBQSz+OZOiHeGLwGXULd2TSai4lh7AFA=; b=SiMl7Hh2XYsDdP4IUPtgso5Ss+r8ZSLqzUvzl2BD4ni+95HCG1dCQt99Mq0c09c83B 3dntOho6ryrIw0hVuW+AsBQqQtbzlCQfvIBAT0FysOFaSRAtbZWX6TOTRBpIv9uM+LWN RnhrnzirPn6a2OaaznL5n+juUuu+Obj7hOLNlxH+Tg6S4Zbt4+nEg0E9FjwyUKa1KO2R GqbVOHSvmoufCI6G19SikVdhMugNcrCIggrNC/mEJkieN9+SsBowC506OzyseU3uFVca 8B9PPwzeQNROnztU19zKeXpBEUdX8HZwEqLi5ngjVuBL1znfHkn+4NGVi8Jzdo4ZGDqk cSEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=oEpBhcj3BeNmBQSz+OZOiHeGLwGXULd2TSai4lh7AFA=; b=EZFjc+KAF5xDam/TLsfCYALW5T532E4nYBiek/oh4DCLwk0IgyImg4u7q3fQxa0Nls vN0F0TBuHUYNajYp+5Hdn+WevNWfwxBhkCjY1Y+U/9jViKToLoPlm6ScIzWeBvmffNLO FrVpEssRciW+wn39JSqoJlbaUTmZQjQR7NUygY1Ljv0t5w5evCh5VQh4/B3cCxqHGwz9 2MUtSOourbf8igSRDqMCU4aiq0R0pJ2rFhCokA1KvP4yqurKuXDgrbGcn8ex8+S3oPh7 XR4y3hPNtfuefovynWU/3QATi/UIEuOvUsEGzjul3XpACxalwDFVl43xHZ5q4pZi3ukU OuTA== X-Gm-Message-State: AKaTC01cBk6mStRPQOPzkRKapqDXkCfNFjdpk6qImRPvzcD3p5Mb/N4IU+A2+LIY21MIRFOkJTRZBjlain3B/Q== X-Received: by 10.129.60.11 with SMTP id j11mr14444179ywa.246.1479751138590; Mon, 21 Nov 2016 09:58:58 -0800 (PST) MIME-Version: 1.0 Received: by 10.129.153.206 with HTTP; Mon, 21 Nov 2016 09:58:58 -0800 (PST) X-Originating-IP: [24.44.110.108] In-Reply-To: References: From: Mark Saad Date: Mon, 21 Nov 2016 12:58:58 -0500 Message-ID: Subject: Re: Help! two machines ran out of swap and corrupted their zpools! To: FreeBSD-Stable ML Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 17:59:00 -0000 On Mon, Nov 21, 2016 at 12:50 PM, Jan Bramkamp wrote: > On 21/11/2016 18:47, Pete French wrote: >> >> So, I am off sick and my colleagues decided to load test our set of five >> servers excesively. All ran out of swap. So far so irritating, but whats >> has >> happened is that twoof them now will not boot, as it appears the ZFS pool >> they are booting from has become corrupted. >> >> One starts to boot, then crases importing the root pool. The other doenst >> even get that far with gptzfsboot saying it can't find the pool to boot >> from! >> >> Now I can recover these, but I am a bit worried, that it got like this at >> all, as I havent ever seen ZFS corrupt a pool like this. Anyone got any >> insights, >> or suggstions as to how to stop it happening again ? >> >> We are swapping to a separate partition, not to the pool by theway. > > > How much trust do you put in your hardware? Have you ever put the hardware > under full load for extended periods before e.g. run poudriere to build pkg > repos? > > -- Jan Bramkamp > > _________ Pete I am thinking like Jan, that this points to a hardware issue. What can you tell us about the servers ? What sort of cpu, do you have ecc ram ? How much ram what was running at the time ? Do you use a slog, l2arc , whats the zpool status -v show? ______________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" -- mark saad | nonesuch@longcount.org