From owner-freebsd-hubs@FreeBSD.ORG  Mon Mar 13 08:31:14 2006
Return-Path: <owner-freebsd-hubs@FreeBSD.ORG>
X-Original-To: hubs@FreeBSD.org
Delivered-To: freebsd-hubs@FreeBSD.ORG
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id DF06416A400;
	Mon, 13 Mar 2006 08:31:14 +0000 (UTC)
	(envelope-from jan@chrillesen.dk)
Received: from mailhotel.chrillesen.dk (vax.chrillesen.dk [193.88.12.35])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 8FC4F43D45;
	Mon, 13 Mar 2006 08:31:14 +0000 (GMT)
	(envelope-from jan@chrillesen.dk)
Received: by mailhotel.chrillesen.dk (Postfix, from userid 1000)
	id B56F15B08; Mon, 13 Mar 2006 09:31:10 +0100 (CET)
Date: Mon, 13 Mar 2006 09:31:10 +0100
From: Jan Chrillesen <jan@chrillesen.dk>
To: hubs@FreeBSD.org
Message-ID: <20060313083109.GV17148@vax.chrillesen.dk>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
X-editor: vim, http://www.vim.org/
User-Agent: Mutt/1.5.9i
Cc: beastie@tdk.net
Subject: Disk failure in danish part of ftp.freebsd.org
X-BeenThere: freebsd-hubs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: "FreeBSD Distributions Hubs: mail sup ftp" <freebsd-hubs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hubs>,
	<mailto:freebsd-hubs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hubs>
List-Post: <mailto:freebsd-hubs@freebsd.org>
List-Help: <mailto:freebsd-hubs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hubs>,
	<mailto:freebsd-hubs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 13 Mar 2006 08:31:15 -0000

(I'm sending this mail since erwin@freebsd.org is on vacation).

It seems the European side of ftp.freebsd.org suffored a double disk
failure during the weekend and this means we lost all data on the
mirror.

At first only disk was showing up as missing but the controller reported
the RAID failed. We replaced the disk, and then a second disk showed up
as missing. We're currently replacing the second disk.

However this leaves us with no other options than building the RAID from
scratch, doing a newfs and resync all data from ftp-master. 

I'll keep ftpd disabled until we're back in sync.

We are continously monitoring the state of the RAID (using nagios) but
got no warning of a failed disk, so I fear the first disk might have
failed silently since two disk failures in 5 minutes (time between
nagios polls) is highly unlikely.

Regards
Jan