From owner-freebsd-stable@FreeBSD.ORG Fri Jun 14 11:00:57 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 726CE80B; Fri, 14 Jun 2013 11:00:57 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-lb0-f176.google.com (mail-lb0-f176.google.com [209.85.217.176]) by mx1.freebsd.org (Postfix) with ESMTP id C181F10D3; Fri, 14 Jun 2013 11:00:56 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id z5so453531lbh.35 for ; Fri, 14 Jun 2013 04:00:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=jCKtUJQG1oIn/J1GPhD2GxSSIaMiTzdi43gmUk5mmOY=; b=DWPhNWoUPjDVfYX0tXEHzWpA5lOaKcJI+c1KxE13qJjq8tjV9NNYvHn7iiITkv06ut u/aNFyxss0EZ0BRdzLivS7YDdZx+pzHKVoBuJ+ecgZBVTgR0dYW1kuwhhz7pAEqnLC/Z pHYIHJ4VARUVXQPgZAyKIeu/N+YxHwYBxOp1gndK5/CXy3lrlusPZqRmmUeD3pWmLCMn siWATlGRlNu6lm/zQ5Wvmo8TPiwRAXPHT4Gbv18funeRvwiRT1D4kl/zXMug/rNDb7iE o0DdJuiXJNekNFljpKEHbXCjEkYoab+u9iZFsoYBRe/a2hxZVLa+R6q9dL9UAyeT5KhR wrRQ== X-Received: by 10.152.20.66 with SMTP id l2mr920749lae.30.1371207655326; Fri, 14 Jun 2013 04:00:55 -0700 (PDT) Received: from [192.168.1.125] (mau.donbass.com. [92.242.127.250]) by mx.google.com with ESMTPSA id p20sm663301lbb.17.2013.06.14.04.00.52 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 14 Jun 2013 04:00:54 -0700 (PDT) Message-ID: <51BAF7E3.4020401@gmail.com> Date: Fri, 14 Jun 2013 14:00:51 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:20.0) Gecko/20100101 Firefox/20.0 SeaMonkey/2.17.1 MIME-Version: 1.0 To: Dr Josef Karthauser , fs@freebsd.org Subject: Re: Help! :( ZFS panic on boot, importing pool after server crash. References: <301B4131-F677-4B8D-ABF6-A6D269FE604E@gmail.com> In-Reply-To: <301B4131-F677-4B8D-ABF6-A6D269FE604E@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Jun 2013 11:00:57 -0000 14.06.2013 12:55, Dr Josef Karthauser: > Hi, I'm a bit at the end of my tether. > > We had a ZFS panic last night on a machine that hosts all my mail and web; it was rebooted and it now panics mounting the ZFS root filesystem. > > The call stack info is: > > solaris assert: ss == NULL, file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensource/uts/common/fs/zfs/space_map.c, line: 109 > > kdb_backtrace > panic > space_map_add > space_map_load > metaslab_activate > metaslab_allocate > zio_dva_allocate > zio_execute > taskqueue_run_locked > taskqueue_thread_loop > fork_exit > fork_trampoline > > I can boot from the live DVD filesystem, but I can only mount the pool read-only without getting the same kernel panic. This is with FreeBSD 9.0. > > The machine is remote, and I don't have access other than through a DRAC console port (so I can't cut and paste; sorry for the poor stack trace). > > Is anyone here in the position to advice me how I might process to get this machine mounting and running again in multi-user mode? There's no official way. > p.s. the config, btw, is a ZFS mirror on two ad devices. It's got a ZFS root file system. If you are fairly sure about your devices you can: 1. Remove second disk from pool or create another pool on top of it. 2. Recreate all FS structure on the second disk. You can dump al your FS with something like: zfs list -Ho name | xargs -n1 zfs get -H all | awk 'BEGIN{shard="";output=""}{if(shard!=$1 && shard!=""){output="zfs create";for(param in params)output=output" -o "param"="params[param];print output" "shard;delete params;shard=""}}$4~/local/{params[$2]=$3;shard=$1;next}$2~/type/{shard=$1}END{output="zfs create";for(param in params)output=output" -o "param"="params[param];print output" "shard;}' Be sure to rename the pool and change the first line. 3. Rsync all data to the second disk. 4. Try to boot from the second disk. If everything worked you are free to attach first disk to second one to create a mirror again. -- Sphinx of black quartz, judge my vow.