From owner-freebsd-fs@FreeBSD.ORG Tue Oct 23 04:06:59 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1F268DA0 for ; Tue, 23 Oct 2012 04:06:59 +0000 (UTC) (envelope-from jurgen.weber@theiconic.com.au) Received: from exprod6og114.obsmtp.com (exprod6og114.obsmtp.com [64.18.1.33]) by mx1.freebsd.org (Postfix) with SMTP id 7F4AD8FC17 for ; Tue, 23 Oct 2012 04:06:57 +0000 (UTC) Received: from mail-da0-f72.google.com ([209.85.210.72]) (using TLSv1) by exprod6ob114.postini.com ([64.18.5.12]) with SMTP ID DSNKUIYX274cQGULBIAr+YM53BTb7SF2LiLw@postini.com; Mon, 22 Oct 2012 21:06:58 PDT Received: by mail-da0-f72.google.com with SMTP id r28so5915769daj.7 for ; Mon, 22 Oct 2012 21:06:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:x-gm-message-state; bh=ioSTgOHdiEQrcj4UTGeBXZCV0B+hVPQGkDkbsV5PAfw=; b=QVC+uW6gwS+lHPnZepBNzLXjJr9PGBHw8HvDyrs623jZWJJlIFqxyQlsoS7SRGBEIN yqFyIXaHsos/zm8tEx+AKl+3FsbPYL/8o3IRiFD3i81n/ZeMDBGynVbICfSm1fcpn4EG kExdQIoQtYZpZaQQ/NRrS+7Ol3Ma4o3bDhLf/7KbkpV9WtxufDFkVpWY4I5wPzse0Ji7 dM3Z2HCN1OsFolBKt6ecXRp4wSVu0ZUXpJN/K6lUSTbwo2Yw4JS2ab/lkyt92HfGWH5/ dNEPZ4G0fIhKtcBNNU6dP7/3dW4soRnnybI0ZniAmL7rdhZqA6xJHQk2D3w0Igfhojqk 78bQ== Received: by 10.66.87.132 with SMTP id ay4mr1685424pab.67.1350964902476; Mon, 22 Oct 2012 21:01:42 -0700 (PDT) Received: by 10.66.87.132 with SMTP id ay4mr1685410pab.67.1350964902333; Mon, 22 Oct 2012 21:01:42 -0700 (PDT) Received: from [172.20.24.157] ([202.126.107.170]) by mx.google.com with ESMTPS id sa2sm6993493pbc.4.2012.10.22.21.01.40 (version=SSLv3 cipher=OTHER); Mon, 22 Oct 2012 21:01:41 -0700 (PDT) Message-ID: <508616A2.60609@theiconic.com.au> Date: Tue, 23 Oct 2012 15:01:38 +1100 From: =?ISO-8859-1?Q?J=FCrgen_Weber?= User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: mfi0 timeout error zfs boot mount problem References: <508090E8.4010300@theiconic.com.au> <5081CE05.1010108@theiconic.com.au> <50830EA3.6020001@theiconic.com.au> <508471E0.9010805@theiconic.com.au> <5084A19A.5050905@theiconic.com.au> In-Reply-To: <5084A19A.5050905@theiconic.com.au> X-Gm-Message-State: ALoCoQm7WZ3KQxiW18qiy4oakG4a69sQndFrhrqgPwcer3RFkagTAD7ebtPAZPPUIx60n9BSsk2xzE+lk5wmRZGv7RXIURvDJ3glAK+IDsJg8ej3Rkh6UFh6f2iLCC23tUAueFDRQogpvHnmN9CmGI5vSEg20MidkQ== Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 04:06:59 -0000 Hi Well, while this thread has been very quiet I have resolved my issues. With patience changing: kern.maxfiles=5000000 kern.maxvnodes=5000000 vfs.zfs.zil_disable="1" vfs.zfs.prefetch_disable="1" vfs.zfs.txg.timeout="5" The above solves the system unable to import or mount the pool. I have also gone into the Card settings BIOS and changed under advanced settings "Forward Read" to "none". This solves the mfi0 timeout. Once I had the system up, I then added a l2arc cache via a usb2 SSD HDD. I then shut hte system down and it took 3 hours to shut down.. but it eventually did. When I turned the system back on again, it booted as normal. The lesson learnt?! Do not turn on deduping on a large file system unless you have a lot of RAM or L2ARC! I would say 32GB of RAM/L2ARC for every 10TB as a good rule of thumb, if not... double. Thanks Jurgen On 22/10/12 12:30, Jürgen Weber wrote: > Some more updates! > > on the bootloader I have also tried: > kern.maxfiles=5000000 > kern.maxvnodes=5000000 > > I have also gone into the Card settings BIOS and changed under > advanced settings "Forward Read" to "none". > > Now the systems gets to > > "Trying to mount root from zfs:tank/root []..... " and then after > maybe 1 to 5 minutes the next couple of lines load like its working! > > eg: > "Setting hostuuid: xxxxx" > "Setting hostid: xxxxxx" > "Entropy harvesting:interrupts ethernet point_to_point kickstart" > "Starting file system checks:" > "Mounting local file systems:." > > and stops. I have had the machine on my desk all morning observing it > and I can see the disk access is going crazy,, it is doing something. > > I have found this article: > > http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe > > I have a 15TB file system which has dedup on from the start (10TB. I > feel its trying to load the DDT and its going to swap/there is not > enough RAM (only have 16GB's). Hopefully my 64GB RAM upgrade is enough. > > Thanks > > Jurgen > > > > On 22/10/12 09:06, Jürgen Weber wrote: >> This is still a problem for me, is anyone there? :) >> >> I have tried the following at the bootime loader. >> >> vfs.zfs.zil_disable="1" >> vfs.zfs.prefetch_disable="1" >> vfs.zfs.txg.timeout="5" >> >> Any other suggestions on how to get this zpool to import and mount >> again? >> >> Thanks >> >> On 21/10/12 07:50, Jurgen Weber wrote: >>> Hi >>> >>> Lastly, is there a way at boot time, some sysctl's or something I >>> can set to bring zfs to a minimalistic state? Turn off features, etc >>> to get this to mount? >>> >>> Any ideas appreciated. >>> >>> Thanks >>> >>> Jurgen >>> On 20/10/2012 9:02 AM, Jurgen Weber wrote: >>>> Guys >>>> >>>> Some more details on this, some insight would be greatly appreciated. >>>> >>>> As my day wore on trying to get this zpool to import or mount I >>>> have learnt a few things. I think over time this issue has came >>>> about as more and more data was added to the file systems. >>>> >>>> Some further details: >>>> >>>> Its a 8 disk raidz pool that the system boots from as well. The >>>> disk are all 2TB. >>>> The server has 16GB Of RAM, I notcied the day before this happen >>>> the server was struggling with its RAM griding to a halt and >>>> dumping its RAM. >>>> The issue is not hardware because I found another server (same one) >>>> swapped the harddrives out took another 8GB of RAM and I have the >>>> same problem. >>>> The main data file systems have dedup and gzip compression on. >>>> >>>> I have booted from open/Oracle Solars 11 adn attempted to import >>>> and the Solaris live CD will not import either. In the Solaris >>>> system the disk detach from the system. >>>> >>>> I get the feeling that ZFS is hitting some root limit when >>>> attempting to mount and its not finishing the job. >>>> >>>> Thanks >>>> >>>> Jurgen >>>> >>>> On 19/10/2012 10:29 AM, Jürgen Weber wrote: >>>>> Team >>>>> >>>>> I have googled around for a solution and I see a lot of posts >>>>> about firmware versions and patches for FreeBSD 8.*. >>>>> >>>>> I have a FreeBSD 9.1rc1 system, which was beta1 orginally and has >>>>> been running for months. >>>>> >>>>> Now it will not boot, I get the following: >>>>> >>>>> "Trying to mount root from zfs:tank/root []..... >>>>> mfi0: COMMAND 0Xffffff8000cb83530 TIMEOUT AFTER xxx SECONDS >>>>> (this just repeats). >>>>> >>>>> I have not seen this error before during normal runtime, _only_ >>>>> during boot. >>>>> >>>>> Originally when I had the problem I could boot off a USB stick >>>>> (9.1beta1 or rc1), run a 'zpool import -f tank' and it would work >>>>> on the livecd. Rebooting and the main system would work. >>>>> >>>>> This time this work around does not work for me. When I am on the >>>>> USB stick I can run a 'zpool import' and all of the disk are >>>>> recognised, the pool is recognised and the file system is healthy. >>>>> >>>>> The Card is a H700 PERC, with 12.10.3 firmware in a Dell R515. >>>>> Running FreeBSD 9.1-RC1, latest zfs and zpool versions. >>>>> >>>>> I have tried disabling the cache (mfiutil cache xxx disable). I >>>>> have also gone into the Card settings and changed under advanced >>>>> settings "adaptive forward read" to "read only". >>>>> >>>>> Any help, appreciated. >>>>> >>>>> Thanks >>>>> >>>> >>> >> > > -- > Jürgen Weber > > Systems Engineer > IT Infrastructure Team Leader > > THE ICONIC | Ejurgen.weber@theiconic.com.au |www.theiconic.com.au -- Jürgen Weber Systems Engineer IT Infrastructure Team Leader THE ICONIC | E jurgen.weber@theiconic.com.au | www.theiconic.com.au