Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 23 Oct 2012 15:01:38 +1100
From:      =?ISO-8859-1?Q?J=FCrgen_Weber?= <jurgen.weber@theiconic.com.au>
To:        freebsd-fs@freebsd.org
Subject:   Re: mfi0 timeout error zfs boot mount problem
Message-ID:  <508616A2.60609@theiconic.com.au>
In-Reply-To: <5084A19A.5050905@theiconic.com.au>
References:  <508090E8.4010300@theiconic.com.au> <5081CE05.1010108@theiconic.com.au> <50830EA3.6020001@theiconic.com.au> <508471E0.9010805@theiconic.com.au> <5084A19A.5050905@theiconic.com.au>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi

Well, while this thread has been very quiet I have resolved my issues. 
With patience changing:

kern.maxfiles=5000000
kern.maxvnodes=5000000
vfs.zfs.zil_disable="1"
vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="5"
The above solves the system unable to import or mount the pool.

I have also gone into the Card settings BIOS and changed under advanced 
settings "Forward Read" to "none". This solves the mfi0 timeout.

Once I had the system up, I then added a l2arc cache via a usb2 SSD HDD.

I then shut hte system down and it took 3 hours to shut down.. but it 
eventually did.

When I turned the system back on again, it booted as normal.

The lesson learnt?! Do not turn on deduping on a large file system 
unless you have a lot of RAM or L2ARC! I would say 32GB of RAM/L2ARC for 
every 10TB as a good rule of thumb, if not... double.

Thanks

Jurgen
On 22/10/12 12:30, Jürgen Weber wrote:
> Some more updates!
>
> on the bootloader I have also tried:
> kern.maxfiles=5000000
> kern.maxvnodes=5000000
>
> I have also gone into the Card settings BIOS and changed under 
> advanced settings "Forward Read" to "none".
>
> Now the systems gets to
>
> "Trying to mount root from zfs:tank/root []..... " and then after 
> maybe 1 to 5 minutes the next couple of lines load like its working!
>
> eg:
> "Setting hostuuid: xxxxx"
> "Setting hostid: xxxxxx"
> "Entropy harvesting:interrupts ethernet point_to_point kickstart"
> "Starting file system checks:"
> "Mounting local file systems:."
>
> and stops. I have had the machine on my desk all morning observing it 
> and I can see the disk access is going crazy,, it is doing something.
>
> I have found this article:
>
> http://constantin.glez.de/blog/2011/07/zfs-dedupe-or-not-dedupe
>
> I have a 15TB file system which has dedup on from the start (10TB. I 
> feel its trying to load the DDT and its going to swap/there is not 
> enough RAM (only have 16GB's). Hopefully my 64GB RAM upgrade is enough.
>
> Thanks
>
> Jurgen
>
>
>
> On 22/10/12 09:06, Jürgen Weber wrote:
>> This is still a problem for me, is anyone there? :)
>>
>> I have tried the following at the bootime loader.
>>
>> vfs.zfs.zil_disable="1"
>> vfs.zfs.prefetch_disable="1"
>> vfs.zfs.txg.timeout="5"
>>
>> Any other suggestions on how to get this zpool to import and mount 
>> again?
>>
>> Thanks
>>
>> On 21/10/12 07:50, Jurgen Weber wrote:
>>> Hi
>>>
>>> Lastly, is there a way at boot time, some sysctl's or something I 
>>> can set to bring zfs to a minimalistic state? Turn off features, etc 
>>> to get this to mount?
>>>
>>> Any ideas appreciated.
>>>
>>> Thanks
>>>
>>> Jurgen
>>> On 20/10/2012 9:02 AM, Jurgen Weber wrote:
>>>> Guys
>>>>
>>>> Some more details on this, some insight would be greatly appreciated.
>>>>
>>>> As my day wore on trying to get this zpool to import or mount I 
>>>> have learnt a few things. I think over time this issue has came 
>>>> about as more and more data was added to the file systems.
>>>>
>>>> Some further details:
>>>>
>>>> Its a 8 disk raidz pool that the system boots from as well. The 
>>>> disk are all 2TB.
>>>> The server has 16GB Of RAM, I notcied the day before this happen 
>>>> the server was struggling with its RAM griding to a halt and 
>>>> dumping its RAM.
>>>> The issue is not hardware because I found another server (same one) 
>>>> swapped the harddrives out took another 8GB of RAM and I have the 
>>>> same problem.
>>>> The main data file systems have dedup and gzip compression on.
>>>>
>>>> I have booted from open/Oracle Solars 11 adn attempted to import 
>>>> and the Solaris live CD will not import either. In the Solaris 
>>>> system the disk detach from the system.
>>>>
>>>> I get the feeling that ZFS is hitting some root limit when 
>>>> attempting to mount and its not finishing the job.
>>>>
>>>> Thanks
>>>>
>>>> Jurgen
>>>>
>>>> On 19/10/2012 10:29 AM, Jürgen Weber wrote:
>>>>> Team
>>>>>
>>>>> I have googled around for a solution and I see a lot of posts 
>>>>> about firmware versions and patches for FreeBSD 8.*.
>>>>>
>>>>> I have a FreeBSD 9.1rc1 system, which was beta1 orginally and has 
>>>>> been running for months.
>>>>>
>>>>> Now it will not boot, I get the following:
>>>>>
>>>>> "Trying to mount root from zfs:tank/root [].....
>>>>> mfi0: COMMAND 0Xffffff8000cb83530 TIMEOUT AFTER xxx SECONDS
>>>>> (this just repeats).
>>>>>
>>>>> I have not seen this error before during normal runtime, _only_ 
>>>>> during boot.
>>>>>
>>>>> Originally when I had the problem I could boot off a USB stick 
>>>>> (9.1beta1 or rc1), run a 'zpool import -f tank' and it would work 
>>>>> on the livecd. Rebooting and the main system would work.
>>>>>
>>>>> This time this work around does not work for me. When I am on the 
>>>>> USB stick I can run a 'zpool import' and all of the disk are 
>>>>> recognised, the pool is recognised and the file system is healthy.
>>>>>
>>>>> The Card is a H700 PERC, with 12.10.3 firmware in a Dell R515.
>>>>> Running FreeBSD 9.1-RC1, latest zfs and zpool versions.
>>>>>
>>>>> I have tried disabling the cache (mfiutil cache xxx disable). I 
>>>>> have also gone into the Card settings and changed under advanced 
>>>>> settings "adaptive forward read" to "read only".
>>>>>
>>>>> Any help, appreciated.
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>
>>
>
> -- 
> Jürgen Weber
>
> Systems Engineer
> IT Infrastructure Team Leader
>
> THE ICONIC | Ejurgen.weber@theiconic.com.au  |www.theiconic.com.au

-- 
Jürgen Weber

Systems Engineer
IT Infrastructure Team Leader

THE ICONIC | E jurgen.weber@theiconic.com.au | www.theiconic.com.au




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?508616A2.60609>