Date: Mon, 30 Apr 2012 14:39:22 -0400 From: Alejandro Imass <ait@p2ee.org> To: jb <jb.1234abcd@gmail.com> Cc: freebsd-questions@freebsd.org Subject: Re: UFS Crash and directories now missing Message-ID: <CAHieY7QX6VGPwbfFHXK2n5aiiAOV8yoHbz95tcJttGX0JjgZFg@mail.gmail.com> In-Reply-To: <loom.20120430T194831-917@post.gmane.org> References: <201204301136.q3UBa8fj083478@mail.r-bonomi.com> <CAHieY7RyFZsDpwsaADrbgKes=i0GhCHmTNeprY-qn6SmAmANwg@mail.gmail.com> <4F9EC5E9.6060604@gmail.com> <CAHieY7QYmQKaXyCYfG2Jnc3eTu59BDAnSGgjwbiEv9ReUQWtQg@mail.gmail.com> <loom.20120430T194831-917@post.gmane.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Apr 30, 2012 at 1:57 PM, jb <jb.1234abcd@gmail.com> wrote: > Alejandro Imass <ait <at> p2ee.org> writes: > >>... >> If you have really followed the thread, all I have done is try to find >> some explanation for a strange behavior of the system under normal >> use. It hung, and some directories were moved, period. I have posted >> some ideas to share with other people expecting some insight and maybe >> similar experience from other users, which there probably are many, >> but many times afraid to speak up and avoid getting insulted. >> ... > > > I looked at problem reports for nullfs and there are quite few. > Hierarchical Jails > NOTES > > You said you have your jail env on a separate disk. > Yes. > I looked at problem reports for nullfs and there are quite few. > http://www.freebsd.org/cgi/query-pr-summary.cgi?category=3D&severity=3D&p= riority=3D&cl > ass=3D&state=3D&sort=3Dnone&text=3Dnullfs&responsible=3D&multitext=3D&ori= ginator=3D&release=3D > > As a matter of fact I just mounted a nullfs but was not able to unmount i= t > (device busy) - a Google search shows it as a problem reported for many m= any > years. > Nullfs does not seem to be stable. > Dirk Engling guessed that somehow nullfs was involved. > Anyway, I found one PR > http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/147420 > > that is about troubles with jails, nullfs, UFS, and NFS. > Synopsis: =A0 =A0 =A0 [ufs] [panic] ufs_dirbad, nullfs, jail panic (corru= pt inode) > > Take a look at this paragraphs: > "... > After two more failures, I now found the offending inode ..." > "... > As one point, I found the inode in a directory which usually is mounted f= or > an (ez-) jail via nullfs." > > This proves that problems with jails, nullfs, and fs corruption are possi= ble. > So, they can not be excluded up front in your case too because nullfs is = just > a simple "path translation". > Up until yesterday (and Dirk's answer) I didn't look for specific references to nullfs, and today I was busy getting vicious myself ;) Thanks for pointing a plausible cause. What I have done so far is limit the offending jail to a specific cpuset and I wanted to add another disk to avoid contention with other jails. MySQL not only consumes the whole CPUs but also limits the whole drive, while it's doing some crazy full scan query on a very large database. I don't have any control of the code or the MySQL myself and the client said it's known problem with VTiger CRM and the way it implements some reports that I guess were not designed for the amount of data they are handling. I have already recommended they move to a dedicated server altogether because their system simply outgrew what we sold them. I really appreciate the time you dedicated to search for a possible explanation and at the very least it helps in taking some immediate steps to avoid it from happening again. Hopefully, someone with deep knowledge will find the root cause and a long-term fix. What is true, that if it happened to me, it can happen to anyone, so maybe your findings will help someone pin-point the problem and fix it. Thanks, --=20 Alejandro
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHieY7QX6VGPwbfFHXK2n5aiiAOV8yoHbz95tcJttGX0JjgZFg>